Monotasks: Architecting for Performance Clarity in Data Analytics Frameworks

Authors: Kay Outerhout, Christopher Canel, Sylvia Ratnasamy, Scott Shenker
Venue: SOSP 2017

This paper presents a data analytics framework which focuses or providing performance clarity first. Consider a user running there analytics framework on EC2. They need to improve performance. Do they invest in more vCPUs, more memory, more disks per node or more network bandwidth? If they do upgrade, what will the expected performance improvement be? To solve this, Monotasks centralizes on building a framework which decomposes all tasks into single-resource utilization: disk use, network I/O, or CPU.

The framework is integrated into Spark, and referred to as "MonoSpark". By simply decomposing tasks at the worker level, the existing Spark API is maintained. By dividing tasks into individual units of disk/network/CPU, the framework can track the precise total amount of work for each, and then determine the performance changes based on the new resource constraints. Most queries are with 9% of the predicted performance. One limitation MonoSpark has, is that when you change the number of machines, the amount of network communication changes due to data spread. As a result, MonoSpark is less accurate in this scenario.

The system is mainly targets at achieving predictable performance and scaling, but the paper claims that the system is within 9% of Spark performance. It is also pointed out that the paper uses a number of simple implementations which could be improved, such as scheduling network tasks, queuing monotasks, and managing concurrency. In terms of replacing Spark with MonoSpark practically, the biggest pitfall is increased disk writes. Spark uses fine-grained pipelining, which interleaves compute and I/O operations, which not only boosts performance but allow for operation on tasks that are larger than what fits into the host's memory. MonoSpark separates tasks into coarse-grained operations, breaking the built-in pipeline. The result is increased disk writes, particularly with big data benchmark -- arguably the most relevant to the context of Spark deployments. This would not only wear disks faster, but reduce performance and potentially cause OOMs. This isn't to say MonoTasks can't increase performance, it does when the default would have caused significant contention.

My overall take away is that users should run an instance of MonoSpark to measure bottlenecks and make changes, but use traditional Spark for actual deployment.

Full text

Comments

Post a Comment

Popular posts from this blog

Fundamental Latency Trade-offs in Architecting DRAM Caches (Alloy Cache)

Authors: Moinuddin K. Qureshi and Gabriel H. Loh Venue:    MICRO 2012 The authors present a new architecture for DRAM cache which targets optimizing latency rather than hit rate. The authors show that for every optimization made to improve hit rate, one must also consider the added cache latency, thus resulting in a new "break even hit rate". The authors compare to three key baselines: an infeasibly large SRAM-tag array, the prior "LH-Cache", and a ideal latency-optimized cache. They find that "unoptimizations" improve latency, while minimally impacting hit-rate. Their final result is a cache which boosts performance 35% improvement over the baseline, and nearly 20% better than the previous LH-cache. The specific contributions are: Moving to a direct mapped cache provides significant improvement (by reducing latency) Combining tag-store and data-store into a single entity improves performance 21% Implementing a small per-core predictor enhances per...
Read more

ZCOMP: Reducing DNN Cross-Layer Memory Footprint Using Vector Extensions

Authors: Berkin Akin, Zeshan A. Christi, and Alaa R. Alameldeen Venue: MICRO, 2019 With accelerators dominating the deep learning space in architecture conferences, this paper stands out as it focuses on reducing DNN inference/training overhead while using a CPU. An obvious question is, with the wide-spread use of GPUs/TPUs for deep learning, why should we focus on optimizing CPUs for deep learning. In a recent paper published at HPCA 2019, Facebook claims that CPUs are preferred for applications where a tight integration is required between DNN and non-DNN tasks. Also, Intel's recent AVX512 has specialized support for DNNs in the form of new instructions called Vector Neural Network Instructions (VNNI). Broadly, optimizing DNNs can be viewed from two different perspectives; computation, and communication. This paper targets reducing communication overhead, more specifically, activation or feature-map communication overhead by compressing them. Compressing activations/weights...
Read more

AutoFDO: Automatic Feedback-Directed Optimization for Warehouse-Scale Applications

Authors: Dehao Chen, David Xinliang Li, Tipp Moseley Venue:    CGO 2016 This paper presents AutoFDO, a system used which profiles warehouse-scale applications, and applies feedback to the compilation for the next release. AutoFDO works by profiling and storing profiles in an aggregate database, annotating the profiles via an intermediate representation, and finally providing feedback. On average, this technique boosts performance by around 10% and works well even with stale releases. For profiles to be useful, they must be in an intermediate representation. This is built from binary-level profile, which uses LBR to map instruction frequencies. By using these frequencies, program counters, and the source, a source profile can be generated. The source profile can be used to build an annotated call-graph with edge frequencies, with some inaccuracy. These annotated call-graphs can then be used as feedback to the compiler. Because of the size of the applications and scale of d...
Read more