Today, the interface between a high speed network and a high performance computation node is the ... more Today, the interface between a high speed network and a high performance computation node is the least mature hardware technology in scalable general purpose cluster computing. Currently, the one-interface-fits-all philosophy prevails. This approach performs poorly in some cases because of the complexity of modern memory hierarchy and the wide range of communication sizes and patterns. Today's message passing NIU's are also unable to utilize the best data transfer and coordination mechanisms due to poor integration into the computation node's memory hierarchy. These shortcomings unnecessarily constrain the performance of cluster systems. Our thesis is that a cluster system NIU should support multiple communication interfaces layered on a virtual message queue substrate in order to streamline data movement both within each node as well as between nodes. The NIU should be tightly integrated into the computation node's memory hierarchy via the cachecoherent snoopy system bus so as to gain access to a rich set of data movement operations. We further propose to achieve the goal of a large set of high performance communication functions with a hybrid NIU micro-architecture that combines custom hardware building blocks with an off-the-shelf embedded processor. These ideas are tested through the design and implementation of the StarT-Voyager NES, an NIU used to connect a cluster of commercial PowerPC based SMP's. Our prototype demonstrates that it is feasible to implement a multi-interface NIU at reasonable hardware cost. This is achieved by reusing a set of basic hardware building blocks and adopting a layered architecture that separates protected network sharing from software visible communication interfaces. Through different mechanisms, our 35MHz NIU (140MHz processor core) can deliver very low latency for very short messages (under 2ps), very high bandwidth for multi-kilobyte block transfers (167 MBytes/s bi-directional bandwidth), and very low processor overhead for multi-cast communication (each additional destination after the first incurs 10 processor clocks). We introduce the novel idea of supporting a large number of virtual message queues through a combination of hardware Resident message queues and firmware emulated Non-resident message queues. By using the Resident queues as firmware This dissertation would not have been possible without the encouragement, support, patience and cooperation of many people. Although no words can adequately express my gratitude, an acknowledgement is the least I can do. First and foremost, I want to thank my wife, Wee Lee, and our families for standing by me all these years. They gave me the latitude to seek my calling, were patient as the years passed but I was no closer to enlightenment, and provided me a sanctuary to retreat to whenever my marathon-like graduate school career wore me thin. To you all, my eternity gratitude. I am greatly indebted to my advisors, Arvind and Larry, for their faith in my abilities and for standing by me throughout my long graduate student career. They gave me the opportunity to co-lead a large systems project, an experience which greatly enriched my systems building skills. To Larry, I want to express my gratitude for all the fatherly/brotherly advice and the cheering sessions in the last leg of my graduate school apprenticeship. I would also like to thank the other members of my thesis committee: Frans and Anant, for helping to refine this work. I want to thank Derek Chiou for our partnership through graduate school, working together on Monsoon, StarT, StarT-NG and StarT-Voyager. I greatly enjoy bringing vague ideas to you, and jointly developing them into well thought out solutions. This work on StarT-Voyager NES is as much yours as it is mine. Thank you too for the encouragement and counselling you gave me all these years. The graduate students and staff in Computation Structures Group gave me a home away from home.
International Colloquium on Automata, Languages and Programming, Jul 11, 1988
Theoretical research on parallel algorithms has focused on NC theory. This motivates the developm... more Theoretical research on parallel algorithms has focused on NC theory. This motivates the development of parallel algorithms that are extremely fast, but possibly wasteful in their use of processors. Such algorithms seem of limited interest for real applications currently run on parallel computers. This paper explores an alternative approach that emphasizes the efficiency of parallel algorithms. We define a complexity
International Conference on Learning Representations, Apr 30, 2020
We study how the behavior of deep policy gradient algorithms reflects the conceptual framework mo... more We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a finegrained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "true" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods. * Equal contribution. Work done in part as an intern at Two Sigma.
International Conference on Parallel Processing, 1985
Abstract The prefix computation problem is to compute all n initial products a 1∘…∘ ai, i= 1,…, n... more Abstract The prefix computation problem is to compute all n initial products a 1∘…∘ ai, i= 1,…, n, of a set of n elements, where∘ is an associative operation. We present an 0 (((log n)/log (2n/p))·(n/p)) time deterministic parallel algorithm using p≤ n processors to solve the prefix computation problem, when the order of the elements is specified by a linked list. For p≤ 0 (n 1-∊)(∊> 0 any constant), this algorithm achieves linear speedup. Such optimal speedup was previously achieved only by probabilistic algorithms. Our algorithms assume ...
Distributed transactions suffer from poor performance due to two major limiting factors. First, d... more Distributed transactions suffer from poor performance due to two major limiting factors. First, distributed transactions suffer from high latency because each of their accesses to remote data incurs a long network delay. Second, this high latency increases the likelihood of contention among distributed transactions, leading to high abort rates and low performance. We present Sundial, an in-memory distributed optimistic concurrency control protocol that addresses these two limitations. First, to reduce the transaction abort rate, Sundial dynamically determines the logical order among transactions at runtime, based on their data access patterns. Sundial achieves this by applying logical leases to each data element, which allows the database to dynamically calculate a transaction's logical commit timestamp. Second, to reduce the overhead of remote data accesses, Sundial allows the database to cache remote data in a server's local main memory and maintains cache coherence. With logical leases, Sundial integrates concurrency control and cache coherence into a simple unified protocol. We evaluate Sundial against state-of-the-art distributed concurrency control protocols. Sundial outperforms the next-best protocol by up to 57% under high contention. Sundial's caching scheme improves performance by up to 4.6× in workloads with high access skew.
2019 IEEE International Conference on Big Data (Big Data), 2019
Current caching methods for improving the performance of big-data jobs assume high (e.g., full bi... more Current caching methods for improving the performance of big-data jobs assume high (e.g., full bi-section) bandwidth; however many enterprise data centers and co-location facilities have large network imbalances due to over-subscription and incremental networking upgrades. We describe D3N, a multi-layer cooperative caching architecture that mitigates network imbalances by caching data on the access side of each layer of a hierarchical network topology, adaptively adjusting cache sizes of each layer based on observed workload patterns and network congestion. We have added (and submitted upstream) a 2-layer D3N cache to the Ceph RADOS Gateway; read bandwidth achieves the 5GB/s speed of our SSDs, and we show that it substantially improves big-data job performance while reducing network traffic.
Kariz is a new architecture for caching data from datalakes accessed, potentially concurrently, b... more Kariz is a new architecture for caching data from datalakes accessed, potentially concurrently, by multiple analytic platforms. It integrates rich information from analytics platforms with global knowledge about demand and resource availability to enable sophisticated cache management and prefetching strategies that, for example, combine historical run time information with job dependency graphs (DAGs), information about the cache state and sharing across compute clusters. Our prototype supports multiple analytic frameworks (Pig/Hadoop and Spark), and we show that the required changes aremodest.We have implemented three algorithms in Kariz for optimizing the caching of individual queries (one from the literature, and two novel to our platform) and three policies for optimizing across queries from, potentially, multiple di erent clusters.With an algorithm that fully exploits the rich information available from Kariz, we demonstrate major speedups (as much as 3×) for TPC-H and TPC-DS.
International Journal for Multiscale Computational Engineering, 2020
We write to introduce our novel group formed to confront some of the issues raised by the COVID-1... more We write to introduce our novel group formed to confront some of the issues raised by the COVID-19 pandemic. Information about the group, which we named "cure COVid for Ever and for All" (RxCOVEA), its dynamic membership (changing regularly), and some of its activities-described in more technical detail for expert perusal and commentary-are available upon request.
Parallel Job Scheduling - A Status Report.- Scheduling on the Top 50 Machines.- Parallel Computer... more Parallel Job Scheduling - A Status Report.- Scheduling on the Top 50 Machines.- Parallel Computer Workload Modeling with Markov Chains.- Enhancements to the Decision Process of the Self-Tuning dynP Scheduler.- Reconfigurable Gang Scheduling Algorithm.- Time-Critical Scheduling on a Well Utilised HPC System at ECMWF Using Loadleveler with Resource Reservation.- Inferring the Topology and Traffic Load of Parallel Programs Running in a Virtual Machine Environment.- Multi-toroidal Interconnects: Using Additional Communication Links to Improve Utilization of Parallel Computers.- Costs and Benefits of Load Sharing in the Computational Grid.- Workload Characteristics of a Multi-cluster Supercomputer.- A Dynamic Co-allocation Service in Multicluster Systems.- Exploiting Replication and Data Reuse to Efficiently Schedule Data-Intensive Applications on Grids.- Performance Implications of Failures in Large-Scale Cluster Scheduling.- Are User Runtime Estimates Inherently Inaccurate?.- Improving Speedup and Response Times by Replicating Parallel Programs on a SNOW.- LOMARC - Lookahead Matchmaking for Multi-resource Coscheduling.
Today, the interface between a high speed network and a high performance computation node is the ... more Today, the interface between a high speed network and a high performance computation node is the least mature hardware technology in scalable general purpose cluster computing. Currently, the one-interface-fits-all philosophy prevails. This approach performs poorly in some cases because of the complexity of modern memory hierarchy and the wide range of communication sizes and patterns. Today's message passing NIU's are also unable to utilize the best data transfer and coordination mechanisms due to poor integration into the computation node's memory hierarchy. These shortcomings unnecessarily constrain the performance of cluster systems. Our thesis is that a cluster system NIU should support multiple communication interfaces layered on a virtual message queue substrate in order to streamline data movement both within each node as well as between nodes. The NIU should be tightly integrated into the computation node's memory hierarchy via the cachecoherent snoopy system bus so as to gain access to a rich set of data movement operations. We further propose to achieve the goal of a large set of high performance communication functions with a hybrid NIU micro-architecture that combines custom hardware building blocks with an off-the-shelf embedded processor. These ideas are tested through the design and implementation of the StarT-Voyager NES, an NIU used to connect a cluster of commercial PowerPC based SMP's. Our prototype demonstrates that it is feasible to implement a multi-interface NIU at reasonable hardware cost. This is achieved by reusing a set of basic hardware building blocks and adopting a layered architecture that separates protected network sharing from software visible communication interfaces. Through different mechanisms, our 35MHz NIU (140MHz processor core) can deliver very low latency for very short messages (under 2ps), very high bandwidth for multi-kilobyte block transfers (167 MBytes/s bi-directional bandwidth), and very low processor overhead for multi-cast communication (each additional destination after the first incurs 10 processor clocks). We introduce the novel idea of supporting a large number of virtual message queues through a combination of hardware Resident message queues and firmware emulated Non-resident message queues. By using the Resident queues as firmware This dissertation would not have been possible without the encouragement, support, patience and cooperation of many people. Although no words can adequately express my gratitude, an acknowledgement is the least I can do. First and foremost, I want to thank my wife, Wee Lee, and our families for standing by me all these years. They gave me the latitude to seek my calling, were patient as the years passed but I was no closer to enlightenment, and provided me a sanctuary to retreat to whenever my marathon-like graduate school career wore me thin. To you all, my eternity gratitude. I am greatly indebted to my advisors, Arvind and Larry, for their faith in my abilities and for standing by me throughout my long graduate student career. They gave me the opportunity to co-lead a large systems project, an experience which greatly enriched my systems building skills. To Larry, I want to express my gratitude for all the fatherly/brotherly advice and the cheering sessions in the last leg of my graduate school apprenticeship. I would also like to thank the other members of my thesis committee: Frans and Anant, for helping to refine this work. I want to thank Derek Chiou for our partnership through graduate school, working together on Monsoon, StarT, StarT-NG and StarT-Voyager. I greatly enjoy bringing vague ideas to you, and jointly developing them into well thought out solutions. This work on StarT-Voyager NES is as much yours as it is mine. Thank you too for the encouragement and counselling you gave me all these years. The graduate students and staff in Computation Structures Group gave me a home away from home.
International Colloquium on Automata, Languages and Programming, Jul 11, 1988
Theoretical research on parallel algorithms has focused on NC theory. This motivates the developm... more Theoretical research on parallel algorithms has focused on NC theory. This motivates the development of parallel algorithms that are extremely fast, but possibly wasteful in their use of processors. Such algorithms seem of limited interest for real applications currently run on parallel computers. This paper explores an alternative approach that emphasizes the efficiency of parallel algorithms. We define a complexity
International Conference on Learning Representations, Apr 30, 2020
We study how the behavior of deep policy gradient algorithms reflects the conceptual framework mo... more We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a finegrained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "true" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods. * Equal contribution. Work done in part as an intern at Two Sigma.
International Conference on Parallel Processing, 1985
Abstract The prefix computation problem is to compute all n initial products a 1∘…∘ ai, i= 1,…, n... more Abstract The prefix computation problem is to compute all n initial products a 1∘…∘ ai, i= 1,…, n, of a set of n elements, where∘ is an associative operation. We present an 0 (((log n)/log (2n/p))·(n/p)) time deterministic parallel algorithm using p≤ n processors to solve the prefix computation problem, when the order of the elements is specified by a linked list. For p≤ 0 (n 1-∊)(∊> 0 any constant), this algorithm achieves linear speedup. Such optimal speedup was previously achieved only by probabilistic algorithms. Our algorithms assume ...
Distributed transactions suffer from poor performance due to two major limiting factors. First, d... more Distributed transactions suffer from poor performance due to two major limiting factors. First, distributed transactions suffer from high latency because each of their accesses to remote data incurs a long network delay. Second, this high latency increases the likelihood of contention among distributed transactions, leading to high abort rates and low performance. We present Sundial, an in-memory distributed optimistic concurrency control protocol that addresses these two limitations. First, to reduce the transaction abort rate, Sundial dynamically determines the logical order among transactions at runtime, based on their data access patterns. Sundial achieves this by applying logical leases to each data element, which allows the database to dynamically calculate a transaction's logical commit timestamp. Second, to reduce the overhead of remote data accesses, Sundial allows the database to cache remote data in a server's local main memory and maintains cache coherence. With logical leases, Sundial integrates concurrency control and cache coherence into a simple unified protocol. We evaluate Sundial against state-of-the-art distributed concurrency control protocols. Sundial outperforms the next-best protocol by up to 57% under high contention. Sundial's caching scheme improves performance by up to 4.6× in workloads with high access skew.
2019 IEEE International Conference on Big Data (Big Data), 2019
Current caching methods for improving the performance of big-data jobs assume high (e.g., full bi... more Current caching methods for improving the performance of big-data jobs assume high (e.g., full bi-section) bandwidth; however many enterprise data centers and co-location facilities have large network imbalances due to over-subscription and incremental networking upgrades. We describe D3N, a multi-layer cooperative caching architecture that mitigates network imbalances by caching data on the access side of each layer of a hierarchical network topology, adaptively adjusting cache sizes of each layer based on observed workload patterns and network congestion. We have added (and submitted upstream) a 2-layer D3N cache to the Ceph RADOS Gateway; read bandwidth achieves the 5GB/s speed of our SSDs, and we show that it substantially improves big-data job performance while reducing network traffic.
Kariz is a new architecture for caching data from datalakes accessed, potentially concurrently, b... more Kariz is a new architecture for caching data from datalakes accessed, potentially concurrently, by multiple analytic platforms. It integrates rich information from analytics platforms with global knowledge about demand and resource availability to enable sophisticated cache management and prefetching strategies that, for example, combine historical run time information with job dependency graphs (DAGs), information about the cache state and sharing across compute clusters. Our prototype supports multiple analytic frameworks (Pig/Hadoop and Spark), and we show that the required changes aremodest.We have implemented three algorithms in Kariz for optimizing the caching of individual queries (one from the literature, and two novel to our platform) and three policies for optimizing across queries from, potentially, multiple di erent clusters.With an algorithm that fully exploits the rich information available from Kariz, we demonstrate major speedups (as much as 3×) for TPC-H and TPC-DS.
International Journal for Multiscale Computational Engineering, 2020
We write to introduce our novel group formed to confront some of the issues raised by the COVID-1... more We write to introduce our novel group formed to confront some of the issues raised by the COVID-19 pandemic. Information about the group, which we named "cure COVid for Ever and for All" (RxCOVEA), its dynamic membership (changing regularly), and some of its activities-described in more technical detail for expert perusal and commentary-are available upon request.
Parallel Job Scheduling - A Status Report.- Scheduling on the Top 50 Machines.- Parallel Computer... more Parallel Job Scheduling - A Status Report.- Scheduling on the Top 50 Machines.- Parallel Computer Workload Modeling with Markov Chains.- Enhancements to the Decision Process of the Self-Tuning dynP Scheduler.- Reconfigurable Gang Scheduling Algorithm.- Time-Critical Scheduling on a Well Utilised HPC System at ECMWF Using Loadleveler with Resource Reservation.- Inferring the Topology and Traffic Load of Parallel Programs Running in a Virtual Machine Environment.- Multi-toroidal Interconnects: Using Additional Communication Links to Improve Utilization of Parallel Computers.- Costs and Benefits of Load Sharing in the Computational Grid.- Workload Characteristics of a Multi-cluster Supercomputer.- A Dynamic Co-allocation Service in Multicluster Systems.- Exploiting Replication and Data Reuse to Efficiently Schedule Data-Intensive Applications on Grids.- Performance Implications of Failures in Large-Scale Cluster Scheduling.- Are User Runtime Estimates Inherently Inaccurate?.- Improving Speedup and Response Times by Replicating Parallel Programs on a SNOW.- LOMARC - Lookahead Matchmaking for Multi-resource Coscheduling.
Uploads
Papers by Larry Rudolph