We have built a new framework that enables independent metadata updates within HDF5 files. Our fr... more We have built a new framework that enables independent metadata updates within HDF5 files. Our framework includes the following components: an overlay network that's optimized for 1-to-All and All-to-All communication; root-less/server-less Bcast and AllReduce operations for reduced synchronization; a distributed consensus-making platform; an execution order keeping system, and a progress engine for generic threadless concurrence handling.
H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O per... more H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O performance from various aspects.
Abstract—With the increased scale of systems in use and the need to quickly store and retrieve in... more Abstract—With the increased scale of systems in use and the need to quickly store and retrieve information, key/value stores are becoming an important element in the design of large-scale storage systems. Key/value stores are well known for their simplistic interfaces, persistent nature, and excellent operational efficiency – they are also known as NoSQL databases. This paper presents the design and implementation of a non-volatile hash table (NoVoHT). NoVoHT was designed from the ground up to be lightweight, fast, and dependency-free. Our goal was to create a fast persistent key/value store that could be easily integrated and operated in lightweight Linux OS typically found on today’s supercomputers. We also aimed to develop a system that performed as close as possible to an in-memory hash map, but with the added benefit of being persistent. We also extended the traditional key/value store interface (e.g. insert, lookup, remove) to include a novel operation (e.g. append) that has a...
ZHT is a zero-hop distributed hash table, which has been tuned for the requirements of high-end c... more ZHT is a zero-hop distributed hash table, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales of millions of nodes. ZHT has some important properties, such as being lightweight, dynamically allowing nodes to join and leave, fault tolerant through replications, persistent, scalable, and supporting unconventional operations such as append. ZHT scaled up to 32K-cores with latencies of 1.1ms and 18M operations/sec throughput on IBM Blue Gene/P supercomputer, and 96 nodes on Amazon EC2 cloud with 800ns latency and 1.2M ops/s throughput. In previous work we proved ZHT’s excellent performance and scalability on supercomputers, and in this work we show that it also works great on cloud environment from both performance and cost perspective.
Over the last decade, storage systems have experienced a 10fold increase between their capacity a... more Over the last decade, storage systems have experienced a 10fold increase between their capacity and bandwidth. This gap is predicted to grow faster with exponentially growing concurrency levels, with future exascales delivering millions of nodes and billions of threads of execution. A critical component of future file systems for high-end computing is metadata management. This extended abstract presents ZHT, a zero-hop distributed hash-table, which has been tuned for the specific requirements of high-end computing. The primary goal of ZHT is excellent availability, fault tolerance, high throughput, and low latencies. 1.
Today with the rapidly accumulated data, datadriven applications are emerging in science and comm... more Today with the rapidly accumulated data, datadriven applications are emerging in science and commercial areas. On both HPC systems and clouds the continuously widening performance gap between storage and computing resource prevents us from building scalable data-intensive systems. Distributed NoSQL storage systems are known for their ease of use and attractive performance and are increasingly used as building blocks of large scale applications on cloud or data centers. However there are not many works on bridging the performance gap on supercomputers with NoSQL data stores. This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputers. It firstly presents ZHT, a dynamic scalable zero-hop distributed key-value store, that aims to be a building block of large scale systems on clouds and supercomputers. This work also presents several real systems that have adopted ZHT as well as other NoSQL systems, namely ZHT/Q (a Flexible QoS Fortified Distribute...
Today’s science is generating datasets that are increasing exponentially in both complexity and v... more Today’s science is generating datasets that are increasing exponentially in both complexity and volume, making their analysis, archival, and sharing one of the grand challenges of the 21st century. Exascale computing, i.e. 10 FLOPS, is predicted to emerge by 2019 with current trends. Millions of nodes and billions of threads of execution, producing similarly large concurrent data accesses, are expected with the exascale. Current state-of-the-art yet decades long storage architecture of high-performance computing (HPC) systems would unlikely provide the support for the expected level of concurrent data access. The main critique comes from the topological allocation of compute and storage resources that are interconnected as two cliques. Even though the network between compute and storage has high bandwidth and is sufficient for compute intensive petascale applications, it would not be adequate for data-intensive petascale computing or the emerging exascale computing (regardless if it...
Although the traditional relational database has been used and dominated for many year, the limit... more Although the traditional relational database has been used and dominated for many year, the limitation of it has appeared with the huge number of connected data which is generated by today’s Internet, Web2.0 and social networks. The query operation will be tremendously slow to query by the traditional relational database. The state-of-the-art graph database is usually Master/Slave architecture thus these systems cannot achieve high scalability and they will become harder and harder to handle the huge number of connected data. In this paper we have design and implement a graph database using ZHT as a block. This graph database is also following the principle of Bulk Synchronous Parallel(BSP) model. We have overcome several problems of using ZHT as basic component in BSP model. We will give the information of our design and implementation in the following part. At last we will also give a comprehensive performance evaluation. Keywords—ZHT; Graph Database; Graph Processing System;
The interoperability across multiple or many blockchains would play a critical role in the forthc... more The interoperability across multiple or many blockchains would play a critical role in the forthcoming blockchain-based data management paradigm. In particular, how to ensure the ACID properties of those transactions across an arbitrary number of blockchains remains an open problem in both academic and industry: Existing solutions either work for only two blockchains or requires a centralized component, neither of which would meet the scalability requirement in practice. This short paper shares our vision and some early results toward scalable cross-blockchain transactions. Specifically, we design two distributed commit protocols and, both analytically and experimentally, demonstrate their effectiveness.
It has become increasingly important to capture and understand the origins and derivation of data... more It has become increasingly important to capture and understand the origins and derivation of data (its provenance). A key issue in evaluating the feasibility of data provenance is its performance, overheads, and scalability. In this paper, we explore the feasibility of a management layer for parallel file systems, in which metadata includes both file operations and provenance metadata. We design and implement a provenance layer within a distributed file system —FusionFS, which implements a distributed file metadata management based on distributed hash tables. Our results show that FusionFS with its own storage layer for provenance capture is able to scale up to 1K nodes on BlueGene/P supercomputer.
In Big Data era, applications are generating orders of magnitude more data in both volume and qua... more In Big Data era, applications are generating orders of magnitude more data in both volume and quantity. While many systems emerge to address such data explosion, the fact that these data’s descriptors, i.e., metadata, are also “big” is often overlooked. The conventional approach to address the big metadata issue is to disperse metadata into multiple machines. However, it is extremely difficult to preserve both load-balance and data-locality in this approach. To this end, in this work we propose hierarchical indirection layers for indexing the underlying distributed metadata. By doing this, data locality is achieved efficiently by the indirection while load-balance is preserved. Three key challenges exist in this approach, however: first, how to achieve high resilience; second, how to ensure flexible granularity; third, how to restrain performance overhead. To address above challenges, we design Dindex, a distributed indexing service for metadata. Dindex incorporates a hierarchy of coarse-grained aggregation and horizontal key-coalition. Theoretical analysis shows that the overhead of building Dindex is compensated by only two or three queries. Dindex has been implemented by a lightweight distributed key-value store and integrated to a fully-fledged distributed filesystem. Experiments demonstrated that Dindex accelerated metadata queries by up to 60 percent with a negligible overhead.
2015 IEEE International Conference on Big Data (Big Data), 2015
In the era of Big Data and Cloud, distributed keyvalue stores are increasingly used as building b... more In the era of Big Data and Cloud, distributed keyvalue stores are increasingly used as building blocks of large scale applications. Comparing to traditional relational databases, keyvalue stores are particularly compelling due to their low latency and excellent scalability. Many big companies, such as Facebook and Amazon, run multiple different applications and services on top of a single key-value store deployment to reduce the deployment and maintenance complexity as well as economic cost. However every application has its own performance requirement but most of current key-value store systems are designed to serve every application request equally. This design works well when a single application accesses the key-value store, but it is not as good for the emerging concurrent multi-application scenario. In this paper we present ZHT/Q, a flexible QoS (Quality of Service) fortified distributed key-value storage system for clouds and data centers. It improves the overall throughput by an order of magnitude and still satisfies different applications' latency requirements with QoS by means of dynamic and adaptive request batching mechanisms. The experiment results show that our new system delivers up to 28 times higher throughput than the base solution while more than 99% of requests' latency requirements are satisfied.
2015 IEEE International Conference on Big Data (Big Data), 2015
In this paper, we propose and implement a keyvalue store that supports MPI while allowing applica... more In this paper, we propose and implement a keyvalue store that supports MPI while allowing application access at any time without having to declaring in the same MPI communication world. This feature may significantly simplify the application design and allow programmers leverage the power of key-value store in an intuitive way. In our preliminary experiment results captured from a supercomputer at Los Alamos National Laboratory, our prototype shows linear scalability at up to 256 nodes.
2015 IEEE International Conference on Cluster Computing, 2015
The emerging applications in big data and social networks issue rapidly increasing demands on gra... more The emerging applications in big data and social networks issue rapidly increasing demands on graph processing. Graph query operations that involve a large number of vertices and edges can be tremendously slow on traditional databases. The state of the art graph processing systems and databases usually adopt master/slave architecture that potentially impairs their scalability. This work describes the design and implementation of a new graph processing system based on Bulk Synchronous Parallel model. Our system is built on top of ZHT, a scalable distributed key-value store, which benefits the graph processing in terms of scalability, performance and persistency. The experiment results imply excellent scalability.
Concurrency and Computation: Practice and Experience, 2015
This paper presents a convergence of distributed Key-Value storage systems in clouds and supercom... more This paper presents a convergence of distributed Key-Value storage systems in clouds and supercomputers. It specifically presents ZHT, a zero-hop distributed key-value store system, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. ZHT has some important properties, such as being lightweight , dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append, compare and swap, callback in addition to the traditional insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 64-nodes, an Amazon EC2 virtual cluster up to 96-nodes, to an IBM Blue Gene/P supercomputer with 8K-nodes. We compared ZHT against other key/value stores and found it offers superior performance for the features and portability it supports. This paper also presents several real systems that have adopted ZHT, namely FusionFS (a distributed file system), IStore (a storage system with erasure coding), MATRIX (distributed scheduling), Slurm++ (distributed HPC job launch), Fabriq (distributed message queue management); all of these real systems have been simplified due to Key-Value storage systems, and have been shown to outperform other leading systems by orders of magnitude in some cases. It's important to highlight that some of these systems are rooted in HPC systems from supercomputers, while others are rooted in clouds and ad-hoc distributed systems; through our work, we have shown how versatile Key-Value storage systems can be in such a variety of environments.
2014 IEEE International Conference on Big Data (Big Data), 2014
State-of-the-art, yet decades-old, architecture of high-performance computing systems has its com... more State-of-the-art, yet decades-old, architecture of high-performance computing systems has its compute and storage resources separated. It thus is limited for modern data-intensive scientific applications because every I/O needs to be transferred via the network between the compute and storage resources. In this paper we propose an architecture that hss a distributed storage layer local to the compute nodes. This layer is responsible for most of the I/O operations and saves extreme amounts of data movement between compute and storage resources. We have designed and implemented a system prototype of this architecturewhich we call the FusionFS distributed file system-to support metadata-intensive and write-intensive operations, both of which are critical to the I/O performance of scientific applications. FusionFS has been deployed and evaluated on up to 16K compute nodes of an IBM Blue Gene/P supercomputer, showing more than an order of magnitude performance improvement over other popular file systems such as GPFS, PVFS, and HDFS.
Proceedings of the 6th Workshop on Scientific Cloud Computing - ScienceCloud '15, 2015
As small, specialized sensor devices become more ubiquitous, reliable, and cheap, increasingly mo... more As small, specialized sensor devices become more ubiquitous, reliable, and cheap, increasingly more domain sciences are creating "instruments at large"-dynamic, often selforganizing, groups of sensors whose outputs are capable of being aggregated and correlated to support experiments organized around specific questions. This calls for an infrastructure able to collect, store, query, and process data set from sensor networks. The design and development of such infrastructure faces several challenges. The challenges reflect the need to interact with and administer the sensors remotely. The sensors may be deployed in inaccessible places and have only intermittent network connectivity due to power conservation and other factors. This requires communication protocols that can withstand unreliable networks as well as an administrative interface to sensor controller. Further, the system has to be scalable, i.e., capable of ultimately dealing with potentially large numbers of data producing sensors. It also needs to be able to organize many different data types efficiently. And finally, it also needs to scale in the number of queries and processing requests. In this work we present a set of protocols and a cloud-based data streaming infrastructure called WaggleDB that address those challenges. The system efficiently aggregates and stores data from sensor networks and enables the users to query those data sets. It address the challenges above with a scalable multi-tier architecture, which is designed in such way that each tier can be scaled by adding more independent resources provisioned on-demand in the cloud.
This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputer... more This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputers. It specifically presents ZHT, a zero-hop distributed keyvalue store system, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales (millions of nodes). ZHT has some important properties, such as being lightweight , dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append, compare and swap, callback in addition to the traditional insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 64-nodes, an Amazon EC2 virtual cluster up to 96-nodes, to an IBM Blue Gene/P supercomputer with 8K-nodes. Using synthetic benchmarks, we scaled ZHT up to 32K-cores with latencies of only 1.1ms and 18M operations/sec throughput on Blue Gene/P. On Amazon EC2 cloud, on 96-node scale it offers 0.8ms latency and 1.2M ops/s throughput. We compared ZHT against other key/value stores (e.g. Casandra, Memcached, DynamoDB) and found it offers superior performance for the features and portability it supports. This work also presents several real systems that have adopted ZHT as well as other NoSQL systems, namely FusionFS (distributed metadata management and data provenance capture/query), IStore (data chunk metadata management), MATRIX (distributed scheduling), Slurm++ (distributed HPC job launch), ZDMQ (distributed message queue management), FREIDA-State (state management for scientific applications on cloud), and WaggleDB (a Cloud-based interactive data infrasvii tructure for sensor network applications); all of these real systems have been simplified due to NoSQL storage systems, and have been shown to outperform other leading systems by orders of magnitude in some cases. It's important to highlight that some of these systems are rooted in HPC systems from supercomputers, while others are rooted in clouds and ad-hoc distributed systems; through our work, we have shown how versatile NoSQL storage systems can be in such a variety of environments. This work will also outline planed future work, in terms of hierarchical architectures for ZHT, and the integration of ZHT into the Swift parallel programming system. viii 1 CHAPTER 1
We have built a new framework that enables independent metadata updates within HDF5 files. Our fr... more We have built a new framework that enables independent metadata updates within HDF5 files. Our framework includes the following components: an overlay network that's optimized for 1-to-All and All-to-All communication; root-less/server-less Bcast and AllReduce operations for reduced synchronization; a distributed consensus-making platform; an execution order keeping system, and a progress engine for generic threadless concurrence handling.
H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O per... more H5bench benchmark suite contains a list of applications that are used to measure the HDF5 I/O performance from various aspects.
Abstract—With the increased scale of systems in use and the need to quickly store and retrieve in... more Abstract—With the increased scale of systems in use and the need to quickly store and retrieve information, key/value stores are becoming an important element in the design of large-scale storage systems. Key/value stores are well known for their simplistic interfaces, persistent nature, and excellent operational efficiency – they are also known as NoSQL databases. This paper presents the design and implementation of a non-volatile hash table (NoVoHT). NoVoHT was designed from the ground up to be lightweight, fast, and dependency-free. Our goal was to create a fast persistent key/value store that could be easily integrated and operated in lightweight Linux OS typically found on today’s supercomputers. We also aimed to develop a system that performed as close as possible to an in-memory hash map, but with the added benefit of being persistent. We also extended the traditional key/value store interface (e.g. insert, lookup, remove) to include a novel operation (e.g. append) that has a...
ZHT is a zero-hop distributed hash table, which has been tuned for the requirements of high-end c... more ZHT is a zero-hop distributed hash table, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales of millions of nodes. ZHT has some important properties, such as being lightweight, dynamically allowing nodes to join and leave, fault tolerant through replications, persistent, scalable, and supporting unconventional operations such as append. ZHT scaled up to 32K-cores with latencies of 1.1ms and 18M operations/sec throughput on IBM Blue Gene/P supercomputer, and 96 nodes on Amazon EC2 cloud with 800ns latency and 1.2M ops/s throughput. In previous work we proved ZHT’s excellent performance and scalability on supercomputers, and in this work we show that it also works great on cloud environment from both performance and cost perspective.
Over the last decade, storage systems have experienced a 10fold increase between their capacity a... more Over the last decade, storage systems have experienced a 10fold increase between their capacity and bandwidth. This gap is predicted to grow faster with exponentially growing concurrency levels, with future exascales delivering millions of nodes and billions of threads of execution. A critical component of future file systems for high-end computing is metadata management. This extended abstract presents ZHT, a zero-hop distributed hash-table, which has been tuned for the specific requirements of high-end computing. The primary goal of ZHT is excellent availability, fault tolerance, high throughput, and low latencies. 1.
Today with the rapidly accumulated data, datadriven applications are emerging in science and comm... more Today with the rapidly accumulated data, datadriven applications are emerging in science and commercial areas. On both HPC systems and clouds the continuously widening performance gap between storage and computing resource prevents us from building scalable data-intensive systems. Distributed NoSQL storage systems are known for their ease of use and attractive performance and are increasingly used as building blocks of large scale applications on cloud or data centers. However there are not many works on bridging the performance gap on supercomputers with NoSQL data stores. This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputers. It firstly presents ZHT, a dynamic scalable zero-hop distributed key-value store, that aims to be a building block of large scale systems on clouds and supercomputers. This work also presents several real systems that have adopted ZHT as well as other NoSQL systems, namely ZHT/Q (a Flexible QoS Fortified Distribute...
Today’s science is generating datasets that are increasing exponentially in both complexity and v... more Today’s science is generating datasets that are increasing exponentially in both complexity and volume, making their analysis, archival, and sharing one of the grand challenges of the 21st century. Exascale computing, i.e. 10 FLOPS, is predicted to emerge by 2019 with current trends. Millions of nodes and billions of threads of execution, producing similarly large concurrent data accesses, are expected with the exascale. Current state-of-the-art yet decades long storage architecture of high-performance computing (HPC) systems would unlikely provide the support for the expected level of concurrent data access. The main critique comes from the topological allocation of compute and storage resources that are interconnected as two cliques. Even though the network between compute and storage has high bandwidth and is sufficient for compute intensive petascale applications, it would not be adequate for data-intensive petascale computing or the emerging exascale computing (regardless if it...
Although the traditional relational database has been used and dominated for many year, the limit... more Although the traditional relational database has been used and dominated for many year, the limitation of it has appeared with the huge number of connected data which is generated by today’s Internet, Web2.0 and social networks. The query operation will be tremendously slow to query by the traditional relational database. The state-of-the-art graph database is usually Master/Slave architecture thus these systems cannot achieve high scalability and they will become harder and harder to handle the huge number of connected data. In this paper we have design and implement a graph database using ZHT as a block. This graph database is also following the principle of Bulk Synchronous Parallel(BSP) model. We have overcome several problems of using ZHT as basic component in BSP model. We will give the information of our design and implementation in the following part. At last we will also give a comprehensive performance evaluation. Keywords—ZHT; Graph Database; Graph Processing System;
The interoperability across multiple or many blockchains would play a critical role in the forthc... more The interoperability across multiple or many blockchains would play a critical role in the forthcoming blockchain-based data management paradigm. In particular, how to ensure the ACID properties of those transactions across an arbitrary number of blockchains remains an open problem in both academic and industry: Existing solutions either work for only two blockchains or requires a centralized component, neither of which would meet the scalability requirement in practice. This short paper shares our vision and some early results toward scalable cross-blockchain transactions. Specifically, we design two distributed commit protocols and, both analytically and experimentally, demonstrate their effectiveness.
It has become increasingly important to capture and understand the origins and derivation of data... more It has become increasingly important to capture and understand the origins and derivation of data (its provenance). A key issue in evaluating the feasibility of data provenance is its performance, overheads, and scalability. In this paper, we explore the feasibility of a management layer for parallel file systems, in which metadata includes both file operations and provenance metadata. We design and implement a provenance layer within a distributed file system —FusionFS, which implements a distributed file metadata management based on distributed hash tables. Our results show that FusionFS with its own storage layer for provenance capture is able to scale up to 1K nodes on BlueGene/P supercomputer.
In Big Data era, applications are generating orders of magnitude more data in both volume and qua... more In Big Data era, applications are generating orders of magnitude more data in both volume and quantity. While many systems emerge to address such data explosion, the fact that these data’s descriptors, i.e., metadata, are also “big” is often overlooked. The conventional approach to address the big metadata issue is to disperse metadata into multiple machines. However, it is extremely difficult to preserve both load-balance and data-locality in this approach. To this end, in this work we propose hierarchical indirection layers for indexing the underlying distributed metadata. By doing this, data locality is achieved efficiently by the indirection while load-balance is preserved. Three key challenges exist in this approach, however: first, how to achieve high resilience; second, how to ensure flexible granularity; third, how to restrain performance overhead. To address above challenges, we design Dindex, a distributed indexing service for metadata. Dindex incorporates a hierarchy of coarse-grained aggregation and horizontal key-coalition. Theoretical analysis shows that the overhead of building Dindex is compensated by only two or three queries. Dindex has been implemented by a lightweight distributed key-value store and integrated to a fully-fledged distributed filesystem. Experiments demonstrated that Dindex accelerated metadata queries by up to 60 percent with a negligible overhead.
2015 IEEE International Conference on Big Data (Big Data), 2015
In the era of Big Data and Cloud, distributed keyvalue stores are increasingly used as building b... more In the era of Big Data and Cloud, distributed keyvalue stores are increasingly used as building blocks of large scale applications. Comparing to traditional relational databases, keyvalue stores are particularly compelling due to their low latency and excellent scalability. Many big companies, such as Facebook and Amazon, run multiple different applications and services on top of a single key-value store deployment to reduce the deployment and maintenance complexity as well as economic cost. However every application has its own performance requirement but most of current key-value store systems are designed to serve every application request equally. This design works well when a single application accesses the key-value store, but it is not as good for the emerging concurrent multi-application scenario. In this paper we present ZHT/Q, a flexible QoS (Quality of Service) fortified distributed key-value storage system for clouds and data centers. It improves the overall throughput by an order of magnitude and still satisfies different applications' latency requirements with QoS by means of dynamic and adaptive request batching mechanisms. The experiment results show that our new system delivers up to 28 times higher throughput than the base solution while more than 99% of requests' latency requirements are satisfied.
2015 IEEE International Conference on Big Data (Big Data), 2015
In this paper, we propose and implement a keyvalue store that supports MPI while allowing applica... more In this paper, we propose and implement a keyvalue store that supports MPI while allowing application access at any time without having to declaring in the same MPI communication world. This feature may significantly simplify the application design and allow programmers leverage the power of key-value store in an intuitive way. In our preliminary experiment results captured from a supercomputer at Los Alamos National Laboratory, our prototype shows linear scalability at up to 256 nodes.
2015 IEEE International Conference on Cluster Computing, 2015
The emerging applications in big data and social networks issue rapidly increasing demands on gra... more The emerging applications in big data and social networks issue rapidly increasing demands on graph processing. Graph query operations that involve a large number of vertices and edges can be tremendously slow on traditional databases. The state of the art graph processing systems and databases usually adopt master/slave architecture that potentially impairs their scalability. This work describes the design and implementation of a new graph processing system based on Bulk Synchronous Parallel model. Our system is built on top of ZHT, a scalable distributed key-value store, which benefits the graph processing in terms of scalability, performance and persistency. The experiment results imply excellent scalability.
Concurrency and Computation: Practice and Experience, 2015
This paper presents a convergence of distributed Key-Value storage systems in clouds and supercom... more This paper presents a convergence of distributed Key-Value storage systems in clouds and supercomputers. It specifically presents ZHT, a zero-hop distributed key-value store system, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. ZHT has some important properties, such as being lightweight , dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append, compare and swap, callback in addition to the traditional insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 64-nodes, an Amazon EC2 virtual cluster up to 96-nodes, to an IBM Blue Gene/P supercomputer with 8K-nodes. We compared ZHT against other key/value stores and found it offers superior performance for the features and portability it supports. This paper also presents several real systems that have adopted ZHT, namely FusionFS (a distributed file system), IStore (a storage system with erasure coding), MATRIX (distributed scheduling), Slurm++ (distributed HPC job launch), Fabriq (distributed message queue management); all of these real systems have been simplified due to Key-Value storage systems, and have been shown to outperform other leading systems by orders of magnitude in some cases. It's important to highlight that some of these systems are rooted in HPC systems from supercomputers, while others are rooted in clouds and ad-hoc distributed systems; through our work, we have shown how versatile Key-Value storage systems can be in such a variety of environments.
2014 IEEE International Conference on Big Data (Big Data), 2014
State-of-the-art, yet decades-old, architecture of high-performance computing systems has its com... more State-of-the-art, yet decades-old, architecture of high-performance computing systems has its compute and storage resources separated. It thus is limited for modern data-intensive scientific applications because every I/O needs to be transferred via the network between the compute and storage resources. In this paper we propose an architecture that hss a distributed storage layer local to the compute nodes. This layer is responsible for most of the I/O operations and saves extreme amounts of data movement between compute and storage resources. We have designed and implemented a system prototype of this architecturewhich we call the FusionFS distributed file system-to support metadata-intensive and write-intensive operations, both of which are critical to the I/O performance of scientific applications. FusionFS has been deployed and evaluated on up to 16K compute nodes of an IBM Blue Gene/P supercomputer, showing more than an order of magnitude performance improvement over other popular file systems such as GPFS, PVFS, and HDFS.
Proceedings of the 6th Workshop on Scientific Cloud Computing - ScienceCloud '15, 2015
As small, specialized sensor devices become more ubiquitous, reliable, and cheap, increasingly mo... more As small, specialized sensor devices become more ubiquitous, reliable, and cheap, increasingly more domain sciences are creating "instruments at large"-dynamic, often selforganizing, groups of sensors whose outputs are capable of being aggregated and correlated to support experiments organized around specific questions. This calls for an infrastructure able to collect, store, query, and process data set from sensor networks. The design and development of such infrastructure faces several challenges. The challenges reflect the need to interact with and administer the sensors remotely. The sensors may be deployed in inaccessible places and have only intermittent network connectivity due to power conservation and other factors. This requires communication protocols that can withstand unreliable networks as well as an administrative interface to sensor controller. Further, the system has to be scalable, i.e., capable of ultimately dealing with potentially large numbers of data producing sensors. It also needs to be able to organize many different data types efficiently. And finally, it also needs to scale in the number of queries and processing requests. In this work we present a set of protocols and a cloud-based data streaming infrastructure called WaggleDB that address those challenges. The system efficiently aggregates and stores data from sensor networks and enables the users to query those data sets. It address the challenges above with a scalable multi-tier architecture, which is designed in such way that each tier can be scaled by adding more independent resources provisioned on-demand in the cloud.
This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputer... more This work presents a convergence of distributed NoSQL storage systems in clouds and supercomputers. It specifically presents ZHT, a zero-hop distributed keyvalue store system, which has been tuned for the requirements of high-end computing systems. ZHT aims to be a building block for future distributed systems, such as parallel and distributed file systems, distributed job management systems, and parallel programming systems. The goals of ZHT are delivering high availability, good fault tolerance, high throughput, and low latencies, at extreme scales (millions of nodes). ZHT has some important properties, such as being lightweight , dynamically allowing nodes join and leave, fault tolerant through replication, persistent, scalable, and supporting unconventional operations such as append, compare and swap, callback in addition to the traditional insert/lookup/remove. We have evaluated ZHT's performance under a variety of systems, ranging from a Linux cluster with 64-nodes, an Amazon EC2 virtual cluster up to 96-nodes, to an IBM Blue Gene/P supercomputer with 8K-nodes. Using synthetic benchmarks, we scaled ZHT up to 32K-cores with latencies of only 1.1ms and 18M operations/sec throughput on Blue Gene/P. On Amazon EC2 cloud, on 96-node scale it offers 0.8ms latency and 1.2M ops/s throughput. We compared ZHT against other key/value stores (e.g. Casandra, Memcached, DynamoDB) and found it offers superior performance for the features and portability it supports. This work also presents several real systems that have adopted ZHT as well as other NoSQL systems, namely FusionFS (distributed metadata management and data provenance capture/query), IStore (data chunk metadata management), MATRIX (distributed scheduling), Slurm++ (distributed HPC job launch), ZDMQ (distributed message queue management), FREIDA-State (state management for scientific applications on cloud), and WaggleDB (a Cloud-based interactive data infrasvii tructure for sensor network applications); all of these real systems have been simplified due to NoSQL storage systems, and have been shown to outperform other leading systems by orders of magnitude in some cases. It's important to highlight that some of these systems are rooted in HPC systems from supercomputers, while others are rooted in clouds and ad-hoc distributed systems; through our work, we have shown how versatile NoSQL storage systems can be in such a variety of environments. This work will also outline planed future work, in terms of hierarchical architectures for ZHT, and the integration of ZHT into the Swift parallel programming system. viii 1 CHAPTER 1
Uploads
Papers by Tonglin Li