Back to Search
Start Over
CSMqGraph: Coarse-Grained and Multi-external-storage Multi-queue I/O Management for Graph Computing
- Source :
- International Journal of Parallel Programming. 48:98-118
- Publication Year :
- 2019
- Publisher :
- Springer Science and Business Media LLC, 2019.
-
Abstract
- As graphs continue growing, external storage graph processing systems serve as a promising alternative to distributed in-memory solutions for low cost and high scalability. To obtain high I/O throughput, these systems usually use multiple external storage devices. They adopt the operating system I/O management method based on striped volume, resulting in unsatisfactory performance, such as low sequential bandwidth utilization of each external storage device, limited I/O parallelism and expensive management overhead. In this paper, we analyzed the problems of the operating system I/O management method based on striped volume. Then we designed CSMqGraph, a graph processing system adopts coarse-grained striping method matching sequential large I/O to fully utilize the maximum sequential bandwidth of each external storage device and an I/O management strategy based on multi-external-storage multi-queue making I/O threads dedicated to each external storage device to further improve I/O throughput and fully exploit the parallelism of multiple external storage devices. For different graph algorithms and datasets, our evaluation shows that CSMqGraph consistently outperforms state-of-the-art engines GridGraph by up to 40%, and has better I/O scalability.
- Subjects :
- 010302 applied physics
Exploit
Computer science
02 engineering and technology
Parallel computing
01 natural sciences
Graph
020202 computer hardware & architecture
Theoretical Computer Science
Management strategy
External storage
0103 physical sciences
Scalability
Theory of computation
0202 electrical engineering, electronic engineering, information engineering
Data striping
Queue
Software
Information Systems
Subjects
Details
- ISSN :
- 15737640 and 08857458
- Volume :
- 48
- Database :
- OpenAIRE
- Journal :
- International Journal of Parallel Programming
- Accession number :
- edsair.doi...........4b36e0651176dfa311007f22a8535736