1. Scalable, reproducible, and cost-effective processing of large-scale medical imaging datasets
- Author
-
Kim, Michael E., Ramadass, Karthik, Gao, Chenyu, Kanakaraj, Praitayini, Newlin, Nancy R., Rudravaram, Gaurav, Schilling, Kurt G., Dewey, Blake E., Archer, Derek, Hohman, Timothy J., Li, Zhiyuan, Bao, Shunxing, Landman, Bennett A., and Khairi, Nazirah Mohd
- Subjects
Computer Science - Distributed, Parallel, and Cluster Computing ,Computer Science - Databases - Abstract
Curating, processing, and combining large-scale medical imaging datasets from national studies is a non-trivial task due to the intense computation and data throughput required, variability of acquired data, and associated financial overhead. Existing platforms or tools for large-scale data curation, processing, and storage have difficulty achieving a viable cost-to-scale ratio of computation speed for research purposes, either being too slow or too expensive. Additionally, management and consistency of processing large data in a team-driven manner is a non-trivial task. We design a BIDS-compliant method for an efficient and robust data processing pipeline of large-scale diffusion-weighted and T1-weighted MRI data compatible with low-cost, high-efficiency computing systems. Our method accomplishes automated querying of data available for processing and process running in a consistent and reproducible manner that has long-term stability, while using heterogenous low-cost computational resources and storage systems for efficient processing and data transfer. We demonstrate how our organizational structure permits efficiency in a semi-automated data processing pipeline and show how our method is comparable in processing time to cloud-based computation while being almost 20 times more cost-effective. Our design allows for fast data throughput speeds and low latency to reduce the time for data transfer between storage servers and computation servers, achieving an average of 0.60 Gb/s compared to 0.33 Gb/s for using cloud-based processing methods. The design of our workflow engine permits quick process running while maintaining flexibility to adapt to newly acquired data.
- Published
- 2024