1. A microservice-based building block approach for scientific workflow engines: Processing large data volumes with dagonstar
- Author
-
Dante D. Sanchez-Gallegos, Diana Di Luccio, J.L. Gonzalez-Compean, and Raffaele Montella
- Subjects
business.industry ,Parallel Processing ,020206 networking & telecommunications ,Cloud computing ,02 engineering and technology ,Cloud Computing ,Data science ,Workflow engine ,Field (computer science) ,Workflows ,Microservices ,Proof of concept ,Virtual Containers ,0202 electrical engineering, electronic engineering, information engineering ,020201 artificial intelligence & image processing ,Natural resource management ,Architecture ,business ,Throughput (business) ,Block (data storage) - Abstract
The impact of machine learning algorithms on everyday life is overwhelming until the novel concept of datacracy as a new social paradigm. In the field of computational environmental science and, in particular, of applications of large data science proof of concept on the natural resources management this kind of approaches could make the difference between species surviving to potential extinction and compromised ecological niches. In this scenario, the use of high throughput workflow engines, enabling the management of complex data flows in production is rock solid, as demonstrated by the rise of recent tools as Parsl and DagOnStar. Nevertheless, the availability of dedicated computational resources, although mitigated by the use of cloud computing technologies, could be a remarkable limitation. In this paper, we present a novel and improved version of DagOnStar, enabling the execution of lightweight but recurring computational tasks on the microservice architecture. We present our preliminary results motivating our choices supported by some evaluations and a real-world use case.
- Published
- 2019