dispel4py: A Python framework for data-intensive scientific computing.

This paper presents <monospace>dispel4py</monospace>, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. <monospace>dispel4py</monospace> enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map <monospace>dispel4py</monospace> abstract workflows optimally onto target platforms chosen dynamically. We present four <monospace>dispel4py</monospace> mappings: Apache Storm, message-passing interface (MPI), multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The <monospace>dispel4py.org</monospace> is an open-source project to which we invite participation. The effective mapping of <monospace>dispel4py</monospace> onto multiple target infrastructures demonstrates exploitation of data-intensive and high-performance computing (HPC) architectures and consistent scalability. [ABSTRACT FROM AUTHOR]