Towards Integrated Hardware/Software Ecosystems for the Edge-Cloud-HPC Continuum

Modern use cases such as autonomous vehicles, digital twins, smart buildings and precision agriculture, greatly increase the complexity of application workflows. They typically combine physics-based simulations, analysis of large data volumes and machine learning and require a hybrid execution infrastructure: edge devices create streams of input data, which are processed by data analytics and machine learning applications in the Cloud, and simulations on large, specialised HPC systems provide insights into and prediction of future system state. From these results, additional steps create and communicate output data across the infrastructure levels, and for some use cases, control devices or cyber-physical systems in the real world are controlled (as in the case of smart factories). All of these steps pose different requirements for the best suited execution platforms, and they need to be connected in an efficient and secure way. This assembly is called the Computing Continuum (CC) (1). It raises challenges at multiple levels: at the application level, innovative algorithms are needed to bridge simulations, machine learning and data-driven analytics; at the middleware level, adequate tools must enable efficient deployment, scheduling and orchestration of the workflow components across the whole distributed infrastructure; and, finally, a capable resource management system must allocate a suitable set of components of the infrastructure to run the application workflow, preferably in a dynamic and adaptive way, taking into account the specific capabilities of each component of the underlying heterogeneous infrastructure. To address the challenges, we foresee an increasing need for integrated software ecosystems which combine current “island” solutions and bridge the gaps between them. These ecosystems must facilitate the full lifecycle of CC use cases, including initial modelling, programming, deployment, execution, optimisation, as well as monitoring and control. It will be important to ensure adequate reproducibility of workflow results and to find ways for creating and managing trust when sharing systems, software and data. All of these will in turn require novel or improved hardware capabilities. This white paper provides an initial discussion of the gaps. Our objective is to accelerate progress in both hardware and software infrastructures to build CC use cases, with the ultimate goals of accelerating scientific discovery, improving timeliness, quality and sustainability of engineering artefacts, and supporting decisions in complex and potentially urgent situations
The authors would like to thank Rafael Mayo-García from CIEMAT and Marion Carrier from CybeleTech for their help in describing relevant use cases for the computing continuum.
{"references":["D. Balouek-Thomert, E. Gibert Renart , A. R. Zamani, A. Simonet and M. Parashar, \"Towards a computing continuum: Enabling edge-to-cloud integration for data-driven workflows,\" The International Journal of High Performance Computing Applications, vol. 33, no. 6, 2019.","G. Nguyen, S. Dlugolinsky , M. Bobák, V. Tran, A. López García, I. Heredia, P. Malík and L. Hluchý, \"Machine Learning and Deep Learning frameworks and libraries for large-scale data mining: a survey,\" Artificial Intelligence Review, vol. 52, p. 77–124, 2019.","O. Peckham, \"15 Years Later, the Green500 Continues Its Push for Energy Efficiency as a First-Order Concern in HPC,\" HPCwire, no. https://www.hpcwire.com/2021/07/15/15-years-later-the-green500-continues-its-push-for-energy-efficiency-as-a-first-order-concern-in-hpc/, 17 07 2021.","E. Masamet, A. Shehabi, N. Lei, S. Smith and J. Koomey, \"Recalibrating global data center energy use estimates,\" Science, vol. 367, no. 6481, pp. 984-986, 2020.","Hyperion Research, [Online]. Available: https://hyperionresearch.com/wp-content/uploads/2021/07/Hyperion-Research-HPC-Briefing-Slides-During-ISC-June-2021.b.pdf.","S. Dewitte, J. P. Cornelis, R. Müller and A. Munteanu, \"Artificial Intelligence Revolutionises Weather Forecast, Climate Monitoring and Decadal Prediction,\" Remote Sensing, vol. 13, no. 16, p. 3209, 2021.","M. Feldman, \"HPC in 2020: AI is no longer an experiment,\" The Next Platform, no. https://www.nextplatform.com/2020/01/09/hpc-in-2020-ai-is-no-longer-an-experiment/, 09 01 2020.","Google Cloud, \"Helping researchers at CERN to analyze powerful data and uncover the secrets of our universe,\" [Online]. Available: https://cloud.google.com/customers/cern.","O. Peckham, \"Ahead of 'Dojo,' Tesla Reveals Its Massive Precursor Supercomputer,\" HPCwire, no. https://www.hpcwire.com/2021/06/22/ahead-of-dojo-tesla-reveals-its-massive-precursor-supercomputer/, 22 06 2021.","K. Fauvel, D. Balouek-Thomert, D. Melgar, P. Silva, A. Simonet, G. Antoniu, A. Costan, V. Masson, M. Parashar, I. Rodero and A. Termier, \"A Distributed Multi-Sensor Machine Learning Approach to Earthquake Early Warning,\" in 34th AAAI Conference on Artificial Intelligence, Outstanding Paper Award - Special Track for Social Impact, New York, https://ojs.aaai.org//index.php/AAAI/article/view/5376, 2020.","M. Asch, F. Bodin, M. Beck, T. Moore, M. Taufer, M. Swany and J.-P. Vilotte, \"Cybercosm: New Foundations for a Converged Science Data Ecosystem,\" https://arxiv.org/abs/2105.10680v3, 2021."]}