Search

This document provides an architectural blueprint for the interTwin Digital Twin Engine, outlining its functional specifications, requirements analysis, and fundamental building blocks. The blueprint is designed to be utilised by the interTwin project's technical Work Packages (WPs) to align with the requirements of associated subsystems. Additionally, it considers other relevant initiatives and projects to identify potential architectural components that can be incorporated within the interTwin context. The blueprint will ultimately serve as a conceptual model of the Digital Twin Engine, following its planned evolution and iterations through collaborative co-creation during the project.

The European Open Science Cloud is a pan-European initiative that aims at developing a federated infrastructure that supports scientific research and Open Science. Having the FAIR principles at the core of Open Science practices, the Environmental Research Infrastructures (ENVRIs) that operate in Europe collaborate at cluster level (the ENVRI cluster) continuously improving the FAIRness of their services and working towards becoming more interoperable among them but also with other clusters. To showcase the benefits of an integration platform that supports scientific workflows, the ENVRIs develop the “Dashboard for the State of the Environment” as a cross-discipline service to address with scientific facts the environmental concerns. The project brings together three scientific domains (Atmosphere, Biodiversity, Ocean) that each have set up analytical workflows to provide environmental indicators in real-time, allowing the users to visualise the “State of the Environment” by interacting with the service interface. The Dashboard is designed to be completely user configurable so that the users can select from a list the indicators to be shown and their order. Providers can add, remove and edit indicators through a standard REST API, that allows transferring all parameters, including the configuration of the indicators and how to provision data values and thumbnail interaction. The Dashboard is implemented and operated using engineering best practices, including YAML for the indicators’ descriptions and a robust and flexible container-based deployment. It builds on EOSC services like AAI, cloud services, and data storage, and the workflows that provide the indicators will also build on the EOSC and Research Infrastructure (RI) computing integration. As a proof of concept, a limited list of indicators is available, and we foresee that the participating RIs will provide many more indicator options in the near future. In addition, through the extension API it will be possible for new RIs to start providing indicators. The Dashboard service is completely open source, and, as the whole concept, it is designed to be flexible and expansible. Therefore, we encourage other clusters that are part of EOSC to use the service as another basis for disseminating their relevant indicators to a wider audience.

NSDFProtect and Access Management.

Computational science depends on complex, data intensive applications operating on datasets from a variety of scientific instruments. A major challenge is the integration of data into the scientist’s workflow. Recent advances in dynamic, networked cloud resources provide the building blocks to construct reconfiguration, end-to-end infrastructure that can increase scientific productivity, but applications are not taking advantage of them. In our previous work, we introduced DyNamo, that enabled CASA scientists to improve the efficiency of their operations and effortlessly leverage capabilities of the cloud resources available to them that previously remained underutilized. However, the provided workflow automation did not satisfy all the operational requirements of CASA. Custom scripts were still in production to manage workflow triggering, while multiple layer 2 connections would have to be allocated to maintain network QoS requirements. To address these issues, we enhance the DyNamo system with advanced network manipulation mechanisms, end-to-end infrastructure monitoring and ensemble workflow management capabilities. DyNamo’s Virtual Software Defined Exchange (vSDX) capabilities have been extended, enabling link adaptation, flow prioritization and traffic control between endpoints. These new features allow us to enforce network QoS requirements for each workflow ensemble and can lead to more fair network sharing. Additionally, to accommodate CASA’s operational needs we have extended the newly integrated Pegasus Ensemble Manager with event based triggering functionality, that improves managing CASA’s workflow ensembles. The Pegasus Ensemble Manager, apart from managing the workflow ensembles can also create conditions for a more fair resource usage, by employing throttling techniques to reduce compute and network resource contention. We evaluate the effects of the DyNamo’s vSDX policies by using two CASA workflow ensembles competing for network resources, and we show that traffic shaping of the ensembles can lead to a fairer sharing of the network links. Finally, we study how changing the Pegasus Ensemble Manager’s throttling for each of the two workflow ensembles affects their performance while they compete for the same network resources, and we assess if this approach is a viable alternative compared to the vSDX policies.

With the growing number and increasing availability of shared-use instruments and observatories, observational data is becoming an essential part of application workflows and contributor to scientific discoveries in a range of disciplines. However, the corresponding growth in the number of users accessing these facilities coupled with the expansion in the scale and variety of the data, is making it challenging for these facilities to ensure their data can be accessed, integrated, and analyzed in a timely manner, and is resulting significant demands on their cyberinfrastructure (CI). In this paper, we present the design of a push-based data delivery framework that leverages emerging in-network capabilities, along with data pre-fetching techniques based on a hybrid data management model. Specifically, we analyze data access traces for two large-scale observatories, Ocean Observatories Initiative (OOI) and Geodetic Facility for the Advancement of Geoscience (GAGE), to identify typical user access patterns and to develop a model that can be used for data pre-fetching. Furthermore, we evaluate our data pre-fetching model and the proposed framework using a simulation of the Virtual Data Collaboratory (VDC) platform that provides in-network data staging and processing capabilities. The results demonstrate that the ability of the framework to significantly improve data delivery performance and reduce network traffic at the observatories’ facilities.