Search

This report provides an overview of the integration of various use cases into the FedEarthData platform under the C-SCALE project, with a focus on Workflow Solutions and new use cases on-boarded via an open call. Workflow Solutions are applications developed to facilitate the use of FedEarthData by offering easily deployable templates and reusable components. Additionally, the inclusion of new use cases from diverse scientific disciplines underlines the adaptability and flexibility of the FedEarthData platform. The user feedback collected throughout the project shows overall satisfaction with the provided C-SCALE services. However, users also highlighted potential areas of improvement, primarily in resource provision and the availability of comprehensive documentation. The report underlines the effectiveness of C-SCALE support and identifies further opportunities to enhance the user experience on the FedEarthData infrastructure.

This document provides the final report on the services created in the C-SCALE project aimed to streamline data coming from Copernicus platforms and EOSC e-infrastructure compute providers. The report includes a description of the key exploitable results and the main services created during C-SCALE to ensure their exploitation, uptake and sustainability beyond the life of the project.

The benefits of Open Science (OS) and FAIR foundational principles - Findable, Accessible, Interoperable and Reusable - are increasingly valued by academia, although what OS and FAIR entail in practice is largely misunderstood. Once researchers manage to grasp OS and FAIR principles, they are often hit by practical difficulties. The European Open Science Cloud (EOSC) is the main initiative in Europe for providing a federated and open multi-disciplinary environment where European researchers, innovators, companies and citizens can share, publish, find and re-use data, tools and services for research, innovation and educational purposes. One of the goals of the EOSC is to co-design with communities the tools and services that are useful for their day to day research work, to facilitate collaboration and to foster wider adoption of Open Science practices. The Pangeo (https://pangeo.io/) community is a world-wide community of scientists and developers, who thrives to facilitate the deployment of ready-to-use and community-driven platforms for big data geoscience. While a number of services based on Jupyter Notebooks were already available, no public Pangeo deployments providing fast access to large amounts of data and compute resources were accessible on EOSC. Most existing cloud-based Pangeo deployments are USA-based, and members of the Pangeo community in Europe did not have a shared platform where scientists or technologists could exchange know-how and experiences. Pangeo teamed up with two EOSC projects, namely EGI-ACE (https://www.egi.eu/project/egi-ace/) and C-SCALE (https://c-scale.eu/) to demonstrate how to deploy and use Pangeo on EOSC and emphasise the benefits for the European community. The Pangeo Europe Community together with EGI deployed a DaskHub, composed of a Dask Gateway (https://gateway.dask.org/) and JupyterHub (https://jupyter.org/hub), with a Kubernetes cluster backend on EOSC using the infrastructure of the EGI Federation (https://www.egi.eu/egi-federation/). The Pangeo EOSC JupyterHub deployment makes use of 1) the EGI Check-In to enable user registration (and thereby authenticated and authorised access to the Pangeo JupyterHub portal and to the underlying distributed compute infrastructure); and 2) the EGI Cloud Compute and the cloud-based EGI Online Storage (to distribute the computational tasks to a scalable compute platform and to store intermediate results produced by the user jobs). To facilitate future Pangeo deployments on top of a wide range of cloud providers (AWS, Google Cloud, Microsoft Azure, EGI Cloud Computing, OpenNebula, OpenStack, and many more), the Pangeo EOSC JupyterHub deployment is now possible through a so-called Infrastructure Manager (IM) Dashboard (https://im.egi.eu/). All the computing and storage resources are currently supplied by CESNET (https://www.cesnet.cz/?lang=en) in the frame of EGI-ACE project (https://www.egi.eu/project/egi-ace/). Several deployments have been made to serve the geoscience community, both for teaching and for research work. One major advantage of these deployments for teaching and on-boarding researchers is the possibility to train them with realistic, large and complex data analysis problems similar to or directly part of their research work. Participants are taught the usage of Xarray, Dask and more generally how to efficiently access and analyse large online datasets. With this approach, attendees have the opportunity to ask questions, collaborate with other researchers as well as Research Software Engineers, and apply Open Science practices without the burden of trying and (sometimes) failing alone and without having to build their own infrastructure. To date, more than 100 researchers have been trained on Pangeo@EOSC deployments. With a growing community, discussions arose about the need to define clear practices for writing and publishing FAIR Jupyter Notebooks that can be reused and built upon for new research. This is where the community of practice comes into play, putting the focus back on actual Open Science practices. Pangeo teamed up with the Environmental Data Science Book (or EDS Book), a pan-european community-driven resource hosted on GitHub and powered by Jupyter Book that provides practical guidelines and templates to help researchers to translate research outputs into curated, interactive, shareable and reproducible executable notebooks. The quality of the FAIR notebooks is ensured by a collaborative and transparent reviewing process supported by GitHub related technologies. Thanks to EDS, Jupyter Notebooks created and published by researchers are much more modular and reusable. In this presentation, we will provide details on the different deployments, how to get access to JupyterHub deployments and contribute to the EDS, and more generally how to contribute to Pangeo@EOSC.

Oral presentation (and demo that is not contained in this presentation) given during thewebinar https://eoscfuture.eu/eventsfuture/eosc-future-use-case-event/ about Use cases from the EOSC Community, 16 May 9.00-12:30 CEST, co-organised by EOSC Future, C-SCALE, DICE, EGI-ACE, OpenAIRE Nexus, and Reliance projects.

Slides to present the e-infrastructure delivered to the Climate Informatics 2023 Reproducibility Challenge: https://github.com/eds-book/reproducibility-challenge-2023 YouTube recording: https://youtu.be/Si-wr7iwTro

This document provides an updated report on the data types that are collected and generated per Work Package (WP), whether they will be shared with the public and how, whether they can be reused and how, and how they will be preserved during and after the end of the project. The first version of the Data Management Plan, submitted in March 2022 to the EC, included the information on the aforementioned topics until March 2022, while in this second and last version, in addition to the above, information on the data that the onboarded use cases collected and generated is provided as well as on the Intellectual Property Rights (IPR) of C-SCALE data is included.

The third and final version of the Cloud Infrastructure, Incentives Management and Data Governance software prototype includes the cloud gateways and APIs, the cloud provisioning mechanisms, the implemented version of the algorithms as well as the data governance tools according to the D3.1 [1], D3.4 [2] and D3.7 [3] specifications and is built upon the first and second versions of the prototype described in D3.2 [4] and D3.5 [5] respectively. The prototype’s cloud infrastructure is supported by RECAS-BARI and is utilized by EGI through cloud gateways. These gateways allow the prototype to gather data from heterogenous data sources, such as Twitter and the global terrorism database and have integrated microservices to serve the needs of the different PolicyCLOUD pilots. The final version of the Incentives Management tool is also provided in this deliverable. This final version has been integrated with the Policy Development Toolkit (PDT) and has been deployed in the EGI Cloud. This third version of the prototype also includes the latest updates of the ABAC based access control mechanism and the Keycloak integration. This is broken down to 8 key components that have been combined to provide fine-tuned and secure access control and authentication. Specifically, Keycloak has been integrated with the Marketplace, the Gateways and the PDT and custom access policies have been developed for the gateways microservices. Finally, both the introduction of a XACML editor to ease access policies creation and the integration of EGI Check-in, an alternative way to authenticate to the prototype with academic and social credential, enhance the user experience in the PolicyCLOUD platform, This deliverable is submitted to the EC, not yet approved.

The third and final version of the PolicyCLOUD Conceptual Model & Reference Architecture (originally submitted as Deliverable D2.2 in September 2020 [20] with the second version submitted as D2.6 in June 2021 [21]) is presented in this document. The PolicyCLOUD Conceptual Model presents the overall project concept along 2 main axes. Along the first data axis PolicyCLOUD delivers Cloud Gateways and APIs to access data sources and adapt to their interfaces so as to simplify interaction and data collection from any source. Along the second main axis, the Policies Management Framework of PolicyCLOUD allows the definition of forward-looking policies as well as their dynamic adaptation and refocusing to the population they are applied on. Based on the project’s offerings along the main two axes of the Concept, five main building blocks (in a layered manner) define its Architecture: (1) The Cloud Based Environment and Data Acquisition, (2) Data Analytics, (3) the Policies Management Framework, (4) the Policy Development Toolkit and (5) The Marketplace. The architecture also includes a Data Governance Model, Protection and Privacy Enforcement and the Ethical Framework as depicted in Figure 2. The architecture allows for integrated data acquisition and analytics. It also allows data fusion with processing and initial analytics (see 7.6.5) as well as seamless analytics (see 7.6.6) on hybrid data at rest. Integration in PolicyCLOUD follows three directions: (i) architecture integration, (ii) integration with the cloud infrastructure and (iii) integration with Use Case scenarios through the implementation of end-to- end scenarios. Additional integration activities take place along the two frameworks of PolicyCLOUD, (a) the Data Governance model, protection and privacy enforcement mechanism and (b) the Ethical and Legal Compliance framework. For end-to-end data path analysis we have used two Use Case scenarios: (i) the scenario of Use Case 1: “Radicalization incidents” and the scenario of Use Case 2: “Visualization of negative and positive opinions on social networks for different products”. The new updates in this final document provide the following: Analysis of how External Frameworks can be integrated with PolicyCLOUD (section 7.6.11.4); Presentation of the overall Conceptual View and architecture of the Data Marketplace (section 7.9.1); Outline of the mechanisms developed for initialising the Policy Development Toolkit with Policy Model components and the visualization of results (section 7.8.3); Analysis of the Ethical and Legal Compliance Framework positive interventions to the PolicyCLOUD architecture, including the addition of specific fields/parameters to the registration Application Programming Interfaces to be populated with details regarding each individual analytics tool and dataset/data source (section 7.5); Presentation of the integration of the Data Governance model, protection and privacy enforcement mechanisms with the Policy Development Toolkit, the cloud gateways and the marketplace (section 7.10.2), and within the same context, the integration of EGI-Check-in with Keycloak including the integration of the Data Governance model, protection and privacy enforcement mechanisms with the Kubernetes cluster. The document also addresses the Reviewers’ comments to the previous version of the deliverable (Deliverable D2.6), included in the second review report. In order to address these comments, additional updates of Deliverable D2.7 include: (i) links to specific user/stakeholder requirements (D2.5), (ii) descriptions and implementation details for the two remaining pilot Use Cases (Sofia and London) and (iii) reference to EOSC and to the role of the Conceptual Model & Reference Architecture document for the identification of the relevant services and of their providers, and description of the onboarding process based on Deliverable D3.4 [22]., This deliverable is submitted to the EC, not yet approved.

The document outlines the initial training plan that will be delivered during the project and will provide a first set of training contents about the C-SCALE services. It discusses the target audiences, creates the structure of the training plan with an initial outline of training material and gives information on how the access to the material will be organised.