Search

interTwin Work Package 7 will provide a set of reusable components, called thematic modules, for the digital twins defined in WP4. In this document, we provide a high-level description of the environmental domain use cases and of the designed thematic modules. We also provide a set of common requirements derived from the analysis of the use cases and thematic modules that will need to be available in the Digital Twin Engine (DTE) core, designed in WP6.

In the context of XF-ACTORS project, a IoT based monitoring system has been established and tested in the field - for an olive grove in Puglia (Italy) affected by Xf - to measure in near real-time some parameters proxies of trees’ conditions and vulnerability under both abiotic (climate) and biotic factors. The system is based on the TreeTalker (TT) technology, comprising multiparametric sensors to monitor water transport in trees, trunk humidity and diametrical oscillations, spectral characteristics of the leaves and microclimatic parameters (temperature, relative humidity). In particular, the sap flow density can be retrieved according to the Heat Balance Method (Granier 1985) after measuring the temperature of two 20 mm long probes inserted into the stem wood at 10 cm distance along the trunk vertical axis; the probe in the higher position is heated while the lower one provides the reference temperature. The TT system collects and transmits data at hourly time frequency, thanks to a LoRa based wireless connection, to a node managed by another microcontroller (TT-Cloud) serving a few tens of devices in a cluster. The TT-Cloud is in turn connected to the internet via the GPRS network and sends data to a computer server. Here, raw data are subject to ETL procedure that allows data Extraction from the TT-Cloud source, data Transformation by cleaning and converting them into variables with eco-physiological meaning, and finally data Loading to insert them into the target spatio-temporal database, adopting proper storage format/structure for querying and analysis purpose. From here, data can be further elaborated and visualized, e.g. into useful statistics, through a tailored Data Science environment. The preliminary results on sap flow density are here presented as they can give important information about the impact of Xf that is known obstructing xylem vessels, reducing hydraulic conductivity and thus affecting evapotranspiration.

Reliable and timely information on the environmental conditions at sea is key to the safety of professional and recreational users as well as to the optimal execution of their activities. The possibility of users obtaining environmental information in due time and with adequate accuracy in the marine and coastal environment is defined as sea situational awareness (SSA). Without adequate information on the environmental meteorological and oceanographic conditions, users have a limited capacity to respond, which has led to loss of lives and to large environmental disasters with enormous consequent damage to the economy, society and ecosystems. Within the framework of the TESSA project, new SSA services for the Mediterranean Sea have been developed. In this paper we present SeaConditions, which is a web and mobile application for the provision of meteorological and oceanographic observation and forecasting products. Model forecasts and satellite products from operational services, such as ECMWF and CMEMS, can be visualized in SeaConditions. In addition, layers of information related to bathymetry, sea level and ocean-colour data (chl a and water transparency) are displayed. Ocean forecasts at high spatial resolutions are included in the version of SeaConditions presented here. SeaConditions provides a user-friendly experience with a fluid zoom capability, facilitating the appropriate display of data with different levels of detail. SeaConditions is a single point of access to interactive maps from different geophysical fields, providing high-quality information based on advanced oceanographic models. The SeaConditions services are available through both web and mobile applications. The web application is available at www.sea-conditions.com and is accessible and compatible with present-day browsers. Interoperability with GIS software is implemented. User feedback has been collected and taken into account in order to improve the service. The SeaConditions iOS and Android apps have been downloaded by more than 105000 users to date (May 2016), and more than 100000 users have visited the web version.

[EN] Climate and biodiversity systems are closely linked across a wide range of scales. To better understand the mutual interaction between climate change and biodiversity there is a strong need for multidisciplinary skills, scientific tools, and access to a large variety of heterogeneous, often distributed, data sources. Related to that, the EUBrazilCloudConnect project provides a user-oriented research environment built on top of a federated cloud infrastructure across Europe and Brazil, to serve key needs in different scientific domains, which is validated through a set of use cases. Among them, the most data-centric one is focused on climate change and biodiversity research. As part of this use case, the BioClimate Science Gateway has been implemented to provide end-users transparent access to (i) a highly integrated user-friendly environment, (ii) a large variety of data sources, and (iii) different analytics & visualization tools to serve a large spectrum of users needs and requirements. This paper presents a complete overview of BioClimate and the related scientific environment, in particular its Science Gateway, delivered to the end-user community at the end of the project., This work was supported by the EU FP7 EUBrazilCloudConnect Project (Grant Agreement 614048), and CNPq/Brazil (Grant Agreement no 490115/2013-6).

The continuous increase in the data produced by simulations, experiments and edge components in the last few years has forced a shift in the scientific research process, leading to the definition of a fourth paradigm in Science, concerning data-intensive computing. This data deluge, in fact, introduces various challenges related to big data volumes, formats heterogeneity and the speed in the data production and gathering that must be handled to effectively support scientific discovery. To this end, High Performance Computing (HPC) and data analytics are both considered as fundamental and complementary aspects of the scientific process and together contribute to a new paradigm encompassing the efforts from the two fields called High Performance Data Analytics (HPDA). In this context, the Ophidia project provides a HPDA framework which joins the HPC paradigm with scientific data analytics. This contribution presents some aspects regarding the Ophidia HPDA framework, such as the multidimensional storage model, its distributed and hierarchical implementation along with a benchmark of a parallel in-memory time series reduction operator.

Journal of grid computing 16(3), 381 - 408 (2018). doi:10.1007/s10723-018-9453-3, This paper describes the achievements of the H2020 project INDIGO-DataCloud. The project has provided e-infrastructures with tools, applications and cloud framework enhancements to manage the demanding requirements of scientific communities, either locally or through enhanced interfaces. The middleware developed allows to federate hybrid resources, to easily write, port and run scientific applications to the cloud. In particular, we have extended existing PaaS (Platform as a Service) solutions, allowing public and private e-infrastructures, including those provided by EGI, EUDAT, and Helix Nebula, to integrate their existing services and make them available through AAI services compliant with GEANT interfederation policies, thus guaranteeing transparency and trust in the provisioning of such services. Our middleware facilitates the execution of applications using containers on Cloud and Grid based infrastructures, as well as on HPC clusters. Our developments are freely downloadable as open source components, and are already being integrated into many scientific applications., Published by Springer Nature, Dordrecht

The MEDiterranean Monitoring and Forecasting Center (Med-MFC) is part of the Copernicus Marine Environment Monitoring Service (CMEMS, http://marine.copernicus.eu/), provided on an operational mode by Mercator Ocean in agreement with the European Commission. Specifically, Med MFC system provides regular and systematic information about the physical state of the ocean and marine ecosystems for the Mediterranean Sea. The Med-MFC service started in May 2015 from the pre-operational system developed during the MyOcean projects, consolidating the understanding of regional Mediterranean Sea dynamics, from currents to biogeochemistry to waves, interfacing with local data collection networks and guaranteeing an efficient link with other Centers in Copernicus network. The Med-MFC products include analyses, 10 days forecasts and reanalysis, describing currents, temperature, salinity, sea level and pelagic biogeochemistry. Waves products will be available in MED-MFC version in 2017. The consortium, composed of INGV (Italy), HCMR (Greece) and OGS (Italy) and coordinated by the Euro-Mediterranean Centre on Climate Change (CMCC, Italy), performs advanced R&D activities and manages the service delivery. The Med-MFC infrastructure consists of 3 Production Units (PU), for Physics, Biogechemistry and Waves, a unique Dissemination Unit (DU) and Archiving Unit (AU) and Backup Units (BU) for all principal components, guaranteeing a resilient configuration of the service and providing and efficient and robust solution for the maintenance of the service and delivery. The Med-MFC includes also an evolution plan, both in terms of research and operational activities, oriented to increase the spatial resolution of products, to start wave products dissemination, to increase temporal extent of the reanalysis products and improving ocean physical modeling for delivering new products. The scientific activities carried out in 2015 concerned some improvements in the physical, biogeochemical and wave components of the system. Regarding the currents, new grid-point EOFs have been implemented in the Med-MFC assimilation system; the climatological CMAP precipitation was replaced by the ECMWF daily precipitation; reanalysis time-series have been increased by one year. Regarding the biogeochemistry, the main scientific achievement is related to the implementation of the carbon system in the Med-MFC biogeochemistry model system already available. The new model is able to reproduce the principal spatial patterns of the carbonate system variables in the Mediterranean Sea. Further, a key result consists of the calibration of the new variables (DIC and alkalinity), which serves to the estimation of the accuracy of the new products to be released in the next version of the system (i.e. pH and pCO2 at surface). Regarding the waves, the system has been validated against in-situ and satellite observations. For example, a very good agreement between model output and in-situ observations has been obtained at offshore and/or well-exposed wave buoys in the Mediterranean Sea.

Climate and biodiversity systems are closely interlaced across a wide range of scales. To better understand the mutual interaction between climate change and biodiversity there is a strong need for multidisciplinary skills, tools and a large variety of heterogeneous, distributed data sources. In this regard, the EUBrazilCloudConnect project provides a user-centric research environment built on top of a federated cloud infrastructure across Europe and Brazil to serve scientific needs. One of the test cases implemented in this project focuses on climate change and biodiversity research. The BioClimate is the Science Gateway of the use case. It aims at providing end-users with a highly integrated environment, addressing mainly data analytics requirements. This paper presents a complete overview about BioClimate and the scientific environment delivered to the user community at the end of the project.

An efficient, secure, and interoperable data platform solution has been developed in the TESSA project to provide fast navigation and access to the data stored in the data archive, as well as a standard-based metadata management support. The platform mainly targets scientific users and the Situational Sea Awareness high-level services such as the Decision Support Systems (DSS). These datasets are accessible through the following three main components: the Data Access Service (DAS), the Metadata Service, and the Complex Data Analysis Module (CDAM). The DAS allows access to data stored into the archive by providing interfaces for different protocols. OPeNDAP, THREDDS, and WMS are just some of the solutions that have been integrated into the TESSA infrastructure. Metadata Service is the heart of the information system of the TESSA products and completes the overall infrastructure for data and metadata management. This component enables data search & discovery, and addresses interoperability by exploiting ISO standards for geospatial data (ISO 19115 and ISO 19139). Finally, the CDAM represents the back-end of the TESSA DSS by performing on-demand complex data analysis tasks.

A case study on climate models intercomparison data analysis addressing several classes of multi-model experiments is being implemented in the context of the EU H2020 INDIGO-DataCloud project. Such experiments require the availability of large amount of data (multi-terabyte order) related to the output of several climate models simulations as well as the exploitation of scientific data management tools for large-scale data analytics. More specifically, the paper discusses in detail a use case on precipitation trend analysis in terms of requirements, architectural design solution, and infrastructural implementation. The experiment has been tested and validated on CMIP5 datasets, in the context of a large scale distributed testbed across EU and US involving three ESGF sites (LLNL, ORNL, and CMCC) and one central orchestrator site (PSNC).

The availability of systems able to process and analyse big amount of data has boosted scientific advances in several fields. Workflows provide an effective tool to define and manage large sets of processing tasks. In the big data analytics area, the Ophidia project provides a cross-domain big data analytics framework for the analysis of scientific, multi-dimensional datasets. The framework exploits a server-side, declarative, parallel approach for data analysis and mining. It also features a complete workflow management system to support the execution of complex scientific data analysis, schedule tasks submission, manage operators dependencies and monitor jobs execution. The workflow management engine allows users to perform a coordinated execution of multiple data analytics operators (both single and massive - parameter sweep) in an effective manner. For the definition of the big data analytics workflow, a JSON schema has been properly designed and implemented. To aid the definition of the workflows, a visual design language consisting of several symbols, named Data Analytics Workflow Modelling Language (DAWML), has been also defined.

Grid applications need to access, integrate, manage and process huge amount of data stored within distributed and heterogeneous databases. In order to develop high level data grid services able to federate/integrate data sources, low level grid data access services must be very modular and extensible as well as very efficient and robust. In the last few years several projects provided different grid enabled services trying to face these important challenges. One of them is the GRelC Project which provides the GRelC Data Access Service solution to access both relational and non-relational data sources in a grid environment. In this paper we highlight the asynchronous query mechanisms provided by the GRelC Data Access Service, a WS-I based and GSI/VOMS enabled data grid access service. We also report on the architectural design issues, one example of web enabled asynchronous query (HTML query), the user support, etc.

Increasingly, grid computing is becoming the paradigm of choice for building large-scale complex scientific applications. These applications are characterized as being computationally and/or data intensive, requiring computational power and storage resources well beyond the capability of a single computer. Grid environments provide distributed, geographically spread computing and storage resources made available to scientists belonging to Virtual Organizations; resource sharing is tightly controlled across multiple administrative domains through established service-level agreements. The adoption of Service-Oriented Architectures leads to grid environments characterized by grid services built using Web Services technologies that can be composed as needed to create arbitrarily complex workflows. In this context, security is a key issue that must be taken into account; another concern is interoperability among grids, a fundamental building block to develop grid-aware applications that can benefit from multiple grid environments. We present the GSI plug-in for gSOAP, an open source solution to the problem of securing Web Services in grid environments providing full interoperability between grid environments based on the Globus Toolkit and gLITE middleware.

The Ophidia project is a research effort addressing big data analytics requirements, issues, and challenges for eScience. We present here the Ophidia analytics framework, which is responsible for atomically processing, transforming and manipulating array-based data. This framework provides a common way to run on large clusters analytics tasks applied to big datasets. The paper highlights the design principles, algorithm, and most relevant implementation aspects of the Ophidia analytics framework. Some experimental results, related to a couple of data analytics operators in a real cluster environment, are also presented.

This work introduces Ophidia, a big data analytics research effort aiming at supporting the access, analysis and mining of scientific (n-dimensional array based) data. The Ophidia platform extends, in terms of both primitives and data types, current relational database system implementations (in particular MySQL) to enable efficient data analysis tasks on scientific array-based data. To enable big data analytics it exploits well-known scientific numerical libraries, a distributed and hierarchical storage model and a parallel software framework based on the Message Passing Interface to run from single tasks to more complex dataflows. The current version of the Ophidia platform is being tested on NetCDF data produced by CMCC climate scientists in the context of the international Coupled Model Intercomparison Project Phase 5 (CMIP5).

Grid portals are web gateways aiming at concealing the underlying infrastructure through a pervasive, transparent, user-friendly, ubiquitous and seamless access to heterogeneous and geographically spread resources (i.e. storage, computational facilities, services, sensors, network and databases). The Climate- G Portal is the web gateway of the Climate-G testbed (an interdisciplinary research effort involving scientists both in Europe and US) and it is devoted to climate change research studies. The main goal of this paper is to present the Climate-G Portal providing a complete understanding of the international context, discussing its main requirements, challenges, architecture and key functionalities, and finally carrying out and presenting a multi-dimensional analysis of the Climate-G Portal, starting from a general schema proposed and discussed in this work.

Over the last 20 years, the open-source community has provided more and more software on which the world’s high-performance computing systems depend for performance and productivity. The community has invested millions of dollars and years of effort to build key components. However, although the investments in these separate software elements have been tremendously valuable, a great deal of productivity has also been lost because of the lack of planning, coordination, and key integration of technologies necessary to make them work together smoothly and efficiently, both within individual petascale systems and between different systems. It seems clear that this completely uncoordinated development model will not provide the software needed to support the unprecedented parallelism required for peta/ exascale computation on millions of cores, or the flexibility required to exploit new hardware models and features, such as transactional memory, speculative execution, and graphics processing units. This report describes the work of the community to prepare for the challenges of exascale computing, ultimately combing their efforts in a coordinated International Exascale Software Project.

Climate-G is a large scale distributed testbed devoted to climate change research. It is an unfunded effort started in 2008 and involving a wide community both in Europe and US. The testbed is an interdisciplinary effort involving partners from several institutions and joining expertise in the field of climate change and computational science. Its main goal is to allow scientists carrying out geographical and cross-institutional data discovery, access, analysis, visualization and sharing of climate data. It represents an attempt to address, in a real environment, challenging data and metadata management issues. This paper presents a complete overview about the Climate-G testbed highlighting the most important results that have been achieved since the beginning of this project.

Grid computing is a well-known technology to share resources such as high-performance computers, sensors, observation devices, data and databases across dynamic and multi-institutional Virtual Organizations (VOs). The complexity of data management in a grid environment comes from the distribution, scale, growing rate, heterogeneity, dynamicity of data sources. The data grid layer (grid data management system) in the overall grid computing software stack must provide a complete support in terms of access, integration, replication, monitoring, mining, management of data sources in a grid environment. The main goal of this work is to present the data access layer of the GRelC system architecture discussing in detail the vision, the main challenges, the internal architecture, the security framework and a real test case in the Earth Science and Environmental domains.

Exascale eScience infrastructures will face important and critical challenges, both from computational and data perspectives. Increasingly complex and parallel scientific codes will lead to the production of a huge amount of data. The large volume of data and the time needed to locate, access, analyze and visualize data will greatly impact on the scientific productivity of scientists and researchers in several domains. Significant improvements in the data management field will increase research productivity in solving complex scientific problems. The exascale scenario will involve a lot of distributed data available at an international level across several countries. Several application domains will produce large volumes of data. For instance, concerning the climate change domain, hundreds of exabytes of data (distributed across several centers) are expected to be available through heterogeneous storage resources (located in data centers as well as in external environments like data grids and data clouds) for access, analysis, post-processing and other scientific activities. Collections of data will be stored at different sites and made available to the users for further analysis and studies. The same will happen for other domains, where exascale high-performance computing applications will generate data at a very high rate (terabytes/s) on million of cores. In the paper, the main challenges that must be taken into account in the exascale context are presented and discussed.

Grid computing is an emerging and enabling technology allowing organizations to easily share, integrate and manage resources in a distributed environment. Computational Grid allows running millions of jobs in parallel, but the huge amount of generated data has caused another interesting problem: the management (classification, storage, discovery etc.) of distributed data, i.e., a Data Grid specific issue. In the last decade, many efforts concerning the management of data (grid-storage services, metadata services, grid-database access and integration services, etc.) identify data management as a real challenge for the next generation petascale grid environments. This work provides an architectural overview of the GRelC DAS, a grid database access service developed in the context of the GRelC Project and currently used for production/tutorial activities both in gLite and Globus based grid environments.

In Bioinformatics, a key issue in sequence analysis is the determination of similarity between biological sequences, that is the percentage of sequence matches among nucleotide or protein sequences. The hypothesis is that similarity relates to functionality: if two sequences are similar, they will have related functionalities. Sequence databases are exploding in size, growing at an exponential rate, currently doubling in about 12 months, exceeding the rate of growth in compute cycles that doubles only every 18 months (Moore’s law). Whenever sequence databases are significantly updated, alignment should be repeated many times to discover new information. Grid computing together with parallel alignment tools, wrapped as Web Services, are crucial techniques to maintain and improve the effectiveness of sequence comparison tools. Partitioning the load into different jobs for each simulation is a good choice for the alignment applied to a large dataset, because the input data, composed by a set of sequences (generally in FastA format), are compared with biological databases that are made of a set of sequences in various formats (flat files and FastA). Therefore, it is possible to parallelize the execution by splitting both the input dataset and/or the database, sending each data partition to a Grid node, and merging the results. The solution proposed in this paper is based on the possibility to manage the untapped processing power of desktop PCs within an enterprise Grid network to process computationally intensive jobs for scientific applications and in particular for the bioinformatics domain. This paper describes BioGAT, Bioinformatics Grid Alignment Toolkit, that offers optimized brokering and a data management system to exploit various bioinformatics alignment tools wrapped as Web Services in a Grid architecture.

Data grid management systems are becoming increasingly important in the context of the recently adopted service oriented science paradigm. The Grid Relational Catalog (GRelC) project is working towards ubiquitous, integrated, seamless and comprehensive data grid management solutions to fully address application specific requirements. This paper describes a GRelC based environment for bioinformatics and its underlying data grid services allowing scientific users (by means of a customized grid portal) to manage data, handle, share and publish metadata, perform search and discovery activities, etc.

Current production Grids involve hundreds of sites and thousands of machines. In this context, P2P solutions are well suited - with regard to existing centralized and hierarchical approaches - to implement highly scalable, decentralized, reliable and manageable grid services. In this paper we describe the GReIC Data Gather Service from an architectural and technological point of view. This service has been developed within the Grid Relational Catalog (GReIC) Project, at the Center for Advanced Computational Technologies (CACT) of the University of Lecce. The GReIC Data Gather architecture aims at integrating transparently and securely distributed and geographically spread heterogeneous grid data sources through Data Gather Service nodes connected in a P2P fashion.

This paper presents a data Grid system, built on top of specific biological data sources in flat file format, which carries out the ingestion into a relational DBMS that integrates these data. The prototype has been implemented for UniProtKB (located at EBI - European Bioinformatics Institute, UK) and UTRdb (located at ITB/CNR Bari, Italy) data banks owing to the following two reasons: a public available relational schema of the UniProtKB and UTRdb does not exist; UniProtKB is the most complete repository of proteins whereas UTRdb contains mRNA nucleotides and although the relation between nucleotides and proteins could be important for several studies, an explicit management of such relationship (cross-referenced link) is not yet available. The system also allows transparent, periodic update of both the DBMS and the involved data banks. Each component is a GSI (Grid Security Infrastructure) enabled Web Service, exploiting the gSOAP Toolkit; the system utilizes several grid nodes to carry out the data ingestion faster whilst reducing the redundance of data present into the flat files.

Today many DataGrid applications need to manage and process a very large amount of data distributed across multiple grid nodes and stored into heterogeneous databases. Grids encourage and promote the publication, sharing and integration of scientifica data (distributed across several Virtual Organizations) in a more open manner than is currently the case, and many e-Science pojects have an urgent need to interconnect legacy and independently operated databases through a set os data access and integration services. The complexity of data management within a Computational Grid comes from the distribution, scale and heterogeneity of data sources. A set of dynamic and adaptive services could address specific issues related to automatic data management providing high performance and transparency as well as fully exploiting a grid infrastructure. These services should involved data migration and integration, discovery of data sources and so on, providing a transparent and dynamic layer of data virtualization. In this pape we introduce the Grid-DBMS concept, a framework for dynamic data management in a grid enviroment, highlighting its requirements, architecture, components and services. We also present an overview about the Grid Relational Catalog Project (GRelC) developed at the CACT/ISUFI of the University of Lecce, which represents a partial implementation of a Grid-DBMS for the Globus Community.

Nowadays many data grid applications need to manage and process a huge amount of data distributed across multiple grid nodes and stored into heterogeneous databases. Grids encourage the publication of scientific data in a more open manner than is currently the case, and many e-Science projects have an urgent need to interconnect legacy, and independently operated databases through a set of data access and integration services. In the data grid area a set of dynamic and adaptive services could address specific issues related to automatic data management aiming at both providing high performance and fully exploiting a grid infrastructure. In this paper we introduce the grid-DBMS concept, a framework for dynamic data management in grid environments, highlighting its requirements, architecture, components and services.

Important issues in bioinformatics are the difficulties for non computer experts to use bioinformatics tools, the transparent access to large biological data sets, and the exploitation of large computing power. Moreover, often such tools and databases are developed by independent groups, so the task of enabling their composition and cooperation is even more difficult. Integrating Computational Grid and Web Services technologies can be a key solution to simplify interaction between bioinformatics tools and biological databases. This paper presents ProGenGrid (Proteomics & Genomics Grid), a distributed and ubiquitous grid environment, accessible through the web, for supporting “in silico” experiments in bioinformatics.

Several Data Grid applications need to manage a lot of data distributed across heterogeneous and wide spreaded resources and stored in Relational Databases. RDBMSs (Relational Database Management Systems) are not grid enabled (with the notable exception of Oracle), so in order to provide security, transparency, robustness, efficiency and dynamic mechanisms in a Grid environment a new concept can be introduced: the Grid-DBMS. After defining it, we talk about an implementation built on top of the Globus Toolkit: the Grid Relational Catalog Project (GRelC) de veloped at the CACT/ISUFI Laboratory of the University of Lecce. Then we present the basic architecture discussing about its main features and components.

The Information Service is a fundamental component in a grid environment. It has to meet a lot of requirements such as access to static and dynamic information related to grid resources, efficient and secure access to dynamic data, decentralized maintenance, fault tolerance etc., in order to achieve better performance, scalability, security and extensibility. Currently there are two different major approaches. One is based on a directory infrastructure and another one on a novel approach that exploits a relational DBMS. In this paper we present a performance comparison analysis between Grid Resource Information Service (GRIS) and Local Dynamic Grid Catalog relational information service (LDGC)

In this paper we present the approach proposed by EU H2020 INDIGO-DataCloud project to orchestrate dynamic workflows over a cloud environment. The main focus of the project is on the development of open source Platform as a Service solutions targeted at scientific communities, deployable on multiple hardware platforms, and provisioned over hybrid e-Infrastructures. The project is addressing many challenging gaps in current cloud solutions, responding to specific requirements coming from scientific communities including Life Sciences, Physical Sciences and Astronomy, Social Sciences and Humanities, and Environmental Sciences. We are presenting the ongoing work on implementing the whole software chain on the Infrastructure as a Service, PaaS and Software as a Service layers, focusing on the scenarios involving scientific workflows and big data analytics frameworks. INDIGO module for Kepler worflow system has been introduced along with the INDIGO underlying services exploited by the workflow components. A climate change data analytics experiment use case regarding the precipitation trend analysis on CMIP5 data is described, that makes use of Kepler and big data analytics services.

The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project., This work has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927). In Germany, it has received complementary funding from the German Federal Ministry of Education and Research (contracts 16HPC016K, 6GPC016K, 16HPC017 and 16HPC018). In France, it has received financial support from Caisse des dépôts et consignations (CDC) under the action PIA ADEIP (project Calculateurs). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). In Norway, it has received complementary funding from the Norwegian Research Council, Norway under project number 323825. In Switzerland, it has been preliminary approved for complimentary funding by the State Secretariat for Education, Research, and Innovation (SERI), Norway. In Poland, it is partially supported by the National Centre for Research and Development under decision DWM/EuroHPCJU/4/2021. The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the “Severo Ochoa Programme for Centres of Excellence in R&D” under Grant CEX2018-000797-S, the Spanish Government, Spain (contract PID2019-107255 GB) and by Generalitat de Catalunya, Spain (contract 2017-SGR-01414). Anna Queralt is a Serra Húnter Fellow., With funding from the Spanish government through the ‘Severo Ochoa Centre of Excellence’ accreditation (CEX2018-000797-S).

Over the last two decades, scientific discovery has increasingly been driven by the large availability of data from a multitude of sources, including high-resolution simulations, observations and instruments, as well as an enormous network of sensors and edge components. In such a dynamic and growing landscape where data continue to expand, advances in Science have become intertwined with the capacity of analysis tools to effectively handle and extract valuable information from this ocean of data. In view of the exascale era of supercomputers that is rapidly approaching, it is of the utmost importance to design novel solutions that can take full advantage of the upcoming computing infrastructures. The convergence of High Performance Computing (HPC) and data-intensive analytics is key to delivering scalable High Performance Data Analytics (HPDA) solutions for scientific and engineering applications. The aim of this paper is threefold: reviewing some of the most relevant challenges towards HPDA at scale, presenting a HPDA-enabled version of the Ophidia framework and validating the scalability of the proposed framework through an experimental performance evaluation carried out in the context of the Centre of Excellence in Simulation of Weather and Climate in Europe (ESiWACE). The experimental results show that the proposed solution is capable of scaling over several thousand cores and hundreds of cluster nodes. The proposed work is a contribution in support of scientific large-scale applications along the wider convergence path of HPC and Big Data followed by the scientific research community.

Presentation of the paper "Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale" at the IEEE BigData2018 Conference (Seattle, WA, USA)

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro-Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP). This paper reports about some ESiWACE WP3 activities described in the deliverable D3.10 "ESiWACE Scheduler development and support activities"

In the context of XF-ACTORS project, a IoT based monitoring system has been established and tested in the field - for an olive grove in Puglia (Italy) affected by Xf - to measure in near real-time some parameters proxies of trees’ conditions and vulnerability under both abiotic (climate) and biotic factors. The system is based on the TreeTalker (TT) technology, comprising multiparametric sensors to monitor water transport in trees, trunk humidity and diametrical oscillations, spectral characteristics of the leaves and microclimatic parameters (temperature, relative humidity). In particular, the sap flow density can be retrieved according to the Heat Balance Method (Granier 1985) after measuring the temperature of two 20 mm long probes inserted into the stem wood at 10 cm distance along the trunk vertical axis; the probe in the higher position is heated while the lower one provides the reference temperature. The TT system collects and transmits data at hourly time frequency, thanks to a LoRa based wireless connection, to a node managed by another microcontroller (TT-Cloud) serving a few tens of devices in a cluster. The TT-Cloud is in turn connected to the internet via the GPRS network and sends data to a computer server. Here, raw data are subject to ETL procedure that allows data Extraction from the TT-Cloud source, data Transformation by cleaning and converting them into variables with eco-physiological meaning, and finally data Loading to insert them into the target spatio-temporal database, adopting proper storage format/structure for querying and analysis purpose. From here, data can be further elaborated and visualized, e.g. into useful statistics, through a tailored Data Science environment. The preliminary results on sap flow density are here presented as they can give important information about the impact of Xf that is known obstructing xylem vessels, reducing hydraulic conductivity and thus affecting evapotranspiration., IT; PPT; monia.santini@cmcc.it

ESiWACE is the Centre of Excellence in SImulation of Weather and Climate in Europe.To improve efficiency and productivity of numerical weather and climate simulation on high performance computing platforms ESiWACE has improved and supported scalability of models, tools and data management on state-of-the-art supercomputer systems, usability of models and tools throughout the European HPC eco-system, and the exploitability of the huge amount of resulting data. In its second phase, ESiWACE will enable leading European weather and climate models to leverage the performance of pre-exascale systems with regard to both compute and data capacity as soon as possible and prepare the weather and climate community to be able to make use of exascale systems when they become available. With respect to the workshop topic, ESiWACE will improve the data management tool chain from weather and climate simulations to address data challenges related to I/O, data analytics,post-processing and visualization at scale, through the development of specific middleware components (ESDM). The talk addresses in particular a set of activities in the data management area, from I/O to HPDA. It gives an overview about what has been already accomplished and what is planned as a future work until the end of the second phase of the project (December 2022). A test case related to the Ophidia HPDA framework is also presented in a real HPC settings on the Athena Cluster available at the CMCC SuperComputing Centre.

Preserving individual privacy is one of the major issues in the context of Big Data, since handling huge volumes of data may contribute to the disclosure of sensitive or personally identifiable information. In fact, even when data is anonymized there is a risk of re-identification through privacy attacks. This paper presents a re-identification risk-based anonymization framework for big data analytics platforms. This framework is based on anonymization policies and allows applying anonymization techniques and models in two stages: during the ETL process and before exporting the statistical results of data analytics. This second stage evaluates the data re-identification risk and increases the anonymity level if it is necessary to reduce this risk. Although generic, the implementation of the framework reported in this work was integrated into Ophidia as a case study. Privacy attacks were performed to check the effectiveness of the re-identification process. Results are promising, showing a low probability of re-identification in two different scenarios.