Search

The exponential increase in data volumes and complexities is causing a radical change in the scientific discovery process in several domains, including climate science. This affects the different stages of the data lifecycle, thus posing significant data management challenges in terms of data archiving, access, analysis, visualization, and sharing. The data space concept can support scientists' workflow and simplify the process towards a more FAIR use of data.In the context of the European Open Science Cloud (EOSC) initiative launched by the European Commission, the ENES Data Space (EDS) represents a domain-specific implementation of the data space concept. The service, developed in the frame of the EGI-ACE project, aims to provide an open, scalable, cloud-enabled data science environment for climate data analysis on top of the EOSC Compute Platform. It is accessible in the European Open Science Cloud (EOSC) through the EOSC Catalogue and Marketplace (https://marketplace.eosc-portal.eu/services/enes-data-space) and it also provides a web portal (https://enesdataspace.vm.fedcloud.eu) including information, tutorials and training materials on how to get started with its main features. The EDS integrates into a single environment ready-to-use climate datasets, compute resources and tools, all made available through the Jupyter interface, with the aim of supporting the overall scientific data processing workflow. Specifically, the data store linked to the ENES Data Space provides access to a multi-terabyte set of variable-centric collections from large-scale global climate experiments. The data pool consists of a mirrored subset of CMIP (Coupled Model Intercomparison Project) datasets from the ESGF (Earth System Grid Federation) federated data archive, collected and kept synchronized with the remote copies by using the Synda tool developed within the scope of the IS-ENES3 H2020 project. Community-based, open source frameworks (e.g., Ophidia) and libraries from the Python ecosystem provide the capabilities for data access, analysis and visualisation. Results and experiment definitions (i.e., Jupyter Notebooks) can be easily shared among users promoting data sharing and application re-use towards a more Open Science approach. An overview of the data space capabilities along with the key aspects in terms of data management will be presented in this work.

Typical end-to-end Earth System Modelling (ESM) workflows rely on different steps including data pre-processing, numerical simulation, output post-processing, as well as data analytics and visualization. The approaches currently available for implementing scientific workflows in the climate context do not properly integrate the entire set of components into a single workflow and in a transparent manner. The increasing usage of High Performance Data Analytics (HPDA) and Machine Learning (ML) in climate applications further exacerbate the issues. A more integrated approach would allow to support next-generation ESM and improve the workflow in terms of execution and energy consumption.Moreover, a seamless integration of components for HPDA and ML into the ESM workflow will open the floor to novel applications and support larger scale pre- and post-processing. However, these components typically have different deployment requirements spanning from HPC (for ESM simulation) to Cloud computing (for HPDA and ML). It is paramount to provide scientists with solutions capable of hiding the technical details of the underlying infrastructure and improving workflow portability.In the context of the eFlows4HPC project, we are exploring the use of innovative workflow solutions integrating approaches from HPC, HPDA and ML for supporting end-to-end ESM simulations and post-processing, with a focus on extreme events analysis (e.g., heat waves and tropical cyclones). In particular, the envisioned solution exploits PyCOMPSs for the management of parallel pipelines, task orchestration and synchronization, as well as PyOphidia for climate data analytics and ML frameworks (i.e., TensorFlow) for data-driven event detection models. This contribution presents the approaches being explored in the frame of the project to address the convergence of HPC, Big Data and ML into a single end-to-end ESM workflows.

New climate regimes, variability and extreme events affect the livestock sector in many aspects, ranging from animal welfare, production, reproduction, diseases and their spread, feed quality and availability. Heat stress, especially when combined with excess or low humidity, exacerbates the perceived temperature or the drought conditions, respectively, increasing hazards for animals. Also, cold extremes, extraordinary windy conditions and altered radiation regimes are detrimental to both animals and fodder.In this context, the EU-funded SEBASTIEN project aims to provide stakeholders with a Decision Support System (DSS) for more efficient and sustainable management, and consequent valuation, of the livestock sector in Italy. SEBASTIEN DSS will integrate GIS, environmental and biological variables to generate updated risk maps for livestock diseases and zoonoses and their spread, alerting about the expected occurrence of stressing conditions for animals due to abiotic and biotic factors.The presence of parasites, vectors, and outbreaks will be combined with environmental data, gathered by spatially distributed meteorological and satellite monitoring, to detect conditions that can potentially favor or trigger the spread of related diseases. Sensor-based monitoring data will be integrated with the above information to determine ranges in animal parameters potentially associated with a higher risk of critical pathogen load or density of vectors potential carriers of diseases. Medium to long-term climate forecasts will support predicting possible shifts of favorable conditions that will open up new areas for parasites and pathogens. The vast amounts of data will be integrated and summarized into user-tailored information through a range of techniques, from empirical/statistical indicators to Machine Learning algorithms.

In the context of the European Open Science Cloud, the ENES Data Space represents a domain-specific implementation of the data space concept, a digital ecosystem supporting scientific communities towards a more sustainable, effective, and FAIR use of data. Such ecosystem has been recently opened to climate users, offering datasets, tools, and services into a single environment with ready-to-use data and programmatic capabilities for the development of data science applications. Presently, the data store of the ENES Data Space provides access to models output from large-scale global experiments for climate model intercomparison. The storage and computational resources for the execution of the data space are provided by the EGI Federated Cloud e-Infrastructure. From a science gateway perspective, the ENES Data Space provides an interactive web-based environment based on the Jupyter project and available through the EOSC MarketPlace.

The scientific discovery process has been deeply influenced by the data deluge started at the beginning of this century. This has caused a profound transformation in several scientific domains which are now moving towards much more collaborative processes. In the climate sciences domain, the ENES Data Space aims to provide an open, scalable, cloud-enabled data science environment for climate data analysis. It represents a collaborative research environment, deployed on top of the EGI federated cloud infrastructure, specifically designed to address the needs of the ENES community. The service, developed in the context of the EGI-ACE project, provides ready-to-use compute resources and datasets, as well as a rich ecosystem of open source Python modules and community-based tools (e.g., CDO, Ophidia, Xarray, Cartopy, etc.), all made available through the user-friendly Jupyter interface. In particular, the ENES Data Space provides access to a multi-terabyte set of specific variable-centric collections from large community experiments to support researchers in climate model data analysis experiments. The data pool of the ENES Data Space consists of a mirrored subset of CMIP datasets from the ESGF federated data archive collected by using the Synda community tool in order to provide the most up to date datasets into a single location. Results and output products as well as experiment definitions (in the form of Jupyter Notebooks) can be easily shared among users through data sharing services, which are also being integrated in the infrastructure, such as EGI DataHub.The service was opened in the second part of 2021 and is now accessible in the European Open Science Cloud (EOSC) through the EOSC Portal Marketplace (https://marketplace.eosc-portal.eu/services/enes-data-space). This contribution will present an overview of the ENES Data Space service and its main features.

The project OFIDIA2 (Operational FIre Danger preventIon plAtform 2), funded by the Interreg Greece-Italy 2014-2020 Programme, proposed a pragmatic approach to improve the operational capacity of the stakeholders to detect and fight forest wildfires. A data analytics system was designed and implemented within the project to manage, transform, and extract knowledge from heterogenous data sources, through forecasting models such as weather, fire danger, and fire behaviour models. The high-resolution weather forecasting network previously developed in OFIDIA1 was enhanced by using a mesoscale configuration of the WRF-ARW model over the Central Mediterranean Sea. A nested domain over the Southern Italy at ~2km horizontal resolution allows getting high-resolution weather forecasts (2x2km) and processing data into fire danger models. Fires, fuel, topography and weather data were collected from several sources and used to run and calibrate fire models (FlamMap and Wildfire Analyst) in Apulia region (Italy). Based on the analyses of recurrent weather conditions leading to large fires, fire metrics’ maps for prevention and fire-fighting activities were produced. Finally, a Decision Support System (DSS) was also developed to provide support for 1) the selection of fire behaviour scenarios by means of mathematical models; and 2) the prevention of emergencies thanks to weather forecast information with fire danger indices at high resolutions.

In the context of XF-ACTORS project, a IoT based monitoring system has been established and tested in the field - for an olive grove in Puglia (Italy) affected by Xf - to measure in near real-time some parameters proxies of trees’ conditions and vulnerability under both abiotic (climate) and biotic factors. The system is based on the TreeTalker (TT) technology, comprising multiparametric sensors to monitor water transport in trees, trunk humidity and diametrical oscillations, spectral characteristics of the leaves and microclimatic parameters (temperature, relative humidity). In particular, the sap flow density can be retrieved according to the Heat Balance Method (Granier 1985) after measuring the temperature of two 20 mm long probes inserted into the stem wood at 10 cm distance along the trunk vertical axis; the probe in the higher position is heated while the lower one provides the reference temperature. The TT system collects and transmits data at hourly time frequency, thanks to a LoRa based wireless connection, to a node managed by another microcontroller (TT-Cloud) serving a few tens of devices in a cluster. The TT-Cloud is in turn connected to the internet via the GPRS network and sends data to a computer server. Here, raw data are subject to ETL procedure that allows data Extraction from the TT-Cloud source, data Transformation by cleaning and converting them into variables with eco-physiological meaning, and finally data Loading to insert them into the target spatio-temporal database, adopting proper storage format/structure for querying and analysis purpose. From here, data can be further elaborated and visualized, e.g. into useful statistics, through a tailored Data Science environment. The preliminary results on sap flow density are here presented as they can give important information about the impact of Xf that is known obstructing xylem vessels, reducing hydraulic conductivity and thus affecting evapotranspiration.

The last few decades have seen a range of advances in climate science and consequential policy initiatives at both national and international levels. These advances have been built on the back of progress in modelling and in part been enabled by the global data sharing initiative - the Earth System Grid Federation (ESGF) - which has underpinned recent phases of the World Climate Research Programme's Coupled Model Intercomparison Projects. The ESGF itself consists of data nodes deployed by individual modelling centres and a backbone of software development and services delivered by a few core institutions. Within Europe, along with some shared development of model components, these core ESGF software development and services are coordinated by the European Network on Earth System Modelling (ENES) and supported by the H2020 IS-ENES Phase 3 research infrastructure project. We provide an historical overview on advances in policy-relevant science, such as the Intergovernmental Panel for Climate Change (IPCC), that have been enabled by long-term underpinning development and funding of the ENES and ESGF infrastructure. We illustrate the recent shift of research funding from physical science objectives alone towards funding services to society (and the necessary underpinning research). We stress the potential dangers of underfunding research infrastructures that need to be simultaneously flexible and reliable enough to serve both ongoing basic research and the growing societal objectives, as emphasised by the development of climate services such as Copernicus Climate Change Service. We conclude by presenting some steps towards sustaining such research infrastructure in the context of the ENES and the possible futures of climate science.

One of the main bottlenecks for NEMO scalability is the time spent performing communications. Two complementary strategies are here proposed to reduce the communication frequency and the communication time: the MPI3 neighbourhood collective communications instead of multiple point to point exchanges and the increasing of the halo region size.NEMO performs Lateral Boundaries Conditions update by using four point to point MPI communications at north, south, east and west for each MPI domain. The model completes east-west exchange before performing north-south communications. The order of the exchanges allows us to preserve both 5-point and 9-point stencils. MPI3 neighbourhood collectives provide a way to have sub-communicators used to perform collective communications. Two different sub-communicators can be defined in order to support the two different stencils. A single MPI message is needed to be built for all neighbours instead of 4 different messages before calling the collective communication, while the received message is used to update the halo region, following the order of the neighbours in the sub-communicator.The new communication strategy has been tested on two computational kernels (i.e. one for 5-point stencil and one for 9-point stencil), selected among the main relevant routines from the computational point of view. Preliminary tests, performed on a domain size of 3000x2000x31 grid points on the Zeus Intel Xeon Gold 6154 machine, available at CMCC, show a gain in communication time for the 5-point stencil use case up to 31% on 2016 cores. The improvement is reduced when communications with processes on the diagonal are activated. However, a modest gain is still achieved, depending on the number of cores.On the other side, the analysis of some NEMO routines shows how the exchange of more than one row/column of halo would allow to move communications outside the routine, preserving data dependencies. A wider halo size reduces the frequency of message exchanges whilst increases the message size at each exchange. It allows us to adopt some optimisation strategies (i.e. loop fusion, tiling, etc.) to improve the data locality. Nevertheless, the use of a wider halo introduces itself some improvements for some kernels like for the MUSCL advection scheme which shows a gain of ~23% in the execution time comparing the original version and the new one with halo extended to 2 lines and the communication moved outside the computing region.The current work has been performed according to the NEMO development strategy plan, defined by the NEMO Consortium, which establish the priorities of the design strategies to reduce the bottlenecks to the scalability and the time to solution. AcknowledgmentsThis work is co-funded by the EU H2020 IS-ENES project Phase 3 (ISENES3) under Grant Agreement number 824084.

In the roadmap of modern parallel architectures development, the computing power of a node grows much more quickly than main memory performance (capacity, bandwidth). This leads to an even much higher gap between computing and memory resources. An efficient use of the cache memory is becoming ever more essential as optimization technique.The NEMO model uses a finite difference integration method and a regular cartesian grid for space discretization. The NEMO code reflects this choice: a generic field is represented in memory as a 3D array; and the code is mainly composed of three-level nested loops. These loops often include only a few operations in the body; the results are stored in a temporary 3D array and then used in subsequent loops until the final calculation.The aim of this work is to make better use of the cache memory by fusing DO loops together. The loop fusion is a transformation which takes two or more adjacent loops that have the same iteration space traversal and combines their bodies into a single loop.The fusion of the loops is not trivial, and it could require introducing additional redundant operations to solve data dependencies. Unfortunately, this leads to a drawback of the overall performance. To avoid the redundant operation, we can adopt pointers to arrays and implement a pointer rotation at each loop iteration.We have developed the loop fusion transformation in an advection kernel extracted from the NEMO oceanic model. We have compared 3 different versions of the optimized advection kernel, with 3 different levels of loop fusion.The first prototype refers to the implementation where the extreme fusion is applied, and all loops in the routine have been fused. In this version, the operations are replicated up to 3 times. In the second prototype the buffer rotation has been applied only in the outermost loop. In the third prototype, the buffer rotation has also been implemented for the second dimension, and this version introduces only a limited amount of redundant operations.The tests have been performed on the Athena cluster located at the CMCC supercomputing center. The supercomputing infrastructure is based on the Intel Xeon E5-2670 processors. The memory hierarchy is composed of 32KB of L1 cache, 256KB of L2 and 20MB L3 cache shared among the cores. The results clearly proved the effectiveness of the loop fusion approach that reaches a speedup of 2x with a high number of cores. The third prototype has proven to be the most promising solution. Prototypes 1 and 2 provide a good improvement up to 256 cores then the redundant operations lead to a loss of performance.A deeper analysis measuring the Last Level Cache misses also showed how the loop transformation significantly reduced the number of cache misses.Despite the good results achieved with the loop fusion optimization, we can remark that this optimization is strictly linked to the computing architecture. A fully portable performance improvement can be ensured by the adoption of a DSL (Domain Specific Language).

Tropical cyclones (TCs) transport energy and moisture along their pathways interacting with the climate system and TCs activities are expected to extend further poleward during the 21st century.For this reason, it is important to assess the ability of state-of-the-art climate models in reproducing an accurate meridional distribution of TCs as well as a reasonable meridional portrait of moisture transport associated with TCs.Since high resolutions are required to reconstruct observed TCs activity, the present work is based on the simulations performed as part of HighResMIP in the framework of the community CMIP6 effort. To inspect this feature, two horizontal resolutions for each climate model are considered. Besides, the impact of boundary conditions, i.e. observed ocean surface state, is examined by considering both coupled and atmosphere-only configurations.In the present work, the north Atlantic region is analyzed as a sample region, while the same approach is applied on a multi-basin basis. In the sample area, climate models present a good ability in reproducing the TCs distribution, with a general underestimation at lower latitudes and a slight overestimation at high-latitudes compared to observed TCs tracks (e.g. IBTRACK).The meridional distribution of moisture transport associated with TCs is evaluated by considering the radial average of the integrated water vapor transport along the TC tracks. When compared to observation (IBTRACS and JRA-55 reanalysis), the simulated moisture transport associated with TCs displays reasonably good performance in atmosphere-only high-resolution models configuration. The interannual variability of water vapor associated with TCs, instead, is poorly represented in climate models.Climate models in high-resolution configuration can then be used in estimating future TCs meridional distribution and changes in meridional moisture transport associated with TCs.This effort is part of HighResMIP and it is developed in the framework of the EU-funded PRIMAVERA project.

Reliable and timely information on the environmental conditions at sea is key to the safety of professional and recreational users as well as to the optimal execution of their activities. The possibility of users obtaining environmental information in due time and with adequate accuracy in the marine and coastal environment is defined as sea situational awareness (SSA). Without adequate information on the environmental meteorological and oceanographic conditions, users have a limited capacity to respond, which has led to loss of lives and to large environmental disasters with enormous consequent damage to the economy, society and ecosystems. Within the framework of the TESSA project, new SSA services for the Mediterranean Sea have been developed. In this paper we present SeaConditions, which is a web and mobile application for the provision of meteorological and oceanographic observation and forecasting products. Model forecasts and satellite products from operational services, such as ECMWF and CMEMS, can be visualized in SeaConditions. In addition, layers of information related to bathymetry, sea level and ocean-colour data (chl a and water transparency) are displayed. Ocean forecasts at high spatial resolutions are included in the version of SeaConditions presented here. SeaConditions provides a user-friendly experience with a fluid zoom capability, facilitating the appropriate display of data with different levels of detail. SeaConditions is a single point of access to interactive maps from different geophysical fields, providing high-quality information based on advanced oceanographic models. The SeaConditions services are available through both web and mobile applications. The web application is available at www.sea-conditions.com and is accessible and compatible with present-day browsers. Interoperability with GIS software is implemented. User feedback has been collected and taken into account in order to improve the service. The SeaConditions iOS and Android apps have been downloaded by more than 105000 users to date (May 2016), and more than 100000 users have visited the web version.

An efficient, secure, and interoperable data platform solution has been developed in the TESSA project to provide fast navigation and access to the data stored in the data archive, as well as a standard-based metadata management support. The platform mainly targets scientific users and the Situational Sea Awareness high-level services such as the Decision Support Systems (DSS). These datasets are accessible through the following three main components: the Data Access Service (DAS), the Metadata Service, and the Complex Data Analysis Module (CDAM). The DAS allows access to data stored into the archive by providing interfaces for different protocols. OPeNDAP, THREDDS, and WMS are just some of the solutions that have been integrated into the TESSA infrastructure. Metadata Service is the heart of the information system of the TESSA products and completes the overall infrastructure for data and metadata management. This component enables data search & discovery, and addresses interoperability by exploiting ISO standards for geospatial data (ISO 19115 and ISO 19139). Finally, the CDAM represents the back-end of the TESSA DSS by performing on-demand complex data analysis tasks.

A new web-based and mobile decision support system (DSS) for search-and-rescue (SAR) at sea is presented, and its performance is evaluated using real case scenarios. The system, named OCEAN-SAR, is accessible via the website http://www.ocean-sar.com. In addition to the website, dedicated applications for iOS and Android have been created to optimise the user experience on mobile devices. OCEAN-SAR simulates drifting objects at sea, using as input ocean currents and wind data provided, respectively, by the CMEMS and ECMWF. The modelling of the drifting objects is based on the leeway model, which parameterises the wind drag of an object using a series of coefficients. These coefficients have been measured in field experiments for different types of objects, ranging from a person in the water to a coastal freighter adrift. OCEAN-SAR provides the user with an intuitive interface to run simulations and to visualise their results using Google Maps. The performance of the service is evaluated by comparing simulations to data from the Italian Coast Guard pertaining to actual incidents in the Mediterranean Sea.

VISIR (discoVerIng Safe and effIcient Routes) is an operational decision support system (DSS) for optimal ship routing designed and implemented in the frame of the TESSA (TEchnology for Situational Sea Awareness) project. The system is aimed to increase safety and efficiency of navigation through the use of forecast environmental fields and route optimization. VISIR can be accessed through a web interface (www.visir-nav.com) and mobile applications for both iOS and Android devices. This paper focuses on the technological infrastructure developed for operating VISIR as a DSS. Its main components are described, the performance of the operational system is assessed through experimental measurements, and a few case studies are presented.

A case study on climate models intercomparison data analysis addressing several classes of multi-model experiments is being implemented in the context of the EU H2020 INDIGO-DataCloud project. Such experiments require the availability of large amount of data (multi-terabyte order) related to the output of several climate models simulations as well as the exploitation of scientific data management tools for large-scale data analytics. More specifically, the paper discusses in detail a use case on precipitation trend analysis in terms of requirements, architectural design solution, and infrastructural implementation. The experiment has been tested and validated on CMIP5 datasets, in the context of a large scale distributed testbed across EU and US involving three ESGF sites (LLNL, ORNL, and CMCC) and one central orchestrator site (PSNC).

Sea Situational Awareness (SSA) is strategically important for management purposes of Italian Seas and coastal areas. The lack of adequate dissemination of marine environmental data and consequent poor knowledge available for operations at sea reduce the response capacity, leading to loss of lives and potential socio-economic damages. The SSA topic is being addressed by "TESSA", an industrial research project funded under the PON "Ricerca & Competitivita 2007–2013" program of Ministero Italiano dell'Istruzione, dell'Universita' e della Ricerca. TESSA is a joint effort of research groups of operational oceanography and scientific computing, and aims to strengthen and consolidate the operational oceanography service and to integrate it with advanced technological platforms in order to disseminate information for the SSA. The first product of TESSA is “SeaConditions”, a public service providing ocean and weather forecasts for the Mediterranean Sea, on the web and mobile applications. Every day, forecasts are produced by operational services, such as the Mediterranean Monitoring and Forecasting Center (www.myocean.eu) for the ocean variables and ECMWF for the atmospheric variables. The service delivers detailed information with high spatial and temporal resolution. Main variables displayed on Google Maps are: bathymetry, weather and oceanographic forecasts and satellite ocean colour data. Ocean forecasts are given at different resolution since nested limited area models for Mediterranean sub-regions are also displayed. SeaConditions provides a user friendly interface with zoom and drag Google Maps' features allowing to display data with different levels of details. SeaConditions' main strength is to provide a single point of access to meteo-marine forecasts, which are based on advanced oceanographic models, remote sensing products and bathymetry, and to deliver high quality information. The SeaConditions products are available through web and mobile channels. The web portal www.sea-conditions.com is compatible with all modern web-browsers on all operating systems. For the mobile users, APPs were also developed to consider the different kind of screens and gesture/interactions. The APPs are available on AppleStore and Google Play.

The Ophidia project aims to provide a big data analytics platform solution that addresses scientific use cases related to large volumes of multidimensional data. In this work, the Ophidia software infrastructure is discussed in detail, presenting the entire software stack from level-0 (the Ophidia data store) to level-3 (the Ophidia web service front end). In particular, this paper presents the big data cube primitives provided by the Ophidia framework, discussing in detail the most relevant and available data cube manipulation operators. These primitives represent the proper foundations to build more complex data cube operators like the apex one presented in this paper. A massive data reduction experiment on a 1TB climate dataset is also presented to demonstrate the apex workflow in the context of the proposed framework.

The Ophidia project is a research effort addressing big data analytics requirements, issues, and challenges for eScience. We present here the Ophidia analytics framework, which is responsible for atomically processing, transforming and manipulating array-based data. This framework provides a common way to run on large clusters analytics tasks applied to big datasets. The paper highlights the design principles, algorithm, and most relevant implementation aspects of the Ophidia analytics framework. Some experimental results, related to a couple of data analytics operators in a real cluster environment, are also presented.

The process for developing a climate model often involves a wide community of developers. All of the code releases can be classified into two groups: improvements and updates related to modeling aspects (new parameterizations, new and more detailed equations, remove of model approximations, and so on); improvement related to the computational aspects (performance enhancement, porting on new computing architectures, fixing of known bugs, and so on). The developing process involves both programmers, scientific experts, and rarely also computer scientists. The new improvements and developments are mainly focused on the scientific aspects and, in second stage, on the computing performance. The developments to improve the physic model often does not care about its impacts on the computational performances. This poses some issue in the developing process; after a new implementation, the code must be revised after new implementation to face out with the performance issues. In this work we analyze 5 different releases starting from the NEMO v3.2 (to be considered as our reference) and evaluate how new developments impact on the computational performances.

The report describes the activities carried out within the NEMO Consortium commitment. They refer to the performance evaluation and the comparison of the NEMO version 3.3 and the NEMO version 3.3.1. The main differences between two versions are related to the memory management and the allocation of the data structures. The NEMO ver. 3.3.1 replaces the static memory array allocation with a dynamic one. This approach brings some relevant benefits, such as the run time evaluation of the best domain decomposition. On the other hand, the dynamic array allocation introduces some lose of computational performance. The aim of this work is to evaluate the difference between the two versions from a computational point of view.

The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project., This work has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, PCI2021-121944, and PCI2021-121927). In Germany, it has received complementary funding from the German Federal Ministry of Education and Research (contracts 16HPC016K, 6GPC016K, 16HPC017 and 16HPC018). In France, it has received financial support from Caisse des dépôts et consignations (CDC) under the action PIA ADEIP (project Calculateurs). In Italy, it has been preliminary approved for complimentary funding by Ministero dello Sviluppo Economico (MiSE) (ref. project prop. 2659). In Norway, it has received complementary funding from the Norwegian Research Council, Norway under project number 323825. In Switzerland, it has been preliminary approved for complimentary funding by the State Secretariat for Education, Research, and Innovation (SERI), Norway. In Poland, it is partially supported by the National Centre for Research and Development under decision DWM/EuroHPCJU/4/2021. The authors also acknowledge financial support by MCIN/AEI /10.13039/501100011033, Spain through the “Severo Ochoa Programme for Centres of Excellence in R&D” under Grant CEX2018-000797-S, the Spanish Government, Spain (contract PID2019-107255 GB) and by Generalitat de Catalunya, Spain (contract 2017-SGR-01414). Anna Queralt is a Serra Húnter Fellow., With funding from the Spanish government through the ‘Severo Ochoa Centre of Excellence’ accreditation (CEX2018-000797-S).

Presentation of the paper "Towards an Open (Data) Science Analytics-Hub for Reproducible Multi-Model Climate Analysis at Scale" at the IEEE BigData2018 Conference (Seattle, WA, USA)

Open Science is key to future scientific research and promotes a deep transformation in the whole scientific research process encouraging the adoption of transparent and collaborative scientific approaches aimed at knowledge sharing. Open Science is increasingly gaining attention in the current and future research agenda worldwide. To effectively address Open Science goals, besides Open Access to results and data, it is also paramount to provide tools or environments to support the whole research process, in particular the design, execution and sharing of transparent and reproducible experiments, including data provenance (or lineage) tracking. This work introduces the Climate Analytics-Hub, a new component on top of the Earth System Grid Federation (ESGF), which joins big data approaches and parallel computing paradigms to provide an Open Science environment for reproducible multi-model climate change data analytics experiments at scale. An operational implementation has been set up at the SuperComputing Centre of the Euro-Mediterranean Center on Climate Change, with the main goal of becoming a reference Open Science hub in the climate community regarding the multi-model analysis based on the Coupled Model Intercomparison Project (CMIP). This paper reports about some ESiWACE WP3 activities described in the deliverable D3.10 "ESiWACE Scheduler development and support activities"

In the context of XF-ACTORS project, a IoT based monitoring system has been established and tested in the field - for an olive grove in Puglia (Italy) affected by Xf - to measure in near real-time some parameters proxies of trees’ conditions and vulnerability under both abiotic (climate) and biotic factors. The system is based on the TreeTalker (TT) technology, comprising multiparametric sensors to monitor water transport in trees, trunk humidity and diametrical oscillations, spectral characteristics of the leaves and microclimatic parameters (temperature, relative humidity). In particular, the sap flow density can be retrieved according to the Heat Balance Method (Granier 1985) after measuring the temperature of two 20 mm long probes inserted into the stem wood at 10 cm distance along the trunk vertical axis; the probe in the higher position is heated while the lower one provides the reference temperature. The TT system collects and transmits data at hourly time frequency, thanks to a LoRa based wireless connection, to a node managed by another microcontroller (TT-Cloud) serving a few tens of devices in a cluster. The TT-Cloud is in turn connected to the internet via the GPRS network and sends data to a computer server. Here, raw data are subject to ETL procedure that allows data Extraction from the TT-Cloud source, data Transformation by cleaning and converting them into variables with eco-physiological meaning, and finally data Loading to insert them into the target spatio-temporal database, adopting proper storage format/structure for querying and analysis purpose. From here, data can be further elaborated and visualized, e.g. into useful statistics, through a tailored Data Science environment. The preliminary results on sap flow density are here presented as they can give important information about the impact of Xf that is known obstructing xylem vessels, reducing hydraulic conductivity and thus affecting evapotranspiration., IT; PPT; monia.santini@cmcc.it