Search

We are experimenting with hybrid access from Cloud and HPC environments using the Pangeo platform to make use of a data lake in an HPC infrastructure “DATARMOR”. DATARMOR is an HPC infrastructure hosting ODATIS services (https://www.odatis-ocean.fr) situated at “Pôle de Calcul et de Données pour la Mer” in IFREMER. Its parallel file system has a disk space dedicated for shared data, called “dataref”. Users of DATARMOR can access these data, and some of those data are cataloged by sextant service (https://sextant.ifremer.fr/Ressources/Liste-des-catalogues-thematiques/Datarmor-Donnees-de-reference ) and is open and accessible from the internet, without duplicating the data. In the cloud environment, the ability to access files in a parallel manner is essential for improving the speed of calculations. The Zarr format (https://zarr.readthedocs.io) enables parallel access to data sets, as it consists of numerous chunked “object data” files and some “metadata” files. Although it enables multiple data access, it is simple to use since all the collections of data stored in a Zarr format are accessible through one access point. For HPC centers, the numerous “object data” files create a lot of metadata on parallel file systems, slowing the data access time. Recent progress on development of Kerchunk (https://fsspec.github.io/kerchunk/), which recognize the chunks in a file (e.g. NetCDF / HDF5) as a Zarr chunk and its capability to recognize a series of files as one Zarr file, is solving these technical difficulties in our PANGEO use cases at DATARMOR. Thanks to Kerchunk and Intake (https://intake.readthedocs.io/) it is now possible to use different sets of data stored in DATARMOR in an efficient and simple manner. We are further experimenting with this workflow using the same use cases on the PANGEO-EOSC cloud. We make use of the same data stored at the data lake in DATARMOR, but based on Kerchunk and Intake catalog through ODATIS access, without duplicating the source data. In the presentation we will share our recent experiences from these experiments.

Research projects heavily rely on the exchange and processing of data and in this context Pangeo (https://pangeo.io/), a world-wide community of scientists and developers, thrives to facilitate the deployment of ready to use and community-driven platforms for big data geoscience. The European Open Science Cloud (EOSC) is the main initiative in Europe for providing a federated and open multi-disciplinary environment where European researchers, innovators, companies and citizens can share, publish, find and re-use data, tools and services for research, innovation and educational purposes. While a number of services based on Jupyter Notebooks were already available, no public Pangeo deployments providing fast access to large amounts of data and compute resources were accessible on EOSC. Most existing cloud-based Pangeo deployments are USA-based, and members of the Pangeo community in Europe did not have a shared platform where scientists or technologists could exchange know-how. Pangeo teamed up with two EOSC projects, namely EGI-ACE (https://www.egi.eu/project/egi-ace/) and C-SCALE (https://c-scale.eu/) to demonstrate how to deploy and use Pangeo on EOSC and emphasise the benefits for the European community. The Pangeo Europe Community together with EGI deployed a DaskHub, composed of Dask Gateway (https://gateway.dask.org/) and JupyterHub (https://jupyter.org/hub), with Kubernetes cluster backend on EOSC using the infrastructure of the EGI Federation (https://www.egi.eu/egi-federation/). The Pangeo EOSC JupyterHub deployment makes use of 1) the EGI Check-In to enable user registration and thereby authenticated and authorised access to the Pangeo JupyterHub portal and to the underlying distributed compute infrastructure; and 2) the EGI Cloud Compute and the cloud-based EGI Online Storage to distribute the computational tasks to a scalable compute platform and to store intermediate results produced by the user jobs. To facilitate future Pangeo deployments on top of a wide range of cloud providers (AWS, Google Cloud, Microsoft Azure, EGI Cloud Computing, OpenNebula, OpenStack, and more), the Pangeo EOSC JupyterHub deployment is now possible through the Infrastructure Manager (IM) Dashboard (https://im.egi.eu/im-dashboard/login). All the computing and storage resources are currently supplied by CESNET (https://www.cesnet.cz/?lang=en) in the frame of EGI-ACE project (https://im.egi.eu/). Several deployments have been made to serve the geoscience community, both for teaching and for research work. To date, more than 100 researchers have been trained on Pangeo@EOSC deployments and more are expected to join, in particular with easy access to large amounts of Copernicus data through a recent collaboration established with the C-SCALE project. In this presentation, we will provide details on the different deployments, how to get access to JupyterHub deployments and more generally how to contribute to Pangeo@EOSC.

In order to make use of a collection of datasets – for example, scenes from a SAR satellite – more efficient, it is important to be able to search for datasets relevant for a specific application. In particular, one might want to search for a specific period in time, for the spatial extent, or perform searches over multiple collections together.For SAR data or data obtained from optical satellites, Spatio-Temporal Asset Catalogs (STAC) have become increasingly popular in the past few years. Defined as JSON and backed by databases with geospatial extensions, STAC servers (endpoints) have the advantage of being efficient, language-agnostic and following a standardized API.Just like satellite scenes, in-situ data is growing in size very quickly and thus would benefit from being catalogued. However, the sequential nature of in-situ data and its sparse distribution in space makes it difficult to fit into STAC's standard model.In the session, we present a experimental STAC extension that defines the most common properties of in-situ data as identified from ArgoFloat and biologging data.

The ever increasing number of scientific datasets made available by authoritative data providers (NASA, Copernicus, etc.) and provided by the scientific community opens new possibilities for advancing the state of the art in many areas of the natural sciences. As a result, researchers, innovators, companies and citizens need to acquire computational and data analysis skills to optimally exploit these datasets. Several educational programs dispense basic courses to students, and initiatives such as “The Carpentries” (https://carpentries.org/) complement this offering but also reach out to established researchers to fill the skill gap thereby empowering them to perform their own data analysis. However, most researchers find it challenging to go beyond these training sessions and face difficulties when trying to apply their newly acquired knowledge to their own research projects. To this regard, hackathons have proven to be an efficient way to support researchers in becoming competent practitioners but organising good hackathons is difficult and time consuming. In addition, the need for large amounts of computational and storage resources during the training and hackathons requires a flexible solution. Here, we propose an approach where researchers work on realistic, large and complex data analysis problems similar to or directly part of their research work. Researchers access an infrastructure deployed on the European Ocean Science Cloud (EOSC) that supports intensive data analysis (large compute and storage resources). EOSC is a European Commission initiative for providing a federated and open multi-disciplinary environment where data, tools and services can be shared, published, found and re-used. We used jupyter book for delivering a collection of FAIR training materials for data analysis relying on Pangeo EOSC deployments as its primary computing platform. The training material (https://pangeo-data.github.io/foss4g-2022/intro.html, https://pangeo-data.github.io/clivar-2022/intro.html, https://pangeo-data.github.io/escience-2022/intro.html) is customised (different datasets with similar analysis) for different target communities and participants are taught the usage of Xarray, Dask and more generally how to efficiently access and analyse large online datasets. The training can be completed by group work where attendees can work on larger scale scientific datasets: the classroom is split into several groups. Each group works on different scientific questions and may use different datasets. Using the Pangeo (http://pangeo.io) ecosystem is not always new for all attendees but applying Xarray (http://xarray.pydata.org) and Dask (https://www.dask.org/) on actual scientific “mini-projects” is often a showstopper for many researchers. With this approach, attendees have the opportunity to ask questions, collaborate with other researchers as well as Research Software Engineers, and apply Open Science practices without the burden of trying and failing alone. We find the involvement of scientific computing research engineers directly in the training is crucial for success of the hackathon approach. Feedback from attendees shows that it provides a solid foundation for big data geoscience and helps attendees to quickly become competent practitioners. It also gives infrastructure providers and EOSC useful feedback on the current and future needs of researchers for making their research FAIR and open. In this presentation, we will provide examples of achievements from attendees and present the feedback EOSC providers have received.

We have developed the ocean-sea ice Arctic regional configuration SEDNA (Sea ice - EDdy resolving ocean paN-Arctic configuration) at ultra-high resolution (800 m in the horizontal and 150 vertical levels) based on the NEMO-SI3 numerical platform, in order to investigate how the dynamics of mesoscale turbulent eddies in the Arctic Ocean interplay with sea ice. The configuration was initially developed on the AMD-based HPC system Joliot-Curie ROME based at CEA in France, which has achieved a Linpack performance of 12 PFlop/s ranked number 33 of June 2020 TOP500 list (https://top500.org/system/179700/).Thanks to a European PRACE allocation of nearly 40 million CPU hours, we were able to run a 8 year-long simulation. Although promising to understand part of the small-scale dynamics at play, this length of simulation will likely be a limiting factor in the investigation of the eddy dynamics which is known to equilibrate over several decades.To overcome this limitation, here we investigate the feasibility of running SEDNA over several decades but at a realistic time cost. To that aim, a benchmark has been performed on the ARM based HPC system Fugaku based at RIKEN in Japan, which has archived a Linpack performance of 442 PFlop/s ranked number 1 of November 2020 TOP500 list (https://www.top500.org/system/179807/). Such a benchmark has required adaptation of the compiling and placements of NEMO and XIOS MPI processes in order to fit on the Fugaku architecture compared to standard X86_64 based HPC systems like Joliot-Curie ROME. In this presentation we will share the tips and lessons learned from our benchmarks and will report the benchmark results. Our insights on model MPI placements for efficient post processing of huge models will be discussed as well.

In biologging, a small device attached to an animal is used to track its behaviour and environment. This data enables biologists to gain a better understanding of its movement, its preferred habitats, and the environmental conditions it needs to thrive, all of which is essential for the future protection of natural resources. For that, it is crucial to have georeferenced data of biological processes, such as fish migration, over a spatial and temporal range.Since it is challenging to track fish directly in the water, models have been developed to geolocate fish from the high resolution temperature and pressure time series obtained from the data storage tag. In particular, reconstructing the trajectories of seabass using the temporal temperature changes obtained from biologging devices has been studied since 2010 (https://doi.org/10.1016/j.ecolmodel.2015.10.024). These fish tracks are computed based on the likelihood of the temperature data obtained from the fish tag and reference geoscience data such as satellite observations and ocean physics model output. A high temporal and spatial resolution of the reference data plays a key role in the quality of the fish trajectories. However, the size and accessibility of these data sets as well as the computing power required to process high resolution data remain technical barriers.As the Pangeo ecosystem has been developed to solve such challenges in geoscience, we can take advantage of it in biologging. We use libraries such as intake, kerchunk, and fsspec to quickly load the data, xarray, pint, and dask to compute and hvplot and jupyter to display the results. The pangeo software stack enables us to easily access the data and compute high resolution fish tracks in a scalable and interactive manner.

OSDYN (Observations and Simulations of the DYNamics) is a Python library that proposes diagnostics to explore the dynamics of the ocean and its interactions with the atmosphere and waves. Its main strengths are its genericity concerning the different types of netCDF files and its ability to handle large volumes of data.Dedicated to large data sets such as in-situ, satellite, and numerical model observations, OSDYN is particularly powerful to manage different types of Arakawa-C grids and vertical coordinates (Nemo, Croco, Mars, Symphonie, WW3, MesoNH). Based on common Pangeo stack (xarray, dask, xgcm), OSDYN provides data readers that standardize the dimensions, coordinates, and variables names and properties of the datasets. Thus, all python diagnostics can be shared regardless of the model outputs.Thanks to progress made using kerchunk and efforts on transforming metadata of Ifremer’s HPC center (auto-kerchunk), the reading of a large amount of netCDF files is fast and the selection of sub-domains or specific variables is almost immediate.Jupyter notebooks will detail the implementation of three kinds of analyses. The first one focuses on climatologic issues. In order to compare modeled and satellite sea surface temperatures, the second one addresses spatial interpolation and comparison of data when some may be missing. Lastly, the third analysis provides an overview of how diagnostics describing the formation of deep water masses can be used from different data sets.

(IF 2.98; Q1); International audience; Modelling three-dimensional wave-current-turbulence interactions in extreme tidal environments is still challenging and necessary for the development of the tidal industry, particularly for the dimensioning of tidal converters. Following this objective, we focus our study on the Alderney Race (France) where the strongest tidal current in Western Europe and various sea states occur. Due to the strong tidal current at this location, wave-current interactions were poorly studied by the past. Therefore, we propose to assess the impacts of wave-current interactions in Alderney Race by the use of a 3D fully-coupled wave-current numerical model and in-situ measurements. The following wave-current interactions were observed : i) Stokes drift e ects inducing an increase/decrease in the current depending on the angle between waves and current, ii) wave enhancement of the bottom friction reducing the tidal current, iii) refraction of waves by the current, generating changes in waves directions, and iv) wave breaking ascribed to tidal current. The impact of such interactions were maximal near the reverse event ( 1h30) and for a weak/medium tidal range. A non-stationary time delay, varying within a same tidal cycle, was noted at both locations, which is reduced by including the local wind e ects and by adjusting the bottom stress formulation. This study shows that wave-current interactions play a non-negligible role in Alderney Race and that they need to consider by the tidal industry.

The Pangeo ecosystem is an interactive computing software stack for HPC and public cloud infrastructures. In this paper, we show benchmarking results of the Pangeo platform on two di erent HPC sys- tems. Four di erent geoscience operations were considered in this bench- marking study with varying chunk sizes and chunking schemes. Both strong and weak scaling analyses were performed. Chunk sizes between 64MB to 512MB were considered, with the best scalability obtained for 512MB. Compared to certain manual chunking schemes, the auto chunk- ing scheme scaled well.

A numerical model coupling the three-dimensional oceanic circulation, thanks to the MARS3D model, and the sea state computed by the WAVEWATCH III model, is developed in order to study the hydrodynamic of the nearshore zone. The main principle of the coupled model based on the coupler PALM and the necessary modifications made for the coupling procedure to both numerical codes are described. First tests in horizontal two-dimension are made and they allowed us to develop a primary version. For three-dimensional flow, we show that the coupled model must solve the quasi-Eulerian velocity. In fact, a coherent analytical form of the wave forcing for the Lagrangian velocity does not exist. This point is illustrated by one example with large errors in comparison with the analytical solution when solving for the Lagrangian mean flow., Un modèle numérique couplant la circulation tridimensionnelle, résolue par le code MARS3D, et l'état de la mer, calculé par le code WAVEWATCH III, est développé pour étudier l'hydrodynamique des zones côtières et littorales. Le principe général du modèle, faisant appel au coupleur PALM est décrit, ainsi que les aménagements apportés aux deux codes existants afin de rendre le couplage possible. Des premiers tests en deux dimensions horizontales ont permis de développer une première version. En trois dimensions, nous montrons que le modèle doit calculer la vitesse quasi-eulérienne, car il n'existe pas de forme analytique cohérente du forçage par les vagues pour la vitesse lagrangienne. Cet aspect est illustré par un résultat comportant de très fortes erreurs par rapport à la solution analytique, obtenu avec une formulation en vitesse lagrangienne.

Resume : Un modele numerique couplant la circulation tridimensionnelle, resolue par le code MARS3D, et l'etat de la mer, calcule par le code WAVEWATCH III, est developpe pour etudier l'hydrodynamique des zones cotieres et littorales. Le principe general du modele, faisant appel au coupleur PALM est decrite, ainsi que les amenagements apportes aux deux codes existants afin de rendre le couplage possible. Des premiers tests en deux dimensions horizontales ont permis de developper une premiere version. En trois dimensions, nous montrons que le modele doit calculer la vitesse quasiEulerienne, car il n'existe pas de forme analytique coherente du forcage par les vagues pour la vitesse Lagrangienne. Cet aspect est illustre par un resultat comportant de tres fortes erreurs par rapport a la solution analytique, obtenu avec le forcage errone propose par MELLOR (2003). Mots-cles : Modelisation couplee – MARS3D – WAVEWATCH III – Circulation cotiere