Search

Managing data from modern high resolution weather and climate simulations is becoming problematic. Here we present a summary of some of the middleware software developed within ESiWACE2 to help manage the deluge of data during and after simulations. Middleware is software that lies between the computer, network, storage fabric and the scientific code that exploits that fabric. The objective of the middleware developed here is to exploit some of the scientific knowledge we have about our data to more efficiently use the underlying fabric. To that end, we present a description of two classes of software which we have developed and/or improved during ESiWACE2: (1) Tools for in-flight analysis of ensembles of simulation which ideally can support cross-ensemble calculations, avoiding the need for writing all variables all the time or at analysis frequency for all ensemble members to disk; and (2) Middleware for managing writing data to disk and managing the migration of data through storage tiers. The particular products we discuss are XIOS and it’s ensemble analysis tools, the Earth System Data Middleware (ESDM), the Near Line Data System (NLDS) and tools which exploit the Climate-Forecast Aggregation conventions (CFA-C, CFA-Python, CFA-CPP and CFStore). Not all of these pieces of software are feature complete, but for each we discuss their current status, where to find code and documentation, and their roadmap to the future., ESiWACE2 stands for Centre of Excellence in Simulation of Weather and Climate in Europe Phase 2. ESiWACE2 is funded by the European Union's Horizon 2020 research and innovation programme (H2020-INFRAEDI-2018-1 call) under grant agreement 823988.

The 2021-2022annual report for the Centre for Environmental Data Analysis (CEDA).This report presents key statistics for the year past (2021- 2022) as well as a series of snapshots of activity, expressed as short highlights and reports.

Managing data on cloud platforms can be an extremely difficult task. The need to store data efficiently, both in terms of energy and monetary cost, dictates the existence of tiered storage for data written and read at different frequencies. Avoiding a complicated user interface is therefore a great challenge, as in most cases the user is tasked with interacting with the different storage media directly, having to learn APIs and complex tape semantics to be able to effectively manage their data. The Near-line Data Store (NLDS), a single interface for moving data between storage solutions, is intended to streamline the process by which users migrate data between storage media, whether on disk, tape or cloud storage. By providing a single, easy to use interface for adding, changing and removing data, we aim to abstract away the complexities involved and instead provide a seamless storage experience. This presentation details progress made to reach milestone 4 of the NLDS.

Managing data on cloud platforms can be an extremely difficult task. The need to store data efficiently, both in terms of energy and monetary cost, dictates the existence of tiered storage for data written and read at different frequencies. Avoiding a complicated user interface is therefore a great challenge, as in most cases the user is tasked with interacting with the different storage media directly, having to learn APIs and complex tape semantics to be able to effectively manage their data. The Near-line Data Store (NLDS), a single interface for moving data between storage solutions, is intended to streamline the process by which users migrate data between storage media, whether on disk, tape or cloud storage. By providing a single, easy to use interface for adding, changing and removing data, we aim to abstract away the complexities involved and instead provide a seamless storage experience. The presentation details the progress made to reach milestone 3 of the NLDS.

Environmental Data Centres, such as the Centre for Environmental Data Analysis (CEDA), are facing two interrelated challenges regarding both archive and user data: the data itself is growing rapidly year on year, and the introduction of new storage technology may require a change in workflows for users. Several software packages already exist to exploit this new storage technology, some of which involve re-writing the data into a new file format, which we believe is not suitable for archiving data. This presentation will detail the implementation of libraries to read and write CFA in C, C++ and Python.

Managing data on cloud platforms can be an extremely difficult task. The need to store data efficiently, both in terms of energy and monetary cost, dictates the existence of tiered storage for data written and read at different frequencies. Avoiding a complicated user interface is therefore a great challenge, as in most cases the user is tasked with interacting with the different storage media directly, having to learn APIs and complex tape semantics to be able to effectively manage their data. The Near-line Data Store (NLDS), a single interface for moving data between storage solutions, is intended to streamline the process by which users migrate data between storage media, whether on disk, tape or cloud storage. By providing a single, easy to use interface for adding, changing and removing data, we aim to abstract away the complexities involved and instead provide a seamless storage experience. Here, we discuss the principles on which this new application was designed, the current state of development, and detail a brief timeline going forward.

Environmental Data Centres, such as the Centre for Environmental Data Analysis (CEDA),are facing two interrelated challenges regarding both archive and user data: the data itself is growing rapidly year on year, and the introduction of new storage technology may require a change in workflows for users. Several software packages already exist to exploit this new storage technology, some of which involve re-writing the data into a new file format, which we believe is not suitable for archiving data. This presentation will present a low level library implementation of a data format, developed in conjunction with CEDA and NCAS CMS, that we believe can exploit the new storage technology and remain suitable as an archival data format.

An exascale simulation requires extreme-scale I/O. Existing workflows will not scale to this extent, particularly as an exascale simulation is likely to involve a smaller number of individuals carrying out large simulations and a larger number of individuals carrying out analysis of simulation products. Such analysis is bound to include data comparison from simulations carried out in multiple locations and stored on a range of media. ESiWACE addresses the entire workflow with software solutions addressing a range of targets, from simulation, through analysis, to information management across institutions. Most of the effort has been devoted to the simulation problem with work on both improving I/O from simulations (using the Earth System Data Middleware, ESDM) and on ensemble data analysis (using the X I/O server, XIOS). The ESDM provides relaxed I/O semantics and manages the heterogeneous storage available in a data centre by mapping datasets dynamically to (possibly multiple) storage locations, addressing both simulation and analysis use cases. This document provides a high-level descriptions of that software stack, including full documentation and information about how to enable ESDM support for existing NetCDF applications and utilise the ESDM API for software development. As part of the development that resulted in the ESDM architecture described and documented here, the ESDM code was hardened, and the I/O path has been investigated for many use cases and optimised. The performance measurements indicate that performance can be close to optimal for large-scale data access. Several additional features had been explored and prototyped that bear a high potential for large-scale performance improvements and co-design with vendors. These include new storage backends for S3, Motr, KOVE, PMEM-IO, and IME, which have been developed to bypass the POSIX API and accelerate performance. Ensemble data analysis can be carried out using existing tools (CYLC and XIOS), albeit with customisations that are model dependent. The necessary software can be built using NetCDF directly, or with the ESDM.

Presented at AGU Fall Meeting 2020, Session IN032-02.

This milestone has the form of a report that provides design considerations and presents design choices. It includes an architectural view of the data management components in ESiWACE2 WP4 (Task 1 to Task 5) and expands upon some designs in WP5. The purpose for the documentation of the design and interaction of the components is to provide a common ground in the ESiWACE2 consortium and to support the development of components that are utilized in different use cases.

The Exploitability work package within ESiWACE contains the objective of: Developing New tape access strategies and software customised for Earth system data by first modelling and simulating possible strategies, and then developing a new software library which provides both higher bandwidth to tape storage and increased storage redundancy. We implemented and released a simulator for hierarchical storage as part of deliverable D4.1 “Business Model with Alternative Scenarios” https://zenodo.org/record/1228749#.W73BbfaYTwk (Chapter 6). The developed simulator was used to model a tape library with a cache system together with a simple network topology. The system allows to run traces recorded on existing tape archive and has been used to predict the DKRZ tape archive using real traces. Additionally, the deliverable provided a high‐level discussion of the models. We have since developed new software, consisting of a multi‐tiered storage library which provides a single API to users to move data to a number of different storage systems, query the data that they have stored on those systems, and retrieve the data. These interactions are carried out using a common user interface, which is a command line tool to be used interactively, and a HTTP API to be used programmatically. The command line tool essentially provides a wrapper for calls to the HTTP API. The source code for the multi‐tiered storage library can be downloaded from: https://github.com/cedadev/django‐jdma_control/releases/tag/0.1.6 This prototype meets the requirements for milestone MS8: Prototype Tape Library., ESiWACE has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No 675191

High performance computers, data analysis systems and data centres typically use large (often parallel) POSIX filesystems. However, with the advent of exascale climate data, such filesystems cannot deliver price/performance across all aspects of the workflow from simulation to analysis. Alternative methods of storing large amounts of data are necessary, potentially as part of tiered storage environments. One alternative option is object storage, which potentially addresses many of the challenges of managing data at exascale, including decoupling from traditional POSIX semantics for user management and presenting a universal interface for access from inside the data centre and remotely across wide area networks. In particular, using the Amazon S3 interface allows hosting data more easily on public cloud or adopting hybrid-cloud scenarios. However, this approach raises a number of problems: 1) Determining a means of storage for large files that will facilitate efficient access; 2) Enabling meaningful search of data, when the data is distributed across objects rather than in a hierarchical file system; 3) Supporting existing applications which depend on POSIX semantics; and 4) Managing the distributed objects. To address these issues large data sets can be broken into "fragments" stored as objects, each fragment containing a subset of the data. This allows for parallel access to the fragments, improving the performance of reading the data across a network - particularly if the application and storage have semantic information about the nature of the objects. Here we present a new method of splitting netCDF files into subarray fragments and storing each fragment as an object in an object storage system with an S3 interface. The metadata and domain for each climate variable are stored in a master array file, as well as the location in the master array and URI of each subarray fragment. This master array file can be written to a location not within the object storage, for example a POSIX file system on a SSD. It allows for fast search and indexing of the metadata without having to reassemble the subarray fragments. Each subarray fragment is stored as a self-contained netCDF file within the object storage, complete with the metadata and subdomain for that subarray, allowing reconstruction of the data, if the master file is lost, using standard climate data tools. The method is implemented in Python and Cython as the S3-netCDF4-python package, which is a subclass of the standard netCDF4-python package. This allows any improvements in the standard package, and lower software stack, to be automatically included in S3-netCDF4-python. Also, as a subclass of netCDF4-python, S3-netCDF4- python presents a well-known interface and data model to the scientist, hiding the physical location of the files and allowing scientists to continue using existing workflows based on netCDF4-python. We conclude by providing some examples and presenting some performance metrics to compare with storing netCDF files on other disk systems or using other interfaces such as OPeNDAP. Some limited comparison with other object store approaches such as the HDF Groups’s Highly Scalable Data Server will also be included.

Using object stores, and cloud based storage utilising the Amazon S3 HTTP API, presents a number of opportunities and challenges in the storing of climate data. The distributed nature of the objects allows large data sets to be broken into "fragments", each fragment containing a subset of the data. This allows for parallel access to the fragments, improving the performance of reading the data across a network. However, this presents a number of problems. Firstly, determining the fragment size and the optimum method of splitting multi-dimensional data. Secondly, enabling meaningful search of data, when the data may be widely distributed. This talk will present a new method of splitting netCDF files into fragments and storing each fragment as an object in an S3 compatible object store. The location of the fragments, the metadata and dimensions for each climate variable are stored in a master file, which can be written to a location not within the object store, for example a POSIX file system on a SSD. This allows fast search of the metadata without having to reassemble the fragments. Each fragment is stored as a self-contained netCDF file allowing reconstruction of the data if the master file is lost., Previously curated at: http://cedadocs.ceda.ac.uk/1361/ Event type: workshop. This item was not refereed before the publication Main files in this record: RCUK_cloud_2018_nrmassey.pptx Item originally deposited with Centre for Environmental Data Analysis (CEDA) document repository by Dr Neil Massey. Transferred to CEDA document repository community on Zenodo on 2022-11-24

JASMIN is a highly successful data analysis system, which is used by thousands of academics and their industrial partners to analyse many petabytes of environmental data. The rapidly increasing volume of data stored on JASMIN, and the steadily increasing number of users, is making it necessary to investigate and implement new methods of providing computing resources to the users, storing the data that they produce from their analyses and storing and maintaining a very large archive of environmental data. To achieve this, two main areas of research are described. Firstly, providing users with virtualised services to best utilise the computing resources available. Secondly, using object storage to provide a large, yet affordable, data store and providing the users with tools and interfaces to common environmental data formats, so as to not unduly affect their current work flows.

A collection of documents submitted by CEDA to the TGICA 25 meeting held in Exeter, July 2017., Work funded by BEIS

Large data sets used to study the impact of anthropogenic climate change on the 2013/14 floods in the UK are provided. The data consist of perturbed initial conditions simulations using the Weather@Home regional climate modelling framework. Two different base conditions, Actual, including atmospheric conditions (anthropogenic greenhouse gases and human induced aerosols) as at present and Natural, with these forcings all removed are available. The data set is made up of 13 different ensembles (2 actual and 11 natural) with each having more than 7500 members. The data is available as NetCDF V3 files representing monthly data within the period of interest (1st Dec 2013 to 15th February 2014) for both a specified European region at a 50 km horizontal resolution and globally at N96 resolution. The data is stored within the UK Natural and Environmental Research Council Centre for Environmental Data Analysis repository.