Search

Moving a world leading numerical weather prediction system that runs on a dedicated, bespoke, high performance computing cluster and supporting infrastructure, into the heart of a digital twin for climate change adaptation and extreme weather events has been a challenging and exciting journey. In this paper we describe this journey with a focus on those aspects required to leverage the pre-exascale EuroHPC systems that have been made available to Destination Earth (DestinE) to run its computational representation[1]. EuroHPC systems can be effectively used for DestinE and are in fact key assets to deliver the computational power required for Earth system digital twins at global km-scale resolution. At the same time, EuroHPC systems were newly installed and procedures to run them efficiently are evolving. We find that each of these systems is operated by a national hosting entity that implements its own procedures, e.g. for identity and access management, specific system configuration like schedulers, filesystems, software management systems, and specific, sometimes vendor associated, toolchains, tooling, and container runtimes. In particular, the different scheduling policies encountered, required us to adapt our workflows for each site. We found that having dedicated resources available, which was trialed in a period from 16th February to 14th April on LUMI, allowed to achieve high occupation rates, with 92% on the reserved GPU allocation and greater than 97% efficiency on the CPU reservation. Also a stronger focus on federation of these systems, with a focus not only on federation of identities and accounts, but also in the areas of data ownership/transfer, observability, services and service accounts, maintenance coordination and performance portability is required. In general, it should become much easier to transfer a workload, or a digital twin system, from one EuroHPC site to the next and run and maintain them across several sites concurrently. [ABSTRACT FROM AUTHOR]

We report on the first multi-year km-scale global coupled simulations using ECMWF's Integrated Forecasting System (IFS) coupled to both the NEMO and FESOM ocean-sea ice models, as part of the Horizon 2020 Next Generation Earth Modelling Systems (nextGEMS) project. We focus mainly on the two unprecedented IFS-FESOM coupled setups, with an atmospheric resolution of 2.8 km and 4.4 km, respectively, and the same spatially varying ocean resolution that reaches locally below 5 km grid-spacing. This is enabled by a refactored ocean model code that allows for more efficient coupled simulations with IFS in a single-executable setup, employing hybrid parallelisation with MPI and OpenMP. A number of shortcomings in the original NWP-focussed model configurations were identified and mitigated over several cycles collaboratively by the modelling centres, academia, and the wider nextGEMS community. The main improvements are (i) better conservation properties of the coupled model system in terms of water and energy balance, which benefit also ECMWF's operational 9 km IFS-NEMO model, (ii) a realistic top-of-the-atmosphere (TOA) radiation balance throughout the year, (iii) improved intense precipitation characteristics, and (iv) eddy-resolving features in large parts of the mid- and high-latitude oceans (finer than 5 km grid-spacing) to resolve mesoscale eddies and sea ice leads. New developments made at ECMWF for a better representation of snow and land use, including a dedicated scheme for urban areas, were also tested on multi-year timescales. We provide first examples of significant advances in the realism and thus opportunities of these km-scale simulations, such as a clear imprint of resolved Arctic sea ice leads on atmospheric temperature, impacts of km-scale urban areas on the diurnal temperature cycle in cities, and better propagation and symmetry characteristics of the Madden-Julian Oscillation. [ABSTRACT FROM AUTHOR]