Search

This work documents version two of the Department of Energy's Energy Exascale Earth System Model (E3SM). E3SMv2 is a significant evolution from its predecessor E3SMv1, resulting in a model that is nearly twice as fast and with a simulated climate that is improved in many metrics. We describe the physical climate model in its lower horizontal resolution configuration consisting of 110 km atmosphere, 165 km land, 0.5° river routing model, and an ocean and sea ice with mesh spacing varying between 60 km in the mid-latitudes and 30 km at the equator and poles. The model performance is evaluated with Coupled Model Intercomparison Project Phase 6 Diagnosis, Evaluation, and Characterization of Klima simulations augmented with historical simulations as well as simulations to evaluate impacts of different forcing agents. The simulated climate has many realistic features of the climate system, with notable improvements in clouds and precipitation compared to E3SMv1. E3SMv1 suffered from an excessively high equilibrium climate sensitivity (ECS) of 5.3K. In E3SMv2, ECS is reduced to 4.0K which is now within the plausible range based on a recent World Climate Research Program assessment. However, a number of important biases remain including a weak Atlantic Meridional Overturning Circulation, deficiencies in the characteristics and spectral distribution of tropical atmospheric variability, and a significant underestimation of the observed warming in the second half of the historical period. An analysis of single-forcing simulations indicates that correcting the historical temperature bias would require a substantial reduction in the magnitude of the aerosol-related forcing.

This paper provides an overview of the United States (US) Department of Energy's (DOE's) Energy Exascale Earth System Model version 2 (E3SMv2) fully coupled Regionally Refined Model (RRM) and documents the overall atmosphere, land, and river results from the Coupled Model Intercomparison Project 6 (CMIP6) DECK (Diagnosis, Evaluation, and Characterization of Klima) and historical simulations – a first-of-kind set of climate production simulations using RRM. The North American (NA) RRM (NARRM) is developed as the high-resolution configuration of E3SMv2 with the primary goal of more explicitly addressing DOE's mission needs regarding impacts to the US energy sector facing Earth system changes. The NARRM features finer horizontal resolution grids centered over NA, consisting of 25→100 km atmosphere and land, 0.125° river routing model, and 14→60 km ocean and sea ice. By design, the computational cost of NARRM is ∼3x of the uniform low-resolution (LR) model at 100 km but only ∼10–20 % of a globally uniform high-resolution model at 25 km. A novel hybrid timestep strategy for the atmosphere is key for NARRM to achieve improved climate simulation fidelity within the high-resolution patch without sacrificing the overall global performance. The global climate, including climatology, time series, sensitivity, and feedback, is confirmed to be largely identical between NARRM and LR as quantified with typical climate metrics. Over the refined NA area, NARRM is generally superior to LR, including for precipitation and clouds over the contiguous US (CONUS), summertime marine stratocumulus clouds off the coast of California, liquid and ice phase clouds near the North polar region, extratropical cyclones, and spatial variability in land hydrological processes. The improvements over land are related to the better resolved topography in NARRM, whereas those over ocean are attributable to the improved air-sea interactions with finer grids for both atmosphere and ocean/sea ice. Some features appear insensitive to the resolution change analyzed here, for instance the diurnal propagation of organized mesoscale convective systems over CONUS, and the warm-season land-atmosphere coupling at the Southern Great Plains. In summary, our study presents a realistically efficient approach to leverage the RRM framework for a standard Earth system model release and high-resolution climate production simulations.

This work documents the first version of the U.S. Department of Energy (DOE) new Energy Exascale Earth System Model (E3SMv1). We focus on the standard resolution of the fully coupled physical model designed to address DOE mission-relevant water cycle questions. Its components include atmosphere and land (110-km grid spacing), ocean and sea ice (60km in the midlatitudes and 30km at the equator and poles), and river transport (55km) models. This base configuration will also serve as a foundation for additional configurations exploring higher horizontal resolution as well as augmented capabilities in the form of biogeochemistry and cryosphere configurations. The performance of E3SMv1 is evaluated by means of a standard set of Coupled Model Intercomparison Project Phase 6 (CMIP6) Diagnosis, Evaluation, and Characterization of Klima simulations consisting of a long preindustrial control, historical simulations (ensembles of fully coupled and prescribed SSTs) as well as idealized CO2 forcing simulations. The model performs well overall with biases typical of other CMIP-class models, although the simulated Atlantic Meridional Overturning Circulation is weaker than many CMIP-class models. While the E3SMv1 historical ensemble captures the bulk of the observed warming between preindustrial (1850) and present day, the trajectory of the warming diverges from observations in the second half of the twentieth century with a period of delayed warming followed by an excessive warming trend. Using a two-layer energy balance model, we attribute this divergence to the model's strong aerosol-related effective radiative forcing (ERFari+aci=−1.65W/m2) and high equilibrium climate sensitivity (ECS=5.3K).

In this study, we analyze the realism with which tropical cyclones (TCs) are simulated in the fully coupled low‐ and high‐resolution Energy Exascale Earth System Model (E3SM) version 1, with a focus on the latter. Compared to the low‐resolution (grid spacing of ∼1°), the representation of TCs improves considerably in the high‐resolution configuration (grid spacing of ∼0.25°). Significant improvements are found in the global TC frequency, TC lifetime maximum intensities, and the relative distribution of TCs among the different basins. However, at both resolutions, spurious TC activity is found in some basins, notably in the subtropical regions. Contrasting the simulated large‐scale TC environment with observations reveals that the model environment is unrealistically conducive for TC development in those regions. Further analysis indicates that these biases are likely related to those in thermodynamic potential intensity, caused by systematic SST biases, and vertical wind shear in the coupled model. TC‐ocean interaction is also examined in the high‐resolution configuration of the model. The salient features of the ocean's response to TC‐induced mixing and the ocean's impact on TC intensification are well‐reproduced. Finally, an evaluation of the influence of El Niño Southern Oscillation (ENSO) on TCs in the high‐resolution configuration of the model reveals that the ENSO‐TC relationship in the model has the right sign and is significant for the North Atlantic and Northwest Pacific, albeit weaker than in observations. In summary, the high‐resolution configuration of the E3SM model simulates TC activity reasonably and hence could be a useful tool for TC‐related research.

Author(s): Rasch, PJ; Xie, S; Ma, PL; Lin, W; Wang, H; Tang, Q; Burrows, SM; Caldwell, P; Zhang, K; Easter, RC; Cameron-Smith, P; Singh, B; Wan, H; Golaz, JC; Harrop, BE; Roesler, E; Bacmeister, J; Larson, VE; Evans, KJ; Qian, Y; Taylor, M; Leung, LR; Zhang, Y; Brent, L; Branstetter, M; Hannay, C; Mahajan, S; Mametjanov, A; Neale, R; Richter, JH; Yoon, JH; Zender, CS; Bader, D; Flanner, M; Foucar, JG; Jacob, R; Keen, N; Klein, SA; Liu, X; Salinger, AG; Shrivastava, M; Yang, Y | Abstract: The Energy Exascale Earth System Model Atmosphere Model version 1, the atmospheric component of the Department of Energy's Energy Exascale Earth System Model is described. The model began as a fork of the well-known Community Atmosphere Model, but it has evolved in new ways, and coding, performance, resolution, physical processes (primarily cloud and aerosols formulations), testing and development procedures now differ significantly. Vertical resolution was increased (from 30 to 72 layers), and the model top extended to 60 km (~0.1 hPa). A simple ozone photochemistry predicts stratospheric ozone, and the model now supports increased and more realistic variability in the upper troposphere and stratosphere. An optional improved treatment of light-absorbing particle deposition to snowpack and ice is available, and stronger connections with Earth system biogeochemistry can be used for some science problems. Satellite and ground-based cloud and aerosol simulators were implemented to facilitate evaluation of clouds, aerosols, and aerosol-cloud interactions. Higher horizontal and vertical resolution, increased complexity, and more predicted and transported variables have increased the model computational cost and changed the simulations considerably. These changes required development of alternate strategies for tuning and evaluation as it was not feasible to “brute force” tune the high-resolution configurations, so short-term hindcasts, perturbed parameter ensemble simulations, and regionally refined simulations provided guidance on tuning and parameterization sensitivity to higher resolution. A brief overview of the model and model climate is provided. Model fidelity has generally improved compared to its predecessors and the CMIP5 generation of climate models.

We present performance results and an analysis of a message passing interface (MPI)/OpenACC implementation of an electromagnetic solver based on a spectral-element discontinuous Galerkin discretization of the time-dependent Maxwell equations. The OpenACC implementation covers all solution routines, including a highly tuned element-by-element operator evaluation and a GPUDirect gather–scatter kernel to effect nearest neighbor flux exchanges. Modifications are designed to make effective use of vectorization, streaming, and data management. Performance results using up to 16,384 graphics processing units of the Cray XK7 supercomputer Titan show more than 2.5× speedup over central processing unit-only performance on the same number of nodes (262,144 MPI ranks) for problem sizes of up to 6.9 billion grid points. We discuss performance-enhancement strategies and the overall potential of GPU-based computing for this class of problems.

Numerical solutions of nonlinear partial differential equations frequently rely on iterative Newton--Krylov methods, which linearize a finite-difference stencil-based discretization of a problem, producing a sparse matrix with regular structure. Knowledge of this structure can be used to exploit parallelism and locality of reference on modern cache-based multi and manycore architectures, achieving high performance for computations underlying commonly used iterative linear solvers. In this paper we describe our approach to sparse matrix data structure design and our implementation of the kernels underlying iterative linear solvers in PETSc. We also describe autotuning of CUDA implementations based on high-level descriptions of the stencil-based matrix and vector operations.

Many factors affect the performance and power characteristics of FPGA designs. Among them are the optimization parameters for synthesis, map, and place-and-route design tools. Choosing the right combination of these parameters can substantially lower power requirements, while still satisfying timing constraints. Finding such an improvement, however, requires significant experimentation by the FPGA designer. Exhaustive search through the parameter space is an automated alternative to experimentation but is intractable because of the large search space and the long build time of each parameter combination. In this paper, we propose a machine-learning-based approach to tune FPGA design parameters. It performs sampling-based reduction of the parameter space and guides the search toward promising parameter configurations. In our experiments, such selective sampling finds parameter configurations that meet the timing constraints and are within 0.2% of the optimal power consumption. Furthermore, these configurations are found in an order of magnitude less time compared with exhaustive search. Such speedups can substantially alleviate bottlenecks in the FPGA design process.

There is a growing number of collaboration technology platforms that support rapid development and deployment of Action Center applications -- collaborative applications that encapsulate both collaboration expertise and tools for effective collaborative work practices using the facilitator-in-a-box strategy. This strategy enables instantiation and diffusion of state-of-the-art collaboration patterns for high-value recurring tasks. A side effect of the growing number of platforms is the potential for incompatibilities among Action Centers that can reduce interoperability, knowledge sharing and reuse. We present an ontology for Action Center-oriented collaboration platforms that formalizes key concepts of the approach using OWL. The resulting ontology can reduce ambiguities and promote knowledge sharing, reuse and standardization.

Earth system models rely on past observations and knowledge to simulate future climate states. Because of the inherent complexity, a substantial uncertainty exists in model-based predictions. Evaluation and improvement of model codes are one of the priorities of climate science research. Automatic Differentiation enables analysis of sensitivities of predicted outcomes to input parameters by calculating derivatives of modeled functions. The resulting sensitivity knowledge can lead to improved parameter calibration. We present our experiences in applying OpenAD to the Fortran-based crop model code in the Community Land Model (CLM). We identify several issues that need to be addressed in future developments of tangent-linear and adjoint versions of the CLM.

Recently, researchers proposed Collaboration Support Systems (CSS) as a new class of collaboration technology with which a collaboration engineer could package collaboration expertise with collaboration technology in a form that non-experts could use to successfully execute a work practice without extensive training on either the technology or the techniques

The programming concept of rewrite strategies supports versatile composition of rewrite rules and control of their application. Such programmability of rewrites can possibly lead to incorrect compositions of rewrites or incorrect applications of rewrites to terms within a strategic rewriting program. In this paper, we explore the analysis of strategic rewriting programs to detect certain programming errors statically. In particular, we introduce fine-grain types to closely approximate the dynamic behavior of rewriting. We develop an expressive type system for a core rewriting language. The resulting system detects programming errors of universally unreachable and failing rewrites. Static detection of such errors can substantially reduce testing and debugging efforts and lead to a more effective use of strategic rewriting in large and complex rewriting problems.

Synchronous collaborative applications typically take several programmer-years to develop because of the need for difficult-to-build capabilities required to resolve issues that emerge when multiple users execute potentially conflicting actions on the same data objects. It is therefore too costly to build bespoke collaborative applications for many tasks. Generic applications, however, are often poorly suited to specific requirements of a given task. This paper presents a conceptual implementation of an integrated collaborative system - ARCADE - to design and deploy complete collaborative applications for any problem domain. The system provides orders of magnitude reduction in development time. The resulting applications - ActionCenters - can guide the practitioners through well-designed collaborative work practices without requiring that they become collaboration experts. Adoption of ActionCenters can significantly increase diffusion of state-of-the-art collaboration in organizations.

Virtual projects are essential components of modern organizations that seek to be flexible and take advantage of distributed resources. A virtual project is a project in which team members are dispersed geographically and potentially on other dimensions, and are working together to accomplish a specific task under time and resource constraints. Because of their dispersion, team members have to rely on computer-mediated communication tools to do their work. Virtual projects are prevalent in software development and increasingly common in research and development, marketing, and customer relationship management, hence they are an important phenomenon for study. The challenge is to discover which practices and perspectives help to enhance the effectiveness of virtual projects, so that team members can leverage the advantages of virtuality while avoiding its pitfalls. The relevant literature on this topic comes from many areas, including studies of virtual teams as well as the body of knowledge in project management. In this article, we bring together disparate fields and provide an integrated view of virtual project management. We begin by defining key terms and concepts in the context of an overall framework and briefly describe relevant knowledge from current research. We then discuss key issues and future trends for research, and conclude with overall observations and implications.

Embedded systems are computational environments having restricted capabilities. These restrictions make the incorporation of high-level general purpose libraries, such as Java.lang and java.util, into the embedded systems software development process problematic. This paper describes a general transformation-based approach that can be used to adapt Java libraries making them compatible with the computational restrictions imposed by embedded environments.

With the increasing complexity of dynamic and collaborative computing environments in grid, security management has become a critical factor. Although several approaches have been proposed, fully decentralized and efficient authorization management is still a challenging problem. We propose an access control scheme based on a group-based RBAC model for grid computing environments. By separating the administrations of users by VO level policies and permissions by resource or service provider policies, our scheme provides decentralized, autonomous, and fine-grained security management which fits the dynamic environment of grids, and can support ad-hoc collaborations. We implement a proof-of-concept prototype system by enhancing the access control module in grid file system (GFS) and specifying different levels of policies with XACML.

Embedded systems can be viewed as scaled-down versions of their stand-alone counterparts. In many cases, the software abstractions and libraries for embedded systems can be derived from libraries for stand-alone systems. One such example is the Java library for Java Virtual Machines. An embedded system does not always support all features as in the case of an embedded JVM that does not support floating-point operations. In such cases, an existing library needs to be migrated to the embedded platform. Libraries are large collections of code and manual migration is a daunting task. In this paper, we provide an automated approach to the library migration problem using program transformations. The solution developed in this paper enables rapid adaptation and re-targeting of Java libraries in the presence of evolving libraries and evolving embedded platforms.

The SSP is a hardware implementation of a subset of the JVM for use in high consequence embedded applications. In this context, a majority of the activities belonging to class loading, as it is defined in the specification of the JVM, can be performed statically. Static class loading has the net result of dramatically simplifying the design of the SSP as well as increasing its performance. The functionality of the class loader can be implemented using strategic programming techniques. The incremental nature of strategic programming is amenable to formal verification. This article gives an overview of the core class loading activities for the SSP, their implementation in the strategic programming language TL, and outlines the approach to formal verification of the implementation.