Search

The advent of exascale computing invites an assessment of existing best practices for developing application readiness on the world's largest supercomputers. This work details observations from the last four years in preparing scientific applications to run on the Oak Ridge Leadership Computing Facility's (OLCF) Frontier system. This paper addresses a range of topics in software including programmability, tuning, and portability considerations that are key to moving applications from existing systems to future installations. A set of representative workloads provides case studies for general system and software testing. We evaluate the use of early access systems for development across several generations of hardware. Finally, we discuss how best practices were identified and disseminated to the community through a wide range of activities including user-guides and trainings. We conclude with recommendations for ensuring application readiness on future leadership computing systems., Comment: Accepted at SC23

In her April 2010 NILOA paper, "Opening Doors to Faculty Involvement in Assessment," Pat Hutchings called for institutions to involve students in assessment, citing the potential to increase faculty engagement. In Curtis and Anderson (2020), the first author interviewed numerous partnership experts in both the United Kingdom and the United States regarding student-faculty partnerships and the extent to which students currently engage in the assessment process. Findings from the study make it clear that there currently exists little student-faculty partnership in assessment at the program- or system-levels. Combining existing research on partnership and the expert responses from the Curtis et al. study, we present a framework, based on prototyping, for developing student-faculty partnerships in program-level student learning outcomes assessment.

The high cost of chemistry integration is a significant computational bottleneck for realistic reactive-flow simulations using operator splitting. Here we present a methodology to accelerate the solution of the chemical kinetic ordinary differential equations using single-instruction, multiple-data vector processing on CPUs using the OpenCL framework. First, we compared several vectorized integration algorithms using chemical kinetic source terms and analytical Jacobians from the pyJac software against a widely used integration code, CVODEs. Next, we extended the OpenFOAM computational fluid dynamics library to incorporate the vectorized solvers, and we compared the accuracy of a fourth-order linearly implicit integrator -- both in vectorized form and a corresponding method native to OpenFOAM -- with the community standard chemical kinetics library Cantera. We then applied our methodology to a variety of chemical kinetic models, turbulent intensities, and simulation scales to examine a range of engineering and scientific scale problems, including (pseudo) steady-state as well as time-dependent Reynolds-averaged Navier--Stokes simulations of the Sandia flame D and the Volvo Flygmotor bluff-body stabilized, premixed flame. Subsequently, we compared the performance of the vectorized and native OpenFOAM integrators over the studied models and simulations and found that our vectorized approach performs up to 33--35x faster than the native OpenFOAM solver with high accuracy., Comment: 43 pages, 6 figures

Accurately predicting key combustion phenomena in reactive-flow simulations, e.g., lean blow-out, extinction/ignition limits and pollutant formation, necessitates the use of detailed chemical kinetics. The large size and high levels of numerical stiffness typically present in chemical kinetic models relevant to transportation/power-generation applications make the efficient evaluation/factorization of the chemical kinetic Jacobian and thermochemical source-terms critical to the performance of reactive-flow codes. Here we investigate the performance of vectorized evaluation of constant-pressure/volume thermochemical source-term and sparse/dense chemical kinetic Jacobians using single-instruction, multiple-data (SIMD) and single-instruction, multiple thread (SIMT) paradigms. These are implemented in pyJac, an open-source, reproducible code generation platform. A new formulation of the chemical kinetic governing equations was derived and verified, resulting in Jacobian sparsities of 28.6-92.0% for the tested models. Speedups of 3.40-4.08x were found for shallow-vectorized OpenCL source-rate evaluation compared with a parallel OpenMP code on an avx2 central processing unit (CPU), increasing to 6.63-9.44x and 3.03-4.23x for sparse and dense chemical kinetic Jacobian evaluation, respectively. Furthermore, the effect of data-ordering was investigated and a storage pattern specifically formulated for vectorized evaluation was proposed; as well, the effect of the constant pressure/volume assumptions and varying vector widths were studied on source-term evaluation performance. Speedups reached up to 17.60x and 45.13x for dense and sparse evaluation on the GPU, and up to 55.11x and 245.63x on the CPU over a first-order finite-difference Jacobian approach. Further, dense Jacobian evaluation was up to 19.56x and 2.84x times faster than a previous version of pyJac on a CPU and GPU, respectively., Comment: 53 pages, 13 figures

The relative importance index (RII) method for determining appropriate target species for dynamic adaptive chemistry (DAC) simulations using the DRGEP method is developed. The accuracy and effectiveness of this RII method is validated for two fuels: n-heptane and isopentanol. The RII method determines appropriate DRGEP target species solely from the local thermo-chemical state of the simulation, ensuring that accuracy will be maintained. Further, the RII method reduces the expertise required of users to select DRGEP target species sets appropriate to the combustion phenomena under consideration. Constant volume autoignition simulations run over a wide range of initial condi- tions using detailed reaction mechanisms for n-heptane and isopentanol show that the RII method is able to maintain accuracy even when traditional static target species sets fail, and are even more accurate than expert-selected target species sets. Additionally, the accuracy and efficiency of the RII method are compared to those of static target species sets in single-cell engine simulations under homogeneous charge compression ignition conditions. For simulations using more stringent DRGEP thresholds, the RII method performs similarly to that of the static target species sets. With a larger DRGEP threshold, the RII method is significantly more accurate than the static target species sets without imposing significant computational overhead. Furthermore, the applicability of the RII method to a DRG-based DAC scheme is discussed.

A fifth-order implicit Runge-Kutta method and two fourth-order exponential integration methods equipped with Krylov subspace approximations were implemented for the GPU and paired with the analytical chemical kinetic Jacobian software pyJac. The performance of each algorithm was evaluated by integrating thermochemical state data sampled from stochastic partially stirred reactor simulations and compared with the commonly used CPU-based implicit integrator CVODE. We estimated that the implicit Runge-Kutta method running on a single GPU is equivalent to CVODE running on 12-38 CPU cores for integration of a single global integration time step of 1e-6 s with hydrogen and methane models. In the stiffest case studied---the methane model with a global integration time step of 1e-4 s---thread divergence and higher memory traffic significantly decreased GPU performance to the equivalent of CVODE running on approximately three CPU cores. The exponential integration algorithms performed more slowly than the implicit integrators on both the CPU and GPU. Thread divergence and memory traffic were identified as the main limiters of GPU integrator performance, and techniques to mitigate these issues were discussed. Use of a finite-difference Jacobian on the GPU---in place of the analytical Jacobian provided by pyJac---greatly decreased integrator performance due to thread divergence, resulting in maximum slowdowns of 7.11-240.96 times; in comparison, the corresponding slowdowns on the CPU were just 1.39-2.61 times, underscoring the importance of use of an analytical Jacobian for efficient GPU integration. Finally, future research directions for working towards enabling realistic chemistry in reactive-flow simulations via GPU\slash SIMD accelerated stiff chemical kinetic integration were identified., Comment: 34 pages, 6 figures; pdfLaTeX

Accurate simulations of combustion phenomena require the use of detailed chemical kinetics in order to capture limit phenomena such as ignition and extinction as well as predict pollutant formation. However, the chemical kinetic models for hydrocarbon fuels of practical interest typically have large numbers of species and reactions and exhibit high levels of mathematical stiffness in the governing differential equations, particularly for larger fuel molecules. In order to integrate the stiff equations governing chemical kinetics, generally reactive-flow simulations rely on implicit algorithms that require frequent Jacobian matrix evaluations. Some in situ and a posteriori computational diagnostics methods also require accurate Jacobian matrices, including computational singular perturbation and chemical explosive mode analysis. Typically, finite differences numerically approximate these, but for larger chemical kinetic models this poses significant computational demands since the number of chemical source term evaluations scales with the square of species count. Furthermore, existing analytical Jacobian tools do not optimize evaluations or support emerging SIMD processors such as GPUs. Here we introduce pyJac, a Python-based open-source program that generates analytical Jacobian matrices for use in chemical kinetics modeling and analysis. As a demonstration, we first establish the correctness of the Jacobian matrices for kinetic models of hydrogen, methane, ethylene, and isopentanol oxidation, then demonstrate the performance achievable on CPUs and GPUs using pyJac via matrix evaluation timing comparisons., Comment: 42 pages, 7 figures