Search

With Moore's law progressively running out of steam, heterogeneous computing architectures have been powering the top supercomputers in the world for many years and are now finding broader adoption across the industry. The trend towards sustainable computing also requires domain-specific heterogeneous hardware architectures, which promise further gains in energy efficiency. At the same time, today's high performance computing applications have evolved from monolithic simulations in a single domain to multidisciplinary complex workflows. In this paper, we explore how these trends affect system design decisions and what this means for future computing system architectures.

Productivity from day one on supercomputers that leverage new technologies requires significant preparation. An institution that procures a novel system architecture often lacks sufficient institutional knowledge and skills to prepare for it. Thus, the "Center of Excellence" (CoE) concept has emerged to prepare for systems such as Summit and Sierra, currently the top two systems in the Top 500. This paper documents CoE experiences that prepared a workload of diverse applications and math libraries for a heterogeneous system. We describe our approach to this preparation, including our management and execution strategies, and detail our experiences with and reasons for using different programming approaches. Our early science and performance results show that the project enabled significant early seismic science with up to a l4X throughput increase over Cori. In addition to our successes, we discuss our challenges and failures so others may benefit from our experience.

CORAL, the Collaboration of Oak Ridge, Argonne and Livermore, is fielding two similar IBM systems, Summit and Sierra, with NVIDIA GPUs that will replace the existing Titan and Sequoia systems. Summit and Sierra are currently ranked No. 1 and No. 3, respectively on the Top500 list. We discuss the design and key differences of the systems. Our evaluation of the systems highlights the following. Applications that fit in HBM see the most benefit and may prefer more GPUs; however, for some applications, the CPU-GPU bandwidth is more important than the number of GPUs. The node-local burst buffer scales linearly, and can achieve a 4X improvement over the parallel file system for large jobs; smaller jobs, however, may benefit from writing directly to the PFS. Finally, several CPU, network and memory bound analytics and GPU-bound deep learning codes achieve up to a 11X and 79X speedup/node, respectively over Titan.

We present results for the static inter-quark potential, lightest glueballs, light hadron spectrum and topological susceptibility using a non-perturbatively improved action on a $16^3\times 32$ lattice at a set of values of the bare gauge coupling and bare dynamical quark mass chosen to keep the lattice size fixed in physical units ($\sim 1.7$ fm). By comparing these measurements with a matched quenched ensemble, we study the effects due to two degenerate flavours of dynamical quarks. With the greater control over residual lattice spacing effects which these methods afford, we find some evidence of charge screening and some minor effects on the light hadron spectrum over the range of quark masses studied ($M_{PS}/M_{V}\ge0.58$). More substantial differences between quenched and unquenched simulations are observed in measurements of topological quantities., Comment: 53 pages, LaTeX/RevTeX, 16 eps figures; corrected clover action expression and various typos, no results changed

The High Order Method Modeling Environment is a scalable, spectral-element-based prototype for the Community Atmospheric Model component of the Community Climate System Model. The 3D moist primitive equations are solved on the cubed sphere with a hybrid pressure η vertical coordinate using an Emanuel convective parametrization for moist processes. Semi-implicit time integration, based on a preconditioned conjugate gradient solver, circumvents the time step restrictions associated with gravity waves. Benchmarks for two standard tests problems at 10 km horizontal resolution have been run on Blue Gene/L. Results obtained on a 32-rack Blue Gene/L system (65,536 processors, 183.5-teraflop peak) show sustained performance of 8.0 teraflops on 32,768 processors for the moist Held–Suarez test problem in coprocessor mode and 11.3 teraflops on 32,768 processors for the aquaplanet test problem, running in virtual node mode.

BlueGene/L (BG/L), developed through a partnership between IBM and Lawrence Livermore National Laboratory (LLNL), is currently the world's largest system both in terms of scale, with 131,072 processors, and absolute performance, with a peak rate of 367 Tflop/s. BG/L has led the last four Top500 lists with a Linpack rate of 280.6 Tflop/s for the full machine installed at LLNL and is expected to remain the fastest computer in the next few editions. However, the real value of a machine such as BG/L derives from the scientific breakthroughs that real applications can produce by successfully using its unprecedented scale and computational power. In this paper, we describe our experiences with eight large scale applications on BG/ L from several application domains, ranging from molecular dynamics to dislocation dynamics and turbulence simulations to searches in semantic graphs. We also discuss the challenges we faced when scaling these codes and present several successful optimization techniques. All applications show excellent scaling behavior, even at very large processor counts, with one code even achieving a sustained performance of more than 100 Tflop/s, clearly demonstrating the real success of the BG/L design.

Quantum chromodynamics (QCD), the theory of the strong nuclear force, can be numerically simulated on massively parallel supercomputers using the method of lattice gauge theory. We describe the special programming requirements of lattice QCD (LQCD) as well as the optimal supercomputer hardware architectures for which LQCD suggests a need. We demonstrate these methods on the IBM Blue Gene/L™ (BG/L) massively parallel supercomputer and argue that the BG/L architecture is very well suited for LQCD studies. This suitability arises from the fact that LQCD is a regular lattice discretization of space into lattice sites, while the BG/L supercomputer is a discretization of space into compute nodes. Both LQCD and the BG/L architecture are constrained by the requirement of short-distance exchanges. This simple relation is technologically important and theoretically intriguing. We demonstrate a computational speedup of LQCD using up to 131,072 CPUs on the largest BG/L supercomputer available in 2007. As the number of CPUs is increased, the speedup increases linearly with sustained performance of about 20% of the maximum possible hardware speed. This corresponds to a maximum of 70.5 sustained teraflops. At these speeds, LQCD and the BG/L supercomputer are able to produce theoretical results for the next generation of strong-interaction physics.

We investigate solidification in metal systems ranging in size from 64,000 to 524,288,000 atoms on the IBM BlueGene/L computer at LLNL. Using the newly developed ddcMD code, we achieve performance rates as high as 103 TFlops, with a performance of 101.7 TFlop sustained over a 7 hour run on 131,072 cpus. We demonstrate superb strong and weak scaling. Our calculations are significant as they represent the first atomic-scale model of metal solidification to proceed, without finite size effects, from spontaneous nucleation and growth of solid out of the liquid, through the coalescence phase, and into the onset of coarsening. Thus, our simulations represent the first step towards an atomistic model of nucleation and growth that can directly link atomistic to mesoscopic length scales.

A general method for optimizing problem layout on the Blue Gene®/L (BG/L) supercomputer is described. The method takes as input the communication matrix of an arbitrary problem as an array with entries C(i, j), which represents the data communicated from domain i to domain j. Given C(i, j), we implement a heuristic map that attempts to sequentially map a domain and its communication neighbors either to the same BG/L node or to near-neighbor nodes on the BG/L torus, while keeping the number of domains mapped to a BG/L node constant. We then generate a Markov chain of maps using Monte Carlo simulation with free energy F =Σi,j C(i, j)H(i, j), where H(i, j) is the smallest number of hops on the BG/L torus between domain i and domain j. For two large parallel applications, SAGE and UMT2000, the method was tested against the default Message Passing Interface rank order layout on up to 2,048 BG/L nodes. It produced maps that improved communication efficiency by up to 45%.

The purpose of this research was to further investigate the hydrodynamics of the United States Pharmacopeia (USP) paddle dissolution apparatus using a previously generated computational fluid dynamics (CFD) model. The influence of paddle rotational speed on the hydrodynamics in the dissolution vessel was simulated. The maximum velocity magnitude for axial and tangential velocities at different locations in the vessel was found to increase linearly with the paddle rotational speed. Path-lines of fluid mixing, which were examined from a central region at the base of the vessel, did not reveal a region of poor mixing between the upper cylindrical and lower hemispherical volumes, as previously speculated. Considerable differences in the resulting flow patterns were observed for paddle rotational speeds between 25 and 150 rpm. The approximate time required to achieve complete mixing varied between 2 to 5 seconds at 150 rpm and 40 to 60 seconds at 25 rpm, although complete mixing was achievable for each speed examined. An analysis of CFD-generated velocities above the top surface of a cylindrical compact positioned at the base of the vessel, below the center of the rotating paddle, revealed that the fluid in this region was undergoing solid body rotation. An examination of the velocity boundary layers adjacent to the curved surface of the compact revealed large peaks in the shear rates for a region within ~3 mm from the base of the compact, consistent with a "grooving" effect, which had been previously seen on the surface of compacts following dissolution, associated with a higher dissolution rate in this region.

The goal of the workshop and this report is to identify common themes and standardize concepts for locality-preserving abstractions for exascale programming models. Current software tools are built on the premise that computing is the most expensive component, we are rapidly moving to an era that computing is cheap and massively parallel while data movement dominates energy and performance costs. In order to respond to exascale systems (the next generation of high performance computing systems), the scientific computing community needs to refactor their applications to align with the emerging data-centric paradigm. Our applications must be evolved to express information about data locality. Unfortunately current programming environments offer few ways to do so. They ignore the incurred cost of communication and simply rely on the hardware cache coherency to virtualize data movement. With the increasing importance of task-level parallelism on future systems, task models have to support constructs that express data locality and affinity. At the system level, communication libraries implicitly assume all the processing elements are equidistant to each other. In order to take advantage of emerging technologies, application developers need a set of programming abstractions to describe data locality for the new computing ecosystem. The new programming paradigm should be more data centric and allow to describe how to decompose and how to layout data in the memory.Fortunately, there are many emerging concepts such as constructs for tiling, data layout, array views, task and thread affinity, and topology aware communication libraries for managing data locality. There is an opportunity to identify commonalities in strategy to enable us to combine the best of these concepts to develop a comprehensive approach to expressing and managing data locality on exascale programming systems. These programming model abstractions can expose crucial information about data locality to the compiler and runtime system to enable performance-portable code. The research question is to identify the right level of abstraction, which includes techniques that range from template libraries all the way to completely new languages to achieve this goal.

We give a detailed general description of a recent geometrical discretisation scheme and illustrate, by explicit numerical calculation, the scheme's ability to capture topological features. The scheme is applied to the Abelian Chern-Simons theory and leads, after a necessary field doubling, to an expression for the discrete partition function in terms of untwisted Reidemeister torsion and of various triangulation dependent factors. The discrete partition function is evaluated computationally for various triangulations of $S^3$ and of lens spaces. The results confirm that the discretisation scheme is triangulation independent and coincides with the continuum partition function, Comment: 27 pages, 5 figures, 6 tables. in latex

We describe a systematic approach to generating approximate actions for the lattice simulation of QCD. Three different tuning conditions are defined to match approximate with true actions, and it is shown that these three conditions become equivalent when the approximate and true actions are sufficiently close. We present a detailed study of approximate actions in the lattice Schwinger model together with an exploratory study of full QCD at unphysical parameter values. We find that the technicalities of the approximate action approach work quite well. However, very delicate tuning is necessary to find an approximate action which gives good predictions for all physical observables. Our best view of the immediate applicability of the methods we describe is to allow high statistics studies of particular physical observables after a low statistics full fermion simulation has been used to prepare the stage., Comment: 34 pages, 5 postscript figures, uses revtex and psfig

In June 2004 QCD was simulated for the first time at sustained speed exceeding 1 TeraFlops in the BlueGene/L supercomputer at the IBM T.J. Watson Research Lab. The implementation and performance of QCD in the BlueGene/L is presented., Talk presented at Lattice2004(machines), Fermilab, 21-26 June 2004, 3 pages, 5 figures

High performance computers are reviewed. Specially built dedicated machines from APE, Columbia, CP-PACS, and RTNN are described. We also consider recent developments in the commercial computer world, and compare machine performances and costs per sustained Gigaflop.

We study the quenched and unquenched lattice Schwinger model with Wilson fermions. The lowest non-trivial order of the systematic expansion recently proposed by Sexton and Weingarten is shown to allow good estimates of long distance physics from quenched configurations. Results for the static potential and the lowest bound state mass are presented.

We compute from lattice QCD in the valence (quenched) approximation the partial decay widths of the lightest scalar glueball to pairs of pseudoscalar quark-antiquark states. These predictions and values obtained earlier for the scalar glueball's mass are in good agreement with the observed properties of $f_J(1710)$ and inconsistent with all other observed meson resonances., Comment: 12 pages of Latex, 3 PostsScript figures as separate uufile

High Power Computing is unanimously recognized today as strategic for research, industry, society and defence. After a long absence, Europe has set up in 2008, as part of PCRD7, the PRACE project to give European research world class computing access. CEA, which has always been a major actor in this field, has a more global vision covering from the multiple usage of HPC to the development of the necessary technology. To support its strategy, CEA established a scientific computing complex, one of the largest in the world with two petaflop/s machines, set up a research platform on machine architectures, parallelism, and software environment shared with industrial and academic partners, and shows an ambitious roadmap to the exaflop/s. The last result of CEA strategy is the BULL TERA100 machine, the first petaflop/s computer ever designed and built in Europe. The TERA100 machine, followed by the Curie machine of the PRACE project, the HELIOS machine of the Fusion program, and a lot of industrial success are demonstrating the capability of Europe to come back on this challenging market. With this background the European Commission has called in 2012, as part of its HORIZON 2020 program, for an ambitious project: • Provide a world-class European HPC infrastructure, benefitting a broad range of academic and industry users, and especially SMEs, including a workforce well trained in HPC; • Ensure independent access to HPC technologies, systems and services for the EU. The talk will cover what has been done in Europe in this period and what is in preparation to answer the European Commission roadmap.

As large scale distributed systems gather and share more and more computing nodes and storage resources, their energy consumption is exponentially increasing. Next generation computing and data centers might require tens of MWatts to be feasible. Thus designing more efficient systems is a major challenge for computer engineers. This challenge is two-folds: saving money and being ecological by using renewable energy. The goal of this panel is to discuss current trends in energy use and energy costs of data centers and servers, metrics to benchmark computing and data centers energy consumption, and opportunities for reducing those costs through improved hardware and software.

We evaluate $f_{\pi}/ m_{\rho}$, $f_K/ m_{\rho}$, $1/f_{\rho}$, and $ m_{\phi}/(f_{\phi} m_{\rho})$, extrapolated to physical quark mass, zero lattice spacing and infinite volume, for lattice QCD with Wilson quarks in the valence (quenched) approximation. The predicted ratios differ from experiment by amounts ranging from 12\% to 17\% equivalent to between 0.9 and 2.8 times the corresponding statistical uncertainties., Comment: uufiles encoded copy of 40 page Latex article, including 14 figures in Postscript. The long version of hep-lat/9302012, IBM/HET 93-3

We discuss a class of reversible, discrete approximations to Hamilton's equations for use in the hybrid Monte Carlo algorithm and derive an asymptotic formula for the step-size-dependent errors arising from this family of approximations. For lattice QCD with Wilson fermions, we construct several different updates in which the effect of fermion vacuum polarization is given a longer time step than the gauge field's self-interaction. On a 44 lattice, one of these algorithms with an optimal choice of step size is 30% to 40% faster than the standard leapfrog update with an optimal step size.

High Throughput Computing (HTC) environments strive "to provide large amounts of processing capacity to customers over long periods of time by exploiting existing resources on the network" according to Basney and Livny [1]. A single Blue Gene/L rack can provide thousands of CPU resources into HTC environments. This paper discusses the implementation of an asynchronous task dispatch system that exploits a recently released feature of the Blue Gene/L control system - called HTC mode - and presents data on experimental runs consisting of the asynchronous submission of multiple batches of thousands of tasks for financial workloads. The methodology developed here demonstrates how systems with very large processor counts and light-weight kernels can be configured to deliver capacity computing at the individual processor level in future petascale computing systems.

We present a novel performance monitor architecture, implemented in the Blue Gene/PTM supercomputer. This performance monitor supports the tracking of a large number of concurrent events by using a hybrid counter architecture. The counters have their low order data implemented in registers which are concurrently updated, while the high order counter data is maintained in a dense SRAM array that is updated from the registers on a regular basis. The per formance monitoring architecture includes support for per- event thresholding and fast event notification, using a two- phase interrupt-arming and triggering protocol. A first implementation provides 256 concurrent 64b counters which offers an up to 64x increase in counter number compared to performance monitors typically found in microprocessors today, and thereby dramatically expands the capabilities of counter-based performance tuning.

The full format data structures of Dense Linear Algebra hurt the performance of its factorization algorithms. Full format rectangular matrices are the input and output of level the 3 BLAS. It follows that the LAPACK and Level 3 BLAS approach has a basic performance flaw. We describe a new result that shows that representing a matrix A as a collection of square blocks will reduce the amount of data reformating required by dense linear algebra factorization algorithms from O(n3) to O(n2). On an IBM Power3 processor our implementation of Cholesky factorization achieves 92% of peak performance whereas conventional full format LAPACK DPOTRF achieves 77% of peak performance. All programming for our new data structures may be accomplished in standard Fortran, through the use of higher dimensional full format arrays. Thus, new compiler support may not be necessary. We also discuss the role of concatenating submatrices to facilitate hardware streaming. Finally, we discuss a new concept which we call the L1 / L0 cache interface.

We describe a new geometrical discretisation method which exactly preserved topological properties of U(1) Chern-Simons theories. A numerical calculation using these new methods correctly evaluates the Reidemeister Torsion for low order Lens spaces.

This document contains information regarding the standard procedures used to calibrate chemical kinetics parameters for the extended Prout-Tompkins model to match experimental data. Two methods for calibration are mentioned: EZM calibration and GLO calibration. EZM calibration matches kinetics parameters to three data points, while GLO calibration slightly adjusts kinetic parameters to match multiple points. Information is provided regarding the theoretical approach and application procedure for both of these calibration algorithms. It is recommended that for the calibration process, the user begin with EZM calibration to provide a good estimate, and then fine-tune the parameters using GLO. Two examples have been provided to guide the reader through a general calibrating process.