Search

Traditionally, high energy physics (HEP) experiments have relied on x86 CPUs for the majority of their significant computing needs. As the field looks ahead to the next generation of experiments such as DUNE and the High-Luminosity LHC, the computing demands are expected to increase dramatically. To cope with this increase, it will be necessary to take advantage of all available computing resources, including GPUs from different vendors. A broad landscape of code portability tools -- including compiler pragma-based approaches, abstraction libraries, and other tools -- allow the same source code to run efficiently on multiple architectures. In this paper, we use a test code taken from a HEP tracking algorithm to compare the performance and experience of implementing different portability solutions.

Next generation High-Energy Physics (HEP) experiments are presented with significant computational challenges, both in terms of data volume and processing power. Using compute accelerators, such as GPUs, is one of the promising ways to provide the necessary computational power to meet the challenge. The current programming models for compute accelerators often involve using architecture-specific programming languages promoted by the hardware vendors and hence limit the set of platforms that the code can run on. Developing software with platform restrictions is especially unfeasible for HEP communities as it takes significant effort to convert typical HEP algorithms into ones that are efficient for compute accelerators. Multiple performance portability solutions have recently emerged and provide an alternative path for using compute accelerators, which allow the code to be executed on hardware from different vendors. We apply several portability solutions, such as Kokkos, SYCL, C++17 std::execution::par and Alpaka, on two mini-apps extracted from the mkFit project: p2z and p2r. These apps include basic kernels for a Kalman filter track fit, such as propagation and update of track parameters, for detectors at a fixed z or fixed r position, respectively. The two mini-apps explore different memory layout formats. We report on the development experience with different portability solutions, as well as their performance on GPUs and many-core CPUs, measured as the throughput of the kernels from different GPU and CPU vendors such as NVIDIA, AMD and Intel., Comment: 26th Intl Conf Computing High Energy & Nuclear Phys (CHEP 2023)

High-energy physics (HEP) experiments have developed millions of lines of code over decades that are optimized to run on traditional x86 CPU systems. However, we are seeing a rapidly increasing fraction of floating point computing power in leadership-class computing facilities and traditional data centers coming from new accelerator architectures, such as GPUs. HEP experiments are now faced with the untenable prospect of rewriting millions of lines of x86 CPU code, for the increasingly dominant architectures found in these computational accelerators. This task is made more challenging by the architecture-specific languages and APIs promoted by manufacturers such as NVIDIA, Intel and AMD. Producing multiple, architecture-specific implementations is not a viable scenario, given the available person power and code maintenance issues. The Portable Parallelization Strategies team of the HEP Center for Computational Excellence is investigating the use of Kokkos, SYCL, OpenMP, std::execution::parallel and alpaka as potential portability solutions that promise to execute on multiple architectures from the same source code, using representative use cases from major HEP experiments, including the DUNE experiment of the Long Baseline Neutrino Facility, and the ATLAS and CMS experiments of the Large Hadron Collider. This cross-cutting evaluation of portability solutions using real applications will help inform and guide the HEP community when choosing their software and hardware suites for the next generation of experimental frameworks. We present the outcomes of our studies, including performance metrics, porting challenges, API evaluations, and build system integration., Comment: 18 pages, 9 Figures, 2 Tables

Gauge covariant smearing based on the 3D lattice Laplacian can be used to create extended operators that have better overlap with hadronic ground states. For staggered quarks, we make use of two-link parallel transport to preserve taste properties. We have implemented the procedure in QUDA. We present the performance of this code on the NVIDIA A100 GPUs in Indiana University's Big Red 200 supercomputer and on the AMD MI250X GPUs in Oak Ridge Leadership Computer Facility's (OLCF's) Crusher and discuss its scalability. We also study the performance improvement from using NVSHMEM on OLCF's Summit. Reusing precomputed two-link products for all sources and sinks, it reduces the total smearing time for a baryon correlator measurement by a factor of 100-120 as compared with the original MILC code and reduces the overall time by 60-70%., Comment: 10 pages, 6 figures, proceedings of the 39th International Symposium on Lattice Field Theory (Lattice 2022), 8-13 August, 2022, Bonn, Germany

Contribution from the USQCD Collaboration to the Proceedings of the US Community Study on the Future of Particle Physics (Snowmass 2021)., Comment: 27 pp. main text, 4 pp. appendices, 29 pp. references, 1 p. index

Today's world of scientific software for High Energy Physics (HEP) is powered by x86 code, while the future will be much more reliant on accelerators like GPUs and FPGAs. The portable parallelization strategies (PPS) project of the High Energy Physics Center for Computational Excellence (HEP/CCE) is investigating solutions for portability techniques that will allow the coding of an algorithm once, and the ability to execute it on a variety of hardware products from many vendors, especially including accelerators. We think without these solutions, the scientific success of our experiments and endeavors is in danger, as software development could be expert driven and costly to be able to run on available hardware infrastructure. We think the best solution for the community would be an extension to the C++ standard with a very low entry bar for users, supporting all hardware forms and vendors. We are very far from that ideal though. We argue that in the future, as a community, we need to request and work on portability solutions and strive to reach this ideal., Comment: 13 pages, 1 figure. Contribution to Snowmass 2021

Programming for a diverse set of compute accelerators in addition to the CPU is a challenge. Maintaining separate source code for each architecture would require lots of effort, and development of new algorithms would be daunting if it had to be repeated many times. Fortunately there are several portability technologies on the market such as Alpaka, Kokkos, and SYCL. These technologies aim to improve the developer productivity by making it possible to use the same source code for many different architectures. In this paper we use heterogeneous pixel reconstruction code from the CMS experiment at the CERNL LHC as a realistic use case of a GPU-targeting HEP reconstruction software, and report experience from prototyping a portable version of it using Kokkos. The development was done in a standalone program that attempts to model many of the complexities of a HEP data processing framework such as CMSSW. We also compare the achieved event processing throughput to the original CUDA code and a CPU version of it., Comment: Submitted to 25th International Conference on Computing in High-Energy and Nuclear Physics (vCHEP 2021)

This work continues our program of lattice-QCD baryon physics using staggered fermions for both the sea and valence quarks. We present a proof-of-concept study that demonstrates, for the first time, how to calculate baryon matrix elements using staggered quarks for the valence sector. We show how to relate the representations of the continuum staggered flavor-taste group $\text{SU}(8)_{FT}$ to those of the discrete lattice symmetry group. The resulting calculations yield the normalization factors relating staggered baryon matrix elements to their physical counterparts. We verify this methodology by calculating the isovector vector and axial-vector charges $g_V$ and $g_A$. We use a single ensemble from the MILC Collaboration with 2+1+1 flavors of sea quark, lattice spacing $a\approx 0.12$ fm, and a pion mass $M_\pi\approx305$ MeV. On this ensemble, we find results consistent with expectations from current conservation and neutron beta decay. Thus, this work demonstrates how highly-improved staggered quarks can be used for precision calculations of baryon properties, and, in particular, the isovector nucleon charges., Comment: 19 pages, 9 figures

We present the first computation in a program of lattice-QCD baryon physics using staggered fermions for sea and valence quarks. For this initial study, we present a calculation of the nucleon mass, obtaining $964\pm16$ MeV with all sources of statistical and systematic errors controlled and accounted for. This result is the most precise determination to date of the nucleon mass from first principles. We use the highly-improved staggered quark action, which is computationally efficient. Three gluon ensembles are employed, which have approximate lattice spacings $a=0.09$ fm, $0.12$ fm, and $0.15$ fm, each with equal-mass $u$/$d$, $s$, and $c$ quarks in the sea. Further, all ensembles have the light valence and sea $u$/$d$ quarks tuned to reproduce the physical pion mass, avoiding complications from chiral extrapolations or nonunitarity. Our work opens a new avenue for precise calculations of baryon properties, which are both feasible and relevant to experiments in particle and nuclear physics., Comment: 33 pages, 19 figures; published in Physical Review D

Critical slowing down in Krylov methods for the Dirac operator presents a major obstacle to further advances in lattice field theory as it approaches the continuum solution. Here we formulate a multi-grid algorithm for the Kogut-Susskind (or staggered) fermion discretization which has proven difficult relative to Wilson multigrid due to its first-order anti-Hermitian structure. The solution is to introduce a novel spectral transformation by the K\"ahler-Dirac spin structure prior to the Galerkin projection. We present numerical results for the two-dimensional, two-flavor Schwinger model, however, the general formalism is agnostic to dimension and is directly applicable to four-dimensional lattice QCD., Comment: 48 pages, 37 figures

Lattice quantum chromodynamics simulations in nuclear physics have benefited from a tremendous number of algorithmic advances such as multigrid and eigenvector deflation. These improve the time to solution but do not alleviate the intrinsic memory-bandwidth constraints of the matrix-vector operation dominating iterative solvers. Batching this operation for multiple vectors and exploiting cache and register blocking can yield a super-linear speed up. Block-Krylov solvers can naturally take advantage of such batched matrix-vector operations, further reducing the iterations to solution by sharing the Krylov space between solves. However, practical implementations typically suffer from the quadratic scaling in the number of vector-vector operations. Using the QUDA library, we present an implementation of a block-CG solver on NVIDIA GPUs which reduces the memory-bandwidth complexity of vector-vector operations from quadratic to linear. We present results for the HISQ discretization, showing a 5x speedup compared to highly-optimized independent Krylov solves on NVIDIA's SaturnV cluster., Comment: 15 pages, 14 figures, in press

The past decade has witnessed a dramatic acceleration of lattice quantum chromodynamics calculations in nuclear and particle physics. This has been due to both significant progress in accelerating the iterative linear solvers using multi-grid algorithms, and due to the throughput improvements brought by GPUs. Deploying hierarchical algorithms optimally on GPUs is non-trivial owing to the lack of parallelism on the coarse grids, and as such, these advances have not proved multiplicative. Using the QUDA library, we demonstrate that by exposing all sources of parallelism that the underlying stencil problem possesses, and through appropriate mapping of this parallelism to the GPU architecture, we can achieve high efficiency even for the coarsest of grids. Results are presented for the Wilson-Clover discretization, where we demonstrate up to 10x speedup over present state-of-the-art GPU-accelerated methods on Titan. Finally, we look to the future, and consider the software implications of our findings., Comment: http://dl.acm.org/citation.cfm?id=3014904.3014995}

We present the implementation of twisted mass fermion operators for the QPhiX library. We analyze the performance on the Intel Xeon Phi (Knights Corner) coprocessor as well as on Intel Xeon Haswell CPUs. In particular, we demonstrate that on the Xeon Phi 7120P the Dslash kernel is able to reach 80\% of the theoretical peak bandwidth, while on a Xeon Haswell E5-2630 CPU our generated code for the Dslash operator with AVX2 instructions outperforms the corresponding implementation in the tmLQCD library by a factor of $\sim 5\times$ in single precision. We strong scale the code up to 6.8 (14.1) Tflops in single (half) precision on 64 Xeon Haswell CPUs., Comment: 7 pages, 6 figures. Proceedings of the 33rd International Symposium on Lattice Field Theory, July 14-18, 2015, Kobe, Japan

Although QUDA allows for an efficient computation of many QCD quantities, it is surprinsingly lacking tools to evaluate disconnected diagrams, for which GPUs are specially well suited. We aim to fill this gap by creating our own branch of QUDA, which includes new kernels and functions required to calculate fermion loops using several methods and fermionic regularizations., Comment: 7 pages, 4 figures. Proceedings of the talk given during the code session of the 31st International Symposium on Lattice Field Theory, July 29 - August 3, 2013, Mainz, Germany. Added a missing reference (number [4])

We discuss an extension of the QUDA library for the Wilson twisted mass operator. A performance analysis is presented for both degenerate and non-degenerate flavor doublets. The degenerate twisted mass fermion operator runs at up to 190, 487 and 856 Gflops, for double, single and half precisions respectively on recent NVIDIA Kepler GPUs, while our implementation for the non-degenerate flavor doublet allows to reach 163, 516 and 879 GFlops, respectively. The code is currently in production for the hadron structure study., Comment: 7 pages, 2 figures, presented at the 31st International Symposium on Lattice Field Theory (Lattice 2013), 29 July - 3 August 2013, Mainz, Germany

Implementation of basic local Monte-Carlo algorithms on ATI Graphics Processing Units (GPU) is investigated. The Ising model and pure SU(2) gluodynamics simulations are realized with the Compute Abstraction Layer (CAL) of ATI Stream environment using the Metropolis and the heat-bath algorithms, respectively. We present an analysis of both CAL programming model and the efficiency of the corresponding simulation algorithms on GPU. In particular, the significant performance speed-up of these algorithms in comparison with serial execution is observed., Comment: 15 pages, 4 figures, 2 tables

We extend the results of Ref. [arXiv:0705.4294] to noncommutative gauge theories at finite temperature. In particular, by making use of the background field method, we analyze renormalization issues and the high-temperature asymptotics of the one-loop Euclidean free energy of the noncommutative U(1) gauge model within imaginary time formalism. As a by-product, the heat trace of the non-minimal photon kinetic operator on noncommutative $S^1 \times R^3$ manifold taken in an arbitrary background gauge is investigated. All possible types of noncommutativity on $S^1 \times R^3$ are considered. It is demonstrated that the non-planar sector of the model does not contribute to the free energy of the system at high temperature. The obtained results are discussed., Comment: Talk given at the 8th Workshop "Quantum Field Theory Under the Influence of External Conditions", Leipzig, Germany, 17-21 September 2007; 7 pages, v2: minor changes

We generalize the Endo formula originally developed for the computation of the heat kernel asymptotic expansion for non-minimal operators in commutative gauge theories to the noncommutative case. In this way, the first three non-zero heat trace coefficients of the non-minimal U(N) gauge field kinetic operator on the Moyal plane taken in an arbitrary background are calculated. We show that the non-planar part of the heat trace asymptotics is determined by U(1) sector of the gauge model. The non-planar or mixed heat kernel coefficients are shown to be gauge-fixing dependent in any dimension of space-time. In the case of the degenerate deformation parameter the lowest mixed coefficients in the heat expansion produce non-local gauge-fixing dependent singularities of the one-loop effective action that destroy the renormalizability of the U(N) model at one-loop level. The twisted-gauge transformation approach is discussed., Comment: 21 pages, misprints corrected