Search

This work presents the GPU acceleration of the open-source code CaNS for very fast massively-parallel simulations of canonical fluid flows. The distinct feature of the many-CPU Navier–Stokes solver in CaNS is its fast direct solver for the second-order finite-difference Poisson equation, based on the method of eigenfunction expansions. The solver implements all the boundary conditions valid for this type of problems in a unified framework. Here, we extend the solver for GPU-accelerated clusters using CUDA Fortran. The porting makes extensive use of CUF kernels and has been greatly simplified by the unified memory feature of CUDA Fortran, which handles the data migration between host (CPU) and device (GPU) without defining new arrays in the source code. The overall implementation has been validated against benchmark data for turbulent channel flow and its performance assessed on a NVIDIA DGX-2 system (16 T V100 32Gb, connected with NVLink via NVSwitch). The wall-clock time per time step of the GPU-accelerated implementation is impressively small when compared to its CPU implementation on state-of-the-art many-CPU clusters, as long as the domain partitioning is sufficiently small that the data resides mostly on the GPUs. The implementation has been made freely available and open source under the terms of an MIT license.

The reliability of cardiovascular computational models depends on the accurate solution of the hemodynamics, the realistic characterization of the hyperelastic and electric properties of the tissues along with the correct description of their interaction. The resulting fluid–structure–electrophysiology interaction (FSEI) thus requires an immense computational power, usually available in large supercomputing centers, and requires long time to obtain results even if multi–CPU processors are used (MPI acceleration). In recent years, graphics processing units (GPUs) have emerged as a convenient platform for high performance computing, as they allow for considerable reductions of the time–to–solution. This approach is particularly appealing if the tool has to support medical decisions that require solutions within reduced times and possibly obtained by local computational resources. Accordingly, our multi–physics solver [1] has been ported to GPU architectures using CUDA Fortran to tackle fast and accurate hemodynamics simulations of the human heart without resorting to large–scale supercomputers. This work describes the use of CUDA to accelerate the FSEI on heterogeneous clusters, where both the CPUs and GPUs are used in synergistically with minor modifications of the original source code. The resulting GPU accelerated code solves a single heartbeat within a few hours (from three to ten depending on the grid resolution) running on premises computing facility made of few GPU cards, which can be easily installed in a medical laboratory or in a hospital, thus opening towards a systematic computational fluid dynamics (CFD) aided diagnostic.

We study the statistics of passive scalars at $Pr=1$ , for turbulent flow within a smooth straight pipe of circular cross section up to $Re_{\tau } \approx 6000$ using direct numerical simulation (DNS) of the Navier–Stokes equations. While featuring a general organisation similar to the axial velocity field, passive scalar fields show additional energy at small wavenumbers, resulting in a higher degree of mixing and in a $k^{-4/3}$ spectral inertial range. The DNS results highlight logarithmic growth of the inner-scaled bulk and mean centreline scalar values with the friction Reynolds number, implying an estimated scalar von Kármán constant $k_{\theta } \approx 0.459$ , which also nicely fits the mean scalar profiles. The DNS data are used to synthesise a modified form of the classical predictive formula of Kader & Yaglom (Intl J. Heat Mass Transfer, vol. 15 (12), 1972, pp. 2329–2351), which points to some shortcomings of the original formulation. Universality of the mean core scalar profile in defect form is recovered, with very nearly parabolic shape. Logarithmic growth of the buffer-layer peak of the scalar variance is found in the Reynolds number range under scrutiny, which well conforms with Townsend's attached-eddy hypothesis, whose validity is also supported by the spectral maps. The behaviour of the turbulent Prandtl number shows good universality in the outer wall layer, with values $Pr_t \approx 0.84$ , as also found in previous studies, but closer to unity near the wall, where existing correlations do not reproduce the observed trends.

We study turbulent flows in a smooth straight pipe of circular cross-section up to friction Reynolds number $({\textit {Re}}_{\tau }) \approx 6000$ using direct numerical simulation (DNS) of the Navier–Stokes equations. The DNS results highlight systematic deviations from Prandtl friction law, amounting to approximately $2\,\%$ , which would extrapolate to approximately $4\,\%$ at extreme Reynolds numbers. Data fitting of the DNS friction coefficient yields an estimated von Kármán constant $k \approx 0.387$ , which nicely fits the mean velocity profile, and which supports universality of canonical wall-bounded flows. The same constant also applies to the pipe centreline velocity, thus providing support for the claim that the asymptotic state of pipe flow at extreme Reynolds numbers should be plug flow. At the Reynolds numbers under scrutiny, no evidence for saturation of the logarithmic growth of the inner peak of the axial velocity variance is found. Although no outer peak of the velocity variance directly emerges in our DNS, we provide strong evidence that it should appear at ${\textit {Re}}_{\tau } \gtrsim 10^4$ , as a result of turbulence production exceeding dissipation over a large part of the outer wall layer, thus invalidating the classical equilibrium hypothesis.

The AFiD code, an open source solver for the incompressible Navier-Stokes equations ({\color{blue}\burl{http://www.afid.eu}}), has been ported to GPU clusters to tackle large-scale wall-bounded turbulent flow simulations. The GPU porting has been carried out in CUDA Fortran with the extensive use of kernel loop directives (CUF kernels) in order to have a source code as close as possible to the original CPU version; just a few routines have been manually rewritten. A new transpose scheme, which is not limited to the GPU version only and can be generally applied to any CFD code that uses pencil distributed parallelization, has been devised to improve the scaling of the Poisson solver, the main bottleneck of incompressible solvers. The GPU version can reduce the wall clock time by an order of magnitude compared to the CPU version for large meshes. Due to the increased performance and efficient use of memory, the GPU version of AFiD can perform simulations in parameter ranges that are unprecedented in thermally-driven wall-bounded turbulence. To verify the accuracy of the code, turbulent Rayleigh-B\'enard convection and plane Couette flow are simulated and the results are in good agreement with the experimental and computational data that published in previous literatures., Comment: 33 pages, 8 figures, submitted

This paper presents a GPU implementation of the graph triangle counting operation based on the set intersection algorithm. The algorithm is implemented in four kernels optimized for different types of graphs in a code delivering performance higher than the current state-of-the-art and without preprocessing the input graph. At runtime, a lightweight heuristic is used to select the kernel to run based on the specific graph taken as input. In contrast to previous works, the presented approach takes advantage of a set intersection operation implemented via bitmaps. Moreover, the simplicity of the approach allows the code to have limited size and engineering complexity.

We present and make available novel implementations of the two-dimensional Ising model that is used as a benchmark to show the computational capabilities of modern Graphic Processing Units (GPUs). The rich programming environment now available on GPUs and flexible hardware capabilities allowed us to quickly experiment with several implementation ideas: a simple stencil-based algorithm, recasting the stencil operations into matrix multiplies to take advantage of Tensor Cores available on NVIDIA GPUs, and a highly optimized multi-spin coding approach. Using the managed memory API available in CUDA allows for simple and efficient distribution of these implementations across a multi-GPU NVIDIA DGX-2 server. We show that even a basic GPU implementation can outperform current results published on TPUs (Yang et al., 2019) and that the optimized multi-GPU implementation can simulate very large lattices faster than custom FPGA solutions (Ortega-Zamorano et al., 2016). Program summary Program title: cuIsing (optimized). CPC Library link to program files: http://dx.doi.org/10.17632/xrb9xtkbcp.1 Licensing provisions: MIT license. Programming languages: CUDA C, Python. Nature of problem: Two dimensional Ising model for spin systems. Solution method: Checkerboard Metropolis algorithm.

We describe the porting of PWscf (Plane-Wave Self Consistent Field), a key component of the Quantum ESPRESSO open-source suite of codes for materials modeling, to GPU systems using CUDA Fortran. Kernel loop directives (CUF kernels) have been extensively used in order to have a single source code for both CPU and GPU implementations. The results of the GPU version have been carefully validated and the performance of the code on several GPU systems (both x86 and POWER8 based) has been compared with traditional Intel multi-core (CPU only) systems. This current GPU version can reduce the time-to-solution by an average factor of 2–3 running two different input cases widely used as benchmarks on small and large high performance computing systems.

Graphics processing unit accelerated supercomputers have proved to be very effective, especially with regard to power efficiency, for accelerating compute intensive applications like the high-performance Linpack used in the TOP500 list. This paper presents the details of a CUDA implementation of the high-performance conjugate gradient, a new proposed benchmark that better represents modern application workloads which rely more heavily on memory system and network performance than high-performance Linpack. The results obtained at full scale on the largest graphics processing unit supercomputers in the world, Titan, the Cray XK7 at ORNL and Piz-Daint, the Cray XC30 at CSCS, indicate that graphics processing unit accelerated supercomputers are also very effective for this type of workload. A comparison with other architectures is also presented, showing that graphics processing units, with their high memory bandwidth, are the highest performing devices for this new benchmark.

This paper presents the details of a CUDA implementation of the Subgraph Isomorphism Graph Challenge, a new effort aimed at driving progress in the graph analytics field. challenge consists of two graph analytics: triangle counting and k-truss. We present our CUDA implementation of the graph triangle counting operation and of the k-truss subgraph decomposition. Both implementations share the same codebase taking advantage of a set intersection operation implemented via bitmaps. The analytics are implemented in four kernels optimized for different types of graphs. At runtime, lightweight heuristics are used to select the kernel to run based on the specific graph taken as input.

We present recent extensions of the MUPHY computational framework for multi-scale simulation of complex bio-fluidic phenomena in real-life geometries. The new framework, which builds on concurrent advances of the computational modeling and parallelization techniques, is able to simulate suspensions with several hundreds of millions of finite-size bodies, interacting with each other and with the surrounding fluid, in geometries of realistic anatomic complexity. Blood flow through the human coronary arteries, at physiological hematocrit values, is simulated with a spatial resolution of 10 micrometers, comparable with the size of red blood cells. The simulation exhibits excellent parallel scalability on a cluster of 4000 M2050 Nvidia GPUs, with an aggregate performance close to 1 Petaflop/s.

This paper presents the details of a CUDA implementation of the PageRank pipeline benchmark [1], a new proposed benchmark aimed to compare and measure the capabilities of big data systems. The reference implementation is only serial at the moment, but our CUDA implementation is parallel. The results indicate that GPU accelerated systems have considerable potential for big data workloads.

We present simulations of blood and cancer cell separation in complex microfluidic channels with subcellular resolution, demonstrating unprecedented time to solution, performing at 65.5% of the available 39.4 PetaInstructions/s in the 18, 688 nodes of the Titan supercomputer. These simulations outperform by one to three orders of magnitude the current state of the art in terms of numbers of simulated cells and computational elements. The computational setup emulates the conditions and the geometric complexity of microfluidic experiments and our results reproduce the experimental findings. These simulations provide sub-micron resolution while accessing time scales relevant to engineering designs. We demonstrate an improvement of up to 45X over competing state-of-the-art solvers, thus establishing the frontiers of simulations by particle based methods. Our simulations redefine the role of computational science for the development of microfluidics -- a technology that is becoming as important to medicine as integrated circuits have been to computers.

There is growing interest in studying large scale graphs having millions of vertices and billions of edges, up to the point that a specific benchmark, called Graph500, has been defined to measure the performance of graph algorithms on modern computing architectures. At first glance, Graphics Processing Units (GPUs) are not an ideal platform for the execution of graph algorithms that are characterized by low arithmetic intensity and irregular memory access patterns. For studying really large graphs, multiple GPUs are required to overcome the memory size limitations of a single GPU. In the present paper, we propose several techniques to minimize the communication among GPUs.

The High Performance Conjugate Gradient (HPCG) benchmark has been recently proposed as a complement to the High Performance Linpack (HPL) benchmark currently used to rank supercomputers in the Top500 list. This new benchmark solves a large sparse linear system using a multigrid preconditioned conjugate gradient (PCG) algorithm. The PCG algorithm contains the computational and communication patterns prevalent in the numerical solution of partial differential equations and is designed to better represent modern application workloads which rely more heavily on memory system and network performance than HPL. GPU accelerated supercomputers have proved to be very effective, especially with regard to power efficiency, for accelerating compute intensive applications like HPL. This paper will present the details of a CUDA implementation of HPCG, and the results obtained at full scale on the largest GPU supercomputers available: the Cray XK7 at ORNL and the Cray XC30 at CSCS. The results indicate that GPU accelerated supercomputers are also very effective for this type of workload.

Graphics processing units (GPU) are currently used as a cost-effective platform for computer simulations and big-data processing. Large scale applications require that multiple GPUs work together but the efficiency obtained with cluster of GPUs is, at times, sub-optimal because the GPU features are not exploited at their best. We describe how it is possible to achieve an excellent efficiency for applications in statistical mechanics, particle dynamics and networks analysis by using suitable memory access patterns and mechanisms like CUDA streams, profiling tools, etc. Similar concepts and techniques may be applied also to other problems like the solution of Partial Differential Equations.

The e ow around an idealized road vehicle at Reynolds numbers up to 10 5 has been simulated using large eddy simulation; the numerical technique is based on the immersed boundary approach, which allows efe cient calculations to be carried out on a Cartesian grid. The effect of the Reynolds number and the wake modie cations produced by drag reduction devices attached to the base have been analyzed and compared with available experimental data. Averaged drag coefe cient and mean velocity proe les are in good agreement with measurement. The effect of subgrid-scale modeling (Smagorinsky and dynamic model ) has also been studied.

We study the energy used by a production-level regional climate and weather simulation code on a distributed memory system with hybrid CPU-GPU nodes. The code is optimised for both processor architectures, for which we investigate both time and energy to solution. Operational constraints for time to solution can be met with both processor types, although on different numbers of nodes. Energy to solution is a factor 3 lower with GPUs, but strong scaling can be pushed to larger node counts with CPUs to minimize time to solution. Our data shows that an affine relationship exists between energy and node hours consumed by the simulation. We use this property to devise a simple and practical methodology for optimising for energy efficiency that can be applied to other applications, which we demonstrate with the HPCG benchmark. We conclude with a discussion about the relationship to the commonly-used GF/Watt metric.

This paper presents the details of a Synthetic Aperture Radar (SAR) imaging on the smallest CUDA-capable platform available, the Jetson TK1. The results indicate that GPU accelerated embedded platforms have considerable potential for this type of workload and in conjunction with low power consumption, light weight and standard programming tools, could open new horizons in the embedded space.

This chapter shows some applications of the Fast Fourier Transform and how to utilize the CUFFT library.

This chapter goes through the steps that one typically takes to optimize a CUDA Fortran application. Reducing the overhead of transferring data between the host and device is discussed first in terms of reducing the amount of data transferred and making such transfers efficient and possible. Once the data are on the device, we discuss at length how to efficiently access these data from kernels, including topics of global memory data coalescing, use of on-chip shared memory, and the read-only constant and texture memories. The topic of launching kernels with enough parallelism, whether in the form of instruction-level or thread-level parallelism, is also discussed. The final section covers instruction optimization.

This case study discusses how a finite-difference code using a large one-dimensional stencil is optimized on a three-dimensional mesh. Achieving good performance requires maximizing data reuse through use of shared memory. Attention is paid to coalescing global memory accesses for derivatives in all directions and using constant memory effectively.

A prerequisite to performance optimization is a means to accurately time portions of a code and subsequently describes how to use such timing information to assess code performance. In this chapter we first discuss how to time kernel execution using CPU timers, CUDA events, and the Command Line Profiler as well as the nvprof profiling tool. We then discuss how timing information can be used to determine the limiting factor of kernel execution. Finally, we discuss how to calculate performance metrics, especially related to bandwidth, and how such metrics should be interpreted.