Search

A numerical algorithm for simulating gas dynamics in rotor–stator systems based on sliding meshes and edge-based schemes is described. The paper pays particular attention to the multilevel MPI + OpenMP parallelization for cluster systems. Parallel efficiency is demonstrated on up to 1400 cores as well as on Intel Xeon Phi accelerators. The scheme is verified by solving a linear acoustic problem. The robustness of the algorithm is demonstrated on the simulation of a model fan.

The present work is devoted to accelerating the NOISEtte code and lowering its memory consumption. This code for scale-resolving supercomputer simulations of compressible turbulent flows is based on higher-accuracy methods for unstructured mixed-element meshes and hierarchical MPI $$+$$ OpenMP parallelization for cluster systems with manycore processors. We demonstrate modifications of the underlying numerical method and its parallel implementation, which consist, in particular, in using a simplified approximation method for viscous fluxes and mixed floating-point precision. The modified version has been tested on several representative cases. The performance measurements and validation results are presented.

In this work, we aim to shed light to the following research question: can we find a nonlinear tensorial subgrid-scale (SGS) heat flux model with good physical and numerical properties, such that we can obtain satisfactory predic- tions for buoyancy-driven turbulent flows? This is motivated by our findings showing that the classical (linear) eddy-diffusivity assumption, qeddy ¿ ¿T, fails to provide a reasonable approximation for the actual SGS heat flux, q = uT - uT: namely, a priori analysis for air-filled Rayleigh-Bénard convection (RBC) clearly shows a strong misalignment. In the quest for more accurate models, we firstly study and confirm the suitability of the eddy-viscosity assumption for RBC carrying out a posteriori tests for different models at very low Prandtl numbers (liquid sodium, Pr = 0.005) where no heat flux SGS activity is expected. Then, different (nonlinear) tensor-diffusivity SGS heat flux models are studied a priori using DNS data of air-filled (Pr = 0.7) RBC at Rayleigh numbers up to 1011. Apart from having good alignment trends with the actual SGS heat flux, we also restrict ourselves to models that are numerically stable per se and have the proper cubic near-wall behavior. This analysis leads to a new family of SGS heat flux models based on the symmetric positive semi-definite tensor GGT where G = ¿u, i.e. q ¿ GGT¿T, and the invariants of the GGT tensor F.X.T. and F.D. are supported by the Ministerio de Economía y Competitividad, Spain (ENE2017-88697-R). F.X.T. and C.O. are supported by the Generalitat de Catalunya RIS3CAT-FEDER, FusionCAT project (001-P-001722). F.X.T. was financially supported by a Ramón y Cajal post-doctoral contract (RYC-2012-11996). F.X.T. and A.G. are supported by the Research project 20-02-01 of the Department of the Moscow Center for Fundamental and Applied Mathematics at the Keldysh Insti-tute of Applied Mathematics of RAS. F.D. is supported by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the K1MET center for metallurgical research in Austria (www.k1-met.com). Calculations have been performed on the MareNostrum 4 supercomputer at the BSC (PRACE 15th Call, Ref. 2016163972, “Exploring new fron-tiers in Rayleigh-Bénard convection”; and RES project FI-2019-1-0040 “Exploring nonlinear subgrid-scale heat flux models for buoyancy driven flows”). Preliminary simulations were carried out using computational resources of MCC NRC “Kurchatov Institute”, http://compu ting.nrcki .ru/. The authors thankfully acknowl-edge these institutions

The use of flux limiters is widespread within the scientific computing community to capture shock dis- continuities and are of paramount importance for the temporal integration of high-speed aerodynamics, multiphase flows and hyperbolic equations in general. Meanwhile, the breakthrough of new computing architectures and the hybridization of supercomputer systems pose a huge portability challenge, particularly for legacy codes, since the computing subroutines that form the algorithms, the so-called kernels, must be adapted to various complex parallel program- ming paradigms. From this perspective, the development of innovative implementations relying on a minimalist set of kernels simplifies the deployment of scientific computing software on state-of-the-art supercomputers, while it requires the reformulation of algorithms, such as the aforementioned flux lim- iters. Equipped with basic algebraic topology and graph theory underlying the classical mesh concept, a new flux limiter formulation is presented based on the adoption of algebraic data structures and kernels. As a result, traditional flux limiters are cast into a stream of only two types of computing kernels: sparse matrix-vector multiplication and generalized pointwise binary operators. The newly proposed formulation eases the deployment of such a numerical technique in massively parallel, potentially hybrid, computing systems and is demonstrated for a canonical advection problem. The work of N. V. and X. Á. F. has been supported by the Government of Catalonia, FI AGAUR predoctoral grants 2019FI_B2_ 000104 and 2019FI_B2_00076. N. V., X. Á. F., J. C., A. O. and F. X. T. have been funded by the Spanish Research Agency, ANUMESOL project ENE2017-88697-R. J. C. has also been funded by Spanish Research Agency, GALIFLOW project ENE2015-70662-P. The studies of this work have been carried out using the MareNostrum 4 supercomputer of the Barcelona Supercomput- ing Center, projects IM-2019-3-0026 and IM-2020-1-0006; the TSUBAME3.0 supercomputer of the Global Scientific Information and Computing Center at Tokyo Institute of Technology; the Lomonosov-2 supercomputer of the shared research facilities of HPC computing resources at Lomonosov Moscow State University; the K-60 hybrid cluster of the collective use center of the Keldysh Institute of Applied Mathematics; the JFF cluster of the Heat and Mass Transfer Technological Center at Technical University of Cat- alonia. The authors thankfully acknowledge these institutions for the compute time and technical support.

This paper presents a new approach for mitigating the unphysical delay in the Reynolds-averaged Navier–Stokes (RANS) to large-eddy simulation (LES) transition, often referred to as the gray area, which is a common issue for hybrid RANS–LES turbulence models such as delayed-detached eddy simulation. An existing methodology designed for improving the LES performance in complex flows is adapted and tested. This is based on reducing the numerical diffusion in critical areas for permitting a more accurate development of turbulence. The new formulation comprises both a two-dimensional sensitive velocity gradient model and an alternative definition of the subgrid length scale, which are tested both individually and in tandem, and compared with the other formulations commonly used for addressing the gray area. Four test cases are examined, a flat plate, two variants of the incompressible backwardfacing step, and an open jet compressible case, all of which are selected to expose the adverse impact of numerical diffusion that this study seeks to address. Furthermore, the proposed changes are implemented in two different codes for the purpose of cross-validation. Encouraging results are observed, supporting the suitability of the new approach as a candidate for addressing the gray area issue in flows of this kind. This work was financially supported by the Ministerio de Economía y Competitividad, Spain (No. ENE2017-88697-R). A.P.V. was supported by an FI-DGR 2016 predoctoral contract (No. 2018FI_ B2_00072) financed by Generalitat de Catalunya, Spain. F.X.T. was supported by a Ramón y Cajal postdoctoral contract (No. RYC-2012- 11996) financed by the Ministerio de Economía y Competitividad, Spain. A.D. and A.G. were supported by Moscow Center for Fundamental and Applied Mathematics, Agreement with the Ministry of Science and Higher Education of the Russian Federation, No. 075- 15-2019-1623. The NOISEtte computations were carried out using the equipment of the shared research facilities of HPC computing resources at Lomonosov Moscow State University and the computing resources of the federal collective usage center Complex for Simulation and Data Processing for Mega-science Facilities at NRC “Kurchatov Institute” (http://ckp.nrcki.ru/).

This work is devoted to evaluating the performance of a multicore CPU Elbrus-8C processor in supercomputer computational fluid dynamics (CFD) applications. Parallel simulation codes based on highly accurate methods on unstructured meshes for modeling the turbulent flows are considered. The main features of the Elbrus architecture are described, and the approaches for adaptating and optimizing the computing software are presented. The performance is investigated for both entire algorithms and their operations separately. The results of comparative testing with different multicore Intel and AMD CPUs are presented.

This work is devoted to acceleration and upgrade of the CFD code NOISEtte for scale-resolving simulations of compressible turbulent flows using edge-based high-accuracy methods on unstructured hybrid meshes. Attempts to extend the baseline multilevel MPI+OpenMP parallelization towards GPU-based hybrid systems have faced the problem: the code is too complex. It is an in-house research code with plenty of numerical methods, schemes, models, most of which are experimental and are not used in practical simulations. This chaotic zoo leads to excessive conditional branches, switches, redundant functional calls that slow down computations. Although the parallel algorithm is fully adapted to the stream processing paradigm, such an immense amount of code is too difficult to port efficiently to OpenCL or CUDA and maintain it in consistency with the CPU version. An approach to survive in the process of adaptation to hybrid systems has been elaborated. It consists of various components, such as creation of a simplified configurations, combining different stages of the algorithm in order to reduce memory traffic, collapsing multiple functions in one function without branches and switches, mixing single and double precision, etc. As a result, the upgraded code is about twice as fast on CPUs and can use GPUs from different manufacturers AMD, NVIDIA, Intel through the OpenCL standard.

The incompressible Navier-Stokes equations stand as the best mathematical model for turbulent flows. However, direct numerical simulations at high Reynolds numbers are not yet feasible because the convective term produces far too many relevant scales of motion, thus remaining limited to relatively low-Reynolds numbers. Dynamically less complex mathematical formulations have been developed for coarse-grain simulations, like the well known eddy-viscosity models. Most of these models are based on the combination of invariants of a symmetric tensor that depends on the gradient of the resolved velocity field, G = Ñu, and should properly detect different flow configurations (laminar and 2D flows, near-wall behavior, transitional regime, etc.). Brand-new models have been constructed considering the first three invariants of the symmetric tensor GGT with excellent results;1 hence, it is formally based on the lowest-order approximation of the subgrid stress tensor, t(u) = D2 12GGT +O(D4), i.e. the gradient model proposed by Clark.2 Furthermore, these models have been implemented on a pseudo-spectral algorithm with a fully-explicit second-order time-integration method.3 The performance of this special configuration has been successfully tested for decaying isotropic turbulence and a turbulent channel flow. It is currently being developed for a semi-infinite boundary layer with periodic conditions as a previous step to carry out wind farm simulations. Details of the implementation and numerical results will be presented. The work has been financially supported by a competitive R+D project (ENE2017-88697-R) by the Spanish Research Agency. The authors thankfully acknowledge these institutions.

Backward-facing step (BFS) constitutes a canonical configuration to study wall-bounded flows subject to massive expansions produced by abrupt changes in geometry. Recirculation flow regions are common in this type of flow, driving the separated flow to its downstream reattachment. Consequently, strong adverse pressure gradients arise through this process, feeding flow instabilities. Therefore, both phenomena are strongly correlated as the recirculation bubble shape defines how the flow is expanded, and how the pressure rises. In an incompressible flow, this shape depends on the Reynolds value and the expansion ratio. The influence of these two variables on the bubble length is widely studied, presenting an asymptotic behaviour when both parameters are beyond a certain threshold. This is the usual operating point of many practical applications, such as in aeronautical and environmental engineering. Several numerical and experimental studies have been carried out regarding this topic. The existing simulations considering cases beyond the above-mentioned threshold have only been achieved through turbulence modelling, whereas direct numerical simulations (DNS) have been performed only at low Reynolds numbers. Hence, despite the great importance of achieving this threshold, there is a lack of reliable numerical data to assess the accuracy of turbulence models. In this context, a DNS of an incompressible flow over a BFS is presented in this paper, considering a friction Reynolds number ($Re_{\unicode[STIX]{x1D70F}}$) of 395 at the inflow and an expansion ratio 2. Finally, the elongation of the Kelvin–Helmholtz instabilities along the shear layer is also studied.

This paper describes the parallel algorithm of the NOISEtte code for computational fluid dynamics and aeroacoustics simulations. It is based on a family of higher-accuracy numerical schemes for unstructured hybrid meshes. The multilevel MPI + OpenMP parallelization is described in detail. Performance results are presented for various supercomputers and applications.

A heterogeneous parallel algorithm for simulation of compressible turbulent flows and its portable software implementation are presented. The underlying numerical method is based on a family of higher accuracy edge-based reconstruction schemes on unstructured mixed-element meshes. The proposed parallel solution can engage a large number of computing devices of most of the existing computing architectures used in modern supercomputers, including manycore CPUs and GPUs. It is capable of co-execution on both CPUs and accelerators simultaneously. The multilevel parallel algorithm combines: MPI for distributing workload among hybrid cluster nodes and between devices inside nodes; OpenMP for manycore CPUs and other supporting devices, such as Intel Xeon Phi; OpenCL for massively-parallel accelerators, such as GPUs of various vendors, including NVIDIA, AMD, Intel. The main focus is on the adaptation of the numerical method and its computational algorithm to the stream processing parallel paradigm. The very limited device memory inherent in GPU computing is also taken into account. A detailed description of the parallel algorithm is presented, as well as the techniques used for its efficient parallel implementation. Special attention is paid to implicit time integration with its linear solver and calculation of convective fluxes and viscous terms. The use of mixed floating-point precision and overlapping communications and computations is also discussed. Parallel performance is demonstrated in practical applications on different kinds of supercomputers using up to 10 thousand cores and multiple GPUs of comparable overall performance.

In this work, we plan to shed light on the following research question: can we find a nonlinear subgrid-scale (SGS) heat flux model with good physical and numerical properties, such that we can obtain satisfactory predictions for buoyancy-driven turbulent flows? This is motivated by our findings showing that the classical (linear) eddy-diffusivity assumption fails to provide a reasonable approximation for the SGS heat flux. This was shown in our work [1] where SGS features have been studied a priori for a Rayleigh–Benard convection (RBC). We also concluded that nonlinear (or tensorial) models can give good approximations of the actual SGS heat flux. Briefly, the large-eddy simulation (LES) equations arise from applying a spatial commutative filter, with filter length \(\delta \), to the incompressible Navier–Stokes and thermal energy equations.

F.X.T., F.D. and A.O. are financially supported by the Ministerio de Economía y Competitividad, Spain (ENE2017-88697-R). F.X.T. is supported by a Ramón y Cajal postdoctoral contract (RYC-2012-11996). F.D. is supported by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development, and the K1MET center for metallurgical research in Austria (www.k1-met.com). Calculations have been performed on the IBM MareNostrum 4 supercomputer at the BSC (PRACE 15th Call, Ref. 2016163972, \Exploring new frontiers in Rayleigh-Bénard convection"). Preliminary simulations were carried out using computational resources of MCC NRC \Kurchatov Institute", http://computing.nrcki.ru/. The authors thankfully acknowledge these institutions.

The paper describes a particular technical solution targeted at improving reliability and quality of a highly-parallel computational fluid dynamics code written in C++. The code considered is based on rather complex high-accuracy numerical methods and models for simulation of turbulent flows on unstructured hybrid meshes. The cost of software errors is very high in largescale supercomputer simulations. Reproducing and localizing errors, especially “magic” unstable bugs related with wrong memory access, are extremely problematic due to the large amount of computing resources involved. In order to prevent, or at least notably filter out memory bugs, an approach of increased reliability is proposed for representing mesh data and organizing memory access. A set of containers is proposed, which causes no overhead in the release configuration compared to plain arrays. At the same time, it provides throughout access control in the safe mode configuration and additional compile-time protection from programming errors. Furthermore, it is fully compatible with heterogeneous computing within the OpenCL standard. The proposed approach provides internal debugging capabilities that allow us to localize problems directly in a supercomputer simulation.

The paper presents the technology of large-scale CFD simulations on supercomputers using finite-volume or finite-difference methods. The study covers various aspects of the numerical experiment and its implementation. The considered problems include the optimal choice of computational domain, minimization of disk space usage, choice of an optimum configuration for a multilevel parallel model, and a method for loading supercomputer nodes. Special attention is paid to the performance problems of multiple access systems taking into account the waiting time in the system queues, the problem of computing reliability, prevention of emergency shutdowns leading to timeconsuming repeated waiting, automatic correction, and optimization of the calculation parameters, etc. Methods are presented for improving the quality of the averaging of the flow fields and the spectral characteristics.

The quest for new portable implementations of simulation algorithms is motivated by the increasing variety of computing architectures. Moreover, the hybridization of high-performance computing systems imposes additional constraints, since heterogeneous computations are needed to efficiently engage processors and massively-parallel accelerators. This, in turn, involves different parallel paradigms and computing frameworks and requires complex data exchanges between computing units. Typically, simulation codes rely on sophisticated data structures and computing subroutines, so-called kernels, which makes portability terribly cumbersome. Thus, a natural way to achieve portability is to dramatically reduce the complexity of both data structures and computing kernels. In our algebra-based approach, the scale-resolving simulation of incompressible turbulent flows on unstructured meshes relies on three fundamental kernels: the sparse matrix-vector product, the linear combination of vectors and the dot product. It is noteworthy that this approach is not limited to a particular kind of numerical method or a set of governing equations. In our code, an auto-balanced multilevel partitioning distributes workload among computing devices of various architectures. The overlap of computations and multistage communications efficiently hides the data exchanges overhead in large-scale supercomputer simulations. In addition to computing on accelerators, special attention is paid at efficiency on manycore processors in multiprocessor nodes with significant non-uniform memory access factor. Parallel efficiency and performance are studied in detail for different execution modes on various supercomputers using up to 9,600 processor cores and up to 256 graphics processor units. The heterogeneous implementation model described in this work is a general-purpose approach that is well suited for various subroutines in numerical simulation codes. The work of A. G. has been funded by the Russian Sci- ence Foundation, project 19-11-00299. The work of X. Á. F. and F. X. T. has been financially supported by the ANUMESOL project (ENE2017-88697-R) by the Spanish Research Agency, and the FusionCAT project (0 01-P-0 01722) by the Government of Catalo- nia RIS3CAT FEDER. X. Á. F. is supported by a predoctoral contract (2019FI_B2-0 0 076) by the Government of Catalonia. The work has been carried out using the MareNostrum 4 supercomputer of the Barcelona Supercomputing Center; the TSUBAME3.0 supercom-puter of the Global Scientific Information and Computing Center at Tokyo Institute of Technology; the Lomonosov-2 supercomputer of the shared research facilities of HPC computing resources at Lomonosov Moscow State University; the K-60 hybrid cluster of the Collective Usage Centre of KIAM RAS. The authors thankfully acknowledge these institutions.

The present work focuses on the study of noise generation and radiation of an infinite open three-dimensional cavity at low Mach number with laminar upstream conditions that is of interest to understand noise generation mechanisms in wall-bounded separated flows. A particular feature of this configuration is the oscillatory mode: shear layer mode or wake mode. For the parameters considered in the present study it is seen that while in shear layer mode the flow shows a two-dimensional behavior, in the wake mode the flow is three-dimensional, resulting in significantly different sound sources. The influence of the acoustic feedback mechanism in the shear layer mode has also been investigated comparing the results between different momentum thickness values at the cavity inlet. This paper presents results of sound radiated by a three-dimensional infinite open cavity with aspect ratio L/D = 4 at Reynolds number based on the cavity depth of ReD = 1500 and Mach number of M = 0.15, both for shear layer (L/θ = 67) and wake (L/θ = 84) oscillation modes. To do so, Curle integral evaluated as a post-process of an incompressible solution will be used. The results are compared with the resulting Curle post-process of a two-dimensional incompressible simulation

Large-eddy simulation (LES) equations result from applying a spatial commutative filter, with filter length \(\varDelta \), to the Navier–Stokes equations $$\begin{aligned} \partial _t \overline{\varvec{u}} + \left( \overline{\varvec{u}} \cdot \nabla \right) \overline{\varvec{u}} = \nu \nabla ^2\overline{\varvec{u}} - \nabla \overline{p} - \nabla \cdot \tau ( \overline{\varvec{u}} ) , \quad \nabla \cdot \overline{\varvec{u}} = 0, \end{aligned}$$ where \(\overline{\varvec{u}}\) is the filtered velocity and \(\tau (\overline{\varvec{u}})\) is the subgrid stress (SGS) tensor and aims to approximate the effect of the under-resolved scales, i.e. \(\tau (\overline{\varvec{u}} ) \approx \overline{\varvec{u}\otimes \varvec{u}} - \overline{\varvec{u}} \otimes \overline{\varvec{u}}\). Most of the difficulties in LES are associated with the presence of walls where SGS activity tends to vanish. Therefore, apart from many other relevant properties, LES models should properly capture this feature [1].

The paper describes a supercomputer application in simulations of fluid-structure interaction problems. A compressible flow solver based on a high-accuracy scheme for unstructured hybrid meshes is considered. It combines an immersed boundary method with a dynamic mesh adaptation method in order to represent motion of solid objects in a turbulent flow. The use of immersed boundaries allows you to dynamically adapt the mesh resolution near moving solid surfaces without changing the mesh topology. Multilevel MPI + OpenMP parallelization of these components fits well with the architecture of modern cluster systems. The proposed implementation can engage thousands of CPU cores in one simulation efficiently. An example application is presented in which a high-speed turbulent flow around a cavity with a deflector is simulated.

Various fundamental studies based on a turbulent duct flow have gained popularity including heat transfer, magnetohydrodynamics as well as particle-laden transportation. An accurate prediction on the turbulent flow field is critical for these researches. However, the database of the mean flow and turbulence statistics is fairly insufficient due to the enormous cost of numerical simulation at high Reynolds number. This paper aims at providing available information by conducting several Direct Numerical Simulations (DNS) on turbulent duct flows at Re τ = 300 , 600 , 900 and 1200 . A quantitative comparison between current and previous DNS results was performed where a good agreement was achieved at Re τ = 300 . However, further comparisons of the present results with the previous DNS results at Re τ = 600 obtained with much coarser meshes revealed some discrepancies which can be explained by the insufficient mesh resolution. At last, the mean flow and turbulent statistics at higher Re τ was presented and the effect of Re τ on the mean flow and flow dynamics was discussed.

A parallel computation technology for modeling fluid dynamics problems by finite-volume and finite-difference methods of high accuracy is presented. The development of an algorithm, the design of a software implementation, and the creation of parallel programs for computations on large-scale computing systems are considered. The presented parallel technology is based on a multilevel parallel model combining various types of parallelism: with shared and distributed memory and with multiple and single instruction streams to multiple data flows.

The present work focuses on the study of noise generation and radiation of unsteady laminar flow over an open three dimensional cavity at low Mach number, that is of interest to understand noise generation mechanisms in wall-bounded separated flows. The length to depth ratio of the cavity is L/D=4. While this configuration has been extensively studied, most of the works assume the flow to be two-dimensional. However, as it will be shown, previous studies confirm that for Reynolds numbers above ˜ 1200, the flow shows a three dimensional behaviour. This results in significantly different sound sources. This paper presents results of sound calculation radiated by a three-dimensional infinite open cavity at Reynolds number based on the cavity depth of Re = 1500 and Mach number of M= 0.15. To do so, two approaches have been used: Curle integral, evaluated as a post-process of an incompressible solution and compressible direct simulation. The results are also compared with the resulting Curle post-process of a two- dimensional incompressible simulation assumed to be constant along the spanwise direction.

Small-scale dynamics is the spirit of turbulence physics. It delivers complex mechanisms when turbulence is purely sustained by buoyancy. This takes place in Rayleigh-Bénard convection (RBC). Most of the existing RBC dynamics is ruled by hard turbulent regime and a deep understanding of small scale dynamics with its relevant nonlinearities, is still unsatis¿ed. To do so, the evolution of QG =-1/2tr(G2) and RG =-1/3tr(G3), invariants of the velocity gradient tensor, G = Gradu u u, in their phase space{QG,RG}, were studied and reported in Dabbagh et al. [1] using DNS of RBC. In the present work, we expand this 2D {QG,RG} evolution to 3D by decomposing RG into its strain production RS = -1/3tr(S3) andenstrophyproductiontr(¿2S) =RS-RG terms, where S and ¿ aretherate-of-strainandrate-of-rotation tensors, respectively. In the {QG,RS,RS-RG} space, the ¿ow topology in a Lagrangian evolution is changing by the conditionalmeantrajectories(CMTs){DQG/Dt,DRS/Dt,D(RS-RG)/Dt}. Usingthedatasetin[1],andfromFigure46: anidenti¿edcyclicalstartoftrajectoriesisdistinguishedinareasofvortex-stretchingRS-RG >0andRS >0inthestrain dominated slots (QG < 0), which becomes stronger and longer expanded at Rayleigh number Ra = 1010. Afterwards, the trajectories move downwards (QG << 0) in areas of vortex-compression RS-RG < 0 and RS > 0, that also become more diverging at Ra = 1010 as a result of the self-ampli¿ed straining [1]. This is followed by rising trajectories upwards (QG > 0) to continue performing the typical planner {QG,RG} cyclical behaviour [1], next to RS = 0, and decaying towards the origin. DNS at Ra = 1011 is currently being computed on the MareNostrum supercomputer [2]. Results will be presented during the conference.

The design of efficient algorithms for large-scale gas dynamics computations with hybrid (heterogeneous) computing systems whose high performance relies on massively parallel accelerators is addressed. A high-order accurate finite volume algorithm with polynomial reconstruction on unstructured hybrid meshes is used to compute compressible gas flows in domains of complex geometry. The basic operations of the algorithm are implemented in detail for massively parallel accelerators, including AMD and NVIDIA graphics processing units (GPUs). Major optimization approaches and a computation transfer technique are covered. The underlying programming tool is the Open Computing Language (OpenCL) standard, which performs on accelerators of various architectures, both existing and emerging.

Direct numerical simulations of the incompressible Navier-Stokes equations are not feasible yet for most practical turbulent flows. Therefore, dynamically less complex mathematical formulations are necessary for coarse-grained simulations. In this regard, eddy-viscosity models for Large-Eddy Simulation (LES) are probably the most popular example thereof. This type of models requires the calculation of a subgrid characteristic length which is usually associated with the local grid size. For isotropic grids this is equal to the mesh step. However, for anisotropic or unstructured grids, such as the pancake-like meshes that are often used to resolve near-wall turbulence or shear layers, a consensus on defining the subgrid characteristic length has not been reached yet despite the fact that it can strongly affect the performance of LES models. In this context, a new definition of the subgrid characteristic length is presented in this work. This flow-dependent length scale is based on the turbulent, or subgrid stress, tensor and its representations on different grids. The simplicity and mathematical properties suggest that it can be a robust definition that minimizes the effects of mesh anisotropies on simulation results. The performance of the proposed subgrid characteristic length is successfully tested for decaying isotropic turbulence and a turbulent channel flow using artificially refined grids. Finally, a simple extension of the method for unstructured meshes is proposed and tested for a turbulent flow around a square cylinder. Comparisons with existing subgrid characteristic length scales show that the proposed definition is much more robust with respect to mesh anisotropies and has a great potential to be used in complex geometries where highly skewed (unstructured) meshes are present., Comment: 27 pages, 13 figures, 1 table

Since 2011, the European project Mont-Blanc has been focused on enabling ARM-based technology for HPC, developing both hardware platforms and system software. The latest Mont-Blanc prototypes use system-on-chip (SoC) devices that combine a CPU and a GPU sharing a common main memory. Specific developments of parallel computing software and well-suited implementation approaches are of crucial importance for such a heterogeneous architecture in order to efficiently exploit its potential. This paper is devoted to the optimizations carried out in the TermoFluids CFD code to efficiently run it on the Mont-Blanc system. The underlying numerical method is based on an unstructured finite-volume discretization of the Navier–Stokes equations for the numerical simulation of incompressible turbulent flows. It is implemented using a portable and modular operational approach based on a minimal set of linear algebra operations. An architecture-specific heterogeneous multilevel MPI+OpenMP+OpenCL implementation of such kernels is proposed. It includes optimizations of the storage formats, dynamic load balancing between the CPU and GPU devices and hiding of communication overheads by overlapping computations and data transfers. A detailed performance study shows time reductions of up to on the kernels’ execution with the new heterogeneous implementation, its scalability on up to 128 Mont-Blanc nodes and the energy savings (around ) achieved with the Mont-Blanc system versus the high-end hybrid supercomputer MinoTauro. The research leading to these results has received funding from the European Community’s Seventh Framework Programme [FP7/2007–2013] and Horizon 2020 under the Mont-Blanc Project (www.montblanc-project.eu), grant agreement n 288777, 610402 and 671697. The work has been financially supported by the Ministerio de Ciencia e Innovación, Spain (ENE- 2014-60577-R), the Russian Science Foundation, project 15-11-30039, CONICYT Becas Chile Doctorado 2012, the Juan de la Cierva posdoctoral grant (IJCI-2014-21034), and the Initial Training Network SEDITRANS (GA number: 607394), implemented within the 7th Framework Programme of the European Commission under call FP7-PEOPLE- 2013-ITN. Our calculations have been performed on the resources of the Barcelona Supercomputing Center. The authors thankfully acknowledge these institutions.

A parallel finite-volume CFD algorithm for modeling of incompressible flows on hybrid supercomputers is presented. It is based on a symmetry-preserving high-order numerical scheme for structured meshes. A multilevel approach that combines di erent parallel models is used for large-scale simulations on computing systems with massively-parallel accelerators. MPI is used on the first level within the distributed memory model to couple computing nodes of a supercomputer. On the second level OpenMP is used to engage multiple CPU cores of a computing node. The third level exploits the computing potential of massively-parallel accelerators such as GPU (Graphics Processing Units) of AMD and NVIDIA, or Intel Xeon Phi accelerators of the MIC (Many Integrated Core) architecture. The hardware independent OpenCL standard is used to compute on accelerators of di erent architectures within a general model for a combination of a central processor and a math co-processor.

A parallel finite-volume algorithm based on a cell-centered high-order polynomial scheme for unstructured hybrid meshes is under consideration. The work is focused on the adaptation and optimization of basic operations of the algorithm to different architec- tures of massively-parallel accelerators including GPU of AMD and NVIDIA. Such an algorithm is especially problematic for the GPU architectures since it has very low FLOP per byte ratio meaning that performance is dominated by the memory bandwidth but not the computing performance of a device. At the same time it has irregular memory access pattern since unstructured meshes are used. The calculation of polynomial coefficients and the calculation of convective fluxes through faces of cells are the most interesting and time consuming operations of the algorithm. Implementations of these operations for accelerators using OpenCL are considered here in detail. The ways to improve the computational efficiency are proposed, performance measurement results reaching up to 160 GFLOPS on a single GPU device are demonstrated.

This paper describes a hybrid MPI-CUDA parallelization strategy for the direct numerical simulation of incompressible flows using unstructured meshes. Our in-house MPI-based unstructured CFD code has been extended in order to increase its performance by means of GPU co-processors. Therefore, the main goal of this work is to take advantage of the current hybrid supercomputers to increase our computing capabilities. CUDA is used to perform the calculations on the GPU devices and MPI to handle the communications between them. The main drawback for the performance is the slowdown produced by the MPI communication episodes. Consequently, overlapping strategies, to hide MPI communication costs under GPU computations, are studied in detail with the aim to achieve scalability when executing the code on multiple nodes.

The main objective of this paper is to study in detail the fluid flow and the heat transfer in plane impinging jets. Mean and fluctuating velocities and global parameters, i.e. the local Nusselt number, are analysed. The study is focused on a Reynolds number 20,000 (based on the bulk inlet velocity and the nozzle width, B) and dimensionless jet-to-surface spacing 4. As a first step, a reliable direct numerical simulation (DNS) has been performed. Then, the DNS results have been used as reference solution to assess the performance of several Reynolds-averaged Navier-Stokes (RANS) models. Namely, explicit algebraic Reynolds stress models and both non-linear and linear eddy viscosity models in conjunction with k- epsilon and k - omega platforms. Moreover, an overview of the numerical methods and the methodology used to verify the code and the simulations is also presented. Time-averaged DNS results have revealed that the main recirculating flow cannot be captured well unless the outflow is placed at least at 4013 from the jet centreline approximately. This suggests that previous experimental data may not be adequate to study the flow configuration far from the jet. Consequently, conclusions previously published by the authors on the performance of the tested RANS models have been necessarily revised. (C) 2011 Elsevier Ltd. All rights reserved.

Direct numerical simulations of the incompressible Navier-Stokes equations are limited to relatively low-Reynolds numbers. Therefore, dynamically less complex mathematical formulations are necessary for coarse-grain simulations. Regularization and eddy-viscosity models for LES are examples thereof. They rely on differential operators that should capture well different flow configurations (laminar and 2D flows, near-wall behavior, transitional regime ...). Most of them are based on the combination of invariants of a symmetric second-order tensor that is derived from the gradient of the resolved velocity field. In the present work, they are presented in a framework where the models are represented as a combination of elements of a 5D phase space of invariants. In this way, new models can be constructed by imposing appropriate restrictions in this space.

where u denotes the velocity field, p represents the pressure, the non-linear convective term is given by C uv = (u · ∇) v, and the diffusive term reads Du = νΔu, where ν is the kinematic viscosity. Direct simulations at high Reynolds numbers are not feasible because the convective term produces far too many scales of motion. Hence, in the foreseeable future numerical simulations of turbulent flows will have to resort to models of the small scales. The most popular example thereof is the Large-Eddy Simulation (LES). Shortly, LES equations result from filtering the NS equations in space ∂t u + C (u,u) = Du −∇ p − ∇ · τ(u) ; ∇ · u = 0, (2)

The effect of collisions on the particle behavior in a fully developed turbulent flow in a straight square duct at Re-r = 300 is numerically investigated. The hydrodynamic modeling of the fluid phase is based on direct numerical simulation. The kinematics and trajectory of the particles as well as the collisions are described by the discrete element method. Three sizes of particles are considered with diameters equal to 50 mu m, 100 mu m and 500 mu m. Firstly, the particle transportation by turbulent flow is studied in the absence of the gravitational effect. It is found that the collisions play an important role in the particle distribution especially in the near-wall regions. The inter-particle collisions enhance the particle diffusion in the direction perpendicular to streamwise flow and make the particles distribute more uniformly near the wall. Then, the particle deposition is studied under the effect of the wall-normal gravity force in which the influence of collisions on the particle resuspension rate and the final stage of particle distribution on the duct floor are discussed, respectively. The collisions are found to have influence on the particle resuspension rate near the duct floor whereas hardly affect the particle behavior near the duct center. Under the gravitational effect the 50 mu m particles deposit more efficiently near the side walls but the 100 mu m and 500 mu m particles preferentially deposit near the center of the duct floor. Moreover, all the sizes of particles tend to concentrate near the center of the duct floor at the final stage of the particle deposition when the inter-particle collisions are considered. This work has been financially supported by the Ministerio de Ciencia e Innovación, Spain (ENE2010-17801). Hao Zhang would like to acknowledge the FI-AGAUR doctorate scholarship granted by the Secretaria d'Universitats i Recerca (SUR) del Departament d'Economia i Coneixement (ECO) de la Generalitat de Catalunya, and by the European Social Fund. F. Xavier Trias would like to thank the financial support by the Ramón y Cajal postdoctoral contracts (RYC-2012- 11996) by the Ministerio de Ciencia e Innovación. Yuanqiang Tan would like to thank the Aid Program for Science and Technology Innovative Research Team in Higher Educational Institutions of Hunan Province [2012] 318. The calculations have been performed on the IBM MareNostrum supercomputer at the Barcelona Supercomputing Center. The authors thankfully acknowledge these institutions. We also thank the anonymous reviewers for their comments and remarks which helped to improve the quality of this work.

This work is devoted to the study of turbulent natural convection flows in differentially heated cavities. The adopted configuration corresponds to an airfilled (Pr = 0.7) cavity of aspect ratio 5 and Rayleigh number Ra = 4.5 × 1010 (based on the cavity height). Firstly, a complete direct numerical simulation (DNS) has been performed. Then, the DNS results have been used as reference solution to assess the performance of symmetry-preserving regularization as a simulation shortcut: a novel class of regularization that restrain the convective production of small scales of motion in an unconditionally stable manner. In this way, the new set of equations is dynamically less complex than the original Navier-Stokes equations, and therefore more amenable to be numerically solved. Direct comparison with the DNS results shows fairly good agreement even for very coarse grids.

The incompressible Navier-Stokes equations constitute an excellent mathematical modelization of turbulence. Unfortunately, attempts at performing direct simulations are limited to relatively low-Reynolds numbers because of the almost numberless small scales produced by the non-linear convective term. Alternatively, a dynamically less complex formulation is proposed here. Namely, regularizations of the Navier-Stokes equations that preserve the symmetry and conservation properties exactly. To do so, both convective and diffusive terms are altered in the same vein. In this way, the convective production of small scales is effectively restrained whereas the modified diffusive term introduces a hyperviscosity effect and consequently enhances the destruction of small scales. In practice, the only additional ingredient is a self-adjoint linear filter whose local filter length is determined from the requirement that vortex-stretching must stop at the smallest grid scale. In the present work, the performance of the above-mentioned recent improvements is assessed through application to turbulent natural convection flows by means of comparison with DNS reference data.

The incompressible Navier-Stokes equations constitute an excellent mathematical modelization of turbulence. Unfortunately, attempts at performing direct simulations are limited to relatively low-Rayleigh numbers because of the almost numberless small scales produced by the non-linear convective term. Alternatively, a dynamically less complex formulation is proposed here. Namely, regularizations of the Navier-Stokes equations that preserve the symmetry and conservation properties exactly. To do so, both convective and diffusive term are altered in the same vein. In this way, the convective production of small scales is effectively restrained whereas the modified diffusive term introduces a hyperviscosity effect and consequently enhances the destruction of small scales. In practice, the only additional ingredient is a selfadjoint linear filter whose local filter length is determined from the requirement that vortex-stretching must stop at the smallest grid scale. The performance of the novel regularization modeling approach is assessed through application to turbulent natural convection flows at very high Rayleigh numbers

Direct numerical simulation (DNS) of incompressible flows is an essential tool for improving the understanding of the physics of turbulence and for the development of better turbulence models. The Poisson equation, the main bottleneck from a parallel point of view, usually also limits its applicability for complex geometries. In this context, efficient and scalable Poisson solvers on fully-3D geometries are of high interest.In our previous work, a scalable algorithm for Poisson equation was proposed. It performed well on both small clusters with poor network performance and supercomputers using efficiently up to a thousand of CPUs. This algorithm named Krylov-Schur-Fourier Decomposition (KSFD) can be used for problems in parallelepipedic 3D domains with structured meshes and obstacles can be placed inside the flow. However, since a FFT decomposition is applied in one direction, mesh is restricted to be uniform and obstacles to be 2D shapes extruded along this direction.The present work is devoted to extend the previous KSFD algorithm to eliminate these limitations. The extension is based on a two-level Multigrid (MG) method that uses KSFD as a solver for second level. The algorithm is applied for a DNS of a turbulent flow in a channel with wall-mounted cube. Illustrative results at Re τ = 590 (based on the cube height and the bulk velocity Re h = 7235) are shown.

The present work is reporting the experience on different aspects of using supercomputers for CFD applications. Several problems that appear with new supercomputers built of multi-core nodes are considered. In particular, it is a problem of increased influence of irregular memory access, problems with efficient use of such a big numbers of processors that are available in present time. A hybrid parallelization using MPI and OpenMP technologies is also considered to improve efficiency of computations. Results of performance tests where the hybrid approach outperformed the MPI-only parallelization are shown. Finally, a performance comparison of Marenostrum and MVS-50000 supercomputers is presented, illustrative numerical results on the DNS of turbulent flows are also included.

We consider regularizations of the convective term that preserve symmetry and conservation properties exactly. This yields a novel class of regularizations that restrain the convective production of small scales in an unconditionally stable manner Numerically, one of the most critical issues is the discrete filtering; properties required are, in general, not preserved by classical LES filters. Alternatively, here we propose to construct filters with the general form F = I + Σm = 1 M dm Dm where D is the discrete diffusive operator. Then, the coefficients, dm , follow from the requirement that, at the smallest grid scale kc , the damping effect to the wavevector-triple (kc , p, kc − p) interactions must be virtually independent of the p-th Fourier-mode. This allows an optimal control of the subtle balance between convection and diffusion to stop the vortex-stretching. Finally, the proposed method is tested for an air-filled differentially heated cavity of aspect ratio 4 by direct comparison with DNS reference results.Copyright © 2010 by ASME

Since direct numerical simulations of natural convection in a differentially heated cavity cannot be performed at high Rayleigh numbers, a dynamically less complex mathematical formulation is sought. In the quest for such a formulation, we consider regularizations (smooth approximations) of the nonlinearity. The regularization method basically alters the convective terms to reduce the production of small scales of motion by means of vortex stretching. In doing so, we propose to preserve the symmetry and conservation properties of the convective terms exactly. This requirement yielded a novel class of regularizations that restrain the convective production of smaller and smaller scales of motion by means of vortex stretching in an unconditional stable manner, meaning that the velocity cannot blow up in the energy-norm (in 2D also: enstrophy-norm). The numerical algorithm used to solve the governing equations preserves the symmetry and conservation properties too. In the present work we propose to determine the filter length dynamically with the requirement that the vortex stretching must be stopped at the scale set by the grid. Finally, the proposed parameter-free regularization model is successfully tested for a turbulent natural convection flow in an air-filled differentially heated cavity of aspect ratio 4 with Rayleigh number up to 10 11.