Porting a large eddy simulation computational fluid dynamics solver to graphics processing units

In this study, multiple approaches for accelerating a large eddy simulation (LES) solver are analyzed. The code routines that comprise the largest fraction of the execution time are offloaded to graphics processing units (GPUs). Large speedups are experienced for ported sections of code when data is able to be kept on the device between timesteps. This study finds that care should be taken in choosing the right programming model as well as the manner in which data is transferred back and forth from the host (CPU) to the device (GPU). NVIDIA GPUs were targeted through the use of the CUDA (Compute Unified Device Architecture) API (Application Programming Interface) and the CUDA backend programming model was used for code written in the heterogeneous programming model Kokkos. Using Kokkos allowed for a much greater ease of use for the developer when using 3D arrays in PETSc (the Portable Extensible Tool for Scientific Computation). Kokkos also allowed for data to remain on the GPU between time steps while using PETSc which yielded greater performance in terms of execution time. Though the entirety of the LES solver’s code was not ported, the benefits of using GPUs for highly parallel routines are discussed and an extensive outline to continue porting the code is provided. At least a 5 times speedup was experienced when using an older generation GPU and 12 cores on a CPU when using the same cell count which saturates the VRAM on the GPU and dividing the execution time of a routine running on the CPU by the same routine ported with Kokkos.