Simultaneous Preaccumulations in OpenMP-Parallel Automatic Differentiation

Automatic differentiation (AD) provides accurate derivatives of computer programs in an efficient manner. With extensions to handle programs that are parallelized with MPI or OpenMP, it can also be applied at large scale. The reverse mode of AD is particularly suitable for applications with low-dimensional output and high-dimensional input, as is typical in optimization contexts. However, it requires reversal of the entire computational graph, which can consume substantial amounts of memory. Several techniques exist to make applications of the reverse mode of AD feasible, one of which is preaccumulation, where Jacobians of parts of the computational graph are already assembled as the graph is recorded. If multiple threads share global derivative variables, simultaneous thread-local preaccumulations with shared input variables introduce data races. We address these data races by using thread-local derivative variables instead. We evaluate different vector- and map-based implementations with respect to memory consumption, memory allocation, and memory access times. We implement these alternatives in the AD tool CoDiPack and benchmark them in the multiphysics simulation suite SU2, which is differentiated with CoDiPack together with the add-ons OpDiLib for AD of OpenMP and MeDiPack for AD of MPI.