Automatic Differentiation of OpenMP with OpDiLib

Automatic differentiation (AD) is an established tool in scientific computing for the efficient retrieval of accurate derivatives. The reverse mode of AD, also known as backpropagation in machine learning, enables, for example, sensitivity analysis, parameter identification, and discrete adjoint procedures for simulation codes from all application domains. To handle large-scale problems, these codes are typically run on high performance computing clusters and written according to parallel paradigms such as MPI or OpenMP, which require dedicated handling in terms of AD. We present OpDiLib, an add-on for operator overloading AD tools that enables the reverse mode differentiation of OpenMP parallel codes. It relies on OpenMP's tools interface (OMPT) to detect and augment OpenMP constructs in a fully automatic fashion. OpDiLib deduces a corresponding reverse parallelism and embeds it into the tape of the underlying operator overloading AD tool. For the sake of generality and applicability, OpDiLib addresses the inherent data races of shared-memory parallel reverse mode AD by defaulting to atomic updates on adjoint variables. To improve the performance, this can be lifted according to user knowledge in a fine-grained manner. Our benchmarks showcase the achievable speedup and scalability and demonstrate the impact of optimizations such as the elimination of atomic updates on adjoint variables, both in OpenMP parallel and in MPI-OpenMP hybrid parallel applications.