Towards optimal softening in three-dimensionalN-body codes - I. Minimizing the force error

In N-body simulations of collisionless stellar systems, the forces are softened to reduce the shot noise. Softening modifies gravity at r=|x-y| smaller than softening length epsilon and the softened forces are increasingly biased for ever larger epsilon. There is, thus, some optimum between reducing the fluctuations and introducing a bias. Here, analytical relations are derived for the amplitudes of the bias and the fluctuations in the limit of small epsilon and large N. It is shown that the fluctuations of the force are generated locally, in contrast to the variations of the potential, which originate from noise in the whole system. Based on the asymptotic relations and using numerical experiments, I study the dependence of the resulting force error on N, epsilon, and on the functional form by which Newtonian gravity is replaced. The Plummer softening, where each body is replaced by a Plummer sphere of scale radius epsilon, yields significantly larger force errors than do methods in which the bodies are replaced by density kernels of finite extent. I also give special kernels, which reduce the errors even further. These kernels largely compensate the errors made with too small inter-particle forces at r<epsilon by exceeding Newtonian forces at r epsilon. Additionally, the possibilities of locally adapting epsilon and of using unequal weights for the bodies are investigated. These various techniques allow, without increasing N, to reduce the rms force error by a factor 2 compared to Plummer softening with constant epsilon. The results of this study are directly relevant to N-body simulations using direct summation techniques or the tree code. (abridged)