Efficient and Unbiased Safety Test for Autonomous Driving Systems

Test the safety of Autonomous Driving Systems (ADS) with realistic traffic conditions is important to the insurance industry, legislators, and third-party technical services. However, the scarcity of risky driving events distributed in real-world driving often makes sampling inefficient. In this paper, we propose a unified and hierarchical testing framework for efficient and unbiased safety tests of ADS. We first extract the risk subspace from the Naturalistic Driving Data (NDD) with defined safety measures. Subsequently, we use the Gaussian Copula method to accurately model the subspace joint probability density function (PDF). We further formulate a Kriging model-based optimization problem for finding the appropriate Importance Sampling (IS) parameters, where the Kriging model is trained iteratively within limited computational resources. Eventually, the risky concrete scenarios library is generated using the obtained Importance Sampling parameters. With the simulation results skewered back to the original risk subspace, the ADS probability of failure (e.g., crash rate) can thus be accurately and unbiasedly estimated. Experiment results show that using 0.94% computational resources, the Kriging surrogate model captures 96.04% of ADS crashes with at least 90.25% precision. On crash rate estimation, our proposed method achieves a consistent result with that of Monte Carlo simulation, provided that the relative error does not exceed 10% while improving the testing efficiency by up to 6,539 times.