Enhancing Stability of Probabilistic Model-Based Reinforcement Learning by Adaptive Noise Filtering

This article proposes stabilized model-based policy optimization (SMBPO) to address the stability and efficiency issues in current probabilistic model-based reinforcement learning (MBRL) approaches. It adaptively filters the noises caused by imperfect models in both model and policy updates: 1) dimensions with abnormal distributions in the prediction are refined to stabilize the training of probabilistic models and 2) predicted states and estimated value functions are clipped to mitigate the negative impact of model bias on the learned policy. Leveraging the stability above, SMBPO employs batch normalization (BN) to significantly boost its learning efficiency. Evaluated on five MuJoCo control benchmarks and one dexterous hand scenario, SMBPO reduces training time by 90% while achieving 50% more cumulative rewards than state-of-the-art model-free and MBRL baselines, thereby extending the feasibility of MBRL in practice. The open source code of SMBPO is available at https://github.com/mrjun123/SMBPO.