Atmospheric cloud detection and cloud base height estimation from the Lidar backscatter: a two-step statistical and machine learning approach

Cloud base height (CBH) is a fundamental atmospheric parameter for weather forecasting, aviation safety, and climate research, as it provides key information on boundary layer structure, atmospheric stability, and cloud–radiation interactions. In this study, a two-step hybrid framework combining statistical cloud detection with machine learning (ML) regression for CBH estimation is developed and evaluated. In the first step, cloud presence is identified using a Variability Index (VI) defined as the ratio of the standard deviation of the backscatter profile to its peak value. This physically interpretable index shows strong class separability, with a large effect size (Cohen’s d ≈ 1.96), a maximum F1 score of about 0.83 at a VI of approximately 0.24, and an area under the ROC curve of 0.88, indicating effective cloud detection performance. In the second step, Multiple Linear Regression, Fine Tree Regression, Random Forest, and Gaussian Process Regression (GPR) models are applied to cloud-present profiles to estimate CBH. Among these, Random Forest, a tree based nonlinear ensemble model perform best, achieving a high correlation coefficients of about 0.94 ± 0.04. GPR, a kernel-based model, demonstrated slightly lower performance compared to Random Forest, achieving correlation coefficients of R = 0.91 ± 0.05, while the Fine Tree model showed the weakest performance among the nonlinear models tested in this study, achieving R = 0.89 ± 0.07. In contrast, Multiple Linear Regression model showed lowest accuracy with R = 0.58 ± 0.13. The results demonstrate that combining a simple, explainable statistical classification approach with advanced machine learning regression significantly improves the reliability and accuracy of CBH retrieval from Lidar backscatter data. The proposed framework is computationally efficient and shows strong potential for operational implementation in real-time atmospheric monitoring networks.