Generation of long-term ground fog time series using harmonized time series cross-calibrating two Meteosat generations

Fog, a meteorological phenomenon resulting in horizontal visibility of less than 1000 meters, has significant socio-economic and environmental consequences. Current long-term research on the fog occurrence based on station data have indicated that the frequency of fog has decreased over Europe since the 1960s. However, due to a limited number of ground-based observations, primarily in low-altitude areas, there is insufficient evidence to support the hypothesis that fog is decreasing across Europe. In order to scientifically investigate different factors which might be responsible in influencing fog formation over the years over space and time, there is a need of long term consistent satellite data time series to analyze the fog distribution. In this study, first a machine learning based methodology has been developed and implemented to harmonize the two generation Meteosat datasets, i.e. Meteosat First Generation (MFG) and Meteosat Second Generation (MSG) to generate a long-term consisent dataset (1991-2020) which can be further used to classify fog over the European domain (WMO region VI). For this, a Random Forest (RF) based model is trained during the overlap period (2004-2006) of MFG and MSG datasets, to synthesize MFG data from MSG data to generate a consistent MFG time series. The results of this model indicates a good match of synthesized MFG datasets with the original MFG datasets during the overlap period with mean absolute error (MAE) of 0.7 K for the WV model and 1.6 K for the IR model and out-of-bag (OOB) R2 score of 0.98 for both models. In the next stage, this harmonized dataset is currently being investigated along with the CM-SAF MSG based cloud mask dataset to generate a homogeneous cloud mask over the domain using a machine learning based eXtreme Gradient Boosting (XGBoost) classifier model. The current version of the cloud mask is able to predict high mid level clouds for both day and night time with high accuracy. In case of fog and low stratus (FLS) clouds, the model exhibits excellent performance during day time but encounters some difficulty in detecting in certain FLS patches during night time. This resultant cloud mask can subsequently be employed to classify fog occurrences by integrating harmonized MFG WV and IR channels with cloud base altitude (CBA) as well as visibility data obtained from Meteorological Aviation Routine Weather Reports (METAR) and synoptic weather observations (SYNOP) within a machine learning-based model. In this context, we present the current ongoing progress and the preliminary results in generating a 30 years fog climatology (1991-2020) for Europe with a temporal resolution of 30 minutes using this dataset.