An Object-Based Evaluation of Output from a Deep Learning Model for Thunderstorm Nowcasting

To evaluate characteristics of output values from machine learning models in bulk, an object-based technique is developed and applied to a deep learning model called ThunderCast (the Thunderstorm Nowcasting Tool). ThunderCast predicts the occurrence of midlatitude lightning-producing convection (thunderstorms) in the next 0–60 minutes from satellite observations. This paper uses the Tracking and Object-Based Analysis of Clouds (tobac) tool to identify tracks of objects comprised of ThunderCast predictions from 7 randomly selected days per month from April to September 2022. For each track, the maximum Multi-Radar Multi-Sensor (MRMS) radar reflectivity at −10°C (ThunderCast’s target) is used to categorize the track as true (≥ 30 dB z at −10°C) or false (< 30 dB z at −10°C) positive. The tracks are further classified by the presence or absence of lightning from the Geostationary Lightning Mapper (GLM-16) or Earth Networks Total Lightning Network (ENTLN). Of the 17,054 tracks identified, 69.4% were true positive, but 62.5% of those true positives lacked associated GLM-16 or ENTLN lightning. A clear radar reflectivity threshold separating lightning-associated tracks from non-lightning tracks was not found. This demonstrates a limitation of using ground-based radar thresholds, indicative of microphysical characteristics associated with thunderstorm electrification, as the target dataset in thunderstorm nowcasting models. Additionally, cloud-top properties from the 10.3 µ m and 1.6 µ m Advanced Baseline Imager (ABI) spectral bands are consistent with cloud-top glaciation. This suggests a need for additional predictors to reduce both false positives and true positives without associated lightning in ThunderCast’s predictions. The object-based analysis technique presented here can be adapted for evaluating output from other machine learning models.