Knowledge Distillation for Improving Deep Learning‐Based Hydrological Prediction and Fine‐Tuning

Abstract This paper introduces knowledge distillation (KD) as a method for improving deep learning (DL)‐based hydrological prediction. The performance of DL models, including long short‐term memory networks (LSTMs), is highly dependent on model structure and training data quality, posing challenges for transfer learning. In the KD framework, a better “teacher” model, trained with higher‐quality input data or constructed as an ensemble, is used to guide the training of a weaker “student” model. This enables improved performance in situations where input data quality or computational resources are limited. We demonstrate the effectiveness of KD in two experiments conducted across 421 catchments throughout the contiguous United States, where LSTM models use daily basin‐averaged meteorological forcings to predict daily streamflow. In the first experiment, KD was applied to improve streamflow prediction when lower‐quality ERA5‐Land reanalysis meteorological inputs were used instead of higher‐quality Daymet observations. This experiment is relevant to forecasting applications, where lower quality (e.g., lower resolution) precipitation forecasts are used to force a DL‐based hydrologic model optimized with high‐resolution observed precipitation. Results demonstrate that KD leads to substantial performance improvements, especially for catchments not used in training (>25% improvement in median Nash‐Sutcliffe efficiency (NSE)). In the second experiment, KD is applied for model compression, distilling the knowledge of an ensemble of five LSTM models into a single model that substantially outperforms LSTMs trained without KD. These findings underscore the value of KD in optimizing DL models for hydrological prediction, offering a model‐agnostic and scalable approach that facilitates efficiency and model transferability.