Model Reliability Under Temporal Distribution Shift: An Empirical Study Using the ELEC2 Dataset

This study investigates how reliable machine learning models stay when real-world data changes over time, using the ELEC2 electricity pricing dataset. To track these shifts, we compared two drift detection methods: the Kolmogorov-Smirnov (KS) statistic and Population Stability Index (PSI), matching what they detect with how much the model's accuracy drops as time goes on. Here’s what stood out: the KS statistic tracks well with when accuracy falls off, while PSI sometimes catches changes earlier than KS, but it also picks up on some noise both KS and PSI gave false alarms during a period when the model was actually recovering. We double-checked our setup with a covariate shift test based on the approach described in "A Survey on Dataset Shift in Machine Learning" to make sure our methods held up.