Ensembling is all you need? Evaluating machine learning models on predicting the energy imbalance of prosumers

Energy imbalance is a situation where there is a mismatch between the expected energy consumption and energy production. Such energy imbalances cause widespread problems in the energy grid, sometimes causing instability of the grid leading to major grid failures. Prosumers, who both consume and produce energy, are one of the main contributors to the issue of energy imbalance. This paper presents a thorough investigation of different machine-learning algorithms for predicting the energy consumption and production behavior of prosumers in Estonia. Linear regression, MLP, TabNet, XGboost, LightGBM, and VotingRegressor-based ensemble of them are compared against each other. While individually, the LightGBM model performed best on this dataset with an MAE of 74.56, overall the VotingRegressor ensemble was the best-performing model with an MAE of 70.16. An investigation into the relative importance of predictive features is presented by exploring the feature importance of the trained LightGBM model. We also propose a target normalization pipeline to handle out-of-bound (OOB) inference issues, which resulted in a significant improvement in the performance of treebased models by an average of 7% on the test set.