Towards a transparent and reproducible evaluation framework of commercial solar forecasting solutions

With the current and expected development of the installed solar capacity, solar forecasting is becoming a crucial part of the electricity supply system to allow an optimal integration of the weather dependent power production. A growing number of algorithms and forecast providers has appeared in the academic and industrial scenes to address this need.The multitude of forecast solutions is beneficial for the industry because it ultimately results in an increase the forecast accuracy. But at the same time, it makes it difficult for forecast users to have a clear view on the strengths and weaknesses of the different solutions. This lack of clarity is reinforced by the use of different evaluation methodologies and proprietary reference solar measurements.This work aims at contributing to the establishment of a standard for solar forecast evaluation based on backtest simulations (also known as hindcasts). The cornerstone of our proposed methodology is to base the forecast evaluation on open-data and FAIR in-situ measurements so that the evaluation is reproducible. Furthermore, we propose to share our evaluation methodology so that any deviation resulting from the data preparation, quality control, filtering or selected baseline are avoided. Finally, we propose in addition to standard error metrics (MAE, RMSE, bias) a set of evaluation procedures aimed to inform end-users on the main characteristics of a forecast:Performance of the forecast in specific weather conditions (cloud free, overcast, variable conditions, regime change) following the approach suggested in Verbois et al. (2020) Comparison of the variability of the forecasted solar irradiance and the reference solar irradiance measurements Performance of ramp detections (sharpness of ramps, time lag) based on the work of Vallance et al. (2018) The numerical, visual or behavioral performance of a particular model against these procedures will be systematically compared with results obtained similarly on typical baseline forecast models (such as persistence, NWP only or CMV forecasts).The current work is limited to deterministic forecast but an extension to probabilistic forecast is planned in the very near future.Verbois, H., Blanc, P., Huva, R., Saint-Drenan, Y.-M., Rusydi, A., Thiery, A., Beyond quadratic error: Case-study of a multiple criteria approach to the performance assessment of numerical forecasts of solar irradiance in the tropics, Renewable and Sustainable Energy Reviews, Volume 117, 2020, 109471, ISSN 1364-0321, https://doi.org/10.1016/j.rser.2019.109471.Vallance, L., Charbonnier, B., Paul, N., Dubost, S., Blanc, P. Towards a standardized procedure to assess solar forecast accuracy: A new ramp and time alignment metric. Solar Energy, 2017, 150, pp.408 - 422. 10.1016/j.solener.2017.04.064. hal-01522453