Performance of Five Machine Learning-based Global Weather Prediction Models in the East Asia Region

Abstract Recently, the development of artificial intelligence (AI) has led to the fruition of machine-learning (ML) based weather prediction (MLWP) systems. Five global models have emerged as notable contenders: Pangu-Weather, FourCastNet v2 (FCN2), GraphCast, FuXi, and FengWu. Despite utilizing different AI/ML methodologies, all five systems rely on the ECMWF Reanalysis v5 (ERA5), as their training dataset. This study evaluates the performance of these five models in the Eastern Asia region from June to November 2023 with our own simulations using identical initial conditions from ERA5. The evaluation comprises three key metrics: 1) Synoptic-scale prediction assessment with Root Mean Square Error and Anomaly Correlation Coefficients, 2) Typhoon predictions as extreme weather cases, and 3) Local rainfalls induced by a typhoon. Results indicate that FengWu emerges as the best-performing model, followed by FuXi and GraphCast, with FCN2 and Pangu-Weather ranking lower. Notably, performance of the ECMWF IFS lies within the MLWP model range. A multi-model ensemble, formed by averaging predictions from all five models, demonstrates superior performance, rivaling that of FengWu. For the 11 typhoons in 2023, FengWu demonstrates the most accurate track prediction. While Pangu-Weather displays the largest overall track error, it still excels in predicting the tracks of 3 typhoons. Meanwhile, FengWu has the largest intensity errors. In addition, both GraphCast and FuXi demonstrate some encouraging precipitation predictions associated with typhoons. Suggestions for future improvements in MLWP models are discussed.