Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation

In recent years, the development of artificial intelligence has led to rapid advances in data-driven weather forecasting models, some of which rival or even surpass traditional methods like Integrated Forecasting System (IFS) in terms of forecasting accuracy. However, existing data-driven weather forecasting models still rely on the analysis fields generated by the traditional assimilation and forecasting system, which hampers the significance of data-driven weather forecasting models regarding both computational cost and forecasting accuracy.Four-dimensional variational assimilation (4DVar) is one of the most popular data assimilation algorithms and has been adopted in numerical weather prediction centers worldwide. This research aims at extending the ability of data-driven weather forecasting models by coupling them with the 4DVar algorithm, i.e., alternatively running the AI forecast and 4DVar to realize a long-term self-contained forecasting system.In the 4DVar algorithm, the forecasting model is embedded into the objective function so that the flow dependencies are taken into account. Realizing the 4DVar algorithm in which the flow dependencies are expressed by the AI weather forecasting model is still a new area to be explored. Previous research has demonstrated the feasibility of using AI forecasting models in 4DVar as flow dependencies with the aid of auto-differentiation in simple dynamic systems, but scaling to the more complicated global weather forecasting faces additional challenges. For example, a differentiable background error covariance matrix needs to be constructed so that auto-differentiation can be implemented. Furthermore, the rapid error accumulation of AI forecasting models reduces the accuracy of flow dependencies in 4DVar and hinders the assimilation accuracy.In this research, we address these challenges by leveraging the following techniques. First, we take advantage of the torch-harmonics package developed by Nvidia to implement the differentiable spherical convolution for representing horizontal correlations in the background error matrix. Second, we reformulate the 4DVar objective function to take into account the cumulative error of AI weather forecasting model so that the objective function can better represent the error statistics. Third, the temporal aggregation strategy with different time-length AI forecasting models is employed to efficiently build flow dependencies so as to reduce the iterative error of AI forecasting model and improve assimilation accuracy.We conduct this research on the global AI weather forecasting model, FengWu, and couple it with 4DVar to implement the AI weather forecasting system prototype, FengWu-4DVar. Our experiments were conducted with the FengWu forecasting model at 1.4 resolution and the ERA5 simulation observations. With an observation proportion of 15% and the assimilation window of 6 hours, FengWu-4DVar is capable of generating reasonable analysis fields and achieving stable and efficiently cyclic assimilation and forecasting for at least one year, and the root mean square error on the potential height of the analysis field at 500hPa is less than 25m2/s2 on average. Moreover, assimilating observations in a 6-hour window can be realized in less than 30 seconds on one GPU of NVIDIA A100.