Statistical, Physical, and Computational Aspects of Massive Data Analysis and Assimilation in Atmospheric Applications

The goals and procedures of the most data-intensive operations in atmospheric sciences, including data assimilation and fusion, are introduced. We explore specific problems that result from the expansion in observing systems from conventional to satellite borne and the corresponding transition from small, medium, and large datasets to massive datasets. The satellite data, their volumes, heterogeneity, and structure are described in specific examples. We illustrate that the atmospheric data analysis and assimilation procedures and the satellite data pose unique problems that do not exist in other applications and are not easily addressed by existing methods and tools. Existing solutions are presented and their performance with massive datasets is critically evaluated. We conclude that since the problems are interdisciplinary, a comprehensive solution must be interdisciplinary as well. We note that components of such a solution already exist in statistics, atmospheric, and computational sciences, but that in isolation they often fail to scale up to the massive data challenge. The prospects of synthesizing an interdisciplinary solution which will scale up to the massive data challenge are thus promising.