A Novel Approach in Improving I/O Performance of Small Meteorological Files on HDFS

With the cloud computing is becoming mature, many of its characteristics for meteorological data processing is extremely important. Since HDFS is designed for reading and writing large files, it’s difficult to be taken advantage of small meteorological files. In this paper, an improved approach on HDFS is proposed for small meteorological files, small files are to be merged, indexed, and blocks are compressed, the pressure of memory on master node occupied by metadata is relieved, the speed of reading and writing small files is increased, read speed is increased by 50%, and write speed is up to 3-4 times of the original, saving about 2/3 of storage space and computing performance has also been improved. Thus, meteorological data processing can make use of cloud computing platform more closely.