A Virtualization-based Hybrid Storage System for a Map-Reduce Framework

A map-reduce framework is popular for big data analysis.In the typical map-reduce framework, both master node and worker nodescan use hard-disk drives (HDDs) as local disks for the map-reduce computation. However, because of the inherit mechanical problems of HDDs,the I/O performance is a bottleneck for the map-reduce framework whenI/O-intensive applications (e.g., sorting) are performed. Replacing HDDswith solid-state drives (SSDs) is not economical, although SSDs have betterperformance than HDDs. In this paper, we propose a virtualization-basedhybrid storage system for the map-reduce framework. The objective of thepaper is to combine the advantages of the fast access property of SSDs andthe low cost of HDDs by realizing an economical design and improvingI/O performance of a map-reduce framework in a virtualization environment. We propose three storage combinations: SSD-based, HDD-based,and a hybrid of SSD-based and HDD-based storage systems which balances speed, capacity, and lifetime. According to experiments, the hybridof SSD-based and HDD-based storage systems offers superior performanceand economy.