Quantifying the performance isolation properties of virtualization systems

In this paper, we present the design of a performance isolation benchmark that quantifies the degree to which a virtualization system limits the impact of a misbehaving virtual machine on other well-behaving virtual machines running on the same physical machine. Our test suite includes six different stress tests - a CPU intensive test, a memory intensive test, a disk intensive test, two network intensive tests (send and receive) and a fork bomb. We describe the design of our benchmark suite and present results of testing three flavors of virtualization systems -- an example of full virtualization (VMware Workstation), an example of paravirtualization (Xen) and two examples of operating system level virtualization (Solaris Containers and OpenVZ). We find that the full virtualization system offers complete isolation in all cases and that the paravirtualization system offers nearly the same benefits -- no degradation in many cases with at most 1.7% degradation in the disk intensive test. The results for operating system level virtualization systems are varied -- illustrating the complexity of achieving isolation of all resources in a tightly coupled system. Our results highlight the difference between these classes of virtualization systems as well as the importance of considering multiple categories of resource consumption when evaluating the performance isolation properties of a virtualization system.