Fault Escaping: Improving Robustness of DPU Enhanced Platform with Mutual Assisted VM Recovery

Modern cloud servers achieve significant performance improvements by exploiting data processing units (DPUs) to offload virtualization overhead. Unlike traditional monolithic hypervisors, this offloading approach splits VM state and distributes hypervisor functions across the DPU and the Host, transforming the cloud server from a single system into a sophisticated orchestration of multiple self-managed processing units. However, the failure rate of these systems has significantly increased, as errors from either the DPU or the Host can crash the entire machine. All these changes necessitate a comprehensive revamp of current VM fault tolerance and isolation mechanisms.This paper explores a novel approach to VM fault tolerance by treating the split hypervisors on the DPU SoC and the Host as redundant peers. We propose a fault-escaping scheme that enables VMs to escape from a failing SoC or Host. With hybrid synchronization, the entire VM state becomes fully accessible on either the SoC or the Host with minimum synchronization overhead (in kilobytes). Instead of recovering the failed hypervisor, this scheme migrates the verified VM state, allowing the VM to escape from the failure and resume on a new machine. The evaluation demonstrates that the escape-based mechanism introduces no runtime performance overhead, achieves performance levels comparable to VM live migration, and reduces the VM recovery time objective (RTO) by 4x to 14x in one of the largest cloud production environments.