TMO: Transparent Memory Offloading in Datacenters

The unrelenting growth in the memory demands of datacenter applications, combined with DRAM’s volatile prices and ever-increasing costs, has made DRAM a major infrastructure expense. Alternative technologies, such as NVMe SSDs and emerging NVM devices, offer higher capacity than DRAM at a fraction of the cost and power. A promising approach is to transparently offload colder memory to cheaper memory technologies via kernel or hypervisor techniques. The key challenge, however, is to develop a datacenter-scale solution that is robust in dealing with diverse workloads and large performance variance of different offload devices, such as compressed memory, SSD, and NVM. This paper presents TMO, Meta’s transparent memory offloading solution for heterogeneous datacenter environments. TMO introduces a new Linux kernel mechanism that directly measures, in realtime, process stalls due to resource shortages across CPU, memory, and I/O. Guided by this information and without any prior application knowledge, TMO automatically adjusts how much memory to offload to heterogeneous devices (e.g., compressed memory or SSD) according to the device’s performance characteristics and the application’s sensitivity to memory-access slowdown. To maximize memory savings, TMO targets both anonymous memory and file cache, balancing the swap-in rate of anonymous memory and the reload rate of file pages that were recently evicted from the file cache. Moreover, it identifies offloading opportunities not only from the application containers but also from the sidecar containers that provide infrastructure-level functions. TMO has been in production since 2021, saving 20–32% of total memory across millions of servers in our hyperscale datacenter fleet. We have successfully upstreamed TMO into the Linux kernel.