Low-overhead General-purpose Near-Data Processing in CXL Memory Expanders

To overcome the memory capacity wall of large-scale AI and big data applications, Compute Express Link (CXL) enables cost-efficient memory expansion beyond the local DRAM of processors. While its CXL.mem protocol stack minimizes interconnect latency, CXL memory accesses can still result in significant slowdowns for memory-bound applications. While near-data processing (NDP) in CXL memory can overcome such limitations, prior works propose application-specific HW units that are not suitable for practical CXL memory-based systems that should support various applications. On the other hand, existing CPU or GPU cores are not cost-effective for NDP because they are not optimized for memory-bound applications. In addition, the communication between the host processor and CXL controller for NDP offloading should achieve low latency, but the CXL$.$io (or PCIe) protocol incurs μs-scale latency and is not suitable for fine-grain NDP. To achieve high-performance NDP end-to-end, we propose a low-overhead general-purpose NDP architecture for CXL memory referred to as Memory-Mapped NDP (M²NDP), which comprises memory-mapped functions (M²func) and memory-mapped μthreading (M²μthr). The M²func is a CXL.mem-compatible low-overhead communication mechanism between the host processor and NDP controller in the CXL memory. The M²μthr enables low-cost, general-purpose NDP unit design by introducing lightweight μthreads that support highly concurrent execution of NDP kernels with minimal resource wastage. By combining them, our M²NDP achieves significant speedups for various applications, including in-memory OLAP, key-value store, large language model, recommendation model, and graph analytics by up to 128× (11.5× overall) and reduces energy by up to 87.9% (80.1% overall) compared to a baseline CPU or GPU host with passive CXL memory.