GPU Memory Reallocation Techniques in Fully Homomorphic Encryption Workloads

Operations in fully homomorphic encryption (FHE) like bootstrapping use large amounts of GPU memory, and typically exceed retail GPU memory capacity. Using CUDA Unified Memory (UM) for memory oversubscription in such FHE frameworks is not enough to overcome these issues because performance is very slow. The new CUDA stream allocator allows more rapid memory allocation than UM, which requires slow global device synchronization when memory is freed. This leads to slow GPU kernel performance even when memory is available. We design a reallocation runtime to take advantage of the stream allocator when device memory is readily available, and can switch to UM on-demand when it is necessary to not crash from out-of-memory (OOM) issues. We design it to be up to 2x faster than a naive reallocation, using hybrid stream allocator and UM heuristics when memory is oversubscribed, and with nearly identical speed to vanilla asynchronous stream allocation when memory is available.