GMM: An Efficient GPU Memory Management-based Model Serving System for Multiple DNN Inference Models

Recent DNN model serving systems have begun to use multi-GPUs and distributed systems to serve a variety of inference models as services to users. However, modern GPU-based model serving systems cannot execute multiple inference models beyond the GPU memory size. This is because inference models occupy and execute on their own pre-allocated GPU memory that cannot be shared with other inference models. As a result, more GPUs or large systems are required as the demand for more inference models increases. This paper proposes an efficient GPU Memory Management-based model serving system, called GMM, to serve multiple inference models beyond the GPU memory limit. The GMM initializes the GPU memory space as a large tensor to allow all models to cache anywhere in the GPU memory without any constraints. Then to ensure that inference models can execute on the GPU without any conflicts, the GMM finds unused GPU memory space and uses the memory overwriting method to cache model's parameters for execution. The proposed system allows for more inference models to be executed parallel on a single GPU than previous systems, resulting in higher throughput and in some cases, shorter inference time.