Latency-Optimized Language Model Inference in Edge Computing Environments

Abstract: Latency optimization is crucial for deploying large language models (LLMs) in edge computing environments, where real-time processing is often required for applications such as autonomous driving, smart healthcare, and industrial automation. This paper presents a comprehensive approach to minimizing inference latency for language models on edge devices. We explore various model compression techniques, including edge caching, model partitioning, task allocation, and lightweight model deployment, alongside advanced containerization and orchestration strategies. Our methodology involves an integrated edge computing platform that dynamically adapts data placement and function orchestration to reduce end-to-end latency. Experimental results demonstrate significant latency reductions and efficient resource utilization compared to traditional approaches. These findings underscore the potential of edge computing to support latency-sensitive applications by leveraging optimized LLM inference.