Large Language Model Inference over Edge-assisted IoT: An Cost-efficient Optimization Method

With the widespread adoption of Internet of Things (IoT) devices and the rapid advancement of edge computing technologies, there is a growing demand for the application of Large Language Models (LLMs) in IoT environments assisted by edge computing. However, unlike the batch processing inference mode of traditional artificial intelligence tasks, the inference process of large language models exhibits significant iterative characteristics. This results in the processing time of a single request not only depending on its task complexity but also being influenced by other requests within the same batch. Therefore, when designing scheduling strategies, it is essential to comprehensively consider the overall processing situation of each batch of tasks. Based on this, this study proposes an innovative finegrained scheduling method. This method first intelligently selects requests that need to be offloaded based on the real-time load status of computing nodes and network conditions. Secondly, by predicting the length of received requests, it dynamically adjusts the number of requests per batch to maximize the alignment of request processing times, thereby optimizing the overall system performance.