ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU

ModeSwitch-LLM is a lightweight request-boundary controller for improving single-GPU large language model inference efficiency by routing each request to an appropriate fixed inference mode. Instead of relying on one static serving configuration, the system selects among FP16, quantized modes, speculative decoding, prefix caching, continuous batching, and hybrid modes such as GPTQ plus prefix caching and INT8 plus continuous batching using cheap workload-level features. We evaluate ModeSwitch-LLM on Meta-Llama-3.1-8B-Instruct served through vLLM on a single NVIDIA A100 GPU. On deployment-style synthetic workloads, the online controller achieves a 2.10× mean latency speedup over FP16 and a 0.48× mean energy ratio, corresponding to 51.7% lower energy per generated token. On automatic benchmarks used as a quality gate, accuracy remains close to FP16 with a mean delta of +0.17 percentage points. The repository includes benchmarking scripts, controller implementation, workload definitions, plotting utilities, and result-generation code. The results suggest that simple request-aware routing can recover substantial inference efficiency from existing serving modes without retraining the model or changing its architecture.