Inference-Time Compute Allocation in LLMs: Dynamic Depth and Token Selection for Efficient Problem Solving

Large language models (LLMs) deliver impressive performance across diverse natural language processing tasks, but their intensive computation and memory requirements severely restrict real-world deployment, especially on edge devices and in latency-sensitive cloud services. Static inference pipelines execute full layers and full tokens for all inputs, resulting in significant resource waste for simple and routine queries. Although dynamic inference methods have emerged to alleviate this issue, most focus only on either layer-level or token-level optimization, and few provide reliable mechanisms to guarantee bounded performance degradation. In this paper, we propose D²TS-LLM, a unified plug-and-play framework for Dynamic Depth and Token Selection that achieves on-demand computational resource allocation for LLM inference. The framework consists of a multi-dimensional lightweight perceptron for estimating input difficulty and token importance, a joint decision module for adaptive layer activation and token-aware computation reduction, and a lightweight performance monitor with dynamic fallback to ensure performance loss remains within a predefined safe threshold. Extensive experiments on representative LLMs (LLaMA-7B, LLaMA-13B, OPT-6.7B) and five common benchmarks (MMLU, GSM8K, SQuAD v2.0, CNN/Daily Mail, Alpaca) demonstrate that D²TS-LLM reduces computational cost by 42.3%–61.3% and accelerates inference by 38.2%–55.7% with performance loss controlled below 1%. Compared with representative state-of-the-art dynamic inference, quantization, and cache optimization methods, our approach achieves superior efficiency-performance trade-offs and strong cross-model and cross-dataset generalization.