ToR-Lite: A Lightweight Semantic Query Decomposition for Multi-Hop Retrieval-Augmented Generation in Cloud-Based AI Systems

Cloud-based AI systems increasingly rely on Retrieval-Augmented Generation (RAG) to handle complex, knowledge-intensive queries. However, query decomposition for multi-hop retrieval—traditionally powered by large language models (LLMs)—incurs significant latency and cost, rendering it impractical for large-scale, cost-sensitive cloud deployments. We propose ToR-Lite, a lightweight, generative LLM-free semantic query decomposition framework designed to enhance multi-hop retrieval efficiency in cloud-based AI systems. ToR-Lite employs a novel Word-Window Splitting algorithm that detects semantic breakpoints via sliding window embeddings, effectively decomposing complex queries without expensive LLM inference. Experiments on the MultiHop-RAG benchmark (n = 2255) demonstrate that ToR-Lite achieves +6.03 pp Hits@10 and +0.89 pp Exact Match improvements over the Baseline, while operating 3.18 times faster than LLM-based Adaptive ToR. Retrieval performance correlates monotonically with decomposition granularity: three sub-query decompositions (#Dq = 3) yields a +7.00 pp Hits@10 improvement, confirming that semantic granularity is a key driver of retrieval performance. Comparison with rule-based Baselines confirms that these gains derive from the precision of semantic boundary detection rather than decomposition quantity alone. ToR-Lite delivers nearly twice the retrieval improvement per unit of computational cost, offering a practical and cost-effective solution for latency-sensitive cloud AI deployments.