Certified Microsecond Answers for an LLM's Repeated Fixed-Choice Decisions

Many production calls to large language models (LLMs) are classification in disguise: an intent, a routing target, a moderation verdict or a tool choice, picked from a fixed list and asked again and again with small variations. We study when a tiny student, trained only on the LLM's own logged answers, can answer such a decision in place of the LLM with a guarantee on how often it disagrees. We describe a system, shad0w, that fits a hashed bag-of-items table to logged (input, answer) pairs and gates it with a Learn-then-Test certificate: on answers it never trained on, it selects a confidence threshold above which disagreement with the LLM is at most α with probability at least 1−δ; below the threshold the LLM answers and its answer becomes training data. With a hosted LLM as the teacher and α = 5%, the table answered 60.8%, 66.8% and 85.5% of fresh traffic on BANKING77, CLINC150 and AG News, with realised disagreement of 3.0%, 6.0% and 4.7%. The CLINC150 figure exceeds α because its test traffic contains far more out-of-scope inputs than the logs, a direct failure of the exchangeability the certificate assumes, which we analyse. A decision costs about one to ten microseconds on a CPU, against roughly 1.4 s for a hosted LLM call. In a CPU-only comparison with five open decision models on six public tasks, the table was the most accurate on four, and placing it in front of the most accurate model matched or exceeded that model's accuracy on every text task at 2.5–7.8× its speed. We report the failures alongside the results. Code (Apache-2.0): https://github.com/2796gaurav/shad0w. Software archive: https://doi.org/10.5281/zenodo.23598814