Merlin: Deterministic Byte-Exact Deduplication for Lossless Context Optimization in Large Language Model Inference

This preprint introduces a deterministic, byte-exact deduplication primitive for LLM context optimization. It operates three to four orders of magnitude below typical inference proxy budgets, achieving a median in-process latency of 1.10 microseconds (compared to standard 10–100 millisecond preprocessing budgets). We establish the lossless safety of this primitive across 240 evaluation cells spanning four production APIs (Google Gemini 2.5 Flash, OpenAI GPT-5.1, Anthropic Claude Sonnet 4.6, Meta Llama 3.3 70B) and three benchmarks (RULER, LongBench, HumanEval-Snowball). The aggregate quality delta attributable to deduplication is statistically null (+0.0 pp on the primary sweep, -0.5 pp on confirmation), with zero significant degradations after Bonferroni correction. Engineered for production, the engine is a zero-dependency, statically linked C++ binary (3.8 MB on Windows, 3.5 MB on Linux ARM64). Mathematical equivalence to Python's set() was verified at scale across 22.2 million public BeIR passages with zero violations. The primary contribution is proving that byte-exact deduplication preserves model quality at evaluation-grade resolution while operating safely within strict real-time inference constraints.