This technical whitepaper proposes Layered Cluster-Compute Architecture, or LCCA, a cost-first and verification-aware lifecycle for reducing repetitive large language model inference. The central claim is not that caching, routing, retrieval, or tool use are new. Rather, LCCA organizes these familiar components into a governed lifecycle for routine AI work: detect repeated workloads, cluster them, route them, verify them, compile them into reusable cluster-kernels, register them with ownership and rollback, monitor them for drift, and retire or rebuild them when necessary. A cluster-kernel is defined as a versioned, callable, testable, and auditable routine compiled from repeated verified traces. LCCA is designed for high-frequency, stable, bounded, and verifiable workloads such as metadata generation, document engineering, support routing, standard code scaffolding, repository descriptions, schema transformations, low-risk summarization, and recurring tool workflows. The primary evaluation metric is cost per verified useful output. Secondary metrics include Wh per verified output, TOTEN per verified output, kernel hit rate, frontier escalation rate, verification pass rate, fallback success rate, latency, and pilot risk magnitude. TOTEN is defined as Total Token-Equivalent Number and accounts for routing prompts, verifier prompts, retrieval context, tool instructions, fallback calls, PromptOps activity, night-cycle distillation prompts, and other token-equivalent overhead. The architecture is sidecar-first rather than replacement-first. It is intended to be tested through offline replay, shadow mode, read-only assist, and limited canary rollout before any production traffic is affected. It also includes PromptOps cards, allowing AI assistants to help engineers perform workload analysis, kernel candidate mining, cost estimation, verifier design, dashboard configuration, and night-cycle distillation under human supervision. Version v0.4 introduced production-edge safeguards for extreme boundary states, including asymmetric low-cost verifier policy, verifier tax accounting, gateway latency budget, fast-path short-circuit routing, fractional canary probing for kernel drift, in-flight request coalescing for burst new loads, probabilistic jittered degradation for hot-kernel eviction, stale-while-revalidate, stale-if-error, and capacity-aware frontier leakage control. Version v0.5 adds logic-closure and flow-accounting patches. It clamps probabilistic leakage into the valid probability range, separates predicted route quantities from posterior verification results, treats gateway cost as traffic-mix dependent, separates initial dispatch fractions from final mutually exclusive service-route fractions, accounts for probe calls as side costs, and adds nonlinear pilot-risk modeling options for production-grade risk assessment. LCCA does not claim to replace foundation models, frontier reasoning, retrieval systems, caches, routers, or tool-calling frameworks. It proposes a practical engineering lifecycle for reducing repetitive LLM inference while reserving frontier models for genuinely novel, uncertain, or high-value tasks. The guiding motto is: compile the routine; reserve the frontier.
Frank F. (Arcman) Meng (Sat,) studied this question.