Counterfactual Replay with Interference-Aware Stratification: A Methodology for Trustworthy A/B Testing of Generative Re-Rankers in OTT Recommendation

Generative re-rankers, i.e., LLM policies that reorder a candidate slate from an upstream retriever, have become a practical fixture in over-the-top (OTT) streaming services, prized for capturing semantic intent and slate-level coherence that pointwise rankers overlook. Evaluation practice, however, has not kept pace with deployment. Standard fixed-horizon A/B tests on cumulative engagement metrics mislead practitioners under two conditions common to generative deployments on shared catalogs: (i) exposure interference, in which treated and control users draw from the same trending shelf seeded by aggregate plays, so titles favored by the treatment arm gain popularity and contaminate the control; and (ii) policy non-stationarity, in which prompt templates, grounding indices, and sampling temperatures shift weekly, voiding the logging propensities that off-policy estimators depend on. We introduce Counterfactual Replay with Interference Stratification (CR-IS), which attacks both pathologies through three mechanisms: (1) propensity recovery via a lightweight discriminative student distilled from the generative teacher; (2) stratified randomization over a co-exposure graph built from shared shelf impressions; and (3) always-valid sequential testing to permit early stopping without alpha inflation. On a controlled simulation adapted from the MIND news-recommendation benchmark, with cluster-localized interference injected via a trending-shelf mechanism, CR-IS cuts average-treatment-effect bias by at least 85% against a vanilla A/B test and by 80–93% against four off-policy baselines (IPS, SNIPS, DR, SNDR), while cutting RMSE by 26–57%. Stratified estimation also surfaces subgroup effects that aggregate metrics mask, which fits naturally into responsible-AI review. A reference implementation layered over standard feature-store and experimentation infrastructure accompanies this paper; production validation on live traffic is future work.