On Sampling Reliability in Offline Recommender Evaluation

When online testing is impractical or risky, offline evaluation remains the primary mechanism for benchmarking recommender systems. Yet it is inherently affected by two sources of distortion: exposure bias, arising from system-driven item visibility, and sampling bias, introduced when evaluation relies on a subset of logged items rather than the full catalog. Prior work has proposed techniques to mitigate sampling bias. However, these techniques are typically assessed on static logs. Such assessments do not establish whether the techniques support reliable model comparisons under varying exposure conditions or against true user preferences. In this paper, we systematically study how logging and sampling choices jointly affect the reliability of offline evaluation. Using a fully observed dataset as ground truth, we simulate diverse exposure regimes and evaluate common sampling strategies along four dimensions: resolution (model separability), fidelity (agreement with full evaluation), robustness (stability under exposure bias), and predictive power (alignment with ground truth). We further examine whether these criteria align or reveal trade-offs in sampler performance. To strengthen external validity, we replicate key analyses on two additional benchmark datasets with fixed missing-completely-at-random (MCAR) test exposure. Our findings characterize when sampling distorts evaluation outcomes and provide controlled empirical evidence regarding how sampling strategies behave under different exposure and sparsity conditions.