What question did this study set out to answer?

This research aims to develop a reliable method for variance estimation in online A/B tests, addressing the limitations of current assumptions.

February 2, 2017

Trustworthy Analysis of Online A/B Tests

View Full Paper

Ask AI

Bookmark

This research aims to develop a reliable method for variance estimation in online A/B tests, addressing the limitations of current assumptions.
Proposed a new method of variance estimation based on plausible assumptions applicable to various randomization mechanisms.
Performed analysis on both simulated and empirical datasets to evaluate the performance of the new method compared to existing ones.
Investigated conditions under which the randomization unit can be treated as independent and identically distributed (iid).
The new variance estimation method shows superior performance compared to two commonly used methods, especially when treatment effect variation is small.
The analysis reveals that treating the randomization unit as iid may lead to under-estimation of variance when treatment effect variation is large.

No takes yet. Share an insight, caveat, or question.

Randomized trial analyzes variance estimation methods in online A/B tests, suggesting improved accuracy for tech companies.

Deng et al. (2017) studied this question.

View Full Paper

Ask AI

Bookmark