122 Statistically valid machine learning fairness evaluation

Objectives/Goals: Within the machine learning “fairness” literature, models audits are point estimates, and only effect size is considered. The well-developed frameworks in biostatistics for diagnostic testing apply directly to classifiers more generally; these principles can be extended to have statistically valid fairness audits. Methods/Study Population: We made original linkages between machine learning methodology and biostatistics, particularly to diagnostic testing, to get analytic biostatistical methods for model evaluation that are applicable to fairness testing. Results/Anticipated Results: The use of odds ratios to compare ratios of binomial proportions is superior to taking ratios of binomial proportions directly, as is currently standard in machine learning. Odds ratios also have an analytic asymptotic approximation for standard errors, by which we can test if a point estimate (effect size) is greater than what we would expect from noise. In diagnostic testing, the odds ratios considered are generally across the cells of a confusion matrix, but we can form odds ratios for specific marginal quantities (positive predictive value, negative predictive value, true positive rate, true negative rate) for specific aspects of model performance or fairness. Discussion/Significance of Impact: Biostatistics can make direct contributions to instantly improve the state of machine learning practice, giving ready-made methods that can take account generations of hard-won lessons about the importance of uncertainty quantification and sample size in making claims and conclusions.