Detecting trade-offs between fairness, stability, and accuracy for responsible machine learning model selection

The subject of the study in the article is the process of machine learning model selection performed by data scientists to build models in critical areas. The purpose of the work: 1) create a software library for measuring the accuracy, stability, and fairness of models; 2) conduct experiments to identify trade-offs between fairness, stability, and accuracy; 3) propose a responsible model selection process to improve the safety of using machine learning models. The article provides for the following tasks: to measure the fairness and stability of machine learning models and to investigate their relationship with the accuracy of models. The following methods are introduced: empirical evaluation, the theory of decomposition of model error into bias and variance, the theory of algorithmic fairness, and methods for quantitative assessment of uncertainty. Results achieved: 1) low predictive variability is proposed as a desirable property to ensure safety and equality of variability between different social groups as a new metric of fairness of machine learning models; 2) it is demonstrated how stability analysis helps specialists overcome the challenges of model multiplicity and choose reliable, stable and fair models; 3) an open-source software framework for community use is created that integrates stability measurements into model development processes. Conclusions. This work proposes a new paradigm of group fairness that combines the issues of correctness/quality and randomness/stability from the research agenda of responsible artificial intelligence. The application of the proposed approaches helps in the responsible selection of machine learning models under conditions of model multiplicity, demonstrating that, although there may be many models with comparable accuracy, there is only one (or a few) "best" model that is reliable, fair and stable, as it should be.