Human decisions about the design, deployment, and governance of AI systems are increasingly complex, involving multiple desiderata such as accuracy, fairness, transparency, cost, and environmental impact. This paper proposes a novel and scalable framework for managing this complexity through application of resources from the field of Multi-Criteria Decision Analysis (MCDA). After an introduction to the “ n > 2 Criteria Problem” in decision-making about AI, an introduction to MCDA methods, and a review of prior applications of MCDA to decision support in decisions involving AI, we lay out our own novel and scalable framework for applying MCDA to such decisions. We illustrate the framework through application to four simulated scenarios involving a medical institution’s choice among AI-powered diagnostic tools with varying performance on accuracy, fairness, transparency, and cost. We apply five multi-criteria decision analysis (MCDA) methods — namely, the Weighted Sum Model (WSM), the Analytic Hierarchy Process (AHP), the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), the VIKOR method (VlseKriterijumska Optimizacija I Kompromisno Resenje), and the Preference Ranking Organization METHod for Enrichment of Evaluations (PROMETHEE) — to rank ten AI models under four distinct weight configurations reflecting realistic deployment contexts. Sensitivity analyses are then conducted to examine how variations in criterion weight assignments affect model rankings across methods and scenarios. In concluding sections we discuss the scalability of these techniques for design, deployment, and policy decisions involving AI, as well as the likely limitations of MCDA methods for decision support involving AI.
Olabanjo et al. (Sat,) studied this question.