Assesses algorithmic fairness metrics in defendants, suggesting a need for improved evaluation methods.
The 2016--2017 controversy over the COMPAS pretrial-risk algorithm exposed a deep ambiguity in the concept of algorithmic fairness. ProPublica showed that COMPAS's false-positive rate is substantially higher for Black defendants. Northpointe (the algorithm's vendor) replied that COMPAS approximately satisfies : among defendants it classifies high-risk, the observed recidivism rate is similar across races. Subsequent technical work proved that the two properties when race-specific recidivism base rates differ---a genuine impossibility theorem. The literature has since proliferated competing metrics (demographic parity, equalized odds, calibration, counterfactual fairness, individual fairness), each capturing one facet of the problem and none of them dispositive. We introduce a third axis of measurement that the impossibility result does not constrain: the (LBI), a matched-pair, case-level of whether an algorithm treats defendants who are similar on selected legitimate predictors similarly. LBI asks: when two defendants agree on legitimate predictors (age, prior offences, juvenile record, charge degree) to within the same Mahalanobis neighbourhood, does the algorithm assign them similar scores regardless of their race? For COMPAS the answer is a robust case-level disparity: at neighbourhood scale k = 20, ( ) = 1.050, 95\,\ defendants. The 5\,\ absolute terms but robust: it survives every legitimate-feature set we can test---including the contested juvenile controls---and at the decision-relevant unit ( = 1.090 on the binary high-risk classification). It is a signal the population-level metrics do not deliver; because matching is on a selected, incomplete feature set, it is a diagnostic to be investigated, not a proof of legal unfairness. LBI is a quantitative implementation of the Aristotelian ``treat-like-cases-alike'' criterion. It is computationally cheap, it applies to any algorithm whose inputs include both legitimate and protected attributes, it does not require knowledge of the algorithm's internals, and it produces a single number with a confidence interval suitable for regulatory and auditing use. The reference implementation is openly archived on Zenodo (doi:10.5281/zenodo.21310251). Target venue: Journal of Science and Law (under review). Author preprint deposited for archival and citation. Draft — pending author review.
No takes yet. Share an insight, caveat, or question.
Andrew Bond (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: