Key points are not available for this paper at this time.
Despite the ubiquity of forensic evidence in criminal cases over many decades, only recently have scholars begun in earnest to consider how to account for inconclusive decisions in error rate calculations for forensic feature-comparison methods (Dror and Langenburg, 2019; Dror and Scurich, 2020). Given the controversy and diverse viewpoints the issue continues to provoke (Biedermann and Kotsoglou, 2021; Hofmann et al., 2021; Dorfman and Valiant, 2022), Arkes and Koehler’s recent paper (Arkes and Koehler, 2022), which seeks to apply a signal detection theory approach to understanding the role of inconclusive decisions, is an important contribution. That said, we are concerned that Arkes and Koehler’s approach: (1) neglects to account for known differences in the way that inconclusive decisions are deployed by practitioners across various feature-comparison methods and (2) appears to posit blind proficiency testing as a near-complete solution to the host of complex problems associated with assessing the validity of such methods. First, there is discernible variation in the use of the inconclusive category across different feature-comparison methods (Tables 1 and 2). However well Arkes and Koehler’s scheme for classifying examiner decision-making—‘confident’, ‘conservative’ and ‘strategic’—might fit some disciplines under certain circumstances, it fails to account for a specific, documented way in which firearms examiners utilize the inconclusive category. Arkes and Koehler’s argument implicitly relies on the assumption that inconclusive decisions are truly a response by examiners to samples lacking information sufficient to reach more definitive conclusions, and thus will distribute relatively evenly between same-source and different-source comparisons. Indeed, in their stylized similarity distributions, Arkes and Koehler portray examiner discrimination ability and threshold placements as identical for same-source and different-source comparisons. Inconclusive rates for same-source samples and different-source samples as represented in friction ridge literature. All calculations are based on decisions made on samples deemed of value Friction ridge conclusions may be based on a 5-point conclusion scale, which includes identification, exclusion and inconclusive, as well as ‘support for different sources’ and ‘support for common source’ (Carter et al., 2020). Table 1 lists the inconclusive rate reported in Carter et al. (2020) for decisions made using a 5-point scale. If ‘support for different sources’ and ‘support for common source’ are instead treated as inconclusive decisions, the same-source inconclusive rate is 70.1% and the different-source inconclusive rate is 68.6%. If the same-source inconclusive rate is calculated based only on comparisons that were deemed of value for identification, the same-source inconclusive rate is 22.7%. If the same-source inconclusive rate is calculated based only on comparisons that were deemed of value for identification, the same-source inconclusive rate is 13.8%. Inconclusive rates for same-source samples and different-source samples as represented in friction ridge literature. All calculations are based on decisions made on samples deemed of value Friction ridge conclusions may be based on a 5-point conclusion scale, which includes identification, exclusion and inconclusive, as well as ‘support for different sources’ and ‘support for common source’ (Carter et al., 2020). Table 1 lists the inconclusive rate reported in Carter et al. (2020) for decisions made using a 5-point scale. If ‘support for different sources’ and ‘support for common source’ are instead treated as inconclusive decisions, the same-source inconclusive rate is 70.1% and the different-source inconclusive rate is 68.6%. If the same-source inconclusive rate is calculated based only on comparisons that were deemed of value for identification, the same-source inconclusive rate is 22.7%. If the same-source inconclusive rate is calculated based only on comparisons that were deemed of value for identification, the same-source inconclusive rate is 13.8%. Inconclusive rates for same-source samples and different-source samples as represented in firearms literature. All calculations are based on: (1) decisions made on samples deemed suitable for comparison and (2) within-class comparisons Inconclusive rates for same-source samples and different-source samples as represented in firearms literature. All calculations are based on: (1) decisions made on samples deemed suitable for comparison and (2) within-class comparisons If supported by research, there may be little danger in such an assumption when evaluating some feature-comparison methods. For example, across multiple pairwise studies conducted to date in the friction ridge realm, inconclusive decisions do frequently distribute relatively evenly between same-source and different-source comparisons (Table 1). But similar studies concerning firearms examination show consistent and substantial differences in the rate of inconclusive decisions between same-source and different-source comparisons. Firearms examiners consistently draw inconclusive decisions far more frequently when performing different-source comparisons than when performing same-source comparisons (Table 2). By treating inconclusive decisions as homogenous across feature-comparison methods, Arkes and Koehler’s approach neglects to grapple with the reality that firearms examiners are neither uniformly ‘confident’ nor ‘conservative’. Instead, whether due to adversarial allegiance bias (Murrie et al., 2013), training and standards that focus more on identifications than exclusions, lab protocols that disallow exclusion decisions unless class characteristics disagree or outside of ‘exceptional’ circumstances (e.g. Baldwin, 2014; Illinois State Police, 2019), or for other reasons, examiners far more readily default to inconclusive decisions where the data supports exclusion than where the data supports identification. In other words, by Arkes and Koehler’s terminology, examiners are ‘confident’ in calling identifications but ‘conservative’ in calling exclusions (Hofmann et al., 2021). We believe that such distinct behaviour concerning inconclusive decisions warrants a fourth category, what we call the Biased Examiner.1 The Biased Examiner’s use of inconclusive decisions, however much it may limit false positive and false negative errors, also depresses specificity, which declines heavily for all comparisons with the same class characteristics and may even fall to zero percent in laboratories that disallow exclusions based on individual characteristics. In other words, because of a combination of ‘conservative’ threshold setting for exclusion decisions and less discriminatory decision-making on different-source comparisons than in same-source comparisons, examiners avoid errors only by deciding not to render exclusion decisions on a large percentage of different-source comparisons. The undervaluing of specificity is not merely an academic problem. An erroneous failure to exclude an individual, even if the failure is based on an inconclusive decision, has important consequences that Arkes and Koehler acknowledge ‘may be serious’. Missed exclusions may leave the innocent languishing in custody while investigators attempt to develop other evidence of guilt, bias investigators against pursuing other credible suspects, or even contribute to a wrongful conviction by depriving the accused of exculpatory evidence. (Cole and Scheck, 2018). This problem is discipline-wide; it extends beyond examiners whose conclusions support the prosecution’s case. Inconclusive decisions made during re-testing by defence experts also limit the ability of the accused to effectively expose and counter false positives committed by government examiners. (People v. Pursley, 2018). It is no overstatement to say that many wrongful convictions are, at least in part, a product of forensic examiners overstating the results of their analyses. (Garrett and Neufeld, 2009; FBI, 2015; Murphy, 2019). Inconclusive decisions can be presented by prosecutors in court to imply that they are probative of guilt even though, particularly in firearms examination, they are far more prevalent when the samples in question originated from different sources. For example, a prosecutor may argue that if the accused was innocent, they (or the gun associated with them) would have been excluded (e.g. Mann, 2022). As attorneys who have long represented the indigent accused in cases involving feature comparison evidence, we can attest that this tack is unfortunately all too common in criminal trials, despite the accused having no burden of proof. And in fact, using a friction ridge example, Arkes and Koehler essentially acknowledge the value of such an approach when discussing the ‘diagnosticity of inconclusives’. Implicit in their discussion is a recognition that government actors can leverage such a strategy in seeking a conviction, even without an affirmative identification result. Prosecutors’ ability to persuade jurors that inconclusive decisions are probative of guilt is yet to be meaningfully addressed by the literature. In particular, a data-informed mechanism for informing jurors that inconclusive decisions are more likely in the case of non-matches has not yet been proposed. Without providing such a mechanism, Arkes and Koehler’s approach to the problem of inconclusive decisions generally, and their discussion of the diagnosticity of inconclusives more specifically, risks entrenching this gap in the literature and encouraging scientifically unsound claims by prosecutors. The Biased Examiner’s approach to inconclusive decisions fundamentally prejudices the accused, who is regularly deprived of exculpatory evidence by firearms examiners’ inability or unwillingness to reach exclusion decisions, while inflicting no commensurate penalty on the prosecution (Hofmann et al., 2021; Dorfman and Valiant, 2022). Mounting evidence from studies of comparison algorithms indicates that such systemic bias is not inevitable given the characteristics of fired bullets and cartridge cases: ‘objective’ comparison algorithms do not struggle to exclude; instead, they provide support for justified exculpatory conclusions at rates comparable to the support they provide for justified inculpatory conclusions without a marked increase in false positive or false negative rates. (Weller et al., 2012; Nally, 2021). Consequently, unless and until firearms examiners remedy the Biased Examiner problem, inconclusive decisions from the field will continue to require rigorous scrutiny and treatment as errors. Second, unfortunately, Arkes and Koehler’s proposed solution to strategic use of inconclusive decisions, blind proficiency testing, will not ameliorate the imbalances and prejudice we have described. As already outlined, the problem of inconclusive decisions being rendered more readily in comparisons of different-source samples stems from a number of sources, not only test-taking bias that may be mitigated through blind testing. Moreover, although, as one of the authors has already explained, we agree that error rate estimates based on non-blind testing cannot be relied upon (Koehler, 2013, 2017), we worry that even Arkes and Koehler’s characterization of blind proficiency testing as a solution to the strategic deployment of inconclusive decisions will leave readers of their paper—including prosecutors seeking the admission of evidence and judges making admissibility determinations—with an impression that could result in significant injustice to the accused. In recent years, firearms examination has come under substantial scrutiny by scholars who have, for a variety of reasons, including but not limited to concerns regarding inconclusive decisions, emphasized that the available scientific literature does not establish a credible estimate of the method’s error rate. (Tobin and Blau, 2013; PCAST, 2016; Dror and Scurich, 2020; Hofmann et al., 2021; Albright, 2022; Dorfman and Valiant, 2022). While blind testing is necessary to establishing trustworthy estimates of error, it is far from sufficient. Blind testing will not resolve the concerns raised by the scholars just mentioned that go beyond strategic deployment of inconclusive decisions: design and administration by parties uninterested in and independent of the field of firearms examination, the inclusion of close non-matches, appropriate sampling of participants and comparison items, attrition rates and data transparency, among others.
Sinha et al. (Wed,) studied this question.