Explainability Can Build Practical Trust, But Not Complete Trust, in AI-Assisted Criminal Investigations

Artificial intelligence is increasingly used to help investigators identify potential suspects, but the most accurate models are often “black boxes” whose reasoning is hidden and therefore not transparent in nature, even if they are seen as ‘trustworthy’. Explainable AI (XAI) tries to address this by giving simpler explanations of model decisions, yet an explanation is itself a model and may not reflect what the original model actually does. This paper asks to what extent highly accurate black-box systems and their explanations can be trusted in criminal investigations. It distinguishes practical trust, which means using AI outputs as leads for human review, from complete trust, which means treating outputs as evidence without further scrutiny. It argues that explainability can support the first but not the second; an explanation simple enough for people to understand must leave out part of the model’s behaviour, internal biases, and dependencies, while one that reproduces the model fully will become a black box model itself and thus is not helpful. To illustrate this, a simple Java simulation compared a black-box scoring function with an explanation that omits one small interaction term, across 1,000 randomized cases. The two agreed on about 98-99% of decisions but disagreed on borderline cases near the decision threshold, and a model with a smaller average gap did not show better agreement. The omitted term was also invisible in the explanation itself. A real black-box model, with unknown structure and many more interacting variables, offers no such bound on the gap. The simulation thus does not prove that a faithful explanation must become the model itself. However, it illustrates the point that any explanation simpler than the model leaves a gap, and that gap decides cases near the threshold. This is an illustrative simulation and model and therefore does not test the argument, but it suggests that explanations should support human review, not replace it.