The role of evaluations in reaching decisions using automated systems supporting forensic analysis

Automated systems allow forensic practitioners to perform analysis tasks that would otherwise be infeasible. However, unless the outputs of such systems are critically evaluated using a scientifically-based framework, there is a risk of undetected errors or bias resulting in wrong decisions. Furthermore, decisions based on automated system outputs that are not well understood or clearly explainable could violate fundamental human rights. These risks can apply to any automated system that supports forensic analysis, and are raised when machine learning is involved. This work presents a framework based on principles of scientific interpretation, and provides an evaluation hierarchy for automated systems, including machine learning approaches, to strengthen forensic conclusions. Specifically, three levels of evaluation are presented: performance, understandability and forensic evaluation. Approaches to clearly conveying the weight of forensic evaluations are discussed. Each level of evaluation is demonstrated in relation to actual automated systems. Finally, requirements for designing automated systems supporting forensic analysis are proposed, and future work is discussed.