A framework for evaluating transparency in AI projects

Transparency has become a key requirement of any artificial intelligence (AI) project because it provides a basis for accountability, reliability, and compliance with regulations. Nonetheless, the current approach views transparency as a principle but does not offer ways to evaluate it. The current paper proposes an evaluation framework of transparency in AI projects based on four types of transparency including data, algorithmic, process, and outcome transparency. The proposed framework is based on external drivers and internal enablers of transparency found in the literature and differentiates between transparency indicators and transparency metrics, where the former is a description of the evidence of transparency while the latter is a quantification of such evidence. The requirements of transparency depend on a particular stage of the AI project including data collection and pre-processing, modeling, evaluation, and deployment. The framework has been validated using a case study and an expert assessment. It has been found to evaluate the level of transparency in AI projects adequately.