Common Metadata Framework: Integrated Framework for Trustworthy AI Pipelines

In this paper we present the capabilities of Common Metadata Framework (CMF) to enable trustworthy AI. CMF is a decentralized framework for tracking metadata and lineages of datasets and machine learning (ML) models in artificial intelligence (AI) pipelines. The framework provides a few unique features for ML practitioners, such as effortless management of distributed AI pipelines that span across the edge, high-performance systems, and public and private clouds. It provides an unbreakable audit trail and model provenance, resulting in trustworthy models. It ensures reproducibility by versioning artifacts and source code. CMF bridges the gap between the pipeline-centric and model-centric views of the AI metadata. This end-to-end approach to metadata logging unlocks a comprehensive understanding of ML workflows, enabling more efficient management and optimization of AI pipelines.