Knowledge Graph Empowered Machine Learning Pipelines for Improved Efficiency, Reusability, and Explainability

Artificial intelligence (AI) pipelines are complex, heavily parameterized, and expensive to execute in terms of time and computational resources. Consequently, it is onerous to run experiments with all possible parameter combinations to achieve an optimal solution. However, these AI experiments can be optimized by recommending relevant parameters to commence the experiments, reducing search space significantly, which can be fine tuned further. The relevant parameters can be identified by observing the metadata of pipelines executed in the past, and the relevant pipeline with relevant parameters can be recommended to the user. Currently, there are various metadata frameworks that automatically record the metadata of AI pipelines. Developing a recommendation system requires understanding pipeline metadata components and their interactions. There is a need to represent the metadata generated by these AI pipelines that capture the relationship among these pipeline entities. This article presents a knowledge-infused recommender that utilizes prior knowledge and metadata of already executed pipelines represented using the proposed metadata schema to recommend a relevant pipeline per user queries. Unlike black-box models, the use of knowledge graphs makes recommendations explainable, improving transparency and trustworthiness for the users.