Four Decades of Symbolic Knowledge Extraction from Sub-Symbolic Predictors. A Survey

Issues deriving from the opaque behaviour of prediction-effective, yet non-interpretable, machine learning predictors are being studied and analysed since many decades. One of the main research branches consists of adopting anyway the unintelligible models, thanks to their predictive performance, but queueing to the learning workflow a dedicated technique aimed at post-hoc extracting human-interpretable symbolic knowledge. Following this research line, a growing number of very different knowledge-extraction procedures have been designed over the last four decades, making it difficult for end-users and researches to orient themselves towards the selection of the most suitable one. Accordingly, this survey aims at providing a guide to perform an aware selection of the knowledge-extraction techniques that most probably fit a given task.