Explaining Tree-Based Regression Model Predictions with Nearest Training Neighbors

In recent years, there has been a lot of discussion around the concept of interpretability in machine learning (ML). The inability to comprehend the reasoning behind model predictions from complex 'black-box models' has been a chronic obstacle for adoption of ML based solutions. The expectations from an ML solution have advanced beyond obtaining automated predictions to also understanding the context of a prediction. Classification and Regression Tree (CART) algorithms have become the staple ML techniques for a large variety of corporate use-cases (customer attrition, product pricing, auto-approval of requests, etc.). To interpret the predictions from these models, techniques such as LIME and SHAP have concurrently found solid footing by evaluating feature importance values across the feature set. However, the fact remains that the mathematical foundation of these interpretability techniques themselves remains complex. This requires the user to have faith in the methodology in order to trust the feature importance values. Hence, this becomes a catch-22 situation where it is attempted to explain black-box predictions further with black-box interpretability techniques. This paper proposes a methodology that is built on top of feature importance values obtained from SHAP (LIME can be used as well). Utilizing these values, data points which the model considers as most similar to the prediction data point (or nearest training neighbors), can be extracted from the respective training data set. As the training data comprises realized target values, the target value of these nearest training neighbors can be used to justify the prediction from the model. This paper suggests, firstly, how these nearest training neighbors can be obtained and, secondly, 3 metrics that can be used to justify how well the nearest training neighbors extracted from this methodology represent a prediction holistically. Predictions from tree-based regression models are obtained by averaging various groups of training data points (through bagging and boosting). The primary goal behind this methodology is to identify which data points from the training data set were most likely to have a presence in those groups.