On Machine Learning models explainability in the banking sector: the case of SHAP

Machine Learning models explainability has recently become a very popular topic in the banking sector. We apply Shapley values and the Python library SHAP to a real credit risk database and show that the two available SHAP types (interventional and path-dependent) provide very similar results even under correlated features. The main drawback of SHAP is that it is not portfolio invariant, this is, the explanation for the prediction provided for each observation depends on the portfolio distribution of the features. This can be a serious problem for customers and banking regulators, who expect that explanations will stay stable as long as the clients characteristics do not change. We conduct several tests and show that the SHAP explanation of an observation may considerably change depending on the rest of the portfolio distribution. As a consequence, the explanation of a client may vary over time even if her characteristics do not change and banks using the same model (for ex. commercial models) may provide different explanations to the same client.