Uncertainty-aware instance-wise feature selection with adaptive graph regularization

Instance-wise feature selection aims at discovering the most relevant set of attributes for each particular sample, providing a natural direction for achieving adaptive learning on inputs of heterogeneous and high-dimensional nature. While recent neural-net models made impressive advances, all current methods share a common drawback: the selected feature mask is considered as a definite one. Such a degree of certainty with respect to selected features is hardly justified in case of imperfect input data, correlations between predictors, and small sample sizes. The resulting too confident interpretation of features’ relevance becomes highly problematic in critical applications, e.g., medical decision-making, risk assessment, and fault diagnosis. Despite such danger, all current methods of instance-wise feature selection have an important common drawback: they assume the selected feature mask to be definite, providing no way of estimating its reliability. In contrast, our paper proposes the first-ever approach that deals with the uncertainty inherent to feature selection itself, not just in model prediction. We propose UDIFS - a novel Uncertainty-aware Deep Instance-wise Feature Selection method with Adaptive Graph Regularization. The fundamental concept is to view feature selection as a stochastic process where, rather than fixing on a specific mask for each instance, a distribution over masks is learned using a Concrete (Gumbel-Sigmoid) distribution. By sampling multiple independent masks at inference time, we can estimate uncertainty in predictions and in feature selection through Monte Carlo estimation. The second key element of our approach is an adaptive Graph Laplacian regularization method which is applied directly to the sampled masks themselves (not the latent representations) in order to ensure that semantically similar instances have consistent explanations without having to resort to a GNN-based approach. We compare UDIFS (Graph) to eight other approaches grouped into four categories: instance-wise selectors such as INVASE, neural feature selectors like STG, LassoNet, CAE, c-STG for 2023, and MCDropout-FS. We show in experiments run on three diverse datasets: Car Evaluation (low dimensionality), Dexter (ultra-high dimensional text corpus), and IMDB Sentiment (high dimensional) that UDIFS (Graph) attains the lowest feature selection uncertainty among all baselines while maintaining classification accuracy. UDIFS is, to the best of our knowledge, the first approach to (i) directly incorporate uncertainty estimation in the feature selection phase itself instead of in predictive models, and (ii) directly use Graph Laplacian regularization on instance-wise feature selection masks instead of on latent space representations. The existence of a non-zero positive gap in terms of uncertainty between correctly and incorrectly classified instances indicates that the learned uncertainty signal holds true semantic meaning. This indicates a need for a paradigm shift; i.e., evaluation of explanations based on accuracy and reliability measures together. The complete implementation and source code are publicly available in the GitHub repository: https://github.com/Vijayavarsini/UDIFS.