Beyond cross-classification: evaluating tabular dataset similarity via feature importance rankings

Background and objective Synthetic data offers a promising solution to data access limitations in healthcare, but evaluating the similarity between real and synthetic datasets remains challenging. Traditional cross-classification metrics have limitations including unbounded values, high variance, and computational cost. This study evaluates feature importance ranking metrics as complements for assessing tabular dataset similarity. Methods We conducted controlled experiments on seven healthcare datasets with five perturbation types (mean-variance drift, noise injection, category flip, correlation mixing, missingness). Feature importance was extracted via permutation importance from multiple classification and regression models (Logistic Regression, Random Forest, Gradient Boosting, K-Nearest Neighbors), and similarity was assessed using ranking metrics (Kendall tau, Weighted tau, Spearman rho, NDCG, Rank-Biased Overlap (RBO)). Results were validated with bootstrap confidence intervals (200 resamples). Results Kendall’s τ was the strongest ranking metric overall, achieving the highest similarity values across perturbation types, the lowest variance under disruptive conditions (missingness: τ SD = 0.12 vs RBO SD = 0.18), and the strongest correlation with utility degradation ( $$r = -0.475$$ , $$p < 0.001$$ ). Rare category collapse left rankings largely intact ( τ ≈ 0.96 ) in the predominantly numeric UCI health datasets, while correlation attenuation and missingness were the most disruptive perturbation types ( τ ≈ 0.66 –0.70). Mean-variance drift produced τ = 0.73 : tree-based models are theoretically invariant to affine transforms, but the mixed-model ensemble (including KNN and logistic regression) yields non-trivial drops. Conclusions Kendall’s τ is the primary recommended ranking metric for tabular dataset similarity assessment, offering the strongest association with downstream utility and the lowest variance under disruptive perturbations. RBO provides a useful complement where bounded $$[0,1]$$ interpretation is required. Both metrics depend on the model ensemble used and must be paired with distributional metrics for comprehensive evaluation.