When Are Linear Classifiers Sufficient? A Re-evaluation and Conditional Boundary Exploration Based on OpenML

The point at which a linear classifier stops being "sufficient" lacks a quantitative definition. To address this, we evaluated 1,687 records from 298 OpenML datasets across three model families. Using a practical significance framework, we defined linear sufficiency as an accuracy drop within a tolerance threshold δ. At δ=0.02, linear sufficiency was observed in 68.9% of SVM datasets, 62.4% of NN datasets, but only 46.1% of DT datasets. Shallow decision tree extraction highlighted low feature-to-instance ratios and small class counts as primary predictors. However, leave-one-out validation exposed a clear split. The extracted rules worked well for the DT family (0.686 accuracy vs. 0.539 baseline) but failed to surpass baselines for SVM and NN. This suggests that while simple meta-features can predict linear sufficiency for decision trees, they cannot fully capture the complex boundaries of SVMs and neural networks.