Paying Too Much Attention? Why the Transformer vs. GBDT Debate is All About the Data

The current hype surrounding Transformer architectures has undeniably reached tabular machine learning. This paper investigates whether these novel models genuinely outperform traditional tree-based ensembles across all scenarios, or if enthusiasm obscures operational reality. A systematic benchmark comparing Deep Learning approaches (FT-Transformer, TabFlex, and the Foundation Model TabPFN) against established Gradient Boosted Decision Trees (XGBoost, LightGBM, and CatBoost) is presented. Evaluating 20 diverse datasets under a strict fixed-budget protocol on consumer-grade hardware (Apple M1 Pro), the findings reveal that model superiority is strictly dictated by data scale, feature modality, and underlying inductive biases. While TabPFN excels in small-data regimes (N < 2,000) on continuous manifolds by leveraging pre-trained priors, tree-based ensembles consistently dominate larger datasets (N > 10,000) due to their axis-aligned decision boundaries and default robustness. Crucially, attention mechanisms incur training and inference latencies up to 1,000 times higher than GBDTs - a severe efficiency gap under Green AI considerations. Finally, interpretability trade-offs (TreeSHAP vs. opaque attention maps) and hybrid ensembling are integrated into a pragmatic practitioner's decision framework.