TabTranSELU: A transformer adaptation for solving tabular data

Tabular data are most prevalent datasets in real world, yet the integration of deep learning algorithms in tabular data often garners less attention despite their widespread utilization in other field. This phenomenon could be attributed to the dominance of the classical algorithms in their simplicity and interpretability, and the superior performance of the gradient boosting tree models in tabular data. In this paper, a simple yet affective adaptation of the Transformer architecture tailored specifically for tabular data is presented, not only achieving good performance but also retains a high degree of explain ability. The model encodes both continuous and categorical features, alongside their respective names, and feed them into an enhanced Transformer structure enriched with Scaled Exponential Linear Unit activation. Through rigorous experimentation, our model not only outperforms classical algorithms and similar Transformer-based counterparts, but also are comparable to the performance of gradient boosting tree models.