Answers are grounded in this paper and cite their sources.
Empirical study demonstrates that subsampling unlabeled data maintains downstream accuracy in tabular self-supervised learning, indicating substantial computational savings are achievable.
Self-supervised learning (SSL) addresses label scarcity by leveraging large amounts of unlabeled data to learn transferable representations. However, pretraining on very large unlabeled datasets can be computationally expensive and may include noisy or unrepresentative samples that degrade learning. In this work, we explore whether selecting a subset of the available unlabeled examples for the pretext task can reduce computational cost while maintaining strong performance of SSL methods for tabular data. In particular, we investigate whether uncertainty-, diversity-, and transport-based criteria can guide this selection and improve representation quality. To this end, we conduct large-scale experiments on 25 tabular benchmark datasets using four SSL models, across varying amounts of labeled data under both biased and unbiased label selection, and multiple strategies for sampling unlabeled data. To better understand when and how unlabeled data subsampling is effective, we relate dataset characteristics to experimentally observed performance gains through a meta-analysis. Our results show that reducing the unlabeled data pool yields substantial computational savings while maintaining, and often even improving, downstream performance.
No takes yet. Share an insight, caveat, or question.
Stevanoska et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: