Context Curation Is Not a Free Lunch: An Empirical Test of Inference-Time Robustness for Tabular Foundation Models Under Distribution Shift

Tabular foundation models such as TabPFN introduced a different deployment paradigm: instead of fitting parameters to a training set, the model performs in-context learning, conditioning a pretrained transformer on a labeled context set at inference time, with no gradient-based training. This training-free deployment is the paradigm's central advantage, but it is threatened by distribution shift: deployment drift degrades predictive performance, and existing remedies retrain the pretraining prior, reintroducing the cost the paradigm was meant to avoid. Because the context set is already a free parameter at inference time, we ask a question that recent concurrent work has approached only indirectly: can selecting which training rows populate the context, using only unlabeled target features, substitute for retraining as a lightweight defense against distribution shift? Unlike concurrent context-selection work motivated primarily by inference efficiency (CRUMB) or cold-start, low-label performance (LUCoS), we treat context curation as a shift-robustness mechanism in its own right and test it directly against a matched retraining baseline across multiple shift types, severities, and a real-world drift benchmark—a comparison neither concurrent work performs. We evaluated five context-selection strategies against TabPFN on five public tabular datasets under three constructed shift types and four severities, with three random seeds and three context sizes, yielding 540 matched evaluation observations for the primary context-selection comparisons, plus one real-world concept-drift benchmark. Contrary to our hypothesis, the best-performing strategy, density-reweighted context selection, did not reliably outperform random selection (p = 0.068), did not outperform an equivalently-informed retraining baseline, performed significantly worse than a tuned retraining baseline (p = 0.008), and underperformed random selection on the real-world drift benchmark. A shift-detectability diagnostic and six candidate moderators showed little reliable relationship with the method's benefit. We did, however, find a robust negative result: nearest-to-target and class-balanced context selection significantly underperformed random selection across the evaluation grid, with paired mean accuracy differences of −3.67 and −3.31 percentage points, respectively. We report this as a robust cautionary result: target-awareness in context selection is not sufficient to improve robustness, and a plausible heuristic can actively harm performance. We release the full experimental pipeline for reproducibility.