Current gaps in downstream applications of fMRI foundation models: lessons learned from BrainLM & BrainHarmonix on Prevent-AD

Foundation models have driven major advancements in the language and vision domains, as seen in the popularity of Artificial Intelligence (AI) tools, and a growing number of studies are bringing the approach to human brain imaging. Whether these models are ready for independent researchers to use on clinical datasets has yet to be examined. We evaluated two recent fMRI foundation models, BrainLM and BrainHarmonix, on PreventAD, a dataset on adults with a family history of Alzheimer's disease. We compared features from frozen and fine-tuned models against two baselines: functional connectivity and a dummy classifier that always predicts the most common class. The sample includes 343 subjects who passed quality control, with a subset of 218 who had amyloid PET imaging. We tested four prediction targets: sex, age, progression to mild cognitive impairment (MCI), and β-amyloid burden. Both models required substantial changes to the released code. BrainLM's own data preparation workflow did not produce stable fine-tuning, which we achieved only after switching to a standard workflow implemented with the library Nilearn. BrainHarmonix's documentation did not specify how to normalize new data, so we resolved this by testing different alternatives. For sex, functional connectivity outperformed all foundation model features. For age group, the best foundation model feature, taken from BrainHarmonix's structural (T1-weighted) encoder, slightly exceeded functional connectivity, but the difference was not significant. For MCI and β-amyloid, no feature exceeded the dummy classifier in accuracy, and the area under the curve (AUC) stayed near chance for all features. For both clinical targets, the most precise feature came from fine-tuned BrainLM, meaning a larger share of its positive predictions were correct than for functional connectivity, but neither difference reached statistical significance. These results suggested that the strong transfer to new tasks seen in large language models cannot yet be assumed for fMRI foundation models, and that both models required code adaptation that would preclude their streamlined adoption by independent groups. We recommend that future studies provide standardized data workflows and always report functional connectivity and dummy classifier baselines. For clinical targets with strong class imbalance, studies should report several metrics, including precision.