When simpler models win: A large-scale computational benchmark of lexical and transformer NLP pipelines for predicting medication effectiveness

Patient-generated drug reviews provide a scalable source of real-world evidence on perceived medication effectiveness, yet robust and reproducible modelling benchmarks remain limited. This study presents a quantitative evaluation of seven natural language processing pipelines for predicting drug-level effectiveness ratings from patient reviews on a 1-10 scale. Reviews were aggregated at the medication level by concatenating texts per drug and constructing a stabilised drug-level target using Bayesian shrinkage (α = 10.0). Models spanning sparse lexical baselines, dense embedding pipelines, frozen biomedical transformers, and a compute-aware fine-tuned transformer baseline (that is, one tuned under an explicitly bounded compute budget) were assessed using Repeated K-Fold cross-validation on Dataset A (5 folds × 2 repeats; out-of-fold prediction) and external validation on four independent datasets (A→B-E). Performance was measured using RMSE and MAE, with R², Pearson correlation, Spearman correlation, and calibration slopes, intercepts, and plots reported to characterise association and reliability under dataset shift. Across internal validation and most external evaluations, TF-IDF with Ridge regression achievedTF-IDF with linear support vector regression (SVR) achieved the strongest overall accuracy (Dataset A internal RMSE = 0.336; R² = 0.813) and generalised well, particularly on Dataset and generalised consistently, achieving the lowest error on every external corpus, including Dataset B (RMSE = 0.438), the only external dataset sharing no reviews with the training data. External results are reported after removing reviews duplicated from the training corpus, which affected 97.8 per cent of Dataset C, 98.9 per cent of Dataset D and 28.7 per cent of Dataset E. Transformer-based approaches, including SBERT and frozen biomedical encoders, did not outperform lexical baselines, and a transparently tuned ClinicalBERT regression baseline using a learning-rate sweep {1e-5, 2e-5, 3e-5} with early stopping did not yield consistent gains under external shift. The best lexical baseline achieved consistently lower RMSE than the best transformer pipeline in every one of the five evaluation settings, with a median RMSE gap of 0.19 rating points. A paired non-parametric test is reported alongside this descriptive pattern (Wilcoxon signed-rank, p = 0.031), but with only five paired settings this is the minimum attainable p-value and it is therefore treated as a caveat-bearing supplement rather than as confirmatory evidence. These findings indicate that, under drug-level aggregation and compute-aware tuning, frequency-based lexical signals capture much of the predictive structure in patient reviews. The benchmark provides a reproducible framework for evaluating drug-review-based effectiveness prediction with internal and external validation and reliability diagnostics relevant to pharmacovigilance and digital health analytics.