Bio Capability Probing: A Confound-Controlled Falsification Study of Linear Probes for Biological Dual-Use Capability Detection

Perfect separability under adversarial in-distribution controls is insufficient evidence of robust feature detection under distribution shift. This report demonstrates that claim in the specific case of linear probes for biological dual-use capability detection in language models. Probes trained on N=10 hand-authored capability pairs achieve in-distribution ROC-AUC of 1.0 across all 25 layers of Pythia-410M and survive vocabulary controls, paraphrase attacks, and scaffolded scenarios that preserve adversarial vocabulary overlap. On held-out benchmark evaluation (N=954 items from WMDP-Bio and MMLU biology), AUROC collapses to 0.488 and Recall@1%FPR peaks at 0.008 against a 0.50 operational threshold. A sparsity analysis shows that perfect in-distribution AUROC is achievable with five features out of 1024, a pattern consistent with memorization in high-dimensional space under small-N conditions. The methodological contribution is a reproducible falsification protocol and a low-cost in-distribution diagnostic whose behaviour was consistent with the later observed generalization failure. The substantive implication for interpretability research is that adversarial in-distribution stress tests — paraphrase attacks, scaffolded scenarios, vocabulary controls — do not discriminate between feature detection and memorization at small N, and held-out benchmark evaluation should be treated as a necessary rather than confirmatory step before any operational claim. What's new in v2.1. Calibrated claims throughout (memorization phrased as "consistent with" rather than asserted). Added Phase 9 benchmark composition table and a graphical train/test protocol diagram. Reframed the nine-phase pipeline as an ablation series with explicit methodological core (Phases 6 and 9).