A Regularization Method for Reducing Cross-Modal Blind Spots in Clinical CLIP Models

____v2: a major revision of v1_____ AbstractMultimodal clinical models built on contrastive learning can treat semantically distinct radiology report texts as inter-changeable in their shared embedding space, creating blind spots invisible to standard retrieval and classification metrics. Clinically distinct phrases such as a left versus a right lung opacity may receive near-identical similarity scores. We show that existing clinical vision–language models exhibit measurable counterfactual sensitivity blind spots on real radiology reports, and that these blind spots can be systematically reduced through targeted regularisation. We probe the shared space with controlled counterfactual probes, which are template-generated text edits that flip a single clinically consequential attribute (laterality, negation, severity, anatomical direction), and treat them as a representation-level stress test rather than as estimates of naturally occurring clinical error prevalence. Extending the symmetry-matching framework from algebraic error-detection theory to cross-modal encoders, we define a cross-modal stabiliser and a Gini-based vulnerability index on the cosine-similarity level. Across two medical contrastive vision–language models (BiomedCLIP and PubMedCLIP) and 3,783 probe pairs from 1,000 chest radiograph reports, every family exhibits substantial baseline vulnerability of 0.46 to 0.64. A similarity-level regularisation applied solely to the frozen-encoder projection head reduces the vulnerability index by 34 to 84 percent on real reports and by up to 95 percent on synthetic probes, and increases counterfactual-detection area under the curve from 0.50–0.59 to 0.51–0.73 on the first model and from 0.49–0.56 to 0.52–0.60 on the second, without retraining either base encoder. A singular value decomposition of per-family sensitivitymatrices yields a closed-form orientation diagnostic. Image-space angles of 2.4 and 0.9 degrees between the negation and severity families reveal an interference criterion that helps explain why a single projection head does not optimise both families simultaneously, which a sequential two-head factorisation overcomes. A pre-specified held-out analysis delimits the result, with the magnitude axis of the gain generalising out-of-sample whereas directional discrimination does not. The resulting V -profile is a counterfactual sensitivity measure and a candidate component of a broader safety evaluation, not a standalone safety guarantee. The single-head intervention adds one loss term while keeping both base encoders frozen; the factorised extension adds a second lightweight projection head when inter-family interference is detected. The reproducibility package is available with the manuscript.