Secure yet fragile: adversarial vulnerabilities of federated vision–language models in medical AI

Abstract Vision–Language Models (VLMs) enable powerful multimodal reasoning for medical image analysis, while federated learning allows collaborative training across institutions without sharing patient data. However, the adversarial robustness of federated medical VLMs remains largely unexplored. This work systematically evaluates the vulnerability of CLIP-based VLMs trained with four federated optimization strategies, FedAvg, FedProx, FedPer, and FedBN, on multiple medical datasets. We assess robustness under FGSM, PGD, BIM, and MI-FGSM attacks at varying strengths and show that client-level adversarial perturbations propagate through federated aggregation, causing severe accuracy degradation and high attack success rates, specially under iterative attacks. We further benchmark two training-free test-time defenses, Test-Time Counter-Attack (TTC) and CLIPure, and demonstrate that both mitigate adversarial effects, with CLIPure providing more consistent improvements across datasets and attack intensities. These results highlight fundamental robustness limitations of federated medical VLMs and underscore the need for effective defense mechanisms in distributed clinical deployments.