Multi-modal personalized federated learning with adaptive differential privacy for medical image classification and a privacy-preserving approach

Deep learning on medical images classification intervention needs to use large data on multi-institutional datasets but privacy laws inhibit sharing of data (GDPR, HIPAA). Federated Learning (FL) facilitates collaborative training without data transfer; until now, the known methods can only address privacy, personalisation, and accuracy not at the same time in a multi-modal environment. We present MM-PFL-ADP, a framework that combines Vision Transformer (ViT) based multi-modal feature extraction in four new elements: (i) privacy budget allocation (independent of number of samples): Fisher information-based adaptive per-parameter privacy budget allocation (ε local / ε shared = 1.5); (ii) personalisation masks: dynamic KL divergence based personalisation masks; (iii) respect The framework gives formal client-level (ε , δ )-DP guarantees on transmitted gradient updates, in $$K = 10$$ simulated medical institutions. On the MRI-MS dataset, MM-PFL-ADP achieves $$97.3%$$ accuracy (95% CI: 96.9–$$97.7%$$) at ε = 1.5, outperforming FedAvg ($$92.1%$$) and DP-FedAvg ($$87.3%$$) by large margins ($$p < 0.001$$). The framework is $$45%$$ faster than FedAvg (47 vs. 85 rounds), has $$47%$$ less total communication and keeps $$95.2%$$ accuracy in case of extreme heterogeneity in data (α = 0.1). The probability of membership inference attack has decreased to 52.1 which was close to the random baseline ($$50%$$). MM-PFL-ADP shows that the concepts of privacy, personalisation, and accuracy are synergistic, but not oppositional to federated medical AI. The single-system Fisher information framework greatly simplifies the hyperparameter tuning problem and can meet formal privacy criteria. Before being deployed, prospective validation against the performance of expert radiologists is desired.