Multi-modal personalized federated learning with adaptive differential privacy for medical image classification and a privacy-preserving approach

Deep learning on medical images classification intervention needs to use large data on multi-institutional datasets but privacy laws inhibit sharing of data (GDPR, HIPAA). Federated Learning (FL) facilitates collaborative training without data transfer; until now, the known methods can only address privacy, personalisation, and accuracy not at the same time in a multi-modal environment. We present MM-PFL-ADP, a framework that combines Vision Transformer (ViT) based multi-modal feature extraction in four new elements: (i) privacy budget allocation (independent of number of samples): Fisher information-based adaptive per-parameter privacy budget allocation (₋₎₂₀₋ / ₒ₇₀ₑ₄₃ = 1. 5) ; (ii) personalisation masks: dynamic KL divergence based personalisation masks; (iii) respect The framework gives formal client-level (, ) -DP guarantees on transmitted gradient updates, in K = 10 simulated medical institutions. On the MRI-MS dataset, MM-PFL-ADP achieves 97. 3\% accuracy (95% CI: 96. 9–97. 7\%) at = 1. 5, outperforming FedAvg (92. 1\%) and DP-FedAvg (87. 3\%) by large margins (p < 0. 001). The framework is 45\% faster than FedAvg (47 vs. 85 rounds), has 47\% less total communication and keeps 95. 2\% accuracy in case of extreme heterogeneity in data (= 0. 1). The probability of membership inference attack has decreased to 52. 1 which was close to the random baseline (50\%). MM-PFL-ADP shows that the concepts of privacy, personalisation, and accuracy are synergistic, but not oppositional to federated medical AI. The single-system Fisher information framework greatly simplifies the hyperparameter tuning problem and can meet formal privacy criteria. Before being deployed, prospective validation against the performance of expert radiologists is desired.