The investigation of robustness differences for variously-sized models in adversarial attacks and defense

Adversarial attacks and defenses are central in deep learning, with various attack methods and defense strategies, including adversarial training, proposed over the years. However, limited research has examined the differences in robustness across models of different sizes. This study seeks to explore these robustness variations through the application of multiple attack methods and attention visualization techniques on four prominent models: VGG16, ResNet18, GoogleNet, and Vision Transformers, employing four popular adversarial attack methods—Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and Carlini-Wagner (CW). Plain adversarial training was used as a defense mechanism. By comparing the resulting changes and discrepancies in correctness, a notable decrease is observed in the robustness of larger models compared to smaller ones after applying this defense strategy. This phenomenon is likely associated with the distinct feature extraction approaches employed by the larger model and its reduced training efficiency. From a practical standpoint, it is advisable to prioritize the use of smaller models in real-world applications. Additionally, techniques like knowledge distillation can be considered to enhance the correctness of smaller models while minimizing computational resource requirements.