A Safety-aware Jailbreak Approach to Bypass Security Filters for Generating Harmful Media with Text-to-Image models

Text-to-image models (T2I) have demonstrated remarkable capabilities in generating realistic and high-quality images, finding applications in fields ranging from entertainment to social networks. However, the ease of access to these tools has raised concerns regarding their potential misuse from ethical and legal perspectives due to the possibility of creating disinformative, manipulative, or offensive content. In this paper, we investigate the effectiveness of the security systems of emerging T2I models to avoid generating harmful media using textual prompts. To this end, we introduce a new jailbreaking technique combining Projected Gradient Descent (PGD) attack with a safety-aware prompt optimization specifically tailored to bypass safety filters in advanced T2I models. Moreover, we propose a new defense mechanism inspired by the splitting of transformers’ attention layers of CLIP, leading to the definition of two classifiers for detecting unsafe and adversarial prompts. The evaluation of the proposed jailbreak approach highlighted the vulnerabilities of the most recent proprietary models, such as DALL·E, IMAGEN, and Nano Banana 2, allowing us to generate thousands of unsafe and unethical images related to seven harmful categories, i.e., harassment, hate, sexual, shocking, violence, self-harm, and illegal activities. Moreover, we have demonstrated that our approach outperforms most popular jailbreaking approaches in bypassing security filters and generating harmful images with an overall performance increase ranging from 2 to 8 times. The results of the proposed defense mechanism have shown that our mechanism is very effective in discriminating unsafe and adversarial prompts, outperforming the detection performance of the most popular commercial safety filters by up to 4 times, even against other types of adversarial prompts. The images and prompts were collected into three new datasets made available for benchmarking safety filters and analyzing the risks associated with T2I generation. Finally, we evaluate the performance of an ad-hoc classifier in detecting and identifying harmful content, achieving similar results to GPT-4o and Gemini 1.5 Pro in detection and enhanced performance in identification.