Harmful Content Generation in Text-to-Image Models: Capabilities and Moderation Limitations

Text-to-image generative models have achieved remarkable capabilities in producing photorealistic images, but raise concerns about misuse for generating harmful content. While safety mechanisms exist, systematic evaluations of their effectiveness against realistic attacks remain limited. This paper presents a comprehensive assessment of harmful content generation across modern text-to-image models using an automated pipeline that transforms legitimate news captions into unsafe prompts targeting sexually explicit content, violent imagery, harmful stereotypes, self-harm, and hate speech. We evaluate five open text-to-image models, including standard models with built-in safety mechanisms and community fine-tuned variants that bypass content restrictions. Our human evaluation of 1,500 generated images reveals that models produce harmful content at concerning rates: 89.2% for gore-related prompts, 47.6% for sexually explicit content, and 43.6% for harmful stereotypes. Hate speech prompts elicit unsafe content in 46.0% of cases, while self-harm prompts yield a lower rate (34.5%), predominantly through graphic violence. All evaluated models demonstrated substantial capabilities for generating violent and stereotypical content, while community fine-tuned models showed particular vulnerability to sexually explicit prompts. Quality assessment shows that models largely maintain generation capabilities for harmful versus benign content, producing synthetic imagery of sufficient fidelity to pose risks for disinformation and abuse. Human evaluation further confirms that unsafe prompting does not severely impact image plausibility and photorealism, with more capable models such as FLUX.1-dev producing clearly realistic harmful images in 30.9% of cases. Analysis of automated moderation systems reveals critical limitations, allowing substantial harmful content to evade detection. Our evaluation of synthetic image detection algorithms reveals that models trained solely on benign datasets perform worse on explicit content, while those incorporating diverse training data improve detection, highlighting semantic distribution gaps in current approaches. These findings highlight gaps between current safety measures and detection capabilities, demonstrating the need for more robust approaches to prevent misuse of text-to-image generation systems at scale. Content Warning: This paper contains instances of sexual content, violence, and hateful material that some readers may find offensive or disturbing. All content is presented for academic research purposes to evaluate safety risks in AI image generation systems.