What type of study is this?

This is a Quantitative Study study.

October 22, 2025Open Access

Zero-Shot Classification of Illicit Dark Web Content with Commercial LLMs: A Comparative Study on Accuracy, Human Consistency, and Inter-Model Agreement

VPVíctor-Pablo Prado-SánchezUniversidad de Alcalá ADAdrián Domínguez‐DíazUniversidad de Alcalá LDLuis de‐MarcosUniversidad de Alcalá

Key Points

Zero-shot classification demonstrated impressive performance, with DeepSeek Chat achieving a leading macro-F1 score of 0.870.
Cohen's Kappa scores above 0.840 indicate high alignment with human annotations across the top models.
Inter-model agreement peaked between Claude 3.5 Haiku and GPT-4o, achieving a Kappa score of 0.911.
The study confirms that state-of-the-art large language models can classify illicit Dark Web content under zero-shot conditions.

Abstract

This study evaluates the zero-shot classification performance of eight commercial large language models (LLMs), GPT-4o, GPT-4o Mini, GPT-3.5 Turbo, Claude 3.5 Haiku, Gemini 2.0 Flash, DeepSeek Chat, DeepSeek Reasoner, and Grok, using the CoDA dataset (n = 10,000 Dark Web documents). Results show strong macro-F1 scores across models, led by DeepSeek Chat (0.870), Grok (0.868), and Gemini 2.0 Flash (0.861). Alignment with human annotations was high, with Cohen’s Kappa above 0.840 for top models and Krippendorff’s Alpha reaching 0.871. Inter-model consistency was highest between Claude 3.5 Haiku and GPT-4o (κ = 0.911), followed by DeepSeek Chat and Grok (κ = 0.909), and Claude 3.5 Haiku with Gemini 2.0 Flash (κ = 0.907). These findings confirm that state-of-the-art LLMs can reliably classify illicit content under zero-shot conditions, though performance varies by model and category.

Ask AI

Helpful

Bookmark

View Full Paper