What question did this study set out to answer?

The aim is to evaluate and categorize various types of adversarial failures in large language models.

May 29, 20260 citationsOpen Access

The Fracture.ai Benchmark Suite: A Taxonomy-Driven Framework for Adversarial Failure Detection in Large Language Models --- A Single-Model Calibration Study

Key Points

The aim is to evaluate and categorize various types of adversarial failures in large language models.
Developed the Fracture.ai Benchmark Suite with 12 mechanistically distinct failure types.
Constructed a 300-probe human-annotated evaluation instrument.
Utilized a hybrid automated judge combining heuristics with LLM for semantic verification.
Achieved an aggregate F1 score of 0.62 on llama-3.3-70b-versatile.
Performance varied by type, from F1 = 1.00 for False Expertise to F1 = 0.00 for OOD Hallucination.
All results are reported transparently.

Abstract

This preprint presents the Fracture.ai Benchmark Suite — a taxonomy-driven adversarial evaluation framework for Large Language Models. We define 12 mechanistically distinct failure types spanning hallucination, sycophancy, authority capitulation, confidence miscalibration, and instruction drift. A 300-probe human-annotated evaluation instrument is constructed and benchmarked against a hybrid automated judge combining deterministic heuristics with LLM-as-a-Judge semantic verification. The judge achieves F1 = 0.62 aggregate on llama-3.3-70b-versatile, with per-type performance ranging from F1 = 1.00 (FT-010 False Expertise) to F1 = 0.00 (FT-006 OOD Hallucination). All results reported transparently including failures.

问 AI

Bookmark

View Full Paper