What question did this study set out to answer?

This research aims to document and analyze a specific refusal pattern in the Claude Opus 4.7 safety classifier.

May 27, 2026Open Access

A Single-Record, Content-Specific Refusal Localised to Opus 4.7: Seven-Judge Falsification on a CTI-Style Prose Sample Falsely Attributing US-CERT Infrastructure to Attackers (safety-note artefacts)

Key Points

This research aims to document and analyze a specific refusal pattern in the Claude Opus 4.7 safety classifier.
Analyzed the performance of Claude Opus 4.7 using a seven-judge cross-vendor panel on a CTI corpus.
Conducted a stochasticity test on multiple evaluation datasets (N=90 and N=194).
Utilized a targeted guardrail-characterization probe (N=44) to explore failure triggers in the safety classifier.
Claude Opus 4.7 refused to score one specific student-output record, incorrectly attributing attacker-controlled channels to US-CERT infrastructure (3.3% refusal on N=90, 1.5% on N=194).
Seven judges from various models engaged with the record, but the refusal was unique to Opus 4.7.
A prompt-swap test revealed a second trigger mode for refusals, indicating complexity in guardrail mechanisms.

Abstract

This deposit accompanies the standalone preprint "A Single-Record, Content-Specific Refusal Localised to Opus 4. 7", a short safety-research note that documents and falsifies a frontier-LLM refusal pattern using a seven-judge cross-vendor panel. Result: Claude Opus 4. 7's safety classifier, called via the structured-output messages. parse () API to rate stylistic fidelity on a public-CTI corpus, refuses to score one specific student-output record (chunkᵢdx=2 ttpₛummary) on both the 2026-05-17 N=90 eval and the 2026-05-24 N=194 wider eval, and reproduces in a 5 / 5 stochasticity test on 2026-05-25. The refused record falsely attributes attacker-controlled distribution channels to real US-CERT / CISA infrastructure. Seven other judges — Claude Sonnet 4. 6, Claude Haiku 4. 5, Gemma 4 E4B, foundation-sec: latest, qwen2. 5: 7b, gpt-oss-120b, and Llama-4-Scout-17B-16E — all engage with the record. The refusal is localised to Opus 4. 7 specifically within the Anthropic family. This is v5 of the note. v5 adds an N=44 targeted guardrail-characterisation probe (data in guardrailₚrobe/) that falsifies the v4-era hypothesis that the guardrail keys on "false attribution of US-CERT / CISA infrastructure" at the content-class level: 24 synthetic records varying defensive-infrastructure entity attribution across US-gov, non-US-gov, and commercial-vendor buckets all engaged under Opus. A prompt-swap test surfaced a second refusal trigger that is prompt-context-conditioned rather than student-response-content-driven. The guardrail therefore has at least two distinct trigger modes that the experiment cannot fully characterise. v4 (2026-05-24) corrected an earlier v3 claim of 53 % refusal rate that had been arithmetically inflated by 15 upstream API credit-balance errors mistaken for refusals; the corrected rate is 1 / 30 = 3. 3 % on N=90 and 1 / 67 = 1. 5 % on N=194. Reuse: CC-BY-4. 0 over the entire deposit. The companion Paper 2 deposit at 10. 5281/zenodo. 20369167 (v2) carries the parallel correction history and the wider eval data this note's correction relies on.

Read Full Paperexternally

AI से पूछें

Bookmark

View Full Paper