What question did this study set out to answer?

This study aims to investigate whether rejection framing increases the fabrication of sources in large language models.

June 2, 2026Open Access

Quality Challenges Do Not Escalate Source Fabrication in Large Language Models: A Pre-Registered Cross-Platform Factorial Study

Key Points

This study aims to investigate whether rejection framing increases the fabrication of sources in large language models.
Conducted a pre-registered 2×5×7 mixed factorial study involving two LLM platforms.
Utilized fictional entities as probes to measure the fabricated source rate (FSR).
Implemented a two-tier validation protocol to assess URL non-existence and content misattribution.
No significant amplification of FSR was found with rejection framing (ChatGPT: OR=1.00, p=0.998; Perplexity: OR=1.19, p=0.822).
Uncertainty-preserving and gap-anchoring framings suppressed fabrication at specific conversation depths (0/240 positive events for UPR).
Platform architecture significantly influenced FSR, with ChatGPT averaging FSR=0.075 and Perplexity FSR=0.008.

Abstract

Large language models (LLMs) frequently fabricate non-existent URLs and citations. A prominent hypothesis holds that rejection framing — prompts expressing dissatisfaction and demanding better sources — amplifies this behaviour by increasing the pressure on the model to produce specific references (schema activation lock hypothesis). We conducted a pre-registered 2×5×7 mixed factorial study across two LLM platforms (ChatGPT gpt-4o; Perplexity sonar-pro) and five rejection-framing arms (BASE, SJR, UPR, NSAF, ECC), with seven conversation depths (k=0–6). Twenty fictional entities per platform (1,400 observations total) were used as probes. The primary outcome was the Fabricated Source Rate (FSR): fabricated URLs divided by total URLs cited. A two-tier validation protocol distinguished URL non-existence (HTTP Tier 1) from content misattribution (TF-IDF Tier 2, 30% stratified sample). The primary hypothesis (H1: SJR rejection framing amplifies FSR relative to BASE, OR≥1.5) was not supported on either platform (ChatGPT: OR=1.00, p=0.998; Perplexity: OR=1.19, p=0.822). Uncertainty-preserving framing (UPR) produced complete suppression of fabrication at k≥1 on both platforms (0/240 positive events), as did gap-anchoring framing (NSAF) on Perplexity (0/120 events). Platform architecture was the dominant determinant of FSR: ChatGPT averaged FSR=0.075, Perplexity FSR=0.008 (9.2× difference). Content-level validation revealed that Perplexity’s near-zero HTTP fabrication rate masks near-universal content misattribution (95.5% of HTTP-validated URLs pointed to topically related but entity-unspecific pages). The expansion framing arm (ECC) showed a significant platform-differential interaction: amplifier on ChatGPT, suppressor on Perplexity (OR=0.157, p=0.042). Quality challenge framing does not amplify URL fabrication; rather, the unrestricted baseline request produces maximal fabrication pressure. Uncertainty-preserving and gap-acknowledging framings provide robust, architecture-independent protection. Parametric and retrieval-grounded systems exhibit distinct failure modes that require different mitigation strategies. Pre-registration: OSF https://osf.io/3es5f/ (v0.4) Code & Data: https://github.com/glasseymour/rice-p1

Ask AI

Helpful

Bookmark

View Full Paper