What question did this study set out to answer?

The study aims to address positional attention bias in language models to improve fact recall.

April 7, 2026Open Access

Positional Restructuring of System Prompts: Mitigating Transformer Attention Bias to Enable Ground Truth Compliance in Sub-Frontier Language Models

Key Points

The study aims to address positional attention bias in language models to improve fact recall.
Systematic evaluation across five models from two model families
Restructuring system prompts to emphasize critical facts
Implementation of ground truth engineering methodology
Development of an automatic correction persistence pipeline
Model score improved from 5.7/10 to 7.5/10 with prompt restructuring
A 32B model's score increased from 6.2/10 to 9.4/10 through ground truth engineering
Improvements achieved without modifying model weights

Abstract

This paper presents evidence that the primary barrier to reliable fact recall in sub-frontier language models deployed with system prompts is positional attention bias — the lost-in-the-middle phenomenon. In a systematic evaluation across five models and two model families, we discovered that restructuring the system prompt to place critical facts at the beginning and end raises a 14-billion parameter model's score from 5. 7/10 to 7. 5/10 on a seven-dimension verification battery, without any modification to model weights. The paper further describes Ground Truth Engineering, a systematic behavioural rule methodology that raised a 32B model from 6. 2/10 to 9. 4/10 with cold-restart persistence, and an automatic correction persistence pipeline. These findings enable sovereign AI deployment on consumer hardware costing under 500.

Read Full Paperexternally

KI fragen

Bookmark

View Full Paper