What question did this study set out to answer?

This research aims to explore the effects of axiomatic prompting on the classification performance of language models.

April 6, 20260 citationsOpen Access

The 70% Rule: When Axiomatic Prompting Helps, and When It Hurts

Key Points

This research aims to explore the effects of axiomatic prompting on the classification performance of language models.
Utilized six classification tasks across eight models from five providers.
Evaluated zero-shot performance to determine the effectiveness of axiomatic prompting.
Conducted ablation studies to assess the impact of axiom styles (pattern-based vs. conceptual).
Axiomatic prompting improved accuracy by 15-24% when zero-shot performance was below 70%.
Axiomatic prompting decreased performance by 5-20% when zero-shot performance was above 70%.
The 70% Rule accurately predicted performance changes in all experiments.

Abstract

We introduce axiomatic prompting, a technique that augments LLM classification prompts with explicit decision rules of the form pattern →category. Across six classification tasks and eight models spanning five providers (OpenAI, Anthropic, Google, Meta, Alibaba), we find a consistent threshold effect: axiomatic prompting significantly improves accuracy when zero-shot performance falls below 70% (gains of +15–24%, p < 0.05), but significantly hurts performance above this threshold (losses of -5 to -20%, p < 0.05). We term this the 70% Rule, a simple heuristic that predicted the direction of axiomatic prompting’s effect with 100% accuracy in our experiments. Ablation studies reveal that axiom style matters: pattern-based rules derived from data outperform conceptual definitions. Our findings suggest practitioners should evaluate zero-shot baselines before investing in rule-based prompt engineering. Code and data available at https://doi.org/10.17605/OSF.IO/PCX2D.

问 AI

Bookmark

View Full Paper