Evaluates bias in language models, indicating a new framework for analysis.
Fairness evaluation in natural language processing is commonly organized around realized outputs: stereotype preference, answer accuracy, disparity, toxicity, sentiment, or related endpoint statistics. These measures are necessary, but they do not by themselves characterize how a system reorganizes its possible responses when ethically relevant evidence changes. We develop Ethical Experience Architecture (E-EA), a domain-specific operationalization of Experience Architecture for bias and fairness evaluation in language models. E-EA preserves six mutually constrained conditions—Difference, Availability, Orientation, Integration, Temporality, and Answerability—and treats Answerability counterfactually as alteration of a declared reachable response field. Experiment 1 evaluates the complete public BBQ release (58,492 items; 29,246 ambiguous/disambiguated pairs) using published predictions from six model/configuration conditions. Absolute ambiguous-context BBQ bias and the E-EA completeness proxy are nearly unassociated across 66 model-category cells (Spearman ρ = −0.053, p = .671; Pearson r = −0.138, p = .271), showing that the two summaries are empirically non-equivalent under this operationalization. UnifiedQA-11B in RACE format has the highest retrospective aggregate proxy, 0.769 (95% category-bootstrap CI [0.732, 0.799]). Experiment 2 applies the same declared architecture to Mistral Small 4 v26.03 on a frozen stratified subset of 1,000 BBQ pairs with three repeated generations and genuine two-turn evidence updates. The run returned 2,985 of 3,000 planned pair-replicate records (99.5%); all returned records were parseable, and the 15 omissions were provider-side rate-limit/service failures. Available-case analysis yields an E-EA proxy of 0.747 (95% CI [0.710, 0.783]), and a complete-replicate sensitivity analysis on 988 pairs yields 0.747 [0.710, 0.783]. These quantities are behavioral structural proxies, not moral scores or the canonical Experience Architecture Completeness Index. The contribution is a reproducible measurement language for testing how language-model behavior changes under controlled, normatively declared interventions.
No takes yet. Share an insight, caveat, or question.
Md. Amir Khusru Akhtar (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: