Answers are grounded in this paper and cite their sources.
Dialect-Marked Response Audit: A Matched-Pair Safety-Support Evaluation of AAVE-Marked Prompt Surfaces. Preprint, version 1.0 (June 2026). Not peer reviewed. Safety evaluation for large language models typically scores a single answer, whether it refuses a harmful request and whether it is correct, yet two users who describe the same safety-critical situation in different language varieties can receive unequally useful help, a failure mode these metrics cannot see. We introduce the Dialect-Marked Response Audit (DMRA), a matched-pair protocol that makes safety-support equivalence a measurable unit of analysis, and characterize where it breaks on Qwen3.5-35B-A3B and a refusal-reduced HauhauCS fine-tune. DMRA pairs 66 deterministic African American Vernacular English (AAVE)-marked and comparison prompts with a 240-row minimal-pair ablation that isolates morphosyntax from lexical, action, and social-address cues, scored by a frame-aware rubric. Top-level safety often converges while support quality, visible-trace cue use, and early routing diverge; the base model's refusals compress these differences into uniformly safe answers, so a refusal-reduced fine-tune is used as a stress test, and the register-conditioned continuation it reveals tracks syntax/register and action/weapon lexis rather than the isolated in-group address term. These single-run, single-rater results are descriptive, and argue that safety evaluation should test support equivalence across matched user surfaces, not refusal alone, and that broad “dialect marking makes models unsafe” claims should give way to driver-specific, model-dependent ones. This deposit contains the paper (LaTeX source and built PDF) and a review change log. Sanitized data artifacts are released separately in the dmra-public-artifacts repository; raw safety-test generations are retained in a controlled archive for verification.
No takes yet. Share an insight, caveat, or question.
Jeffrey W. Shorthill (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: