Auditing LLM Social Bias Through Scenario Generation and Search

Public benchmarks are routinely used to evaluate large language models (LLMs), but repeated exposure makes fixed scores harder to interpret. This paper treats benchmark construction as a continuing process and asks whether new test items can be generated automatically from existing seeds. Sixty ambiguous items from the Bias Benchmark for Question Answering (BBQ) with 20 items each from Age, Gender identity and Nationality, were used in two matched generation arms: an LLM scenario-generation arm and a deterministic template-substitution arm. Candidate samples from both arms were required to pass the same ambiguity audit. Accepted items and a BBQ-derived open-ended baseline were then posed to Claude Haiku 4.5 and GPT-OSS-20B, and free-form answers were labelled by an LLM judge. Across three repeated runs (900 candidates per arm), 278 LLM-generated candidates passed the audit (0.3089) against 112 template candidates (0.1244). The accepted LLM-generated items also showed much higher novelty relative to the seeds (mean novelty per run 0.5414–0.5508 against 0.0951–0.1164). On GPT-OSS-20B, the unsupported-selection rate was 0.2720 on LLM-generated items and 0.6235 on the baseline. Claude Haiku 4.5 stayed low on all three sets. Selection-guided search gave inconsistent results across runs. On a blinded, stratified sample relabelled by one annotator, judge–human agreement was 0.9559 (Cohen's κ = 0.8886). The main contribution is a reproducible dynamic-evaluation pipeline integrating scenario generation, validity auditing, target-model testing and answer judging. This evaluation measures unsupported selection rather than directional stereotype bias, and the results show that observed rates depend on both the target model and the scenario distribution.