Enabling Scalable Evaluation of Unfair Patterns in Medical LLMs

The integration of large language models in medical applications (Med LLMs) has raised critical questions about fairness and potential biased behaviors, which require rigorous evaluation. Due to the high complexity, existing work in this domain has primarily relied on manually crafted datasets for bias evaluation. In this study, we present a new method to scale up such bias evaluations by generating evaluation cases based on rigorous medical evidence. We use knowledge graphs and medical ontologies, coupled with a base LLM in a generative pipeline, to generate red-teaming evaluation scenarios. Such scenarios are commonly used to evaluate Med LLMs and often include medical question-answering cases, in which changes in an LLM's response to the same question with only a sensitive attribute flipped (e.g., a patient with vs. without insurance) can indicate unfair behavior. We specifically target the challenges of domain-specific bias characterization (e.g., in prevention vs. diagnosis), hallucination, and the dependencies between health outcomes and sensitive attributes (e.g., gender-dependent diseases). Through extensive experiments, we show that the test cases generated by our proposed method are reliable and can effectively reveal bias patterns in Med LLMs.Additionally, we publish a bias evaluation dataset, providing a comprehensive platform for testing and improving the fairness of medical LLMs, available at https://github.com/healthylaife/autofair .