Comparing the algorithmic fidelity of large language models in predicting human decision making: a case study of vaccination choice

This study explores the algorithmic fidelity of five well-established LLM architectures when attempting to replicate complex health decisions like vaccination. The analysis leverages individual survey responses and real-world online news content within prescribed model designs that vary in their relative inputs—namely, basic demographic information, revealed attitudes based on survey data, and personalized media diets. We evaluate the predictive accuracy and sensitivity for each model across all five LLMs, and reveal differing biases inherent to each LLM design. We further explore the role of media exposure on predicted vaccination choice within a counterfactual scenario analysis, by varying the ratio of authoritative to low credibility content. The results indicate that the LLM architectures vary substantially in their prior assumptions of human behavior, as well as their sensitivity to curated media exposure. Specifically, three of the five LLMs illustrate a clear pro-science alignment, as evidenced by their predicted pro-vaccination behavior. These findings carry critical implications for future efforts that strive to build LLMs as reliable tools for modeling individual decisions and population-level health outcomes in complex environments.