Implicit bias in large language models: Experimental proof and implications for education

We provide experimental evidence of implicit racial bias in a large language model (specifically ChatGPT 3.5) in the context of an educational task and discuss implications for the use of these tools in educational contexts. Specifically, we presented ChatGPT with identical student writing passages alongside various descriptions of student demographics, including race, socioeconomic status, and school type. Results indicate that when directly prompted to consider race, the model produced higher overall scores than responses to a control prompt, but scores given to student descriptors of Black and White were not significantly different. However, this result belied a subtler form of prejudice that was statistically significant when racial indicators were implied rather than explicitly stated. Additionally, our investigation uncovered subtle sequence effects that suggest the model is more likely to illustrate bias when variables change within a single chat. The evidence indicates that despite the implementation of guardrails by developers, biases are profoundly embedded in ChatGPT, reflective of both the training data and societal biases at large. While overt biases can be addressed to some extent, the more ingrained implicit biases present a greater challenge for the application of these technologies in education. It is critical to develop an understanding of the bias embedded in these models and how this bias presents itself in educational contexts before using LLMs to develop personalized learning tools.