Research Questions: (1) Is there a pattern of racial bias in student advising recommendations made by generative AI? (2) What safeguards can promote equity when using generative AI in high-stakes decision-making? Methodology: Using lists of names associated with various ethnic/racial groups, we asked ChatGPT and Claude AI for recommendations for colleges and majors for each student. Results: ChatGPT was more likely to recommend STEM majors to some student groups. ChatGPT did not show systematic bias in various metrics of school quality, but Claude AI did. There were also overall differences in the colleges recommended by Claude AI and ChatGPT. Implications: We provide cautions and recommendations for using generative AI in high-stakes tasks.
No takes yet. Share an insight, caveat, or question.
Julie Smith (2024) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: