SaudiCulture: A benchmark for evaluating large language models’ cultural competence within Saudi Arabia

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing; however, they often struggle to accurately capture and reflect cultural nuances. This research addresses this challenge by focusing on Saudi Arabia, a country characterized by diverse dialects and rich cultural traditions. We introduce SaudiCulture, a novel benchmark designed to evaluate the cultural competence of LLMs within the distinct geographical and cultural contexts of Saudi Arabia. SaudiCulture is a comprehensive dataset of questions covering five major geographical regions—West, East, South, North, and Center—along with general questions applicable across all regions. The dataset encompasses a broad spectrum of cultural domains, including food, clothing, entertainment, celebrations, and crafts. To ensure a rigorous evaluation, SaudiCulture includes questions of varying complexity, such as open-ended, single-choice, and multiple-choice formats, with some requiring multiple correct answers. Additionally, the dataset distinguishes between common cultural knowledge and specialized regional aspects. We conduct extensive evaluations on six LLMs—GPT-4, Llama 3.3, FANAR, Jais, AceGPT and DeepSeek —analyzing their performance across different question types and cultural contexts. Our findings reveal that all models experience significant performance declines when faced with highly specialized or region-specific questions, particularly those requiring multiple correct responses. Furthermore, we observe that while some regions are better understood by LLMs, others remain largely misrepresented. For instance, GPT-4 achieves the highest accuracy in the western region (66%), whereas Jais records the lowest accuracy in the northern region (16%). Additionally, certain cultural categories are more easily identifiable than others, further highlighting inconsistencies in LLMs’ cultural understanding. These results emphasize the importance of incorporating region-specific knowledge into LLMs training to enhance their cultural competence. We hope that SaudiCulture serves as a foundation for future efforts aimed at improving the ability of LLMs to engage with and accurately represent diverse cultural contexts.