A Mobile Augmented Reality Integrating KCHDM-Based Ontologies with LLMs for Adaptive Q&A and Knowledge Testing in Urban Heritage

A cultural heritage augmented reality system overlays virtual information onto real-world heritage sites, enabling intuitive exploration and interpretation with spatial and temporal contexts. This study presents the design and implementation of a cognitive Mobile Augmented Reality (MAR) system that integrates KCHDM-based ontologies with large language models (LLMs) to facilitate intelligent exploration of urban heritage. While conventional AR guides often rely on static data, our system introduces a Semantic Retrieval-Augmented Generation (RAG) pipeline anchored in a structured knowledge base modeled after the Korean Cultural Heritage Data Model (KCHDM). This architecture enables the LLM to perform dynamic contextual reasoning, transforming heritage data into adaptive question-answering (Q&A) and interactive knowledge-testing quizzes that are precisely grounded in both historical and spatial contexts. The system supports on-site AR exploration and map-based remote exploration to ensure robust usability and precise spatial alignment of virtual content. To deliver a rich, multisensory experience, the system provides multimodal outputs, integrating text, images, models, and audio narration. Furthermore, the integration of a knowledge sharing repository allows users to review and learn from others’ inquires. This ontology-driven LLM-integrated MAR design enhances semantic accuracy and contextual relevance, demonstrating the potential of MAR for socially enriched urban heritage experiences.