Design of augmented reality educational content based on a multi-modal approach for holistic development in early childhood

This paper develops interactive educational content for music-based learning utilizing Augmented Reality (AR). Designed for toddlers and young children, the content aims to foster holistic development through multi-modal stimuli-including visual, auditory, and haptic elements-and the coordination of both fine and gross motor skills. The content employs a rhythm game style and consists of both software and hardware components. Within the software, an AR environment is constructed using Unity to simulate visual and auditory information. Various musical instruments in the AR environment enable user interaction based on rhythm notes generated according to the corresponding musical lyrics. The hardware includes AR glasses, a haptic stick, and a sensing pad. Visual and auditory feedback is relayed to the user through the AR glasses, while the haptic stick allows for striking the sensing pad. Simultaneously, the haptic stick appropriate proper tactile feedback, considering both the specific musical instrument being interacted with and the force of the strike, as measured by an integrated pressure sensor in the sensing pad. We expect that the proposed educational content will contribute to the holistic development of toddlers and young children by offering a more interactive and multi-sensory learning experience.