Cognitive neuroscience-inspired multimodal perceptual fusion framework for enhanced mixed reality environments: a deep learning approach

Mixed reality (MR) systems have to weave several sensory streams, vision, audio, touch and spatial cues, into a single coherent percept, and they have to do it quickly. We present a hybrid framework that pairs deep learning with a small set of concrete principles borrowed from cognitive neuroscience, rather than treating those principles as decoration: lateral inhibition inside the modality encoders, an environment-conditioned stage that re-balances modality weights on the fly, a probabilistic decision layer, and an online predictive-coding-style update on top of that decision. The framework operates on the four modalities actually present in our data, visual, auditory, haptic and spatial; it contains no olfactory channel. Across 25 MR interaction classes spread over 12 scene types, the model reaches a mean accuracy of 94.7% (± 1.2 over five seeds) and runs end-to-end in 12.3 ms on a single GPU, comfortably inside the budget for interactive use. The corpus underlying these numbers is sizeable: 28,055 labelled interaction segments collected from 24 volunteers across the 12 scenes, with the 25 classes balanced to a coefficient of variation of 0.07, and inter-rater agreement of κ = 0.87. Measured against eight baselines, among them a multimodal transformer, a CLIP-style fusion and a Perceiver, accuracy improves by 8.3–15.7% (paired t-test on five matched seeds, t(4) = 9.42, p < 0.01, Cohen’s d = 1.76, 95% CI for the gap against the strongest baseline [1.9, 3.4]; post-hoc power > 0.95). The same framework, evaluated on EPIC-KITCHENS-100 as an external public benchmark, narrows the gap to the strongest competing fusion model to within a percentage point, suggesting the gains are not an artefact of the in-house data. Rather than letting a neuroscience label do the explanatory work, we tie each principle to a switchable module and report what removing it costs: lateral inhibition lives inside the modality encoders, Bayesian integration lives in the decision layer, predictive coding lives in the online weight update, and a component ablation shows each one earns its place, with adaptive weighting and cross-modal alignment carrying the largest share. One caveat is stated up front. User-facing constructs such as immersion and cognitive load are estimated by computational user models rather than measured on people; every user-facing number in this paper is therefore a predicted quantity, and a calibration study against standardised instruments — NASA-TLX for perceived workload [66] and the Igroup Presence Questionnaire (IPQ) for presence [67] — is deferred to future work. Twenty-four adult volunteers contributed sensor data during MR interaction sessions under an IRB-approved protocol (Kyonggi University, ref. KGU-2024-IRB-038), and written informed consent was collected before every session; no physiological monitoring or subjective rating was recorded, so the study remains computational in its outputs even though real participants sit at the input side.