Optimization algorithm for immersive VR spatial interaction based on multimodal perceptual fusion

To solve the problem of “human-machine gap” caused by insufficient understanding of intentions in traditional Virtual Reality (VR) interaction, this study proposes a multimodal perception fusion interaction optimization algorithm called Sparse Attention Transformer Proximal Policy Optimization (SAT-PPO). This method constructs a “perception-decision” closed-loop framework: The perception layer adopts a lightweight Transformer model with sparse attention mechanism, and uses large-scale Ego4D datasets for pre-training to deeply integrate multi-modal data streams such as user’s sight, posture and voice. The decision-making layer trains an agent that can dynamically adjust the interaction strategy through the Proximal Policy Optimization (PPO) Deep Reinforcement Learning (DRL) algorithm. Experimental results in complex virtual assembly tasks show that, compared with many baseline models, SAT-PPO algorithm optimizes the Task Completion Time (TCT) to 110.2 s, reduces the Error Rate (ER) to 4.4%, and significantly reduces the user’s NASA Task Load Index (NASA-TLX) index to 28.1. The study proves that the algorithm successfully realizes efficient, smooth and intuitive adaptive VR interaction through accurate user intention prediction, which provides an effective technical solution for building the next generation immersive experience.