Introduction: Virtual characters that are expressive and sensitive are increasingly in demand across the growing fields of Virtual Reality (VR), gaming, and human-computer interaction. Traditional animation systems rely on pre-recorded or manually designed movements, which lack real-time adaptability and multimodal response. Multimodal perception can dramatically improve animation realism and engagement. Methods: The research aims to develop a real-time animation generation system for virtual characters that uses multimodal perception to improve texture accuracy, motion consistency, and processing efficiency. A novel Mayfly Optimised Variational Autoencoder (MFlyO-VAE) method is proposed to generate animation. Real-time audiovisual and motion data were collected with depth cameras, microphones, and motion capture sensors. Butterworth filtering and min-max normalisation were used to reduce noise and align temporal signals, providing clean and synchronised input signals. Facial and gesture features were extracted using a Convolutional Neural Network (CNN)-based pose and expression tracker. Mel-Frequency Cepstral Coefficients (MFCC) were used to determine speech characteristics. Results: The Multimodal Transformer (MMT) uses attention mechanisms to capture temporal and contextual connections across modalities. The VAE uses a probabilistic latent space to generate smooth animation transitions. The MFlyO was utilised to fine-tune the VAE hyperparameters. The proposed MFlyO-VAE method achieved a response speed of 1.35 seconds at iteration 550 and 98% precision at iteration 305. Discussion: CNN, MFCC, and MOA-optimised VAE combined with multimodal perception provide a solid foundation for creating high-quality, real-time animations for VR and gaming. Conclusion: Multimodal perception in conjunction with the MFlyO-VAE method provides responsive, realistic, and efficient real-time animation generation for advancement in applications in VR, gaming, and human-computer interaction.
Zhifang He (Thu,) studied this question.