AI-Based Speech-Driven 3D Avatar Animation System Integrated Gender, Emotion, and Viseme Analysis for Browser-Rendered Avatars

Speech-driven three-dimensional avatars provide a practical interface between human speech and interactive digital characters. This paper presents an integrated, AI-based pipeline that converts ordinary speech into a synchronized, browser-rendered avatar without specialized capture hardware. A frozen pretrained Wav2Vec2 encoder extracts frame-level speech features, which three lightweight modules interpret to infer speaker gender, emotional state and viseme category. Gender classification fuses a trained GenderCNN with pitch evidence through calibrated weighting, so that avatar selection does not rest on the neural classifier alone. A thirteen-class Transformer predicts visemes, and a second trained convolutional network smooths them before rendering. A FastAPI backend performs inference, and a Three.js frontend renders the avatar with procedural breathing, blinking and audio-reactive motion. Recorded measurements include a mean Wav2Vec2 real-time factor of about 0.18, a GenderCNN accuracy of 0.7130 before augmentation and 0.7207 on the augmented domain, and an eight-class emotion accuracy of about 0.37 for a Random Forest baseline. The BiLSTM emotion model used at inference has no logged accuracy yet, and this is documented as an open item. The system is framed as an explainable, deployment-ready engineering pipeline rather than a direct competitor to continuous speech-to-mesh research.