Embodied 3D perception requires understanding dynamic environments from ego-centric viewpoints using natural language instructions. However, current methods leveraging large language models (LLMs) for embodied 3D perception remain limited. Some approaches suffer from restricted semantic output and localization accuracy, while other LLM approaches lack a unified encoder for effectively aggregating multi-view semantic and geometric features which are essential for precise language alignment and localization. To address this, we introduce M3Fusion, the first end-to-end framework for unified multi-view multi-modality fusion in embodied 3D perception. M3Fusion tightly integrates 2D visual semantics and 3D geometric features from multiple ego-centric views, projecting them into a shared 3D space to form unified M3-tokens. These tokens enable seamless alignment with language for complex instruction understanding and simultaneously decode accurate 3D bounding boxes. We propose a specialized 3-stage training strategy to align modalities. Evaluations on 3D visual grounding and QA datasets demonstrate significant performance improvements, particularly in grounding accuracy while maintaining QA capability, highlighting our unified representation and framework design.
Mao et al. (Thu,) studied this question.