OCC-MLLM-V2: Joint understanding and generation for occluded objects via multi-modal token learning

Comprehending occluded objects remains a critical challenge for multi-modal large language models due to missing visual representations. Current approaches either rely on multi-stage pipelines with error propagation, or use unified encoders that fail to balance understanding and generation. We propose OCC-MLLM-V2, an end-to-end autoregressive framework with three key innovations: (1) Hierarchical Trinity Fusion Architecture integrating multi-view RGB, hand masks, and 3D reconstructions via Adaptive Weight Image Fusion and Spatial Attention Affine Fusion; (2) Visual Dual Encoder employing SigLIP for understanding and VQ tokenizers for generation; (3) Visual Dual Decoder with joint optimization. Unlike pipeline methods, our unified framework eliminates sequential dependencies and enables end-to-end gradient flow. Experiments demonstrate improvements: 6.27% gains on ObMan and 6.35% on DexYCB across 1B-8B models, while reducing FLOPs by 32%–59% and inference time by 9.5%–37%. Our framework surpasses GPT-4o and Gemini 2.5-Pro on multiple occlusion benchmarks. Code and data are available at https://github.com/chaoyiwang09/OCC-MLLM .