MaterialSeg3D++: Large-Scale Material Prediction for 3D Assets from 2D Priors

Recent image diffusion models have enabled automatic 3D object creation from text or image guidance, but their 2D generative priors often bake illumination and shadow into textures, making relighting and physically based rendering (PBR) difficult. To address this issue, we propose MaterialSeg3D, a framework that predicts surface materials for 3D assets by leveraging 2D material semantics. Given a mesh and its albedo UV map, MaterialSeg3D renders multi-view images, performs material segmentation using a 2D prior model, projects the predictions back to UV space, and fuses them through weighted voting and region unification to obtain coherent material maps. To train the prior model, we construct MIO++, a large-scale single-object material segmentation dataset containing 115,542 images, 12 object themes, and 30 fine-grained material categories, substantially extending the previous MIO dataset. Each MIO++ material category is associated with an independent pair of roughness and metallic values for PBR assignment. We further observe that poor topology in AI-generated assets can degrade PBR quality even when plausible materials are assigned, and introduce a plane-simplification strategy as an auxiliary preprocessing step for such meshes. Experiments show that MIO++ improves material segmentation and that the resulting material maps support more consistent relightable renderings for both human-crafted and AI-generated 3D assets.