OmniSketch: Sketch-Guided Text-to-3D Generation with High-Fidelity Geometry and Texture

Sketch-based or text-based 3D content generation, which rapidly converts abstract concepts into 3D models, plays a vital role in accelerating digitalization. However, the sparse sketch, absent texture and unaligned dataset pose considerable challenges for cross-modal synthesis. To address these issues, we introduce OmniSketch, a sketch-guided text-to-3D generation framework that decouples the complex sketch-to-3D task into a three-stage pipeline. Firstly, we employ a conditional diffusion model to fuse structural constraints from the sketch with semantic descriptions, generating a high-quality intermediate reference view. Subsequently, a Rectified Flow model is utilized to produce a 3D model with high-fidelity geometry. Finally, we apply a geometry-guided multi-view image synthesis method to furnish the model with view-consistent, high-quality textural details. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches in geometric fidelity and texture consistency. Leveraging our framework, we also construct a multi-modal sketch-to-3D dataset and an interactive platform to promote the application of digital content generation.