ImmerseSketch: Transforming Creative Prompts into Vivid 3D Environments in VR

We propose ImmerseSketch, a framework designed to transform creative prompts into vivid and detailed 3D content within a Virtual Reality (VR) environment. Our aim is to inspire creative ideation and to provide creators with 3D reference content for painting. However, the computational demands of diffusion models for 3D model generation and the scarcity of high-quality 3D datasets limit the diversity and intricacy of the generated environments. To overcome these obstacles, we focus on generating initial panoramic images through diffusion models and then converting these images into rich 3D environments with the aid of depth estimation. The pilot study shows that the proposed ImmerseSketch can provide an immersive environment and assist the process of creation.