Using a synthetic 3D-environment and model-guided scene refinement to train a deep learning model for mapping fine woody debris from RGB-imagery

Deep learning (DL) algorithms are increasingly used for ecological applications, but manual annotation is still one of the main obstacles in ecological DL, especially for pixel-level segmentation in complex environments like the forest floor. Identifying fine woody debris (FWD, twigs, branches, and small tree stems) is important for understanding carbon cycling, fire risk, and habitat structures, but traditional inventories of FWD are tedious and hard to accomplish in a spatially continuous way. Automatic inventory of FWD from forest-floor images taken by a surveyor or an under-canopy drone flight could notably improve efficiency. Here, we present a workflow to train a DL-based model able to map FWD using synthetic training data only. For this, we created a simulation-based approach that produces photorealistic forest-floor images with perfect labels for training our segmentation models. We first built a detailed 3D environment and filled it with realistic elements such as soil, leaves, moss, stones, and a wide variety of woody fragments. We then rendered high-resolution RGB images along with their binary masks under different lighting setups, camera, and sensor conditions. We trained a SegFormer entirely on this synthetic dataset and improved it through an human-in-the-loop generative refinement process. The model’s mistakes showed us where to adjust the simulated scenes, e.g. increasing environmental complexity, changing debris shapes, refining lighting, or adding more occlusion. After five rounds of refining the simulated forests, the model performed very well on real images (600 patches of size 512 × 512 px) it had never seen before. It reached an F1-score of 0.80 and an Intersection over Union of 0.67, all without using any real annotated data. These results suggest that careful ecological simulation, combined with model-guided improvements, can produce training data for DL models, i.e. realistic enough to support accurate FWD segmentation in photos taken in natural forest conditions. Our approach is likely transferable to other applications.