Resilient Assembly Supervision: A Synthetic-to-Real SemanticTwin Pipeline for Data-Efficient Operator Guidance

Manual assembly remains critical in Industry 5.0 high-mix/low-volume manufacturing, but it introduces resilience challenges due to cognitive load, training demands and frequent product changes. While AI-based supervision can mitigate errors, deploying such systems is often hindered by the cost of collecting and labelling thousands of real images for each product variant. This paper presents a Human-in-the-Loop semantic-twin pipeline that generates approximately 45,000 labelled synthetic images from a single CAD-based configuration and uses them to train an object detection model for real-time assembly supervision. Experiments on seven virtual environment configurations show that removing realistic lighting or camera motion reduces F1-score on real images from 0.87 to 0.46, confirming their critical role for synthetic-to-real transfer. A controlled laboratory study on a single bicycle chainring assembly task with 10 participants and 100 monitored cycles demonstrates the feasibility of automatic KPI extraction, with error events associated with a 25.6% increase in average cycle time (from 58.4 s to 73.3 s) under the tested conditions. Compared to manual annotation, where labelling 3000 images required approximately 4 h, the semantic-twin configuration takes around 4 to 6 h including image generation that enables rapid creation of large labelled datasets for new product variants without additional human annotation. These results provide a proof-of-concept foundation for resilient, data-efficient supervision of high-mix manual workstations, with full industrial validation across multiple products, stations and operator demographics identified as the necessary next step.