Computer vision as a data source for digital twins in manufacturing: A Sim2Real pipeline

The rapid advancement of Industry 4.0 has established Digital Twins (DTs) as crucial technologies for optimizing the management of manufacturing systems. These twins facilitate data-driven decision-making throughout the production lifecycle. Nonetheless, the implementation of DTs in industrial settings faces challenges, particularly in acquiring and integrating accurate, time-synchronized data streams that mirror real-world processes. Traditional sensors often lack the necessary contextual and spatial details, especially in complex manufacturing environments. In this context, computer vision has emerged as a complementary sensing technology, capable of extracting rich semantic information from images and videos. It enables various tasks such as object detection, process monitoring, and activity analysis, whose outputs can be integrated into DT systems. However, creating vision-based solutions for DT applications is hindered by a lack of annotated datasets and the challenge of fine-tuning off-the-shelf CV algorithms that heavily rely on training techniques. To address these challenges, this paper explores the role of computer vision as a primary data source for the development and deployment of Digital Twins. A review of existing techniques and industrial datasets highlights current limitations for assembly-based applications. Beyond these limitations, this work introduces a Sim2Real object detection pipeline that extends existing approaches by combining synthetic and controlled industrial data within a unified training framework using YOLO-World. The model is trained on a mixed dataset to enhance class generalization while respecting data privacy constraints. The approach is validated through a case study on tidal turbine assembly, showcasing its potential to improve DT capabilities for real-time monitoring and manufacturing operational constraints in manufacturing systems.