Autonomous Detection and Assessment of Indoor Building Defects Using Multimodal Learning and GPT

Buildings deteriorate over their service life. The early detection of defects such as cracking, spalling, corrosion, and moisture can benefit the preventative maintenance of building systems. Autonomous robotic systems have enormous potential in automating indoor building defect inspections, along with challenges of inaccurate prediction and unorganized information. With the implementation of state-of-art multimodal learning methods and large language model (LLM) techniques, we present a cutting-edge workflow composed of image captioning, landmark documentation, and real-time on-site human–machine interactive path planning. Compared with previous vision and language navigation (VLN) algorithms, our workflow introduces defect prompts to improve indoor inspection captioning performance. These pivotal defect features are extracted by YOLO (You Only Look Once) v5, a PyTorch-based deep learning model. As the robotic system recognizes the environment clearly, inspectors are capable of providing target-oriented instructions to control the survey path. By implementing the large language model GPT-3, vocal and textual instructions are transferred to the robotic localization system and summarize a brief inspection report. In this way, with the assistance of GPT, numerous inspections that previously demanded substantial effort can be conducted efficiently and expeditiously.