Multimodal Prompt-Tuning Large Language Model for Machinery Fault Diagnosis with Sound-Vibration Signals

Vibration and sound-sensing technology has become the mainstream perception method in the field of mechanical fault diagnosis. However, most current diagnostic models still focus on single-modality data analysis, which to some extent restricts further improvement of diagnostic performance. Multimodal fault-diagnosis models are still relatively lacking in existing research. In addition, most methods usually simply concatenate or early fuse vibration and sound signals and directly input them into the model, failing to deeply explore and effectively fuse the discriminative features of each modality. Moreover, the strong generalization ability of large language models (LLMs) demonstrates their potential for application in other fields. Therefore, it is interesting to perform machinery fault diagnosis based on LLMs with sound and vibration signals. This study proposes a new multimodal DNN, namely a sound-vibration large language model (SVLLM), for multimodal feature fusion and machinery fault diagnosis. Firstly, a multimodal prompt-tuning module is proposed to fine-tune LLMs to learn domain knowledge of fault diagnosis based on vibration signals and sound signals. It can also adaptively fuse sound signals and vibration signals. Secondly, a freeze–thaw module is proposed to freeze the pre-trained key layers and thaw the layers that need to be learned to adapt the model to fault diagnosis tasks. Thirdly, the multihead cross-attention module is proposed to fully extract the characteristics of vibration signals and sound signals through modal expansion interaction. The experimental results on the motor dataset and Ottawa dataset show that SVLLM has good performance in feature extraction and machinery fault diagnosis with sound and vibration signals.