Using Artificial Intelligence for the Early Diagnosis of Vehicle Malfunctions: A Leakage-Aware Evaluation and Meta-Analysis

The early diagnosis of the vehicle malfunctions by the artificial intelligence (AI) is a promising way to reduce the breakdowns and the maintenance cost, but the reported accuracies in the literature are usually very high and they may not reflect the real use. In this paper, six machine learning classifiers were trained on the public Engine Fault DB dataset, which contains 55,999 records of 14 engine and exhaust variables under one normal and three fault conditions. The models were evaluated by two protocols, a random five-fold split and a blocked split in which the test segments represent unseen operating conditions. In addition, a random-effects meta-analysis of nine published studies was conducted. The results showed that the four-class accuracy was about 75% under the random split, and it dropped to between 41.17% and 66.92% under the blocked split. However, the detection of the fault against the normal state was almost perfect (AUC = 1.000), and when the two fault types which cannot be separated were merged, the accuracy became 98.22% with all the signals and 97.34% with six on-board signals only. The meta-analysis gave a pooled reported accuracy of 96.50% (95% CI 92.63% to 98.38%) with a very high heterogeneity (I2 = 98.47%). The findings indicate that, on this dataset, the AI models can detect the presence of a fault reliably, while the isolation of the fault type under new operating conditions remains difficult and it needs a stricter validation practice.