An interpretable real-time outlier detection method for online near infrared tobacco data based on linear regression

In the process of online near infrared (NIR) analysis of tobacco leaves, outliers inevitably appear in the large number of NIR spectra and chemical composition data generated due to factors such as instrumental errors, environmental variations, and sample anomalies. These outliers can severely affect the reliability and validity of subsequent data analysis, and consequently the grading and quality assessment of tobacco leaves. Many existing algorithms are unable to meet the requirements of real-time analysis and interpretability, and some studies considered only anomalies of NIR spectra and neglected the outliers in the predicted chemical composition data. We propose an unsupervised method for online NIR outlier detection based on linear regression using offline data, leveraging the strong positive correlation between nicotine and total nitrogen content in tobacco chemical composition. The method was applied to detect outliers in online tobacco NIR spectra and chemical composition data from different origins in Yunnan Province to evaluate its effectiveness and it was compared with several existing outlier detection algorithms. The results demonstrate that the linear regression method based on offline data can identify and remove outliers in online NIR analysis, thereby significantly improving the reliability and stability of tobacco chemical composition analysis. Compared with several existing outlier detection algorithms, the proposed method exhibits superior real-time performance, lower computational resource consumption, and strong interpretability, making it well-suited for real-time outlier detection in online NIR applications. Notably, in practical application, combining the method with the Mahalanobis distance-based outlier detection can ensure the reliability of the obtained online data, although it will be more stringent.