Text data preprocessing module in an intelligent classification system

Purpose of research. The purpose of this study is to develop and conduct a comprehensive analysis of a text data preprocessing module intended for use within an intelligent classification system. The primary focus is on improving the quality of processing complex textual information, which may be unstructured, multilingual, and may also contain errors or partial corruption. Within the scope of this work, the task is to ensure stable and efficient processing of such data under real-world information flow conditions. Methods. To achieve the stated objective, modern methods of intelligent text processing were employed. In particular, document parsing algorithms for various formats, text cleaning and normalization procedures, as well as tokenization techniques were applied. Special attention was given to the use of an n-gram approach for automatic language detection. Additionally, statistical analysis methods were utilized, including character frequency analysis, which made it possible to construct an informative feature space based on textual representations. Results. As a result of the conducted research, a module was developed that is capable of efficiently extracting textual information from various sources, including complex formats such as JSON and XML, even in the presence of corrupted data. Experimental evaluation demonstrated that the use of the proposed module increases classification accuracy to approximately 89% and ensures stable processing of more than 90% of input data. Furthermore, the application of the n-gram algorithm enabled achieving high accuracy in language detection, exceeding 91%. Conclusion. The developed preprocessing module demonstrates high efficiency, enhances the reliability and versatility of the intelligent classification system, and can be successfully applied to the analysis of textual information in practical tasks.