PU-Based Quality Classifier for LLM Training Texts

In this study, we examine positive-unlabeled (PU) learning for classifying Polish texts intended for large language model training. We evaluate six PU methods, Naive, Clust, Strict-LassClust, Non-Strict-LassClust, LassoJoint, and Spy, under both classic and MVC non-SCAR labeling schemes. Across the benchmark, the best F1 reaches 0.906 on an open subtitles corpus under classic labeling and 0.900 under MVC labeling; the largest absolute gain over the Naive baseline is 0.811 F1 points on ulotki medyczne and 0.758 on plwiki in the classic setting. The experimental workflow supports multiple labeling rates, c calc, and repeated random seeds, and evaluates models with AUC, PR-AUC, and F1 as the main metrics. Our study focuses on six SpeakLeash datasets spanning different domains: plwiki, open subtitles corpus, job offers corpus, ISAP corpus, ulotki medyczne, and wolne lektury corpus. For each dataset, we construct a binary PU setup from the quality field, where HIGH is treated as positive and LOW is negative, while other values are removed. The resulting preprocessed tables retain only the selected linguistic and structural features, rescale numeric variables to the [0, 1] range, and remove the raw text column.