Hybrid Approach to Patient Review Classification at Scale: From Expert Annotations to Production-Ready Machine Learning Models for Sustainable Healthcare

Patients leave millions of medical reviews annually, providing critical data for quality management. However, manual processing is infeasible, and existing systems fail to distinguish medical from organizational problems—a distinction essential for complaint routing. The consequences of misrouting are significant: clinical issues may go unaddressed when medical complaints reach administrative staff, while systemic service problems remain unresolved when organizational complaints reach medical directors. We developed a hybrid approach combining expert annotation with Large Language Models (LLMs). Fifteen prompt iterations on 1500 reviews with expert validation (modified Cohen’s kappa (κ_mod), which weights errors hierarchically, reached 0.745) preceded the LLM annotation of 15,000 mixed-sentiment and positive reviews. These were combined with 7417 expert-annotated negative reviews to form a corpus of 22,417 reviews. Eight architectures, ranging from Logistic Regression to a BERT + TF-IDF + LightGBM ensemble, were compared using both standard metrics and domain-specific practical metrics tailored to complaint routing. The best model, scaled to 4.3 million Russian-language reviews from the Prodoctorov.ru platform, achieved 92.9% Practical Accuracy—the proportion of reviews classified without critical medical–organizational misclassification errors (M ↔ O)—compared to 68.0% standard accuracy, which treats all errors equally. Critical errors were reduced to 1.4%, yielding 144,000 more correctly processed complaints than traditional methods (TF-IDF + Logistic Regression). Analysis of the scaled data revealed the following: 46.1% M (medical), 21.0% O (organizational), and 32.9% C (combined) reviews; medical ratings were highest (4.75 vs. 4.59 for organizational, p < 0.001); combined reviews were longest (802 characters); zero-star reviews comprised 3.8% of feedback, with organizational complaints dominating (38.2%) among extreme negatives; and average ratings rose by 1.24 points over 14 years. This hybrid approach yields expert-comparable corpora, automates 93% of feedback processing, ensures correct complaint routing, and contributes to healthcare sustainability by reducing administrative burden, accelerating resolution, and enabling data-driven quality management without proportional increases in human resources. All analyses were conducted on Russian-language patient reviews.