Development and validation of a natural language processing system to assess quality of physician communication in prostate cancer consultations

Abstract Background AUA guidelines for shared decision making (SDM) in prostate cancer recommend discussion of five content areas in consultations: (1) cancer severity (tumor risk (TR), pathology results (PR)); (2) life expectancy (LE); (3) cancer prognosis (CP); (4) baseline urinary and erectile function (UF and EF); and (5) treatment side effects (erectile dysfunction (ED), urinary incontinence (UI), and irritative urinary symptoms (LUTS)). However, patient retention of information after the visit and inconsistent risk communication by physicians are barriers to informed SDM. We sought to develop natural language processing (NLP) models based on recorded consultations to provide key information to patients and audit quality of physician communication. Methods We used 50 consultation transcripts to train and validate NLP models to identify sentences related to key concepts. We then tested whether communication quality across entire consultations could be determined by sentences with the highest model-predicted topic concordance in 20 separate consultation transcripts. Results Our development dataset included 28,927 total sentences, with 75% reserved for training and 25% for internal validation. The Random Forest model had the highest accuracy for identifying topic-concordant sentences, with area under the curve 0.98, 0.94, 0.89, 0.92, 0.84, 0.96, 0.98, 0.97, and 0.99 for TR, PR, LE, CP, UF, EF, ED, UI, and LUTS compared with manual coding across all concepts in the internal validation dataset. In 20 separate consultations, the top 10 model-identified sentences correctly graded communication quality across entire consultations with accuracies of 100%, 90%, 95%, 95%, 80%, 95%, 85%, 100%, and 95% for TR, PR, LE, CP, UF, EF, ED, UI, and LUTS compared with manual coding, respectively. Conclusions NLP models accurately capture key information and grade quality of physician communication in prostate cancer consultations, providing the foundation for scalable quality assessment of risk communication.