Identification of Syndrome Types in Pancreatic Cancer Patients from Electronic Medical Record Free Text: Model Development and Validation (Preprint)

BACKGROUND Syndrome differentiation is fundamental to diagnosis and treatment planning in traditional Chinese medicine (TCM). However, this process is highly dependent on expert experience, thereby limiting systematically standardization. OBJECTIVE This study developed and validated a BERT-based model, TCMPCSD-BERT, using in-house pancreatic cancer medical records, to digitalize expert knowledge and support standardized syndrome differentiation in TCM. METHODS A retrospective dataset of pancreatic cancer cases (2011–2024) from Fudan University Shanghai Cancer Center was annotated into four TCM syndrome types by two experts (Cohen’s kappa = 0.913). The proposed TCMPCSD-BERT model was compared with conventional models (LSTM, Text-CNN) embedded in TCM diagnostic tools and with large language models (LLMs: ChatGPT 4o, Kimi, Ernie Bot 4.0 Turbo, Zhipu Qingyan) under a Prompt Engineering framework. Performance evaluation on in-house data was supplemented with attention visualizations and Integrated Gradients analyses for interpretability. McNemar’s test was used to assess differences in classification accuracy. bootstrap 95% confidence intervals (CIs) were calculated to quantify statistical uncertainty and stability, and Welch’s t-test was applied to evaluate mean differences between TCMPCSD-BERT and comparator models. RESULTS Among 6,830 records, case counts were: damp-heat syndrome (n = 1,694), spleen-deficiency syndrome (n = 1,185), damp-heat with spleen-deficiency syndrome (n = 1,178), and others (n = 2,773). On the test set, McNemar’s test indicated statistically significantly higher accuracy for TCMPCSD-BERT than the three baseline models, and generally better performance than LLMs. Across all comparisons, TCMPCSD-BERT yielded higher mean Macro-Precision, Macro-Recall, Macro-F1, and Accuracy, with non-overlapping 95% bootstrap CIs and statistically significant Welch’s t-test results (P < 0.01). The model achieved a Macro-Precision of 0.935 (95% CI 0.918–0.951), Macro-Recall of 0.921 (95% CI 0.900–0.942), Macro-F1 of 0.927 (95% CI 0.908–0.945), and Accuracy of 0.919 (95% CI 0.899–0.939). Attention visualizations suggested the model could capture less common TCM term associations, while Integrated Gradients highlighted high-attribution diagnostic features (e.g., “gray-white stool” 0.933 in damp-heat syndrome; “indigestion” 1.204 in spleen-deficiency syndrome). Misclassification analyses indicated challenges in handling overlapping or atypical symptom presentations. Compared with LLMs, online platforms, and diagnostic instruments, TCMPCSD-BERT appeared to provide relatively higher accuracy, interpretability, and efficiency in processing long unstructured texts for syndrome differentiation. CONCLUSIONS The TCMPCSD-BERT model shows potential for automated syndrome differentiation from unstructured clinical texts and broader application in TCM. Based on this study, it appears to improve operability over four-diagnostic instruments and online platforms, and offers greater stability and accuracy than LLMs in specific tasks. However, these findings should be interpreted cautiously, given the subjectivity of syndrome definitions, data imbalance, and reliance on preprocessed, expert-annotated data. Further studies involving larger and more diverse populations are needed to validate its generalizability and support its broader application in real-world applicability.