1 Developing an annotated database to facilitate machine learning in rare pediatric cancer

Objectives/Goals: Rare pediatric cancer research has been hindered by limited sample size and disease-specific databases. The training and infrastructure required to create these datasets have been cost prohibitive. Automated electronic medical record (EMR) extraction and natural language processing (NLP) hold the potential to overcome these limitations. Methods/Study Population: In a mixed population of adults and children with thyroid cancer, we implemented an automated data extraction process for multiple data types, including structured demographic, laboratory, and billing data and unstructured free text from pathology reports. Data extraction and cleaning software transformed EMR data into a research-ready database. NLP using regular expressions extracted meaningful, structured data from free-text pathology reports. Results from the automated data extraction and NLP were compared to manual extraction, and results were quantified using accuracy, precision, recall, F1-score, and Cohen’s Kappa to account for random agreement. Additional iterations using machine learning and large language models will be performed to compare accuracy and identify an optimal strategy. Results/Anticipated Results: From 2016 to 2023, 2,899 patients with thyroid cancer were identified. In a 50-patient subset, manual extraction of pathology reports was compared to NLP-derived data. Histology type was correctly identified in 90% of cases (kappa = 0.85), and thyroid procedure identification accuracy was 89% (kappa 0.81). Focality was 98% accurate (kappa=0.96). Vascular, lymphatic, and perineural invasion, along with extrathyroidal extension and pathologic tumor stage, were 100% accurate. Pathologic nodal staging was 96% accurate (kappa=0.89). Kappa greater than 0.8 is considered almost perfect agreement. Discussion/Significance of Impact: Our results indicate that automated EMR extraction and NLP can be used to develop a thyroid cancer dataset that is highly detailed, accurate, and efficient to collect. After optimizing our system of data extraction, we will replicate this process in a multi-center study with the eventual goal to expand these algorithms across rare pediatric cancers.