Interactive Pathology Reporting Using Eye Gaze, Natural Speech, and Large Language Models

Context.— Pathology reporting is essential to clinical workflow, but reports are entirely text based, and report completion disturbs pathologists’ workflow. This article proposes an alternative pathology reporting workflow that addresses these issues. Objective.— To determine the feasibility of using a pathologist’s eye gaze and natural speech during whole slide image (WSI) assessment with a large language model (LLM) to generate an interactive pathology report. Design.— Ten pathologists assessed 3 WSIs in a controlled setting identifying preset tissue features, and 3 WSIs in an uncontrolled setting identifying any tissue features and measurements they desired. LLM accuracy, temporal eye gaze–natural speech delay, and number of report modifications were recorded. Eye gaze and voice data were collected to create an interactive multimedia pathology report; each finding linked to the corresponding WSI region. Pathologists subsequently completed a qualitative survey on their experience. Results.— LLM feature identification accuracy was 99.7% and 98.3% for the controlled and uncontrolled setting, respectively. Measurement identification accuracy for the uncontrolled setting was 72.2%. All report modifications were made by using a custom interface after the uncontrolled setting; of the 19 report modifications, 18 (95%) supported pathologists’ preferences rather than correcting report errors. Median temporal eye gaze–natural speech delay was 1.47 ± 0.96 seconds. All observers expressed that the proposed workflow could improve reporting efficiency and interphysician communication. Conclusions.— The LLM was better at identifying features than measurements, presumed to be due to the temporal eye gaze–natural speech delay. Future work is needed to further characterize this temporal delay.