A novel methodology for clinical semantic annotations assessment

Recently, there have been big leaps of advances in key natural language processing tasks that support semantic analysis, achieving some success in real-world clinical tasks. However, there is still a gap between the development of advanced semantical annotation resources for real-world plain text electronic health records (EHRs), generated by Language Annotating Systems (LASs), and their utilization in clinical settings. One common obstacle is that medical staff have no time to assess the quality of the annotations due to their workload. This paper introduces a novel method to tackle the quality assessment problem overcoming the limitations of classical approaches (i.e the need of manual annotations, which are unfeasible in real-life clinical environments), providing a mechanism to obtain quantitative accuracy indicators of the semantic annotations generated by a LAS from EHRs as a previous step to a human validation. The method is supported by a statistical analysis, and we demonstrate in real cases that a LAS previously introduced by our research group is competitive or even better regarding the state-of-the-art LAS approaches, needing less computational resources because no pre-annotations are needed in our case.