Natural language processing: use in EBM and a guide for appraisal

Studies using natural language processing (NLP) techniques are increasingly being published. Evidence-based medicine (EBM) users need to learn the basics of NLP to be able to appraise these types of studies. We propose a set of criteria to evaluate the quality of studies that have used NLP, focusing on the methods of sample selection, coding, the gold standard, algorithm training, algorithm testing and measures of accuracy (such as recall and precision). NLP has proven critical for conducting biomedical research and has the potential to improve healthcare practice and facilitate EBM. Stakeholders (healthcare providers and policymakers) interested in using evidence derived from studies that used NLP need to know the basics of NLP and need to be able to appraise this type of study.