Developing robust benchmarks for driving forward AI innovation in healthcare

Machine learning technologies have seen increased application to the healthcare domain. The main drivers are openly available healthcare datasets, and a general interest from the community to use its powers for knowledge discovery and technological advancements in this more conservative field. However, with this additional volume comes a range of questions and concerns — are the obtained results meaningful and conclusions accurate; how do we know we have improved state of the art; is the clinical problem well defined and does the model address it? We reflect on key aspects in the end-to-end pipeline that we believe suffer the most in this space, and suggest some good practices to avoid reproducing these issues. Finding good benchmarks is an important and pervasive problem in machine learning for healthcare. This Perspective highlights key aspects that require scrutiny in the whole process of benchmark generation and use, including problem formulation, creation of datasets, development of a suite of machine learning models and evaluation of these models.