Evaluation of machine learning algorithms for health and wellness applications: A tutorial

Research on decision support applications in healthcare, such as those related to diagnosis, prediction, treatment planning, etc., has seen strongly growing interest in recent years. This development is thanks to the increase in data availability as well as to advances in artificial intelligence and machine learning research and access to computational resources. Highly promising research examples are published daily. However, at the same time, there are some unrealistic, often overly optimistic, expectations and assumptions with regard to the development, validation and acceptance of such methods. The healthcare application field introduces requirements and potential pitfalls that are not immediately obvious from the 'general data science' viewpoint. Reliable, objective, and generalisable validation and performance assessment of developed data-analysis methods is one particular pain-point. This may lead to unmet schedules and disappointments regarding true performance in real-life with as result poor uptake (or non-uptake) at the end-user side. It is the aim of this tutorial to provide practical guidance on how to assess performance reliably and efficiently and avoid common traps especially when dealing with application for health and wellness settings. Instead of giving a list of do's and don't s, this tutorial tries to build a better understanding behind these issues and presents both the most relevant performance evaluation criteria as well as approaches how to compute them. Along the way, we will indicate common mistakes and provide references discussing various topics more in-depth.