Challenges of Testing Machine Learning Based Systems

Machine learning is being increasingly deployed in many industrial and societal applications. Given its widespread use, it is of utmost importance to ensure the quality of software systems supported by machine learning. Although researchers have applied some concepts from traditional software testing to testing of machine learning based systems, the latter introduce a range of challenges not typical for traditional software systems, thus making traditional software testing techniques ineffective. In this paper, we discuss the challenges intrinsic to testing of machine learning based systems. We highlight the promising role of machine learning based testing to alleviate some of these challenges. We also discuss directions for future research in this domain. This paper focuses on the testing aspects of machine learning based systems from the quality assurance perspective, rather than model performance perspective.