A Systematic Review of IoT Systems Testing: Objectives, Approaches, Tools, and Challenges

Internet of Things (IoT) systems are becoming prevalent in various domains, from healthcare to smart homes. Testing IoT systems is critical in ensuring their reliability. Previous papers studied separately the objectives, approaches, tools, and challenges of IoT systems testing. However, despite the rapid evolution of the IoT domain, no review has been undertaken to investigate all four aspects collectively. This paper presents a systematic literature review that aggregates, synthesizes, and discusses the results of 83 primary studies (PSs) concerning IoT testing objectives, approaches, tools, and challenges. We followed the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) protocol to report our findings and answer research questions (RQs). To select PSs, we applied inclusion and exclusion criteria to relevant studies published between 2012 and 2022. We extracted and analyzed the data from PSs to understand IoT systems testing. The results reveal that IoT systems testing embraces traditional software quality attributes but also introduces new ones like connectivity, energy efficiency, device lifespan, distributivity, and dynamicity. They also show that existing IoT systems testing approaches are limited to specific aspects and should be expanded for more comprehensive testing. They also show 19 testing tools and 15 testbeds for testing IoT systems with their limitations, necessitating the development or enhancement for wider coverage. The large number of heterogeneous devices generating data in different formats, along with the need for testing in real-world scenarios, poses a challenge. Thus, our study offers insights into the testing objectives, approaches, tools, and challenges associated with IoT systems. Based on the results, we also provide practical guidance for IoT practitioners by cataloging existing tools and approaches, while also identifying new research opportunities for interested researchers.