Black-Box Mutation Testing of Web APIs

The key role of web Application Programming Interfaces (APIs) in software integration has led to a surge in the development of test case generation tools for web APIs. Most of these tools work in black-box mode, without access to the source code, and are primarily assessed based on their ability to detect crashes and violations of API descriptions. As a result, their effectiveness in identifying other types of failures (e.g., incorrect outputs caused by functional bugs) remains largely unknown. In this article, we propose a black-box mutation testing approach to assess the fault-detection capabilities of testing tools for web APIs. Unlike traditional mutation testing, which introduces faults in the source code, our approach generates faulty but realistic variants of HTTP responses, simulating the observable effects of bugs in the underlying implementation. These mutated responses can then be used to evaluate the effectiveness of testing tools in detecting incorrect outputs, making our approach readily applicable to both closed- and open-source web APIs. The approach is implemented in an extensible tool, HttpMutator , which supports 23 mutation operators for HTTP responses, including mutations on status codes, headers, and JSON payloads. Evaluation results on 30 API operations from 19 open-source and industrial APIs show that HttpMutator is effective in generating diverse and realistic erroneous outputs, with a positive correlation between black-box and white-box mutation coverage. More importantly, HttpMutator has already proven instrumental in assessing the effectiveness of test case generation tools for REST APIs, highlighting its practical relevance.