When AI-Generated Unit Tests Validate Bugs: The Risk of Faulty Assertions

There is increasing research and commercial exploration into tools for automated unit test generation using Large Language Models (LLMs). This paper critically examines how recent LLM-based test generation tools pitched for unit testing such as Qodo Cover and GitHub Copilot handle buggy code. Considering bugs are only exposed by failing test cases, we explore the question: can these tools truly achieve the intended objectives of unit testing - find bugs? Using real human-written buggy code as input, we evaluate these tools, showing how LLM-generated tests can fail to detect bugs and, more alarmingly, validate bugs in the generated test suite. These findings raise important questions about the usefulness of LLM-based unit test generation tools today and their impact on software quality and test suite reliability.