LLM-driven test and test-repair automation for legacy Perl systems: an empirical study at IBM

This is the authors' abstract. We don't add key points for this paper.

Automated unit test generation using Large Language Models (LLMs) has shown promise for widely used programming languages such as Java and Python. It may further benefit legacy systems lacking a test base, and the investigation of less widespread legacy languages remains relevant as many companies still operate systems built on them. We develop a modular workflow combining static code analysis, multi-stage prompt engineering with business logic extraction, automated error analysis, and iterative test-fix cycles for legacy systems, evaluated by pass rates, coverage, and repair effectiveness. We evaluate our approach on four productive IBM Perl repositories (186 functions) and three CPAN distributions (139 functions) using two open-source LLMs: Llama-3.3-70B-Instruct and Mistral-Small-3.1-24B-Instruct-2503. A key methodological contribution is a defensive wrapper that records a crash in the code under test as a failed assertion rather than terminating the run. A git-diff mode enables selective test generation for changed functions, supporting incremental maintenance workflows. The larger model achieves initial pass rates of 66–82% (industrial) and 74–81% (CPAN), improving to 74–86% and 76–89% after repair, with statement coverage of 34–92% and 65–100%. The smaller model performs substantially worse on both sets. Repair reduces structural errors on both ( from 0–44% to 0–6% and 0–7% to 0–5%) and improves pass rates for both models in most repositories. We show the transferability of LLM-based test generation to low-resource legacy languages, establishing Perl as a viable target for automated testing pipelines.