Trust and Hallucinations: A Study of 39 Experts on AI-Assisted Requirements Reverse Engineering

With regard to the evolution of software systems, the process is hindered by the poor state of documentation, as software systems continuously evolve, which thereafter increases the maintenance costs to around 90% of development lifecycle spending. In addition, although the extraction of embedded business logic through the reverse engineering of requirements is essential, a gap in meaning remains between the source code and the high-level objectives, which means a need for addressing this issue. Therefore, currently, many artificial intelligence tools are in place for such actions. This research evaluates the performance of specialized Retrieval-Augmented Generation (RAG), general-purpose large language models, and hybrid static AI systems by focusing on the expert observations of practitioners within industrial environments. To achieve this, the study gathers data to measure hallucination rates and the accuracy of business rule recovery based on the actual professional experience of those managing legacy code. In particular, these experts used EPAM ART, GitHub Copilot, and IBM ADDI to provide percentage-based error estimates and rate rule identification on a standard scale. Ultimately, this empirical approach ensures that the research questions are addressed through the practical insights and lived experiences of professionals. In this research, a study of perspectives of 39 senior professionals observed that, while general models are successful at abstracting meaning with a score of 4.05 out of 5, a shortfall in traceability is retained. Furthermore, it was discovered that hybrid tools such as IBM ADDI allow for superior formal mapping with a score of 4.23 out of 5, although a struggle in verification is produced because high rates of incorrect data generation or hallucination exceeding 20% were reported by 66.7% of the participants. In light of these findings, this research proposes a strategy of multiple tool coordination in order to make the evolution of software systems feasible over long periods.