Empirical evaluation demonstrates high accuracy and substantial agreement across four large language models classifying software requirements, highlighting feasibility for automated analysis.
The adoption of large language models (LLMs) in software engineering has enabled the potential to automate complex activities such as requirements analysis. This paper presents an empirical performance analysis of four modern LLMs: GPT-4o, Aya, Gemma and Phi-4 on the task of automated classification of atomic software requirements. The LLMs were invoked under two different scenarios to solve a multilabel classification of 296 requirements extracted from the PROMISE[Formula: see text] dataset. The baseline scenario relies solely on internal model knowledge, whereas the rubric-augmented scenario uses formal definitions derived from the SQuaRE product quality model. The results indicate that GPT-4o consistently attains the highest overall classification accuracy under both scenarios. Moreover, all LLMs exhibit strong and stable performance in identifying functional, performance efficiency and security requirements. Inter-rater agreement assessments using Cohen’s and Fleiss’ Kappa coefficients further demonstrate moderate to substantial agreement among the outputs of the evaluated LLMs.
No takes yet. Share an insight, caveat, or question.
Ayed et al. (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: