Automated classification of non-functional requirements twenty years later: why this isn’t a solved problem

This paper revisits “Automated Classification of Non-Functional Requirements”, published in the Requirements Engineering Journal in 2007, and reflects on how the underlying problem has evolved over two decades. The original work addressed the central challenge in requirements engineering that quality concerns are difficult to identify because they are distributed across heterogeneous artifacts and only partially documented. It introduced one of the earliest machine-learning approaches for NFR classification and produced the NFR PROMISE dataset, which became a widely used benchmark in this area. To understand the paper’s long-term impact, we conducted a citation review of 55 papers and replicated representative NFR classification approaches spanning two decades. The results show substantial advances in automated classification, culminating in modern LLM and agent-based techniques that outperform earlier approaches on PROMISE. However, improved accuracy has not solved the underlying problem. Contemporary AI-assisted development increasingly blurs the boundaries between Requirements Engineering, architectural design, and implementation. Quality concerns continue to emerge throughout development, but now appear as conversational fragments, implementation discussions, and architectural tradeoffs rather than explicit requirements statements. We argue that the next generation of requirements engineering tools must move beyond classifying documents toward identifying, surfacing, and preserving architecturally significant quality concerns as they emerge throughout development. We then outline a research agenda centered on detecting emerging and missing quality concerns, selective intervention, living documentation of quality decisions, and evaluation beyond classification accuracy. Finally we take a retrospective look at two decades of sharing and reusing the dataset, including the importance of dataset lineage, experimental comparability, and benchmark evolution.