From LLMs to AI agents: a systematic benchmark for SAO structure extraction in patent analytics

The continuous growth of patent literature has placed new demands on technological innovation and automated processing, particularly in the task of extracting Subject-Action-Object (SAO) relationships. This study constructs an information extraction benchmark framework tailored for the collaborative robotics domain, comparing three large language models—GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet—and their corresponding agents. The experimental design employs a closed-loop “reflection and refinement” model, integrating the thought chain and ReAct method strategies, and configures decoding parameters to prioritize factual quality over linguistic diversity in extraction results. The experimental results indicate that the agent architecture has a positive impact on the performance of all models. Among them, the Gemini 1.5 Pro agent performed the best, achieving an F1 score of 89.58% (95% confidence interval: 87.9%, 91.2%) and a BERTScore of 87.73%. A t-test statistical significance analysis based on 501 independent test sentence samples confirmed that the agent significantly outperforms baseline large language models, particularly in the parsing of recursively nested clauses and the filtering of legal jargon. Resource consumption analysis indicates that although the agent framework results in a 40- to 60-fold increase in computational cost—consuming an average of 11,000 to 15,000 tokens per sentence—this overhead is justified in specific patent analysis scenarios due to its superior recall performance and risk mitigation effects. Furthermore, by classifying typical failure modes and proposing a hierarchical deployment strategy, this study provides a methodological framework and empirical foundation for deploying and applying the agent in specific technical domains.