IDI-Bench: A Benchmark for Evaluating Intent Understanding, Capability Selection, and Autonomous Internet Task Execution

The increasing development of generative artificial intelligence (GenAI), autonomous agents, search engines, and interconnected Internet services has created new possibilities for systems that can interpret human intentions and autonomously coordinate heterogeneous digital capabilities. However, evaluating such systems remains a challenging research problem. Conventional artificial intelligence benchmarks frequently emphasize answer accuracy, reasoning performance, or task-specific completion, while web-agent benchmarks increasingly examine interaction with websites and multi-step task execution. Nevertheless, evaluating an intent-driven Internet system requires a broader perspective that considers not only whether a requested task is completed, but also whether the system correctly understands the user's underlying intention, identifies the required capabilities, selects an appropriate execution strategy, coordinates multiple capabilities, and verifies the resulting outcome. This paper proposes IDI-Bench (Intent-Driven Internet Benchmark), a benchmark framework designed to evaluate the fundamental capabilities of intent-driven human–Internet interaction systems. The proposed benchmark treats human Internet requests as outcome-oriented tasks rather than isolated questions or predefined tool commands. It introduces a multidimensional evaluation structure covering eight task categories: information retrieval, content generation, comparative analysis, multi-step research, transactional tasks, hybrid tasks, ambiguous tasks, and dynamic tasks. These categories are designed to represent different levels of intent complexity, capability requirements, execution dependencies, and environmental uncertainty. The proposed benchmark introduces eight principal evaluation dimensions: Intent Understanding Accuracy, Capability Selection Accuracy, Task Completion Rate, Execution Efficiency, Evidence Traceability, User Intervention Rate, Recovery Success Rate, and End-to-End Task Utility. Unlike evaluation approaches that primarily assess final outputs, IDI-Bench separately examines the quality of intent interpretation, capability-selection decisions, execution trajectories, evidence relationships, and task outcomes. It further introduces task-level annotations, capability requirements, execution-state records, failure classifications, and reference evaluation procedures to support reproducibility and comparative analysis. A central feature of IDI-Bench is its separation of task outcome quality from execution-process quality. This distinction makes it possible to identify systems that produce correct answers through inefficient or inappropriate execution strategies, as well as systems that select appropriate capabilities but fail to complete their intended tasks. The benchmark also proposes controlled experimental conditions for evaluating single-capability execution, hybrid capability orchestration, clarification strategies, dynamic replanning, and failure recovery. This paper develops the conceptual foundations, benchmark architecture, task taxonomy, evaluation metrics, annotation methodology, experimental protocols, and research propositions of IDI-Bench. It also examines the benchmark's potential contributions to autonomous Internet computing, intelligent capability orchestration, and the empirical development of Intent-Driven Interaction. The proposed framework is a benchmark research design and does not claim that a completed dataset, benchmark implementation, or experimental performance advantage has already been established.