CompToolBench: A Compositional Benchmark for Fine-Grained Tool-Use Evaluation of Large Language Models

Key Points

• This research aims to evaluate the performance of large language models (LLMs) in using tools across various compositional complexities.
• Introduced CompToolBench as a benchmark for tool-use evaluation.
• Assessed 18 LLMs drawn from cloud and local deployments.
• Evaluated 200 tasks using 106 public APIs across four composition levels.
• Identified performance gaps in tool selection and call validity.
• Found an average Selection Gap of 13.2 percentage points in tool call validity.
• Compositional complexity does not consistently degrade model performance.
• Local models are now nearing the accuracy of cloud models on structured tasks.