Picture a group chat where friends are planning dinner. One person says they want Thai food, another suggests meeting in San Francisco, and a good while later someone asks the AI to find a restaurant. The name of the function the AI should call, and its arguments, are not sitting together in one sentence. They are spread across several people and several rounds of conversation.
Most function-calling benchmarks, however, cover the case where one person states everything needed in a single turn. That is far from how tools get used in practice. So we built DICE-BENCH.
Measuring what is hard first: DICE-SCORE
Before building a benchmark, we wanted to measure how much of the real situation existing benchmarks capture. So we made a metric called DICE-SCORE. It measures how widely tool-related information, such as the function name and parameter values, is dispersed across a dialogue. The higher the score, the more scattered the information, and the harder the function call.
Measured this way, existing benchmarks score notably low: their information sits almost in one place. DICE-SCORE also correlates strongly (r ≈ -0.984) with how well people do on the same task. The dialogues that score higher are harder for humans too.
How it was built
DICE-BENCH builds dialogues in three stages.
- Tool graph construction: the dependencies between tools are modeled as a graph, so the result of one round can become the input of a later one.
- Scenario configuration: the number of rounds, the number of participants, and each participant's persona are set.
- Dialogue simulation: several agents with distinct personas actually talk to each other, which makes the conversation natural.
The dialogues were then filtered three times. First, G-Eval with GPT-4o scored six criteria, including coherence, fluency, and persona consistency, and dialogues averaging below 4.0 were removed. Next came rule-based filtering, with the authors manually reviewing ambiguous cases. Finally, human evaluators scored 15 sub-criteria and only dialogues with 10 or more were kept. Of the initial 1,800 dialogues, 193 were filtered out, leaving 1,607.
Results: still a long way to go
We evaluated 19 LLMs. The number that stands out most: even GPT-4o reaches only 64% exact match on average. And performance drops as the number of rounds or the number of participants grows.
Models that do well when one person says everything at once lose track when the information is spread over people and time. Significant advances are still needed before such models can be deployed effectively in real settings.
What this means for my research
I now work on robots. A group chat and a robot seem far apart, but to me the question carries over: when the needed information does not arrive neatly in one place, can a model still use the right tool with the right arguments? In a chat, that information is scattered across what several people said. For a robot, it is scattered across the scene in front of it and what it has done so far.
HumanoidToolBench, which we built after DICE-BENCH, measures whether a humanoid can choose the right tool and then finish the job with it. The judgment of choosing a tool remains a hard problem, for agents that talk and for agents that have a body.
Paper: arXiv 2506.22853, project page: DICE-BENCH, code: GitHub, data: Hugging Face
