When we cannot reach something, we grab a stick; when we need to pull a ball closer, we choose a hook. For a humanoid to be useful in everyday life, it has to go beyond the limits of its own body in the same way, with tools. That takes two things at once: the judgment to choose the tool that fits the situation, and the motion to handle that tool all the way to the end, walking with it when needed.
Existing benchmarks did not evaluate these two together on a humanoid. So we built HumanoidToolBench.
What it measures
HumanoidToolBench has three scenarios.
- BallMove: push a ball with a long stick into a goal 0.12 m across. Reach matters.
- BallRetrieve: hook the far side of a ball and pull it in. The tool's shape matters.
- IceBreak: strike two ice blocks hard enough to break them. The tool's material matters.
Each scenario has three levels. L0 is choosing the suitable tool and lifting it, L1 is finishing the task with the tool without moving, and L2 is a task that can only be finished by moving while holding the tool. There are also two modes. In Standard mode the suitable tool sits among unrelated objects; in Decoy mode a competing tool that differs only in the property that matters is added, such as a 0.30 m stick next to the 0.50 m one, or a fly swatter next to the metal hammer. Three scenarios × three levels × two modes gives 18 tasks.
The tools are 55 assets from three functional categories. Simulation runs in MuJoCo, and the real robot is a Unitree G1.
ToolBook: the demonstrations
We also built ToolBook, a demonstration dataset for training and evaluation: 3,094 trajectories collected by eight experienced operators teleoperating with a Meta Quest, 3,003 in simulation and 91 on the real G1.
What we found
We evaluated seven policies in simulation and three on the real robot. The clearest result is this: there is a large gap between choosing the right tool and completing the task. Success drops further when a task needs locomotion and manipulation together.
A single final success rate hides this gap. So we recorded interaction events such as contact, whether the correct tool was lifted, and whether the tool touched the target. They let us tell whether a failure came from not picking the tool up, or from touching it often but never finishing. The tool a policy first chose often differed from how the episode ended.
A closer look at GR00T N1.7 showed lower selection accuracy on tools it had not seen in training, and that it kept executing the same task even when the instruction was changed to something unrelated.
What this means for my research
I see what drives a robot as one loop. System 2 reasons about what to do, System 1 turns that into action, and System 0 controls the body. HumanoidToolBench shows that, in this loop, knowing what to pick up and actually getting it done with that tool are different abilities.
That is why I think we need action models that understand actions themselves deeply, an idea I wrote about in Agentic AI for Robotics and Action Generation. Tying the judgment of choosing a tool and the motion of finishing with it into one system is the next step.
Paper: arXiv 2610.02089 · Project page and data: HumanoidToolBench
