Skip to main content

Action Space

Research, ideas, and everyday notes on AI that acts.

📅 November 23, 2026 · 7 min read

Running HumanoidToolBench Zero-Shot with a Physical AI Agentic Loop

Physical Agentic AIHumanoid Tool UseLLM Agents
The same scene (BallMove L1) solved with two loops and two models. 2× speed. The runtime clock counts the episode's real elapsed time evenly from 0.

Every policy we evaluated on HumanoidToolBench was trained on ToolBook demonstrations. This time we looked at how far an LLM agent with no training gets when it drives the humanoid directly. The results follow four questions.

How we ran it

The agent looks at the head-camera image and the joint state, sends robot commands, and looks at the result again. I call this loop a physical AI agentic loop. The commands are about as low as it gets: inverse kinematics to move a palm to a point, finger poses, walking and turning, and a function that locates tools. The docs give a recipe for picking a tool up and none for using it.

We ran the loop in two ways.

  • CAP (Code-as-Policies): from one image, the model writes one Python program and runs it. Up to three tries per episode.
  • RUA (Robot-Use Agent): the model runs one command at a time and reads each result before choosing the next. Claude Code and Codex run on the robot as they are.

The models were Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra, with three episodes on each of the 18 tasks and 60 s of robot motion per episode. The simulation pauses while the model thinks.

Main question: Does a zero-shot agent beat a policy trained on the task?

At picking the tool up, yes. At using it, no.

HumanoidToolBench leaderboard with the success rate, runtime and cost of seven trained policies and six zero-shot LLM agents
  • At L0 (choose the right tool and pick it up), the agents did better. Sol and Astra with RUA succeeded in all 18 episodes. The best trained policies, GR00T N1.7 and FastWAM, reach about 73% at L0.
  • At L1 and L2, where the tool has to be used, 1 of 216 episodes succeeded. FastWAM reaches 50% at the same levels and GR00T N1.7 38%.
  • Ranked by overall success, the agents land in the middle, but nearly all of that score comes from L0.

Question 1: How do CAP and RUA differ, and the models?

The two loops fail for different reasons

The loop mattered as much as the model. GPT-6.1 Sol succeeded at L0 in 2 of 18 episodes with CAP and in all 18 with RUA.

  • CAP has no chance to revise a decision written in advance. Sol and Astra added a check to their programs: stop if the IK does not converge. With the palm facing down, the hand sags about 4 cm even when the IK reports convergence; raising the target by that much drives the wrist into its limit, and the IK fails. The programs stopped themselves before the hand ever closed.
  • RUA can correct itself, but only locally. The same models read the same warning, carried on and picked the tool up every time. But when a step stalled they sent the same step again, and when the tool slipped they picked it up and carried it the same way. Every retry came out of the 60 s budget.

Each model has its habits

  • Claude Opus 5.5 looks at the image often and changes its plan often. The one L1 success came from it: when sidestepping stalled, it turned its body to realign the ball, the stick and the ring, and swept the ball in. It also gives up early with time left, and it is the only model that picked the wrong tool. Asked to pull the ball back, it took the straight stick instead of the hook in 2 of 3 decoy episodes.
  • GPT-6.1 Sol follows the documented recipe down to the numbers. When it reports a failure, it stops at the symptom ("it slipped") and does not trace it to the command it sent.
  • GPT-6 Astra behaves almost exactly like Sol at about eight times the cost. With RUA over 54 episodes, both succeeded 18 times; Sol cost $24 and Astra $198.
RUA, Claude Opus 5.5. BallRetrieve L2 with a decoy. The hook lies on the left, and it takes the straight stick. 4× speed.

Question 2: How far do agents get, and where is the bottleneck?

The tool usually reaches the hand and the target. Then the hand fails first, and the reasoning does not change course.

  • Grip: a one-hand pinch lets the tool rotate or fall while it is carried.
  • Tool tip: agents place the hand, not the tip of the stick or the head of the hammer, so sweeps and strikes miss.
  • Force: commands set position, not force or speed. Lowering a hammer slowly does not break ice.
  • One hand: every model does everything with the left hand, even when the goal is on the right.
  • No change of strategy: the symptom is named correctly, then the same approach is retried with new numbers.
RUA, GPT-6.1 Sol. It picks the stick up, then drops it while carrying it sideways. 4× speed.
RUA, Claude Opus 5.5. IceBreak L1. One block breaks; the second survives every strike. 4× speed.

Question 3: How long does it take, and what does it cost?

The robot moves for 60 s, but with the wait for model responses an episode took up to 23 minutes. That works only because the simulation waits. A real robot does not. The API cost of one episode ranged from $0.29 to $3.86 at the median.

Setup L0 L1 and L2 Runtime (max) Cost per episode Cost per success Total cost
CAP, Opus 5.5 14/18 0/36 6.3 min $1.35 $4.24 $59
CAP, GPT-6.1 Sol 2/18 0/36 10.6 min $0.29 $7.90 $16
CAP, GPT-6 Astra 2/18 0/36 6.9 min $1.51 $40.12 $80
RUA, Opus 5.5 16/18 1/36 23.0 min $1.40 $4.44 $76
RUA, GPT-6.1 Sol 18/18 0/36 11.7 min $0.46 $1.31 $24
RUA, GPT-6 Astra 18/18 0/36 14.2 min $3.86 $11.00 $198

What I think

A low-level control model is needed. GUI agents needed little physical knowledge, and time did not matter. Tool use needs both. However good the LLM, it needs a separate model for low-level control. Whether that is classical control such as PD and IK or a neural action generator such as a VLA is still open. An LLM that could handle force and walking itself would solve this, but running a large LLM for every contact is wasteful. A model specialized in control is the better split. People work this way too: we decide what to do without reasoning through every muscle.

The reasoning failures are harness problems. I see keeping the same strategy as an implementation problem rather than a limit of the model: the agent has no memory of what it tried before. Getting no feedback while a command runs and repeating the same order are limits of the agentic pipeline too. The harness should supply that memory and that feedback.

Time alone calls for System 2 and System 1. A real robot does not wait 23 minutes. A slow planner and a fast controller have to run together.

The full analysis and tables are on the HumanoidToolBench project page.

CAP and RUA agents with Claude Opus 5.5 and GPT-6.1 Sol on the same HumanoidToolBench scene, shown in a 2 by 2 grid