What makes robotics benchmarks comparable?

Start with the question

Comparing two models on one robot is different from comparing two complete robot systems. In the first case, changes to the gripper, camera, or control interface can make it harder to attribute a difference to the model. In the second, those differences are part of what you want to measure. Both comparisons can be useful, provided the report makes the distinction clear.

A benchmark should state which of these questions it answers before presenting a ranking. A result on one arm and one set of objects does not, by itself, describe performance across other robots or tasks.

Define more than the task name

“Put the object in the container” leaves a lot unspecified. The object might start near the gripper or at the edge of its reach. The container might be open or need to be opened. A trial might count as successful when the object crosses its rim, or only after the robot releases it and it stays inside.

A practical comparison needs a written success condition, a time limit, and a rule for choosing starting conditions. The rules do not have to make every trial identical. They do need to give each model comparable opportunities. A fixed set of starts and a shared sampling procedure are two ways to make that explicit.

Keep a record of the setup

Record the robot and gripper, camera placement, model checkpoint, instructions, and control settings alongside the score. If something changes during evaluation, keep that change in the record. Otherwise, a later reader cannot tell whether they are comparing models or comparing setups.

SIMPLER is a useful example of why this matters. Its authors identify visual differences and differences in control behavior as obstacles to matching simulated and real robot evaluations. They evaluate how well simulation reproduces real-world policy performance and behavior under changes to the environment.

Source: Li et al. — Evaluating Real-World Robot Manipulation Policies in Simulation (2024).

Include the attempts that went wrong

A report should include the number of attempted trials as well as the success rate. The same percentage based on a few attempts and on many attempts carries different uncertainty. Looking at results by task also helps explain an overall score: two systems can reach the same average while failing on different tasks.

Decide in advance how to handle a timeout, a dropped object, a safety stop, or help from an operator. If a trial is excluded because the equipment failed, record that exclusion and its reason. Keeping only the clean runs would answer a different question from whether the system can complete the task reliably.

Check what simulation predicts

The SIMPLER paper uses paired real and simulated evaluations to test whether its environments reflect policy performance on physical robots. That relationship is something to evaluate for a setup, rather than assume from a simulation score alone.

A useful report can show the simulated result, the physical result, and the conditions for each. When they disagree, the disagreement is worth documenting. It gives the next experiment a more specific question than simply asking which model is best.

Source: Li et al. — Evaluating Real-World Robot Manipulation Policies in Simulation (2024).

What to publish with a score

A compact evaluation record should let someone else understand what the score means:

• The task, success condition, starting conditions, and time limit.
• The robot, sensors, model version, and control settings.
• Attempt counts and outcomes by task, including interventions and exclusions.
• Whether the trials ran in simulation or on physical hardware, with enough setup detail to interpret the comparison.

I'm a co-founder of Coop, where we're building standardized, third-party robotics benchmarks for humanoid and manipulation model developers. Giving different models a shared set of tasks is the starting point; the accompanying evaluation record is what makes a score useful to someone else.

References

• Li et al. — Evaluating Real-World Robot Manipulation Policies in Simulation (2024)

  Primary research on paired real and simulated evaluations; the project materials describe visual matching, control matching, and policy-ranking comparisons.

Related work

• Coop

  Standardized benchmarks for humanoid and manipulation models.

• BCI & assistive robotics

  EEG motor-imagery pipelines and robotic-arm systems for assistive feeding.