RoboLab-120 · Simulated Franka Panda
NVIDIA's 120 RoboLab tasks in Isaac Sim, on a simulated Franka Panda with a Robotiq 2F-85 gripper (NVIDIA RoboLab).
FLUX 3 Action
Put one (1) more bananas in the grey bin.
Success in 7 s of simulated time
π₀.₅
Make sure there are 3 (three) bananas in the grey bin.
Success in 7 s of simulated time
Cosmos3 Nano
Put 2 bananas in the crate
Success in 13 s of simulated time
GR00T
Pick up a fruit
Success in 5 s of simulated time
MolmoAct2
Put the red bbq sauce bottle in the crate
Success in 10 s of simulated time
G0.5
Put the mustard in the right bin
Success in 15 s of simulated time
GPT-6 Astra
Take the mugs off the shelf
Success in 157 s of simulated time
| # | Model | Episodes | Reliability @ 300 s | Speed @ 300 s |
|---|---|---|---|---|
| % subtask score | tasks/min | |||
| Loading the benchmark… | ||||
Reliability is RoboLab's mean subtask score in simulated time, over 120 tasks × 3 episodes. OpenAI GPT-6 Astra: Ran on positronic ec20466d with the model on Azure OpenAI (gpt-6-astra, medium reasoning effort) and its inference time charged, to 480 s. Each episode and its replay are cut at its task's budget. OpenAI GPT-6 Astra, paused world: The same run timed without the model's thinking, as if the simulated world had paused while it computed, with the 25 episodes the 480 s cut short rerun in a paused world. Its replays are the same recordings with the thinking cut out, and the reruns'. It has no rank and no Speed. ± is half the 95% range.
How reliable is it at each time limit?
The mean subtask score the episodes had reached within each time limit, with its 95% range over the tasks. An episode that has ended keeps its final score. Drag the marker to rank the table at another limit.
More charts
When do the successes come?
The success marks of each model's episodes, smoothed, as a share of its episodes per second of simulated time.
Is it faster, or more reliable?
Each line is a model at every time limit from 1 s to 300 s: its Speed in tasks/min across, its Reliability up. The dot is the current time limit, and the dashed lines are its 95% ranges. The axis ends at 6 tasks/min, so the quickest successes at a short limit fall beyond it.
Do two models really differ?
Episodes
Each episode and its own result, the newest first.