PhAIL – About

Where this is going

Today, PhAIL is a leaderboard with 4 benchmarks. The larger goal is that PhAIL does the real-robot evaluation of VLA (vision-language-action) models, and you do the research. A lab can publish its results on the leaderboard, or keep them private. The harness that we run will become a toolkit that you can run yourself.

See what we are building →

Read the paper

Our paper on arXiv gives the PhAIL method: what PhAIL measures, and why.

Benchmarks

Each benchmark has a page with its ranking and its charts.

DROID: Everyday tasks. Language-instructed tabletop tasks, each worded as a DROID annotation, on a Franka FR3 (DROID).
Pick and Place, v1.0. Bin-to-bin pick and place of household objects on a Franka FR3 with a Robotiq 2F-85 gripper (DROID).
RoboLab-120. NVIDIA's 120 RoboLab tasks in Isaac Sim, on a simulated Franka Panda with a Robotiq 2F-85 gripper (NVIDIA RoboLab).
MolmoSpaces-320. 320 tasks of Ai2's MolmoSpaces benchmarks in MuJoCo, on a simulated Franka FR3 with a Robotiq 2F-85 gripper (Ai2 MolmoSpaces).

DROID: Everyday tasks has 286 tasks. An operator watches each episode and marks each stage that the robot reaches. At the end, the operator gives a verdict: Success, Fail, Ran out of time, or Safety. The console stops an episode at 240 s. An annotator then reviews the recording. Where the annotator labelled an episode, the annotator's marks decide.

In Pick and Place, the robot moves a known number of items from one tote to a second tote. Each run uses one object. The v1.0 release page gives the full protocol.

RoboLab-120 and MolmoSpaces-320 run in simulation. On RoboLab-120, the DROID benchmark's models and GPT-6 Astra run NVIDIA's 120 RoboLab tasks in Isaac Sim, on a simulated Franka Panda with a Robotiq 2F-85 gripper. RoboLab checks each episode and scores its subtasks itself, so no person marks it. The simulated world keeps moving while the policy computes, as the real world does. GPT-6 Astra's paused-world entry, where the world waits for the model, is shown for reference and has no rank.

MolmoSpaces-320 runs the same models in simulation on 320 tasks of Ai2's MolmoSpaces benchmarks, in MuJoCo, on a simulated Franka FR3 with a Robotiq 2F-85 gripper. Each model runs each task once. MolmoSpaces checks each episode itself, so no person marks it, and Reliability is the share of tasks that pass the check. The simulated world keeps moving while the policy computes.

Metrics

Each benchmark gives a model two numbers, Reliability and Speed. The leaderboard ranks the models by Reliability. Both numbers count only what the model does within a time limit. On a benchmark page, you can move the time limit.

DROID: Everyday tasks. The leaderboard shows both numbers at 240 s.

Reliability – Percent of the task done, from the ranking model: each model gets a skill and each task a difficulty, fitted to how far every episode got within 240 s. Reliability is the expected credit over the tasks: 1 for Success, 0.4 for At target, 0.3 for Control, 0.1 for Contact, else 0. A task of several steps scores the steps done plus the credit of the current step, over the number of steps.

Speed – Tasks per minute while the model succeeds: 60 over the mean time to the success mark, over the successes within 240 s.

Pick and Place. The leaderboard shows both numbers at 120 s.

Reliability – Percent of items placed, each within 120 s of the previous item.

Speed – Items per minute while the model places them: 60 over the mean time of the items placed within 120 s.

RoboLab-120. The leaderboard shows both numbers at 300 s.

Reliability – RoboLab's own partial credit: the mean subtask score each episode had reached within 300 s of simulated time, in percent. Each task has its own budget of 20 to 300 s, so at 300 s every episode has ended and the number is the mean final score, the score of NVIDIA’s leaderboard.

Speed – Tasks per minute while the model succeeds: 60 over the mean simulated time to success, over the successes within 300 s.

MolmoSpaces-320. The leaderboard shows both numbers at 46 s.

Reliability – The share of tasks that pass MolmoSpaces' own success check within 46 s of simulated time. Each task has its own time limit of 20 to 45 s, so at 46 s every episode has ended and the number is the success rate.

Speed – Tasks per minute while the model succeeds: 60 over the mean simulated time to success, over the successes within 46 s.

On Pick and Place, each object has the same weight.

On DROID: Everyday tasks, the leaderboard's Reliability comes from a ranking model. The model works like the scoring of an exam: each model gets a skill, and each task gets a difficulty. The chance that an episode gets at least as far as a stage in a step rises with the skill minus the difficulty, and each model has its own spacing of the stages. Each step of a task is one more try on the same stages. The fit picks the skills and difficulties that make the recorded episodes most likely. A model's Reliability is then its expected credit over the tasks, where a task of several steps scores the steps done plus the credit of the current step, over the number of steps. The episode lists name each task's difficulty by how much of the task a model of average skill gets done: Easy 60% or more, Medium 40 to 60%, Hard 20 to 40%, Very hard under 20%. An episode that the operator stopped early keeps how far it got. The charts on the benchmark page show the episodes as recorded, over the tasks that every model ran.

Each number shows ± half of its 95% range. The range comes from 1,000 bootstrap draws, and each draw samples the data again, with replacement. On Pick and Place, a draw samples the runs of each object. On DROID: Everyday tasks, a draw samples the tasks and fits the ranking model again, and all the models use the same draw.

Two models share a rank when the data cannot separate them. On Pick and Place, a logrank test compares the item times of two models, and it must give p below 0.05 after a Bonferroni correction: p times the number of model pairs. On DROID: Everyday tasks, a model ranks above another when it has the higher Reliability in at least 95% of the 1,000 draws. A shared rank shows as a range, for example 2–3.

Getting started

To see the v1.0 fine-tuning dataset, run one command:

uv run --with positronic \
  python -m positronic.cfg.phail.v1_0

pip install positronic
python -m positronic.cfg.phail.v1_0

The leaderboard shows the scores of each model on each benchmark. On each benchmark page, you can see the ranking and the charts at a different time limit.
The v1.0 release page gives the model weights, the recorded evaluation runs, the protocol, the hardware specification, and the steps to reproduce the results.
To get an evaluation, read the options on the eval page, or send an email to [email protected]. The results can go on the leaderboard, or stay private to your lab. For v1.0, we do the onboarding by hand.

Pick and Place release: v1.0

The task is bin-to-bin order picking of towels, wooden spoons, scissors, and batteries. The hardware is one Franka Research 3 arm with a Robotiq 2F-85 gripper and two Stereolabs ZED cameras. The release evaluates OpenPI π_0.5, NVIDIA GR00T N1.6, Hugging Face SmolVLA, and Action Chunking Transformer.

The v1.0 release page gives the full hardware specification, the protocol, the dataset, the model weights, and the steps to reproduce the results. The v1.0 white paper gives the evaluation method of the release.

RoboLab credits and licences

RoboLab-120 runs in NVIDIA RoboLab v0.2.0, © 2026 NVIDIA Corporation & affiliates, on Isaac Sim. RoboLab's code is under the Apache 2.0 licence. Its 3D assets, which every RoboLab replay and video on this site shows, keep their own licences:

The scenes, fixtures and materials, the simulated robot, and the basic, fruit and vegetable, and Objaverse-derived objects: © 2026 NVIDIA Corporation, CC BY-NC-SA 4.0.
Objects derived from YCB-Video (© 2017 Yu Xiang): MIT. VOMP objects (© 2026 NVIDIA Corporation): CC BY 4.0. Backgrounds from Poly Haven: CC0 1.0.

The replays and videos are renders of these assets from the simulator's cameras, unchanged except for what the robot does to them. We share them under CC BY-NC-SA 4.0, for non-commercial use. RoboLab's third-party notices list every asset folder and its licence.

MolmoSpaces credits and licences

MolmoSpaces-320 runs in Ai2 MolmoSpaces, © 2026 Allen Institute for AI, on MuJoCo. MolmoSpaces' code is under the Apache 2.0 licence. Its assets, which every MolmoSpaces replay and video on this site shows, keep their own licences:

The objects converted from Objaverse: ODC-BY 1.0. Some of these objects keep a non-commercial licence of their own, CC BY-NC or CC BY-NC-SA.
The other scenes and objects: CC BY 4.0.
The simulated Franka FR3 and Robotiq 2F-85 come from MuJoCo Menagerie, with changes by Ai2: the FR3 by Franka Robotics, the 2F-85 © 2013 ROS-Industrial.

The replays and videos are renders of these assets from the simulator's cameras, unchanged except for what the robot does to them. We share them under CC BY-NC-SA 4.0, for non-commercial use. MolmoSpaces' README lists the data licences and attributions.

Open consortium

An open consortium governs PhAIL, so no single company controls what PhAIL measures or how. Nebius is the founding compute partner, and Toloka is the data partner. Partners and how to join →

Contact

Send questions, feedback, or a model for evaluation to [email protected].