Benchmarking Robotic Manipulation under Physically Realistic Dynamics

Accepted at the NeurIPS 2026 Workshop on Robot Learning with World Models
Paper Code Data & assets Arcade
Abstract

Acting in a world that does not wait

Object manipulation in the dynamic real world requires understanding how the environment functions and evolves, and acting accordingly. Existing benchmarks, however, do not adequately address these needs. Physics-oriented benchmarks evaluate predictive fidelity but not closed-loop action. Dynamic robotic manipulation benchmarks often use simplified, physically implausible object motions, limiting evaluation of reasoning and action under realistic dynamics. Evaluation is also typically synchronized, freezing the environment during inference and making computation time irrelevant to task outcomes.

To this end, we propose RoboDyna, a benchmark for evaluating the physical understanding and action capabilities of manipulation policies in dynamic scenes. RoboDyna comprises 20 conceptual dynamic tasks spanning five capability categories, with simple objects and minimal scene clutter, under a two-factor difficulty design, and 10 household tasks that instantiate the same capabilities in visually and semantically richer simulated real-world environments. RoboDyna provides 3,500 expert demonstrations, paired synchronous and asynchronous evaluation protocols, metrics for task success and partial completion, and interactive tools for human baseline evaluation.

Evaluating vision-language-action models and a world-action model across 5.4K episodes, we find that all policies achieve success rates below 20%, revealing substantial limitations in dynamic manipulation. The world-action model captures simple physical dynamics but struggles with complex interactions, and its prediction fidelity does not consistently align with task success. Sync–async comparisons show that inference latency flips 5–12% of episode outcomes while changing average success rates by at most 3%. Human baselines substantially outperform all policies under matched conditions.

The benchmark

RoboDyna at a glance

Every task is a language-conditioned dual-arm manipulation problem in SAPIEN. The scene evolves continuously under gravity, friction, collision and contact, independently of the policy.

Overview of RoboDyna: capability categories, sync vs async, complex physics, conceptual and household tasks, human baselines, and difficulty levels
Fig. 1. 20 conceptual tasks in five capability categories and 10 household tasks with physics-governed motion; four difficulty settings per conceptual task; paired sync/async protocols; and a human study on the same seeds.

Five capability categories

Each category isolates one physical competence. Four conceptual tasks per category use minimal scenes with only task-relevant objects.

Trajectory prediction

Anticipate where a moving object will be, including motion redirected by contact or collision.

  • Catch Ramp Ball
  • Catch Valley Ball
  • Stop Valley Ball
  • Save Goal

Periodic pattern

Identify recurring motion and time actions to short opportunity windows.

  • Catch Marbles Trapdoors
  • Catch Cuboid
  • Drop Ball Hole
  • Whack Moles

Dynamic avoidance

Plan robot motion around independently moving objects and obstacles.

  • Put Cup Belt
  • Hit Target
  • Load Train
  • Place Block Belt

State transition

Track changes not captured by pose alone, such as ripeness, doneness or availability.

  • Cook Meat Timer
  • Pick Ripe Apple
  • Sort Apples Belt
  • Pack Fruits

Static to dynamic

Anticipate motion the robot itself triggers in an initially static scene by pushing, striking or releasing.

  • Catch Shelf Marble
  • Dispense Gummy
  • Marble Shelf Maze
  • Play Billiard

Composable difficulty

Each conceptual task has four conditions: base easy, var1 and var2 medium, each adding one task-specific variation, and var1+2 hard, which composes both and is withheld from training. In Load Train, the robot drops a ball into a wagon of a circling train.

Load train basebaseany wagon
Load train var1var1red wagon only
Load train var2var2tunnels occlude
Load train var1+2var1+2both

Household tasks

Ten office and kitchen scenes test the same capabilities with clutter, richer semantics, and everyday physics beyond rigid bodies: rising liquids, overflow, dispensing and spreading spills.

Boil milk sceneBoil Milk: switch off the stove before the rising milk overflows.
Catch mouse object drop sceneCatch Mouse Object Drop: predict which object the mouse knocks off, then catch it.
Results

No policy solves even one in five episodes

We fine-tune two vision-language-action models (π0.5, X-VLA) and one world-action model (FastWAM) on all 3,500 demonstrations, then evaluate each from a single checkpoint. Easy is base, Medium averages var1 and var2, and Hard is the held-out var1+2 composition. Human participants played the Hard conditions and the household tasks.

Metric
Protocol

Bold marks the best policy in each column. Mean inference latency is shown under each policy name. Overall covers Easy, Medium and household, and excludes Hard. Humans are scored on success only, in real time, so the protocol switch does not apply to them.

Human study

People are still far ahead

Twelve participants played the same tasks on the same seeds: the hard var1+2 condition for conceptual tasks, and household tasks at base. They used two control modes. Teleoperation drives the arm with the keyboard, like a policy. Scene-interactive control acts on objects directly with point and click, removing the cost of low-level control.

77.1%humans, interactive
50.2%humans, teleoperation
14%best policy, same conditions

The gap between control modes is largest on time-critical tasks. Policies do best where tracking a visible object is enough, and fail on periodic patterns, which humans solve 75% of the time.

Human vs policy success by category
Fig. 2. Success rate by category for humans and policies on the human-evaluated slice.

RoboDyna Arcade

Play all 30 RoboDyna tasks in your browser with the same point-and-click controls our human participants used. Five episodes per game, then see your success rate next to π0.5, X-VLA and FastWAM. Simplified 2D physics, not the SAPIEN benchmark.

Launch the arcade →
Get started

Install and explore

git clone https://github.com/robodyna/robodyna.git
cd robodyna
bash script/install_robodyna.sh
conda activate robodyna
# Download the minimal runtime asset package (~1.2 GiB)
bash script/_download_assets.sh

# Explore the benchmark interactively
python interactive/robodyna_gui.py
Citation

BibTeX

@inproceedings{li2026robodyna,
  title     = {Benchmarking Robotic Manipulation under
               Physically Realistic Dynamics},
  author    = {Li, Zhiyuan and Yang, Ruiheng and Zhao, Xuan
               and Rasouli, Amir},
  booktitle = {NeurIPS 2026 Workshop on Robot Learning
               with World Models},
  year      = {2026}
}