ToyBlocks offers a controlled testbed for robot learning

Ai2 has introduced ToyBlocks, an open simulation framework designed to study robot learning in a deliberately limited setting. It models a robot arm working with blocks on a tabletop, using realistic physics and appearance while supporting many tasks, including pushing, flipping, sorting, searching and stacking. Images and letters on block faces can require the robot to reveal hidden sides before acting.

ToyBlocks automatically creates short skills and full task demonstrations through reinforcement learning, motion planning and a task-and-motion planner. Hundreds of plans can run in parallel on one GPU, at up to 24 times real-time speed. The generated data can be adjusted to emphasize particular skills or expose gaps in training sets, while evaluation records success, speed, applied forces and task progress.

The framework is intended to examine generalization, planning and transfer to physical robots. Initial tests found that several vision-language-action models generalized poorly to changes in workspace, block arrangements and task sequences, despite large-scale pretraining. ToyBlocks also supports hierarchical experiments in which Molmo 2 selects skills and ManiFlow controls the simulated arm.