ToyBlocks

Molmo2-4B-tb

Molmo2-4B-tb is the high-level model of ToyBlocks, a robot block-stacking project. It is Molmo2-4B fine-tuned to read a stacking goal, look at the robot's front camera and choose the next step. It states the goal's progress, names the next step and points to the block or spot to act on. A low-level model, such as a motion planner or a learned policy, then carries out the step.

Quick links:

Model details

Model type: vision-language model with about 4.5B parameters, running at about 6 Hz.
Input: one 640x360 front-camera image and a text goal.
Output: a short plan, then the action and its target as a Molmo2 point.
Actions: pick, place (on a block or the table), flip (left, right, forward, backward), rotate (cw, ccw) and done.
Training: 6,000 steps at batch size 256 on simulated ToyBlocks three-block stacking episodes. We mix in 25% image-pointing data (PixMo-Points, PixMo-Count, CoSyn-Point) to retain Molmo's general pointing ability.
Files: model.pt (float32 weights in the native Molmo2 format, not Transformers) and config.yaml (full model and training configuration).
Developed and shared by: Ai2. Funded by: Fund for Science and Technology.
Model date: November 2026. Language: English. Fine-tuned from: Molmo2-4B.

Prompt format

The goal lists the tower from top to bottom:

stack A 4 B. plan. point to target. predict action.

now: A not on 4, 4 on B. next: move A to 4. Do: block A <points coords="1 1 367 623">block A</points> pick

The question can also start with the previous step, in the same format as the answer:

previous action: block A <points coords="1 1 367 623">block A</points> pick. stack A 4 B. plan. point to target. predict action.

Points are in 0–1000 of the image width and height. Each target is 5 px (in a 640x360 frame) below the block's center, so it points to the center of the block face in front of the camera.

Here is the Molmo2 output when combined with the low level Maniflow-tb running on a real robot.

Quick start

hf download allenai/Molmo2-4B-tb --local-dir Molmo2-4B-tb

Inference and evaluation code is in the ToyBlocks repository.

Software: transformers, torch>=2.3.1, torchvision, omegaconf, tokenizers, sentencepiece, einops, huggingface_hub, cached_path and accelerate.
Hardware: about 20 GB of GPU memory in bfloat16.

Evaluation

Closed-loop success rate (%) in ToyBlocks simulation, with Molmo2-4B-tb as the high-level model and either a motion planner or a learned ManiFlow policy as the low-level model. There are three test sets: Easy tests stacking, Medium tests re-orienting blocks and stacking, and Hard tests searching for blocks, re-orienting and stacking. Each evaluation set has exactly 200 episodes. Seen: blocks seen in training, in new configurations. Unseen: held-out blocks.

Level	Motion Planner, seen	Motion Planner, unseen	ManiFlow, seen	ManiFlow, unseen
Easy	97.5	81.0	55.5	44.5
Medium	53.5	36.5	35.5	12.5
Hard	31.5	14.5	17.0	10.0

On a real robot, with ManiFlow as the low-level model, it succeeds in 60% of 10 trials. The main metric is task success; the tech report also measures optimality (how efficiently the task is completed).

License and use

This model is licensed under Apache 2.0 and is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. Its base model, Molmo2-4B, is trained on third-party datasets subject to academic and non-commercial research use only. Please review them to decide whether this model fits your use case.

Direct use: robotics research, in simulation and on real robots. Expected outcomes are described in the tech report.
Downstream use: fine-tuning on your own training data, or pairing the model with your own planning systems for skill execution. See the ToyBlocks repository for instructions.
Out-of-scope use and misuse: anything other than robotics research. The model's scope is moving blocks on a table.

Training details

Training data and preprocessing: see the ToyBlocks repository.
Settings: all training settings are in config.yaml.

Technical specifications

The training objective is next-token prediction of the plan, the action and the target point (which block to act on).

Vision encoder: SigLIP 2 so400m/14 at 378 px, a 27-layer, 1152-wide ViT.
Connector: features from two ViT layers (third- and ninth-from-last), pooled with 2x2 attention pooling (mean query), then an MLP projector into the language model.
Language model: Qwen3-4B, dense: 36 layers, width 2560, 32 query heads with 8 key/value heads (grouped-query attention, head dim 128), RoPE, RMSNorm with QK-norm and a SwiGLU MLP.

Environmental impact

Hardware: one node with 8 NVIDIA H100 80GB GPUs, on Ai2's in-house compute cluster.
Hours used: about 8.
Carbon emitted: the node emits about 3.84 kg CO2e per hour (8 kW × 1.2 PUE × 0.4 kg CO2e/kWh), so the 8-hour run emits 30.7 kg, doubled to about 61 kg CO2e per run.

Model card safety disclosure

This AI model has not undergone safety benchmarking and must be used exclusively in accordance with the provided research protocol.

Citation

Preliminary reference:

@techreport{toyblocks2026,
  title       = {{ToyBlocks}: A Minimalistic {FoundationWorld} for Robot Foundation Models},
  author      = {{ToyBlocks Team}},
  institution = {Allen Institute for AI},
  year        = {2026},
  type        = {Technical Report},
  note        = {Draft},
  url         = {http://allenai.org/blog/toyblocks-techreport.pdf}
}