1006labs/so101-table-clean-gr00t

SO-101 pick-into-tray policy (GR00T N1.7, MuJoCo sim)

NVIDIA Isaac GR00T N1.7 (nvidia/GR00T-N1.7-3B) fine-tuned on simulated demonstrations of the SO-101 arm in MuJoCo. The policy sees only the wrist camera (the real robot has no other camera) and the joint state, picks one named object into a tray, and comes back home, so tasks can be given one after another on the same table.

This revision (tag pick-r14): run pick-r14, checkpoint 4000, for the scene with the striped wall behind the table (as at the real table). Earlier revisions stay under their tags: pick-r10 (wall), pick-r8 (no wall), pick-r6, pick-r2 (single tasks only), pick-r0, and r0c (a different task).

The policy chaining three tasks

Final-evaluation seeds re-run with video (front camera left, sim only for viewing; wrist camera right, the policy's only image). Each task starts where the last one ended; after each success the policy returns home by itself.

Seed 10003: cylinder, then sphere, then cube (16.4 s).

Seed 10001: sphere, then cube, then cylinder (27.4 s). The front camera (sim only) sits behind the striped wall and sees through its back face.

The scripted expert that labels the training data.

Task

A cylinder, a cube (4×3×4 cm box) and a sphere, all white, on a table next to a fixed tray, with a striped wall along the far edge of the table (stripes ~2.2 cm, measured from a photo of the real table; objects pushed against it are stopped). One of three fixed instructions per task:

• pick up the cube and put it in the tray
• pick up the sphere and put it in the tray
• pick up the cylinder and put it in the tray

A task succeeds when the named object is in the tray; other objects may already be there, so tasks chain ("pick up the cube" → success → "pick up the sphere" from that state).

Inputs and outputs

GR00T NEW_EMBODIMENT, the layout of GR00T's SO100 example with video = wrist only:

	Key	Shape / units
Video	video.wrist	RGB, recorded at 640×480, 30 fps
State	state.single_arm	5: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, degrees (LeRobot use_degrees=True)
State	state.gripper	1: 0 (closed) .. 100 (open)
Language	annotation.human.task_description	one of the three instructions above
Action	action.single_arm, action.gripper	16-step chunk at 30 Hz; arm relative to the current state, gripper absolute (same units)

In evaluation the first 8 steps of each chunk are executed before the next query.

Usage

from huggingface_hub import snapshot_download
path = snapshot_download("1006labs/so101-table-clean-gr00t", revision="pick-r14")

Load path as a GR00T checkpoint with embodiment tag new_embodiment (GR00T's policy server / PolicyClient, or the project's gr00t-runner) and send observations in the format above. Loading the backbone requires access to nvidia/Cosmos-Reason2-2B (gated).

Training

Data: 1006labs/so101-table-clean-mujoco revision pick-r14: 14,413 episodes, 2,043,077 frames (18.9 h) at 30 Hz, every action from a scripted expert that knows the true object poses.

• Scripted episodes (1,389, wall scene): clean table / 1–2 objects already in the tray / random arm pose / the cylinder lying on its side, each recorded on past the drop: the expert lets go, lifts away and returns home.
• DAgger (13,024): the previous policies ran single tasks and three-task chains (pick-r2, pick-r4, pick-r5, pick-r6 without the wall: 7,104 episodes; pick-r8 and pick-r10 in the wall scene: 5,847; pick-r12 on sphere tasks only: 73); at chosen frames the sim was restored and the expert took over, recording only its part: about to miss or push the target, after a failure, right after the policy knocked the cylinder over, at the start of the next task in a chain (the object often still held in the tray), and while coming back home.

Lineage (each fine-tune: 2,000 steps unless noted, global batch 32, LR 2e-5 with 5% warmup and cosine decay, all datasets of the round sampled equally; 1× NVIDIA L40S):

Round	Start from	Added
pick-r2	nvidia/GR00T-N1.7-3B (10k steps, LR 1e-4)	scripted episodes
pick-r4	pick-r2	DAgger from pick-r2: single tasks + chains
pick-r5	pick-r4	DAgger from pick-r4
pick-r6	pick-r5	scripted episodes recorded back home; DAgger from pick-r5 with the return home
pick-r8	pick-r6	the knocked-over cylinder: scripted episodes starting with it lying, and DAgger from pick-r6 right after it tipped the cylinder (cylinder tasks only)
pick-r10	pick-r8	the striped wall: the scripted episodes re-recorded in the wall scene (same seeds), DAgger from pick-r8 in the wall scene; the earlier DAgger data (no wall) kept
pick-r12	pick-r10	DAgger from pick-r10 (wall scene), and 6,000 steps instead of 2,000 (2,000 steps covered ~3% of the frames)
pick-r14	pick-r12	spheres pushed out of reach: DAgger from pick-r12 on sphere tasks only, about to miss or push it and after a failure; 4,000 steps

Evaluation (simulation only)

Closed loop in the same MuJoCo scene on 100 held-out seeds (10000–10099), 30 s per task.

• Single task: start condition mixed per seed (clean table / objects already in the tray / random arm pose).
• Chain: clean table, all three tasks in a row (each of the 6 orders on 16–17 seeds), each from where the last ended; the first failure ends the chain. Chain + return: the same, but after each success the policy keeps going until it is back home (within 10° per joint, at most 10 s), then the next task starts.

Model	Scene	Single task	Chain, all 3	Chain + return, all 3	Returns that reached home
pick-r14 (this)	wall	99/100	–	98/100	242/296 (1.3 s)
pick-r12	wall	96/100	–	94/100	161/285 (1.8 s)
pick-r10	wall	97/100	–	89/100	225/280 (1.2 s)
pick-r8	wall	91/100	–	76/100	172/246 (1.3 s)
pick-r8	no wall	98/100	–	85/100	208/268 (1.4 s)
pick-r6	no wall	90/100	–	80/100	202/250 (1.4 s)
pick-r5	no wall	90/100	87/100	72/100	10/235
pick-r4	no wall	90/100	75/100	–	–
pick-r2	no wall	87/100	11/100	–	–

pick-r14 in detail (wall scene):

• Single task: cube 31/31, sphere 35/36, cylinder 33/33; clean table 37/37, after other picks 29/29, random arm pose 33/34; 1 timeout, no object off the table.
• Chain + return: first task 100/100, first two 98/100, all three 98/100; the target never left the tray on the way home. The 2 failed chains both stopped in the second task: a cube grasp that did not close, and a sphere pushed out of reach.

The policy samples its actions, so the same weights on the same seeds vary by several points between runs (the first task, identical in both chain evaluations of pick-r5, scored 95 and 87).

Limitations

• Simulation only. Not tested on a real SO-101; camera intrinsics, servo response and gripper calibration of the real robot are not matched yet.
• One fixed scene (table, striped wall, tray position, three white objects); instructions outside the three above are out of distribution.
• 98/100 is one run of 100 seeds; the same weights vary by several points between runs (the policy samples its actions), and the result has not been reproduced by a run trained from scratch.

License

Fine-tuned from nvidia/GR00T-N1.7-3B, released under the NVIDIA Open Model License Agreement; these weights are distributed under the same terms.