GAM-WildDet3D

The benchmark-independent weights of GAM (Grounded Action Model): the general WildDet3D detector (SAM3 backbone, text / click prompts, 2D + 3D box heads, depth head) and the universal span head that turns a task string into the object phrases to ground. They are the defaults of the LeRobot / real-robot line of the released code (configs/lerobot_bimanual.yaml): a LeRobot dataset with a task string can be trained on without manual labelling, but these weights are not tuned to any particular scene — on a new robot / camera / object set the detector (and the span head) may need fine-tuning for reliable grounding.

Resolution matters. WildDet3D's native input is about 1k pixels (images are letter-boxed to 1008 x 1008 before the backbone); low-resolution datasets (e.g. 256 x 256 or 224 x 224) are upsampled, small objects lose detail and detection / 3D box quality drops noticeably. Record or choose datasets at 640 x 480 or higher.

Quick Links

• 📄 Paper: Grounded Action Model: 3D Grounding as a Foundation for Robotics
• 💻 Code: GehaoZhang6/Grounded-Action-Model
• 📂 Benchmark releases: GAM-LIBERO-PRO, GAM-RoboTwin

Intended Use

python benchmarks/lerobot/inspect_dataset.py user/dataset                  # cameras, resolution, tasks -> config lines
python benchmarks/lerobot/train.py --config configs/lerobot_bimanual.yaml --mode precompute   # tracks -> feature cache once -> fast training
python benchmarks/lerobot/train.py --config configs/lerobot_bimanual.yaml --mode e2e          # tracks -> perception inside the training loop

Two ways to train. precompute (default) runs the frozen detector over every frame once and stores the features (36x36 patch tokens, 3D boxes, point clouds; roughly 0.3-1 GB per 1000 frames), so training itself is fast — at the price of keeping that cache on disk and rebuilding it when perception settings change. e2e keeps the detector in the training loop (about 0.6-1 s per sample) and stores nothing.

GAM.tools.track runs the span head on each episode's task string, detects the phrases on the first frame with this detector and follows them through the episode with the offline CoTracker; training then prompts the detector with those points frame by frame. The monocular recipe (detector_depth: zero, geometry_depth: pred) needs no depth and no calibration beyond the camera intrinsics in the yaml.

Files

path	what
wilddet3d_general_v1.pt	WildDet3D trained on the general mix including DROID — the default detector for new datasets; extents axis order [h, l, w] (box_whl_perm: [1, 2, 0]) — 4.7 GB
span_head_universal.pt	frozen flan-t5-large encoder + 2-layer transformer head, trained on RoboTwin templates, LIBERO, DROID annotations and ~80k task strings from LeRobot datasets on the Hub — 26 MB

The span head names every object mentioned; GAM.text.phrases.manipulation_objects() then drops reference landmarks ("from the top drawer", "beside the box") so only the objects to act on become slots. Retrain it on your own instructions with python -m GAM.text.train_span_head --labels my_labels.json --init span_head_universal.pt ({"instruction": ..., "span": [object phrases]}).

License

The GAM code is MIT (GehaoZhang6/Grounded-Action-Model). The WildDet3D weights here are fine-tuned from WildDet3D (Allen Institute for AI), which builds on SAM 3 and LingBot-Depth, and are distributed under the SAM License; the policy checkpoints, span heads and caches are MIT.

Citation

@misc{zhang2026groundedactionmodel3d,
  title         = {Grounded Action Model: 3D Grounding as a Foundation for Robotics},
  author        = {Gehao Zhang and Weikai Huang and Shailesh Shailesh and Yiyan Peng and Jiafei Duan and Ranjay Krishna},
  year          = {2026},
  eprint        = {2609.23863},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2609.23863}
}