Ground First, Then Grasp: checkpoints
Language-driven grasp detection split into two experts, trained separately:
a language-free prior (GR-ConvNet, 1.90 M params) and a grounder (CLIPSeg, fine-tuned decoder, 1.13 M trainable of 150.7 M).
Composition: soft product of experts, alpha=0.75, temperature=2, rerank weight=0 (chosen on val, see composition/best_compose.json).
Code: https://github.com/coutMinh/Ground-then-Grasp (commit 9929373).
| File | What | Note |
|---|---|---|
prior/best.pt |
R1 prior, best val epoch | full training state, loads with g2g.eval.evaluate.load_expert |
grounder/best.pt |
R2 grounder, best val epoch (8) | full state incl. frozen CLIP weights, 612 MB |
grounder/decoder_only.pt |
trainable decoder weights only (1,127,009 params) | key decoder_state; needs pretrained CIDAS/clipseg-rd64-refined for the rest; not loadable by load_expert as is |
composition/best_compose.json |
frozen composition + checkpoint hashes |
SHA-256 of each .pt is in SHA256.json.
Results (top-1 grasp success, %, single seed)
| System | Seen | Unseen | H |
|---|---|---|---|
| Prior only | 33.65 | 32.28 | 32.95 |
| Zero-shot CLIPSeg + prior, hard | 32.05 | 33.28 | 32.65 |
| Fine-tuned grounder + hard | 36.41 | 33.44 | 34.86 |
| Mask peak | 18.39 | 13.81 | 15.77 |
| Ours (= soft, lambda=0) | 37.65 | 35.77 | 36.69 |
| Oracle GT mask + prior (diagnostic) | 46.58 | 41.60 | 43.95 |
Test: Seen 1376 rows / 412 images, Unseen 601 rows / 193 images. Success = rotated IoU > 0.25 and angle error <= 30 deg.
Limits
Trained on 8,144 images of Grasp-Anything++ (synthetic images), one seed, one grasp per instruction. No comparison to joint baselines or the official LGD checkpoint. Not tested on real robots or real photos.
Demo (single image + instruction)
demo/infer.py pads the image to a square, runs prior and grounder, and returns one rectangle (x, y, w, h, theta in radians, input pixels).
pip install -r demo/requirements.txt gradio
export HF_TOKEN=<read token> # only while this repo is private
python demo/app.py # Gradio UI
python demo/infer.py image.jpg "Grab the handle of the mug" out.png
Checkpoints are fetched from this repo. infer.py reproduces the saved predictions of the evaluated system on 8 test samples (max coordinate difference 7e-6). Non-square or real photos are outside the training distribution (square 416 px synthetic images); results there are not validated.