Instructions to use mickeykang/smolvla-multiframe-DOM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use mickeykang/smolvla-multiframe-DOM with LeRobot:
# See https://github.com/huggingface/lerobot?tab=readme-ov-file#installation for more details git clone https://github.com/huggingface/lerobot.git cd lerobot pip install -e .[smolvla]
# Launch finetuning on your dataset python lerobot/scripts/train.py \ --policy.path=mickeykang/smolvla-multiframe-DOM \ --dataset.repo_id=lerobot/svla_so101_pickplace \ --batch_size=64 \ --steps=20000 \ --output_dir=outputs/train/my_smolvla \ --job_name=my_smolvla_training \ --policy.device=cuda \ --wandb.enable=true
# Run the policy using the record function python -m lerobot.record \ --robot.type=so101_follower \ --robot.port=/dev/ttyACM0 \ # <- Use your port --robot.id=my_blue_follower_arm \ # <- Use your robot id --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \ # <- Use your cameras --dataset.single_task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording --dataset.repo_id=HF_USER/dataset_name \ # <- This will be the dataset name on HF Hub --dataset.episode_time_s=50 \ --dataset.num_episodes=10 \ --policy.path=mickeykang/smolvla-multiframe-DOM - Notebooks
- Google Colab
- Kaggle
FINAL: 10-epoch checkpoint (step 735270)
Browse files- README.md +13 -8
- config.json +1 -1
- model.safetensors +1 -1
README.md
CHANGED
|
@@ -9,18 +9,21 @@ tags:
|
|
| 9 |
- dom
|
| 10 |
---
|
| 11 |
|
| 12 |
-
# SmolVLA-MultiFrame on DOM —
|
| 13 |
|
| 14 |
Multi-frame **SmolVLA** fine-tuned on the **DOM** (Dynamic Object Manipulation) dataset.
|
| 15 |
-
|
|
|
|
| 16 |
|
| 17 |
## What this is
|
| 18 |
- **Backbone:** `lerobot/smolvla_base` — SmolVLM2-500M-Video-Instruct (SigLIP vision + SmolLM2)
|
| 19 |
-
+ flow-matching action expert. VL-aligned + robot-pretrained.
|
|
|
|
| 20 |
- **Training:** **full fine-tune** (403M / 450M trainable, vision encoder unfrozen) on `hzxie/DOM`
|
| 21 |
-
(Franka, cameras `opst_cam` + `wrist_cam`, state 6-d, action 7-d, chunk 50).
|
| 22 |
- **Multi-frame:** temporal window **{t-2, t}** (`DELTA_TIMESTAMPS observation: [-2, 0]`) — each frame is
|
| 23 |
-
fed to SmolVLM2 as a **separate image** so the model perceives object motion (DOM is dynamic
|
|
|
|
| 24 |
- **Setup:** 8×H200, global batch 640 (40 × grad_accum 2 × 8), AdamW lr 1e-4, cosine + 1000 warmup, bf16.
|
| 25 |
|
| 26 |
## ⚠️ Important — load with MultiFrameSmolVLAPolicy
|
|
@@ -45,7 +48,9 @@ github.com/mickeykang16/DynamicVLA — branch **`smolvla-multiframe-dom`**
|
|
| 45 |
(`policies/smolvla_multiframe.py`, `configs/smolvla.yaml`, `utils/helpers.py`).
|
| 46 |
|
| 47 |
## Notes
|
| 48 |
-
-
|
| 49 |
-
|
| 50 |
-
-
|
|
|
|
| 51 |
- Built to test whether a VL-aligned backbone + multi-frame closes the DOM sim gap seen with DynamicVLA.
|
|
|
|
|
|
| 9 |
- dom
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# SmolVLA-MultiFrame on DOM — FINAL (10 epochs, step 735,270)
|
| 13 |
|
| 14 |
Multi-frame **SmolVLA** fine-tuned on the **DOM** (Dynamic Object Manipulation) dataset.
|
| 15 |
+
This is the **final checkpoint** — training stopped at exactly **10 epochs** (global step / batch
|
| 16 |
+
**735,270** = 10 × 73,527), loss ≈ 0.0015.
|
| 17 |
|
| 18 |
## What this is
|
| 19 |
- **Backbone:** `lerobot/smolvla_base` — SmolVLM2-500M-Video-Instruct (SigLIP vision + SmolLM2)
|
| 20 |
+
+ flow-matching action expert. VL-aligned + robot-pretrained. **All weights loaded from
|
| 21 |
+
smolvla_base (no random init)**, then full fine-tuned.
|
| 22 |
- **Training:** **full fine-tune** (403M / 450M trainable, vision encoder unfrozen) on `hzxie/DOM`
|
| 23 |
+
(Franka, cameras `opst_cam` + `wrist_cam`, state 6-d, action 7-d, chunk 50). 10 epochs, ~12 days on 8×H200.
|
| 24 |
- **Multi-frame:** temporal window **{t-2, t}** (`DELTA_TIMESTAMPS observation: [-2, 0]`) — each frame is
|
| 25 |
+
fed to SmolVLM2 as a **separate image** so the model perceives object motion (DOM is dynamic; some DOM
|
| 26 |
+
instructions literally reference object velocity, e.g. "the moving object with the higher initial velocity").
|
| 27 |
- **Setup:** 8×H200, global batch 640 (40 × grad_accum 2 × 8), AdamW lr 1e-4, cosine + 1000 warmup, bf16.
|
| 28 |
|
| 29 |
## ⚠️ Important — load with MultiFrameSmolVLAPolicy
|
|
|
|
| 48 |
(`policies/smolvla_multiframe.py`, `configs/smolvla.yaml`, `utils/helpers.py`).
|
| 49 |
|
| 50 |
## Notes
|
| 51 |
+
- Final 10-epoch checkpoint. Loss is deeply converged (~0.0015) but **loss does not guarantee sim
|
| 52 |
+
success** — judge by DOM sim success-rate.
|
| 53 |
+
- Intermediate checkpoints (steps 40,000 / 275,581 / 427,635 / 529,689) previously occupied root;
|
| 54 |
+
recoverable from git history.
|
| 55 |
- Built to test whether a VL-aligned backbone + multi-frame closes the DOM sim gap seen with DynamicVLA.
|
| 56 |
+
Evaluate vs DynamicVLA (our mid-train + the released checkpoint) on the same DOM sim harness.
|
config.json
CHANGED
|
@@ -91,5 +91,5 @@
|
|
| 91 |
0
|
| 92 |
]
|
| 93 |
},
|
| 94 |
-
"epoch":
|
| 95 |
}
|
|
|
|
| 91 |
0
|
| 92 |
]
|
| 93 |
},
|
| 94 |
+
"epoch": 11
|
| 95 |
}
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 1197790032
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:48a3d64aba3b8020b2e343cb520025aa89b369f564eeef570e27ec523cc6731c
|
| 3 |
size 1197790032
|