Instructions to use mickeykang/smolvla-multiframe-DOM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use mickeykang/smolvla-multiframe-DOM with LeRobot:
# See https://github.com/huggingface/lerobot?tab=readme-ov-file#installation for more details git clone https://github.com/huggingface/lerobot.git cd lerobot pip install -e .[smolvla]
# Launch finetuning on your dataset python lerobot/scripts/train.py \ --policy.path=mickeykang/smolvla-multiframe-DOM \ --dataset.repo_id=lerobot/svla_so101_pickplace \ --batch_size=64 \ --steps=20000 \ --output_dir=outputs/train/my_smolvla \ --job_name=my_smolvla_training \ --policy.device=cuda \ --wandb.enable=true
# Run the policy using the record function python -m lerobot.record \ --robot.type=so101_follower \ --robot.port=/dev/ttyACM0 \ # <- Use your port --robot.id=my_blue_follower_arm \ # <- Use your robot id --robot.cameras="{ front: {type: opencv, index_or_path: 8, width: 640, height: 480, fps: 30}}" \ # <- Use your cameras --dataset.single_task="Grasp a lego block and put it in the bin." \ # <- Use the same task description you used in your dataset recording --dataset.repo_id=HF_USER/dataset_name \ # <- This will be the dataset name on HF Hub --dataset.episode_time_s=50 \ --dataset.num_episodes=10 \ --policy.path=mickeykang/smolvla-multiframe-DOM - Notebooks
- Google Colab
- Kaggle
Final checkpoint: 10 epochs (step 735270)
Browse files
README.md
CHANGED
|
@@ -9,22 +9,21 @@ tags:
|
|
| 9 |
- dom
|
| 10 |
---
|
| 11 |
|
| 12 |
-
# SmolVLA-MultiFrame on DOM —
|
| 13 |
|
| 14 |
Multi-frame **SmolVLA** fine-tuned on the **DOM** (Dynamic Object Manipulation) dataset.
|
| 15 |
-
|
| 16 |
-
|
| 17 |
|
| 18 |
## What this is
|
| 19 |
- **Backbone:** `lerobot/smolvla_base` — SmolVLM2-500M-Video-Instruct (SigLIP vision + SmolLM2)
|
| 20 |
-
+ flow-matching action expert. VL-aligned + robot-pretrained.
|
| 21 |
-
smolvla_base (no random init)**, then full fine-tuned.
|
| 22 |
- **Training:** **full fine-tune** (403M / 450M trainable, vision encoder unfrozen) on `hzxie/DOM`
|
| 23 |
-
(Franka, cameras `opst_cam` + `wrist_cam`, state 6-d, action 7-d, chunk 50).
|
| 24 |
- **Multi-frame:** temporal window **{t-2, t}** (`DELTA_TIMESTAMPS observation: [-2, 0]`) — each frame is
|
| 25 |
-
fed to SmolVLM2 as a **separate image** so the model perceives object motion (DOM is dynamic
|
| 26 |
-
instructions literally reference object velocity, e.g. "the moving object with the higher initial velocity").
|
| 27 |
- **Setup:** 8×H200, global batch 640 (40 × grad_accum 2 × 8), AdamW lr 1e-4, cosine + 1000 warmup, bf16.
|
|
|
|
| 28 |
|
| 29 |
## ⚠️ Important — load with MultiFrameSmolVLAPolicy
|
| 30 |
`config.json` has `type: "smolvla"`, but this checkpoint was trained to consume **two frames per camera**.
|
|
@@ -48,9 +47,7 @@ github.com/mickeykang16/DynamicVLA — branch **`smolvla-multiframe-dom`**
|
|
| 48 |
(`policies/smolvla_multiframe.py`, `configs/smolvla.yaml`, `utils/helpers.py`).
|
| 49 |
|
| 50 |
## Notes
|
| 51 |
-
- Final 10
|
| 52 |
-
success** — judge by DOM sim success-rate.
|
| 53 |
-
- Intermediate checkpoints (steps 40,000 / 275,581 / 427,635 / 529,689)
|
| 54 |
-
recoverable from git history.
|
| 55 |
- Built to test whether a VL-aligned backbone + multi-frame closes the DOM sim gap seen with DynamicVLA.
|
| 56 |
-
Evaluate vs DynamicVLA (our mid-train + the released checkpoint) on the same DOM sim harness.
|
|
|
|
| 9 |
- dom
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# SmolVLA-MultiFrame on DOM — final, 10 epochs (step 735,270)
|
| 13 |
|
| 14 |
Multi-frame **SmolVLA** fine-tuned on the **DOM** (Dynamic Object Manipulation) dataset.
|
| 15 |
+
Root holds the **final** checkpoint: **10 full epochs** of DOM (step 735,270 = 10 × 73,527), final loss ≈ 0.0015.
|
| 16 |
+
Training completed and auto-stopped at the 10-epoch target.
|
| 17 |
|
| 18 |
## What this is
|
| 19 |
- **Backbone:** `lerobot/smolvla_base` — SmolVLM2-500M-Video-Instruct (SigLIP vision + SmolLM2)
|
| 20 |
+
+ flow-matching action expert. VL-aligned + robot-pretrained.
|
|
|
|
| 21 |
- **Training:** **full fine-tune** (403M / 450M trainable, vision encoder unfrozen) on `hzxie/DOM`
|
| 22 |
+
(Franka, cameras `opst_cam` + `wrist_cam`, state 6-d, action 7-d, chunk 50).
|
| 23 |
- **Multi-frame:** temporal window **{t-2, t}** (`DELTA_TIMESTAMPS observation: [-2, 0]`) — each frame is
|
| 24 |
+
fed to SmolVLM2 as a **separate image** so the model perceives object motion (DOM is dynamic).
|
|
|
|
| 25 |
- **Setup:** 8×H200, global batch 640 (40 × grad_accum 2 × 8), AdamW lr 1e-4, cosine + 1000 warmup, bf16.
|
| 26 |
+
~12 days wall-clock for 10 epochs.
|
| 27 |
|
| 28 |
## ⚠️ Important — load with MultiFrameSmolVLAPolicy
|
| 29 |
`config.json` has `type: "smolvla"`, but this checkpoint was trained to consume **two frames per camera**.
|
|
|
|
| 47 |
(`policies/smolvla_multiframe.py`, `configs/smolvla.yaml`, `utils/helpers.py`).
|
| 48 |
|
| 49 |
## Notes
|
| 50 |
+
- **Final** checkpoint (10 epochs). Loss is deeply converged (~0.0015) but **loss does not guarantee sim
|
| 51 |
+
success** — judge by DOM sim success-rate (vs DynamicVLA and the released DynamicVLA checkpoint).
|
| 52 |
+
- Intermediate checkpoints (steps 40,000 / 275,581 / 427,635 / 529,689) are in git history.
|
|
|
|
| 53 |
- Built to test whether a VL-aligned backbone + multi-frame closes the DOM sim gap seen with DynamicVLA.
|
|
|