mickeykang commited on
Commit
b65b4ad
·
verified ·
1 Parent(s): 5a2a8ff

Final checkpoint: 10 epochs (step 735270)

Browse files
Files changed (1) hide show
  1. README.md +10 -13
README.md CHANGED
@@ -9,22 +9,21 @@ tags:
9
  - dom
10
  ---
11
 
12
- # SmolVLA-MultiFrame on DOM — FINAL (10 epochs, step 735,270)
13
 
14
  Multi-frame **SmolVLA** fine-tuned on the **DOM** (Dynamic Object Manipulation) dataset.
15
- This is the **final checkpoint** — training stopped at exactly **10 epochs** (global step / batch
16
- **735,270** = 10 × 73,527), loss ≈ 0.0015.
17
 
18
  ## What this is
19
  - **Backbone:** `lerobot/smolvla_base` — SmolVLM2-500M-Video-Instruct (SigLIP vision + SmolLM2)
20
- + flow-matching action expert. VL-aligned + robot-pretrained. **All weights loaded from
21
- smolvla_base (no random init)**, then full fine-tuned.
22
  - **Training:** **full fine-tune** (403M / 450M trainable, vision encoder unfrozen) on `hzxie/DOM`
23
- (Franka, cameras `opst_cam` + `wrist_cam`, state 6-d, action 7-d, chunk 50). 10 epochs, ~12 days on 8×H200.
24
  - **Multi-frame:** temporal window **{t-2, t}** (`DELTA_TIMESTAMPS observation: [-2, 0]`) — each frame is
25
- fed to SmolVLM2 as a **separate image** so the model perceives object motion (DOM is dynamic; some DOM
26
- instructions literally reference object velocity, e.g. "the moving object with the higher initial velocity").
27
  - **Setup:** 8×H200, global batch 640 (40 × grad_accum 2 × 8), AdamW lr 1e-4, cosine + 1000 warmup, bf16.
 
28
 
29
  ## ⚠️ Important — load with MultiFrameSmolVLAPolicy
30
  `config.json` has `type: "smolvla"`, but this checkpoint was trained to consume **two frames per camera**.
@@ -48,9 +47,7 @@ github.com/mickeykang16/DynamicVLA — branch **`smolvla-multiframe-dom`**
48
  (`policies/smolvla_multiframe.py`, `configs/smolvla.yaml`, `utils/helpers.py`).
49
 
50
  ## Notes
51
- - Final 10-epoch checkpoint. Loss is deeply converged (~0.0015) but **loss does not guarantee sim
52
- success** — judge by DOM sim success-rate.
53
- - Intermediate checkpoints (steps 40,000 / 275,581 / 427,635 / 529,689) previously occupied root;
54
- recoverable from git history.
55
  - Built to test whether a VL-aligned backbone + multi-frame closes the DOM sim gap seen with DynamicVLA.
56
- Evaluate vs DynamicVLA (our mid-train + the released checkpoint) on the same DOM sim harness.
 
9
  - dom
10
  ---
11
 
12
+ # SmolVLA-MultiFrame on DOM — final, 10 epochs (step 735,270)
13
 
14
  Multi-frame **SmolVLA** fine-tuned on the **DOM** (Dynamic Object Manipulation) dataset.
15
+ Root holds the **final** checkpoint: **10 full epochs** of DOM (step 735,270 = 10 × 73,527), final loss ≈ 0.0015.
16
+ Training completed and auto-stopped at the 10-epoch target.
17
 
18
  ## What this is
19
  - **Backbone:** `lerobot/smolvla_base` — SmolVLM2-500M-Video-Instruct (SigLIP vision + SmolLM2)
20
+ + flow-matching action expert. VL-aligned + robot-pretrained.
 
21
  - **Training:** **full fine-tune** (403M / 450M trainable, vision encoder unfrozen) on `hzxie/DOM`
22
+ (Franka, cameras `opst_cam` + `wrist_cam`, state 6-d, action 7-d, chunk 50).
23
  - **Multi-frame:** temporal window **{t-2, t}** (`DELTA_TIMESTAMPS observation: [-2, 0]`) — each frame is
24
+ fed to SmolVLM2 as a **separate image** so the model perceives object motion (DOM is dynamic).
 
25
  - **Setup:** 8×H200, global batch 640 (40 × grad_accum 2 × 8), AdamW lr 1e-4, cosine + 1000 warmup, bf16.
26
+ ~12 days wall-clock for 10 epochs.
27
 
28
  ## ⚠️ Important — load with MultiFrameSmolVLAPolicy
29
  `config.json` has `type: "smolvla"`, but this checkpoint was trained to consume **two frames per camera**.
 
47
  (`policies/smolvla_multiframe.py`, `configs/smolvla.yaml`, `utils/helpers.py`).
48
 
49
  ## Notes
50
+ - **Final** checkpoint (10 epochs). Loss is deeply converged (~0.0015) but **loss does not guarantee sim
51
+ success** — judge by DOM sim success-rate (vs DynamicVLA and the released DynamicVLA checkpoint).
52
+ - Intermediate checkpoints (steps 40,000 / 275,581 / 427,635 / 529,689) are in git history.
 
53
  - Built to test whether a VL-aligned backbone + multi-frame closes the DOM sim gap seen with DynamicVLA.