anbinh39 commited on
Commit
8a9c498
·
verified ·
1 Parent(s): a3949cd

MAVT v2 Grain-Prior checkpoints and notes (2026-10-30)

Browse files
README.md ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ tags:
4
+ - image-tokenizer
5
+ - video-tokenizer
6
+ - autoencoder
7
+ - dinov3
8
+ ---
9
+
10
+ # MAVT v2 — Grain-Prior tokenizer (checkpoints + notes, 2026-10-30)
11
+
12
+ Research checkpoints of the Grain-Prior C-D tokenizer from
13
+ [Computer-Vision-Lab-LinhDang/Antokenizer](https://github.com/Computer-Vision-Lab-LinhDang/Antokenizer)
14
+ (`src/mavt/grain_prior`, branch `feat/grain-splat-decoder`, code at commit `a5ab394`; the flags are documented in
15
+ `src/mavt/grain_prior/README.md`). Work in progress, not a release.
16
+
17
+ ## What the model is
18
+
19
+ A 256×256 image is encoded on top of a **frozen DINOv3 ViT-L/16** into a continuous state of **11,300 numbers**
20
+ (the comparison ceiling is 12,288 = 16 × 16 × 48, AToken's image latent):
21
+
22
+ | Part | Shape | Numbers | Role |
23
+ |---|---|---|---|
24
+ | Grain geometry | 16 × 16 × 4 (u 2, ρ 1, σ 1) | 1,024 | position in the cell, log inverse depth, radius |
25
+ | Grain content `f` | 16 × 16 × **32** | 8,192 | local appearance (`d_f = 32`) |
26
+ | Coarse memory `c_0` | 4 × 4 × 128 | 2,048 | layout / colour / semantics of 64 px regions |
27
+ | Header | 36 | 36 | global scalars |
28
+
29
+ Decoder (`rgb_decoder=splat`): a geometry-biased context transformer over the grains and coarse slots, per-grain
30
+ frequency atoms, a depth-aware Gaussian splat onto a 64 × 64 query grid, then a pointwise RGB head with a coarse
31
+ start image. Video adds per-frame records (camera, motion, innovation slots) on top of the anchor state.
32
+
33
+ **DINOv3 weights are not included** (gated licence). The loader skips the missing `stems.dino.model.*` keys; pass
34
+ your own copy with `dino_repo=<local dir or facebook/dinov3-vitl16-pretrain-lvd1689m>`.
35
+
36
+ ## Files
37
+
38
+ | File | Step | What | Use |
39
+ |---|---|---|---|
40
+ | `image/bs_step38113_ema.pt` | 38,113 | **Best image model** (image-only from scratch, stopped mid-decay), EMA | eval / inference |
41
+ | `image/bs_step38113.pt` | 38,113 | same, raw weights | `--init` for further stages |
42
+ | `image/bs_step35000.pt` | 35,000 | same run, before the LR decay, raw | `--init` for a fresh schedule |
43
+ | `image_video/s2_step5500_ema.pt` | 5,500 | stage 2 (image + video) from `bs_step38113`, EMA | eval |
44
+ | `image_video/s2_step5500_resume.pt` | 5,500 | stage 2 full state (optimizer, EMA, scheduler) | `--resume` |
45
+ | `ablations/df48_ema.pt` | +3,000 | `d_f = 48` (15,396 numbers), widened from `bs_step38113` | rate study |
46
+ | `ablations/df64_ema.pt` | +3,000 | `d_f = 64` (19,492 numbers) | rate study |
47
+
48
+ All checkpoints: `cfg` keys `d_f=32` (48 / 64 for the ablations), `rgb_decoder=splat`, `query_grid=64`,
49
+ `rgb_head=pointwise`, `w_lpips=0.1`.
50
+
51
+ ```python
52
+ from mavt.grain_prior import load_grain_prior
53
+ model, cfg, step = load_grain_prior("image/bs_step38113_ema.pt", "cuda", dino_repo="<dinov3-vitl16 dir>")
54
+ out = model(x, "image", return_outputs=True) # x: [B, 3, 256, 256] in [-1, 1]
55
+ rgb = out["outputs"]["rgb"]
56
+ ```
57
+
58
+ ## Results (ImageNet val, first 1,000 images, 256², centre crop)
59
+
60
+ | Model | Numbers | PSNR ↑ | SSIM ↑ | LPIPS ↓ | rFID ↓ |
61
+ |---|---|---|---|---|---|
62
+ | `bs_step38113_ema` | 11,300 | **27.76** | 0.790 | 0.130 | 17.6 |
63
+ | `bs_step38113` (raw) | 11,300 | 27.69 | 0.788 | 0.131 | 17.1 |
64
+ | `s2` step 3,500 (EMA, image after adding video) | 11,300 | 27.58 | 0.784 | 0.136 | 18.0 |
65
+ | `df48_ema` | 15,396 | 28.16 | 0.804 | 0.121 | 16.3 |
66
+ | `df64_ema` | 19,492 | 28.43 | 0.813 | 0.114 | 15.5 |
67
+ | *reference: AToken-So/c (paper)* | 12,288 | 29.72 | 0.848 | 0.085 | 0.21 |
68
+
69
+ rFID on 1,000 images is high-variance and not comparable to 50k-image rFID; use it only between rows of this table.
70
+
71
+ ## What was learned this round (all 3k-step probes from `bs_step38113`, lr 1e-4, same protocol, EMA PSNR)
72
+
73
+ | Probe | PSNR | Δ vs control |
74
+ |---|---|---|
75
+ | control | 27.74 | — |
76
+ | more frequency atoms per grain (6 → 12 / 24) | 27.67 / 27.65 | 0 |
77
+ | atoms evaluated at the pixels (no 64-grid aliasing) | 27.74 | 0 |
78
+ | + pixel ResBlocks at 128 / 256 | 27.75 | +0.01 |
79
+ | + ConvNeXt context at 64 | 27.77 | +0.03 |
80
+ | + Open Images 50/50 for 5k steps | 27.72 | 0 |
81
+ | **`d_f` 35 / 48 / 64** | **27.83 / 28.16 / 28.43** | **+0.09 / +0.42 / +0.69** |
82
+
83
+ * The decoder is not the bottleneck: three ways of adding decoder capacity at the pixel level change nothing,
84
+ although the new modules do train.
85
+ * The information in the state is: PSNR grows by about +1 dB per doubling of `d_f` (raw: +0.13 / +0.61 / +0.95),
86
+ and SSIM / LPIPS / rFID improve with it.
87
+ * Channel ablations at step 20k: replacing `c_0` by its mean → 15.0 dB, shuffling `f` across cells → 13.3 dB,
88
+ `f` uses all 32 dimensions (first PC 16 % of variance), 94 % of the remaining error energy is above 1/8 cycle/px.
89
+ * Adding video (stage 2) costs about 0.2 dB on images early on; part of it is the LR going back up from a decayed
90
+ point.
91
+ * Remaining error is high-frequency texture (fur, foliage, small text). Perceptual losses (edge / LPIPS / Gram)
92
+ are expected to help LPIPS and rFID rather than PSNR; flags `w_edge` and `feat_sg_f` exist at `a5ab394`, not yet
93
+ trained.
94
+
95
+ ## Training recipe of `bs_step38113`
96
+
97
+ Image only, ImageNet-1k train, 4 GPUs, batch 256, Muon / AdamW, lr 2e-4, warmup 1k, WSD with the decay over the
98
+ last 4k of 40k steps (stopped at 38,113, mid-decay), loss = L1 + 0.1 LPIPS + DINOv3 feature loss (+ init / code
99
+ terms), about 11 h. Stage 2 adds OpenVid + SSv2 clips (batch 128 images + 32 clips).
ablations/df48_ema.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eeb4fb9fb354c58d664f2004e2c7b186dc6181d01c81b62a9677203f26d73f8c
3
+ size 825939357
ablations/df64_ema.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2a1f60f655e9c9d2985b023a1fc64186dd70e4c98eea495749b5fd821fcfa43b
3
+ size 826234397
image/bs_step35000.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7e5b88da6d9211e00e5ee196db9891e29d0df1777c15c7b33cec00c315395ff8
3
+ size 825649493
image/bs_step38113.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:96d54fc404f118d40ea95df3e1270855e5f083722f14229197f53aaf3d9406c3
3
+ size 825649493
image/bs_step38113_ema.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:54a13dead5bc4b6a0c65832c358d958d72dab920acaa6d9386165745ecf9c0ee
3
+ size 825654797
image_video/s2_step5500_ema.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d8698272399c84e66cfe0dd94a41b412d2db37a64e85953b0a1461c8fa5af1cf
3
+ size 825653663
image_video/s2_step5500_resume.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30731e620a1cc498bcacb4be4372b9c2531ab8112aa825208eeccb81a6c37e47
3
+ size 2342971315