MAVT v2 Grain-Prior checkpoints and notes (2026-10-30)
Browse files- README.md +99 -0
- ablations/df48_ema.pt +3 -0
- ablations/df64_ema.pt +3 -0
- image/bs_step35000.pt +3 -0
- image/bs_step38113.pt +3 -0
- image/bs_step38113_ema.pt +3 -0
- image_video/s2_step5500_ema.pt +3 -0
- image_video/s2_step5500_resume.pt +3 -0
README.md
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
tags:
|
| 4 |
+
- image-tokenizer
|
| 5 |
+
- video-tokenizer
|
| 6 |
+
- autoencoder
|
| 7 |
+
- dinov3
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# MAVT v2 — Grain-Prior tokenizer (checkpoints + notes, 2026-10-30)
|
| 11 |
+
|
| 12 |
+
Research checkpoints of the Grain-Prior C-D tokenizer from
|
| 13 |
+
[Computer-Vision-Lab-LinhDang/Antokenizer](https://github.com/Computer-Vision-Lab-LinhDang/Antokenizer)
|
| 14 |
+
(`src/mavt/grain_prior`, branch `feat/grain-splat-decoder`, code at commit `a5ab394`; the flags are documented in
|
| 15 |
+
`src/mavt/grain_prior/README.md`). Work in progress, not a release.
|
| 16 |
+
|
| 17 |
+
## What the model is
|
| 18 |
+
|
| 19 |
+
A 256×256 image is encoded on top of a **frozen DINOv3 ViT-L/16** into a continuous state of **11,300 numbers**
|
| 20 |
+
(the comparison ceiling is 12,288 = 16 × 16 × 48, AToken's image latent):
|
| 21 |
+
|
| 22 |
+
| Part | Shape | Numbers | Role |
|
| 23 |
+
|---|---|---|---|
|
| 24 |
+
| Grain geometry | 16 × 16 × 4 (u 2, ρ 1, σ 1) | 1,024 | position in the cell, log inverse depth, radius |
|
| 25 |
+
| Grain content `f` | 16 × 16 × **32** | 8,192 | local appearance (`d_f = 32`) |
|
| 26 |
+
| Coarse memory `c_0` | 4 × 4 × 128 | 2,048 | layout / colour / semantics of 64 px regions |
|
| 27 |
+
| Header | 36 | 36 | global scalars |
|
| 28 |
+
|
| 29 |
+
Decoder (`rgb_decoder=splat`): a geometry-biased context transformer over the grains and coarse slots, per-grain
|
| 30 |
+
frequency atoms, a depth-aware Gaussian splat onto a 64 × 64 query grid, then a pointwise RGB head with a coarse
|
| 31 |
+
start image. Video adds per-frame records (camera, motion, innovation slots) on top of the anchor state.
|
| 32 |
+
|
| 33 |
+
**DINOv3 weights are not included** (gated licence). The loader skips the missing `stems.dino.model.*` keys; pass
|
| 34 |
+
your own copy with `dino_repo=<local dir or facebook/dinov3-vitl16-pretrain-lvd1689m>`.
|
| 35 |
+
|
| 36 |
+
## Files
|
| 37 |
+
|
| 38 |
+
| File | Step | What | Use |
|
| 39 |
+
|---|---|---|---|
|
| 40 |
+
| `image/bs_step38113_ema.pt` | 38,113 | **Best image model** (image-only from scratch, stopped mid-decay), EMA | eval / inference |
|
| 41 |
+
| `image/bs_step38113.pt` | 38,113 | same, raw weights | `--init` for further stages |
|
| 42 |
+
| `image/bs_step35000.pt` | 35,000 | same run, before the LR decay, raw | `--init` for a fresh schedule |
|
| 43 |
+
| `image_video/s2_step5500_ema.pt` | 5,500 | stage 2 (image + video) from `bs_step38113`, EMA | eval |
|
| 44 |
+
| `image_video/s2_step5500_resume.pt` | 5,500 | stage 2 full state (optimizer, EMA, scheduler) | `--resume` |
|
| 45 |
+
| `ablations/df48_ema.pt` | +3,000 | `d_f = 48` (15,396 numbers), widened from `bs_step38113` | rate study |
|
| 46 |
+
| `ablations/df64_ema.pt` | +3,000 | `d_f = 64` (19,492 numbers) | rate study |
|
| 47 |
+
|
| 48 |
+
All checkpoints: `cfg` keys `d_f=32` (48 / 64 for the ablations), `rgb_decoder=splat`, `query_grid=64`,
|
| 49 |
+
`rgb_head=pointwise`, `w_lpips=0.1`.
|
| 50 |
+
|
| 51 |
+
```python
|
| 52 |
+
from mavt.grain_prior import load_grain_prior
|
| 53 |
+
model, cfg, step = load_grain_prior("image/bs_step38113_ema.pt", "cuda", dino_repo="<dinov3-vitl16 dir>")
|
| 54 |
+
out = model(x, "image", return_outputs=True) # x: [B, 3, 256, 256] in [-1, 1]
|
| 55 |
+
rgb = out["outputs"]["rgb"]
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
## Results (ImageNet val, first 1,000 images, 256², centre crop)
|
| 59 |
+
|
| 60 |
+
| Model | Numbers | PSNR ↑ | SSIM ↑ | LPIPS ↓ | rFID ↓ |
|
| 61 |
+
|---|---|---|---|---|---|
|
| 62 |
+
| `bs_step38113_ema` | 11,300 | **27.76** | 0.790 | 0.130 | 17.6 |
|
| 63 |
+
| `bs_step38113` (raw) | 11,300 | 27.69 | 0.788 | 0.131 | 17.1 |
|
| 64 |
+
| `s2` step 3,500 (EMA, image after adding video) | 11,300 | 27.58 | 0.784 | 0.136 | 18.0 |
|
| 65 |
+
| `df48_ema` | 15,396 | 28.16 | 0.804 | 0.121 | 16.3 |
|
| 66 |
+
| `df64_ema` | 19,492 | 28.43 | 0.813 | 0.114 | 15.5 |
|
| 67 |
+
| *reference: AToken-So/c (paper)* | 12,288 | 29.72 | 0.848 | 0.085 | 0.21 |
|
| 68 |
+
|
| 69 |
+
rFID on 1,000 images is high-variance and not comparable to 50k-image rFID; use it only between rows of this table.
|
| 70 |
+
|
| 71 |
+
## What was learned this round (all 3k-step probes from `bs_step38113`, lr 1e-4, same protocol, EMA PSNR)
|
| 72 |
+
|
| 73 |
+
| Probe | PSNR | Δ vs control |
|
| 74 |
+
|---|---|---|
|
| 75 |
+
| control | 27.74 | — |
|
| 76 |
+
| more frequency atoms per grain (6 → 12 / 24) | 27.67 / 27.65 | 0 |
|
| 77 |
+
| atoms evaluated at the pixels (no 64-grid aliasing) | 27.74 | 0 |
|
| 78 |
+
| + pixel ResBlocks at 128 / 256 | 27.75 | +0.01 |
|
| 79 |
+
| + ConvNeXt context at 64 | 27.77 | +0.03 |
|
| 80 |
+
| + Open Images 50/50 for 5k steps | 27.72 | 0 |
|
| 81 |
+
| **`d_f` 35 / 48 / 64** | **27.83 / 28.16 / 28.43** | **+0.09 / +0.42 / +0.69** |
|
| 82 |
+
|
| 83 |
+
* The decoder is not the bottleneck: three ways of adding decoder capacity at the pixel level change nothing,
|
| 84 |
+
although the new modules do train.
|
| 85 |
+
* The information in the state is: PSNR grows by about +1 dB per doubling of `d_f` (raw: +0.13 / +0.61 / +0.95),
|
| 86 |
+
and SSIM / LPIPS / rFID improve with it.
|
| 87 |
+
* Channel ablations at step 20k: replacing `c_0` by its mean → 15.0 dB, shuffling `f` across cells → 13.3 dB,
|
| 88 |
+
`f` uses all 32 dimensions (first PC 16 % of variance), 94 % of the remaining error energy is above 1/8 cycle/px.
|
| 89 |
+
* Adding video (stage 2) costs about 0.2 dB on images early on; part of it is the LR going back up from a decayed
|
| 90 |
+
point.
|
| 91 |
+
* Remaining error is high-frequency texture (fur, foliage, small text). Perceptual losses (edge / LPIPS / Gram)
|
| 92 |
+
are expected to help LPIPS and rFID rather than PSNR; flags `w_edge` and `feat_sg_f` exist at `a5ab394`, not yet
|
| 93 |
+
trained.
|
| 94 |
+
|
| 95 |
+
## Training recipe of `bs_step38113`
|
| 96 |
+
|
| 97 |
+
Image only, ImageNet-1k train, 4 GPUs, batch 256, Muon / AdamW, lr 2e-4, warmup 1k, WSD with the decay over the
|
| 98 |
+
last 4k of 40k steps (stopped at 38,113, mid-decay), loss = L1 + 0.1 LPIPS + DINOv3 feature loss (+ init / code
|
| 99 |
+
terms), about 11 h. Stage 2 adds OpenVid + SSv2 clips (batch 128 images + 32 clips).
|
ablations/df48_ema.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:eeb4fb9fb354c58d664f2004e2c7b186dc6181d01c81b62a9677203f26d73f8c
|
| 3 |
+
size 825939357
|
ablations/df64_ema.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2a1f60f655e9c9d2985b023a1fc64186dd70e4c98eea495749b5fd821fcfa43b
|
| 3 |
+
size 826234397
|
image/bs_step35000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7e5b88da6d9211e00e5ee196db9891e29d0df1777c15c7b33cec00c315395ff8
|
| 3 |
+
size 825649493
|
image/bs_step38113.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:96d54fc404f118d40ea95df3e1270855e5f083722f14229197f53aaf3d9406c3
|
| 3 |
+
size 825649493
|
image/bs_step38113_ema.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:54a13dead5bc4b6a0c65832c358d958d72dab920acaa6d9386165745ecf9c0ee
|
| 3 |
+
size 825654797
|
image_video/s2_step5500_ema.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d8698272399c84e66cfe0dd94a41b412d2db37a64e85953b0a1461c8fa5af1cf
|
| 3 |
+
size 825653663
|
image_video/s2_step5500_resume.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:30731e620a1cc498bcacb4be4372b9c2531ab8112aa825208eeccb81a6c37e47
|
| 3 |
+
size 2342971315
|