File size: 2,243 Bytes
139ec15 9aba566 139ec15 9aba566 139ec15 9aba566 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | ---
license: apache-2.0
tags:
- video
- self-supervised-learning
- diffusion
pipeline_tag: other
---
# TT-VidT pretrained DiT decoders
Pretrained DiT decoders for [TT-VidT](https://github.com/KohakuBlueleaf/TTVidT)
(*Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining*,
NeurIPS 2026). [Paper](https://huggingface.co/papers/2609.33419) ·
[Project page](https://kohakublueleaf.github.io/TTVidT/)
Each TT-VidT encoder pretraining run initialises its decoder from one of
these; the training code downloads them automatically through `DECODER_PRETRAINED`,
e.g. `"KBlueLeaf/TTVidT-decoders/pretrain_video_S_qknorm"`.
The decoders work in the 32x32x4 latent space of the frozen f8 frame VAE
[`KBlueLeaf/latentmaid-vae`](https://huggingface.co/KBlueLeaf/latentmaid-vae) and are
conditioned on DINOv3 ViT-B/16 features (768-d).
- **ImageNet** decoders reconstruct an ImageNet image from its DINOv3 class token.
- **Video** decoders reconstruct a later video frame from its class token plus, through
cross-attention, the spatial features of an earlier frame.
## Files
Each decoder is `<name>.safetensors` plus `<name>.json`, which holds the decoder
config (`decoder_config`), the latent size and the latent statistics used in training.
`pretrain_{imgnet,video}_{S,B,L}_<variant>`:
| Size | Hidden | Layers | Heads |
|---|---:|---:|---:|
| S | 768 | 12 | 12 |
| B | 1024 | 16 | 16 |
| L | 1152 | 28 | 16 |
| Variant | Meaning |
|---|---|
| `qknorm` | QK-norm attention, diffusion head |
| `qknorm_nofinal` | as above, without the final norm |
| `qknorm_reg`, `qknorm_nofinal_reg` | latent-regression head instead of diffusion |
| `768d12l`, `1024d16l`, `1152d28l` (`_80k`) | earlier runs without QK-norm |
The ones used in the paper's configs:
| Decoder | Used by |
|---|---|
| `pretrain_imgnet_S_qknorm` | objective / encoder sweep |
| `pretrain_video_S_qknorm` | TT-VidT |
| `pretrain_imgnet_S_qknorm_nofinal` | DisMo-2D3D with dual augmentation |
| `pretrain_{imgnet,video}_B_qknorm_nofinal`, `pretrain_{imgnet,video}_L_qknorm` | decoder ablation |
Recipe: 100k steps (80k for `_80k`), global batch 256, AdamW lr 1e-4. See the
[code repository](https://github.com/KohakuBlueleaf/TTVidT) for training and usage. |