TT-VidT pretrained DiT decoders
Pretrained DiT decoders for TT-VidT
(Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining,
NeurIPS 2026). Each TT-VidT encoder pretraining run initialises its decoder from one of
these; the training code downloads them automatically through DECODER_PRETRAINED,
e.g. "KBlueLeaf/TTVidT-decoders/pretrain_video_S_qknorm".
The decoders work in the 32x32x4 latent space of the frozen f8 frame VAE
KBlueLeaf/latentmaid-vae and are
conditioned on DINOv3 ViT-B/16 features (768-d).
- ImageNet decoders reconstruct an ImageNet image from its DINOv3 class token.
- Video decoders reconstruct a later video frame from its class token plus, through cross-attention, the spatial features of an earlier frame.
Files
Each decoder is <name>.safetensors plus <name>.json, which holds the decoder
config (decoder_config), the latent size and the latent statistics used in training.
pretrain_{imgnet,video}_{S,B,L}_<variant>:
| Size | Hidden | Layers | Heads |
|---|---|---|---|
| S | 768 | 12 | 12 |
| B | 1024 | 16 | 16 |
| L | 1152 | 28 | 16 |
| Variant | Meaning |
|---|---|
qknorm |
QK-norm attention, diffusion head |
qknorm_nofinal |
as above, without the final norm |
qknorm_reg, qknorm_nofinal_reg |
latent-regression head instead of diffusion |
768d12l, 1024d16l, 1152d28l (_80k) |
earlier runs without QK-norm |
The ones used in the paper's configs:
| Decoder | Used by |
|---|---|
pretrain_imgnet_S_qknorm |
objective / encoder sweep |
pretrain_video_S_qknorm |
TT-VidT |
pretrain_imgnet_S_qknorm_nofinal |
DisMo-2D3D with dual augmentation |
pretrain_{imgnet,video}_B_qknorm_nofinal, pretrain_{imgnet,video}_L_qknorm |
decoder ablation |
Recipe: 100k steps (80k for _80k), global batch 256, AdamW lr 1e-4. See the
code repository for training and usage.