Blame for 1.3B/docs/assets.md

# Asset Preparation

The source release includes training and evaluation code, the training configuration, and example prompts. Model weights, training reference data, and generated videos are stored separately.

## Required assets

| Use | Assets |
|---|---|
| Inference | Wan2.1-T2V-1.3B base weights, text encoder, tokenizer, VAE, and a trained Anonymous Video Generation checkpoint |
| Training | Inference base assets, the Self-Forcing ODE initialization, three frozen feature encoders, reference features, and Nyström summaries |

The trained Anonymous Video Generation checkpoint and reference collection are not distributed in this repository. Encoder weights alone are insufficient to reproduce training with the provided configuration.

## Model families (local files required)

- Wan2.1-T2V-1.3B
- Self-Forcing initialization
- [DINOv3](https://github.com/facebookresearch/dinov3)
- V-JEPA 2 ViT-L
- VideoMAEv2-Large

For the supplied configuration, DINOv3 uses ViT-L/16 features from its final layer; V-JEPA 2 and VideoMAEv2 use layer 15. VideoMAEv2 is pinned to revision `9981a9c8f77118c421e5228e1b219468a4b0238d`, and the DINOv3 source is pinned to commit `6876159a11b4df116f30f667f8c9888617df0751`.

## Directory layout

Place assets under a common root. The following filenames are retained for compatibility with the supplied configuration:

```text
ASSET_ROOT/
├── checkpoints/ode_init.pt
├── models/
│   ├── Wan2.1-T2V-1.3B/
│   ├── dinov3-src/
│   ├── dinov3_vitl16_lvd1689m.pth
│   ├── dinov3_vitl16_lvd1689m.pth.provenance.json
│   ├── vjepa2-vitl-fpc64-256/
│   └── VideoMAEv2-Large/
├── teacher_banks/triple_video_new8682_824/
│   ├── manifest.json
│   └── ...                         # All feature shards referenced by the manifest
└── experiments/triple_video_new8682_824/
    ├── source/new8682_824_matched.jsonl
    └── references/
        ├── dinov3_vitl16_f20mean5_k1024_m256_matched5k.pt
        ├── vjepa2_l15_last64_k1024_m1024_matched5k.pt
        └── videomae_v2_l15_fixed5x16_k256_m256_matched5k.pt
```

The reference collection contains 8,682 videos stored in 272 feature shards. The `matched5k` suffix is a legacy filename; the manifest determines the actual collection size.

Keep the feature shards, manifests, and summaries together without changing their contents or sample ordering. Preserve the DINOv3 provenance file and the VideoMAEv2 `download_manifest.json` alongside their weights.

## Configure asset paths

From the `1.3B/` directory:

```bash
export ASSET_ROOT=/path/to/video-generation-assets
python scripts/prepare_config.py --asset-root "$ASSET_ROOT"
```

The setup script generates a local configuration with relocated asset paths. Generated configurations are excluded from version control. Asset and source-hash preflight checks are not run by the launchers.

## Reference feature preparation

Preprocessing is performed once, before generator training. It requires the raw captioned reference videos, the frozen encoders, and a CUDA device. Training subsequently reads the saved features instead of decoding those reference videos again.

1. **Prepare the prompt/video manifest.** Each entry binds a caption to an MP4 basename in a fixed sample order. The loader in [`matched_video_manifest.py`](../code/utils/matched_video_manifest.py) expects a JSONL manifest and its `.meta.json` sidecar, including the source-data identity. A plain prompt text file is not a training manifest.
2. **Extract all three representations.** [`extract_triple_matched_teacher_bank.py`](../code/tools/extract_triple_matched_teacher_bank.py) reads the manifest and raw videos and writes feature shards plus a `manifest.json`. Supply the source manifest, video root, encoder paths, and model identities. For the released collection, set `--expected-count 8682` and `--shard-rows 32`; the default sample count in this lower-level tool is different.
3. **Build the fixed summaries.** [`build_triple_nystrom_references.py`](../code/tools/build_triple_nystrom_references.py) reads the extracted feature bank and writes one Nyström reference per encoder. Supply the bank and model identities and set `--expected-sample-count 8682`. Keep the reference files together with the feature bank from which they were built.

Run the preprocessing tools from the `1.3B/` directory with `PYTHONPATH="$PWD/code"`. Their `--help` output lists the required manifests, encoder identities, and output paths.

The feature protocol used by the training configuration is:

| Encoder | Video sampling | Representation | Nyström landmarks |
|---|---|---|---:|
| DINOv3 ViT-L/16 | 20 frames across the 81-frame video, pooled in groups of 5 | Final-layer features, 1,024 tokens per video | 256 |
| V-JEPA 2 ViT-L | Last 64 contiguous frames | Layer 15, 1,024 tokens per video | 1,024 |
| VideoMAEv2-Large | Skip the first frame, then use five 16-frame clips | Layer 15, 256 tokens per clip | 256 |

Encoder revisions, feature sampling, source ordering, and preprocessing metadata must agree between the reference assets and training configuration. Re-extraction can change serialized hashes; it should not be assumed to recreate the supplied assets byte for byte.

## Using another reference collection

A new collection requires a new manifest, feature bank, Nyström summaries, and a training configuration with matching paths and identities.

Update the local configuration for a new experiment. Changing the reference collection should not be expected to reproduce the reported VBench score.

## Local snapshot preparation

All model loaders require existing local directories. Hosted model repository URLs, repository IDs and automatic downloads are omitted from this source release. The legacy-named `download_videomae_v2_large.py` stages an existing local snapshot using `--source-dir` and an empty `--output-dir`; it does not download files. If the source includes a valid pinned `download_manifest.json`, that manifest is preserved for compatibility with frozen reference banks. A snapshot without a manifest gets a new local manifest; rebuild matching reference assets and update their configuration identities before training. `prepare_config.py` reads model identities from an available local feature bank into private local configuration files.