Elastic-forcing commited on
Commit
066102e
Β·
verified Β·
1 Parent(s): 072b111

Update documentation

Browse files
Files changed (3) hide show
  1. 1.3B/README.md +47 -136
  2. 14B/README.md +59 -79
  3. README.md +30 -38
1.3B/README.md CHANGED
@@ -1,183 +1,94 @@
1
- # Anonymous Video Generation β€” 1.3B
2
 
3
- [Project overview](../README.md) Β· [14B release](../14B/README.md)
4
 
5
- ---
6
 
7
- For a guided implementation walkthrough, see [Reading the code](../docs/paper-to-code.md).
8
- The YAML uses the paper's distribution batch, differentiation batch, and reference minibatch terminology; `code/utils/config.py` keeps legacy configurations compatible.
9
 
10
- Anonymous Video Generation post-trains few-step autoregressive video generators by matching generated and reference videos in frozen representation spaces. It combines a hybrid NystrΓΆm–Monte Carlo distribution estimator with gradient replay, allowing distribution estimation to use more samples than backpropagation. Only the generator is optimized; diffusion score models are not needed during post-training.
11
-
12
- This repository provides the **Wan2.1-T2V-1.3B three-encoder configuration**, using **DINOv3, V-JEPA 2, and VideoMAEv2**. This configuration achieves **84.25 VBench Total** in the paper.
13
-
14
- ## Supported Configuration
15
-
16
- | Backbone | Feature encoders | Sampling | Training GPUs | Updates |
17
- |---|---|---|---|---:|
18
- | Wan2.1-T2V-1.3B | DINOv3 + V-JEPA 2 + VideoMAEv2 | Causal, 4 steps per chunk | 4 | 150 |
19
-
20
- Training uses a distribution batch of **256 generated videos**, a differentiation batch of **128 generated videos**, and a reference minibatch of **128 videos**. Inference produces **81-frame, 832 Γ— 480 videos at 16 FPS**.
21
-
22
- ## 1. Environment
23
-
24
- Use Linux, Python 3.10, and NVIDIA GPUs with BF16 support. Run the commands below from the `1.3B/` directory.
25
 
26
  ```bash
27
- conda create -n video-generation python=3.10 -y
28
- conda activate video-generation
29
-
30
  python -m pip install torch==2.5.1 torchvision==0.20.1 \
31
  --index-url https://download.pytorch.org/whl/cu121
32
  python -m pip install -r requirements.txt
33
  python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
34
-
35
  export PYTHON_BIN="$(command -v python)"
36
  ```
37
 
38
- ## 2. Models and Data
39
 
40
- ### Inference assets
41
 
42
- Inference needs the Wan2.1 base model, text encoder, tokenizer, VAE, and a trained Anonymous Video Generation checkpoint. Place the matching Wan2.1 assets in the local model directory:
43
 
44
- ```bash
45
- export ASSET_ROOT=/path/to/video-generation-assets
 
46
 
47
- # Place the matching model files at the local paths described below.
 
 
 
 
48
  ```
49
 
50
- The base model is stored under `$ASSET_ROOT/models/Wan2.1-T2V-1.3B/`. The trained generator checkpoint can be stored separately and selected with `CHECKPOINT` when running inference.
51
-
52
- ### Training assets
53
-
54
- Training starts from the Self-Forcing ODE initialization, stored locally at `$ASSET_ROOT/checkpoints/ode_init.pt`.
55
-
56
- It additionally requires:
57
-
58
- - **Frozen feature encoders:** DINOv3 ViT-L/16, V-JEPA 2 ViT-L, and VideoMAEv2-Large.
59
- - **Reference features:** precomputed features for the 8,682-video reference collection, with the aligned prompt/video manifest.
60
- - **NystrΓΆm references:** one fixed summary per encoder, built from the same collection.
61
-
62
- Reference preparation consists of extracting the three representations and building their NystrΓΆm summaries. During training, reference features are read directly from disk. See [Asset Preparation](docs/assets.md) for model sources, the directory layout, and preprocessing tools.
63
-
64
- **The trained Anonymous Video Generation weights and reference collection are not distributed with this source release.** Reproducing the reported training run requires the corresponding reference assets in addition to the public model downloads.
65
-
66
- ### Configure asset paths
67
-
68
- Generate the local configuration once the asset root is set:
69
 
70
  ```bash
71
- "$PYTHON_BIN" scripts/prepare_config.py --asset-root "$ASSET_ROOT"
 
72
  ```
73
 
74
- This creates `configs/local.yaml` from [the three-encoder configuration](configs/three_encoder.yaml). Local asset paths are excluded from version control. Download dependencies before launching a run; the training and inference scripts use local models in offline mode.
75
-
76
- ## 3. Quick Inference
77
 
78
- Create a text file with one prompt per line, then select a trained checkpoint:
79
 
80
  ```bash
81
- export CHECKPOINT=/path/to/video-generation/model.pt
82
-
83
  GPUS=0 \
84
- PROMPTS=/path/to/prompts.txt \
85
- OUTPUT_DIR=outputs/demo \
86
- bash scripts/infer.sh
87
  ```
88
 
89
- The launcher generates one video per prompt with seed 0, using the generator weights without EMA. The four-step inference path has no additional classifier-free guidance branch. Example prompts are provided in [test_8.txt](code/prompts/test_8.txt).
90
 
91
- ## 4. Training
92
 
93
- Prepare the environment and training assets above, then start the three-encoder run on four GPUs:
94
 
95
  ```bash
96
- export LOGDIR=runs/video-generation
97
- GPUS=0,1,2,3 bash scripts/train.sh
98
- ```
99
-
100
- The launcher uses FSDP and gradient checkpointing. Each worker generates 64 videos for distribution estimation and replays 32 with gradients. The feature encoders remain frozen, but gradients pass through them to the generated videos. W&B logging is disabled by the launcher.
101
-
102
- Use a separate `LOGDIR` for each run. Training starts from the ODE initialization and runs for **150 updates**, saving model weights, optimizer state, and per-worker random states every five updates. All checkpoints are retained. The final generator checkpoint is:
103
-
104
- ```text
105
- runs/video-generation/checkpoint_model_000150/model.pt
106
  ```
107
 
108
- Monitor progress with `tail -f "$LOGDIR/train.log"`. To generate videos from the trained model:
109
-
110
- ```bash
111
- export CHECKPOINT="$LOGDIR/checkpoint_model_000150/model.pt"
112
- GPUS=0 PROMPTS=/path/to/prompts.txt \
113
- OUTPUT_DIR=outputs/video-generation bash scripts/infer.sh
114
- ```
115
-
116
- <details>
117
- <summary><b>Training configuration</b></summary>
118
-
119
  | Setting | Value |
120
  |---|---|
121
- | Initialization | Self-Forcing ODE checkpoint |
122
- | Distribution batch | 256 generated videos |
123
- | Differentiation batch | 128 generated videos |
124
- | Reference minibatch | 128 videos, sampled with replacement |
125
  | Encoder weights | DINOv3 : V-JEPA 2 : VideoMAEv2 = 10 : 1 : 1 |
126
- | Loss aggregation | Sum of weighted, normalized branch losses divided by 3 |
127
- | Reference estimator | NystrΓΆm + Monte Carlo, weights 2 : 1 |
128
  | Optimizer | AdamW, betas (0, 0.999), weight decay 0 |
 
129
  | Gradient clipping | Maximum norm 10 |
130
- | Learning rate | Cosine decay from 2.5e-6 to 5e-7 over 125 updates, then constant |
131
- | Training length | 150 updates |
132
- | Checkpoint interval | Every 5 updates |
133
-
134
- The full configuration is in [configs/three_encoder.yaml](configs/three_encoder.yaml).
135
-
136
- </details>
137
 
138
- ### Short training run
139
 
140
- To check the complete training and checkpoint path before a full run:
141
 
142
- ```bash
143
- export LOGDIR=runs/video-generation-smoke
144
- GPUS=0,1,2,3 bash scripts/train.sh --smoke
145
- ```
146
-
147
- This runs **one optimizer update with the same batch sizes and all three encoders**. The resulting checkpoint is `$LOGDIR/checkpoint_model_000001/model.pt` and can be loaded with the inference command above.
148
 
149
- <details>
150
- <summary><b>Running in tmux and CPU tests</b></summary>
151
-
152
- On a machine with four allocated GPUs, start a persistent session:
153
 
154
  ```bash
155
- tmux new -s video-generation
 
156
  ```
157
-
158
- Inside the session, activate the environment, set `PYTHON_BIN` and `LOGDIR`, and run the training command. Detach with `Ctrl-b`, then `d`; reconnect with `tmux attach -t video-generation`.
159
-
160
- Run the CPU numerical tests separately:
161
-
162
- ```bash
163
- "$PYTHON_BIN" -m pip install pytest
164
- "$PYTHON_BIN" scripts/test_cpu.py
165
- ```
166
-
167
- See the [validation record](docs/validation.md) for revision-specific test results and the GPU environment used.
168
-
169
- </details>
170
-
171
- ## 5. Evaluation
172
-
173
- The paper reports the following result for the three-encoder generator after 150 updates:
174
-
175
- | Model | VBench Total ↑ | Quality ↑ | Semantic ↑ |
176
- |---|---:|---:|---:|
177
- | Anonymous Video Generation, 1.3B, three encoders | **84.25** | **85.06** | **80.99** |
178
-
179
- The per-dimension scores are available in the [reference summary](results/reference/three-encoder-summary.json). Use the upstream [VBench](https://github.com/Vchitect/VBench) evaluator to evaluate generated videos; VBench and its evaluation models are maintained separately from this training and inference release. A short training run verifies execution and does not reproduce the reported benchmark score.
180
-
181
- ## Acknowledgments
182
-
183
- This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing) and [Wan2.1](https://github.com/Wan-Video/Wan2.1). We also thank the authors of [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2), and [VBench](https://github.com/Vchitect/VBench) for making their work available. Bundled source retains its upstream license notices; model weights and datasets follow their respective upstream terms.
 
1
+ # Elastic Forcing β€” 1.3B
2
 
3
+ [Overview](../README.md) Β· [14B guide](../14B/README.md) Β· [Checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint/tree/main/checkpoints/1.3B)
4
 
5
+ The 1.3B model uses Wan2.1-T2V-1.3B with DINOv3, V-JEPA 2, and VideoMAEv2 for distribution matching. It generates 81-frame videos at 832 Γ— 480 resolution and 16 FPS, with four causal denoising steps per chunk.
6
 
7
+ ## Environment
 
8
 
9
+ Use Linux, Python 3.10, and NVIDIA GPUs with BF16 support. Run these commands from `1.3B/`:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
 
11
  ```bash
12
+ conda create -n elastic-forcing python=3.10 -y
13
+ conda activate elastic-forcing
 
14
  python -m pip install torch==2.5.1 torchvision==0.20.1 \
15
  --index-url https://download.pytorch.org/whl/cu121
16
  python -m pip install -r requirements.txt
17
  python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
 
18
  export PYTHON_BIN="$(command -v python)"
19
  ```
20
 
21
+ ## Models and assets
22
 
23
+ Inference requires the Wan2.1-T2V-1.3B base model, T5 text encoder, tokenizer, VAE, and the Elastic Forcing generator checkpoint. Place the base-model bundle under `$ASSET_ROOT/models/Wan2.1-T2V-1.3B/`. Training also requires the Self-Forcing ODE initialization, local DINOv3, V-JEPA 2, and VideoMAEv2 models, and precomputed reference features.
24
 
25
+ Download the generator checkpoint:
26
 
27
+ ```python
28
+ from pathlib import Path
29
+ from huggingface_hub import snapshot_download
30
 
31
+ folder = Path(snapshot_download(
32
+ "Elastic-forcing/Elastic-forcing-checkpoint",
33
+ allow_patterns=["checkpoints/1.3B/*"],
34
+ ))
35
+ print(folder / "checkpoints/1.3B/model.pt")
36
  ```
37
 
38
+ Set the local asset root and generate the configuration:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
 
40
  ```bash
41
+ export ASSET_ROOT=/path/to/assets
42
+ python scripts/prepare_config.py --asset-root "$ASSET_ROOT"
43
  ```
44
 
45
+ ## Inference
 
 
46
 
47
+ Use the downloaded checkpoint and a text file containing one prompt per line:
48
 
49
  ```bash
50
+ CHECKPOINT=/path/to/model.pt \
 
51
  GPUS=0 \
52
+ PROMPTS=code/prompts/test_8.txt \
53
+ OUTPUT_DIR=outputs/demo \
54
+ bash scripts/infer.sh
55
  ```
56
 
57
+ The launcher produces one video per prompt with seed 0 and non-EMA generator weights. Prepare base-model assets before launching; model loading uses local files.
58
 
59
+ ## Training
60
 
61
+ Training requires an ODE initialization, all three frozen encoders, a matched prompt/video manifest, a reference feature bank, and one NystrΓΆm summary per encoder. Prepare the model and reference assets at the locations specified in [configs/three_encoder.yaml](configs/three_encoder.yaml), generate the local configuration, and start training:
62
 
63
  ```bash
64
+ GPUS=0,1,2,3 LOGDIR=runs/train bash scripts/train.sh
 
 
 
 
 
 
 
 
 
65
  ```
66
 
 
 
 
 
 
 
 
 
 
 
 
67
  | Setting | Value |
68
  |---|---|
69
+ | Training updates | 150 |
70
+ | Distribution / differentiation / reference batch | 256 / 128 / 128 |
71
+ | Per-GPU generated / differentiated videos | 64 / 32 |
72
+ | Reference videos | 8,682 |
73
  | Encoder weights | DINOv3 : V-JEPA 2 : VideoMAEv2 = 10 : 1 : 1 |
74
+ | Loss divisor | 3 |
75
+ | NystrΓΆm / Monte Carlo weights | 2 : 1 |
76
  | Optimizer | AdamW, betas (0, 0.999), weight decay 0 |
77
+ | Learning rate | Cosine from 2.5e-6 to 5e-7 over 125 updates, then constant |
78
  | Gradient clipping | Maximum norm 10 |
79
+ | Checkpoint interval | Every five updates |
 
 
 
 
 
 
80
 
81
+ The launcher uses FSDP and gradient checkpointing. Training settings are in [configs/three_encoder.yaml](configs/three_encoder.yaml). Use a distinct `LOGDIR` for each run; checkpoints include optimizer and random-state files for resuming training.
82
 
83
+ ## Evaluation
84
 
85
+ | VBench Total ↑ | Quality ↑ | Semantic ↑ |
86
+ |---:|---:|---:|
87
+ | 84.25 | 85.06 | 80.99 |
 
 
 
88
 
89
+ Use [VBench](https://github.com/Vchitect/VBench) to evaluate generated videos. Per-dimension scores are in the [reference results](results/reference/three-encoder-summary.json). Run the CPU numerical tests from `1.3B/`:
 
 
 
90
 
91
  ```bash
92
+ python -m pip install pytest
93
+ python scripts/test_cpu.py
94
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14B/README.md CHANGED
@@ -1,22 +1,16 @@
1
- # Anonymous Video Generation β€” 14B
2
 
3
- [Project overview](../README.md) Β· [1.3B release](../1.3B/README.md)
4
 
5
- This directory provides the Wan2.1-T2V-14B implementation for **80 updates** of Anonymous Video Generation with **DINOv3, V-JEPA 2, and VideoMAEv2**. It includes the causal generator, frozen feature encoders, distribution-matching objective, gradient replay, CPU offload, checkpoint saving/resume, and inference code.
6
 
7
- This is the non-EMA checkpoint used for the human study and VLM evaluations. See [evaluated settings](docs/evaluated-settings.md) for the model identity and distinct sampling protocols.
8
 
9
- The step-80 recipe starts from the **causal ODE initialization**. It uses the first 80 updates of a 200-step cosine learning-rate schedule; shortening the learning-rate schedule to 80 steps changes the recipe.
10
-
11
- For an implementation walkthrough, see [Reading the code](../docs/paper-to-code.md). The configuration uses the paper's distribution and differentiation terminology; `code/utils/config.py` preserves legacy configuration compatibility.
12
-
13
- ## 1. Environment
14
-
15
- Run the following commands from `14B/`. Use Linux, Python 3.10, and CUDA-capable GPUs with BF16 support. The original training used **8 H200 GPUs** with CPU offload and approximately **1 TB of host RAM** allocated. Inference uses one GPU; this release does not establish a minimum VRAM requirement.
16
 
17
  ```bash
18
- conda create -n video-generation-14b python=3.10 -y
19
- conda activate video-generation-14b
20
  python -m pip install torch==2.5.1 torchvision==0.20.1 \
21
  --index-url https://download.pytorch.org/whl/cu121
22
  python -m pip install -r requirements.txt
@@ -24,106 +18,92 @@ python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
24
  export PYTHON_BIN="$(command -v python)"
25
  ```
26
 
27
- ## 2. Models and Data
28
 
29
- See [Asset preparation](docs/assets.md) for the directory layout, frozen encoder revisions, and reference-preparation tools.
30
 
31
- Inference requires the Wan2.1-T2V-14B base model, shared T5/tokenizer/VAE, and a trained Anonymous Video Generation `model.pt`. Training additionally requires the original causal ODE initialization and the **8,000-video three-encoder reference collection**. These experiment-specific weights and reference assets are not bundled in this source repository; downloading the public Wan model alone does not reproduce the trained checkpoint.
32
 
33
- ```bash
34
- export ASSET_ROOT=/path/to/video-generation-assets
35
- # Place the matching model files at the local paths described below.
36
-
37
- # After placing all training assets according to docs/assets.md:
38
- python scripts/prepare_config.py --asset-root "$ASSET_ROOT"
39
 
40
- # For inference alone, omit the training-bank requirement:
41
- # python scripts/prepare_config.py --asset-root "$ASSET_ROOT" --inference-only
 
 
 
 
 
 
 
42
  ```
43
 
44
- The setup script writes `configs/local.yaml` and `configs/local.env`, which are excluded from Git. It preserves the frozen feature-bank identity while separating the VideoMAEv2 load location from the identity stored in its reference metadata.
45
 
46
- ## 3. Inference
47
 
48
  ```bash
 
 
 
 
 
 
 
 
49
  GPUS=0 \
50
- CHECKPOINT=/path/to/checkpoint_model_000080/model.pt \
51
  PROMPTS=code/prompts/test_8.txt \
52
- OUTPUT_DIR=outputs/step80 \
53
  bash scripts/infer.sh
54
  ```
55
 
56
- The default matches the previous inference run: seed `0`, **five samples per prompt**, regular generator weights, and **no classifier-free guidance**. All five samples' initial and transition noises are drawn together; the generator then executes one video at a time to limit memory. Set `NUM_SAMPLES=1` for a single-video demo; this changes the random-number sequence. The seed advances continuously across prompts. Exact output reproduction also requires the same prompt order and shard split. The supplied pipeline performs four causal denoising steps followed by a terminal `t=0` cleanup, and uses the training-matched context-cache lifetime. The auxiliary noise seed is separately fixed in `configs/inference.yaml`.
57
-
58
- The `21` latent frames requested by the launcher decode to **81 frames at 832 Γ— 480 and 16 FPS**.
59
 
60
- ## 4. Training
61
 
62
- After preparing the training assets and local configuration:
63
 
64
  ```bash
65
- GPUS=0,1,2,3,4,5,6,7 \
66
- LOGDIR=runs/three-encoder \
67
- bash scripts/train.sh
68
  ```
69
 
70
- Training stops after update 80. Full model, optimizer, and per-rank random-state checkpoints are saved every 10 updates. Use `runs/three-encoder/checkpoint_model_000080/model.pt` for inference. Keep all companion files for resuming training. Use a fresh `LOGDIR` for each new experiment; eight full 14B checkpoints require substantial disk space.
71
-
72
  | Setting | Value |
73
  |---|---|
74
- | Generator | Wan2.1-T2V-14B, causal softmax attention |
75
- | Initialization | Causal ODE checkpoint |
76
- | GPUs | 8 |
77
  | Distribution / differentiation / reference batch | 256 / 128 / 128 |
78
- | Per-rank generated / differentiated videos | 32 / 16 |
79
- | Physical generation / logical feature / physical replay batch | 2 / 16 / 1 |
80
- | Reference collection | 8,000 videos |
81
- | Frozen feature encoders | DINOv3 final block; V-JEPA 2 and VideoMAEv2 block index 15 |
82
- | Loss aggregation | `(10 Γ— DINOv3 + V-JEPA 2 + VideoMAEv2) / 3`, after per-branch normalization |
83
- | Fixed / sampled reference attraction weights | 2 / 1 |
84
- | Optimizer | AdamW, betas `(0, 0.999)`, weight decay `0` |
85
- | Learning rate | Cosine `2.5e-6 β†’ 5e-7` over **200** updates; stop at **80** |
86
- | Generator gradient clipping | 10 |
87
- | EMA | Disabled |
88
- | Memory management | FSDP full sharding, activation checkpointing, optimizer/activation/frozen-encoder CPU offload |
89
-
90
- All three encoders are frozen, but their input gradients are retained for generator training. The original configuration uses `seed: 0`, which selects an effective seed at launch; set an explicit positive training seed for a repeatable new run. Matching the recipe alone does not recover the historical RNG state.
91
-
92
- For a one-update smoke test with the full batch geometry:
93
 
94
- ```bash
95
- LOGDIR=runs/smoke bash scripts/train.sh --smoke
96
- ```
97
 
98
- Resume a checkpoint produced by this release using the same eight-rank topology and asset identities:
99
 
100
  ```bash
101
  python scripts/prepare_config.py --asset-root "$ASSET_ROOT" \
102
- --resume-checkpoint "$PWD/runs/three-encoder/checkpoint_model_000070/model.pt"
103
- LOGDIR=runs/resumed bash scripts/train.sh
104
  ```
105
 
106
- This continues through update 80. Historical checkpoints may contain deployment-specific resume identities; the inference loader accepts their generator weights, but exact training resume across repackaged identities is not guaranteed.
107
 
108
- To run in tmux after activating the environment:
109
 
110
- ```bash
111
- tmux new-session -s video-generation-14b
112
- bash scripts/train.sh
113
- # Detach: Ctrl-b, then d. Reattach: tmux attach -t video-generation-14b
114
- ```
115
-
116
- ## 5. Validation and Evaluation
117
 
118
  ```bash
119
  python -m pip install pytest
120
  python scripts/test_cpu.py
121
  ```
122
-
123
- The CPU suite covers three-encoder configuration and feature-bank loading, partial-gradient geometry, MMD gradients, replay, offload state restoration, terminal cleanup/context-cache inference, sample-batch noise ordering, and the 200-step cosine curve. See [validation notes](docs/validation.md) for the distinction between the original GPU run and checks of this source package.
124
-
125
- For VBench, supply the benchmark prompt list to `scripts/infer.sh` and evaluate the resulting videos with the official [VBench](https://github.com/Vchitect/VBench) implementation. A sample-video smoke test is not a benchmark reproduction.
126
-
127
- ## Acknowledgments
128
-
129
- This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing), [Wan2.1](https://github.com/Wan-Video/Wan2.1), [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), and [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2). Upstream source notices are preserved in `code/LICENSE`; model weights and datasets retain their respective terms.
 
1
+ # Elastic Forcing β€” 14B
2
 
3
+ [Overview](../README.md) Β· [1.3B guide](../1.3B/README.md) Β· [Checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint/tree/main/checkpoints/14B)
4
 
5
+ The 14B model uses Wan2.1-T2V-14B with DINOv3, V-JEPA 2, and VideoMAEv2 for distribution matching. It generates 81-frame videos at 832 Γ— 480 resolution and 16 FPS. Inference uses four causal denoising steps followed by terminal cleanup.
6
 
7
+ ## Environment
8
 
9
+ Use Linux, Python 3.10, and NVIDIA GPUs with BF16 support. Run these commands from `14B/`:
 
 
 
 
 
 
10
 
11
  ```bash
12
+ conda create -n elastic-forcing-14b python=3.10 -y
13
+ conda activate elastic-forcing-14b
14
  python -m pip install torch==2.5.1 torchvision==0.20.1 \
15
  --index-url https://download.pytorch.org/whl/cu121
16
  python -m pip install -r requirements.txt
 
18
  export PYTHON_BIN="$(command -v python)"
19
  ```
20
 
21
+ ## Models and assets
22
 
23
+ Inference requires the Wan2.1-T2V-14B base model, T5 text encoder, tokenizer, VAE, and the Elastic Forcing generator checkpoint. Place the base-model bundle under `$ASSET_ROOT/models/Wan2.1-T2V-14B/`.
24
 
25
+ The generator checkpoint is stored as eight FP32 safetensors shards. Download them and create the `.pt` file expected by the inference launcher:
26
 
27
+ ```python
28
+ import json
29
+ from pathlib import Path
30
+ import torch
31
+ from huggingface_hub import snapshot_download
32
+ from safetensors.torch import load_file
33
 
34
+ folder = Path(snapshot_download(
35
+ "Elastic-forcing/Elastic-forcing-checkpoint",
36
+ allow_patterns=["checkpoints/14B/*"],
37
+ )) / "checkpoints/14B"
38
+ index = json.loads((folder / "model.safetensors.index.json").read_text())
39
+ state = {}
40
+ for shard in sorted(set(index["weight_map"].values())):
41
+ state.update(load_file(str(folder / shard), device="cpu"))
42
+ torch.save({"generator": state}, "/path/to/local/model.pt")
43
  ```
44
 
45
+ Conversion preserves tensor values and precision. Allow enough host memory and disk space for the approximately 57 GB FP32 generator and the output file.
46
 
47
+ For inference, prepare the base-model assets and generate the local configuration:
48
 
49
  ```bash
50
+ export ASSET_ROOT=/path/to/assets
51
+ python scripts/prepare_config.py --asset-root "$ASSET_ROOT" --inference-only
52
+ ```
53
+
54
+ ## Inference
55
+
56
+ ```bash
57
+ CHECKPOINT=/path/to/local/model.pt \
58
  GPUS=0 \
 
59
  PROMPTS=code/prompts/test_8.txt \
60
+ OUTPUT_DIR=outputs/demo \
61
  bash scripts/infer.sh
62
  ```
63
 
64
+ The defaults are seed 0 and five samples per prompt with non-EMA generator weights. Set `NUM_SAMPLES=1` for a single sample. The generator executes one video at a time while retaining the sampling order across prompts. Model loading uses local base-model assets.
 
 
65
 
66
+ ## Training
67
 
68
+ Training requires the causal ODE initialization, frozen encoders, matched prompt/video manifest, reference feature bank, and NystrΓΆm summaries. Prepare the model and reference assets at the locations specified in [configs/train.yaml](configs/train.yaml), then run:
69
 
70
  ```bash
71
+ python scripts/prepare_config.py --asset-root "$ASSET_ROOT"
72
+ GPUS=0,1,2,3,4,5,6,7 LOGDIR=runs/train bash scripts/train.sh
 
73
  ```
74
 
 
 
75
  | Setting | Value |
76
  |---|---|
77
+ | Training updates | 80 |
 
 
78
  | Distribution / differentiation / reference batch | 256 / 128 / 128 |
79
+ | Per-GPU generated / differentiated videos | 32 / 16 |
80
+ | Reference videos | 8,000 |
81
+ | Encoder weights | DINOv3 : V-JEPA 2 : VideoMAEv2 = 10 : 1 : 1 |
82
+ | Loss divisor | 3 |
83
+ | NystrΓΆm / Monte Carlo weights | 2 : 1 |
84
+ | Optimizer | AdamW, betas (0, 0.999), weight decay 0 |
85
+ | Learning rate | Cosine from 2.5e-6 to 5e-7 over 200 updates |
86
+ | Gradient clipping | Maximum norm 10 |
87
+ | Checkpoint interval | Every ten updates |
88
+ | Memory management | FSDP, activation checkpointing, optimizer and encoder CPU offload |
 
 
 
 
 
89
 
90
+ The learning-rate schedule spans 200 updates while training stops at update 80. Use a separate `LOGDIR` for each run. Keep checkpoint companion files for resuming training with the same distributed topology and asset configuration.
 
 
91
 
92
+ To resume:
93
 
94
  ```bash
95
  python scripts/prepare_config.py --asset-root "$ASSET_ROOT" \
96
+ --resume-checkpoint /path/to/checkpoint/model.pt
97
+ GPUS=0,1,2,3,4,5,6,7 LOGDIR=runs/resume bash scripts/train.sh
98
  ```
99
 
100
+ ## Evaluation and tests
101
 
102
+ Generate benchmark videos with consistent prompt ordering and sampling settings, then evaluate them with the official [VBench](https://github.com/Vchitect/VBench) implementation.
103
 
104
+ Run the CPU numerical tests from `14B/`:
 
 
 
 
 
 
105
 
106
  ```bash
107
  python -m pip install pytest
108
  python scripts/test_cpu.py
109
  ```
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -5,69 +5,61 @@ tags:
5
  - video-generation
6
  - pytorch
7
  ---
8
- <div align="center">
9
 
10
- # Elastic-forcing-code
11
 
12
- Anonymous source-code repository.
13
 
14
- [Model checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint)
15
-
16
- [1.3B Training & Inference](1.3B/README.md) Β· [14B Training & Inference](14B/README.md)
17
-
18
- </div>
19
-
20
- ---
21
-
22
-
23
- Anonymous Video Generation post-trains few-step autoregressive video generators by matching generated and reference videos in frozen representation spaces. It combines a hybrid NystrΓΆm–Monte Carlo distribution estimator with gradient replay. Only the generator is optimized; diffusion score models are not needed during post-training.
24
- Both released recipes use the same three-encoder MMD objective; no auxiliary frame-level distribution objective is enabled or included.
25
 
26
- Read the implementation through the [paper-to-code guide](docs/paper-to-code.md), which maps the paper's notation to configuration fields, the training loop, the hybrid MMD objective, and gradient replay.
 
 
 
27
 
28
- ## Models
29
 
30
- Each model directory contains its own implementation, configuration, dependency list, asset guide, and training/inference scripts. Run its commands from that directory.
31
 
32
- | Release | Frozen feature encoders | Training | Inference | Guide |
33
- |---|---|---|---|---|
34
- | **1.3B** | DINOv3 + V-JEPA 2 + VideoMAEv2 | 4 GPUs, 150 updates | 4 causal steps per chunk | [Training and inference](1.3B/README.md) |
35
- | **14B** | DINOv3 + V-JEPA 2 + VideoMAEv2 | 8 GPUs, 80 updates | 4 trained steps + terminal zero-step cleanup | [Training and inference](14B/README.md) |
36
 
37
- Both configurations use a distribution batch of 256 generated videos, a differentiation batch of 128 videos, and a reference minibatch of 128 videos. They generate 81-frame, 832 Γ— 480 videos at 16 FPS.
 
 
 
38
 
39
- The 1.3B three-encoder configuration achieves **84.25 VBench Total** in the paper. The 14B release targets the **step-80 checkpoint**, retaining the first 80 updates of the original **200-step cosine learning-rate schedule**. This is the **three-encoder, 8,000-reference-video model used in the human and VLM evaluations**; see [evaluated settings](14B/docs/evaluated-settings.md).
40
 
41
  ```text
42
  Elastic-forcing-code/
43
- β”œβ”€β”€ 1.3B/ # Three-encoder training and inference
44
  β”‚ β”œβ”€β”€ code/
45
  β”‚ β”œβ”€β”€ configs/
46
  β”‚ β”œβ”€β”€ scripts/
47
  β”‚ └── README.md
48
- β”œβ”€β”€ 14B/ # Three-encoder training and step-80 inference
49
  β”‚ β”œβ”€β”€ code/
50
  β”‚ β”œβ”€β”€ configs/
51
  β”‚ β”œβ”€β”€ scripts/
52
  β”‚ └── README.md
53
- └── docs/ # Implementation and validation guides
54
  ```
55
 
56
- ## Getting Started
57
-
58
- ```bash
59
- git clone https://huggingface.co/Elastic-forcing/Elastic-forcing-code
60
- cd Elastic-forcing-code
61
 
62
- # Choose one model, then follow its README.
63
- cd 14B
64
- # Or: cd 1.3B
65
- ```
66
 
67
- See the [1.3B asset guide](1.3B/docs/assets.md) or [14B asset guide](14B/docs/assets.md) for model and reference-data requirements. This release contains source code; trained generator weights, the 14B ODE initialization, and frozen reference assets are stored separately and are not bundled here.
 
 
68
 
 
69
 
 
70
 
71
- ## Acknowledgments
72
 
73
- This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing) and [Wan2.1](https://github.com/Wan-Video/Wan2.1). We also thank the authors of [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2), and [VBench](https://github.com/Vchitect/VBench) for making their work available. Bundled source retains its upstream license notices; model weights and datasets follow their respective upstream terms.
 
5
  - video-generation
6
  - pytorch
7
  ---
8
+ # Elastic Forcing
9
 
10
+ Elastic Forcing post-trains autoregressive video generators by matching generated and reference videos in frozen representation spaces. It combines hybrid NystrΓΆm–Monte Carlo distribution estimation with gradient replay.
11
 
12
+ [Model checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint) Β· [1.3B guide](1.3B/README.md) Β· [14B guide](14B/README.md)
13
 
14
+ ## Models
 
 
 
 
 
 
 
 
 
 
15
 
16
+ | Backbone | Frozen feature encoders | Training GPUs | Training updates | Inference |
17
+ |---|---|---:|---:|---|
18
+ | Wan2.1-T2V-1.3B | DINOv3 + V-JEPA 2 + VideoMAEv2 | 4 | 150 | Four causal denoising steps per chunk |
19
+ | Wan2.1-T2V-14B | DINOv3 + V-JEPA 2 + VideoMAEv2 | 8 | 80 | Four causal denoising steps and terminal cleanup |
20
 
21
+ Both models generate 81-frame videos at 832 Γ— 480 resolution and 16 FPS. Training uses a distribution batch of 256 generated videos, a differentiation batch of 128 generated videos, and a reference minibatch of 128 videos.
22
 
23
+ ## Getting started
24
 
25
+ ```bash
26
+ git clone https://huggingface.co/Elastic-forcing/Elastic-forcing-code
27
+ cd Elastic-forcing-code
 
28
 
29
+ # Select the model size and follow its guide.
30
+ cd 1.3B
31
+ # Or: cd 14B
32
+ ```
33
 
34
+ Each model directory includes its dependencies, configuration, asset guide, training launcher, and inference launcher. Download the matching generator checkpoint from the checkpoint repository and prepare the Wan2.1 base-model assets locally. Training additionally requires an ODE initialization, frozen feature encoders, and reference features.
35
 
36
  ```text
37
  Elastic-forcing-code/
38
+ β”œβ”€β”€ 1.3B/
39
  β”‚ β”œβ”€β”€ code/
40
  β”‚ β”œβ”€β”€ configs/
41
  β”‚ β”œβ”€β”€ scripts/
42
  β”‚ └── README.md
43
+ β”œβ”€β”€ 14B/
44
  β”‚ β”œβ”€β”€ code/
45
  β”‚ β”œβ”€β”€ configs/
46
  β”‚ β”œβ”€β”€ scripts/
47
  β”‚ └── README.md
48
+ └── docs/
49
  ```
50
 
51
+ The model guides describe dependencies, required assets, training settings, and inference commands.
 
 
 
 
52
 
53
+ ## Evaluation
 
 
 
54
 
55
+ | Model | VBench Total ↑ | Quality ↑ | Semantic ↑ |
56
+ |---|---:|---:|---:|
57
+ | Elastic Forcing 1.3B | 84.25 | 85.06 | 80.99 |
58
 
59
+ Per-dimension scores are provided in the [reference results](1.3B/results/reference/three-encoder-summary.json). Evaluate generated videos using the official [VBench](https://github.com/Vchitect/VBench) implementation.
60
 
61
+ ## Acknowledgments and license
62
 
63
+ This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing), [Wan2.1](https://github.com/Wan-Video/Wan2.1), [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), and [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2).
64
 
65
+ Source-code terms are provided in [LICENSE](LICENSE). Upstream source notices are retained; model assets and datasets follow their respective upstream terms.