Update documentation
Browse files- 1.3B/README.md +47 -136
- 14B/README.md +59 -79
- README.md +30 -38
1.3B/README.md
CHANGED
|
@@ -1,183 +1,94 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
-
[
|
| 4 |
|
| 5 |
-
---
|
| 6 |
|
| 7 |
-
|
| 8 |
-
The YAML uses the paper's distribution batch, differentiation batch, and reference minibatch terminology; `code/utils/config.py` keeps legacy configurations compatible.
|
| 9 |
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
This repository provides the **Wan2.1-T2V-1.3B three-encoder configuration**, using **DINOv3, V-JEPA 2, and VideoMAEv2**. This configuration achieves **84.25 VBench Total** in the paper.
|
| 13 |
-
|
| 14 |
-
## Supported Configuration
|
| 15 |
-
|
| 16 |
-
| Backbone | Feature encoders | Sampling | Training GPUs | Updates |
|
| 17 |
-
|---|---|---|---|---:|
|
| 18 |
-
| Wan2.1-T2V-1.3B | DINOv3 + V-JEPA 2 + VideoMAEv2 | Causal, 4 steps per chunk | 4 | 150 |
|
| 19 |
-
|
| 20 |
-
Training uses a distribution batch of **256 generated videos**, a differentiation batch of **128 generated videos**, and a reference minibatch of **128 videos**. Inference produces **81-frame, 832 Γ 480 videos at 16 FPS**.
|
| 21 |
-
|
| 22 |
-
## 1. Environment
|
| 23 |
-
|
| 24 |
-
Use Linux, Python 3.10, and NVIDIA GPUs with BF16 support. Run the commands below from the `1.3B/` directory.
|
| 25 |
|
| 26 |
```bash
|
| 27 |
-
conda create -n
|
| 28 |
-
conda activate
|
| 29 |
-
|
| 30 |
python -m pip install torch==2.5.1 torchvision==0.20.1 \
|
| 31 |
--index-url https://download.pytorch.org/whl/cu121
|
| 32 |
python -m pip install -r requirements.txt
|
| 33 |
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
|
| 34 |
-
|
| 35 |
export PYTHON_BIN="$(command -v python)"
|
| 36 |
```
|
| 37 |
|
| 38 |
-
##
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
|
| 43 |
|
| 44 |
-
```
|
| 45 |
-
|
|
|
|
| 46 |
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
```
|
| 49 |
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
### Training assets
|
| 53 |
-
|
| 54 |
-
Training starts from the Self-Forcing ODE initialization, stored locally at `$ASSET_ROOT/checkpoints/ode_init.pt`.
|
| 55 |
-
|
| 56 |
-
It additionally requires:
|
| 57 |
-
|
| 58 |
-
- **Frozen feature encoders:** DINOv3 ViT-L/16, V-JEPA 2 ViT-L, and VideoMAEv2-Large.
|
| 59 |
-
- **Reference features:** precomputed features for the 8,682-video reference collection, with the aligned prompt/video manifest.
|
| 60 |
-
- **NystrΓΆm references:** one fixed summary per encoder, built from the same collection.
|
| 61 |
-
|
| 62 |
-
Reference preparation consists of extracting the three representations and building their NystrΓΆm summaries. During training, reference features are read directly from disk. See [Asset Preparation](docs/assets.md) for model sources, the directory layout, and preprocessing tools.
|
| 63 |
-
|
| 64 |
-
**The trained Anonymous Video Generation weights and reference collection are not distributed with this source release.** Reproducing the reported training run requires the corresponding reference assets in addition to the public model downloads.
|
| 65 |
-
|
| 66 |
-
### Configure asset paths
|
| 67 |
-
|
| 68 |
-
Generate the local configuration once the asset root is set:
|
| 69 |
|
| 70 |
```bash
|
| 71 |
-
|
|
|
|
| 72 |
```
|
| 73 |
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
## 3. Quick Inference
|
| 77 |
|
| 78 |
-
|
| 79 |
|
| 80 |
```bash
|
| 81 |
-
|
| 82 |
-
|
| 83 |
GPUS=0 \
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
```
|
| 88 |
|
| 89 |
-
The launcher
|
| 90 |
|
| 91 |
-
##
|
| 92 |
|
| 93 |
-
Prepare the
|
| 94 |
|
| 95 |
```bash
|
| 96 |
-
|
| 97 |
-
GPUS=0,1,2,3 bash scripts/train.sh
|
| 98 |
-
```
|
| 99 |
-
|
| 100 |
-
The launcher uses FSDP and gradient checkpointing. Each worker generates 64 videos for distribution estimation and replays 32 with gradients. The feature encoders remain frozen, but gradients pass through them to the generated videos. W&B logging is disabled by the launcher.
|
| 101 |
-
|
| 102 |
-
Use a separate `LOGDIR` for each run. Training starts from the ODE initialization and runs for **150 updates**, saving model weights, optimizer state, and per-worker random states every five updates. All checkpoints are retained. The final generator checkpoint is:
|
| 103 |
-
|
| 104 |
-
```text
|
| 105 |
-
runs/video-generation/checkpoint_model_000150/model.pt
|
| 106 |
```
|
| 107 |
|
| 108 |
-
Monitor progress with `tail -f "$LOGDIR/train.log"`. To generate videos from the trained model:
|
| 109 |
-
|
| 110 |
-
```bash
|
| 111 |
-
export CHECKPOINT="$LOGDIR/checkpoint_model_000150/model.pt"
|
| 112 |
-
GPUS=0 PROMPTS=/path/to/prompts.txt \
|
| 113 |
-
OUTPUT_DIR=outputs/video-generation bash scripts/infer.sh
|
| 114 |
-
```
|
| 115 |
-
|
| 116 |
-
<details>
|
| 117 |
-
<summary><b>Training configuration</b></summary>
|
| 118 |
-
|
| 119 |
| Setting | Value |
|
| 120 |
|---|---|
|
| 121 |
-
|
|
| 122 |
-
| Distribution batch | 256
|
| 123 |
-
|
|
| 124 |
-
| Reference
|
| 125 |
| Encoder weights | DINOv3 : V-JEPA 2 : VideoMAEv2 = 10 : 1 : 1 |
|
| 126 |
-
| Loss
|
| 127 |
-
|
|
| 128 |
| Optimizer | AdamW, betas (0, 0.999), weight decay 0 |
|
|
|
|
| 129 |
| Gradient clipping | Maximum norm 10 |
|
| 130 |
-
|
|
| 131 |
-
| Training length | 150 updates |
|
| 132 |
-
| Checkpoint interval | Every 5 updates |
|
| 133 |
-
|
| 134 |
-
The full configuration is in [configs/three_encoder.yaml](configs/three_encoder.yaml).
|
| 135 |
-
|
| 136 |
-
</details>
|
| 137 |
|
| 138 |
-
|
| 139 |
|
| 140 |
-
|
| 141 |
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
```
|
| 146 |
-
|
| 147 |
-
This runs **one optimizer update with the same batch sizes and all three encoders**. The resulting checkpoint is `$LOGDIR/checkpoint_model_000001/model.pt` and can be loaded with the inference command above.
|
| 148 |
|
| 149 |
-
|
| 150 |
-
<summary><b>Running in tmux and CPU tests</b></summary>
|
| 151 |
-
|
| 152 |
-
On a machine with four allocated GPUs, start a persistent session:
|
| 153 |
|
| 154 |
```bash
|
| 155 |
-
|
|
|
|
| 156 |
```
|
| 157 |
-
|
| 158 |
-
Inside the session, activate the environment, set `PYTHON_BIN` and `LOGDIR`, and run the training command. Detach with `Ctrl-b`, then `d`; reconnect with `tmux attach -t video-generation`.
|
| 159 |
-
|
| 160 |
-
Run the CPU numerical tests separately:
|
| 161 |
-
|
| 162 |
-
```bash
|
| 163 |
-
"$PYTHON_BIN" -m pip install pytest
|
| 164 |
-
"$PYTHON_BIN" scripts/test_cpu.py
|
| 165 |
-
```
|
| 166 |
-
|
| 167 |
-
See the [validation record](docs/validation.md) for revision-specific test results and the GPU environment used.
|
| 168 |
-
|
| 169 |
-
</details>
|
| 170 |
-
|
| 171 |
-
## 5. Evaluation
|
| 172 |
-
|
| 173 |
-
The paper reports the following result for the three-encoder generator after 150 updates:
|
| 174 |
-
|
| 175 |
-
| Model | VBench Total β | Quality β | Semantic β |
|
| 176 |
-
|---|---:|---:|---:|
|
| 177 |
-
| Anonymous Video Generation, 1.3B, three encoders | **84.25** | **85.06** | **80.99** |
|
| 178 |
-
|
| 179 |
-
The per-dimension scores are available in the [reference summary](results/reference/three-encoder-summary.json). Use the upstream [VBench](https://github.com/Vchitect/VBench) evaluator to evaluate generated videos; VBench and its evaluation models are maintained separately from this training and inference release. A short training run verifies execution and does not reproduce the reported benchmark score.
|
| 180 |
-
|
| 181 |
-
## Acknowledgments
|
| 182 |
-
|
| 183 |
-
This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing) and [Wan2.1](https://github.com/Wan-Video/Wan2.1). We also thank the authors of [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2), and [VBench](https://github.com/Vchitect/VBench) for making their work available. Bundled source retains its upstream license notices; model weights and datasets follow their respective upstream terms.
|
|
|
|
| 1 |
+
# Elastic Forcing β 1.3B
|
| 2 |
|
| 3 |
+
[Overview](../README.md) Β· [14B guide](../14B/README.md) Β· [Checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint/tree/main/checkpoints/1.3B)
|
| 4 |
|
| 5 |
+
The 1.3B model uses Wan2.1-T2V-1.3B with DINOv3, V-JEPA 2, and VideoMAEv2 for distribution matching. It generates 81-frame videos at 832 Γ 480 resolution and 16 FPS, with four causal denoising steps per chunk.
|
| 6 |
|
| 7 |
+
## Environment
|
|
|
|
| 8 |
|
| 9 |
+
Use Linux, Python 3.10, and NVIDIA GPUs with BF16 support. Run these commands from `1.3B/`:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
```bash
|
| 12 |
+
conda create -n elastic-forcing python=3.10 -y
|
| 13 |
+
conda activate elastic-forcing
|
|
|
|
| 14 |
python -m pip install torch==2.5.1 torchvision==0.20.1 \
|
| 15 |
--index-url https://download.pytorch.org/whl/cu121
|
| 16 |
python -m pip install -r requirements.txt
|
| 17 |
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
|
|
|
|
| 18 |
export PYTHON_BIN="$(command -v python)"
|
| 19 |
```
|
| 20 |
|
| 21 |
+
## Models and assets
|
| 22 |
|
| 23 |
+
Inference requires the Wan2.1-T2V-1.3B base model, T5 text encoder, tokenizer, VAE, and the Elastic Forcing generator checkpoint. Place the base-model bundle under `$ASSET_ROOT/models/Wan2.1-T2V-1.3B/`. Training also requires the Self-Forcing ODE initialization, local DINOv3, V-JEPA 2, and VideoMAEv2 models, and precomputed reference features.
|
| 24 |
|
| 25 |
+
Download the generator checkpoint:
|
| 26 |
|
| 27 |
+
```python
|
| 28 |
+
from pathlib import Path
|
| 29 |
+
from huggingface_hub import snapshot_download
|
| 30 |
|
| 31 |
+
folder = Path(snapshot_download(
|
| 32 |
+
"Elastic-forcing/Elastic-forcing-checkpoint",
|
| 33 |
+
allow_patterns=["checkpoints/1.3B/*"],
|
| 34 |
+
))
|
| 35 |
+
print(folder / "checkpoints/1.3B/model.pt")
|
| 36 |
```
|
| 37 |
|
| 38 |
+
Set the local asset root and generate the configuration:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
```bash
|
| 41 |
+
export ASSET_ROOT=/path/to/assets
|
| 42 |
+
python scripts/prepare_config.py --asset-root "$ASSET_ROOT"
|
| 43 |
```
|
| 44 |
|
| 45 |
+
## Inference
|
|
|
|
|
|
|
| 46 |
|
| 47 |
+
Use the downloaded checkpoint and a text file containing one prompt per line:
|
| 48 |
|
| 49 |
```bash
|
| 50 |
+
CHECKPOINT=/path/to/model.pt \
|
|
|
|
| 51 |
GPUS=0 \
|
| 52 |
+
PROMPTS=code/prompts/test_8.txt \
|
| 53 |
+
OUTPUT_DIR=outputs/demo \
|
| 54 |
+
bash scripts/infer.sh
|
| 55 |
```
|
| 56 |
|
| 57 |
+
The launcher produces one video per prompt with seed 0 and non-EMA generator weights. Prepare base-model assets before launching; model loading uses local files.
|
| 58 |
|
| 59 |
+
## Training
|
| 60 |
|
| 61 |
+
Training requires an ODE initialization, all three frozen encoders, a matched prompt/video manifest, a reference feature bank, and one NystrΓΆm summary per encoder. Prepare the model and reference assets at the locations specified in [configs/three_encoder.yaml](configs/three_encoder.yaml), generate the local configuration, and start training:
|
| 62 |
|
| 63 |
```bash
|
| 64 |
+
GPUS=0,1,2,3 LOGDIR=runs/train bash scripts/train.sh
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
```
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
| Setting | Value |
|
| 68 |
|---|---|
|
| 69 |
+
| Training updates | 150 |
|
| 70 |
+
| Distribution / differentiation / reference batch | 256 / 128 / 128 |
|
| 71 |
+
| Per-GPU generated / differentiated videos | 64 / 32 |
|
| 72 |
+
| Reference videos | 8,682 |
|
| 73 |
| Encoder weights | DINOv3 : V-JEPA 2 : VideoMAEv2 = 10 : 1 : 1 |
|
| 74 |
+
| Loss divisor | 3 |
|
| 75 |
+
| NystrΓΆm / Monte Carlo weights | 2 : 1 |
|
| 76 |
| Optimizer | AdamW, betas (0, 0.999), weight decay 0 |
|
| 77 |
+
| Learning rate | Cosine from 2.5e-6 to 5e-7 over 125 updates, then constant |
|
| 78 |
| Gradient clipping | Maximum norm 10 |
|
| 79 |
+
| Checkpoint interval | Every five updates |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
+
The launcher uses FSDP and gradient checkpointing. Training settings are in [configs/three_encoder.yaml](configs/three_encoder.yaml). Use a distinct `LOGDIR` for each run; checkpoints include optimizer and random-state files for resuming training.
|
| 82 |
|
| 83 |
+
## Evaluation
|
| 84 |
|
| 85 |
+
| VBench Total β | Quality β | Semantic β |
|
| 86 |
+
|---:|---:|---:|
|
| 87 |
+
| 84.25 | 85.06 | 80.99 |
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
+
Use [VBench](https://github.com/Vchitect/VBench) to evaluate generated videos. Per-dimension scores are in the [reference results](results/reference/three-encoder-summary.json). Run the CPU numerical tests from `1.3B/`:
|
|
|
|
|
|
|
|
|
|
| 90 |
|
| 91 |
```bash
|
| 92 |
+
python -m pip install pytest
|
| 93 |
+
python scripts/test_cpu.py
|
| 94 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
14B/README.md
CHANGED
|
@@ -1,22 +1,16 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
-
[
|
| 4 |
|
| 5 |
-
|
| 6 |
|
| 7 |
-
|
| 8 |
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
For an implementation walkthrough, see [Reading the code](../docs/paper-to-code.md). The configuration uses the paper's distribution and differentiation terminology; `code/utils/config.py` preserves legacy configuration compatibility.
|
| 12 |
-
|
| 13 |
-
## 1. Environment
|
| 14 |
-
|
| 15 |
-
Run the following commands from `14B/`. Use Linux, Python 3.10, and CUDA-capable GPUs with BF16 support. The original training used **8 H200 GPUs** with CPU offload and approximately **1 TB of host RAM** allocated. Inference uses one GPU; this release does not establish a minimum VRAM requirement.
|
| 16 |
|
| 17 |
```bash
|
| 18 |
-
conda create -n
|
| 19 |
-
conda activate
|
| 20 |
python -m pip install torch==2.5.1 torchvision==0.20.1 \
|
| 21 |
--index-url https://download.pytorch.org/whl/cu121
|
| 22 |
python -m pip install -r requirements.txt
|
|
@@ -24,106 +18,92 @@ python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
|
|
| 24 |
export PYTHON_BIN="$(command -v python)"
|
| 25 |
```
|
| 26 |
|
| 27 |
-
##
|
| 28 |
|
| 29 |
-
|
| 30 |
|
| 31 |
-
|
| 32 |
|
| 33 |
-
```
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
|
| 40 |
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
```
|
| 43 |
|
| 44 |
-
|
| 45 |
|
| 46 |
-
|
| 47 |
|
| 48 |
```bash
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
GPUS=0 \
|
| 50 |
-
CHECKPOINT=/path/to/checkpoint_model_000080/model.pt \
|
| 51 |
PROMPTS=code/prompts/test_8.txt \
|
| 52 |
-
OUTPUT_DIR=outputs/
|
| 53 |
bash scripts/infer.sh
|
| 54 |
```
|
| 55 |
|
| 56 |
-
The
|
| 57 |
-
|
| 58 |
-
The `21` latent frames requested by the launcher decode to **81 frames at 832 Γ 480 and 16 FPS**.
|
| 59 |
|
| 60 |
-
##
|
| 61 |
|
| 62 |
-
|
| 63 |
|
| 64 |
```bash
|
| 65 |
-
|
| 66 |
-
LOGDIR=runs/
|
| 67 |
-
bash scripts/train.sh
|
| 68 |
```
|
| 69 |
|
| 70 |
-
Training stops after update 80. Full model, optimizer, and per-rank random-state checkpoints are saved every 10 updates. Use `runs/three-encoder/checkpoint_model_000080/model.pt` for inference. Keep all companion files for resuming training. Use a fresh `LOGDIR` for each new experiment; eight full 14B checkpoints require substantial disk space.
|
| 71 |
-
|
| 72 |
| Setting | Value |
|
| 73 |
|---|---|
|
| 74 |
-
|
|
| 75 |
-
| Initialization | Causal ODE checkpoint |
|
| 76 |
-
| GPUs | 8 |
|
| 77 |
| Distribution / differentiation / reference batch | 256 / 128 / 128 |
|
| 78 |
-
| Per-
|
| 79 |
-
|
|
| 80 |
-
|
|
| 81 |
-
|
|
| 82 |
-
|
|
| 83 |
-
|
|
| 84 |
-
|
|
| 85 |
-
|
|
| 86 |
-
|
|
| 87 |
-
|
|
| 88 |
-
| Memory management | FSDP full sharding, activation checkpointing, optimizer/activation/frozen-encoder CPU offload |
|
| 89 |
-
|
| 90 |
-
All three encoders are frozen, but their input gradients are retained for generator training. The original configuration uses `seed: 0`, which selects an effective seed at launch; set an explicit positive training seed for a repeatable new run. Matching the recipe alone does not recover the historical RNG state.
|
| 91 |
-
|
| 92 |
-
For a one-update smoke test with the full batch geometry:
|
| 93 |
|
| 94 |
-
``
|
| 95 |
-
LOGDIR=runs/smoke bash scripts/train.sh --smoke
|
| 96 |
-
```
|
| 97 |
|
| 98 |
-
|
| 99 |
|
| 100 |
```bash
|
| 101 |
python scripts/prepare_config.py --asset-root "$ASSET_ROOT" \
|
| 102 |
-
--resume-checkpoint
|
| 103 |
-
LOGDIR=runs/
|
| 104 |
```
|
| 105 |
|
| 106 |
-
|
| 107 |
|
| 108 |
-
|
| 109 |
|
| 110 |
-
``
|
| 111 |
-
tmux new-session -s video-generation-14b
|
| 112 |
-
bash scripts/train.sh
|
| 113 |
-
# Detach: Ctrl-b, then d. Reattach: tmux attach -t video-generation-14b
|
| 114 |
-
```
|
| 115 |
-
|
| 116 |
-
## 5. Validation and Evaluation
|
| 117 |
|
| 118 |
```bash
|
| 119 |
python -m pip install pytest
|
| 120 |
python scripts/test_cpu.py
|
| 121 |
```
|
| 122 |
-
|
| 123 |
-
The CPU suite covers three-encoder configuration and feature-bank loading, partial-gradient geometry, MMD gradients, replay, offload state restoration, terminal cleanup/context-cache inference, sample-batch noise ordering, and the 200-step cosine curve. See [validation notes](docs/validation.md) for the distinction between the original GPU run and checks of this source package.
|
| 124 |
-
|
| 125 |
-
For VBench, supply the benchmark prompt list to `scripts/infer.sh` and evaluate the resulting videos with the official [VBench](https://github.com/Vchitect/VBench) implementation. A sample-video smoke test is not a benchmark reproduction.
|
| 126 |
-
|
| 127 |
-
## Acknowledgments
|
| 128 |
-
|
| 129 |
-
This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing), [Wan2.1](https://github.com/Wan-Video/Wan2.1), [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), and [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2). Upstream source notices are preserved in `code/LICENSE`; model weights and datasets retain their respective terms.
|
|
|
|
| 1 |
+
# Elastic Forcing β 14B
|
| 2 |
|
| 3 |
+
[Overview](../README.md) Β· [1.3B guide](../1.3B/README.md) Β· [Checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint/tree/main/checkpoints/14B)
|
| 4 |
|
| 5 |
+
The 14B model uses Wan2.1-T2V-14B with DINOv3, V-JEPA 2, and VideoMAEv2 for distribution matching. It generates 81-frame videos at 832 Γ 480 resolution and 16 FPS. Inference uses four causal denoising steps followed by terminal cleanup.
|
| 6 |
|
| 7 |
+
## Environment
|
| 8 |
|
| 9 |
+
Use Linux, Python 3.10, and NVIDIA GPUs with BF16 support. Run these commands from `14B/`:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
```bash
|
| 12 |
+
conda create -n elastic-forcing-14b python=3.10 -y
|
| 13 |
+
conda activate elastic-forcing-14b
|
| 14 |
python -m pip install torch==2.5.1 torchvision==0.20.1 \
|
| 15 |
--index-url https://download.pytorch.org/whl/cu121
|
| 16 |
python -m pip install -r requirements.txt
|
|
|
|
| 18 |
export PYTHON_BIN="$(command -v python)"
|
| 19 |
```
|
| 20 |
|
| 21 |
+
## Models and assets
|
| 22 |
|
| 23 |
+
Inference requires the Wan2.1-T2V-14B base model, T5 text encoder, tokenizer, VAE, and the Elastic Forcing generator checkpoint. Place the base-model bundle under `$ASSET_ROOT/models/Wan2.1-T2V-14B/`.
|
| 24 |
|
| 25 |
+
The generator checkpoint is stored as eight FP32 safetensors shards. Download them and create the `.pt` file expected by the inference launcher:
|
| 26 |
|
| 27 |
+
```python
|
| 28 |
+
import json
|
| 29 |
+
from pathlib import Path
|
| 30 |
+
import torch
|
| 31 |
+
from huggingface_hub import snapshot_download
|
| 32 |
+
from safetensors.torch import load_file
|
| 33 |
|
| 34 |
+
folder = Path(snapshot_download(
|
| 35 |
+
"Elastic-forcing/Elastic-forcing-checkpoint",
|
| 36 |
+
allow_patterns=["checkpoints/14B/*"],
|
| 37 |
+
)) / "checkpoints/14B"
|
| 38 |
+
index = json.loads((folder / "model.safetensors.index.json").read_text())
|
| 39 |
+
state = {}
|
| 40 |
+
for shard in sorted(set(index["weight_map"].values())):
|
| 41 |
+
state.update(load_file(str(folder / shard), device="cpu"))
|
| 42 |
+
torch.save({"generator": state}, "/path/to/local/model.pt")
|
| 43 |
```
|
| 44 |
|
| 45 |
+
Conversion preserves tensor values and precision. Allow enough host memory and disk space for the approximately 57 GB FP32 generator and the output file.
|
| 46 |
|
| 47 |
+
For inference, prepare the base-model assets and generate the local configuration:
|
| 48 |
|
| 49 |
```bash
|
| 50 |
+
export ASSET_ROOT=/path/to/assets
|
| 51 |
+
python scripts/prepare_config.py --asset-root "$ASSET_ROOT" --inference-only
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
## Inference
|
| 55 |
+
|
| 56 |
+
```bash
|
| 57 |
+
CHECKPOINT=/path/to/local/model.pt \
|
| 58 |
GPUS=0 \
|
|
|
|
| 59 |
PROMPTS=code/prompts/test_8.txt \
|
| 60 |
+
OUTPUT_DIR=outputs/demo \
|
| 61 |
bash scripts/infer.sh
|
| 62 |
```
|
| 63 |
|
| 64 |
+
The defaults are seed 0 and five samples per prompt with non-EMA generator weights. Set `NUM_SAMPLES=1` for a single sample. The generator executes one video at a time while retaining the sampling order across prompts. Model loading uses local base-model assets.
|
|
|
|
|
|
|
| 65 |
|
| 66 |
+
## Training
|
| 67 |
|
| 68 |
+
Training requires the causal ODE initialization, frozen encoders, matched prompt/video manifest, reference feature bank, and NystrΓΆm summaries. Prepare the model and reference assets at the locations specified in [configs/train.yaml](configs/train.yaml), then run:
|
| 69 |
|
| 70 |
```bash
|
| 71 |
+
python scripts/prepare_config.py --asset-root "$ASSET_ROOT"
|
| 72 |
+
GPUS=0,1,2,3,4,5,6,7 LOGDIR=runs/train bash scripts/train.sh
|
|
|
|
| 73 |
```
|
| 74 |
|
|
|
|
|
|
|
| 75 |
| Setting | Value |
|
| 76 |
|---|---|
|
| 77 |
+
| Training updates | 80 |
|
|
|
|
|
|
|
| 78 |
| Distribution / differentiation / reference batch | 256 / 128 / 128 |
|
| 79 |
+
| Per-GPU generated / differentiated videos | 32 / 16 |
|
| 80 |
+
| Reference videos | 8,000 |
|
| 81 |
+
| Encoder weights | DINOv3 : V-JEPA 2 : VideoMAEv2 = 10 : 1 : 1 |
|
| 82 |
+
| Loss divisor | 3 |
|
| 83 |
+
| NystrΓΆm / Monte Carlo weights | 2 : 1 |
|
| 84 |
+
| Optimizer | AdamW, betas (0, 0.999), weight decay 0 |
|
| 85 |
+
| Learning rate | Cosine from 2.5e-6 to 5e-7 over 200 updates |
|
| 86 |
+
| Gradient clipping | Maximum norm 10 |
|
| 87 |
+
| Checkpoint interval | Every ten updates |
|
| 88 |
+
| Memory management | FSDP, activation checkpointing, optimizer and encoder CPU offload |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
The learning-rate schedule spans 200 updates while training stops at update 80. Use a separate `LOGDIR` for each run. Keep checkpoint companion files for resuming training with the same distributed topology and asset configuration.
|
|
|
|
|
|
|
| 91 |
|
| 92 |
+
To resume:
|
| 93 |
|
| 94 |
```bash
|
| 95 |
python scripts/prepare_config.py --asset-root "$ASSET_ROOT" \
|
| 96 |
+
--resume-checkpoint /path/to/checkpoint/model.pt
|
| 97 |
+
GPUS=0,1,2,3,4,5,6,7 LOGDIR=runs/resume bash scripts/train.sh
|
| 98 |
```
|
| 99 |
|
| 100 |
+
## Evaluation and tests
|
| 101 |
|
| 102 |
+
Generate benchmark videos with consistent prompt ordering and sampling settings, then evaluate them with the official [VBench](https://github.com/Vchitect/VBench) implementation.
|
| 103 |
|
| 104 |
+
Run the CPU numerical tests from `14B/`:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
```bash
|
| 107 |
python -m pip install pytest
|
| 108 |
python scripts/test_cpu.py
|
| 109 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README.md
CHANGED
|
@@ -5,69 +5,61 @@ tags:
|
|
| 5 |
- video-generation
|
| 6 |
- pytorch
|
| 7 |
---
|
| 8 |
-
|
| 9 |
|
| 10 |
-
|
| 11 |
|
| 12 |
-
|
| 13 |
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
[1.3B Training & Inference](1.3B/README.md) Β· [14B Training & Inference](14B/README.md)
|
| 17 |
-
|
| 18 |
-
</div>
|
| 19 |
-
|
| 20 |
-
---
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
Anonymous Video Generation post-trains few-step autoregressive video generators by matching generated and reference videos in frozen representation spaces. It combines a hybrid NystrΓΆmβMonte Carlo distribution estimator with gradient replay. Only the generator is optimized; diffusion score models are not needed during post-training.
|
| 24 |
-
Both released recipes use the same three-encoder MMD objective; no auxiliary frame-level distribution objective is enabled or included.
|
| 25 |
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
-
|
| 29 |
|
| 30 |
-
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
| **14B** | DINOv3 + V-JEPA 2 + VideoMAEv2 | 8 GPUs, 80 updates | 4 trained steps + terminal zero-step cleanup | [Training and inference](14B/README.md) |
|
| 36 |
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
-
|
| 40 |
|
| 41 |
```text
|
| 42 |
Elastic-forcing-code/
|
| 43 |
-
βββ 1.3B/
|
| 44 |
β βββ code/
|
| 45 |
β βββ configs/
|
| 46 |
β βββ scripts/
|
| 47 |
β βββ README.md
|
| 48 |
-
βββ 14B/
|
| 49 |
β βββ code/
|
| 50 |
β βββ configs/
|
| 51 |
β βββ scripts/
|
| 52 |
β βββ README.md
|
| 53 |
-
βββ docs/
|
| 54 |
```
|
| 55 |
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
```bash
|
| 59 |
-
git clone https://huggingface.co/Elastic-forcing/Elastic-forcing-code
|
| 60 |
-
cd Elastic-forcing-code
|
| 61 |
|
| 62 |
-
#
|
| 63 |
-
cd 14B
|
| 64 |
-
# Or: cd 1.3B
|
| 65 |
-
```
|
| 66 |
|
| 67 |
-
|
|
|
|
|
|
|
| 68 |
|
|
|
|
| 69 |
|
|
|
|
| 70 |
|
| 71 |
-
|
| 72 |
|
| 73 |
-
|
|
|
|
| 5 |
- video-generation
|
| 6 |
- pytorch
|
| 7 |
---
|
| 8 |
+
# Elastic Forcing
|
| 9 |
|
| 10 |
+
Elastic Forcing post-trains autoregressive video generators by matching generated and reference videos in frozen representation spaces. It combines hybrid NystrΓΆmβMonte Carlo distribution estimation with gradient replay.
|
| 11 |
|
| 12 |
+
[Model checkpoints](https://huggingface.co/Elastic-forcing/Elastic-forcing-checkpoint) Β· [1.3B guide](1.3B/README.md) Β· [14B guide](14B/README.md)
|
| 13 |
|
| 14 |
+
## Models
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
+
| Backbone | Frozen feature encoders | Training GPUs | Training updates | Inference |
|
| 17 |
+
|---|---|---:|---:|---|
|
| 18 |
+
| Wan2.1-T2V-1.3B | DINOv3 + V-JEPA 2 + VideoMAEv2 | 4 | 150 | Four causal denoising steps per chunk |
|
| 19 |
+
| Wan2.1-T2V-14B | DINOv3 + V-JEPA 2 + VideoMAEv2 | 8 | 80 | Four causal denoising steps and terminal cleanup |
|
| 20 |
|
| 21 |
+
Both models generate 81-frame videos at 832 Γ 480 resolution and 16 FPS. Training uses a distribution batch of 256 generated videos, a differentiation batch of 128 generated videos, and a reference minibatch of 128 videos.
|
| 22 |
|
| 23 |
+
## Getting started
|
| 24 |
|
| 25 |
+
```bash
|
| 26 |
+
git clone https://huggingface.co/Elastic-forcing/Elastic-forcing-code
|
| 27 |
+
cd Elastic-forcing-code
|
|
|
|
| 28 |
|
| 29 |
+
# Select the model size and follow its guide.
|
| 30 |
+
cd 1.3B
|
| 31 |
+
# Or: cd 14B
|
| 32 |
+
```
|
| 33 |
|
| 34 |
+
Each model directory includes its dependencies, configuration, asset guide, training launcher, and inference launcher. Download the matching generator checkpoint from the checkpoint repository and prepare the Wan2.1 base-model assets locally. Training additionally requires an ODE initialization, frozen feature encoders, and reference features.
|
| 35 |
|
| 36 |
```text
|
| 37 |
Elastic-forcing-code/
|
| 38 |
+
βββ 1.3B/
|
| 39 |
β βββ code/
|
| 40 |
β βββ configs/
|
| 41 |
β βββ scripts/
|
| 42 |
β βββ README.md
|
| 43 |
+
βββ 14B/
|
| 44 |
β βββ code/
|
| 45 |
β βββ configs/
|
| 46 |
β βββ scripts/
|
| 47 |
β βββ README.md
|
| 48 |
+
βββ docs/
|
| 49 |
```
|
| 50 |
|
| 51 |
+
The model guides describe dependencies, required assets, training settings, and inference commands.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
+
## Evaluation
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
+
| Model | VBench Total β | Quality β | Semantic β |
|
| 56 |
+
|---|---:|---:|---:|
|
| 57 |
+
| Elastic Forcing 1.3B | 84.25 | 85.06 | 80.99 |
|
| 58 |
|
| 59 |
+
Per-dimension scores are provided in the [reference results](1.3B/results/reference/three-encoder-summary.json). Evaluate generated videos using the official [VBench](https://github.com/Vchitect/VBench) implementation.
|
| 60 |
|
| 61 |
+
## Acknowledgments and license
|
| 62 |
|
| 63 |
+
This implementation builds on [Self-Forcing](https://github.com/guandeh17/Self-Forcing), [Wan2.1](https://github.com/Wan-Video/Wan2.1), [DINOv3](https://github.com/facebookresearch/dinov3), [V-JEPA 2](https://github.com/facebookresearch/vjepa2), and [VideoMAEv2](https://github.com/OpenGVLab/VideoMAEv2).
|
| 64 |
|
| 65 |
+
Source-code terms are provided in [LICENSE](LICENSE). Upstream source notices are retained; model assets and datasets follow their respective upstream terms.
|