Duplicate from JonathanColetti/LiveWan

Co-authored-by: Jonathan Coletti <JonathanColetti@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,43 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+samples/w44_diff_0_to_80.png filter=lfs diff=lfs merge=lfs -text
+samples/w60_diff_0_to_80.png filter=lfs diff=lfs merge=lfs -text
+samples/w60_edge_over_time.png filter=lfs diff=lfs merge=lfs -text
+samples/w60_filmstrip.png filter=lfs diff=lfs merge=lfs -text
+samples/world_p0_woman_16fps.mp4 filter=lfs diff=lfs merge=lfs -text
+samples/world_p44_owl_16fps.mp4 filter=lfs diff=lfs merge=lfs -text
+samples/world_p60_waterfall_16fps.mp4 filter=lfs diff=lfs merge=lfs -text
+samples/world_p82_court_16fps.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,163 @@
+---
+license: apache-2.0
+base_model: Wan-AI/Wan2.1-T2V-1.3B
+pipeline_tag: text-to-video
+library_name: livewan
+tags:
+  - text-to-video
+  - streaming
+  - real-time
+  - video-generation
+  - distillation
+  - dmd
+  - wan2.1
+  - wanstreamer
+---
+
+# LiveWan: streaming text 2 video, 3000 steps
+
+> **Unofficial community project.** Not affiliated with, endorsed by, or produced
+> by Alibaba Group or the Wan-Video team. Built on their Apache-2.0
+> [Wan2.1](https://github.com/Wan-Video/Wan2.1) release.
+
+A 1.3 B student distilled from a Wan2.1-14B teacher that generates video
+**continuously** rather than as a fixed clip: 750 ms of 640x368 video per block,
+generated in 428 ms on one A100 80GB, extended block by block for as long as you
+let it run. The text conditioning can be changed mid-stream without clearing the
+K/V cache, so the scene continues instead of cutting.
+
+**Code, setup and documentation: [JonathanColetti/LiveWan](https://github.com/JonathanColetti/LiveWan).** This repository holds
+weights and data only. It is not usable on its own.
+
+## Not audio
+
+The model is text-to-video and was never trained to articulate speech. There is no
+audio, no lip sync and no speech anywhere in this project. Text steers **the scene**.
+A face that appears to talk is generating plausible mouth motion, not saying your
+sentence.
+
+## Files
+
+**To run the model — 6.0 GB.** This is what `setup.sh` fetches.
+
+| path | size | what |
+|---|---|---|
+| `checkpoints/t14b_b64/milestones/step002250_noema.pt` | 5.3 GB | **the model** — 825 tensors, bf16, no EMA copy |
+| `checkpoints/t14b_b64/history.json` | 100 KB | the full training history of the run |
+| `data/prompts.pt` | 388 MB | umt5-xxl embeddings for 96 prompts, indices 0–95 |
+| `out/world_p{0,44,60,82}.pt` | 4 × 59 MB | the four cached evaluation worlds — skip base-model generation entirely |
+| `samples/` | 16 MB | reference clips and analysis filmstrips from those worlds |
+
+**To continue the run** Not needed for inference. do not
+download these unless you intend to train.
+
+| path | size | what |
+|---|---|---|
+| `checkpoints/t14b_b64/latest.pt` | 10.6 GB | step 3000, **carries its EMA copy**. The comparison arm, and the weights a resume starts from |
+| `checkpoints/t14b_b64/opt_rank00{0..7}.pt` | 8 × 2.2 GB | FSDP-sharded AdamW state at step 3000, one shard per rank |
+
+Nothing here duplicates a public download. The Wan2.1 base model, its VAE and the
+umt5-xxl encoder are not mirrored in this repository. `setup.sh` pulls
+[Wan-AI/Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) (17 GB)
+from its own repo, and the 14B teacher is only needed if you are distilling.
+
+The layout mirrors the code repository, so `setup.sh` downloads straight into place.
+To take only part of it:
+
+```bash
+hf download JonathanColetti/LiveWan --local-dir . \
+  --exclude "checkpoints/t14b_b64/latest.pt" "checkpoints/t14b_b64/opt_rank*"
+```
+
+## Use
+
+```bash
+git clone https://github.com/JonathanColetti/LiveWan && cd LiveWan
+pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
+./setup.sh                       # pulls this repo, checksum-verified
+
+python scripts/demo.py \
+  --weights checkpoints/t14b_b64/milestones/step002250_noema.pt \
+  --world-cache out/world_p60.pt --prompt-idx 60 \
+  --latent-norm 1.0 --block 3 --steps 2 --window 6 --units 60 --fps 16 \
+  --out out/demo_w60
+```
+
+Needs ~10 GiB of VRAM. `scripts/live_demo.py` serves the same model as a steerable
+browser stream.
+
+**`--prompt-idx` must match the world**: `world_p60.pt` goes with `--prompt-idx 60`.
+Indices run 0–95; anything above fails.
+
+
+## Training
+
+3000 iterations of SF-DMD distillation from a Wan2.1-T2V-14B teacher into a
+Wan2.1-T2V-1.3B student, effective batch 64 (8×H200, accum 8, FSDP), **41.6 hours**
+at 66.5 s/it, zero interventions. Losses do not decrease in this trainer and should
+not (the critic is retrained every step, so the generator holds position against a
+strengthening opponent).
+
+## Continuing the run
+
+The optimizer shards make extending the run a **resume rather than a restart**.
+Put `latest.pt`, all eight `opt_rank*.pt` and `history.json` in the same run
+directory and:
+
+```bash
+ITERS=4000 ./run_b64_resume.sh
+```
+
+Four things decide whether this works:
+
+- **Exactly 8 ranks.** These are per-rank FSDP shards, not a consolidated state
+  dict. World size 8 is what wrote them and world size 8 is what can read them;
+  fewer ranks means *larger* shards, not smaller, so there is no repacking
+  shortcut. `run_b64_resume.sh` checks for all eight up front and refuses rather
+  than half-starting.
+- **You resume from step 3000, not from the released step 2250.** Optimizer state
+  and the EMA copy exist only for step 3000 — 2250 was archived `_noema` and its
+  EMA is gone. The released checkpoint is the better *model*; `latest.pt` is the
+  only *resumable* one.
+- **`--resume` overwrites the checkpoint it resumed from.** It reads
+  `OUT/latest.pt` and writes that same path at the first save.
+  `run_b64_resume.sh` preserves it as `step003000.pt` first — if you drive the
+  trainer directly instead, copy it aside yourself.
+- **~30 GiB must be free transiently at each save**, because the shards are
+  written as `.tmp` and renamed, so old and new coexist. Prune before the save,
+  not after; a post-save janitor cannot help, since the save is what runs out of
+  space.
+
+The resume path is verified rather than assumed: killing a run at step 12 brought
+AdamW's internal counters back as 16/18 rather than reset, and the real 750→3000
+resume held `loss_gen`, `loss_critic`, `dmd_grad_norm` and `gn_gen` within one
+standard deviation of the 60-point pre-crash reference. A subtly broken restore
+shows up as a shifted mean or changed variance; neither appeared.
+
+## Verified
+
+- all 825 tensors load, **zero non-finite values**, weight norms 0.018–115.4
+- three correctness gates pass with their deliberately broken controls failing, both
+  before and after training
+- 24 streaming cells across every arm and world met the real-time budget
+- same box, same seed → **byte-identical** output
+
+## Limits
+
+- **Quality holds for roughly a minute.** By 160 s sharpness has fallen to ~73% of
+  the world's.
+- **Maximum 1024 latent frames** (4.3 minutes at block 3), set by `WanModel.freqs`.
+- **The pixel check has not been done.** Checkpoint selection rests on proxy metrics,
+  and in this project nine automatic measurements have pointed the wrong way — every
+  one caught by looking at pixels at 1:1.
+- **World 82 is unexplained**: every trained checkpoint drifts on it while the
+  untrained baseline sits at −0.001.
+- **The batch-64 premise is unsettled** — confounded with having simply trained
+  longer.
+
+
+
+## Licence
+
+Apache-2.0. Derived from [Wan2.1](https://github.com/Wan-Video/Wan2.1)
+(Apache-2.0, Alibaba Group).

checkpoints/t14b_b64/latest.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4fe559eeaff4c9e545ca4c5a9fe7e39c1a51cc49e5e992a4697700ec9dc02a5
+size 11352491549

checkpoints/t14b_b64/milestones/step002250_noema.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f7619d2f96f253c9ef18c6c026e961a82c0da7798f098943325719121bcbb17
+size 5676251781

checkpoints/t14b_b64/opt_rank000.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f0f9e7214e108f0cf63a4ea9039113b2438ac7911f1fc5a640fad8f79d2ec2b4
+size 2379940269

checkpoints/t14b_b64/opt_rank001.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f6925320fb258359e01eeec107699e3d6742c1141ccc6c1ac35400401721239
+size 2376579565

checkpoints/t14b_b64/opt_rank002.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef94e9c70f6cd0227f4a9f9437bc242846ab904760cbabbe02b4e2d0a4a1f024
+size 2376579565

checkpoints/t14b_b64/opt_rank003.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6831bfd1c73f45de07213cba5cb8db89cc3ed5b5e6ac11a7cc8244bc0d16c83
+size 2376579565

checkpoints/t14b_b64/opt_rank004.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7212cb3da6624fc752c3861ff05c53b8d9aad9f515e872e126a49b98067ae80d
+size 2376579565

checkpoints/t14b_b64/opt_rank005.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13e4ae9ff3479dd8f6937a19a2ce7935fe28fb059661f11541aff520180d2495
+size 2376579565

checkpoints/t14b_b64/opt_rank006.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:936f712c9d14ca762d93f3871c29a9360959cf5fff38ac7f609fbde8f93a5669
+size 2376579565

checkpoints/t14b_b64/opt_rank007.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e352a437b3881fd5de834d48008e573d906d1279476635bdb978a8e3e45e992
+size 2376579565

checksums.sha256

@@ -0,0 +1,26 @@
+4256a6d4cedfc38fdbb4ccf9e21307f592737e844e0e05456ac9b6e8e493e43f  checkpoints/t14b_b64/history.json
+c4fe559eeaff4c9e545ca4c5a9fe7e39c1a51cc49e5e992a4697700ec9dc02a5  checkpoints/t14b_b64/latest.pt
+9f7619d2f96f253c9ef18c6c026e961a82c0da7798f098943325719121bcbb17  checkpoints/t14b_b64/milestones/step002250_noema.pt
+f0f9e7214e108f0cf63a4ea9039113b2438ac7911f1fc5a640fad8f79d2ec2b4  checkpoints/t14b_b64/opt_rank000.pt
+4f6925320fb258359e01eeec107699e3d6742c1141ccc6c1ac35400401721239  checkpoints/t14b_b64/opt_rank001.pt
+ef94e9c70f6cd0227f4a9f9437bc242846ab904760cbabbe02b4e2d0a4a1f024  checkpoints/t14b_b64/opt_rank002.pt
+d6831bfd1c73f45de07213cba5cb8db89cc3ed5b5e6ac11a7cc8244bc0d16c83  checkpoints/t14b_b64/opt_rank003.pt
+7212cb3da6624fc752c3861ff05c53b8d9aad9f515e872e126a49b98067ae80d  checkpoints/t14b_b64/opt_rank004.pt
+13e4ae9ff3479dd8f6937a19a2ce7935fe28fb059661f11541aff520180d2495  checkpoints/t14b_b64/opt_rank005.pt
+936f712c9d14ca762d93f3871c29a9360959cf5fff38ac7f609fbde8f93a5669  checkpoints/t14b_b64/opt_rank006.pt
+0e352a437b3881fd5de834d48008e573d906d1279476635bdb978a8e3e45e992  checkpoints/t14b_b64/opt_rank007.pt
+79a2a8edfcf34999b3928382e3b891cad0dbcbcf999c20433331cefea76ac631  data/prompts.pt
+4a0b0067309a143cee3b853b047041412b6e7556d5f668c97434a76cdd6f4b5e  out/world_p0.pt
+26de659980d9441766fd167aa1966e97e529579bd2552d1ac8721a55987f45f8  out/world_p44.pt
+b01ee8e99fccbe899d4363a886ba6e0c224d73d5461bb3648f313ca541a2a08d  out/world_p60.pt
+a5a301b04a8feb3a135f6e655da2d4459a0af76898419d3fa16927395cbe35b2  out/world_p82.pt
+cd2c0e30f607c2630b60393d7a56f47ab1697603429adf37f833bdb8e39eef17  samples/w0_bottom_zoom.png
+01682d0b64714737b5c7b3c94331b9cf0b5855d31ff1f4057725cb072fb746d5  samples/w44_diff_0_to_80.png
+77efa06356dfc3b9ce9d244ce1a552c41c40286304b35ce59df4afe34e71f2b2  samples/w60_diff_0_to_80.png
+24fcb4fbe55b868f82c0143fcc5ae4b9aafe20a9bb369cb77d7543999ac69e01  samples/w60_edge_over_time.png
+b1d79ef272d3bb52bc4e49809d1d16a00826ae522b33d2374ec000ad3d748730  samples/w60_edges_zoom.png
+6ef7c38e2dd0c8433c5580b6bb56a9b6cb93ea2635d6bdc649a7ab57a9910c63  samples/w60_filmstrip.png
+67db1cdd80db99b0a583d2c874bfe1873476ba3e5876ea97a56e19b330029c4c  samples/world_p0_woman_16fps.mp4
+60eeead771c8b178eab3419d6c17fb9ca9a261cfebd06089f2dab9eb91ab8a49  samples/world_p44_owl_16fps.mp4
+f02cc3ca0ccbd31cb2a07e152365f030ac76de106bb4a4d866c293300b5199f7  samples/world_p60_waterfall_16fps.mp4
+9d5129dccd0cee17989d0039c3ce96be6370855a6056e727368e35cc8e1e4c61  samples/world_p82_court_16fps.mp4

data/prompts.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79a2a8edfcf34999b3928382e3b891cad0dbcbcf999c20433331cefea76ac631
+size 406857637

out/world_p0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a0b0067309a143cee3b853b047041412b6e7556d5f668c97434a76cdd6f4b5e
+size 62179245

out/world_p44.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26de659980d9441766fd167aa1966e97e529579bd2552d1ac8721a55987f45f8
+size 62179253

out/world_p60.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b01ee8e99fccbe899d4363a886ba6e0c224d73d5461bb3648f313ca541a2a08d
+size 62179253

out/world_p82.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5a301b04a8feb3a135f6e655da2d4459a0af76898419d3fa16927395cbe35b2
+size 62179253

samples/world_p0_woman_16fps.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67db1cdd80db99b0a583d2c874bfe1873476ba3e5876ea97a56e19b330029c4c
+size 2520053

samples/world_p44_owl_16fps.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60eeead771c8b178eab3419d6c17fb9ca9a261cfebd06089f2dab9eb91ab8a49
+size 2890418

samples/world_p60_waterfall_16fps.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f02cc3ca0ccbd31cb2a07e152365f030ac76de106bb4a4d866c293300b5199f7
+size 4348958

samples/world_p82_court_16fps.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d5129dccd0cee17989d0039c3ce96be6370855a6056e727368e35cc8e1e4c61
+size 4793435