multimodalart HF Staff Claude Opus 4.8 (1M context) commited on
Commit
7cca9ce
·
1 Parent(s): 3cc6f42

gradio.Server demo: real LTX-2 t2v backend + DreamVerse UI (ZeroGPU-shaped)

Browse files

app.py: gradio.Server serving index.html at "/" and @app .api("/generate")
running the real FastVideo LTX-2 pipeline behind @spaces.GPU (queued,
single-GPU safe). Validated end-to-end on the RTX 6000: model load ~26s
cached, generation ~9s for 512x768x65f with audio; verified via gradio_client.

Written in the ZeroGPU idiom (import spaces first, GPU-only work in
@spaces.GPU, generator cached, lazy load for spawn-safety). ZEROGPU_NOTES.md
documents the port path (in-process pipeline + aokit weight-less DiT blocks)
and the dev-mode deploy (restart, not rebuild; ephemeral storage).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files changed (3) hide show
  1. ZEROGPU_NOTES.md +80 -0
  2. app.py +145 -5
  3. requirements.txt +41 -0
ZEROGPU_NOTES.md ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # DreamVerse LTX-2 · gradio.Server demo — notes
2
+
3
+ A custom **DreamVerse**-styled frontend (vanilla HTML/CSS/JS in `index.html`)
4
+ served by **`gradio.Server`**, with a real **FastVideo LTX-2** text-to-video
5
+ backend behind `@app.api("/generate")` and `@spaces.GPU`.
6
+
7
+ ## Architecture (per the gradio.Server blog)
8
+
9
+ ```
10
+ index.html ──Gradio JS client──▶ @app.api("/generate") ──▶ @spaces.GPU
11
+ (DreamVerse UI) (queued, single-GPU safe) real LTX-2
12
+ ▲ │
13
+ └────────────── transparent mp4 / video url ◀─────────────┘
14
+ @app.get("/") serves index.html ; @app.get("/healthz") for readiness
15
+ ```
16
+
17
+ `@app.api()` (not a plain `@app.post()`) puts generation through Gradio's
18
+ queue, so concurrent users are serialized onto the one GPU instead of
19
+ colliding, and the endpoint is also callable via `gradio_client`
20
+ (`api_name="/generate"`).
21
+
22
+ ## Validated (on the RTX PRO 6000 Blackwell dev Space)
23
+
24
+ - `gradio.Server` boots and serves the DreamVerse UI at `/` instantly.
25
+ - `/generate` ran the **real** `FastVideo/LTX2-Distilled-Diffusers` pipeline:
26
+ model load **26 s** (cached) / **77 s** cold, generation **~9 s** for a
27
+ 512×768, 65-frame clip with synced audio (h264 + aac mp4). First call ~40 s
28
+ total, subsequent calls ~9 s (model stays warm).
29
+ - Verified end-to-end through `gradio_client.predict("/generate", ...)`.
30
+ - Attention forced to **TORCH_SDPA** (sm_120 can't run FA4 — see the aokit
31
+ digest), `num_gpus=1`.
32
+
33
+ ## ZeroGPU porting
34
+
35
+ The app is already in the ZeroGPU idiom: `import spaces` first, GPU work only
36
+ inside `@spaces.GPU`, generator created once and cached. On a dedicated GPU
37
+ `@spaces.GPU` is a passthrough so this just works.
38
+
39
+ The one real blocker for actual ZeroGPU hardware: FastVideo's `VideoGenerator`
40
+ spawns a **persistent** GPU worker (`MultiprocExecutor`, start method
41
+ `spawn`). ZeroGPU instead gives you the GPU only inside a forked `@spaces.GPU`
42
+ call and snapshots **module-scope** CUDA state across forks — a persistent
43
+ external worker process doesn't fit that model, and a `spawn` worker re-imports
44
+ the app module (which is why the generator load is lazy, never at module
45
+ scope).
46
+
47
+ The ZeroGPU-native path:
48
+ 1. Run the LTX-2 pipeline **in-process** (no MultiprocExecutor), loading
49
+ components at module scope so `spaces` can snapshot them.
50
+ 2. Use the **aokit-compiled, weight-less** DiT blocks from
51
+ `package/{cu128,cu130}/` for the transformer — aokit is built exactly for
52
+ ZeroGPU (lazy `ContextVar` execution, fork cleanup), so the heavy compiled
53
+ graph loads correctly inside each `@spaces.GPU` fork.
54
+
55
+ That reuses the artifacts from the earlier task and removes the persistent-worker
56
+ assumption.
57
+
58
+ ## Deploy (dev mode, ephemeral storage — be careful)
59
+
60
+ The Space supervisor runs, on rebuild only:
61
+ `git reset --hard $COMMIT_SHA && git clean -fdx && python app.py`.
62
+ So **`git clean -fdx` wipes any uncommitted file** and a full rebuild wipes the
63
+ ephemeral env (the source-installed FastVideo, the CUDA-matched torch, and the
64
+ ~38 GB cached weights). Therefore:
65
+
66
+ - Everything is committed to git (survives the reset).
67
+ - Deploy by **restarting the app process** in the warm env — not a clean
68
+ rebuild. On restart `python app.py` picks up the new app; the first
69
+ `/generate` lazy-loads the (already cached) model in ~30 s.
70
+ - `requirements.txt` is best-effort for a cold rebuild; FastVideo (main) isn't
71
+ on PyPI and pins `gradio==5.32.0` (conflicts with the gradio≥6 that
72
+ `gradio.Server` needs), so the warm env installs it via
73
+ `pip install -e ./FastVideo --no-deps`.
74
+
75
+ ## Render settings (env-overridable in `app.py`)
76
+
77
+ `DREAMVERSE_HEIGHT=512 DREAMVERSE_WIDTH=768 DREAMVERSE_FRAMES=65`
78
+ `DREAMVERSE_STEPS=5 DREAMVERSE_FPS=24 DREAMVERSE_GPU_DURATION=300`
79
+ (`num_frames-1` must be divisible by 8). Bump toward DreamVerse's
80
+ 1920×1088 / 121-frame profile for higher quality at higher latency.
app.py CHANGED
@@ -1,7 +1,147 @@
1
- import gradio as gr
2
 
3
- def greet(name):
4
- return "Hello " + name + "!"
 
 
 
 
5
 
6
- demo = gr.Interface(fn=greet, inputs="text", outputs="text")
7
- demo.launch()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """DreamVerse-style LTX-2 text-to-video demo on `gradio.Server`.
2
 
3
+ Architecture (per the gradio.Server blog):
4
+ * `@app.get("/")` serves the custom DreamVerse UI (index.html)
5
+ * `@app.api("/generate")` runs REAL FastVideo LTX-2 text-to-video and
6
+ returns an mp4. It goes through Gradio's queue, so concurrent users are
7
+ serialized onto the single GPU instead of colliding.
8
+ * `@spaces.GPU` ZeroGPU allocation around the GPU work.
9
 
10
+ The heavy FastVideo `VideoGenerator` (the same engine DreamVerse runs) is
11
+ initialized lazily on the first request so the Server boots instantly and
12
+ serves the UI immediately — the model load (and a slow first generation,
13
+ DreamVerse-style warmup) happens on first `/generate`.
14
+
15
+ Notes for this box:
16
+ * RTX PRO 6000 Blackwell (sm_120): FA4/CuTeDSL kernels aren't built for it,
17
+ so we force `FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA`.
18
+ * The compiled aokit LTX-2 block lives under `package/{cu128,cu130}/` — it
19
+ is the ZeroGPU-ready (lazy, weight-less) piece; this app runs the full
20
+ FastVideo pipeline via its own persistent GPU worker.
21
+ """
22
+
23
+ import os
24
+ import threading
25
+ import time
26
+ import uuid
27
+ import tempfile
28
+
29
+ # Must be set before any fastvideo / CUDA import.
30
+ os.environ.setdefault("FASTVIDEO_ATTENTION_BACKEND", "TORCH_SDPA")
31
+ os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
32
+
33
+ import spaces
34
+ from gradio import Server
35
+ from gradio.data_classes import FileData
36
+ from fastapi.responses import HTMLResponse
37
+
38
+ MODEL_ID = os.getenv("LTX2_MODEL_PATH", "FastVideo/LTX2-Distilled-Diffusers")
39
+ HERE = os.path.dirname(os.path.abspath(__file__))
40
+ OUT_DIR = os.path.join(tempfile.gettempdir(), "dreamverse_outputs")
41
+ os.makedirs(OUT_DIR, exist_ok=True)
42
+
43
+ # Demo render settings — modest for single-GPU latency. Override via env.
44
+ # num_frames must satisfy (num_frames - 1) % 8 == 0.
45
+ GEN_HEIGHT = int(os.getenv("DREAMVERSE_HEIGHT", "512"))
46
+ GEN_WIDTH = int(os.getenv("DREAMVERSE_WIDTH", "768"))
47
+ GEN_FRAMES = int(os.getenv("DREAMVERSE_FRAMES", "65"))
48
+ GEN_STEPS = int(os.getenv("DREAMVERSE_STEPS", "5"))
49
+ GEN_FPS = int(os.getenv("DREAMVERSE_FPS", "24"))
50
+ GPU_DURATION = int(os.getenv("DREAMVERSE_GPU_DURATION", "300"))
51
+
52
+ app = Server()
53
+
54
+ # --- ZeroGPU porting notes ---------------------------------------------
55
+ # This app is written in the ZeroGPU idiom so the port is mechanical:
56
+ # * `import spaces` happens before any torch/CUDA import (top of file);
57
+ # * the only GPU-touching code lives inside the `@spaces.GPU` function;
58
+ # * the generator is created once and reused (cached in a module global).
59
+ # On this dedicated RTX 6000 dev Space `@spaces.GPU` is a passthrough, so the
60
+ # cached generator persists in-process and the first call pays the load cost.
61
+ #
62
+ # The ONE thing to change for real ZeroGPU: FastVideo's `VideoGenerator`
63
+ # spawns a *persistent* GPU worker (MultiprocExecutor, start method = spawn).
64
+ # ZeroGPU instead hands you the GPU only inside a forked `@spaces.GPU` call
65
+ # and snapshots module-scope CUDA state across forks — a persistent external
66
+ # worker doesn't fit that. The ZeroGPU-native path is an in-process LTX-2
67
+ # pipeline whose DiT blocks are the aokit-compiled, weight-less artifacts in
68
+ # `package/{cu128,cu130}/` (aokit is built for exactly this: lazy, fork-safe
69
+ # loading). See ZEROGPU_NOTES.md. Until then this runs great on a dedicated
70
+ # GPU and keeps the ZeroGPU-shaped surface.
71
+ _generator = None
72
+ _gen_lock = threading.Lock()
73
+
74
+
75
+ def _get_generator():
76
+ """Lazily build the FastVideo LTX-2 generator (single GPU, in-process worker).
77
+
78
+ Lazy (not module-scope) on purpose: FastVideo's worker uses `spawn`, which
79
+ re-imports this module in the child; a module-scope load would re-trigger
80
+ here. Building inside the request keeps import side-effect-free.
81
+ """
82
+ global _generator
83
+ if _generator is not None:
84
+ return _generator
85
+ with _gen_lock:
86
+ if _generator is not None:
87
+ return _generator
88
+ from fastvideo import VideoGenerator
89
+ print(f"[dreamverse] loading {MODEL_ID} (first request — this is slow) ...", flush=True)
90
+ t0 = time.perf_counter()
91
+ gen = VideoGenerator.from_pretrained(MODEL_ID, num_gpus=1)
92
+ print(f"[dreamverse] generator ready in {time.perf_counter()-t0:.0f}s", flush=True)
93
+ _generator = gen
94
+ return _generator
95
+
96
+
97
+ @spaces.GPU(duration=GPU_DURATION)
98
+ def generate_video(prompt: str) -> str:
99
+ """Run real LTX-2 text-to-video; return the path to an mp4."""
100
+ gen = _get_generator()
101
+ out_path = os.path.join(OUT_DIR, f"{uuid.uuid4().hex}.mp4")
102
+ t0 = time.perf_counter()
103
+ gen.generate_video(
104
+ prompt=prompt,
105
+ output_path=out_path,
106
+ save_video=True,
107
+ height=GEN_HEIGHT,
108
+ width=GEN_WIDTH,
109
+ num_frames=GEN_FRAMES,
110
+ num_inference_steps=GEN_STEPS,
111
+ fps=GEN_FPS,
112
+ guidance_scale=1.0,
113
+ seed=int(time.time()) % 100000,
114
+ )
115
+ print(f"[dreamverse] generated in {time.perf_counter()-t0:.1f}s -> {out_path}", flush=True)
116
+ if not os.path.exists(out_path):
117
+ raise RuntimeError("Generation did not produce an output file")
118
+ return out_path
119
+
120
+
121
+ @app.api(name="generate")
122
+ def generate(prompt: str) -> FileData:
123
+ """Text-to-video endpoint. Goes through Gradio's queue (single-GPU safe).
124
+
125
+ Callable from the browser via the Gradio JS client and from Python via
126
+ `gradio_client` (api_name="/generate").
127
+ """
128
+ prompt = (prompt or "").strip()
129
+ if not prompt:
130
+ raise ValueError("Prompt must be non-empty")
131
+ path = generate_video(prompt)
132
+ return FileData(path=path)
133
+
134
+
135
+ @app.get("/", response_class=HTMLResponse)
136
+ async def homepage():
137
+ with open(os.path.join(HERE, "index.html"), "r", encoding="utf-8") as f:
138
+ return f.read()
139
+
140
+
141
+ @app.get("/healthz")
142
+ async def healthz():
143
+ return {"ok": True, "model": MODEL_ID, "loaded": _generator is not None}
144
+
145
+
146
+ if __name__ == "__main__":
147
+ app.launch(server_name="0.0.0.0", server_port=int(os.getenv("PORT", "7860")), show_error=True)
requirements.txt ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # DreamVerse LTX-2 demo on gradio.Server
2
+ #
3
+ # NOTE ON DEPLOY: this Space runs in dev mode with EPHEMERAL storage. The live
4
+ # environment is pre-warmed (FastVideo installed from source, torch built for
5
+ # the box's CUDA, the LTX-2 weights cached) and the demo is deployed by
6
+ # RESTARTING the app process — not by a clean rebuild. A clean rebuild from
7
+ # this file is best-effort only, because FastVideo (main) is not on PyPI and
8
+ # pins gradio==5.32.0, which conflicts with the gradio>=6 that gradio.Server
9
+ # needs. See ZEROGPU_NOTES.md.
10
+
11
+ # Web / serving (gradio.Server lives in gradio>=6)
12
+ gradio>=6.14.0
13
+ spaces
14
+
15
+ # Torch for the box's CUDA (cu128 shown; use cu130 to match a CUDA-13 runtime).
16
+ --extra-index-url https://download.pytorch.org/whl/cu128
17
+ torch==2.11.0
18
+ torchvision
19
+
20
+ # FastVideo (LTX-2 lives on main, not in the 0.1.7 PyPI release). Installed in
21
+ # the warm env via: pip install -e ./FastVideo --no-deps
22
+ # (kept here as a reference; --no-deps avoids the gradio==5.32 / torch pins)
23
+ # fastvideo @ git+https://github.com/hao-ai-lab/FastVideo.git
24
+
25
+ # FastVideo runtime deps actually needed to load + run LTX-2 inference
26
+ transformers==4.57.3
27
+ diffusers>=0.33.1
28
+ safetensors
29
+ sentencepiece
30
+ tokenizers>=0.20.1
31
+ accelerate==1.0.1
32
+ einops
33
+ omegaconf
34
+ loguru
35
+ cloudpickle
36
+ imageio
37
+ imageio-ffmpeg
38
+ scipy
39
+ ftfy
40
+ huggingface_hub
41
+ hf_transfer