Hugging Face's logo

desiree
/
spi

Model card Files
xet
 Community
spi / docs /research_report.md
desiree's picture
desiree
Phase 1: research report (3 candidates, primary sources)
9d69194 verified
preview code
|
Raw History Blame Contribute Delete
4.55 kB

SPI Research Report — Video Backends (Phase 2 selection)

Date: 2026-09-23 · Mode: ultra_low (narrow research: 3 candidates + VBench) · Sources: official HF model cards, diffusers docs, vendor repos.

Selection policy (from project brief): openness > controllability > architecture access > fine-tunability > ecosystem > quality > compute efficiency.

Comparison

Wan 2.1/2.2 (Wan-AI) LTX-2 / LTX-2.5 (Lightricks) Fallback: CogVideoX (zai-org)
License Apache-2.0 (all Wan2.1/2.2 cards) LTX Open Weights License (license: other, gated=auto): free for entities < $10M annual revenue; paid license required for Commercial Entities (≥$10M) on revenue-generating use or commercial fine-tuning 2b: Apache-2.0; 5b: custom (restrictive)
Params Wan2.2 A14B: 27B MoE (two 14B experts); TI2V-5B dense LTX-2/2.5: 19B (13B video DiT + audio stream); legacy LTX-Video 2B/13B 2B / 5.57B
Min VRAM (low-res) Wan2.1 T2V-1.3B: 8.2 GB (official); TI2V-5B: 24 GB (RTX 4090, 720p@24fps); A14B: ~80 GB native, less with offload/FP8 LTX-Video 2B: ~10 GB (offloading/fp8); LTX-2 19B: bf16 48 GB+, fp8 ~24 GB 2b: ~5 GB; 5b: ~10–12 GB (offload)
Diffusers pipelines WanPipeline, WanImageToVideoPipeline, WanVACEPipeline, WanAnimatePipeline (stable ≥0.40.0) LTXPipeline, LTXConditionPipeline, LTXLatentUpsamplePipeline; LTX2Pipeline in v0.40.0 (diffusers.pipelines.ltx2) CogVideoXPipeline, CogVideoXImageToVideoPipeline (mature)
Conditioning Broadest: T2V, I2V, FLF2V (first/last frame), VACE (pose/depth/mask/reference/inpaint/outpaint/video-extend in one pipeline), Animate, S2V; alibaba-pai Fun-Control (pose/depth/trajectory/camera) T2V, I2V, video-to-video/extend via LTXConditionPipeline (decode_timestep=0.05); official IC-LoRAs (Ingredients, Clean-Plate, Union-Control, Slow-Motion, upscalers); no ControlNet-style adapter T2V, I2V; ControlNet only via community CogVideoX-Fun; no official video-extend
Fine-tuning LoRA + full: DiffSynth-Studio (official), community diffusers LoRA LoRA-first: official distilled/IC LoRAs; LTX2LoraLoaderMixin LoRA via diffusers examples; full FT hard
Long video Wan-VAE handles arbitrary-length 1080p; I2V chaining; 81-frame clips LTX-Video 0.9.8+ long video; LTX-2 multiscale 2-stage; ~18× faster than Wan2.2 on H100 (1.22 vs 22.3 s/step, 121f 720p) 49-frame fixed; weakest
Ecosystem Largest (ComfyUI, DiffSynth, Wan2GP, VBench-topping) Fast-moving (ComfyUI/Kijai, GGUF); gated HF repos add friction Stale (last update Nov 2024)

Recommendation

Phase 2 primary: Wan 2.2 (+ Wan 2.1 VACE for control jobs). Reasons:

  1. Only candidate that is genuinely Apache-2.0 with no revenue clause — matches openness priority.
  2. Deepest conditioning surface: one WanVACEPipeline covers pose/depth/mask/reference/extend, mapping 1:1 onto SPI's common conditioning API.
  3. Official diffusers pipelines for every task family; mock interface translates directly.
  4. Cheap lanes for tests (1.3B/5B) and quality lane (A14B MoE).

Secondary (speed): LTX-2/2.5 — ~18× faster per step, native audio+video stream. Constraints recorded in registry: gated HF access (token with accepted license needed) and the $10M-revenue commercial cap.

Fallback only: CogVideoX-2b (Apache-2.0, stale). HunyuanVideo rejected: custom license bans EU/UK/KR use, 13B needs 45+ GB without offload, no first-class diffusers pipeline.

Known gaps / caveats

  • Wan VACE is Wan2.1-based; no official Wan2.2+VACE in diffusers yet → route control jobs to Wan2.1-VACE, quality jobs to Wan2.2.
  • LTX-2.x repos are gated → renderer adapter must verify token/license at load time.
  • Wan LoRA training official path is DiffSynth-Studio, not diffusers Trainer.
  • Pipeline names drift fast; pin diffusers>=0.40.0 and re-verify model_index.json tags at install time.

VBench / VBench-2.0 (evaluation reference)

  • VBench: pip vbench (0.1.5, Apache-2.0), 16 dimensions. Programmatic: from vbench import VBench; VBench(device, info_json, out_dir).evaluate(videos_path=..., name=..., dimension_list=[...]). Needs detectron2 from git for some dims.
  • VBench-2.0: pip vbench2 (Apache-2.0, Apr 2025), 18 dimensions (adds physics, human interaction, commonsense). Heavy install (torch 2.5.1 pins, flash-attn, mmcv 2.2.0). Plan: VBench v1 as cheap CI regression gate; vbench2 reserved for release-level evals. SPI uses them as adapters, not as the whole DirectorBench.