Upload folder using huggingface_hub
e488ce7 verified about 1 month ago
23 kB
# Low VRAM, low RAM: the de-rope on a 16 GB card

Status 2026-08-18: alpha. Measured on one machine fenced down to a 16 GB
card / 32 GB box (how, and how the fence was read, at the bottom); not yet
run on a physically different 16 GB card, which is the next thing.

**Get started** (16 GB card, 32 GB machine; measured, alpha):

1. Start ComfyUI **without** `--gpu-only` and **with**
   `--fast-disk --fp16-intermediates --disable-pinned-memory` (and
   `--cache-none` if RAM is tight; it re-runs every node on each queue).
   What each does, measured on the 702-frame de-rope (RSS peak, same VRAM and
   speed in every case): fp32 IMAGEs and a pinned DiT = 41.9 GB; `--fp16-intermediates`
   halves every held IMAGE = 31.8; `--disable-pinned-memory` drops the DiT's
   ~10 GB pinned staging copy = 21.6; `--cache-none` frees node outputs after
   their consumers = 24.2 alone, **14.6 GB with all four**. `--fast-disk` reads
   the weights from the NVMe page cache instead of RAM copies. Speed did not
   move (369-374 s/step throughout).
2. Load `examples/motion_pipeline_lowvram.json` (API twin alongside), point
   the LoadImage at your reference, set the prompt and canvas.
3. Queue. The 124-frame reference clip takes about 7 minutes end to end on
   the fenced 15 GiB card (RSS 25 GB).

If `H3 Streamed Blocks` is not in the node search (category `MAINodes/VRAM
Lab`, display name "H3 Streamed Blocks (exact low-VRAM, alpha)"), look at
the ComfyUI console for one line at startup: `[MAINodes] vram_lab not
loaded: ...`. The module needs a recent ComfyUI (developed on 0.33.0; it
imports `comfy.ldm.minimax.model` and comfy_kitchen's native rope op via
`comfy.quant_ops.ck`) and PyTorch flash attention; the pack loads the rest
of its nodes without it and prints why. Paste that line in an issue.

**What the graph does to make a small card happy:**

- the DiT at 4-bit weights / 8-bit activations (`minimax_h3_ref2va_pruned_w4a8_mixed`, 11 GB), the text encoder in NVFP4, the video VAE as int8_convrot (2.2 GiB less than fp16, decode 1.5x faster, 60 dB PSNR against it);
- `H3 Streamed Blocks` runs every DiT block in 16k-token chunks so the whole-sequence QKV and SwiGLU tensors (8.6 and 15.4 GiB at 217k tokens) never exist, and streams the output head; same math as stock (bit-equal on int8 / W4A8 with the exact K/V store);
- **`kv_store` = kvi8r**: the K/V of the whole sequence held as rotated int8, forward peak +9.5 GiB at 217k tokens instead of +11.9 (with `trim_forward`, on by default, another 2.3 GiB comes off); a sibling take of the exact result that passed the operator's eyes on the de-rope ("perfect for both"); flip it to `bf16 (exact)` for the bit-equal path;
- `H3 Evict Text Encoder` after each prompt encode, `H3 Free Cache` before the pass-2 decode: the 15 GB TE and the allocator pool leave the card before the VAE needs it;
- the turbo LoRA with a 12-step base pass and a 6-step / inject 0.7 de-rope pass, so the long pass is a handful of forwards.

Everything below is the why and the numbers.

## The workflow that ships: `examples/motion_pipeline_lowvram.json`

On a Blackwell card there is a free speed lever on top of all of this: swap
the example's W4A8 UNET for the NVFP4 build
([MATLOWAI/minimax-h3-nvfp4](https://huggingface.co/MATLOWAI/minimax-h3-nvfp4)).
Same file size, measured 0.84 to 0.91x wall on 25-step graphs and 0.88x with
the turbo LoRA stacked, stock ComfyUI. A same-seed render is a sibling take;
judge it on your own content. Non-Blackwell cards should not use it.

Two things the example already includes that are easy to miss: the AUDIO INIT
(`H3 V2V Init` takes the source performance's audio latent at strength 0.5, so
pass 2 follows the original delivery) and the smear-aware audio recover. For
clips whose dilated pass does not fit your card in one piece, the same stack is
wired into the windowed graph:
[`examples/motion_pipeline_rolling_window_lowvram.json`](examples/motion_pipeline_rolling_window_lowvram.json)
(API twin alongside; plan report first, then queue per window; inject 0.48). Each
window pins its first and last frame to the baseline via `MiniMax H3 Add Guide`,
which holds the regenerated windows on the baseline's motion and look so the
splice seams carry no state jumps; the pin frames come straight from
`H3 Window Plan`'s first/last frame outputs.

One graph, the whole motion pipeline for a small card, with the rotated int8
K/V store on. What is in it and why, in the order it runs (2026-08-18):

| node | setting | what it buys |
|---|---|---|
| `UNETLoader` | `minimax_h3_ref2va_pruned_w4a8_mixed` (11 GB) | the DiT at 4-bit weights / 8-bit activations, exact under chunking (int32 accumulate) |
| `CLIPLoader` | `qwen3vl_32b_minimax_h3_nvfp4_awq` | the text encoder in NVFP4 (~15 GB file) |
| `VAELoader` (video) | `minimax_h3_video_vae_int8_convrot` | 2.2 GiB less on the card than fp16, decode 1.5x faster, 60 dB PSNR against the fp16 decoder on the same latent |
| `VAELoader` (audio) | `minimax_h3_audio_vae_fp32` | unchanged, 0.6 GB |
| `LoraLoaderModelOnly` | lightx2v turbo 4-step v1.0 768p, strength 1.0 | pass 1 at 12 steps (`linear_quadratic`), pass 2 on a 6-step `beta` schedule with `inject 0.7` (the "faithful detail" preset), so the long de-rope pass costs a handful of forwards |
| **`H3 Streamed Blocks`** | q/kv/mlp chunks 16384, `min_tokens` 32768, `final_layer_chunk` 16384, `final_layer_gemm` exact, **`kv_store` = kvi8r** | every DiT block in token chunks (never the whole-sequence QKV or SwiGLU), the output head streamed, and the K/V of the whole sequence held as rotated int8: forward peak +9.5 GiB at 217k tokens instead of +11.9 (exact store) or +21 (stock). Bit-equal to stock with `kv_store` = bf16; with kvi8r a sibling take that the operator judged "perfect for both" on the de-rope side by side |
| `H3 Evict Text Encoder` (x2) | after each prompt encode | the 15 GB TE leaves the card as soon as its conditioning exists (1 to 2.5 GiB of peak on a 16 GB card, ~3 s per new prompt to bring back) |
| `H3 Jerk Oracle`, `H3 Time Smear`, `H3 Audio Smear`, `H3 V2V Init` | `d_max 4`, dilation 4, audio 0.5 | the de-rope itself (see README) |
| `H3 Free Cache` | between the pass-2 sampler and its decodes | returns the allocator pool before the VAE runs (17 GiB on the long pass under `--gpu-only`; the small-card win is that decode never competes with a stale pool) |
| `H3 Exact Recover`, `H3 Audio Recover` | | back to real time, frame-exact |
Measured as it ships (kvi8r + int8 VAE) at the 15.0 GiB fence / 48 GB
cgroup: the 124-frame reference clip end to end in **6:56**, VRAM 15.4 (the
fence, filled during the TE load), torch reserved max 6.2 GiB, RSS 25.2 GB,
zero allocator trims; pass 1 14.2 s/step, the de-rope pass 56 s/step. (With
the exact store and the fp16 VAE at the earlier 15.4 fence: 6:14, RSS 24.8.)
On the 702-frame de-rope, a bigger graph than this example, the exact store
no longer fits at 15.0 GiB while kvi8r (374 s/step) and kvi8s (219) render.
The KJNodes sage-attention patches and FFN chunker from the parent graph
are not in it: sage attention is an approximation and the exactness gates
here were run on PyTorch's flash attention.
3. Set your prompt, reference image and canvas as usual.
## What to expect
**VRAM.** The activation peak inside a forward is card-independent:
`0.3 GiB + ~53 KB per packed token` (K/V buffers 28.7 KB/token, the residual
10.8, chunk transients and two tensors ComfyUI's forward keeps alive).
At 702 frames / 217k tokens that is +11.9 GiB. Dynamic VRAM fills whatever is
left with resident weights, so the process peak reads close to the card size
on every card; what decides whether a length fits is `weights it must keep +
activations + ~1 GB context`. Measured: 702 frames fits a 16 GB card, 906 does
not (275k tokens: +14.9 GiB of activations plus the minimum resident weights).
Frames here are 1376x768; other canvases scale by tokens
(`latent_frames x (W/32) x (H/32)`).
**RAM. This is the tight budget, not VRAM.** With `--fast-disk`, process RSS
was 27.4 GB on the 16 GB card and 25.8 GB on the 24 GB card for a single
702-frame pass, 36 to 40 GB for the full de-rope pipeline (it holds the pass-1
decodes, the smear IMAGE, the heat map: a 702-frame fp32 IMAGE is 8.9 GB), and
59 GB without `--fast-disk`. Add your own headroom: an idle Ubuntu desktop is
~2 GB, an idle Windows 11 desktop 4 to 8 GB, ComfyUI itself 3 to 5 GB before
any model. So, measured: a single 702-frame pass and the 124-frame example
graph fit a 16 GB card + 32 GB RAM (lean Linux; already paging on Windows);
**the full 702-frame de-rope pipeline does not fit 32 GB** (it was OOM-killed at
the smear under a 32 GB fence, 2026-08-18 19:00) and needs ~48 GB today, 64 GB
to be comfortable. Holding intermediates as fp16/uint8 by consumer type is
the next change and should bring the full pipeline back under 32 GB. Under our 64 GB fence
the kernel reclaimed page cache before killing anything; a Windows box would
pagefile instead, which is slower but alive.

**Time.** 316 to 318 s per pass-2 step at 217k tokens on the fenced 16, 24 and
32 GB cards versus 311 resident on 96 GB. On the 90-frame clip the 16 GB card
did 8.2 s/step. The text encoder costs 3 to 4.5 s per new prompt from a warm
page cache (a cold read of the 15 GB file is longer), and 1 to 2.5 GiB of
VRAM peak on a 16 GB card, which the evict node removes.

**The text encoder on tiny cards**, measured on the fenced 16 GB card, 90 frames:

| arm | wall | encode | VRAM peak | RSS |
|---|---|---|---|---|
| stock CLIPLoader | 221 s | 3.0 s | 15.7 | 14.3 |
| second prompt in the same process | 218 s | 1.7 s | 16.6 | 15.5 |
| `H3 Evict Text Encoder` after the encode | 221 s | 3.1 s | 13.9 | 14.7 |
| `CLIPLoader device=cpu` | 272 s | 55 s | 14.0 | 26.9 |
| conditioning bank hit (encoder never runs) | 218 s | none | 14.7 | 12.4 |

Nothing thrashed. Evict is the default in the example graph because a normal
run types a new prompt; `H3 Conditioning Bank` is for repeated prompts (seed
hunts, windows, extension chains) and needs the sampler's latent to come from
somewhere other than the encode node (`EmptyMiniMaxH3LatentAV` for T2VA, the
latent bank for a banked pass 1), or the encode node runs anyway for its
latent output.

## The numbers

The de-rope pass at `d_max 4` on an 8 to 12 second clip is ~200k packed tokens.
The stock H3 block builds its fused QKV and SwiGLU tensors for the whole
sequence at once, 8.6 and 15.4 GiB at that length, and that is what OOMs a
24 GB card, and at 1376x768 a 96 GB one. `H3 Streamed Blocks` runs every DiT
block in token chunks with the same math. Together with two things ComfyUI
already ships (dynamic VRAM, `--fast-disk`) it changes the small-card story
from "OOM" to "same speed":

| card | machine | 294 f -> 702 f de-rope at 1376x768 (~217k tokens) | s/step (pass 2) |
|---|---|---|---|
| 96 GB, everything resident (`--gpu-only`) | 91 GB | renders | 311 |
| 32 GB (fenced), normal mode, `--fast-disk` | 64 GB (cgroup) | renders, 25 GiB VRAM peak | 318 |
| 24 GB (fenced) | 32 GB (cgroup) | renders, 23.5 GiB VRAM peak, 25.8 GB RSS | 318 |
| **16 GB (fenced)** | **32 GB (cgroup)** | **renders, 15.5 GiB VRAM peak, 27.4 GB RSS** | **316** |
| **15.0 GiB fence (a 5070 Ti with its desktop up)** | 48 GB (cgroup) | exact store: **out of memory** at the second forward (+12.1 GiB needed above the resident weights, 1990 allocator trims first); **kvi8r: renders** (+9.5 GiB, 22:33); **kvi8s: renders** (+9.5 GiB, 14:50); **kvi8s + `trim_forward`: renders with zero allocator trims (+7.2 GiB, 14:51)** | exact - / kvi8r 374 / kvi8s 219 |
| 16 GB (fenced) | 32 or 64 GB | 906 f (~275k tokens): out of memory | |
| 24 GB (fenced) | 32 GB | 906 f: renders, RSS 29.8 GB (RAM at the edge) | 515 |
Same seed, the streamed result is bit-equal to the stock result on the int8
and W4A8 checkpoints (video and audio latents compared tensor by tensor). It
is not a quality dial. It is also not slower: at these lengths the step is
attention-bound and the weight traffic hides under it (see "Why no speed
penalty").
## How far you can go (extrapolated from the lines above)
The activation line predicted the 906-frame point on the 24 GB card before it
was measured (14.9 GiB both ways), so the table below is the line, not a
guess; treat it as +-1 frame-block. "Max tokens" assumes dynamic VRAM keeps
the minimum of resident weights (2 to 3 GB measured) and ~1 GB of context.
| card | max packed tokens | frames at 1376x768, single de-rope pass | RAM at that length, single pass / full pipeline |
|---|---|---|---|
| 16 GB | ~230k | ~740 (measured: 702 renders, 906 does not) | ~28 / ~40 GB |
| 24 GB | ~380k | ~1230 (measured: 906 renders at 515 s/step) | ~33 / ~45 GB |
| 32 GB | ~530k | ~1720 | ~39 / ~51 GB |
RAM per frame is ~12 MB (one fp32 IMAGE at 1376x768) on top of a ~19 GB base
for a single pass with `--fast-disk`; the full pipeline holds two to three
more IMAGEs of the clip. Past ~1100 frames the step time (attention grows with
the square of the sequence: 316 s at 702 frames, 515 at 906, ~650 at 1110)
is the practical limit before memory is. Other canvases: convert to tokens
with `latent_frames x (W/32) x (H/32)`, where `latent_frames = (frames-5)/17*5+2`.
**The example graph itself, end to end on the fenced 16 GB card in a 32 GB
machine** (2026-08-18 18:25): rendered in 6:14 wall. Pass 1 (Ref2VA, 12 base
steps, 124 frames) 147 s, oracle + smear + audio smear + VAE encode 30 s,
pass 2 (6 turbo steps) 150 s, decode 21 s, recover and three saves. Process
VRAM peak 16.1 GiB (dynamic VRAM sized itself to the card), RSS peak 24.8 GB,
cgroup peak 30.9 GB (RSS plus the reclaimable page cache of the weight files:
that is how little air a 32 GB box has even at 124 frames). The text encoder
held 13 GB of the card until `H3 Evict Text Encoder` freed it
(`device free 1.3 -> 14.6 GiB`).
## Why no speed penalty
At 217k tokens about 250 of the ~315 seconds per forward are attention, which
grows with the square of the sequence. Streaming the whole 11 to 12.5 GB DiT
once per forward is ~0.5 s over PCIe from RAM and ~2 s from an NVMe page
cache, and dynamic VRAM prefetches the next layers while the current ones
compute. Weight traffic is a fixed cost per forward; attention is quadratic;
the long clip hides the traffic completely, and on the short one prefetch
did the same (8.2 s/step on 16 GB versus ~9 resident on 96 GB). This box has
PCIe 5 x16 and a fast NVMe; a PCIe 4 desktop halves the bandwidth, which is
still nothing against a 300 s step. A machine whose RAM cannot hold the
weight file's page cache and whose SSD is slow could show it on short clips.
## The K/V store options (approximation tier, default off)
At 217k tokens the exact block's forward peak is +11.9 GiB above the resident
weights, and 5.8 GiB of that is the K/V of the whole sequence held in bf16
while the query chunks attend against it. `H3 Streamed Blocks` has a
`kv_store` option with two ways of halving those bytes. Neither is bit-equal
to stock: a same-seed render is a sibling take. The node's default is the
exact store; the low-VRAM example graph ships with kvi8r on (operator's
call); kvi8s has now been rendered and seen ("looks perfect", 219 s/step
against kvi8r's 374) and is the faster option for anyone with the
`sageattention` package installed; the example keeps kvi8r because it needs
no extra package.
| `kv_store` | what it is | forward peak at 217k tokens | s/step (pass 2, 702 f de-rope) | quality so far |
|---|---|---|---|---|
| `bf16 (exact)` | stock math, chunked | +11.9 GiB | 318 | bit-equal |
| `kvi8r: rotated int8 K/V` | K and V int8 with one fp16 scale per (token, head) row, in a fixed Hadamard-rotated basis of the head dim; the query chunk is rotated to match and the output un-rotated once, so a K/V block dequant is one int8->fp16 cast times a scale, no GEMM; attention in fp16 blockwise (`kv_block`, default 16384) with an online-softmax combine | **+9.5 GiB** (second cut, measured live 2026-08-18 evening; the first cut was +12.5) | **374** (first cut 400) | first cut: operator's eyes "almost perfect" on the de-rope side by side, "looks fine" on a 90 f T2VA; second cut: side by side rendered, verdict pending; sensor bank over seeds owed |
| `kvi8s: Sage int8/fp8 K/V, rotated` | the same bytes kept in SageAttention's kernel layout (int8 K with one scale per 64-token block after mean smoothing, fp8 e4m3 V per channel), Q and K Hadamard-rotated first, attended on int8/fp8 tensor cores straight from the store, no dequant; needs the `sageattention` package (2.x, sm120 works) | **+9.5 GiB** (measured live 2026-08-18 evening) | **219** (31% faster than the exact store, 42% faster than kvi8r) | one rung more approximate than kvi8r on a synthetic proxy (rel-rms 3.1e-2 vs 1.3e-2; the shipped `sageattn()` scores 5.5e-2 on the same inputs, so the rotation is worth 1.8x to Sage); operator's eyes on the de-rope side by side: "looks perfect" (one clip, one viewer; sensor bank over seeds owed) |

Why the rotation: the fixed orthonormal Hadamard spreads the outlier channels
of K and V over all 128 dims before rounding, so a per-row int8 scale wastes
less range; scores are invariant (q.k = (qH).(kH)) and P.V comes back through
H^T once on the output. Why fp16 and not bf16 inside kvi8r: in the rotated
basis every value has the same magnitude and bf16's 8-bit mantissa alone costs
a 4.8e-3 rel-rms floor; fp16's 11 bits cost 1.7e-3 (measured on random data).

## What is exact, and what is not

- `H3 Streamed Blocks` on int8 and W4A8 checkpoints: bit-equal to stock. The
  mechanism is that comfy_kitchen quantises activations per row and the int8
  GEMM accumulates in int32, which is associative, so chunking rows cannot
  change the sum. NVFP4/FP8 would need one shared activation scale and are
  untested; bf16 checkpoints are numerically equivalent (fp accumulation
  order), not identical.
- Query chunking is exact only while every chunk keeps PyTorch's flash
  kernel off its split-KV path (measured boundary `heads x ceil(L/64) >=
  0.8 x 2 x SMs`; a 259-token tail once made a different clip). The node sizes
  chunks from the SM count with a 2.6x margin and has a `self_check` that
  compares stock and streamed on the first block's real input.
- The output head has an exact mode (default) and a chunked-GEMM mode that
  differs by ~1e-6 per step in fp32 and produced a different clip after 25
  steps. It is labelled and off. `kv_block` on the exact store is experimental
  and as built does not lower memory; leave it at 0 (it is the block size for
  the kvi8r store).
- `H3 Free Cache` and `H3 Evict Text Encoder` change no math.
- `trim_forward` (default on) runs a copy of the stock model forward that
  releases the patch-embed buffers once the packed sequence is assembled
  (stock holds them for the whole forward: 2.3 GiB at 217k tokens, measured
  live). Same math, bit-equal on the patcher gate; it checks the installed
  ComfyUI's forward against the copy by source hash and switches itself off
  with a log line if ComfyUI has changed. With kvi8s it takes the forward
  peak to +7.2 GiB (the streamed output head is now the peak, not attention).
- The int8_convrot video VAE is an approximation by construction (60 dB PSNR
  against the fp16 decoder on the same latent, encoder identical); the fp16
  VAE is what the exactness gates were run with.
## What is next (the roadmap, in the order it will be tested)
1. **Host RAM was the binding limit on the small machine; four flags moved it.**
   Every held IMAGE is 8.9 GB per 702 frames at 1376x768 in fp32, ComfyUI pins
   a ~10 GB staging copy of the DiT, and the execution cache holds every
   output until the prompt ends: 41.9 GB. With `--fp16-intermediates
   --disable-pinned-memory --cache-none` (all measured one at a time and
   together) the same pipeline peaks at 14.6 GB. Note the fp16 IMAGE is
   normalised in fp16 before the encoder, so the pass-2 result is a sibling
   take of the fp32 run (28 dB between them, judged equal by eye). What is
   left for code: streaming the smear straight into the encoder (never
   holding the dilated IMAGE), uint8 for view-only outputs, and an fp32
   normalise on the encode input if anyone wants the fp32 sibling back. VAE
   activations themselves are not the problem (0.6 GiB decode / 2.2 GiB
   encode transient at the 256 px tiles).
2. **kvi8s live**, then the sensor bank over seeds for both stores; a per-32-value
   scale variant of kvi8r (finer than per row); fp8 K/V if a kernel takes it
   without dequant; int4/nvfp4 K/V behind the same gate.
3. **The weight side**: where the streamed block forces extra weight decode
   per token chunk (W4A8's int4->int8 decode is repeated per chunk: measured
   +25% wall at 29k tokens, +6% at 217k); decode once per block and reuse,
   larger chunks where the flash split-KV rule allows, kitchen's fused paths
   (SwiGLU fold, `convrot_w4a4` for the MLP).
4. **A real 16 GB card**, someone else's, end to end on the example graph.
5. Later, after the static rungs: dynamic precision routing (per block, per
   step, FP4 vs FP8 by measured reconstruction error; the first experiment is
   a heatmap on real activations, no kernels), and the ViT VAE decoder's own
   low-bit rungs (already on int8 tensor cores in the int8_convrot file; the
   remaining case is speed, ~40 s per 702 f decode).
## The environment this was measured in
Everything below was measured on one
machine, fenced down to look like small ones (real renders on the real GPU with
the VRAM held by a balloon and the RAM capped by a cgroup). **How the fence
was read:** the first day's "16 GB" rows ran with 15.4 GiB device-free as
`torch.cuda.mem_get_info` sees it (a headless 5070 Ti reports 15.92 GiB total
and about 15.5 free); the evening's kvi8r run used a stricter 15.0 GiB fence,
which is a 5070 Ti with its desktop attached. Dynamic VRAM fills whatever the
card has, so process peaks always read as the fence; the numbers that carry a
memory claim are the in-process forward peaks from `H3 Memory Probe`. Nothing
has been run on a physically different 16 GB card yet; that is the next thing,
and it is a dogfood ask, not a research question.
One workstation: two RTX PRO 6000 Blackwell (96 GB, 188 SMs), EPYC 9554,
91 GB RAM, NVMe, Ubuntu, ComfyUI 0.33.0, PyTorch 2.14 nightly cu132,
comfy_kitchen native ops, PyTorch flash attention (comfy default), MiniMax-H3
W4A8 and int8 checkpoints. Small cards were simulated by holding VRAM with a
balloon process so ComfyUI, dynamic VRAM and torch all saw a real 16 / 24 /
32 GB card; small machines by running ComfyUI in a cgroup with a hard memory
limit and swap off. Only ComfyUI ran during the measurements. Not measured:
Windows, other GPUs, ROCm, non-flash attention backends, real PCIe 4 boxes.

The graph of the lines (VRAM activation line with per-card ceilings, RSS by
graph type) is `LOWVRAM_lines.html` next to this file; it grows as the sweep
does.