docs/attention.md

# Per-layer attention config LoRAs

On an RTX 5090, dense attention is ~2.4 s of every ~3.8 s step at 960×544 (5 s clip), so it, not the weights,
decides speed. ComfyUI can pick an attention kernel per layer from a weightless "config LoRA":

- **kitchen INT8 attention**: ~2.5× faster, ~1% error per layer;
- **kitchen SOL block-sparse attention**: each 64-token query block attends a routed subset of key blocks exactly
  and the rest through one pooled term; `tau` (in standard deviations of a proxy score) sets the sparsity — 4–14×
  faster, 11–36% error per layer.

comfyanonymous published a test LoRA that sets 23 layers to INT8 and 27 to SOL. We searched for better allocations.

## How the allocations were found

1. **Profile** the exact stack (INT8 ConvRot + dmad 4-step LoRA, 4 steps, 960×544): at every DiT attention call the
   dense result is kept and every candidate kernel (INT8, SOL at six `tau`) also runs on the same q/k/v, recording
   kernel time and error separately on the **video rows and the audio rows** of H3's packed sequence.
2. **Search** with an exact knapsack: least `video error + w × audio error` for an attention-time budget.
3. **Verify** each allocation end to end in ComfyUI: one-step output error against dense attention on held-out
   prompts, video and audio separately.

## Results on the tuned stack (INT8 ConvRot + dmad 4-step)

<!-- BEGIN:attn -->
| Allocation | Attention speed-up | Video error | Audio error | Layers (INT8 / SOL) |
|---|---|---|---|---|
| comfyanonymous test LoRA (`fast_sol_kitchen_attention`) | 3.32× | 18.4% | 13.6% | 23 / 27 |
| ours, same speed, video-row objective | 3.24× | 14.5% | 13.8% | 32 / 18 |
| ours, same speed, audio weight 1 | 3.25× | 14.6% | 13.3% | 32 / 18 |
| **balanced**: same speed, audio weight 3 | 3.24× | 15.1% | 11.6% | 31 / 19 |
| ours, same speed, audio weight 10 | 3.25× | 15.7% | 11.7% | 32 / 18 |
| **quality**: x3 | 2.94× | 12.2% | 10.7% | 39 / 11 |
| ours, x3, audio weight 1 | 2.96× | 12.5% | 10.7% | 39 / 11 |
| ours, x3, audio weight 3 | 2.96× | 12.5% | 10.7% | 39 / 11 |
| ours, x3, audio weight 10 | 2.96× | 14.8% | 9.9% | 39 / 11 |
| ours, x3.5 | 3.40× | 16.0% | 17.5% | 29 / 21 |
| ours, x3.5, audio weight 1 | 3.41× | 15.9% | 15.0% | 28 / 22 |
| ours, x3.5, audio weight 3 | 3.42× | 17.0% | 14.3% | 28 / 22 |
| **fast**: x3.5, audio weight 10 | 3.42× | 17.1% | 13.2% | 28 / 22 |
<!-- END:attn -->

Measured speed on the same stack (RTX 5090, 4 timed generations each):

<!-- BEGIN:attn_speed -->
| Attention LoRA | s/step | Time per clip (4 steps) |
|---|---|---|
| none (dense attention) | 3.83 | 25 s |
| comfyanonymous test LoRA | 2.16 | 21 s |
| **balanced** | 2.16 | 20 s |
| **quality** | 2.24 | 19 s |
<!-- END:attn_speed -->

At the same real speed as the reference LoRA, **balanced** has ~18% less video error and ~15% less audio error.
**Quality** gives up ~4% of the step time for ~35% less video and ~20% less audio error.

## Without the 4-step LoRA (base INT8 ConvRot, 20 steps)

The same LoRAs, unchanged, on a stack they were not tuned on:

<!-- BEGIN:attn_base20 -->
| Attention LoRA | Attention speed-up | Video error | Audio error |
|---|---|---|---|
| comfyanonymous test LoRA | 3.32× | 15.1% | 12.5% |
| **fast** (dmad 4-step) | 3.42× | 15.4% | 11.4% |
| **balanced** | 3.24× | 13.5% | 10.4% |
| **quality** | 2.94× | 11.2% | 10.6% |
<!-- END:attn_base20 -->

**Balanced** and **quality** stay better than the reference on both video and audio without the distillation LoRA;
the **fast** allocation is only worth it on the 4-step stack.

## Files

| File | Allocation | Use with |
|---|---|---|
| `loras/minimax_h3_attn_balanced.safetensors` | same speed as the reference, less error | any INT8/NVFP4 H3 stack, 4 or 20 steps |
| `loras/minimax_h3_attn_quality.safetensors` | ~4% slower, clearly less error | any INT8/NVFP4 H3 stack |
| `loras/minimax_h3_attn_fast_dmad4step.safetensors` | slightly faster than the reference, less audio error, about the same video error | INT8 ConvRot + dmad 4-step |

Load with **Load LoRA (Model Only)** at strength 1.0, after any weight LoRA. They hold no weights, so they also keep a
calibrated NVFP4 checkpoint's scales. Disable attention-override nodes, or the configs are ignored.

## Experimental: audio/text protection (needs a ComfyUI patch, not yet upstream)

ComfyUI's config passes only `tau` to SOL. A small patch ([ComfyUI PR #16994](https://github.com/Comfy-Org/ComfyUI/pull/16994)) lets H3 publish its packed row
spans so SOL keeps every query's text/conditioning/audio keys exact (`"sink_keys": "prefix"`) and the audio queries
fully exact (`"sink_queries": "audio"`): a few hundred of ~19.5K rows. It also passes through the other SOL options
and, separately, allows per-step configs (`"sigma_range"`). Same stack, RTX 5090:

| Attention LoRA | s/step | Video error | Audio error |
|---|---|---|---|
| comfyanonymous test LoRA | 2.15 | 18.4% | 13.6% |
| sinks, same speed | 2.17 | **10.9%** | **7.5%** |
| sinks, fast (4× attention kernels; ~6% per step) | **2.03** | 13.7% | 8.6% |

Removing the sinks from the same allocations raises the error to 14.1% / 11.4% and 17.5% / 15.5%: the search and the sinks
each account for roughly half of the gain over the reference. Per-step configs added at most ~8% on top and did worse at 4×. 

To test, check out the PR (`gh pr checkout 16994` in a ComfyUI clone) and load a sink LoRA as above, after the
dmad 4-step LoRA:

| File | Allocation |
|---|---|
| `loras/sinks/minimax_h3_attn_sinks_same_speed_dmad4step.safetensors` | reference speed, ~40% less error |
| `loras/sinks/minimax_h3_attn_sinks_fast_dmad4step.safetensors` | ~6% faster per step than the reference, less error |
| `loras/sinks/minimax_h3_fast_sol_kitchen_attention_sinks.safetensors` | comfyanonymous' test LoRA with sinks added, otherwise unchanged (see below) |

**comfyanonymous' LoRA + sinks.** The cleanest test of the patch: the original
[`fast_sol_kitchen_attention_test_lora`](https://huggingface.co/comfyanonymous/minimax_h3_experiments/blob/main/loras/minimax_h3_fast_sol_kitchen_attention_test_lora.safetensors)
by comfyanonymous with `sink_keys`/`sink_queries` added to its 27 SOL layers and nothing else changed (same layers,
same `tau`, same speed). Credit for the allocation is theirs; it is redistributed here unmodified apart from the sink
keys, for testing the PR.

| Attention LoRA | Attention speed-up | Video error | Audio error |
|---|---|---|---|
| comfyanonymous test LoRA | 3.31× | 18.4% | 13.6% |
| same LoRA + sinks | 3.31× | **12.3%** | **8.1%** |

Without the 4-step LoRA (base INT8 ConvRot, 20 steps), unchanged LoRAs:

| Attention LoRA | Attention speed-up | Video error | Audio error |
|---|---|---|---|
| comfyanonymous test LoRA | 3.31× | 15.1% | 12.5% |
| same LoRA + sinks | 3.31× | 12.3% | 7.1% |
| sinks, same speed (dmad-tuned) | 3.22× | **9.7%** | 7.2% |
| sinks, fast (dmad-tuned) | 3.98× | 12.9% | 8.1% |

Errors are per model call. Final clips from any attention LoRA diverge from dense attention about equally, so judge
them by eye and ear. Two blind tests (3 prompts × 2 seeds each, 8-step workflow at 1280×736): our same-speed sink
LoRA vs the reference 4–2; the reference with vs without sinks 3–3 on concert prompts and 5–1 on fine-texture prompts
(lace, ribbed swimsuit, mesh), 12–6 for sinks overall. Audio was indistinguishable throughout; the wins were coherence
in busy scenes (broken motion without sinks) and fine detail (hands, fabric noise). Small sample, consistent direction.

On stock ComfyUI these load and run, but the sink keys are ignored and the error is the "sinks removed" figures above:
use the LoRAs in the table further up instead.

## Ref2VA at 768×1344, 124 frames (sinks; needs the PR above)

Tuned on the ref2va pruned INT8 ConvRot transformer + dmad 4-step full-critic LoRA, `dpmpp_sde_gpu` / beta, 8 steps,
one reference image. At this size dense attention is ~78% of each model call, so faster allocations pay off more than
at 960×544. All files measured in one run (2 held-out prompts, same seed; DiT time per clip on an RTX 5090 with
DynamicVRAM, as in a ComfyUI server):

| Attention LoRA | Time per clip | Speed-up | Video error | Audio error |
|---|---|---|---|---|
| none (dense) | 198 s | 1.00× | – | – |
| comfyanonymous test LoRA | 91 s | 2.19× | 16.4% | 24.6% |
| **quality** (ref2va) | 97 s | 2.05× | 9.2% | 10.2% |
| **balanced** (ref2va) | 90 s | 2.20× | 11.1% | 11.3% |
| sinks fast (dmad 4-step, 960×544 tune) | 83 s | 2.40× | 14.4% | 14.7% |
| **x6** (ref2va) | 72 s | 2.75× | 22.2% | 19.3% |

At the reference LoRA's speed, **balanced** has a third less video error and half the audio error. Blind tests on two
prompts: dense, fast, balanced and quality could not be told apart; between fast, x6 and an 8× allocation, only the 8×
one was picked out as different, in both clips. Past ~6× the rest of the model dominates the step, so x6 is the fast
end.

### The same LoRAs on the SMD 8-step LoRA

[SMD Ref2VA 8-step](https://huggingface.co/zjwfufu/SMD-MiniMax-H3-Ref2VA-8step-ComfyUI) at its own settings (euler,
its ManualSigmas, shift 12/3, 1344×768, two references) gives a different stack: different LoRA, sampler, schedule and
reference count. We tuned it from scratch and compared:

![Per-layer attention allocations and sensitivity](../assets/attn_layer_map.png)

Which layers tolerate sparse attention barely changes between the two stacks (rank correlation 0.93–0.98 per kernel),
so a fresh SMD tune found nothing better than **x6** as published: at the same speed it measured 63.9% / 23.8% final
video / audio error against 67.2% / 26.3% for the SMD tune (2 prompts). Use the ref2va LoRAs on SMD as they are. Measured on SMD,
DiT time per clip:

| Attention LoRA | Time per clip | Speed-up |
|---|---|---|
| none (dense) | 102 s | 1.00× |
| **x6** (ref2va) | 36 s | 2.81× |

"Final error" is how far the finished latent lands from dense attention on the same seed. A few-step sampler turns
small early differences into a different but equally valid sample, so these numbers are large for every attention
LoRA. They separate allocations far apart in speed, but not ones at a similar speed (see below); they are not a
quality scale.

| File | Allocation |
|---|---|
| `loras/sinks/minimax_h3_attn_sinks_x6_ref2va_hires.safetensors` | ~6× attention, ~2.8× per clip: the fast end |
| `loras/sinks/minimax_h3_attn_sinks_balanced_ref2va_hires.safetensors` | 3.5× attention |
| `loras/sinks/minimax_h3_attn_sinks_quality_ref2va_hires.safetensors` | 3× attention, least error |

Load after the dmad or SMD LoRA, at strength 1.0. They were tuned at 768×1344 × 124 frames and work at other sizes,
but there they are not the measured best.

## All released LoRAs, measured together

Every file above re-measured in one run per stack, from the released files themselves: measured speed-up per clip
against one-step error (2 held-out prompts, RTX 5090, DynamicVRAM).

![Speed vs error of the released attention LoRAs](../assets/attn_loras.png)

At 960×544 × 4 steps the non-attention part of the model is a large share of the step, so every LoRA lands at
1.7–1.9× per clip; the sink LoRAs are the clear wins there. This run's margins over the reference are smaller than
in the first table on this page (different prompts and loader): **balanced** has 9% less video and 25% less audio
error at the same speed, and the non-sink **fast** is no better than the reference on video. The ranking of the sink
LoRAs holds.

Final-latent error did not rank these LoRAs: at 960×544 the reference ended closer to dense (55%) than our non-sink
LoRAs (67–73%) despite more one-step error. Between LoRAs of similar speed it is dominated by how a few-step sampler
amplifies any difference, so judge those by eye and ear.