File size: 43,044 Bytes

e488ce7

# Tuning the Motion Lab pipeline

A working guide for dialing the pipeline to a specific user's content and
taste. Written for humans and for AI assistants doing the tuning on a
user's behalf. Everything here comes from same-seed comparisons on our
test clips; expect your content to move the numbers a little.

## First: pick the graph

The start-here index with diagrams is `examples/README.md`; the hardware
rows are `HARDWARE.md`. The short table:

| priority | graph | cost (vs one baseline render) |
|---|---|---|
| the normal starting point (since 2026-08-19): 12-step base pass 1, turbo 6-step pass 2, audio seeded | `examples/motion_pipeline_ref2va_audioinit.json` | ~1.3x; 192 frames at 1 MP in ~12 min on our card |
| best quality, base model throughout, audio dial available | `examples/motion_pipeline.json` | ~3.5x; the same clip at 25 steps both passes is ~3x the row above |
| a 16 to 24 GB card (12 reported working in the field, just slower) | `examples/motion_pipeline_lowvram.json` | as the first row; see `LOWVRAM.md` for the fenced measurements |
| prompt and seed scouting, not a final | `examples/motion_pipeline_fast_iterate.json` | ~1x or less (0.2 -> 0.4 MP, ~95 s) |
| user knows where the problem is (two-pass: review the oraclemap, type ranges, requeue) | `examples/motion_pipeline_targeted.json` | baseline + regen cost of YOUR spans only |
| GUI editing on the node (blocks, painting, automation) | `examples/motion_pipeline_editor.json` | as targeted |
| maximum speed for one burst in a longer clip (segment crop + splice) | `examples/motion_pipeline_editor_segment.json` | baseline + regen of window+handles only; the crop report states the ratio |
| cheap pass 1, de-rope at the delivery size | `examples/motion_pipeline_upscale_derope.json` | 89% of native detail in 83% of the time (0.4 -> 1.5 MP) |
| de-roping footage that already exists (no baseline render) | `examples/archive/motion_pipeline_v2v_source.json` [alpha] | regen only, ~2.5x one baseline-equivalent render; there is no baseline pass |
| the dilated pass does not fit your card (OOM, or steps balloon while weights stream) | `examples/motion_pipeline_rolling_window.json` [alpha] | peak memory scales with the biggest WINDOW, not the clip. At the budget that forces a split: ~1.4x the generated frames, and on a card at the offload cliff wall time comes back to parity because the DiT stays resident instead of streaming. Measured on a fenced 32 GB budget: 30.7 GB peak + layer streaming one-pass vs 24.6 GB resident windowed, 460 s vs 440 s |
| earlier recipes (turbo-in-the-old-graph, probe + expert, featherweight, i50, split LoRA) | `examples/archive/` | kept so old write-ups resolve; do not start from these |

On turbo, the rule moved on 2026-08-19 and it is worth stating exactly.
Pass 1 decides the choreography and wants the base model at 12 steps or
more. Pass 2 repairs bursts from a known init and is where the turbo LoRA
belongs, **provided the whole pass-2 recipe comes with it**: `gradient_estimation`
and `linear_quadratic` on pass 1, a 6-step `beta` schedule on pass 2, the
audio seeded through `H3 Audio Smear`. The turbo LoRA dropped into the
25-step `res_multistep` graph with nothing else changed renders jerky and
pixelated (measured 2026-08-22, seven arms, all scrapped). That is what the
old "no turbo in the pipeline" rule was actually measuring.

The working rhythm: iterate prompts and seeds on the fast-iterate graph to
learn what a prompt gives you globally, then run the keeper through the
starting-point graph. The base model on pass 2 (the 25-step graph) is
better quality still, at three times the wall; reach for it when a final
has earned the time.

## Second: ask the user what they actually care about

The dials trade four things against each other: sharpness in the bursts,
pose fidelity to the source, render time, and audio feel. Get a ranking
before touching anything. A user who says "the flip looks melted" wants a
different dial than one who says "it changed my character's pose."

## Symptom to dial

| user says | do this |
|---|---|
| the reference image stops influencing pass 2 (or an upscale pass) after a graph change | before touching any reference dial, put `BasicScheduler` / stock sigmas back. One field report (2026-08-23) lost all reference influence to a third-party scheduler node and got it back with `BasicScheduler`; not a same-sigma comparison, so it is a diagnostic first step, not a theorem about that scheduler. Only then: reference strength, prompt roles, `ref_image_size` |
| dialogue or audio sounds wrong: mangled words, tags read aloud, audio that copies the audio reference instead of borrowing its voice | check the core first: `H3 Capability Probe` prints `tokenizer_special_tokens`; before ComfyUI PR #15808 (2026-08-22) `<d>` and six other H3 tags tokenized as two tokens each, and every dialogue reading on such a core is suspect. Update, re-render, then tune steps, turbo and audio refs |
| motion still smears inside a burst | raise `d_max` toward 4 if lowered; lower `q` toward 0.70; check `bridge` is not 0 |
| brief stutter or soft frames mid-burst | `bridge: 8` (fills the dip between burst peaks) |
| poses drift from the source (head angle, hand position) | `bridge: 0` first; if not enough, lower `inject` toward 0.5 |
| output ignores the source choreography, invents moves | `inject` is too high; come down toward 0.6 |
| source artifacts leak into the output | `inject` is too low; go up toward 0.7 |
| too slow / too expensive | raise `q` toward 0.85 (tighter spans); switch to the turbo graph; then the probe graph |
| coarse mosaic / tiling in a turbo pass 2 when pass 1 was ALSO turbo (the seed-hunt iterate-then-finish flow) | a turbo init only tolerates a SHALLOW turbo pass 2. Judged in playback, one fight clip, fl2va, 0.7 to 1.5 MP, rank-21 at 1.0: 1 step at inject 0.20 clean and good; 3 steps at 0.50 and 6 steps at 0.50 both out of quality bounds. A later playback check on a second content class found 1 step at 0.30 already visibly griddy, so 0.20 is the measured ceiling, not a conservative pick, and 0.30 to 0.50 is progressive tiling onset. The same 3-at-0.50 recipe is clean over a bare 12-step pass 1, so the init is the variable, not the recipe. For keepers from turbo seed picks: stay at 1 step / 0.20, or hand pass 2 to the base model. Do NOT stack a second shallow tap through pixel space: each extra tap pays a VAE encode/decode roundtrip that costs more than a 0.20 solve returns (judged: 2 taps a step below 1) | split the pass with the rolling-window graph: `H3 Window Plan` + `H3 Window Collect`. Set `max_dilated_frames` below the dilated count your card last survived, read the plan report before rendering, queue one window per item. Leave `coverage` on `full clip` on upscale graphs or the calm spans stay at baseline resolution |
| turbo output changes appearance (adds ornament, shifts style) | lower LoRA strength 0.8 toward 0.65; or raise `base_head` in the expert schedule so more structure forms on the base model |
| audio feels thin | raise `reference_mix` (needs the non-probe graphs); it is happiest near 0 or 1, mid values can double misaligned impacts |
| audio impacts feel soft | known vocoder trait; try `reference_mix: 1` for baseline foley, or accept lean |
| unprompted speech or vocal sounds appear | any voice your prompt describes without verbatim words gets gap-filled: sometimes a coherent invented line, sometimes non-language vocalizing. Script the words in the dialogue format, declare the voice indistinct by design, or cut the voice mention from the prompt |
| dialog sounds processed | check whether the speech overlaps a burst; unheld spans pass through untouched, so only speech during bursts is affected. `reference_mix: 1` restores the original line |
| probe init loses choreography | raise `probe_steps` from 6 toward 10 |
| background details change in regenerated spans (a flag recolors, props swap) | known limitation: detailed backgrounds re-roll during dilation. Try `inject` toward 0.5 (closer init tracking); simple backgrounds barely show it. A subject-only "foveated" mode that never regenerates the background is on the roadmap |
| smooth or slowly panning footage gets dilated spans much wider than the action | measured cause: on smooth content the latent jerk profile is nearly flat (max/min about 1.7 where the pixel jerk spans 20+), so a quantile threshold marks a fixed fraction of tokens hot by construction and the ramp+bridge fuse them into one wide window. Camera motion itself explains only a few percent of the jerk, so camera compensation does not help. Raise `q` toward 0.85, or gate the oracle through `H3 Manual Hold Map` with the ranges where the real action is |
| the oracle dilates spans the user does not care about / the pass is too expensive for one burst | `H3 Manual Hold Map` in gate mode: wire the oracle's hold_map, type the ranges that matter, everything else recovers to hold 1. The report output shows the effective length before the expensive pass runs |
| an action repeats or doubles in regenerated spans (two backflips become four) | the model fills dilated time with extra beats instead of slowing the existing ones; its prior for action density wins over the init. Try `inject` toward 0.5 (closer init tracking), and state exact counts positively in the prompt ("performs exactly two backflips in total"). On the roadmap as beat-anchored recovery |
| the clip contains a mirror or a big reflection and the beats double | do NOT blame the oracle first. Synthetic probe, 2026-08-09: a reflection showing the SAME motion at the SAME time leaves the oracle's output bit-identical (its threshold is a quantile of the profile, so duplicating the motion scales every token equally and selects exactly the same ones; profile shape correlation 0.9999). What does bite is a reflection prominent at a DIFFERENT time than the body, which a camera tilt produces (mirror early, subject late): that leaves two separated hot clusters and `bridge` welds them into one long plateau, +26% dilated time in the probe, of which `bridge` alone is +18%. So on mirror content try `bridge: 0` first, then `inject` toward 0.5, then explicit counts in the prompt. Everything left over after that is the model's action-density prior, not the oracle |
| background elements (birds, crowds, traffic) speed up during bursts | the AUTOMATIC remedies stay rejected (oracle-mask compositing popped at the boundary, blanket freeze degraded other artifacts). The manual path is new and promising: draw the keep-baseline region yourself (`show_drift` on the heatmap shows what to lasso in blue), wire it into `H3 Motion Composite` `mask` with `invert_mask` on, feather 48-64 smoothstep, and put the seam on a real edge (horizon, rooftop) where nothing moves. Use the `H3 V2V Init` `mask` freeze instead when background and subject share lighting or contact. First internal A/B (demo clip, invented flock + pagoda both removed, static seam) awaits playback ratification |
| half a second of an otherwise good clip is wrong (a face, a prop, one bad beat) | do not re-render the clip. `H3 Repair Plan` + `H3 Repair Splice`: type the bad frame range, let the plan snap it outward to whole tokens, and splice the rest back as original pixels. Type any nearby shot cut into `shot_cuts`; the exit belongs ON a cut. AUDIO IS STILL A WORK IN PROGRESS here, the splice is video-exact and passes audio through untouched |
| the de-roped span still plays accelerated: background agents run fast inside the held region | that is the stock clock, not the oracle. `H3 DyRoPE` gives the true-duration grid to some blocks or some steps. Two settings worth trying, `physical_blocks` 30-49 and 40-49; see the dose section below, and expect to judge it in playback because the meters cannot rank the two |
| a de-rope with DyRoPE on shows flicker at the span boundary | flash is an early-block effect: keep the physical grid out of blocks 0-9. Any `*_blocks` range starting at 25 or above meters at the stock control's flash level. If flicker still bothers you more than jitter does, switch to `fade_physical_to_compact` at `fade_end 0.7`, which is flash-free in playback |
| the sound restarts, jumps level or changes room at a segment join | wire the previous segment's audio as `audio_latent` and set `H3 V2V Init`'s `audio_prefix_ticks` to the handle (a 39-frame handle is exactly 65 ticks). Add `audio_prefix_release_ticks 8` for a soft handover, and then the ASSEMBLY must let the continuation own the overlap tail or the trim discards the ramp. It carries level, timbre and beat, not phrase structure |
| contrast and texture wash out further with every link of a chain | `H3 Drift Control` on the pass-1 model: schedule-matched noise on the carried video prefix so the model stops treating it as impossibly clean. `prefix_frames` must equal the handle and `matched_steps` + `taper_steps` must equal its latent step count (39 f = 12 = 8 + 4). replicated across two chains and new seeds; joins hold 0.85-0.89 through link 4 (previously 0.65-class at link 2 without it).4% and handles accumulate it. `H3 Delta Color Carry` cancels the encode bias by construction. Active path exercised on a real 4-link render 2026-08-24: it fires, corrects in the anchor direction, and stays sub-visible under its clamps on mildly-drifted content; on the deepest tested link it fires on both channels at or near its clamp. A strong-drift bench (where the 2.4% per-round-trip signature actually accrues) is still owed. |
| a window edit renders garbage, and the damage sits where the window USED to be | the conditioning length was not moved with the crops and the hold map. All four numbers move together: see `docs/WINDOW_MODE.md` |

## VRAM expectations (measured)

An honesty note on the peak numbers in this section: they were read
from `nvidia-smi` on a large card, where the reading includes whatever
the caching allocator felt like keeping, so treat them as upper bounds
rather than requirements (we have since measured a smaller model
peaking HIGHER than a bigger one this way). The portable units are
latent token count for activations, and the `loaded partially` /
`lowvram patches` lines in the ComfyUI log for whether your card is
actually streaming weights. When the dilated pass is what does not
fit, the rolling-window graph caps the token count per pass: see the
table above.

Weights: the int8 DiT is 20 GB, the text encoder 15 GB (offloads after
encoding), the video VAE 4.9 GB. Activations at 1.0 MP run about 0.1 GB
per latent token: a 5 s clip (37 tokens) needs ~5 GB, and its dilated
regeneration pass (87 tokens) ~9 GB. Roughly half that at 0.5 MP.

- 96 GB: everything stays resident, ~11.5 s/step.
- 32 GB (5090 class): the DiT fits, but the dilated pass of a 5 s clip
  lands around 30 GB total, right at the cliff, so ComfyUI starts
  offloading and steps balloon (~20 s/it reports). What helps, in order:
  start with 2 to 3 second clips (the dilated span is what hurts), drop
  the regeneration to 0.5 MP, or use the probe graph so there are far
  fewer of those expensive steps. Or switch stacks entirely: see the
  featherweight numbers below.
- 24 GB: featherweight plus 0.5 MP plus short clips territory. Untested
  by us so far.

Cost scales with the burst spans, not the clip length, so a long clip
with one short burst is far cheaper than these worst-case numbers.

### Featherweight stack, measured (ComfyUI 0.31+)

The smallest community-published variant of each piece: w4a8 DiT
(12.5 GB, Kijai/MiniMax-H3-experimental), int8_convrot video VAE
(3.2 GB), nvfp4 AWQ text encoder (15.7 GB, offloads after encoding).
Needs ComfyUI 0.31 with comfy-kitchen 0.2.28. Full de-roping pipeline
measured on one card (peaks are torch-allocated for the whole process):

| run | wall time | peak, TE resident | peak with TE offloaded |
|---|---|---|---|
| 3 s at 0.4 MP | 4.0 min | 34.4 GB | ~19 GB |
| 3 s at 0.7 MP | 6.3 min | 35.2 GB | ~20 GB |
| 5 s at 1.0 MP | 29 min | 43.8 GB | ~28 GB peak, ~20 GB sustained |

The last column is what consumer cards see: without `--gpu-only` the
text encoder leaves VRAM after encoding, which is default behavior.
The first column is the one **H3 Conditioning Bank** attacks: it banks
the encoded conditioning to disk and takes its `conditioning` input
lazily, so a queue item that hits the bank never executes the encode
node and the text encoder is not loaded at all. Wire it on any flow
that re-runs one prompt: rolling windows, seed hunts, extension chains.
Requeueing an unchanged graph does not need it (ComfyUI's node cache
already serves the conditioning); queueing a second workflow in
between, restarting, or editing anything upstream of the encode does,
and that is the normal shape of a session. **H3 Latent Bank** does the
same for a sampled pass: on the rolling-window graphs the pass-1
baseline is lost to the same evictions and re-renders in full before
window item 2 can start, so bank its LATENT (4.8 MB for a 107-frame
480x832 clip; the decoded frames would be 513 MB) and wire the noise
seed into the node's `seed` input so a new seed misses instead of
serving the old take. So
a 5 s, 1.0 MP de-rope fits a 32 GB card with a few GB of headroom (the
28 GB figure is a brief spike, under a minute total), and 3 s clips run
comfortably. Resolution barely moves the peak (0.4 to 0.7 MP added less
than 1 GB); frame count is what costs.

Two honest notes. Quality: the featherweight output is coherent and
sharp in our runs, but a given seed renders a DIFFERENT take of the
scene than the int8 stack; quantization changes the trajectory, so
compare quality, not pixels. Speed: on our card the w4a8 pipeline ran
somewhat slower per clip than int8 (29 vs low-20s minutes for the 5 s
case); on a 32 GB card that is the wrong comparison, because int8 does
not fit and offload-thrash costs far more.

## Chained clips and the audio clock

H3's video runs at 24 fps but its audio latent runs a 40 Hz clock, and
the two only agree on durations that are multiples of 1/8 s. Of the
legal `17k+5` frame counts, only k = 2, 5, 8, 11 ... (39, 90, 141, 192
frames) land exactly; every other length ships an audio stream up to
12.5 ms longer or shorter than its video (124 frames: audio 8.3 ms
long; 107: 8.3 ms short). One clip: harmless. Concatenate segments for
a long continuation and the error compounds into audible A/V drift --
we have watched a 10-minute extension from another tool visibly
separate by the end.

What to do about it:
- for anything destined for chaining, prefer the aligned lengths
  (90 / 141 / 192 frames);
- when assembling segments, trim or pad each segment's audio to exactly
  `frames / 24` seconds before concatenation, so no segment exports its
  rounding error to the next;
- inside this pack you are already covered: H3 Audio Recover, H3
  Segment Splice, and H3 Window Collect place and size audio on the
  absolute world clock (sample-exact per segment), so seams do not
  accumulate error.

`tests/test_audio_recover.py` is the regression guard for all of this:
run it bare for the synthetic sample-exact checks (no GPU), or hand it
a `baseline.mp4 recovered.mp4` pair from any `reference_mix: 1` render
and it verifies the recovered audio really is the baseline track --
duration within one AAC frame, zero lag, full correlation. It caught a
pre-fix render 64 ms short on its first outing.

### Known-good environment (this works for me)

Every number in this document was measured on this exact stack, on
Blackwell silicon, so it is a safe reference point for 5090-class cards:

```
GPU:      RTX PRO 6000 Blackwell / driver 610.43.02
Python:   3.12.13
PyTorch:  2.14.0.dev20260801+cu132 (nightly; Blackwell wants the cu13x builds)
CUDA:     13.2, cuDNN 9.24, Triton 3.8.0
Attention: sageattention (no flash-attn, no xformers)
ComfyUI:  0.31.1 with comfy-kitchen 0.2.28
```

If a 5090 runs far slower than the table above, check the torch build
first: a stable cu126/cu128 wheel on Blackwell can cost you more than
any dial in this document.

### Conditioning modes (I2VA, FL2VA, Ref2VA)

The pipeline works with all of them, unmodified, at the shipped
defaults. Tested end to end on the featherweight stack: wire your
`first_frame` / `last_frame` / reference inputs into BOTH conditioning
nodes (the baseline's and the regeneration's) and run.

- I2VA: the first-frame pin lands at t=0 on any clock, so the
  regeneration honors it exactly. Recovered frame 0 matched the
  reference in our test.
- FL2VA: the last-frame pin names a timestamp that lands mid-clip on
  the regeneration's dilated clock, and it does not matter: at inject
  0.70 the injected trajectory owns the timing and the pins act as
  pose guidance. The recovered clip hit the anchor pose at the true
  end. At much lower inject values this could in principle fight; if
  your FLF de-rope stalls mid-clip, raise inject back toward 0.70.
- Ref2VA (the ref2va checkpoint, w4a8 available): reference tokens
  carry no timeline, so they pass through dilation untouched. Subject
  identity held through the full de-rope in our test.

## Manual targeting (time ranges and spatial masks)

When the user can point at the problem, let them. Two tools, one per
axis:

- Time: `H3 Manual Hold Map` converts typed ranges (`36-60,
  1.5s-2.4s:3`; frames or seconds, ends inclusive) into an
  oracle-format hold map. Gate mode (oracle hold_map wired in) is the
  default recommendation: the oracle decides hold strength, the user
  decides where it is allowed to spend. Ranges snap outward to the
  token grid, so trust the segments output over the typed numbers, and
  show the user the report output: it states world length vs effective
  regeneration length, and estimates minutes when given a measured
  s/step. Cost scales with held spans, so this is also the speed
  answer for long clips with one burst.
- Space: `H3 Motion Composite` `mask` (pixel space, full feather
  control: size, linear/smoothstep/gaussian profile, in/out/centered
  direction) pastes baseline timing back outside the mask after
  recovery. `H3 V2V Init` `mask` freezes the region during generation
  instead; its default is HARD latent cells (mask_feather 0): every
  ~16 px cell fully frozen or fully live, no fractional blend cells.
  Rationale: half-frozen cells denoise as frozen/live mixtures that can
  themselves mush the seam, while the decode's receptive-field overlap
  smooths a hard cell edge for free. mask_feather above 0 restores the
  pooled fractional ramp if a hard seam ever shows in playback (report
  it; that reading is one A/B from ratified). Either way this path
  cannot do fine pixel feathering; the composite is the fine-feather
  tool.

The GUI for both axes at once is the `H3 Motion Editor` node
(`examples/motion_pipeline_editor.json`): timeline blocks, per-frame
painting, per-block dials, and automation envelopes (hold, feather,
strength) compiled into the same hold_map/mask wires. Queue once to
load the filmstrip, edit on the node, queue again; the baseline stays
cached. Its mask output is already feathered, so the composite runs
with `mask_is_soft` on and its own feather at 0. When tuning for a
user who cannot or will not paint, author `editor_state` JSON for
them; the contract is in the node docstring.

Compute honesty when a user asks where the speed comes from: TIME
targeting is the lever. Sampler cost scales with the dilated frame
count, so gating holds to the user's window helps, and `H3 Segment
Crop` multiplies that by dropping the un-held world from the regen
pass entirely (its report states the ratio). SPATIAL masks do not
reduce FLOPs; the DiT still processes every token and the mask only
controls blending. Sell the mask as quality control and the window as
the speedup.

A mask authored without frame-by-frame attention WILL cut off props
that swing with the subject (a sword mid-cartwheel leaves a static
blob and gets truncated at the boundary). Fixes, in order: paint
per-frame in the editor following the prop, enlarge the static region
to the prop's full arc, or feather outward. This is exactly the
static-vs-per-frame trade; a human on the timeline beats any static
region for rotating props.

Method rules for masks: static union masks cannot pop (the boundary
never moves); route the seam along a real image edge, not through sky
mid-gradient; lasso generously and let feather work; per-frame mask
batches reintroduce the moving-boundary pop and need harder feather
and a playback check. The two-pass rhythm for the targeted graph is
queue once, read `mainodes_targeted_oraclemap`, type ranges, requeue.

## When the expansion stops before the clip does (expand_to_end)

A hold map that expands a burst and then drops back to rate 1 for the
last few world frames reads as a small jump at the end of the shot: the
picture is in slow motion and then, a beat before the cut, it is not.
`H3 Time Smear` and `H3 Temporal Insert` both carry an `expand_to_end`
toggle, default ON, for exactly that shape.

What it does when it fires: the trailing rate-1 run is lifted to the
rate of the span in front of it, and the resulting length is put back
on the 17k+5 grid inside that same span, spending the deficit on the
LAST frames so rates only rise toward the end. The t2c fight window
`[1]*17+[2]*34+[1]*5` (56 world frames, 90 dilated) becomes
`[1]*17+[2]*27+[3]*12` (107 dilated); the same shape with the burst
later, `[1]*34+[2]*17+[1]*5`, becomes `[1]*34+[2]*10+[3]*12` (90
dilated). Mixed rates are the normal outcome, not a fallback: the added
frame count has to be a multiple of 17 and a single rate rarely lands
there.

What it never touches: uniform dilation, a map that already ends inside
an expansion, a map that is rate 1 all the way, and any rate-1 tail
longer than 17 world frames. That last one is the tail guard: one whole
group of frames or more of real time at the end of a shot is intended
rest, not an end jump, and adaptive oracle maps produce it constantly
(a 124-frame map ending in 39 quiet frames would otherwise go from 250
to 294 dilated frames to slow down a section nobody asked to slow
down). Those come back bit-identical, so existing graphs keep their
results. It also never
lowers a hold, so the user never loses expansion they asked for, and it
does not move where the expansion begins: only the tail is rewritten.
A rewrite prints one line to the console with the before and after map
and repeats it in the node's report, so nobody has to guess whether it
fired. `expand_to_end` off is the old behaviour exactly.

Cost warning worth saying out loud before turning it on: running the
span to the end usually buys frames. 90 to 107 dilated frames is about
19% more frame data and more than that in time, because per-step cost
is superlinear in tokens. The report output prices it.

Two traps that cost a round each while this was being built:

1. `H3 Inject Schedule`'s `preset` overrides the `inject` knob unless
   it is set to `custom`. Typing 0.45 into `inject` and leaving the
   preset at "balanced 0.70" renders 0.70, silently.
2. `H3 Time Smear` pads any segment under 39 frames. `_legal_ceil`
   floors at 39, so a 20-frame window smeared at rate 2 comes out at 39
   frames with the whole pad folded into the final hold, which is a
   long freeze on the last frame. With `expand_to_end` on, that pad is
   absorbed by the expansion instead (`[1]*5+[2]*10+[1]*5` becomes
   `[1]*5+[2]*11+[3]*4`, still 39 frames, no freeze). Either way, do
   not hand a sub-39-frame window to the smear and expect the length
   you asked for.

## Source footage instead of a baseline render (alpha)

`examples/archive/motion_pipeline_v2v_source.json` swaps the first pass for a file
on disk: `LoadVideo -> GetVideoComponents -> H3 Video Fit -> ImageScale ->
VAEEncode`, and from there the graph is the standard one. The oracle reads
the source's own encoded latent; nothing else in the chain changes.

Two things to get right, in this order.

1. Frame count. H3's legal lengths are 5, 22, 39 ... 17k+5. An arbitrary
   file lands between them, and the VAE does NOT complain: `encode_temporal`
   pads the last chunk by repeating the final frame, so a 312-frame clip is
   encoded as 323 with 11 frozen frames on the end. The oracle then reads
   those tokens as calm and under-dilates the real ending. `H3 Video Fit`
   trims to 311 instead and hands the count straight to the oracle's
   `length`. Read its report; it states which end lost frames.
2. Canvas. The source is resized to the `ResolutionSelector` dimensions with
   a centre crop, and the same width/height feed the conditioning. Pick the
   aspect ratio that matches your footage or you will crop the subject out.

`max_frames` on the fit node is the cost lever. Per-step time goes as
tokens**1.7, so capping a long source before fitting saves more than it
looks: 192 frames capped to 150 fits to 141, which is 42 tokens instead of
57, about 1.7x faster per step.

Audio: the graph keeps YOUR track. Exact recovery restores the world clock
frame for frame, so the source audio still lines up with the regenerated
video; it is wired into `H3 Audio Recover`'s `reference` at `reference_mix
1.0`. Lower the mix to hear the model's own foley for the slowed
performance instead, or blend. If your source has no audio stream, the fit
node's audio output carries nothing and that link must be removed.

## Repairing a few frames instead of the clip

The repair verb is for a finished clip that is good except for a moment of
it. Two nodes: `H3 Repair Plan` does integer arithmetic, `H3 Repair Splice`
does frame selection.

The tuning is almost all in the plan's inputs.

- `bad_start` / `bad_end` are 0-BASED and inclusive. A file-numbered frame
  46 is 45 here, and getting this wrong by one is the most common way to
  repair the wrong token.
- `hold` is the dilation on the repaired span only; everything else stays
  real time. Set `H3 Time Smear`'s `dilation` to the same number, and turn
  its `expand_to_end` OFF: an end-expansion rewrite moves every dilated
  coordinate the plan just computed, and the plan's report says so when it
  detects the case.
- `shot_cuts` is where the quality is. Type the 0-based first frame of any
  new shot near the span. A cut inside the span, or within `cut_reach`
  frames of its end, becomes the exit, and the splice hands back to the
  original AT the cut so the seam is an edit the shot already had. On the
  validation clip, not doing this left the exit seam popping at 2.6x the
  clip's median frame delta while the model reinvented the next shot's
  opening frames; extending the mask through the cut did not fix it (2.3x);
  exiting on the cut did.
- Wire `images` if you have the frames handy. It costs nothing and the
  report then tells you whether your entry frame is quiet, and names a
  quieter frame within four if it is not. The plan never moves the entry
  for you: change `bad_start` yourself if you want it.

Read both reports rather than eyeballing the result. The plan states the
snapped token span, the splice bounds and the cost; the splice states the
entry and exit seam deltas next to the ORIGINAL clip's own delta at the same
frames, because at a hard cut a big number is the right answer. The measured
cell (90 frames, bad 45 to 47, hold 3, cut at 53) read entry 3.99/255 =
0.26x median, exit 138.04 against the original's own 135.57 (1.8% apart),
and maximum absolute difference 0 outside the splice, which the node
asserts rather than reports.

**Audio is still a work in progress for repair.** The video splice is
frame-exact; audio handling across the splice is not finished. The node
passes an AUDIO input through untouched, same length and same clock, which
is correct when the repair did not change the performance and is not a
general answer. If the bad frames carried dialogue, plan on repairing the
picture and keeping your original track.

## DyRoPE: choosing the dose

`H3 DyRoPE` is a dial on one question: how much of the run sees the clip's
true world duration instead of the stock uniform grid. That dose is what
buys the timing correction on a de-roped span, and it is also what costs
shimmer, so tuning it is picking a point on one curve rather than finding a
setting that is good at everything.

Two findings shape the whole dial. **Flash is an early-block effect**: the
physical grid in blocks 0 to 9 reproduces the entire flash penalty (7.84x
against a stock control's 2.09x), while every arm that keeps it out of those
blocks meters at the control's level. **Jitter is a dose effect**: once the
early blocks are excluded, it scales with total physical-time exposure
(blocks 25-49 2.61x, 30-49 2.33x, 40-49 1.39x; fading by sigma 0.3 2.72x,
0.5 2.07x, 0.7 2.06x).

The two settings worth starting from, with no winner between them:

| setting | what you get | what it costs |
|---|---|---|
| `physical_blocks`, blocks 30-49 | world speed held neutrally through the span | slight flicker or hitch inside the span |
| `physical_blocks`, blocks 40-49 | the smoothest span, and the cleanest meters measured (1.17x to 1.39x jitter across three takes) | on some content it slightly overshoots and reads over-smooth or slowed |
| `fade_physical_to_compact`, `fade_end 0.7` | flash-free in playback, timing held | visible jitter (2.06x to 2.87x across takes) |

Below those doses the world-time correction is gone entirely: `fade_end 0.8`
loses it because the window is too short, and stacking a block restriction
on top of a sigma window loses it too, even though both parents keep it.
Above them, shimmer rises. There is no quieter setting that is still on
time, so a clip whose timing slips wants MORE dose, not a cleverer one.

Three method notes.

- **The meters cannot pick for you.** They rank flash and jitter, and they
  say nothing about whether the timing benefit survived. Every ranking
  above came from a meter; every benefit boundary came from playback.
- **Match the arms.** `physical_all` reproduces `H3 True Clock` exactly and
  `compact_all` reproduces having no node in the graph, so an A/B is one
  widget rather than two graphs. Keep the seed, the hold map and the
  schedule fixed across arms.
- **Watch out for the patch collision.** The `*_blocks` modes and
  `H3 Streamed Blocks` both own the double-block replacement slot and
  ComfyUI keeps one per block; the capability probe reports it. The fade
  modes use no block patches and compose fine.

## Audio across a join (alpha)

`audio_prefix_ticks` on `H3 V2V Init` freezes the first n audio-latent ticks
to the seeded `audio_latent` and leaves every later tick at
`audio_strength`. It is the audio twin of the video prefix freeze, and it is
how a continuation carries the previous segment's sound instead of starting
a new performance.

Four constraints, all of which have bitten:

1. **It needs `audio_latent` wired.** The ticks are the audio latent's clock,
   not the video's: a 39-frame carried handle is exactly 65 ticks.
2. **An AddGuide AUDIO guide and `ref_audios` cannot coexist** in the same
   graph; they collide in conditioning-audio packing. (Measured on core 82f839f5; treat as a current-core rule, not a model property.) Pick one route.
3. **A release changes assembly.** With `audio_prefix_release_ticks` above 0
   the last n ticks of the frozen prefix ramp up on a half cosine (8 ticks,
   0.2 s, is the upstream lineage's tested recipe), and the assembly step must let the
   CONTINUATION own the overlap tail. Trim the other way and the ramp is
   discarded, which is worse than never asking for it.
4. **Scope.** It carries local continuity: level, timbre, beat. It does not
   carry global phrase structure. A longer handle helps at the seam (90
   frames measured 0.953 overlap correlation vs 0.92 at 39) and gives the
   model a full musical phrase of context, but only a pinned master track
   GUARANTEES the structure; that guarantee is by construction.

## Chained segments: drift and colour (alpha)

Two separate decays, two separate nodes, and neither is a substitute for the
other.

`H3 Drift Control` is for contrast and texture washing out down a chain. The
carried prefix is bit-clean conditioning, and repeated clean conditioning
accumulates error: the seam probe reads join one around 0.9 and join two
around 0.65 across every variant tested. The node gives the carried VIDEO
prefix a small schedule-matched amount of the sampler's own noise at every
model evaluation, tapering to exact at the seam end. Audio is untouched by
design; the frozen per-tick audio prefix stays hard.

Wiring and limits, in the order they trip people:

- Wire the `H3 V2V Init` latent in for shape (it must already carry the
  noise mask; the node rewrites that mask, it does not create one), feed it
  the SAME base model your pass-1 guider uses, and take the guider's model
  from this node's output.
- `prefix_frames` must equal the plan's handle and sit on the 17k+5 grid.
  39 frames is 12 latent steps, and `matched_steps` + `taper_steps` must sum
  to exactly that: 8 + 4 is the field-validated recipe.
- It **refuses to stack** on another dynamic denoise-mask patch, and it does
  not support sigma-split samplers.
- **Replicated across two chains and new seeds.** Joins hold 0.85-0.89
  through link 4 on the newer chain; the earlier chain fell to the 0.65
  class at its second join before the recipe existed. Cross-chain
  comparison, not a paired drift-off arm. The recipe numbers are still
  one recipe, not a swept space.

`H3 Delta Color Carry` (fed by two `H3 Scene Color Stats`) is for the other
decay: each carried handle is a VAE round trip, each round trip darkens
about 2.4%, and a chain accumulates it. It decodes the prefix once, applies
a weak scene-one exposure and saturation correction in RGB, encodes both the
original and the corrected frames, and adds only E(corrected) - E(original)
to the latent, so the encode bias cancels and only the grade survives. The
delta is low-passed spatially and tapered from zero at the old edge to full
strength beside the generated future; audio is never touched. Anchor is
scene one's delivered tail, current is the current predecessor's tail, and
on link one they are the same tail, so the transform is the identity.

Status, so nobody tunes against a number that does not exist yet:
Active path exercised on a real 4-link render 2026-08-24: it fires, corrects in the anchor direction, and stays sub-visible under its clamps on mildly-drifted content; on the deepest tested link it fires on both channels at or near its clamp. A strong-drift bench (where the 2.4% per-round-trip signature actually accrues) is still owed.

Both are adapted from ComfyUI-MiniMaxH3-Contex-Loop (GPL-3.0, the same
licence as this pack); the per-module credit lines in `h3_drift.py` and
`h3_color_carry.py` state what was taken and what was deliberately not
ported.

## Window mode (retiming an excerpt)

When only part of a long clip needs retiming, the window lives in four
coupled places: the prefix crop, the excerpt crop, the hold map, and the
regeneration's conditioning length. Move three of them and the render breaks
with the damage parked at the OLD boundary, because a latent longer than its
conditioning is garbage and the downstream masks resample to the old
proportions. Window starts also quantize to a 17-frame lattice (0.708 s at
24 fps), so a request to start at a given second lands on the nearest rung.
The arithmetic and the procedure are in `docs/WINDOW_MODE.md`.

## Method

- Change one dial at a time, same seed, and compare in playback. Still
  frames lie about temporal quality in both directions; we have been
  burned by this repeatedly.
- The oracle heatmap (`H3 Jerk Heatmap`) shows where the pipeline will
  spend its budget. If the hot region misses the artifact the user is
  pointing at, fix `q` before touching anything else.
- Sharpness metrics rank frames usefully within one clip but do not
  decide between settings. The user's eyes decide.
- Cost scales with the burst span, not the clip length. A long clip with
  one short burst is cheap to fix.

## Trying the indecision oracle (experimental)

`H3 Indecision Oracle` is a second signal for the same hold map: the
model's own x0 jitter instead of the clip's jerk. It is not on by
default and nothing else moved for it.

- Cost: free if pass 1 already ran through `X0 Tap` with two steps
  dumped. It reads files, it does not sample.
- Compare, do not switch. Put it where the jerk oracle sits, flip `mode`
  between `indecision`, `jerk passthrough` and `blend max` with
  everything else fixed, and judge in playback. `jerk passthrough` is
  the same numbers the jerk oracle produces, so the only variable is
  the signal.
- Keep `q`, `d_max`, `ramp` and `bridge` matched across the arms. They
  feed the same compiler, and an unmatched knob turns the A/B into a
  comparison of two hold-map compilers.
- The `comparison` output is the measurement, not the vibe: whole-map
  Spearman, top-decile IoU, and the token-times where the two sources
  disagree most. Look at those token-times in playback first, since
  that is where the choice actually shows.
- If the map looks like a rectangle rather than like the picture, check
  the report for the degeneracy line. Masked and repaint runs pin token
  rows to exactly zero jitter and the oracle then draws the mask.
- Expected failure mode: fast small props. Jitter under-ranks them
  relative to frame-diff, which is the argument for `blend max` over
  `indecision` alone.

## Defaults, and why

`q 0.75, d_max 4, ramp on, bridge 8, inject 0.70` is the playback-ratified
starting point. The shipped default is now `inject 0.48`: at 0.50 and above some
scenes measurably loosen reference adherence, and 0.48 keeps the sharpness
without that risk. `inject 0.50` measured sharper with closer motion tracking
on our clips and is one preset away; some of us prefer it. The expert
schedule defaults (`total 8, inject 0.70, base_head 2`) give the turbo
tail its native 4 steps.

## Identity vs strength: the reference rule

Whether a higher inject or denoise is safe depends on what is anchoring
the subject, not on the route.

- No substantial references in the graph: keep the repaint strength
  under 0.5 when the subject's identity must match the source. Identity
  drift is measurable from about 0.30 and structural past 0.5; below
  that line the init still owns the subject, above it the model's prior
  does, and no prompt wording buys it back.
- With substantial references (identity refs, FLF pins, a strong anchor
  frame): higher strength is fine, and on the high-step routes (the
  full de-rope at 25+ steps) it often helps more, because the
  references carry identity so the extra strength buys motion quality
  instead of drift. The pinned FLF de-rope holding identity at inject
  0.70 above is the worked example.

## Refining this guide

When a tuning session finds a symptom/dial pair this table lacks, add it.
That is the point of the file.