dzmdd's picture
Upload folder using huggingface_hub
e488ce7 verified
|
Raw History Blame Contribute Delete
9.45 kB

De-rope for any model (branch generic-derope, not shipped)

This branch carries two things: the LTX-2.5 graphs and tools behind the public de-rope deck (https://matlowai.github.io/flipbook/derope.html), and the any-model layer that lets one hold map drive any regenerating model through per-model presets. It is a branch on purpose: what ships on main is the H3 Motion Lab; this merges once the oracle-shaped LTX arm has been measured.

What it does today: a FLAT clock

An H3 de-rope (H3JerkOracle -> H3TimeSmear -> regenerate -> H3ExactRecover) slows each frame by its own hold (1 to 4, with ramps and bridges), which is what makes the regenerated motion land on the right frames. The LTX-2.5 graphs here use the H3 oracle only to choose WHERE the window sits (its hot span), and inside the window every frame is held exactly d times (ImageBatchRepeatInterleaving, d = 8 or 16), padded to LTX's legal length (8k+1), regenerated from a given starting sigma, and recovered by taking every d-th frame.

Why flat: LTX compresses time by 8, so a held frame only owns its own latent block when its hold is a multiple of 8 frames. Holds of 1 to 4 share blocks, and measured on two scenes that is exactly where d=8 barely responds to the denoise dial (0.94 to 0.90 across 0.1 to 0.9) while d=16 moves it (0.90 to 0.28).

Per-pass cap: 57 source frames at d=16 (positional_embedding_max_pos[0] = 20 with time in absolute seconds). Full coverage of a longer clip is several overlapping windows joined by tools/ltx25/splice_windows.py, which measures PSNR across each overlap before it blends.

The any-model layer (new nodes)

The smear, recover and audio nodes were already pixel-level and model-agnostic; what was H3-specific was the grid law (17k+5 lengths, holds in H3 frames). That law is now data, model_profiles.py, and one node applies it:

  • H3 Clock Remap (any model, presets) - wire any hold map (H3 Jerk Oracle, H3 Manual Hold Map, H3 Drawn Plan), pick the model that will regenerate the frames. Holds of 2+ are scaled by the profile's hold_scale and quantised to whole latent time blocks; holds of 1 stay 1 (plain video anywhere). The output map carries the target's legal grid, so H3 Time Smear pads to that grid (LTX-2.5: 8k+1, not 17k+5) and H3 Exact Recover / H3 Audio Recover invert it unchanged. minimax-h3 is the identity: existing graphs do not change by construction (tests assert it). The report is the price tag: frames, seconds at the model's fps, windows needed under its cap, and a loud UNMEASURED line for any profile whose numbers did not come from a ladder.

  • H3 Save Hold Map / H3 Load Hold Map (sidecar) - write <prefix>.holdmap.json beside the render; read it back by prefix (newest wins) in another graph, so the H3 pass decides the clock and an LTX-2.5 or Wan 2.2 graph regenerates on it without both models loaded at once. Load hands back the ORIGINAL clock (source_holds), never a previous target's grid. Without it a render's clock is lost when the graph closes; with it the staircase ruler can read H3 arms and a later remap can retime the same plan.

  • H3 Jerk Oracle profile_mode: "value |d3| camera-compensated" - aligns each latent frame to its predecessor by the best small integer shift before differencing, so a pan or scroll stops scoring as jerk (the documented cause of panrun dilating 124 -> 345 frames). Same compiler, different signal.

  • H3 Jerk Oracle model_profile - the oracle can read ANOTHER model's video latent: pick the preset whose latent is wired in (LTX-2.5: 128 channels on a 1+8k token clock; Wan 2.2: 16 channels, 1+4k). Same planner, that model's frame<->token mapping, no H3 phase normalisation, holds per source frame as before. Wrong channel counts and wrong lengths are refused with a message that says what was expected. minimax-h3 is the shipped path, bit-identical (tested). An independent port of the three motion nodes onto LTX-2.3/2.5 latents appeared publicly in August 2026 and confirmed this is the right seam: the oracle, the smear and the recover carried over with only the grid law changed, which is exactly what the profile holds.

Profiles

id block hold_scale legal fps cap latent (channels, clock) measured
minimax-h3 1 1 17k+5 24 VRAM 24, (1,4,4,4,4) per 17 (H3 code path) yes
ltx-2.5 8 4 8k+1 48 (the conditioning frame_rate) 20 s / pass = 960 frames 128, 1+8k yes (2026-08-21 ladder)
wan-2.2 (unmeasured) 4 4 4k+1 16 none 16, 1+4k NO - placeholder

hold_scale is measured, never derived: LTX-2.5 needed two 8-frame blocks per hot hold (d=16) before the dial responded at all, which its compression ratio alone would not have told you.

Adding a model without touching code

Pick custom (use the fields below) on the node and type the five numbers; the report stamps the result unmeasured. When a row works, save it as a preset: create <ComfyUI user dir>/mainodes_models.json (or point $MAINODES_MODELS_JSON at a file) with

{"my-model": {"name": "My Model", "block": 4, "hold_scale": 4,
              "legal": [4, 1], "fps": 16, "cap_seconds": null, "measured": false,
              "latent": {"channels": 16, "first": 1, "block": 4}}}

and restart: it appears in the dropdown, and a row with a preset's id overrides that preset. Send us the row and the ladder behind it.

First measurements (2026-08-21, one pass on two lab instances)

  • H3 two-stage graph with the identity remap in the chain: bit-identical to the production render (PSNR inf on baseline, dilated and recovered).
  • LTX-2.5 on the camera-compensated H3 clock at x4: panrun needs a window (the live clock is 1201 dilated frames at x4, over the 960 cap; a 98-frame window fits at 951), swordspin fits whole (866). Hold-map staircase 0.864 (panrun) / 0.818 (swordspin) at starting sigma 0.4, against 0.80 / 0.78 for the uniform d=16 windows: the oracle-shaped clock does not fix LTX's weak response at that denoise; the denoise is the limiter, not the clock. 298 s / 247 s, peak 59-60 GiB.
  • Wan 2.2, first ladder cell (x1, 21-source-frame window on the densest hold-4 run, low-noise 14B expert, denoise 0.4, 640^2): staircase 0.693 on swordspin (the lowest of the set), 0.877 on panrun's window; 119-126 s including the model load, peak 33-52 GiB. The wan-2.2 preset stays unmeasured until the x1/x2 ladder settles its hold_scale; x4 cannot fit Wan's 81-frame training length.
  • Camera-compensated oracle: identical to the default on the static swordspin shot; trims panrun's held span 80 -> 76 frames.

Sizing the oracle-shaped LTX arm

On the saved oracle profiles the x4 rule gives panrun 712 / swordspin 816 dilated frames over the hot span, under the 960-frame cap (913 was the largest legal length used), so it fits one pass at the same cost as the uniform d=16 window with the time budget spent where the oracle put it. That arm is the one measurement left before this merges. The x8 rule (hold 4 -> 32x) needs two windows per scene.

Files

  • examples/ltx25/derope_any_model_ltx25_api.json - the any-model path on LTX-2.5: H3 Manual Hold Map (or an oracle) -> H3 Clock Remap (ltx-2.5) -> H3 Time Smear -> the LTX-2.5 pass -> H3 Exact Recover, with H3 Save Hold Map beside the render. API JSON (drag onto the canvas); the UI twin lands once an instance with these nodes can convert it.
  • examples/ltx25/derope_any_model_h3_oracles_api.json - the H3 base pass with both oracle modes (default, camera-compensated), each saved as a sidecar, plus the identity remap saved beside them for a bit-exact check.
  • examples/ltx25/derope_any_model_ltx25_fromsidecar_api.json - the whole clip on LTX-2.5 in ONE pass from the camera-compensated sidecar (panrun: 124 frames -> ~785 dilated under the 960-frame cap, quiet frames at native rate).
  • examples/wan22/derope_any_model_wan22_api.json - the same clock on Wan 2.2 (low-noise 14B expert, v2v from the smeared frames at denoise 0.4, 640^2, 16 fps, 4k+1 grid from the wan-2.2 (unmeasured) preset): the first ladder cell that turns that preset's numbers into measured ones.
  • examples/ltx25/ - four graphs as run for the deck, API JSON (*_api.json, what actually executed) and UI JSON (paste on the canvas): panrun_or_d16_d040 (uniform d=16 de-rope of the oracle window), swordspin_or_d8_d050 (d=8), swordspin_or_x2_d8_d050_d030 (one-pass x2 latent upsample of a d=8 pass), panrun_ltx_d040_full (first of the three windows behind the spliced full-length clip). Source clips are referenced by basename: put yours in ComfyUI input/.
  • tools/ltx25/mint_ltx25_derope.py - mints a uniform de-rope graph from the stock LTX-2.5 T2V template (--frames --fps --dilation --seed ...).
  • tools/ltx25/mint_ltx25.py - the plain LTX-2.5 graph minter the ladder used.
  • tools/ltx25/ltx25_oracle_window.py - reads a saved H3 oracle profile and returns the legal LTX window over its hot span.
  • tools/ltx25/splice_windows.py - joins window renders, checking alignment.
  • tools/ltx25/ui2api.py - the UI -> API converter the minters depend on.

Model files are the stock Comfy-Org LTX-2.5 set (22B transformer, its VAE and audio VAE, the x2 latent upsampler) plus the MiniMax-H3 set for the H3 arms.