File size: 21,906 Bytes

e488ce7

# Roadmap: methods we are investigating

> Status ledger for what has shipped vs what is open:
> [`docs/roadmaps/ATOS_ALPHA_STATUS.md`](docs/roadmaps/ATOS_ALPHA_STATUS.md).
> The persistent-world design and its status live there too.

Companion to `RESEARCH_NOTES_ATOS.md` (the claim and the evidence plan),
`TUNING.md` (symptom to dial, including the rejections) and
`TESTING_ALPHA.md` (what the alpha nodes still need from human hands).

Everything below is a direction, not a promise. Items are grouped by the
problem they attack, and each says what would count as success and what the
cheapest first measurement is. Where we have already measured something that
contradicts an earlier claim of ours, it is written down as such.

House rules that constrain every item here: motion is judged in playback
because stills lie in both directions; a measurement earns its compute only
if it beats the cheap baseline it replaces; and anything shipped as a default
has to survive a same-seed A/B.

---

## 1. The oracle: from allocator to meter

**The problem, measured and published in the research notes:** the jerk
oracle ranks but cannot abstain. Because the threshold is a quantile of the
profile, a clip with exactly zero trajectory jerk still received 3.06x
dilation. And in the value domain the third difference is nearly the same
signal as the first (corr 0.960 and 0.977 on real clips), so the score is
contaminated by motion energy. This is the single most important open
problem in the method.

Directions, roughly in order of expected value per hour:

- **Paired-speed reachability.** Generate the same choreography at two
  content speeds and diff the latents. Tokens whose state actually changes
  between the two are the tokens that motion speed reaches; tokens that look
  hot in both are hot for some other reason, which is exactly the
  textured-object-passing-a-location confound. Borrowed from the practice of
  probing a system with paired semantic perturbations rather than noise.
  Success: a per-token relevance score that separates the two synthetic
  clips (zero jerk versus lurching) where the quantile does not. Cost: two
  renders plus offline analysis of saved latents.
- **Demand over capacity as an absolute unit.** A token spans four frames;
  the failure is that those four frames need more distinct poses than one
  token can hold. So the natural absolute quantity is required pose change
  divided by representable pose change, with the hold count falling out as
  that ratio rounded up. Calibrated once against the speed at which smear
  actually appears, instead of fixed by a quantile forever. Success: the
  meter says "dilate nothing" on a smooth clip and "dilate everything" on a
  uniformly violent one, both of which the quantile gets wrong today.
- **Trajectory-domain measurement.** Showing promise on real clips
  (top-decile share 0.29 versus 0.22 for velocity, with only moderate
  correlation between them). Track a region centroid and differentiate the
  trajectory rather than the latent values. Now selectable on the oracle as
  `profile_mode`. **Caveat measured while shipping it:** on a smooth
  synthetic blob the centroid path is nearly noise-free, so its peak-to-mean
  contrast is not comparable to the value domain's there (1.21x versus 1.88x
  separation between a constant-velocity and a lurching toy). Treat it as an
  ablation option, not as an established improvement, until it is measured on
  real textured content.
- **Knee-finding instead of a fixed quantile.** Run the cheap early-x0 probe
  at several dilation factors and take the knee of the curve per clip.
  Unlike a quantile, a knee can land at 1.0x, which is the abstention we
  currently cannot express.
- **Absolute calibration by temporal capacity profiling.** Feed the VAE
  synthetic controlled-speed content (moving gratings, rotating props, speed
  sweeps) and measure reconstruction fidelity against speed to get the
  model's temporal transfer curve. Useful beyond this repo: it would be a
  reusable way to compare base models on temporal capacity.

**Ablation owed regardless:** delta order 1 versus 2 versus 3, phase
normalization on and off, against external baselines (optical flow magnitude
and flow acceleration on decoded frames). If the third difference wins or
ties while being free, that is the result. If it loses, the honest move is to
say so and rename the thing.

## 2. Detection quality

- **Camera-compensated jerk.** Subtract the dominant global motion component
  per token before thresholding, to stop pans inflating the dilated spans.
  Progression: global jerk, then camera-compensated, then spatially
  localized overload.
- **Stabilize, repair, unstabilize.** The alternative to teaching the oracle
  to ignore pans: remove the pan first with a coarse global warp, run the
  pipeline where subject motion is the only motion, then invert the warp.
  A classic VFX move that turns the hardest confound into preprocessing.
  Worth comparing directly against camera compensation.
- **Cross-modal detection.** The model co-generates audio, so audio onset
  density is a second, independent account of where the action beats are.
  Where video-hot and audio-quiet disagree is a good candidate signature for
  camera-pan inflation. This also gives the oracle a **free baseline it must
  beat**: a latent detector that cannot outperform onset detection on
  co-generated audio has not earned its complexity.
- **Mirror and reflection double counting.** Published correction in TUNING:
  a synchronized reflection leaves the hold map bit-identical, and the real
  amplifier is `bridge` welding a temporally separated reflection to the
  body's burst. A real detector needs approximate reflection symmetry of the
  spatial jerk map, not temporal correlation, which we probed and found
  cannot work (one body's own limbs correlate at ~1.0, so the confuser floor
  sits above the signal). Untested on real mirror footage.

## 3. Allocation and cost

- **Fixed-budget bidirectional retiming.** Today the method BUYS temporal
  capacity for bursts. The general object is a monotone retiming function
  whose two signs are dilation and decimation: compress the quiet spans and
  spend exactly what you saved on the bursts, for a constant token budget.
  "Repair at zero marginal cost" is a much stronger claim than "3.5x for
  quality." Open risk: decimated frames have no generated source, so
  recovery there is nearest-held-frame and may judder in quiet spans.
  Playback decides.
- **Honest cost accounting, already shipped.** Per-step time is superlinear
  in token count (measured exponent 1.75 here, ~1.64 in a field report,
  `COST_EXP 1.7` in the code). Reports now state time multipliers rather than
  frame ratios. The practical consequence is counter-intuitive and worth
  repeating: many small windows can be cheaper than one large one, so
  pass-count is the wrong unit for comparing any of these approaches.
- **Rolling-window regeneration, for cards that OOM on the dilated pass.**
  What runs people out of memory is not the clip, it is the dilated span: a
  5 s action clip regenerates as 11 to 13 s of frame data, so peak VRAM
  tracks the burst budget rather than the runtime. Field reports put the
  wall around 1 MP past 10 s on a 3090, 128 GB of system RAM
  notwithstanding. Sliding a fixed-size window across world time,
  regenerating each window and splicing, turns peak memory into a knob
  instead of a consequence: the user sets the window, and clip length stops
  mattering. `H3 Segment Crop` and `H3 Segment Splice` already do exactly
  this for one window, with context handles and video plus sample-accurate
  audio crossfades, so what is missing is orchestration and a seam policy
  across N windows, not new machinery. A community continuation graph
  (`seamless_extension_derope.json`, 2026-08-12) already wraps the de-rope
  chain in a segment loop for a different purpose and is worth reading
  before designing ours.
  It is also not the slow option, though by less than the bare exponent
  suggests. Naive arithmetic says N windows cost N^-0.75 of the single-window
  time, four windows for about a third. Fitting 266 corpus runs and then
  charging for duplicated handle frames and grid padding gives **0.69x at 300
  dilated frames, 0.45x at 600 and 0.37x at 1000**: real, and roughly half
  the advertised saving. Handle duplication, not per-window overhead, is what
  eats it, and it turns back into a loss somewhere around 8 to 12 windows.
  The same fit found the step-independent cost is not constant either, it
  scales at t^1.66 against the step's t^1.70, so it splits along with
  everything else.
  Two things are already true that this item assumed were missing. Chaining
  `H3 Segment Splice` so each output becomes the next one's baseline
  reconstructs the world to 6e-8 today, with no new code, so N-window
  reassembly is not the gap. And core's graph-expansion API means owning the
  iteration does not require a fat node that swallows the sampler.
  The real difficulty is the seam, and specifically the HOT cut, a window
  boundary landing inside a burst, which is exactly the case the feature is
  requested for. Handle frames are forced to hold 1, so a crossfade there
  blends repaired against un-repaired content at peak motion, and the audio
  problem is worse: two independent foley takes of one impact, which no
  crossfade width fixes. Cold cuts, where the boundary sits at hold 1, are
  just the shipped single-window path N times and carry no new risk.
  Success: peak VRAM flat in clip length, seams invisible in playback. Note
  we cannot currently measure the first half of that, `peak_vram_mb` in our
  own telemetry is flat at 55-62 GB from 5 to 294 frames. Cheapest first
  measurement: run the shipped crop/splice pair over two adjacent windows of
  one clip and look at the join. That needs no new code at all.
  Worth saying plainly: two cheaper answers to "I OOM" already ship, lowering
  `d_max` or raising `q`, and `H3 Segment Crop` as it stands. Rolling windows
  only win for someone insisting on full `d_max` on a burst larger than their
  card, and nobody has produced that clip yet.
- **Extension: make a long clip from short renders (shipped alpha 2026-08-23).**
  The other long-video job: not one long world chunked, but a chain of
  short worlds with a carried handle. `examples/motion_pipeline_extend_api.json`
  and `ALPHA.md`. What the first cells taught, in order: protecting the
  handle from RETIMING is not enough, pass 2 still re-denoises it (20/255
  drift) until it is also frozen (3.3); the REPRESENTATION beats the
  prompt - the tail written into the next segment's own latent under a
  per-token mask makes the join an ordinary frame transition (0.86x to
  1.1x vs 5x to 6.5x for the image guide) and keeps the camera's velocity
  continuous, while an explicit camera-transition sentence changed nothing;
  each VAE round trip darkens the frozen prefix ~2.4% (per-channel gains
  fitted on the hidden overlap fix it, an audio rms gain measured
  negative); and a burst that runs into the cut comes back fast unless the
  last token group of a non-final segment is protected (1.55x -> 1.06x).
  Open: the drift curve over many segments, 192/90 vs 141/39, inject on
  continuation segments, dialogue across a join, the audio ambience bed
  (wants a per-tick audio mask).
- **A ceiling on dilated frames** (field request, 2026-08-12). A max-length
  option that drops lower-priority frames to keep people off the OOM rail
  while the highest-priority ones still reach pass 2. In our terms a hard
  cap on the dilated budget, filled greedily from the top of the jerk
  profile, so the pass fits the card and the hottest spans keep their holds.
  Small change at the `_compile_hold_map` seam. The blocker sits upstream of
  the cap: at today's false-positive rate a budget can be spent entirely on
  a camera pan, so this wants the detector work in section 2 first, or it
  will confidently protect the wrong frames.
- **Spatial crop regeneration.** Time targeting and segment crop reduce
  FLOPs; spatial masks do not, they only blend. A spatial crop analog of
  segment crop would make "foveated" literal. Risks are context loss and
  resolution mismatch at the seam.

## 4. Beat duplication and anchoring

The model fills dilated time with extra beats instead of slowing the existing
ones (two backflips become four). This is the most visible remaining artifact
and it has a clear line of attack.

- **Sparse pinned time.** We already have the choreography: the baseline.
  Sampling sparse pose anchors from it and pinning them at their retimed
  positions gives the regeneration a ladder to climb instead of room to
  improvise. Success: beat count stops scaling with dilation factor.
- **Dependency: resolved upstream (checked against core 2026-08-20).**
  When this item was written, stock ComfyUI restricted H3 keyframe anchors
  to first and last, and two community packs rewrote the temporal column
  after packing to get around it. Upstream
  [PR #15439](https://github.com/Comfy-Org/ComfyUI/pull/15439) (merged
  commit `e01fb4c5`) made arbitrary guide positions native:
  `MiniMaxH3AddGuide` takes a still or a clip at any frame index
  (negative counts from the end), a clip guide snaps DOWN the 17k+5 grid
  (5, 22, 39, ...; fewer than five frames becomes a single-frame guide),
  and an audio guide rides along. The two bugs we had recorded here (a
  keyframe mis-timed by the reference advance; the refs branch overwriting
  the keyframe `cond_video_latents` list) are both fixed: references now
  APPEND to the keyframe list in `model_base.py`, and `PackedLayout` places
  each keyframe at its resolved frame index. On v0.33.1 and earlier the
  slide still happens, so a graph that combines references with anchors
  should probe the running core rather than trust a version string. Since
  2026-08-18, [PR #15375](https://github.com/Comfy-Org/ComfyUI/pull/15375)
  also offers per-token video and audio latent masks (snapped to the 2x2
  latent patch grid and to whole audio latent frames, rounded to binary by
  the generic mask path), which is the better tool for PINNING existing
  content; guides remain the tool for creative anchors.
- **Audio onsets as beat anchors.** Align the retimed audio's impacts to the
  baseline's impact times and let video recovery inherit those anchor points.
  Attacks the same problem from the other modality.

## 5. Byproducts worth shipping

- **The slow-motion insert reel.** Exact recovery discards most of what the
  expensive pass generated, but those are genuine intermediate poses at up to
  4x temporal density, with pitch-preserved slowed audio available from the
  same machinery. Exporting each repaired burst as a slow-motion insert is
  nearly free and is production-useful on its own.
- **Quantization agreement as an artifact pre-filter.** Running the same seed
  through two quantizations of the same weights is a controlled perturbation:
  content that survives it is something the model committed to, content that
  moves is underdetermined. If that holds, it is an automatic way to
  pre-sort renders before a human looks, which is the real bottleneck in
  any comparison-heavy workflow. Speculative and cheap to test.

## 6. Validation we owe

- **VAE-only roundtrip probe.** Take real fast-motion footage, encode and
  decode at native speed versus slowed variants, and measure reconstruction
  error against local motion magnitude. This tests "temporal compression is
  the bottleneck" with no sampler and no DiT in the loop. Until it runs, the
  causality language in the README stays soft.
- **Cross-model transfer.** The largest external-validity risk is that
  everything here is fitted to one model's packing geometry. Porting the
  oracle, the smear and the recovery to a second temporally compressed video
  model is the experiment that upgrades this from a trick to a method, even
  as a partial result.
- **Blinded playback A/B.** Repair score and preservation score reported
  separately, because a method that removes the smear by removing the motion
  is not a fix.

## 7. An adjacent approach we are evaluating, not pursuing yet

Rather than dilating time so the compressed representation can hold the
motion, one could refuse the compression: generate every frame as its own
standalone image latent by sliding the model's native five-slot stencil
across world time and anchoring each pass on the previous pass's frames.
Four passes shifted by one frame cover 17 contiguous frames at full rate.

We think this is a genuinely interesting sibling to ATOS rather than a rival,
and the two would compose (use the cheap detector to decide where the
expensive representation is worth it). Two things to know before anyone
sinks time into it: priced in passes it sounds absurd, but priced in the
measured superlinear cost model it lands much closer to adaptive dilation
than the pass count suggests; and its likely failure mode is per-frame decode
incoherence, since decoding each frame independently discards the temporal
smoothing that overlapping receptive fields normally provide.

## 8. Explicitly not doing, with reasons

Kept here so nobody re-derives them:

- **RoPE time warping.** Warping per-token time spans was implemented and
  rejected: it produced boundary stutter, and the root cause was traced to
  chunked VAE decode on rate-warped content. Content-level slowdown on a
  uniform model clock is the approach that survived.

  **Reopened as a narrower question, 2026-08-13.** Two facts sharpen it.
  First, verified by executing `comfy/ldm/minimax/model.py:_video_t_grid`:
  H3's MM-RoPE time coordinate is already PHYSICAL, 5/3 RoPE units per
  world frame (token spans 5/3 x (1, 4, 4, 4, 4)), so the positional
  clock is a frame clock, not a token index. That means content dilation
  currently misreports time to the model: a dilated pass gets RoPE time
  proportional to dilated frames, and the model is generating what it
  believes is a longer clip rather than slow motion. Second, the original
  rejection was rooted in the decode path, not in RoPE itself, so it does
  not settle this. The narrow experiment now on the table: keep content
  dilation exactly as shipped, divide the RoPE t-grid by the local hold
  density so the time axis reports TRUE world time, decode normally
  (nothing rate-warped reaches the decoder), and A/B against the shipped
  pipeline for beat duplication, which is the failure this would most
  plausibly dissolve. Proposed, not run.
- **Automatic spatial masks.** Rejected in playback. Automatic
  oracle-heat-driven compositing degraded other artifacts even where it
  fixed background timing. Hand-authored masks are a different question and
  are still open.
- **Post-hoc frame interpolation.** It is a baseline to compare against, not
  a component. Interpolating a smeared frame cannot recover a pose the
  generator never created.
- **Feathering latent freeze masks.** Now defaulting to hard cells, on the
  argument that fractional edge cells denoise as half-frozen blends while the
  decode smooths a hard cell edge for free. Ratification A/B is on the alpha
  checklist and the default reverts if the ramp wins.

---

## How much work each of these is

Rough sizing against this codebase, for anyone deciding where to jump in.
The architecture helps more than it looks: `_jerk_profile` is a dozen lines
of numpy and `_compile_hold_map` is a shared seam, so **any new detector is a
drop-in** and most detection work is hours rather than days. The expensive
items are the ones that change the time REPRESENTATION or reach into ComfyUI
internals.

**Already shipped on this branch (alpha, defaults unchanged):** the
abstention gate (`abstain_below`) and alternative detectors (`profile_mode`:
value |d3|, value |d1| baseline, trajectory centroid) on H3 Jerk Oracle. The
default path is bit-identical to what shipped before.

| item | size | why |
|---|---|---|
| delta-order ablation | done | now a dropdown on the oracle |
| absolute abstention gate | done | contrast floor, off by default |
| camera-compensated profile | hours | another `_jerk_profile` mode, same seam |
| audio-onset profile as the cheap baseline | ~half day | solved DSP, emits the same profile shape |
| slow-motion insert export | ~half day | the recover node already selects frames; emit the discarded ones |
| paired-speed reachability | ~1 day | two renders plus offline latent diff, no new node |
| knee-finding over dilation factors | ~1 day | orchestration around the existing probe schedule |
| quantization agreement pre-filter | ~1 day | two renders per clip plus a divergence map |
| mirror symmetry detector | ~1 day | needs real mirror footage; synthetics pass trivially |
| spatial crop regeneration | ~2 days | mirrors the existing time crop/splice pair; seam risk |
| fixed-budget bidirectional retiming | ~2 days | holds below 1 change the hold-map representation AND recovery semantics |
| sparse pinned time / anchor ladder | ~2-3 days | needs the packed-layout temporal patch or upstream #15439, plus maintenance risk against upstream churn |
| stabilize, repair, unstabilize | ~2-3 days | a whole preprocessing stage with warp and inverse warp |
| VAE roundtrip probe | ~1 day | a script, not a node |
| temporal capacity profiling | ~2-3 days | synthetic content generation plus a sweep |
| cross-model transfer | large | a port, not a feature |

## What would actually help

If you are running this pack, the most useful things you can send back are
same-seed before and after pairs on content that breaks it, a note on which
dial you reached for, and your seconds-per-step so cost estimates can be
calibrated across hardware. Failure cases on content unlike ours are worth
more to us than successes on content like ours.