| # Roadmap: methods we are investigating |
|
|
| > Status ledger for what has shipped vs what is open: |
| > [`docs/roadmaps/ATOS_ALPHA_STATUS.md`](docs/roadmaps/ATOS_ALPHA_STATUS.md). |
| > The persistent-world design and its status live there too. |
|
|
| Companion to `RESEARCH_NOTES_ATOS.md` (the claim and the evidence plan), |
| `TUNING.md` (symptom to dial, including the rejections) and |
| `TESTING_ALPHA.md` (what the alpha nodes still need from human hands). |
|
|
| Everything below is a direction, not a promise. Items are grouped by the |
| problem they attack, and each says what would count as success and what the |
| cheapest first measurement is. Where we have already measured something that |
| contradicts an earlier claim of ours, it is written down as such. |
|
|
| House rules that constrain every item here: motion is judged in playback |
| because stills lie in both directions; a measurement earns its compute only |
| if it beats the cheap baseline it replaces; and anything shipped as a default |
| has to survive a same-seed A/B. |
|
|
| --- |
|
|
| ## 1. The oracle: from allocator to meter |
|
|
| **The problem, measured and published in the research notes:** the jerk |
| oracle ranks but cannot abstain. Because the threshold is a quantile of the |
| profile, a clip with exactly zero trajectory jerk still received 3.06x |
| dilation. And in the value domain the third difference is nearly the same |
| signal as the first (corr 0.960 and 0.977 on real clips), so the score is |
| contaminated by motion energy. This is the single most important open |
| problem in the method. |
|
|
| Directions, roughly in order of expected value per hour: |
|
|
| - **Paired-speed reachability.** Generate the same choreography at two |
| content speeds and diff the latents. Tokens whose state actually changes |
| between the two are the tokens that motion speed reaches; tokens that look |
| hot in both are hot for some other reason, which is exactly the |
| textured-object-passing-a-location confound. Borrowed from the practice of |
| probing a system with paired semantic perturbations rather than noise. |
| Success: a per-token relevance score that separates the two synthetic |
| clips (zero jerk versus lurching) where the quantile does not. Cost: two |
| renders plus offline analysis of saved latents. |
| - **Demand over capacity as an absolute unit.** A token spans four frames; |
| the failure is that those four frames need more distinct poses than one |
| token can hold. So the natural absolute quantity is required pose change |
| divided by representable pose change, with the hold count falling out as |
| that ratio rounded up. Calibrated once against the speed at which smear |
| actually appears, instead of fixed by a quantile forever. Success: the |
| meter says "dilate nothing" on a smooth clip and "dilate everything" on a |
| uniformly violent one, both of which the quantile gets wrong today. |
| - **Trajectory-domain measurement.** Showing promise on real clips |
| (top-decile share 0.29 versus 0.22 for velocity, with only moderate |
| correlation between them). Track a region centroid and differentiate the |
| trajectory rather than the latent values. Now selectable on the oracle as |
| `profile_mode`. **Caveat measured while shipping it:** on a smooth |
| synthetic blob the centroid path is nearly noise-free, so its peak-to-mean |
| contrast is not comparable to the value domain's there (1.21x versus 1.88x |
| separation between a constant-velocity and a lurching toy). Treat it as an |
| ablation option, not as an established improvement, until it is measured on |
| real textured content. |
| - **Knee-finding instead of a fixed quantile.** Run the cheap early-x0 probe |
| at several dilation factors and take the knee of the curve per clip. |
| Unlike a quantile, a knee can land at 1.0x, which is the abstention we |
| currently cannot express. |
| - **Absolute calibration by temporal capacity profiling.** Feed the VAE |
| synthetic controlled-speed content (moving gratings, rotating props, speed |
| sweeps) and measure reconstruction fidelity against speed to get the |
| model's temporal transfer curve. Useful beyond this repo: it would be a |
| reusable way to compare base models on temporal capacity. |
|
|
| **Ablation owed regardless:** delta order 1 versus 2 versus 3, phase |
| normalization on and off, against external baselines (optical flow magnitude |
| and flow acceleration on decoded frames). If the third difference wins or |
| ties while being free, that is the result. If it loses, the honest move is to |
| say so and rename the thing. |
|
|
| ## 2. Detection quality |
|
|
| - **Camera-compensated jerk.** Subtract the dominant global motion component |
| per token before thresholding, to stop pans inflating the dilated spans. |
| Progression: global jerk, then camera-compensated, then spatially |
| localized overload. |
| - **Stabilize, repair, unstabilize.** The alternative to teaching the oracle |
| to ignore pans: remove the pan first with a coarse global warp, run the |
| pipeline where subject motion is the only motion, then invert the warp. |
| A classic VFX move that turns the hardest confound into preprocessing. |
| Worth comparing directly against camera compensation. |
| - **Cross-modal detection.** The model co-generates audio, so audio onset |
| density is a second, independent account of where the action beats are. |
| Where video-hot and audio-quiet disagree is a good candidate signature for |
| camera-pan inflation. This also gives the oracle a **free baseline it must |
| beat**: a latent detector that cannot outperform onset detection on |
| co-generated audio has not earned its complexity. |
| - **Mirror and reflection double counting.** Published correction in TUNING: |
| a synchronized reflection leaves the hold map bit-identical, and the real |
| amplifier is `bridge` welding a temporally separated reflection to the |
| body's burst. A real detector needs approximate reflection symmetry of the |
| spatial jerk map, not temporal correlation, which we probed and found |
| cannot work (one body's own limbs correlate at ~1.0, so the confuser floor |
| sits above the signal). Untested on real mirror footage. |
|
|
| ## 3. Allocation and cost |
|
|
| - **Fixed-budget bidirectional retiming.** Today the method BUYS temporal |
| capacity for bursts. The general object is a monotone retiming function |
| whose two signs are dilation and decimation: compress the quiet spans and |
| spend exactly what you saved on the bursts, for a constant token budget. |
| "Repair at zero marginal cost" is a much stronger claim than "3.5x for |
| quality." Open risk: decimated frames have no generated source, so |
| recovery there is nearest-held-frame and may judder in quiet spans. |
| Playback decides. |
| - **Honest cost accounting, already shipped.** Per-step time is superlinear |
| in token count (measured exponent 1.75 here, ~1.64 in a field report, |
| `COST_EXP 1.7` in the code). Reports now state time multipliers rather than |
| frame ratios. The practical consequence is counter-intuitive and worth |
| repeating: many small windows can be cheaper than one large one, so |
| pass-count is the wrong unit for comparing any of these approaches. |
| - **Rolling-window regeneration, for cards that OOM on the dilated pass.** |
| What runs people out of memory is not the clip, it is the dilated span: a |
| 5 s action clip regenerates as 11 to 13 s of frame data, so peak VRAM |
| tracks the burst budget rather than the runtime. Field reports put the |
| wall around 1 MP past 10 s on a 3090, 128 GB of system RAM |
| notwithstanding. Sliding a fixed-size window across world time, |
| regenerating each window and splicing, turns peak memory into a knob |
| instead of a consequence: the user sets the window, and clip length stops |
| mattering. `H3 Segment Crop` and `H3 Segment Splice` already do exactly |
| this for one window, with context handles and video plus sample-accurate |
| audio crossfades, so what is missing is orchestration and a seam policy |
| across N windows, not new machinery. A community continuation graph |
| (`seamless_extension_derope.json`, 2026-08-12) already wraps the de-rope |
| chain in a segment loop for a different purpose and is worth reading |
| before designing ours. |
| It is also not the slow option, though by less than the bare exponent |
| suggests. Naive arithmetic says N windows cost N^-0.75 of the single-window |
| time, four windows for about a third. Fitting 266 corpus runs and then |
| charging for duplicated handle frames and grid padding gives **0.69x at 300 |
| dilated frames, 0.45x at 600 and 0.37x at 1000**: real, and roughly half |
| the advertised saving. Handle duplication, not per-window overhead, is what |
| eats it, and it turns back into a loss somewhere around 8 to 12 windows. |
| The same fit found the step-independent cost is not constant either, it |
| scales at t^1.66 against the step's t^1.70, so it splits along with |
| everything else. |
| Two things are already true that this item assumed were missing. Chaining |
| `H3 Segment Splice` so each output becomes the next one's baseline |
| reconstructs the world to 6e-8 today, with no new code, so N-window |
| reassembly is not the gap. And core's graph-expansion API means owning the |
| iteration does not require a fat node that swallows the sampler. |
| The real difficulty is the seam, and specifically the HOT cut, a window |
| boundary landing inside a burst, which is exactly the case the feature is |
| requested for. Handle frames are forced to hold 1, so a crossfade there |
| blends repaired against un-repaired content at peak motion, and the audio |
| problem is worse: two independent foley takes of one impact, which no |
| crossfade width fixes. Cold cuts, where the boundary sits at hold 1, are |
| just the shipped single-window path N times and carry no new risk. |
| Success: peak VRAM flat in clip length, seams invisible in playback. Note |
| we cannot currently measure the first half of that, `peak_vram_mb` in our |
| own telemetry is flat at 55-62 GB from 5 to 294 frames. Cheapest first |
| measurement: run the shipped crop/splice pair over two adjacent windows of |
| one clip and look at the join. That needs no new code at all. |
| Worth saying plainly: two cheaper answers to "I OOM" already ship, lowering |
| `d_max` or raising `q`, and `H3 Segment Crop` as it stands. Rolling windows |
| only win for someone insisting on full `d_max` on a burst larger than their |
| card, and nobody has produced that clip yet. |
| - **Extension: make a long clip from short renders (shipped alpha 2026-08-23).** |
| The other long-video job: not one long world chunked, but a chain of |
| short worlds with a carried handle. `examples/motion_pipeline_extend_api.json` |
| and `ALPHA.md`. What the first cells taught, in order: protecting the |
| handle from RETIMING is not enough, pass 2 still re-denoises it (20/255 |
| drift) until it is also frozen (3.3); the REPRESENTATION beats the |
| prompt - the tail written into the next segment's own latent under a |
| per-token mask makes the join an ordinary frame transition (0.86x to |
| 1.1x vs 5x to 6.5x for the image guide) and keeps the camera's velocity |
| continuous, while an explicit camera-transition sentence changed nothing; |
| each VAE round trip darkens the frozen prefix ~2.4% (per-channel gains |
| fitted on the hidden overlap fix it, an audio rms gain measured |
| negative); and a burst that runs into the cut comes back fast unless the |
| last token group of a non-final segment is protected (1.55x -> 1.06x). |
| Open: the drift curve over many segments, 192/90 vs 141/39, inject on |
| continuation segments, dialogue across a join, the audio ambience bed |
| (wants a per-tick audio mask). |
| - **A ceiling on dilated frames** (field request, 2026-08-12). A max-length |
| option that drops lower-priority frames to keep people off the OOM rail |
| while the highest-priority ones still reach pass 2. In our terms a hard |
| cap on the dilated budget, filled greedily from the top of the jerk |
| profile, so the pass fits the card and the hottest spans keep their holds. |
| Small change at the `_compile_hold_map` seam. The blocker sits upstream of |
| the cap: at today's false-positive rate a budget can be spent entirely on |
| a camera pan, so this wants the detector work in section 2 first, or it |
| will confidently protect the wrong frames. |
| - **Spatial crop regeneration.** Time targeting and segment crop reduce |
| FLOPs; spatial masks do not, they only blend. A spatial crop analog of |
| segment crop would make "foveated" literal. Risks are context loss and |
| resolution mismatch at the seam. |
|
|
| ## 4. Beat duplication and anchoring |
|
|
| The model fills dilated time with extra beats instead of slowing the existing |
| ones (two backflips become four). This is the most visible remaining artifact |
| and it has a clear line of attack. |
|
|
| - **Sparse pinned time.** We already have the choreography: the baseline. |
| Sampling sparse pose anchors from it and pinning them at their retimed |
| positions gives the regeneration a ladder to climb instead of room to |
| improvise. Success: beat count stops scaling with dilation factor. |
| - **Dependency: resolved upstream (checked against core 2026-08-20).** |
| When this item was written, stock ComfyUI restricted H3 keyframe anchors |
| to first and last, and two community packs rewrote the temporal column |
| after packing to get around it. Upstream |
| [PR #15439](https://github.com/Comfy-Org/ComfyUI/pull/15439) (merged |
| commit `e01fb4c5`) made arbitrary guide positions native: |
| `MiniMaxH3AddGuide` takes a still or a clip at any frame index |
| (negative counts from the end), a clip guide snaps DOWN the 17k+5 grid |
| (5, 22, 39, ...; fewer than five frames becomes a single-frame guide), |
| and an audio guide rides along. The two bugs we had recorded here (a |
| keyframe mis-timed by the reference advance; the refs branch overwriting |
| the keyframe `cond_video_latents` list) are both fixed: references now |
| APPEND to the keyframe list in `model_base.py`, and `PackedLayout` places |
| each keyframe at its resolved frame index. On v0.33.1 and earlier the |
| slide still happens, so a graph that combines references with anchors |
| should probe the running core rather than trust a version string. Since |
| 2026-08-18, [PR #15375](https://github.com/Comfy-Org/ComfyUI/pull/15375) |
| also offers per-token video and audio latent masks (snapped to the 2x2 |
| latent patch grid and to whole audio latent frames, rounded to binary by |
| the generic mask path), which is the better tool for PINNING existing |
| content; guides remain the tool for creative anchors. |
| - **Audio onsets as beat anchors.** Align the retimed audio's impacts to the |
| baseline's impact times and let video recovery inherit those anchor points. |
| Attacks the same problem from the other modality. |
|
|
| ## 5. Byproducts worth shipping |
|
|
| - **The slow-motion insert reel.** Exact recovery discards most of what the |
| expensive pass generated, but those are genuine intermediate poses at up to |
| 4x temporal density, with pitch-preserved slowed audio available from the |
| same machinery. Exporting each repaired burst as a slow-motion insert is |
| nearly free and is production-useful on its own. |
| - **Quantization agreement as an artifact pre-filter.** Running the same seed |
| through two quantizations of the same weights is a controlled perturbation: |
| content that survives it is something the model committed to, content that |
| moves is underdetermined. If that holds, it is an automatic way to |
| pre-sort renders before a human looks, which is the real bottleneck in |
| any comparison-heavy workflow. Speculative and cheap to test. |
|
|
| ## 6. Validation we owe |
|
|
| - **VAE-only roundtrip probe.** Take real fast-motion footage, encode and |
| decode at native speed versus slowed variants, and measure reconstruction |
| error against local motion magnitude. This tests "temporal compression is |
| the bottleneck" with no sampler and no DiT in the loop. Until it runs, the |
| causality language in the README stays soft. |
| - **Cross-model transfer.** The largest external-validity risk is that |
| everything here is fitted to one model's packing geometry. Porting the |
| oracle, the smear and the recovery to a second temporally compressed video |
| model is the experiment that upgrades this from a trick to a method, even |
| as a partial result. |
| - **Blinded playback A/B.** Repair score and preservation score reported |
| separately, because a method that removes the smear by removing the motion |
| is not a fix. |
|
|
| ## 7. An adjacent approach we are evaluating, not pursuing yet |
|
|
| Rather than dilating time so the compressed representation can hold the |
| motion, one could refuse the compression: generate every frame as its own |
| standalone image latent by sliding the model's native five-slot stencil |
| across world time and anchoring each pass on the previous pass's frames. |
| Four passes shifted by one frame cover 17 contiguous frames at full rate. |
|
|
| We think this is a genuinely interesting sibling to ATOS rather than a rival, |
| and the two would compose (use the cheap detector to decide where the |
| expensive representation is worth it). Two things to know before anyone |
| sinks time into it: priced in passes it sounds absurd, but priced in the |
| measured superlinear cost model it lands much closer to adaptive dilation |
| than the pass count suggests; and its likely failure mode is per-frame decode |
| incoherence, since decoding each frame independently discards the temporal |
| smoothing that overlapping receptive fields normally provide. |
|
|
| ## 8. Explicitly not doing, with reasons |
|
|
| Kept here so nobody re-derives them: |
|
|
| - **RoPE time warping.** Warping per-token time spans was implemented and |
| rejected: it produced boundary stutter, and the root cause was traced to |
| chunked VAE decode on rate-warped content. Content-level slowdown on a |
| uniform model clock is the approach that survived. |
|
|
| **Reopened as a narrower question, 2026-08-13.** Two facts sharpen it. |
| First, verified by executing `comfy/ldm/minimax/model.py:_video_t_grid`: |
| H3's MM-RoPE time coordinate is already PHYSICAL, 5/3 RoPE units per |
| world frame (token spans 5/3 x (1, 4, 4, 4, 4)), so the positional |
| clock is a frame clock, not a token index. That means content dilation |
| currently misreports time to the model: a dilated pass gets RoPE time |
| proportional to dilated frames, and the model is generating what it |
| believes is a longer clip rather than slow motion. Second, the original |
| rejection was rooted in the decode path, not in RoPE itself, so it does |
| not settle this. The narrow experiment now on the table: keep content |
| dilation exactly as shipped, divide the RoPE t-grid by the local hold |
| density so the time axis reports TRUE world time, decode normally |
| (nothing rate-warped reaches the decoder), and A/B against the shipped |
| pipeline for beat duplication, which is the failure this would most |
| plausibly dissolve. Proposed, not run. |
| - **Automatic spatial masks.** Rejected in playback. Automatic |
| oracle-heat-driven compositing degraded other artifacts even where it |
| fixed background timing. Hand-authored masks are a different question and |
| are still open. |
| - **Post-hoc frame interpolation.** It is a baseline to compare against, not |
| a component. Interpolating a smeared frame cannot recover a pose the |
| generator never created. |
| - **Feathering latent freeze masks.** Now defaulting to hard cells, on the |
| argument that fractional edge cells denoise as half-frozen blends while the |
| decode smooths a hard cell edge for free. Ratification A/B is on the alpha |
| checklist and the default reverts if the ramp wins. |
|
|
| --- |
|
|
| ## How much work each of these is |
|
|
| Rough sizing against this codebase, for anyone deciding where to jump in. |
| The architecture helps more than it looks: `_jerk_profile` is a dozen lines |
| of numpy and `_compile_hold_map` is a shared seam, so **any new detector is a |
| drop-in** and most detection work is hours rather than days. The expensive |
| items are the ones that change the time REPRESENTATION or reach into ComfyUI |
| internals. |
|
|
| **Already shipped on this branch (alpha, defaults unchanged):** the |
| abstention gate (`abstain_below`) and alternative detectors (`profile_mode`: |
| value |d3|, value |d1| baseline, trajectory centroid) on H3 Jerk Oracle. The |
| default path is bit-identical to what shipped before. |
|
|
| | item | size | why | |
| |---|---|---| |
| | delta-order ablation | done | now a dropdown on the oracle | |
| | absolute abstention gate | done | contrast floor, off by default | |
| | camera-compensated profile | hours | another `_jerk_profile` mode, same seam | |
| | audio-onset profile as the cheap baseline | ~half day | solved DSP, emits the same profile shape | |
| | slow-motion insert export | ~half day | the recover node already selects frames; emit the discarded ones | |
| | paired-speed reachability | ~1 day | two renders plus offline latent diff, no new node | |
| | knee-finding over dilation factors | ~1 day | orchestration around the existing probe schedule | |
| | quantization agreement pre-filter | ~1 day | two renders per clip plus a divergence map | |
| | mirror symmetry detector | ~1 day | needs real mirror footage; synthetics pass trivially | |
| | spatial crop regeneration | ~2 days | mirrors the existing time crop/splice pair; seam risk | |
| | fixed-budget bidirectional retiming | ~2 days | holds below 1 change the hold-map representation AND recovery semantics | |
| | sparse pinned time / anchor ladder | ~2-3 days | needs the packed-layout temporal patch or upstream #15439, plus maintenance risk against upstream churn | |
| | stabilize, repair, unstabilize | ~2-3 days | a whole preprocessing stage with warp and inverse warp | |
| | VAE roundtrip probe | ~1 day | a script, not a node | |
| | temporal capacity profiling | ~2-3 days | synthetic content generation plus a sweep | |
| | cross-model transfer | large | a port, not a feature | |
|
|
| ## What would actually help |
|
|
| If you are running this pack, the most useful things you can send back are |
| same-seed before and after pairs on content that breaks it, a note on which |
| dial you reached for, and your seconds-per-step so cost estimates can be |
| calibrated across hardware. Failure cases on content unlike ours are worth |
| more to us than successes on content like ours. |
|
|
