anime cooking clipi

same script but photo realism

Both halves of this report are in cortiq 0.5.80 (out now: cargo install cortiq-cli, or the release binaries):

• The mm_kill hack is no longer needed. The contention kill is disarmed for the whole prompt-encode phase and arms only when the denoise loop starts — the encoder is a one-shot pass over 12 GB, and on a 24 GB box it streams from disk for reasons that are not contention. Three consecutive strikes are still required after that. CMF_MM_KILL=0 remains as the big switch.
• P-cores for the keyframe encode. Your asitop (E-cores at 100%, P-cores asleep during the --first-frame VAE encode) was the pool's worker threads landing wherever the scheduler put them; they now ask for QOS_CLASS_USER_INITIATED. That shortens the ~100–140 s encode; a device path for the encoder's 3-D convolutions is the real fix and is on the list.
• Your frame budgets are in the model card (Field notes): 25.7 GB file — 512×256 safe to ~70–90 frames, 768×448 caps at 40–50; v2 14.5 GB file — 448×768 × 50 frames at 136 s/step, 90 frames is the paging edge. And the v2 face-consistency confirmation, in your words.

Thank you — please break it again.

Cortiq v0.5.82 Field Report: I2V Pipeline & Memory Stress Testing

Hardware: Apple Mac Mini (M4, 24GB Unified Memory)
Model: mmh3-turbo-clipproj4b-fl2va-v2-q4tp.cmf (14.5 GB)
Resolution: 768x448

1. Validating the v0.5.82 Upgrades

The latest adjustments to core scheduling and contention detection are working exactly as intended. We ran several Image-to-Video (--first-frame) benchmarks to validate the pipeline:

• P-Core Wakeup (QOS_CLASS_USER_INITIATED): Successfully verified. During the massive 12GB prompt-encode phase, macOS properly routed the workload. asitop readings showed P-cores pinned at 100% @ ~3900 MHz, while E-cores hovered around 70-96%.
• mm_kill Disarmament: The engine no longer throws false-positive contention kills during the heavy initial VAE encode. The 164–174 second image encode phase completes smoothly, allowing the engine to successfully transition into the denoise loop.
• Performance Parity: For a 39-frame I2V generation, the overall denoise speed stabilized at ~18.0 seconds/step. This perfectly matches our historical averages for standard Text-to-Video runs, meaning the engine is absorbing the heavy I2V encode overhead without degrading overall batch performance.

2. Pushing the "Paging Edge" (How We Break It)

We specifically set out to test your hypothesis regarding the memory paging edge. While you estimated the edge to be around 90 frames for the 14.5 GB v2 model, our telemetry suggests the true cliff for a 24GB machine is much lower—likely right around 50 frames.

The "Nuke Test" (120 Frames)
To completely break the engine, we launched a 120-frame I2V generation. Here is the lifecycle of the system collapse:

1. Encode Phase: Survived. RAM sat comfortably at ~5.7GB.
2. Transition to Denoise: As the engine allocated the massive 120-frame latent tensor, the system instantly flatlined into swap death.
3. Telemetry at Collapse:
   • Physical RAM: 22.0 / 24.0 GB
   • Swap: 4.0 GB used / 5.0 GB capacity (macOS forcefully doubled the swap capacity in real-time to prevent a kernel panic).
   • GPU Usage: Plummeted from ~77% down to 53% (and falling) as the cores starved waiting for data to be dragged across the swap boundary.

We manually bailed out (SIGINT) right as denoise 1/4 began to spare the SSD, so we didn't wait to see if the 3-strike mm_kill would formally execute the process. However, the system was undeniably thrashing.

Conclusion:
The 0.5.82 engine is remarkably stable and fast. 40-frame runs are the undisputed "sweet spot" for 24GB hardware (~20.1GB RAM used, 0.9GB swap). Pushing beyond 50 frames invites steep swap penalties, and 120 frames is a guaranteed system stall.

Thank you for the excellent update!

"Claire's Day"

Your 0.5.82 report is in the model card now, including the correction: I had estimated the paging edge at ~90 frames for the 14.5 GB file and you measured ~50 — the card says 50. The I2V number (18.0 s/step at 39 frames, 768×448, parity with T2V) is there too, and so is the collapse profile at 120 frames.

0.5.92 adds runtime adapters (#8) — worth a try on the faces: fal's Realism adapter is attention-only, so all 104 of its branches bind, and on my M4 it costs about a tenth of a step.

Follow-up with a cleaner measurement, because the first one was taken on a machine that was swapping.

With memory freed and the two runs back to back — 512×288, 22 frames, four steps, M4 24 GB:

1.06× on the denoise, and the VAE moved 6.5% between the same two runs — so on this machine an adapter costs at or below the run-to-run noise. My earlier 1.13× and 1.37× came from the base render itself drifting from 22.2 to 32.5 s a step under 6 GB of swap.

Where weights stream off disk, the adapter's extra work hides inside the streaming. Where the DiT is fully resident, the attention fusion it stands down should show up — that number is still owed, and I will post it when a card with headroom runs it.

Since this thread is where the Mac numbers live, the honest note for it: 0.5.96 does nothing for you.

It is a 10% win on a four-bit denoise (RTX 5090, 768×448: 69.6 s → 62.4 s, output byte-identical) and it is entirely on the wgpu/Vulkan path. Your M4 runs the Metal one.

What is worth carrying over from it is the diagnosis, because it was not where anyone was looking — including me. Profiling one fused FFN call on the four-bit path:

DiT shape 5376x14336, b=2447 : host 126.2 ms | card 41.4 ms
VAE shape 2048x8192,  b=1797 : host 100.2 ms | card 12.6 ms

Three quarters of the call was host work — buffer setup, bind groups, and a scan over the activation panel for max|x| on one core — before the GPU was asked for anything. On the pod's 27-core box that was still 10% of the render. On a laptop-class CPU sharing memory with the GPU it is likely worse, not better.

So the Metal side now has a specific thing to look for rather than "make the kernel faster", and I would rather tell you that than quietly ship a release note you cannot use.

Two more things from this evening that touch your machine directly:

• The 24 GB frame budget. Chunked rendering (--stream-chunk 13, container mmh3-raven-streaming-q4tp.cmf) stops the activation cost growing with clip length. On the 5090 a 321-frame clip did not render at all bidirectionally — a single 2 GiB buffer binding — and took 354 s chunked. That limit is the same on your machine; the fix is in 0.5.95.
• The eight-bit file (q8_2f) went from 171.5 s to 43 s at 512×288 in 0.5.95. At 768×448 it is now faster than the four-bit one (69 s against 131 s) with the more faithful codec. If you want the next quality step on the M1/64 GB, that is the file to try — though the same caveat applies, those numbers are the Vulkan path.

Correction to what I posted an hour ago, and this time with a Mac in front of me.

I said the host-preparation cost that dominates the Vulkan path was "likely worse, not better" on a laptop-class CPU sharing memory with the GPU. I have now measured it on an M4 with mmh3-turbo-clipproj4b-fl2va-v2-q4tp.cmf — 384×256, 9 frames, four steps — and that guess was wrong:

q4tp mm x1091: upload 0.7 s · submit+wait 51.1 s · readback 1.3 s
stages: text encode 0.7s · denoise 52.1s · video vae 12.4s

Two seconds of host work out of 53. On Metal the copies are already nothing — unified memory does what it says — and effectively all of it is the kernel. The opposite of the Vulkan finding, and worth knowing before anyone spends a week porting today's fix to Metal: there is nothing there to win.

The harder number is what the kernel is doing. One four-step render at that size is about 210 TFLOP of arithmetic (50 blocks × 1.05 TFLOP per block at ~1400 tokens). Against 51.1 s that is **4.1 TFLOPS sustained**, and the kernels already use Apple's simdgroup_matrix. An M4's GPU is in that neighbourhood. So the render is not waiting on bad code — it is doing the arithmetic the model asks for, at roughly the speed the machine can.

Which changes the advice I would give you for the 24 GB machine, and I would rather be useful than encouraging:

1. Do less arithmetic, do not hope for a faster kernel. The levers that actually move it are the ones that shrink the token count: render small and --upscale (the latent upscaler decodes at the larger size without a second pass through the 5 B VAE), and --stream-chunk so the cost grows with the clip rather than with its square.
2. The 45 minutes has a shape now. It is ~210 TFLOP per four-step pass at a small size, and it scales with tokens — double the frames, double the work, plus attention's own growth. That is why your frame budget feels like a cliff.
3. The M1/64 GB is not faster per token, it is only bigger. Use it for the parts that need room (long clips, the VAE decode), not as a way to make one clip finish sooner.

I will keep the wgpu work where it belongs — it helps the NVIDIA users in #1 — and stop implying it will reach your machine.