One 32 GB card (RTX 5090) for the agent; fix the hf download commands; which setup for which GPU

COMFYUI.md

@@ -13,6 +13,7 @@ starting noise for a seed. Everything else is stock ComfyUI. Tested with ComfyUI
 
 ```bash
 cd ComfyUI
+pip install -U "huggingface_hub>=1.0"   # the hf CLI; older versions keep only the last --include
 
 # the base model, as Comfy-Org's single-file repack (about 124 GB)
 hf download Comfy-Org/MiniMax-H3 \
@@ -23,7 +24,7 @@ hf download Comfy-Org/MiniMax-H3 \
   --local-dir models
 
 # the LoRAs, the nodes, the workflows and the example inputs from this repository
-hf download Viggle/Viggle-Remake --include "comfyui/*" "examples/inputs/*" --local-dir ./Viggle-Remake
+hf download Viggle/Viggle-Remake --include "comfyui/*" --include "examples/inputs/*" --local-dir ./Viggle-Remake
 cp -r Viggle-Remake/comfyui/ComfyUI-Viggle-Remake custom_nodes/
 cp Viggle-Remake/comfyui/viggle_remake_*_comfyui.safetensors models/loras/
 ```
@@ -205,7 +206,8 @@ We measured on B200s. A second process held the rest of the card, so ComfyUI saw
 - **Why GGUF needs the flag.** ComfyUI-GGUF puts as much of the model on the card as ComfyUI's memory estimate allows, and for MiniMax-H3 that estimate leaves out the reference and keyframe tokens.
   - When the weights fit but the activations do not, ComfyUI aborts with `CUDA error: invalid argument` instead of falling back to offloading.
   - `--reserve-vram N` keeps N GB free for the activations. If a run aborts, raise N.
-- **Why int8 stops at 1280×736 × 226 frames.** Without a CUDA 13 build of PyTorch, comfy-kitchen falls back to a plain int8 matmul. That matmul keeps 32-bit sums for the whole layer and assembles the output in pieces, so one layer briefly needs several times the memory of its output. At 1280×736 × 226 frames this ran out of memory even with 80 GB free, while bf16 + LoRA fit in 80 GB and Q8_0 ran in 24 GB with `--reserve-vram 20`. We have not tested a CUDA 13 build.
+- **Why int8 stops at 1280×736 × 226 frames.** Without a CUDA 13 build of PyTorch, comfy-kitchen falls back to a plain int8 matmul. That matmul keeps 32-bit sums for the whole layer and assembles the output in pieces, so one layer briefly needs several times the memory of its output. At 1280×736 × 226 frames this ran out of memory even with 80 GB free, while bf16 + LoRA fit in 80 GB and Q8_0 ran in 24 GB with `--reserve-vram 20`. We have not tested the graphs above with a CUDA 13 build, but see the next point.
+- **A CUDA 13 build on a 32 GB card.** With torch 2.9.1+cu130 on an RTX 5090, int8 renders 1280×736 at up to 345 frames (about 260 s per step) through the MCP's ComfyUI worker, [`mcp/comfy_worker/`](mcp/comfy_worker), which also runs each block's feed-forward in row chunks and starts ComfyUI with `--vram-headroom 4`.
 - **Q4_K_M is the smallest file.** It is visibly further from the reference than int8 (see Quality).
 
 ### Quality

README.md

@@ -53,15 +53,28 @@ swaps, removes and inserts objects, and changes the setting, the weather or the
 
 > **Just want to use it?** The [Viggle app](https://viggle.ai/meme/app/home?ui=v2) runs Remake for you. The rest of this section is for running it on your own GPUs.
 
+**Which setup for your machine:**
+
+| your GPU | what to run |
+|---|---|
+| none | the [Viggle app](https://viggle.ai/meme/app/home?ui=v2) |
+| one 12–24 GB card | ComfyUI with an int8 or GGUF file, workflows by hand (no agent): [Smaller GPUs](COMFYUI.md#smaller-gpus-int8-and-gguf) |
+| one 32 GB card (RTX 5090) | the agent with the ComfyUI worker, 1280×736 up to 345 frames, with about 60 GB of RAM: [One 32 GB card](mcp/README.md#one-32-gb-card-the-comfyui-worker) |
+| 80 GB | `sample.py` below, or ComfyUI with bf16 + LoRA: [ComfyUI](COMFYUI.md#comfyui) |
+| 96 GB or more | the same, or the [agent](#agent-the-mcp-server) with `viggle-remake-worker` |
+
 You need the base model (about 144 GB of its files) and this repository's LoRAs and sampler:
 
 ```bash
+pip install -U "huggingface_hub>=1.0"   # the hf CLI; older versions keep only the last --include
 hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
-  --include "modular_model_index.json" "transformer/*" "text_encoder/*" "tokenizer/*" \
-            "processor/*" "vae/*" "audio_vae/*" "scheduler/*" "audio_scheduler/*" "docs/*"
+  --include "modular_model_index.json" --include "transformer/*" --include "text_encoder/*" \
+  --include "tokenizer/*" --include "processor/*" --include "vae/*" --include "audio_vae/*" \
+  --include "scheduler/*" --include "audio_scheduler/*" --include "docs/*"
 
 hf download Viggle/Viggle-Remake --local-dir ./Viggle-Remake \
-  --include "lora/*" "inference/*" "examples/*" "requirements.txt"   # 5.5 GB; the whole repo is ~210 GB
+  --include "lora/*" --include "inference/*" --include "examples/*" \
+  --include "requirements.txt"   # 5.5 GB; the whole repo is ~210 GB
 pip install -r Viggle-Remake/requirements.txt   # plus an ffmpeg binary on PATH
 
 python Viggle-Remake/inference/sample.py --mode swap \
@@ -83,6 +96,11 @@ the LoRA did not load.
 
 Node pack, workflows, and int8/GGUF files down to 12 GB: [`COMFYUI.md`](COMFYUI.md).
 
+To run the agent below on one 32 GB card such as an RTX 5090, use the ComfyUI render worker in
+[`mcp/comfy_worker/`](mcp/comfy_worker) instead of `viggle-remake-worker`: a 1280×736 window of 345 frames renders in
+about 28 minutes, and White House Funk runs end to end in about 3 hours. See
+[One 32 GB card](mcp/README.md#one-32-gb-card-the-comfyui-worker).
+
 ## Agent: the MCP server
 
 [`mcp/`](mcp) lets a coding agent (Claude Code, codex) remake a whole film: it cuts the windows, paints

comfyui/ComfyUI-Viggle-Remake/nodes.py

@@ -9,6 +9,7 @@ RemakeConditioning   the source clip as <Video 1>, each image as <Picture N> and
                      clean target audio for the whole denoise
 RemakeKeyframeUntil  drops the keyframes once sigma falls below `keyframe_until` (0.9 = the last two of six forwards)
 RemakeNoise          draws the starting noise in the reference script's order, so a seed gives the same video
+RemakePrefix         a hop: the previous take's frames open this render, held clean like the soundtrack
 
 Load the LoRA with the stock LoraLoaderBypassModelOnly. LoraLoaderModelOnly merges it into the bf16 weights,
 where rounding drops part of the update and the output comes out softer.
@@ -211,8 +212,48 @@ class RemakeNoise:
         return (Noise(seed, skip),)
 
 
+class RemakePrefix:
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {"required": {
+            "vae": ("VAE",),
+            "latent": ("LATENT",),
+            "prefix": ("IMAGE", {"tooltip": "The previous take's frames, on this render's canvas."}),
+            "prefix_frames": ("INT", {"default": 17, "min": 17, "max": 323, "step": 17,
+                                      "tooltip": "How many of its frames open this render, a multiple of 17."}),
+            "prefix_at": ("INT", {"default": -1, "min": -1, "max": 100000,
+                                  "tooltip": "The previous take's frame they start at; -1: its last prefix_frames."}),
+        }}
+
+    RETURN_TYPES = ("LATENT",)
+    FUNCTION = "run"
+    CATEGORY = "Viggle Remake"
+
+    def run(self, vae, latent, prefix, prefix_frames, prefix_at):
+        # frames [at, at + k) of the previous take are this render's first k frames: their latents go in clean with
+        # noise_mask 0, so like the held soundtrack they enter at the condition timestep, are never stepped and come
+        # out unchanged (the reference worker's prefix rows)
+        k = prefix_frames
+        at = len(prefix) - k if prefix_at < 0 else prefix_at
+        px = prefix[at:at + k, ..., :3]
+        video, audio = latent["samples"].unbind()
+        vmask, amask = latent["noise_mask"].unbind()
+        n = k // 17 * 5
+        assert k % 17 == 0 and n < video.shape[2], f"prefix_frames must be 17m and shorter than the render, got {k}"
+        assert tuple(px.shape[:3]) == (k, video.shape[3] * 16, video.shape[4] * 16), \
+            f"the prefix must be {k} frames on the {video.shape[4] * 16}x{video.shape[3] * 16} canvas, got {tuple(px.shape)}"
+        # the VAE encodes 17-frame clips and drops the last 3 latents: pad to 17m + 5, keep the first 5m
+        clean = vae.encode(torch.cat([px, px[-1:].repeat(5, 1, 1, 1)]))[:, :, :n]
+        video, vmask = video.clone(), vmask.clone()
+        video[:, :, :n] = clean.to(video)
+        vmask[:, :, :n] = 0
+        return ({**latent, "samples": comfy.nested_tensor.NestedTensor((video, audio)),
+                 "noise_mask": comfy.nested_tensor.NestedTensor((vmask, amask))},)
+
+
 NODE_CLASS_MAPPINGS = {"RemakeConditioning": RemakeConditioning, "RemakeKeyframeUntil": RemakeKeyframeUntil,
-                       "RemakeNoise": RemakeNoise}
+                       "RemakeNoise": RemakeNoise, "RemakePrefix": RemakePrefix}
 NODE_DISPLAY_NAME_MAPPINGS = {"RemakeConditioning": "Viggle Remake Conditioning",
                               "RemakeKeyframeUntil": "Viggle Remake Keyframe Until",
-                              "RemakeNoise": "Viggle Remake Noise"}
+                              "RemakeNoise": "Viggle Remake Noise",
+                              "RemakePrefix": "Viggle Remake Prefix"}

mcp/README.md

@@ -12,6 +12,15 @@ painted, writes the six-section prompt per render window, queues the renders, re
 final video. The server gives it 27 tools for that (28 in pool image mode; see [Tools](#tools)) and enforces the
 budgets. The rules the agent follows are in [`AGENTS.md`](viggle_remake_mcp/data/AGENTS.md).
 
+**What you need:**
+- **an agent**: Claude Code, codex, or any MCP client;
+- **a painter** for the keyframes: either a codex login with a ChatGPT subscription (`REMAKE_PAINT=pool`, no API
+  spend), or a key for an OpenAI-compatible images API (the default; see [Cost](#cost));
+- **render workers**: `viggle-remake-worker` on a card with 96 GB or more, or the
+  [ComfyUI worker](#one-32-gb-card-the-comfyui-worker) on one 32 GB card such as an RTX 5090 (Linux, about 60 GB of
+  RAM). The server itself needs no GPU. Cards below 32 GB are not tested with the agent; use ComfyUI by hand
+  ([`COMFYUI.md`](../COMFYUI.md#smaller-gpus-int8-and-gguf)).
+
 ## How it fits together
 
 ```
@@ -57,7 +66,8 @@ budgets. The rules the agent follows are in [`AGENTS.md`](viggle_remake_mcp/data
 
 ## Install
 
-Requires Python ≥ 3.10, and `ffmpeg` and `ffprobe` on `PATH`.
+Requires Python ≥ 3.10, and `ffmpeg` 5.1 or newer and `ffprobe` on `PATH` (Ubuntu 22.04 ships 4.4: use a static
+build).
 
 ```bash
 hf download Viggle/Viggle-Remake --include "mcp/*" --local-dir viggle-remake   # under 1 MB plus the example's 210 MB
@@ -173,8 +183,39 @@ One worker per GPU, bf16. A worker measures its card's free memory at start and
 A longer one-take becomes a chain of windows, each continuing the previous one's take, so a smaller card renders the same
 clip with more seams. Each worker also keeps the text encoder in host RAM: about 58 GiB of RAM per worker.
 
-On a smaller GPU you would offload or quantize the model yourself; the worker does not. Peak memory per length and the
-measurements behind the limits are in [`WORKER_NOTES.md`](WORKER_NOTES.md#memory).
+This worker does not offload or quantize; on a smaller GPU use the ComfyUI worker below. Peak memory per length and
+the measurements behind the limits are in [`WORKER_NOTES.md`](WORKER_NOTES.md#memory).
+
+#### One 32 GB card: the ComfyUI worker
+
+[`comfy_worker/`](comfy_worker) is a second worker for the same queue, for one consumer card. It imports ComfyUI 0.42.0
+as a library and runs the int8 per-mode transformers from [`int8/`](../int8) with ComfyUI's dynamic VRAM, which
+memory-maps the checkpoints and streams in, layer by layer, whatever does not fit on the card. The server and the agent
+are unchanged.
+
+```bash
+cd viggle-remake/mcp/comfy_worker
+./setup.sh          # once: ComfyUI 0.42.0, torch 2.9.1 + CUDA 13.0, the node pack, the int8 weights, the text encoder
+bash run_worker.sh  # in place of viggle-remake-worker, in the foreground; --gpu N picks the card
+```
+
+It needs Linux, an NVIDIA driver for CUDA 13, `uv`, `ffmpeg` 5.1 or newer, `hf`, about 70 GB of disk and 60 GB of RAM.
+Measured on one RTX 5090 (32 GB) with 60 GB of RAM, an int8 swap job, six steps:
+
+| canvas | frames | render |
+|---|---|---|
+| 832×480 | 226 | 3 min |
+| 832×480 | 345 | 6 min |
+| 1280×736 | 175 | 8 min |
+| 1280×736 | 226 | 13 min |
+| 1280×736 | 345 | 28 min; with four keyframes, VRAM at most 27.8 GB |
+
+A step is almost all attention, so a 345-frame window costs four times a 175-frame one; `--max-736 175` trades more
+seams for speed. White House Funk ran end to end on one 5090 in 2 h 58 min. How it differs from the bf16 worker:
+- the int8 transformer of the job's mode, one at a time; a job of the other mode swaps it;
+- ComfyUI keeps 4 GB of VRAM free (`--vram-headroom 4`);
+- after an OOM or a CUDA error the worker writes the `.err` and exits, and `run_worker.sh` restarts it;
+- hops go through the node pack's `RemakePrefix` node.
 
 ### Writing your own worker
 

mcp/comfy_worker/remake_worker.py

@@ -0,0 +1,262 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Viggle Remake render worker on ComfyUI as a library: one process, no ComfyUI server.
+
+    ComfyUI/venv/bin/python remake_worker.py [--gpu 0] [--home ~/.viggle-remake] [--max-480 345] [--max-736 345] \
+        [-- <ComfyUI flags, e.g. --vram-headroom 6>]
+
+Renders the viggle-remake MCP's file queue with the graph of comfyui/workflows/remake_*_api.json: the mode's int8
+transformer -> RemakeKeyframeUntil -> BasicGuider, euler on the mode's six-step sigmas, RemakeConditioning +
+RemakeNoise, and RemakePrefix for a hop. The queue protocol is the official worker's (viggle_remake_mcp/worker.py).
+"""
+import os
+import sys
+from pathlib import Path
+
+if "--gpu" in sys.argv[:-1]:  # before torch
+    os.environ["CUDA_VISIBLE_DEVICES"] = sys.argv[sys.argv.index("--gpu") + 1]
+COMFY = Path(__file__).resolve().parent / "ComfyUI"
+split = sys.argv.index("--") if "--" in sys.argv else len(sys.argv)
+# --vram-headroom: dynamic VRAM keeps 4 GB free, else a 311-frame 736 window OOMed on a 2.8 GB attention output
+# before the weights make room (a later flag after -- wins)
+ours, sys.argv = sys.argv[1:split], [str(COMFY / "main.py"), "--vram-headroom", "4", *sys.argv[split + 1:]]
+sys.path.insert(0, str(COMFY))
+sys.path.insert(0, str(COMFY / "custom_nodes"))
+
+import comfy.options  # noqa: E402
+
+comfy.options.enable_args_parsing()
+import cuda_malloc  # noqa: E402,F401  as ComfyUI's own main does before torch
+import main  # noqa: E402,F401  ComfyUI's setup (logging, dynamic VRAM, model paths); its server starts only as __main__
+
+import argparse  # noqa: E402
+import gc  # noqa: E402
+import importlib  # noqa: E402
+import json  # noqa: E402
+import resource  # noqa: E402
+import socket  # noqa: E402
+import subprocess  # noqa: E402
+import threading  # noqa: E402
+import time  # noqa: E402
+import traceback  # noqa: E402
+from fractions import Fraction  # noqa: E402
+
+import numpy as np  # noqa: E402
+import torch  # noqa: E402
+from PIL import Image  # noqa: E402
+
+import comfy.ldm.minimax.model as H3  # noqa: E402
+import comfy.model_management as mm  # noqa: E402
+import comfy.samplers  # noqa: E402
+import nodes  # noqa: E402
+from comfy_api.latest import InputImpl, Types  # noqa: E402
+from comfy_extras.nodes_audio import vae_decode_audio  # noqa: E402
+from comfy_extras.nodes_custom_sampler import Guider_Basic  # noqa: E402
+
+R = importlib.import_module("ComfyUI-Viggle-Remake.nodes")
+
+# Each block's MLP over the packed sequence FF_ROWS rows at a time, as the reference worker does: its S x 28672 bf16
+# fc1 output alone is 11 GiB at 345 frames on the 736 canvas, which a 32 GB card does not have. int8 quantizes the
+# activations per row, so the output is the same.
+FF_ROWS = 16384
+_mlp = H3.MLP.forward
+
+
+def mlp_rows(self, x):
+    if x.shape[-2] <= FF_ROWS:
+        return _mlp(self, x)
+    out = torch.empty_like(x)
+    for a in range(0, x.shape[-2], FF_ROWS):
+        out[..., a:a + FF_ROWS, :] = _mlp(self, x[..., a:a + FF_ROWS, :])
+    return out
+
+
+H3.MLP.forward = mlp_rows
+
+MODES = ("swap", "edit")
+FRAMES = range(90, 346, 17)  # 17n + 5, 24 fps
+AREA = {480: 832 * 480, 736: 1280 * 736}
+SIGMAS = {"swap": [1.0, 0.990476, 0.965243, 0.923077, 0.8, 0.631579, 0.0],
+          "edit": [1.0, 0.983607, 0.96, 0.923077, 0.857143, 0.705882, 0.0]}
+UNET = {m: f"viggle_remake_{m}_pruned_int8_convrot.safetensors" for m in MODES}
+TE = "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors"
+VAE, AUDIO_VAE = "minimax_h3_video_vae_fp16.safetensors", "minimax_h3_audio_vae_fp32.safetensors"
+
+
+def main_loop():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--gpu", default=None, help="card index; default: CUDA_VISIBLE_DEVICES as set")
+    parser.add_argument("--home", default=os.environ.get("REMAKE_HOME", "~/.viggle-remake"))
+    parser.add_argument("--max-480", type=int, default=345, help="longest render at 832x480 (17n+5, 0 = none)")
+    parser.add_argument("--max-736", type=int, default=345, help="longest render at 1280x736 (17n+5, 0 = none)")
+    args = parser.parse_args(ours)
+    gpu = os.environ.get("CUDA_VISIBLE_DEVICES", "0")
+    host = socket.gethostname()
+    home = Path(args.home).expanduser()
+    queue, workers = home / "queue", home / "workers"
+    for p in (queue, workers):
+        p.mkdir(parents=True, exist_ok=True)
+    limit = {480: args.max_480, 736: args.max_736}
+    state = {"job": None}
+    beat_path, lock = workers / f"{host}_{gpu}.json", threading.Lock()
+
+    def beat():
+        with lock:
+            tmp = beat_path.with_suffix(f".{os.getpid()}.tmp")
+            tmp.write_text(json.dumps({"t": time.time(), "host": host, "gpu": gpu, "pid": os.getpid(),
+                                       "job": state["job"], "modes": list(MODES),
+                                       "max_frames": {str(k): v for k, v in limit.items()}}))
+            tmp.rename(beat_path)
+
+    def beat_loop():
+        while True:
+            try:
+                beat()
+            except OSError:  # a transient fs error must not stop the heartbeat
+                pass
+            time.sleep(10)
+
+    t0 = time.time()
+    clip = nodes.CLIPLoader().load_clip(TE, type="minimax")[0]
+    vae = nodes.VAELoader().load_vae(VAE)[0]
+    audio_vae = nodes.VAELoader().load_vae(AUDIO_VAE)[0]
+    loaded = {}
+
+    def model(m):
+        # one transformer at a time (21 GB each): a mode switch drops the other one first
+        if m not in loaded:
+            loaded.clear()
+            mm.unload_all_models()
+            gc.collect()
+            mm.soft_empty_cache()
+            loaded[m] = nodes.UNETLoader().load_unet(UNET[m], "default")[0]
+        return loaded[m]
+
+    model("swap")
+    print(f"loaded in {time.time() - t0:.0f}s; max_frames {limit}", flush=True)
+    threading.Thread(target=beat_loop, daemon=True).start()
+
+    @torch.inference_mode()  # as ComfyUI's executor runs every node; without it autograd keeps the VAE's activations
+    def render(job):
+        m, nf, tier, refs, seed = job["mode"], job["num_frames"], job["tier"], job["refs"], job["seed"]
+        assert m in MODES, f"mode must be one of {MODES}, got {m!r}"
+        assert nf in FRAMES, f"num_frames must be 17n+5 in 90..345, got {nf}"
+        assert tier in limit, f"tier must be one of {list(limit)}, got {tier}"
+        if nf > limit[tier]:
+            raise ValueError(f"num_frames {nf} exceeds this worker's max {limit[tier]} at tier {tier}")
+        assert len(refs) <= 4, f"the node pack takes at most 4 refs, got {len(refs)}"
+        pin = job.get("pin") or []
+        assert len(pin) in (0, len(refs)) and all(0 <= k < nf for k in pin), (pin, len(refs), nf)
+
+        source = InputImpl.VideoFromFile(job["cond"]).get_components().images
+        h, w = source.shape[1], source.shape[2]
+        assert h % 32 == 0 and w % 32 == 0 and h * w <= AREA[tier], \
+            f"cond must be multiples of 32 with at most {AREA[tier]} pixels at tier {tier}, got {w}x{h}"
+        # the soundtrack, as RemakeConditioning takes it (it pads and cuts to the render)
+        pcm = subprocess.run(["ffmpeg", "-loglevel", "error", "-i", job["audio"], "-t", f"{nf / 24:.3f}", "-vn",
+                              "-f", "f32le", "-ac", "2", "-ar", str(R.AUDIO_SR), "-"],
+                             capture_output=True, check=True).stdout
+        audio = {"waveform": torch.frombuffer(bytearray(pcm), dtype=torch.float32).view(-1, 2).T[None],
+                 "sample_rate": R.AUDIO_SR}
+        ims = [torch.from_numpy(np.array(Image.open(x).convert("RGB"))).float()[None] / 255 for x in refs]
+        ims += [None] * (4 - len(ims))
+        # short_edge = the cond's own: it is already on its canvas, so nothing is resized
+        positive, latent = R.RemakeConditioning().run(clip, vae, audio_vae, source, job["prompt"], ims[0],
+                                                      " ".join(map(str, pin)), nf, min(h, w), audio, *ims[1:])
+        if job.get("prefix"):
+            prev = InputImpl.VideoFromFile(job["prefix"]).get_components().images
+            latent = R.RemakePrefix().run(vae, latent, prev, job.get("prefix_frames", 17), job.get("prefix_at", -1))[0]
+
+        guider = Guider_Basic(R.RemakeKeyframeUntil().run(model(m), job.get("keyframe_until", 0.9))[0])
+        guider.set_conds(positive)
+        noise = R.RemakeNoise().run(positive, seed)[0]
+        samples = guider.sample(noise.generate_noise(latent), latent["samples"], comfy.samplers.sampler_object("euler"),
+                                torch.tensor(SIGMAS[m]), denoise_mask=latent["noise_mask"], seed=seed)
+        out = {"samples": samples.to(mm.intermediate_device())}
+        video = InputImpl.VideoFromComponents(Types.VideoComponents(
+            images=nodes.VAEDecode().decode(vae, out)[0], audio=vae_decode_audio(audio_vae, out), frame_rate=Fraction(24)))
+        dst = Path(job["out"])
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        tmp = dst.with_name(f".{dst.stem}.{os.getpid()}.mp4")
+        video.save_to(str(tmp), format=Types.VideoContainer.MP4, codec=Types.VideoCodec.H264)
+        tmp.rename(dst)
+
+    def live_beats():
+        beats = []
+        for p in workers.glob("*.json"):
+            try:
+                hb = json.loads(p.read_text())
+            except (OSError, ValueError):
+                continue
+            if time.time() - hb["t"] < 120:
+                beats.append(hb)
+        return beats
+
+    def requeue_orphans(beats):
+        live = {hb["job"] for hb in beats}
+        for run in queue.glob("*.run"):
+            try:
+                if time.time() - run.stat().st_mtime > 120 and run.stem not in live:
+                    run.rename(run.with_suffix(".json"))
+                    print(f"{run.stem}: orphaned, requeued", flush=True)
+            except FileNotFoundError:
+                pass
+
+    def next_job(beats):
+        others = [hb for hb in beats if (hb["host"], hb["gpu"]) != (host, gpu)]
+        for p in sorted(queue.glob("*.json")):
+            try:
+                job = json.loads(p.read_text())
+            except (OSError, ValueError):  # claimed meanwhile, or still being written
+                continue
+            if job.get("prefix") and not Path(job["prefix"]).exists():
+                continue
+            tier, nf = job.get("tier"), job.get("num_frames", 0)
+            if nf > limit.get(tier, 0) and any(
+                    job.get("mode") in hb["modes"] and hb["max_frames"].get(str(tier), 0) >= nf for hb in others):
+                continue  # too long for this card, not for another live worker: leave it to that one
+            run = p.with_suffix(".run")
+            try:  # touch first: rename keeps the mtime, and a .run older than 120 s that no heartbeat names is requeued
+                os.utime(p)
+                p.rename(run)
+            except FileNotFoundError:
+                continue
+            return run
+        return None
+
+    while True:
+        beats = live_beats()
+        requeue_orphans(beats)
+        run = next_job(beats)
+        if run is None:
+            time.sleep(2)
+            continue
+        state["job"] = run.stem
+        beat()
+        try:
+            job = json.loads(run.read_text())
+            t = time.time()
+            render(job)
+            info = {"out": job["out"], "secs": round(time.time() - t, 1),
+                    "peak_rss_gib": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 2**20, 1),
+                    "host": host, "gpu": gpu, "mode": job["mode"], "sigmas": SIGMAS[job["mode"]]}
+            tmp = run.with_suffix(f".{os.getpid()}.tmp")
+            tmp.write_text(json.dumps(info))
+            tmp.rename(run.with_suffix(".done"))
+            print(f"{run.stem}: {info}", flush=True)
+        except Exception:
+            tb = traceback.format_exc()
+            run.with_suffix(".err").write_text(tb)
+            run.unlink(missing_ok=True)
+            print(f"{run.stem}: FAILED\n{tb}", flush=True)
+            # the context is gone after a CUDA error, and after an OOM freeing the exception's tensors aborts the
+            # process: exit before that happens (run_worker.sh restarts the worker)
+            if "CUDA error" in tb or "OutOfMemoryError" in tb:
+                os._exit(1)
+        run.unlink(missing_ok=True)
+        state["job"] = None
+        gc.collect()
+        mm.soft_empty_cache()
+
+
+if __name__ == "__main__":
+    main_loop()

mcp/comfy_worker/run_worker.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Viggle Remake worker in the foreground; restarted whenever it exits (it exits on purpose after a CUDA error or an OOM).
+#   bash run_worker.sh [worker flags] [-- ComfyUI flags]     Ctrl-C stops it.
+cd "$(dirname "$0")"
+trap exit INT
+ulimit -c 0  # a crash would otherwise dump a core of the whole address space (minutes, GBs)
+while true; do
+    ComfyUI/venv/bin/python remake_worker.py "$@"
+    echo "worker exited ($?), restarting in 5 s"
+    sleep 5
+done

mcp/comfy_worker/setup.sh

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+# One-time install of the ComfyUI render worker, next to this script: ComfyUI 0.42.0 with torch 2.9.1 / CUDA 13.0, the
+# repository's node pack linked in, the int8 transformers and the text encoder and VAEs. SKIP_MODELS=1 skips the weights.
+set -e
+cd "$(dirname "$0")"
+REPO=$(realpath ../..)
+export UV_HTTP_TIMEOUT=600
+[ -d ComfyUI ] || git clone -q --depth 1 --branch v0.42.0 https://github.com/comfyanonymous/ComfyUI.git
+[ -x ComfyUI/venv/bin/python ] || uv venv -q --python 3.12 ComfyUI/venv
+uv pip install -q --python ComfyUI/venv/bin/python torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
+    --index-url https://download.pytorch.org/whl/cu130
+uv pip install -q --python ComfyUI/venv/bin/python -r ComfyUI/requirements.txt
+M=(--include "comfyui/ComfyUI-Viggle-Remake/*")
+[ -n "$SKIP_MODELS" ] || M+=(--include "int8/*")
+hf download Viggle/Viggle-Remake "${M[@]}" --local-dir "$REPO"
+ln -sfn "$REPO/comfyui/ComfyUI-Viggle-Remake" ComfyUI/custom_nodes/ComfyUI-Viggle-Remake
+if [ -z "$SKIP_MODELS" ]; then
+    hf download Comfy-Org/MiniMax-H3 text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors \
+        vae/minimax_h3_video_vae_fp16.safetensors vae/minimax_h3_audio_vae_fp32.safetensors --local-dir ComfyUI/models
+    ln -sf "$REPO"/int8/*.safetensors ComfyUI/models/diffusion_models/
+fi
+echo "done. Start the worker with: bash $PWD/run_worker.sh"

mcp/viggle_remake_mcp/common.py

@@ -60,14 +60,14 @@ def frames(video, w, h):
 def pick(video, fs, W, H):
     """Only frames `fs` of a video, decoded at W x H (a full decode of a 2-minute cond is 3.4 GB)."""
     sel = "+".join(f"eq(n\\,{f})" for f in sorted(set(fs)))
-    raw = subprocess.run(["ffmpeg", "-v", "error", "-i", str(video), "-vf", f"select={sel},scale={W}:{H}", "-vsync", "0",
+    raw = subprocess.run(["ffmpeg", "-v", "error", "-i", str(video), "-vf", f"select={sel},scale={W}:{H}", "-fps_mode", "passthrough",
                           "-f", "rawvideo", "-pix_fmt", "rgb24", "-"], capture_output=True, check=True).stdout
     return dict(zip(sorted(set(fs)), np.frombuffer(raw, np.uint8).reshape(-1, H, W, 3)))
 
 
 def seg(video, a, b, W, H):
     raw = subprocess.run(["ffmpeg", "-v", "error", "-i", str(video), "-vf", f"select=between(n\\,{a}\\,{b - 1}),scale={W}:{H}",
-                          "-vsync", "0", "-f", "rawvideo", "-pix_fmt", "rgb24", "-"], capture_output=True, check=True).stdout
+                          "-fps_mode", "passthrough", "-f", "rawvideo", "-pix_fmt", "rgb24", "-"], capture_output=True, check=True).stdout
     return np.frombuffer(raw, np.uint8).reshape(-1, H, W, 3)
 
 

mcp/viggle_remake_mcp/kit.py

@@ -604,7 +604,7 @@ def unbar(d, v, canvas):
         if not m:
             continue
         raw = subprocess.run(["ffmpeg", "-v", "error", "-i", str(cond(d, canvas)), "-vf", f"scale={W}:{H},crop={crop(m)},"
-                              "scale=48:48,format=gray", "-vsync", "0", "-f", "rawvideo", "-pix_fmt", "gray", "-"],
+                              "scale=48:48,format=gray", "-fps_mode", "passthrough", "-f", "rawvideo", "-pix_fmt", "gray", "-"],
                              capture_output=True, check=True).stdout
         dark = np.percentile(np.frombuffer(raw, np.uint8).reshape(-1, 48 * 48)[:n], 98, axis=1) < 14
         for i in np.flatnonzero(dark):