Two-adapter release: docs and code

MODIFICATIONS.md

@@ -55,6 +55,14 @@ Not MiniMax files. Written by us against the public `diffusers` API (`recam/h3.p
 own layout builder and scheduler; nothing in `diffusers` is patched). Licensed under Apache 2.0
 (`LICENSE-CODE`); each Python file carries an `SPDX-License-Identifier: Apache-2.0` header.
 
+## `comfyui/` — new
+
+Not MiniMax files. `comfyui/meridian_*_lora.safetensors` are the two adapters above rewritten into
+ComfyUI's generic LoRA layout by `recam/to_comfyui.py` — the same numbers in different keys, with the
+qkv projections fused and the SwiGLU halves reordered to match, nothing retrained. They load on
+ComfyUI's own `minimax_h3_fl2va_bf16.safetensors`, which is not redistributed here.
+`comfyui/meridian_embed.py` is a ComfyUI node we wrote, Apache 2.0 like the rest of the code.
+
 ## Not included: VGGT-Omega
 
 Inference depends on Meta's VGGT-Omega for geometry. It is not redistributed here (FAIR Noncommercial

README.md

@@ -201,6 +201,45 @@ final video generation are separate GPU operations, not real-time generative vid
 The service has no authentication. The command above binds to loopback; do not expose this
 prototype directly to the internet.
 
+## ComfyUI
+
+Both adapters are also published in ComfyUI's generic LoRA format, under `comfyui/`.
+
+Load **`minimax_h3_fl2va_bf16.safetensors`** from
+[Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3) — Meridian is trained on
+MiniMax-H3's `fl2va` partition, so the `ref2va` file is the wrong base — then apply
+`comfyui/meridian_teacher_lora.safetensors` and `comfyui/meridian_turbo_lora.safetensors`, in that
+order, both at strength 1.0. Sampling is the same as here: `MiniMaxH3SigmaShift` at 3.0 with 4 steps
+for the turbo pair, or 12.0 with 50 steps for the teacher alone.
+
+Conditioning goes through `MiniMaxH3ReferenceToVideo` with **two reference videos**: the source clip
+first, the geometric render second, and the text of `assets/prompt.txt` as the prompt. That is the
+`<Video 1>` / `<Video 2>` presentation the adapters were trained on. Give both reference videos at
+**`render.mp4`'s resolution** — the 480-class canvas of `recam/h3.py`'s ladder, which is what the
+adapters were conditioned on. The node picks a 768-class canvas by default but never upscales, so a
+reference already at the smaller size is passed through untouched; a source clip left at the target
+canvas would be encoded 2.5x too large.
+
+**ComfyUI cannot produce the render.** No node reconstructs the point cloud or splats it along an
+authored camera path, and the adapters do nothing without one. Produce it here first — every
+`inference/sample.py` run writes `render.mp4` next to its output — and load that file as the second
+reference.
+
+One more node is needed, and it ships here: **`comfyui/meridian_embed.py`**, dropped into
+`ComfyUI/custom_nodes/`, adds *Meridian Frozen Prompt*. Wire it between `MiniMaxH3ReferenceToVideo`
+and the sampler and point `assets_dir` at this repo's `assets/`. Inference here conditions on a frozen
+text embedding — `assets/fixed_embed_{n}.pt`, the presentation above with timestamp markers and no
+pixels — and that is what the adapters were trained against: the reference videos reach the model only
+as condition rows, never through the text encoder. ComfyUI instead samples both reference videos at
+2 fps and feeds those frames to Qwen3-VL, so without this node its text conditioning carries vision
+tokens this training never saw. The node reads the clip length off the latent, so it cannot load the
+wrong embedding.
+
+The weight conversion is exact (`recam/to_comfyui.py`, checked key by key against the published
+ComfyUI weights), and the rest of the graph was checked by reading ComfyUI's H3 nodes rather than
+guessing: the condition rows get the same 0.999 noise augmentation and the same pinned timestep there
+as here. We do not run ComfyUI ourselves, so the graph is described from its source, not measured.
+
 ## Performance
 
 Reported results on **one B200 with the service resident**, using the default adapter:

comfyui/meridian_embed.py

@@ -0,0 +1,53 @@
+# Copyright 2026 Viggle AI. Licensed under the Apache License, Version 2.0 (see LICENSE-CODE).
+# SPDX-License-Identifier: Apache-2.0
+"""A ComfyUI node that swaps in Meridian's frozen text conditioning.
+
+Drop this file into `ComfyUI/custom_nodes/` and **Meridian Frozen Prompt** appears under `conditioning`.
+Wire it between `MiniMaxH3ReferenceToVideo` and the sampler; leave the rest of the graph alone.
+
+Why it is needed. Meridian was trained on a text-only presentation: `assets/fixed_embed_{n}.pt` holds
+`hidden_states[50]` of Qwen3-VL for the `<Video 1>` / `<Video 2>` text with timestamp markers and no
+pixels, and the two reference videos reach the model only as DiT condition rows. ComfyUI's reference
+node instead samples both videos at 2 fps and feeds those frames to Qwen3-VL, so its text conditioning
+carries vision tokens these adapters never saw. This node replaces that tensor and its
+`minimax_token_tags` with the frozen pair and keeps everything else in the conditioning, `minimax_refs`
+included.
+
+The two sides meet at the same tensor: `layer="last"` on ComfyUI's text encoder, whose stack is
+truncated to 50 layers, with `layer_norm_hidden_state=False`, is `hidden_states[50]` unnormalised --
+which is what `recam/make_embed.py` saved. Both paths then run the model's own `condition_proj` and
+token refiner, so the substitution happens one stage before anything H3-specific.
+
+`assets_dir` is this repo's `assets/` (an absolute path if ComfyUI does not run from here). The clip
+length is read off the latent rather than typed again, so the embedding cannot disagree with what is
+being sampled.
+"""
+import os
+
+import torch
+
+
+class MeridianFrozenPrompt:
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {"required": {"conditioning": ("CONDITIONING",),
+                             "latent": ("LATENT",),
+                             "assets_dir": ("STRING", {"default": "assets"})}}
+
+    RETURN_TYPES = ("CONDITIONING",)
+    FUNCTION = "apply"
+    CATEGORY = "conditioning"
+
+    def apply(self, conditioning, latent, assets_dir):
+        samples = latent["samples"]
+        video = samples.unbind()[0] if getattr(samples, "is_nested", False) else samples
+        frames = (video.shape[2] - 2) // 5 * 17 + 5  # the inverse of H3's video_latent_t
+        embed = torch.load(os.path.join(assets_dir, f"fixed_embed_{frames}.pt"),
+                           map_location="cpu", weights_only=True)
+        print(f"Meridian: {frames} frames -> {embed['prompt_embeds'].shape[1]} frozen text tokens")
+        return ([[embed["prompt_embeds"], {**d, "minimax_token_tags": embed["text_token_tags"]}]
+                 for _, d in conditioning],)
+
+
+NODE_CLASS_MAPPINGS = {"MeridianFrozenPrompt": MeridianFrozenPrompt}
+NODE_DISPLAY_NAME_MAPPINGS = {"MeridianFrozenPrompt": "Meridian Frozen Prompt"}

recam/to_comfyui.py

@@ -0,0 +1,85 @@
+# Copyright 2026 Viggle AI. Licensed under the Apache License, Version 2.0 (see LICENSE-CODE).
+# SPDX-License-Identifier: Apache-2.0
+"""Rewrite a Meridian adapter from diffusers PEFT layout into ComfyUI's generic LoRA layout.
+
+    python -m recam.to_comfyui teacher_lora/pytorch_lora_weights.safetensors comfyui/meridian_teacher.safetensors
+
+Same delta, different packing. ComfyUI loads MiniMax-H3 from its own repack
+(`Comfy-Org/MiniMax-H3`), which keeps the reference implementation's module names and two of its
+fusions, so three things have to change and nothing else does:
+
+  names    `transformer_blocks.N` -> `diffusion_model.blocks.N`; `proj_in` -> `video_patch_proj`,
+           `proj_out` -> `final_layer.video_out`, `attn.to_out.0` -> `attn.out_proj`,
+           `ff.net.0.proj` -> `mlp.fc1`, `ff.net.2` -> `mlp.fc2`.
+  qkv      diffusers keeps `to_q/to_k/to_v` separate, ComfyUI fuses them into one `attn.qkv_proj`
+           whose rows are `[q; k; v]` in that order. Concatenating A down the rank axis and putting
+           the three B's on the block diagonal keeps each output slab reading only its own A, so
+           the fused delta equals the three separate ones stacked. Rank triples, and alpha with it.
+  SwiGLU   both fuse the gate and value projections into one `fc1`, in opposite order: the reference
+           computes `fc2(silu(gate) * value)` from `[gate; value]`, diffusers' `SwiGLU` computes
+           `value * silu(gate)` from `[value; gate]`. B's two row halves swap; A is untouched.
+
+Neither the row order nor the half swap is a guess: `blocks.0` of ComfyUI's
+`minimax_h3_fl2va_bf16.safetensors` is bit-identical to this repo's base transformer once both are
+applied, and the official ComfyUI H3 LoRA documents the same recipe in its own metadata.
+
+Note the base: Meridian trains on MiniMax-H3's **fl2va** partition, so the ComfyUI file to load is
+`minimax_h3_fl2va_bf16.safetensors`, not the ref2va one.
+
+`alpha` is written equal to rank so ComfyUI's `alpha / rank` scale is 1.0, which is what diffusers
+applies here (`lora_alpha` 128, `r` 128). Load the teacher and the turbo adapter together at
+strength 1.0, in that order; the turbo adapter was distilled against the teacher and does nothing
+sensible without it.
+"""
+import sys
+
+import torch
+from safetensors.torch import load_file, save_file
+
+BLOCKS, DIM, QKV, FFN = 50, 5376, 7168, 14336
+
+
+def convert(src, dtype=torch.float16):
+    w = load_file(src)
+    r = w["proj_in.lora_A.weight"].shape[0]
+    out = {}
+
+    def put(name, A, B, rank):
+        out[f"diffusion_model.{name}.lora_A.weight"] = A.to(dtype).contiguous()
+        out[f"diffusion_model.{name}.lora_B.weight"] = B.to(dtype).contiguous()
+        out[f"diffusion_model.{name}.alpha"] = torch.tensor(float(rank))
+
+    put("video_patch_proj", w["proj_in.lora_A.weight"], w["proj_in.lora_B.weight"], r)
+    put("final_layer.video_out", w["proj_out.lora_A.weight"], w["proj_out.lora_B.weight"], r)
+
+    for i in range(BLOCKS):
+        p, q = f"transformer_blocks.{i}", f"blocks.{i}"
+
+        A = torch.cat([w[f"{p}.attn.to_{x}.lora_A.weight"] for x in "qkv"])
+        B = A.new_zeros(3 * QKV, 3 * r)
+        for j, x in enumerate("qkv"):
+            B[j * QKV:(j + 1) * QKV, j * r:(j + 1) * r] = w[f"{p}.attn.to_{x}.lora_B.weight"]
+        put(f"{q}.attn.qkv_proj", A, B, 3 * r)
+
+        put(f"{q}.attn.out_proj", w[f"{p}.attn.to_out.0.lora_A.weight"], w[f"{p}.attn.to_out.0.lora_B.weight"], r)
+
+        value, gate = w[f"{p}.ff.net.0.proj.lora_B.weight"].chunk(2)
+        put(f"{q}.mlp.fc1", w[f"{p}.ff.net.0.proj.lora_A.weight"], torch.cat([gate, value]), r)
+
+        put(f"{q}.mlp.fc2", w[f"{p}.ff.net.2.lora_A.weight"], w[f"{p}.ff.net.2.lora_B.weight"], r)
+
+    return out
+
+
+if __name__ == "__main__":
+    src, dst = sys.argv[1], sys.argv[2]
+    out = convert(src)
+    save_file(out, dst, metadata={
+        "format": "pt",
+        "source_format": "Diffusers PEFT LoRA",
+        "target_format": "ComfyUI generic LoRA",
+        "source_file": src,
+        "qkv_fusion": "concat A; block diagonal B; alpha multiplied by 3",
+        "swi_glu_mapping": "Diffusers [value;gate] -> ComfyUI [gate;value]",
+    })
+    print(f"{len(out)} tensors -> {dst}")