| # OmniRef native ComfyUI export |
|
|
| The INT8 file packaged in this repository is an unchanged copy of the existing |
| Comfy-produced artifact, SHA256 |
| `851b2dc9670ed864b7f1e951ac9da047d83b1c04ded9bfcc1a9e5bba0ee33f80`. |
| Its source identity and sampled PDD/quantized-row checks are in `../provenance/`. |
| The converter below produced the BF16 variant. Running the optional quantizer |
| commands below creates a new PTQ variant; they are not a claim of byte-for-byte |
| reproduction of the supplied INT8 file, whose reduced AdaLN weights are FP16 |
| rather than this converter's FP32. |
|
|
| Source: `FastVideo/FastVideo-FastH3-Omni-8-Step-V1` at |
| `88d512c5e7cc38dda499fbb65688ce58611f9c80`, component `transformer_ref`. |
| The caller must verify the source shards against the frozen source identity |
| before conversion. The converter checks every indexed tensor and its geometry; |
| that check alone does not prove the files match the Hub revision. |
|
|
| The target is native ComfyUI's MiniMax H3 architecture, pinned to |
| [`b1a247c9cf9dbfda4b2c0a0c1d50950667eeaa0c`](https://github.com/Comfy-Org/ComfyUI/tree/b1a247c9cf9dbfda4b2c0a0c1d50950667eeaa0c). |
| Official model tools are pinned to |
| [`d6797787e6bdb1a1fb0094d588a26f8e71a1c757`](https://github.com/Comfy-Org/comfy-model-tools/tree/d6797787e6bdb1a1fb0094d588a26f8e71a1c757). |
| The small pinned architecture, detector, quantizer, source URLs, hashes and GPL |
| licenses are under `upstream/`. No upstream source tree or local virtualenv is |
| required in the model repository. |
|
|
| ## Transformations |
|
|
| - Rename all 688 source tensors to the consumer's names; unknown/missing keys |
| or mismatched geometry abort conversion. |
| - Fuse Q, K, V in that row order, including the token refiner. Swap the two |
| SwiGLU input halves from source value/gate to native Comfy gate/value. |
| - Preserve all 32 PDD heads. Native Comfy stores the first full head and each |
| later head's offset from the first. Compute those offsets in FP32; source |
| BF16 values and their differences are exactly representable at this stage. |
| - Generate the nonpersistent source RoPE frequencies as the persistent buffer |
| expected by Comfy, using the original FP32 formula. |
| - Preserve source dtypes otherwise. Native Comfy casts its input, timestep and |
| output projection islands to FP32 when loading. |
| - `--adaln-curve-rank 0` retains the dense timestep MLP and full AdaLN matrices. |
| `--adaln-curve-rank 8 --curve-grid 1025` uses the same kind of time-embedding |
| curve compression as the V2 Comfy pruned checkpoint: fit a shared SVD basis, |
| project all 51 modulation matrices and replace the timestep MLP by a |
| linearly interpolated coordinate table on `[0,1]`. This is an approximation, |
| not removal of unused model blocks. The export stores table and reduced |
| matrices in FP32; the V2 checkpoint stores reduced matrices in FP16. Its |
| receipt records fit residual and 129 independent holdout timesteps on 64 |
| sampled output rows per projection. No output-quality claim follows from |
| these modulation checks alone. |
|
|
| Every source tensor contributes to the output or the curve projection; no |
| trained transformer blocks or PDD heads are discarded. The four timestep-MLP |
| tensors are replaced only in the explicit curve variant. |
|
|
| ## Run |
|
|
| The conversion requires CPU PyTorch, NumPy and safetensors. It uses bounded |
| buffers and a scratch directory, instead of loading the 70.1 GB source into |
| RAM. The output writer hashes bytes as it streams them. A partial file is |
| renamed only after a complete write; existing outputs and scratch directories |
| are not overwritten. |
|
|
| ```bash |
| python convert_omniref_comfy.py \ |
| --src /verified/source/transformer_ref \ |
| --contract /verified/source/fastvideo_inference.json \ |
| --output /staging/diffusion_models/fasth3_omniref_8step_pruned_bf16.safetensors \ |
| --scratch /allocated/scratch/omniref-comfy-curve-r8 \ |
| --adaln-curve-rank 8 --curve-grid 1025 --threads 4 |
| ``` |
|
|
| This produces 582 tensors with the curve recipe or 585 without it. The full |
| source contains 688 tensors and 70,142,174,464 tensor bytes. Source and output |
| heads are not interchangeable by renaming: the PDD representation differs. |
|
|
| After validating the BF16-format export, run the pinned official quantizer's |
| `--dry-run`, then quantize on an allocated GPU: |
|
|
| ```bash |
| python upstream/quant_int8_convrot.py \ |
| /staging/diffusion_models/fasth3_omniref_8step_pruned_bf16.safetensors \ |
| --dry-run |
| python upstream/quant_int8_convrot.py \ |
| /staging/diffusion_models/fasth3_omniref_8step_pruned_bf16.safetensors \ |
| /staging/diffusion_models/fasth3_omniref_8step_pruned_int8_convrot.safetensors \ |
| --verify-report /staging/int8-convrot-verification.txt |
| ``` |
|
|
| The tool defaults to absmax INT8, a minimum GEMM dimension of 256 and the largest |
| eligible ConvRot group in 256/64/16. Do not add MSE clipping, FP32 downcasting, |
| W4A8 or W6A8 when claiming this default recipe. Final/output heads and the |
| small reduced AdaLN matrices pass through. Execution is CUDA-only in this |
| version and retains the output dictionary in CPU RAM; allocate enough RAM for |
| the roughly 22 GB INT8 result plus work buffers. The dry run itself is CPU. |
|
|
| ## Validation and inference limits |
|
|
| `python test_conversion.py` runs six CPU regression checks against actual |
| pinned ComfyUI architecture and detector definitions. Ordinary CPU Linear and |
| RMSNorm replace CUDA operations for these checks. Tests cover strict state-dict |
| loading for dense and curve forms, fused QKV order, native SwiGLU output, |
| native PDD head reconstruction, rejected unexpected tensors and streamed-file |
| hash/overwrite protection. `audit_native_geometry.py --src ...` additionally |
| checks every real-source key and shape against dense and rank-8 native model |
| skeletons on the meta device, without allocating model weights. |
|
|
| These are checkpoint format and CPU arithmetic checks. They do not validate |
| CUDA quantization kernels or an end-to-end generated clip. |
|
|
| Native Comfy inference differs from the frozen FastVideo source contract: |
|
|
| - Native PDD uses a unit `[1,0]` fine grid; the source's grid maximum is 0.999. |
| Native PDD storage is handled by this converter, but its sampler grid is |
| consumer behavior and is not changed by model metadata. |
| - Native Comfy's VSA node uses `(4,4,4)` video cubes, while the source contract |
| specifies tile size 128 and reference policy `p2_multi_region`. |
| - Native attention is dense until a sparse-attention patch is selected. |
| - Source generation requires 8 forwards, guidance 1, video/audio shifts 12/3 |
| and the declared 32-interval head partition. Do not copy V2's video shift 10 |
| or present the V2 text-to-video template as an OmniRef workflow. |
|
|
| The original inference contract is retained verbatim in safetensors metadata |
| and in the conversion receipt. Do not claim bit-identical inference or reuse |
| GB200 benchmark numbers for native Comfy without a matched runtime check. |
|
|
| ## Other checkpoints |
|
|
| The existing V2 Comfy text encoders and complete original 36L video/audio VAEs |
| are valid native-Comfy artifacts and can be copied with their original hashes |
| and license attribution. MiniMax-H3 weights retain the MiniMax-H3 Community |
| License; the tooling's GPL license is separate from the model license. |
|
|
| The FastVideo FP8 source needs a separate format bridge for fused QKV/scales, |
| SwiGLU/scales and native quantization metadata. The QAD NVFP4 source uses |
| FlashInfer packed weights, swizzled block scales and per-linear global scales; |
| it cannot be copied or renamed into native Comfy's fused format. In particular, |
| Q/K/V may have different global scales. Requantizing BF16 masters would be a |
| new PTQ treatment, not byte-preserving conversion of the released QAD kernel |
| representation. Neither unimplemented bridge should be advertised as a |
| working Comfy checkpoint. |
|
|
