Instructions to use BennyDaBall/PAN2-NVFP4-ComfyUI with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusion Single File
How to use BennyDaBall/PAN2-NVFP4-ComfyUI with Diffusion Single File:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
⚡ PAN 2 - NVFP4 for ComfyUI
Follow me on X @BennyDaBall_OG !
Want to support my work? Buy me a coffee on Ko-fi !
PAN 2 on a single consumer RTX 5090. A 29B world model in 16.3 GB, full W4A4, sampling up to 20 seconds of video on the card.
A native ComfyUI NVFP4 build of PAN 2, the general-purpose world model from the Institute of Foundation Models at MBZUAI, made from the Comfy-Org repack. Text to video and image to video in one file. Weights and activations are both 4-bit.
- One consumer card: the whole transformer sits in the VRAM of a single RTX 5090 with room left for the video. A 5 second clip peaks at 20.4 GB, and even a 20 second clip samples on the card.
- Built for Blackwell, on Blackwell: made on three RTX PRO 6000 Blackwell cards and one RTX 5090, for every Blackwell machine (RTX 50 series, RTX PRO Blackwell, DGX Spark).
- True W4A4: all 528 linear layers of the 44 joint blocks are NVFP4, with 4-bit weights and 4-bit activations. They are the same layers the official INT8 file quantizes.
- 10 GB lighter: 16.33 GB against 26.68 GB for the official INT8 and 67.37 GB for BF16.
- Faster than INT8 at every length: 16% on a 5 second clip, 10% at 10 seconds, 26% at 20 seconds.
- 20 seconds without spilling: on the same card the official INT8 pushes about 10 GB into shared memory while it samples a 20 second clip. This file samples it on the card.
- Every length calibrated: text to video and image to video, 2, 5, 10 and 20 second clips, on the real 50-step guided sampler.
- Drop-in: stock ComfyUI, core nodes only. Three workflows are in the repo.
- With sound: PAN 2 is silent. One of the workflows lets stock LTX-2.5 write the soundtrack from the finished frames, in the same run.
Image to video, 5 seconds. One picture in, the next five seconds out.
Every clip on this page came out of this file in ComfyUI at 1248x704, 50 steps, CFG 2.5. The prompts are in assets/PROMPTS.md, the full captions in assets/captions, the start picture in input.
Text to video, 5 seconds. Run, jump, catch.
Text to video, 10 seconds. Dig, wave, regret.
Text to video, 20 seconds. One continuous shot, start to finish, on one RTX 5090.
With sound: PAN 2 and LTX-2.5
PAN 2 makes video only. The third workflow adds the trick from my Hy-Nomas x LTX-2.5 audio workflows: the finished frames are encoded with LTX's own video VAE and frozen, and the stock LTX-2.5 distilled model denoises only an empty audio track next to them. LTX-2.5 is a joint audio and video model, so the sound follows the picture. No training, no extra nodes. Turn the volume on.
Text to video with LTX-2.5 audio, 5 seconds. Crack, drop, sizzle.
Text to video with LTX-2.5 audio, 5 seconds. She holds the camera and turns.
Text to video with LTX-2.5 audio, 5 seconds. One push, then physics.
Picture and sound of each clip came out of one run of the workflow. The raw soundtracks come out at very different levels (the egg loud, the island quiet), so the copies on this page are leveled to about the same loudness. Nothing else was done to them.
The numbers
Timed on my RTX 5090 in stock ComfyUI with core nodes: 1248x704, 24 fps, euler, simple, 50 steps, CFG 2.5. Time is the sampling of one clip: the warm median of three for 5 seconds, single runs for 10 and 20.
| Transformer file | Size | 5s clip | 10s clip | 20s clip |
|---|---|---|---|---|
| Original BF16 | 67.37 GB | does not fit | does not fit | does not fit |
| Official INT8 | 26.68 GB | 262s | 672s | 2,450s |
| NVFP4, this repo | 16.33 GB | 220s | 602s | 1,819s |
- Peak VRAM on a 5 second clip: 20.4 GB against 30.1 GB for the official INT8. BF16 needs 58 GB or more.
- 28% closer to BF16 than plain NVFP4: along the way I built a plain round-to-nearest NVFP4 of the same layers as my baseline. Over 10 held-out prompts, rendered on the RTX PRO 6000 Blackwell cards, its mean LPIPS against the BF16 render is 0.283. This file is at 0.205, closer on 10 of 10 prompts, at the same size. The official INT8, with 8-bit weights and activations and 10 GB more, is at 0.100.
- No drift on the 20 second test clip: its last quarter is as close to BF16 as the rest of it.
Whole prompt from the ComfyUI interface on the same card, one run of each workflow (text encoder, sampling, decode and MP4 save): text to video 5s in 244s, image to video 5s in 248s, text to video with LTX-2.5 audio 5s in 258s.
How I did it
The shape of it, without the recipe.
- Teacher and student: the full-precision model is the teacher, the parent. The 4-bit file is the student, the child. Nothing is rounded and hoped for: each of the 528 matrices is solved so the student does what the teacher does on real sampling runs.
- Calibrated the way the model is used: text to video and image to video, clips from 2 to 20 seconds, both branches of the guided sampler, across the whole 50-step schedule.
- GPTQ, taken further: the weights come out of my own complex derivatives of GPTQ (Frantar et al., 2022). Every rounding choice is corrected against what the layer actually sees, instead of rounding weight by weight.
- Schur Replay scoring with Four Over Six: each 4-bit block scale is picked by Schur Replay scoring (arXiv 2609.36654), with the Four Over Six points (arXiv 2512.02010) among the options.
- An E4 pass to finish: one more pass over the 4-bit codes and their E4M3 block scales after the solve.
- Only the layers that should move: the embedders, the caption refiner, every adaLN and modulation tensor, the norms and the output layer stay exactly as in the official INT8 file.
- Checked like a release: every tensor is read back after writing and decoded with the same math ComfyUI uses, then the file is rendered on prompts it never saw.
Use it
| Download | ComfyUI folder | Size |
|---|---|---|
| PAN 2 transformer (NVFP4) | models/diffusion_models/ |
16.33 GB |
Text encoder qwen3.5_9b_int8_convrot.safetensors, from Comfy-Org/PAN2 |
models/text_encoders/ |
9.75 GB |
VAE wan2.2_vae.safetensors, from Comfy-Org/PAN2 |
models/vae/ |
1.41 GB |
| For the audio workflow only: the LTX-2.5 distilled INT8 transformer, its Gemma 4 text encoder and the video and audio VAEs, from Lightricks/LTX-2.5 | their usual folders |
- Update ComfyUI to a build with PAN 2 support (merged 2026-11-15, pull request 17755).
- Put the files in their folders.
- Drag a workflow from workflows into ComfyUI and click Run. For image to video, copy the picture from input into
ComfyUI/input/first.
| Workflow | Example |
|---|---|
| Text to Video | the dog and the ball |
| Image to Video | the sea turtle |
| Text to Video with LTX-2.5 Audio | the egg, with sound |
All three use core nodes only: Load Diffusion Model, Load CLIP (type pan2), Load VAE, a text node for the caption, CLIP Text Encode, Wan22ImageToVideoLatent, KSampler, VAE Decode. I ran each of them from the ComfyUI interface before putting it here.
- Prompts: PAN 2 is trained on structured world-state captions (JSON), not on plain one-line prompts. Write the caption with IFM's prompt rewriter, which any good LLM can follow, and keep their fixed negative prompt.
- Where the caption goes: paste it into the PAN 2 CAPTION text node of the workflow, not straight into CLIP Text Encode. ComfyUI's CLIP Text Encode box reads
{a|b}as a random choice and strips the braces out of the JSON. The plain text node passes it through untouched. - Settings: 1248x704, 24 fps, 50 steps, euler, simple, CFG 2.5.
- Length: 121 frames is 5 seconds, 241 is 10, 481 is 20. The event times in the caption should cover the same length.
- Sound prompt: one flowing paragraph for LTX-2.5. Describe the scene briefly, then every sound and when it happens.
- Hardware: the FP4 speed-up needs an NVIDIA Blackwell GPU (RTX 50 series, RTX PRO Blackwell, DGX Spark).
Files
| File | Size | sha256 |
|---|---|---|
diffusion_models/PAN2-NVFP4-ComfyUI.safetensors |
16,330,390,364 bytes | f643ebc36697c33c8bcbaec1a30083df729ca017ca65db9a1bda08d769b1535a |
- 528 NVFP4 matrices: the linear layers of the 44 joint blocks, in the stock ComfyUI
nvfp4layout. - Everything else untouched: the embedders, the caption refiner, the adaLN (in the compact curve form), norms, biases, the output layer and the per-block INT8 attention tags carry the values of
pan2_int8_convrot.safetensorsfrom Comfy-Org/PAN2.
Test setup
ComfyUI at commit e9afbccc2, the first build after PAN 2 support was merged (version string 0.45.0), comfy-kitchen 0.2.41, torch 2.14.0 + CUDA 13.0, one RTX 5090. Text encoder qwen3.5_9b_int8_convrot and the Wan 2.2 VAE for every file, clips and numbers alike. Same prompt and seed for every file in a comparison.
Quantized by BennyDaBall. Built on three RTX PRO 6000 Blackwell cards and one RTX 5090.
License and credits
PAN 2 is made by the Institute of Foundation Models at MBZUAI and released under Apache-2.0. This repository is a quantized derivative of it under the same license, built from the ComfyUI repack by Comfy-Org. LICENSE, NOTICE and THIRD_PARTY_LICENSES are included. It is not an official IFM or Comfy-Org release.
Thanks to IFM for the model, to Comfy-Org and comfy-kitchen for the ComfyUI support and the NVFP4 runtime, to Lightricks for LTX-2.5, and to the authors of GPTQ, Schur Replay and Four Over Six.
Follow me on X @BennyDaBall_OG !
Want to support my work? Buy me a coffee on Ko-fi !
- Downloads last month
- 32
