BAGEL-7B-MoT ConvRot (INT8, W6/W8 Hybrid & INT6) for ComfyUI

Quantized single-file checkpoints of ByteDance-Seed/BAGEL-7B-MoT using ConvRot (block Hadamard rotation + row-wise symmetric quantization) for ComfyUI, cutting peak inference VRAM from 31.32 GB (BF16) down to 17.17–18.19 GB (fitting comfortably on a single 24 GB GPU) while preserving multimodal Text-to-Image, Thinking-guided Generation, Image Editing, and Image Understanding.

This repository also bundles the patched ComfyUI-BAGEL custom node (both extracted folder ComfyUI-BAGEL/ and reproducible archive ComfyUI-BAGEL-a1a19a40e619.tar.gz) plus a lossless unpack script scripts/unpack_int6.py for the bit-packed INT6 downloads.


Available Checkpoints

File Variant Quantized Layers File Size ComfyUI Ready? Notes
bagel-7b-mot-int8-convrot.safetensors INT8 ConvRot (Recommended) 392 / 392 INT8 16.18 GB Yes (Direct) Closest quality to BF16 across all tasks
int6_experiments/bagel-7b-mot-w6w8-hybrid-convrot.safetensors W6/W8 Hybrid ConvRot 288 INT6 + 104 INT8 16.18 GB Yes (Direct) Keeps layers.0,1,26,27 & o_proj in INT8; stored in INT8 container
int6_experiments/bagel-7b-mot-w6w8-hybrid-convrot-packed.safetensors W6/W8 Hybrid ConvRot (Packed) 288 INT6 + 104 INT8 13.53 GB Unpack first Smaller download; unpack via scripts/unpack_int6.py
int6_experiments/bagel-7b-mot-w6a8-int6-convrot.safetensors W6A8 INT6 ConvRot 392 / 392 INT6 ([-31, 31]) 16.18 GB Yes (Direct) Experimental all-INT6 weights in ComfyUI INT8 container
int6_experiments/bagel-7b-mot-int6-convrot-packed.safetensors INT6 ConvRot (Packed) 392 / 392 INT6 12.91 GB Unpack first Smallest file (4Γ—6-bit values bit-packed into 3 uint8 bytes)

Why do *-w6a8-* / *-w6w8-hybrid-* files still measure 16.18 GB on disk? Stock ComfyUI / PyTorch TensorCoreINT8Layout stores 6-bit constrained weights ([-31, 31]) inside standard torch.int8 tensors so they load with zero custom CUDA kernel changes. Download the bit-packed *-packed.safetensors files to save bandwidth and unpack them locally with scripts/unpack_int6.py.


Visual Comparisons (BF16 Baseline vs INT8 ConvRot vs W6/W8 Hybrid ConvRot vs W6A8 INT6 ConvRot)

All outputs below were rendered on the same ComfyUI v0.45.0 pipeline (1024Γ—1024, 50 steps, cfg_text_scale=4.0, timestep_shift=3.0).

Text-to-Image (No Thinking)

T2I Cabin 4-way T2I Pocket Watch 4-way

Text-to-Image with <think> Chain-of-Thought

T2I Think Robot 4-way

Honest Note on INT6 / Hybrid Thinking Mode: Notice in t2i_think_s42:

  • INT8 ConvRot closely matches the scene coherence of BF16 (<think> rewrites the prompt slightly differently, producing a slightly different framing with crisp object structure).
  • All-INT6 (W6A8) showed occasional composition artifacts under <think> (e.g., an extra small floating robot head). Keeping the 104 boundary/o_proj layers in INT8 (W6/W8 Hybrid ConvRot) eliminated the floating head artifact, though some composition merge still occurred between the watering pot and robot arm. For production multimodal generation and <think> mode, INT8 ConvRot is strongly recommended.

Image Editing (Using Matched BF16 Input Image)

Image Edit Winter Cabin 4-way


Evaluation Metrics (G4 GPU, ComfyUI v0.45.0)

Measured via scripts/eval_bagel_g4.py comparing each quantized checkpoint directly against the unquantized bagel-7b-mot.safetensors (BF16) baseline (stats.json files archived in metrics/):

Variant Peak VRAM (T2I / Edit) t2i_cabin_s42 (Time / Cos / SSIM / PSNR) t2i_watch_s1337 (Time / Cos / SSIM / PSNR) t2i_think_s42 (Time / Cos / SSIM / PSNR) edit_cabin_winter_s42 (Time / Cos / SSIM / PSNR) understand_cabin Time
BF16 Baseline 30.30 / 31.32 GB 36.38s / ref 31.43s / ref 34.55s / ref 40.32s / ref 1.58s
INT8 ConvRot (392/392) 17.17 / 18.19 GB 27.75s / 0.97786 / 0.89359 / 24.42 dB 25.78s / 0.98396 / 0.92897 / 24.13 dB 33.92s / 0.52064 / 0.49434 / 9.72 dB 33.79s / 0.99582 / 0.95865 / 32.08 dB 1.53s
W6/W8 Hybrid ConvRot 17.18 / 18.19 GB 28.20s / 0.95340 / 0.80723 / 21.93 dB 25.77s / 0.96230 / 0.87035 / 20.82 dB 34.13s / 0.51049 / 0.48816 / 9.59 dB 33.78s / 0.99503 / 0.95232 / 31.08 dB 1.85s
W6A8 INT6 ConvRot (392/392) 17.18 / 18.19 GB 28.16s / 0.95483 / 0.81805 / 22.08 dB 25.77s / 0.93379 / 0.83128 / 18.12 dB 35.90s / 0.54385 / 0.49905 / 10.03 dB 33.77s / 0.99477 / 0.95100 / 30.62 dB 1.85s

(Note: For t2i_think_s42, <think> generates autoregressive reasoning tokens before image diffusion, so a slightly differently phrased <think> output naturally shifts pixel layout metrics while editing (edit_cabin_winter_s42) achieves 0.99582 Cosine / 0.95865 SSIM / 32.08 dB PSNR with INT8 ConvRot and 0.99503 / 0.95232 / 31.08 dB with W6/W8 Hybrid ConvRot.)


ComfyUI Setup & Usage

  1. Install the patched ComfyUI-BAGEL custom node: Copy ComfyUI-BAGEL/ (or extract ComfyUI-BAGEL-a1a19a40e619.tar.gz) into:
    ComfyUI/custom_nodes/ComfyUI-BAGEL/
    
  2. Download a checkpoint into ComfyUI/models/bagel/:
    • Direct plug-and-play (recommended): Download bagel-7b-mot-int8-convrot.safetensors straight into ComfyUI/models/bagel/.
    • Packed INT6 download: Download a *-packed.safetensors file and unpack it losslessly before loading in ComfyUI:
      python scripts/unpack_int6.py bagel-7b-mot-w6w8-hybrid-convrot-packed.safetensors ComfyUI/models/bagel/bagel-7b-mot-w6w8-hybrid-convrot.safetensors
      
  3. Ensure BAGEL's VAE (ae.safetensors) is placed in ComfyUI/models/vae/.
  4. Hardware note: Requires a CUDA GPU with native bfloat16 support (Ampere / Ada / Hopper / Blackwell, such as RTX 3090 / 4090 / L4 / G4 / A100) and PyTorch/ComfyUI INT8 quant ops support (ComfyUI >= v0.45.0).

License & Attribution

Licensed under Apache License 2.0, matching the upstream ByteDance-Seed/BAGEL-7B-MoT repository and ComfyUI-BAGEL. See NOTICE for full upstream attribution and a summary of modifications made to the weights and custom node.

Downloads last month
55
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for SUSSYEATING/BAGEL-7B-MoT-ConvRot

Base model

Qwen/Qwen2.5-7B
Quantized
(6)
this model