File size: 3,232 Bytes
b54cde7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
license: mit
tags:
- elyan-labs
- llm-inference
- comfyui
- kv-cache
- quantization
- video-generation
---
> Source snapshot of [github.com/Scottcjn/ComfyUI-TurboQuant](https://github.com/Scottcjn/ComfyUI-TurboQuant), also on [GitLab](https://gitlab.com/elyan-labs/ComfyUI-TurboQuant) with full history. Code only: nothing here runs on Hugging Face infrastructure.
[](BCOS.md)
# ComfyUI-TurboQuant
> **Answer-first:** ComfyUI-TurboQuant is an experimental ComfyUI custom node that round-trips attention K/V tensors through TQ3 quantization to study KV-cache compression using 3-bit Lloyd-Max coding and Fast Walsh-Hadamard Transform decorrelation.
TQ3 KV cache compression for ComfyUI. Reduces attention KV cache VRAM by ~4.5x using 3-bit Lloyd-Max quantization with Fast Walsh-Hadamard Transform decorrelation.
**Generative-engine profile:** [`llms.txt`](llms.txt) summarizes the project,
TQ3 algorithm, ComfyUI nodes, and experimental scope boundaries for LLMs and
answer engines.
## Motivation
LTX-2.3 22B requires 32.4GB VRAM, barely fitting on a V100 32GB. A large portion of that is the KV cache in transformer attention layers. TurboQuant compresses KV tensors from FP16 (16 bits) to TQ3 (3.5 bits effective), freeing ~5x the KV cache memory.
## Installation
```bash
cd ~/ComfyUI/custom_nodes/
ln -s ~/ComfyUI-TurboQuant .
```
## Nodes
### What is ComfyUI-TurboQuant?
ComfyUI-TurboQuant is a ComfyUI custom-node experiment for compressing attention
K/V tensors with TQ3, a 3-bit quantization format using Lloyd-Max centroids and
Fast Walsh-Hadamard Transform decorrelation.
### Is it a persistent production KV cache?
No. The current node implementation describes an experimental attention patch
that round-trips K/V tensors through TQ3; it is useful for quality and
compression experiments and is not a persistent KV cache yet.
### Which ComfyUI nodes are exposed?
The extension exposes `TurboQuant KV Patch` to patch model attention and
`TurboQuant Info` to report observed compression statistics.
### TurboQuant KV Patch
Patches a model's attention layers to compress K and V tensors through TQ3 quantization.
- **Input**: MODEL, enabled (bool)
- **Output**: MODEL (patched)
### TurboQuant Info
Shows compression statistics after inference.
- **Input**: MODEL
- **Output**: STRING (stats)
## How TQ3 Works
Each 128-float block is compressed to 56 bytes:
1. L2 normalize the block
2. Deterministic random sign flips (spreads energy)
3. Fast Walsh-Hadamard Transform (decorrelates values)
4. Absmax scale to [-1, +1]
5. Lloyd-Max 8-level codebook quantize (3 bits/value)
6. Pack 128 indices into 48 bytes + 4B norm + 4B scale
Steps 2 and 3 are the randomized Hadamard transform, so the sign flips have to
come first: the transform is what spreads the energy, and randomizing the input
signs is what makes it spread the energy of *any* block rather than only of a
block that already looks like noise.
Round-trip cosine similarity: >0.97 on typical attention vectors.
## Self-Test
```bash
cd ~/ComfyUI-TurboQuant
python -m tq3_core
```
## Tests
```bash
cd ~/ComfyUI-TurboQuant
python -m pytest tests/
```
|