turboquant-elyan / README.md
elyanlabs's picture
Source snapshot of github.com/Scottcjn/turboquant-elyan@ee32f7d
5d35978 verified
|
Raw History Blame Contribute Delete
7.8 kB
---
license: mit
tags:
- elyan-labs
- llm-inference
- kv-cache
- quantization
- power8
- cuda
---
> Source snapshot of [github.com/Scottcjn/turboquant-elyan](https://github.com/Scottcjn/turboquant-elyan), also on [GitLab](https://gitlab.com/elyan-labs/turboquant-elyan) with full history. Code only: nothing here runs on Hugging Face infrastructure.
[![BCOS Certified](https://img.shields.io/badge/BCOS-Certified-brightgreen?style=flat)](BCOS.md)
# TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 KV Cache Compression
TurboQuant-Elyan is a clean-room Elyan Labs implementation of TurboQuant TQ3 KV-cache quantization for IBM POWER8 VSX and NVIDIA CUDA. It provides `ggml-turboquant-vsx.h` and `turboquant_cuda_kernel.cu` for 128-float blocks compressed into 56-byte TQ3 blocks, with scalar fallback support in the VSX header and llama.cpp-style quantization entry points.
Production-ready implementations of Google's [TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) (ICLR 2026) 3-bit KV cache quantization for **IBM POWER8 VSX** and **NVIDIA CUDA** GPUs.
Built clean-room by [Elyan Labs](https://github.com/Scottcjn) from the published paper. No official Google code was used.
For LLM, search, and answer-engine context, see [`llms.txt`](llms.txt). It summarizes the repository scope, canonical files, supported platforms, benchmark caveats, and citation guidance without changing the implementation.
## What Is TurboQuant?
TurboQuant compresses the Key-Value cache in transformer attention from FP16/FP32 down to **3 bits per value** with near-zero quality loss. This is the biggest memory bottleneck during LLM and video model inference.
| Metric | Before | After TQ3 |
|--------|--------|-----------|
| KV cache per 128 values | 512 bytes (FP32) / 256 bytes (FP16) | **56 bytes** |
| Compression ratio | 1x | **9.1x vs FP32 / 4.6x vs FP16** |
| Quality loss (NMSE) | 0% | **~4.1%** (cosine sim > 0.97) |
| Retraining required | - | **None** (training-free) |
## Platforms
| Platform | File | Status | Throughput |
|----------|------|--------|------------|
| **IBM POWER8 VSX** | `ggml-turboquant-vsx.h` | Tested on POWER8 S824 | 474K vec/s dequant |
| **NVIDIA CUDA** | `turboquant_cuda_kernel.cu` | Tested on V100-SXM2-32GB, M40, RTX 4070 | **97.5M vec/s** dequant |
Cross-platform **bit-exact** compatibility: identical norm, scale and packed indices across backends.
This is a property of the reduction *order*, not just of the arithmetic. Float
addition is not associative, so a sequential accumulator, a 4-lane `vec_madd`
loop and a warp-shuffle tree give three different L2 norms in the last bit —
and `norm` scales the vector the codebook indices are derived from. All three
backends therefore use the same lane layout (accumulator `j` takes elements
`j, j+4, j+8, ...` with a fused multiply-add, combined left to right).
`tests/test_turboquant_vsx.c` pins a golden block; run the suite on each
platform you ship to and the claim stays honest.
## Algorithm
1. **L2 normalize** the 128-float KV vector
2. **Deterministic sign flips** (xoshiro256** PRNG) — the `D` of the rotation
3. **Fast Walsh-Hadamard Transform** (7 butterfly stages) to Gaussianize the distribution
4. **Absmax scale** to [-1, +1] range
5. **Lloyd-Max 8-level codebook** quantization (3 bits per coordinate)
6. **3-bit packing** into 48 bytes + 4-byte norm + 4-byte scale = 56 bytes
Dequantize is the exact reverse.
Steps 2 and 3 are the randomized Hadamard transform `H*D`, and the order is
not free: `D` has to be applied **before** `H`. Applied afterwards the flips
are inert — absmax is invariant under a sign flip and the codebook is
symmetric about zero, so quantizing `s*v` and multiplying the centroid by `s`
again reproduces exactly what quantizing `v` produces. Blocks that `H` alone
concentrates (a DC offset, a low-frequency ramp, a Hadamard row — all ordinary
KV cache shapes) then round-trip at cosine ~0.33 instead of ~0.96.
### Lloyd-Max Centroids (d=128 Beta distribution)
```c
// 8 centroids, MSE-optimal for transformed distribution
{-0.9816f, -0.6168f, -0.3479f, -0.1129f, 0.1129f, 0.3479f, 0.6168f, 0.9816f}
```
These are the levels the code ships, in `TQ3_CENTROIDS` and
`d_TQ3_CENTROIDS`. They are deliberately *not* the Lloyd-Max table for a
plain `N(0,1)` source — that one is `{+/-0.2451, +/-0.7559, +/-1.3438,
+/-2.1519}`, whose outer levels do not fit in `[-1,+1]`. The quantizer input
here is a Hadamard coefficient divided by the block absmax, which
concentrates it near zero, so the levels are denser there.
`tests/check_backends_agree.sh` checks that this table, the header and the
CUDA kernel still say the same thing.
## Quick Start
### POWER8 VSX
```bash
# Self-test (on POWER8)
cat > test.c << 'EOF'
#include "ggml-turboquant-vsx.h"
int main(void) { return tq3_self_test(); }
EOF
gcc -mcpu=power8 -mvsx -maltivec -O2 -std=c11 -o tq3_test test.c -lm
./tq3_test
```
### CUDA (V100/5070/3060/M40)
```bash
# Self-test
nvcc -arch=sm_70 -O2 -o tq3_cuda_test turboquant_cuda_kernel.cu -DTURBOQUANT_SELF_TEST
./tq3_cuda_test
```
### x86 Scalar Fallback
The VSX header includes a scalar fallback that works on any platform:
```bash
gcc -O2 -std=c11 -o tq3_test test.c -lm # no -mvsx needed
./tq3_test
```
### Test suite
```bash
make test # full suite for this machine
make test-vsx # cross-build for POWER8 (run on hardware or under qemu)
sh tests/check_backends_agree.sh # header, CUDA kernel and README agree
```
## Benchmark Results
### POWER8 S824 (512GB RAM, 128 threads)
| Operation | Throughput | Bandwidth |
|-----------|-----------|-----------|
| Quantize | 249,247 vec/s | 127.6 MB/s |
| Dequantize | 474,182 vec/s | 242.8 MB/s |
### V100-SXM2-32GB (CUDA SM_70)
| Operation | Throughput | Bandwidth |
|-----------|-----------|-----------|
| Quantize | 87,576,227 vec/s | 44.8 GB/s |
| Dequantize | 97,539,209 vec/s | 49.9 GB/s |
The V100 dequant bandwidth (49.9 GB/s) means TQ3 decompression is **hidden by HBM2 latency** -- effectively free.
## Use Cases
- **LLM inference**: Fit 6x longer contexts in the same VRAM. A 70B model's KV cache at 128K context drops from ~40GB to ~7GB.
- **Video generation (DiT)**: LTX-2, HunyuanVideo, Wan2.1 use transformer attention. TQ3 enables 6x longer video clips or higher resolution on the same GPU.
- **POWER8 inference**: Combined with PSE vec_perm collapse, enables 512GB RAM to serve as ~4.5TB effective KV cache.
- **Heterogeneous fleets**: Bit-exact cross-platform means a model can quantize KV on GPU and serve from CPU (or vice versa).
## Integration with llama.cpp
The headers follow the ggml quantization convention:
```c
// Quantize k floats into TQ3 blocks
void quantize_row_tq3_vsx(const float *x, void *y, int64_t k);
// Dequantize TQ3 blocks back to floats
void dequantize_row_tq3_vsx(const void *vx, float *y, int64_t k);
```
To add to an existing llama.cpp POWER8 build, include the header in `quants.c`:
```c
#include "ggml-turboquant-vsx.h"
```
## Related Work
- [ram-coffers](https://github.com/Scottcjn/ram-coffers) -- NUMA-aware weight banking and neuromorphic cognitive routing for POWER8
- [Google TurboQuant Paper](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) (ICLR 2026)
- [llama.cpp TurboQuant Discussion](https://github.com/ggml-org/llama.cpp/discussions/20969)
## License
MIT License. (c) 2026 Elyan Labs.
## Citation
If you use this implementation in research or production, please cite:
```
@software{turboquant_elyan_2026,
title={TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 Implementation},
author={Boudreaux, Scott and Elyan Labs},
year={2026},
url={https://github.com/Scottcjn/turboquant-elyan}
}
```