|
Download README.md from elyanlabs/turboquant-elyan: direct link, hf CLI and curl.
- Browser
- Download file 7.8 kB
-
https://huggingface.co/elyanlabs/turboquant-elyan/resolve/main/README.md
- Command line
-
hf download hf://elyanlabs/turboquant-elyan/README.md
-
curl -L -o README.md https://huggingface.co/elyanlabs/turboquant-elyan/resolve/main/README.md
7.8 kB
| license: mit | |
| tags: | |
| - elyan-labs | |
| - llm-inference | |
| - kv-cache | |
| - quantization | |
| - power8 | |
| - cuda | |
| > Source snapshot of [github.com/Scottcjn/turboquant-elyan](https://github.com/Scottcjn/turboquant-elyan), also on [GitLab](https://gitlab.com/elyan-labs/turboquant-elyan) with full history. Code only: nothing here runs on Hugging Face infrastructure. | |
| [](BCOS.md) | |
| # TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 KV Cache Compression | |
| TurboQuant-Elyan is a clean-room Elyan Labs implementation of TurboQuant TQ3 KV-cache quantization for IBM POWER8 VSX and NVIDIA CUDA. It provides `ggml-turboquant-vsx.h` and `turboquant_cuda_kernel.cu` for 128-float blocks compressed into 56-byte TQ3 blocks, with scalar fallback support in the VSX header and llama.cpp-style quantization entry points. | |
| Production-ready implementations of Google's [TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) (ICLR 2026) 3-bit KV cache quantization for **IBM POWER8 VSX** and **NVIDIA CUDA** GPUs. | |
| Built clean-room by [Elyan Labs](https://github.com/Scottcjn) from the published paper. No official Google code was used. | |
| For LLM, search, and answer-engine context, see [`llms.txt`](llms.txt). It summarizes the repository scope, canonical files, supported platforms, benchmark caveats, and citation guidance without changing the implementation. | |
| ## What Is TurboQuant? | |
| TurboQuant compresses the Key-Value cache in transformer attention from FP16/FP32 down to **3 bits per value** with near-zero quality loss. This is the biggest memory bottleneck during LLM and video model inference. | |
| | Metric | Before | After TQ3 | | |
| |--------|--------|-----------| | |
| | KV cache per 128 values | 512 bytes (FP32) / 256 bytes (FP16) | **56 bytes** | | |
| | Compression ratio | 1x | **9.1x vs FP32 / 4.6x vs FP16** | | |
| | Quality loss (NMSE) | 0% | **~4.1%** (cosine sim > 0.97) | | |
| | Retraining required | - | **None** (training-free) | | |
| ## Platforms | |
| | Platform | File | Status | Throughput | | |
| |----------|------|--------|------------| | |
| | **IBM POWER8 VSX** | `ggml-turboquant-vsx.h` | Tested on POWER8 S824 | 474K vec/s dequant | | |
| | **NVIDIA CUDA** | `turboquant_cuda_kernel.cu` | Tested on V100-SXM2-32GB, M40, RTX 4070 | **97.5M vec/s** dequant | | |
| Cross-platform **bit-exact** compatibility: identical norm, scale and packed indices across backends. | |
| This is a property of the reduction *order*, not just of the arithmetic. Float | |
| addition is not associative, so a sequential accumulator, a 4-lane `vec_madd` | |
| loop and a warp-shuffle tree give three different L2 norms in the last bit — | |
| and `norm` scales the vector the codebook indices are derived from. All three | |
| backends therefore use the same lane layout (accumulator `j` takes elements | |
| `j, j+4, j+8, ...` with a fused multiply-add, combined left to right). | |
| `tests/test_turboquant_vsx.c` pins a golden block; run the suite on each | |
| platform you ship to and the claim stays honest. | |
| ## Algorithm | |
| 1. **L2 normalize** the 128-float KV vector | |
| 2. **Deterministic sign flips** (xoshiro256** PRNG) — the `D` of the rotation | |
| 3. **Fast Walsh-Hadamard Transform** (7 butterfly stages) to Gaussianize the distribution | |
| 4. **Absmax scale** to [-1, +1] range | |
| 5. **Lloyd-Max 8-level codebook** quantization (3 bits per coordinate) | |
| 6. **3-bit packing** into 48 bytes + 4-byte norm + 4-byte scale = 56 bytes | |
| Dequantize is the exact reverse. | |
| Steps 2 and 3 are the randomized Hadamard transform `H*D`, and the order is | |
| not free: `D` has to be applied **before** `H`. Applied afterwards the flips | |
| are inert — absmax is invariant under a sign flip and the codebook is | |
| symmetric about zero, so quantizing `s*v` and multiplying the centroid by `s` | |
| again reproduces exactly what quantizing `v` produces. Blocks that `H` alone | |
| concentrates (a DC offset, a low-frequency ramp, a Hadamard row — all ordinary | |
| KV cache shapes) then round-trip at cosine ~0.33 instead of ~0.96. | |
| ### Lloyd-Max Centroids (d=128 Beta distribution) | |
| ```c | |
| // 8 centroids, MSE-optimal for transformed distribution | |
| {-0.9816f, -0.6168f, -0.3479f, -0.1129f, 0.1129f, 0.3479f, 0.6168f, 0.9816f} | |
| ``` | |
| These are the levels the code ships, in `TQ3_CENTROIDS` and | |
| `d_TQ3_CENTROIDS`. They are deliberately *not* the Lloyd-Max table for a | |
| plain `N(0,1)` source — that one is `{+/-0.2451, +/-0.7559, +/-1.3438, | |
| +/-2.1519}`, whose outer levels do not fit in `[-1,+1]`. The quantizer input | |
| here is a Hadamard coefficient divided by the block absmax, which | |
| concentrates it near zero, so the levels are denser there. | |
| `tests/check_backends_agree.sh` checks that this table, the header and the | |
| CUDA kernel still say the same thing. | |
| ## Quick Start | |
| ### POWER8 VSX | |
| ```bash | |
| # Self-test (on POWER8) | |
| cat > test.c << 'EOF' | |
| #include "ggml-turboquant-vsx.h" | |
| int main(void) { return tq3_self_test(); } | |
| EOF | |
| gcc -mcpu=power8 -mvsx -maltivec -O2 -std=c11 -o tq3_test test.c -lm | |
| ./tq3_test | |
| ``` | |
| ### CUDA (V100/5070/3060/M40) | |
| ```bash | |
| # Self-test | |
| nvcc -arch=sm_70 -O2 -o tq3_cuda_test turboquant_cuda_kernel.cu -DTURBOQUANT_SELF_TEST | |
| ./tq3_cuda_test | |
| ``` | |
| ### x86 Scalar Fallback | |
| The VSX header includes a scalar fallback that works on any platform: | |
| ```bash | |
| gcc -O2 -std=c11 -o tq3_test test.c -lm # no -mvsx needed | |
| ./tq3_test | |
| ``` | |
| ### Test suite | |
| ```bash | |
| make test # full suite for this machine | |
| make test-vsx # cross-build for POWER8 (run on hardware or under qemu) | |
| sh tests/check_backends_agree.sh # header, CUDA kernel and README agree | |
| ``` | |
| ## Benchmark Results | |
| ### POWER8 S824 (512GB RAM, 128 threads) | |
| | Operation | Throughput | Bandwidth | | |
| |-----------|-----------|-----------| | |
| | Quantize | 249,247 vec/s | 127.6 MB/s | | |
| | Dequantize | 474,182 vec/s | 242.8 MB/s | | |
| ### V100-SXM2-32GB (CUDA SM_70) | |
| | Operation | Throughput | Bandwidth | | |
| |-----------|-----------|-----------| | |
| | Quantize | 87,576,227 vec/s | 44.8 GB/s | | |
| | Dequantize | 97,539,209 vec/s | 49.9 GB/s | | |
| The V100 dequant bandwidth (49.9 GB/s) means TQ3 decompression is **hidden by HBM2 latency** -- effectively free. | |
| ## Use Cases | |
| - **LLM inference**: Fit 6x longer contexts in the same VRAM. A 70B model's KV cache at 128K context drops from ~40GB to ~7GB. | |
| - **Video generation (DiT)**: LTX-2, HunyuanVideo, Wan2.1 use transformer attention. TQ3 enables 6x longer video clips or higher resolution on the same GPU. | |
| - **POWER8 inference**: Combined with PSE vec_perm collapse, enables 512GB RAM to serve as ~4.5TB effective KV cache. | |
| - **Heterogeneous fleets**: Bit-exact cross-platform means a model can quantize KV on GPU and serve from CPU (or vice versa). | |
| ## Integration with llama.cpp | |
| The headers follow the ggml quantization convention: | |
| ```c | |
| // Quantize k floats into TQ3 blocks | |
| void quantize_row_tq3_vsx(const float *x, void *y, int64_t k); | |
| // Dequantize TQ3 blocks back to floats | |
| void dequantize_row_tq3_vsx(const void *vx, float *y, int64_t k); | |
| ``` | |
| To add to an existing llama.cpp POWER8 build, include the header in `quants.c`: | |
| ```c | |
| #include "ggml-turboquant-vsx.h" | |
| ``` | |
| ## Related Work | |
| - [ram-coffers](https://github.com/Scottcjn/ram-coffers) -- NUMA-aware weight banking and neuromorphic cognitive routing for POWER8 | |
| - [Google TurboQuant Paper](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) (ICLR 2026) | |
| - [llama.cpp TurboQuant Discussion](https://github.com/ggml-org/llama.cpp/discussions/20969) | |
| ## License | |
| MIT License. (c) 2026 Elyan Labs. | |
| ## Citation | |
| If you use this implementation in research or production, please cite: | |
| ``` | |
| @software{turboquant_elyan_2026, | |
| title={TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 Implementation}, | |
| author={Boudreaux, Scott and Elyan Labs}, | |
| year={2026}, | |
| url={https://github.com/Scottcjn/turboquant-elyan} | |
| } | |
| ``` | |