--- license: mit tags: - elyan-labs - llm-inference - kv-cache - quantization - power8 - cuda --- > Source snapshot of [github.com/Scottcjn/turboquant-elyan](https://github.com/Scottcjn/turboquant-elyan), also on [GitLab](https://gitlab.com/elyan-labs/turboquant-elyan) with full history. Code only: nothing here runs on Hugging Face infrastructure. [![BCOS Certified](https://img.shields.io/badge/BCOS-Certified-brightgreen?style=flat)](BCOS.md) # TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 KV Cache Compression TurboQuant-Elyan is a clean-room Elyan Labs implementation of TurboQuant TQ3 KV-cache quantization for IBM POWER8 VSX and NVIDIA CUDA. It provides `ggml-turboquant-vsx.h` and `turboquant_cuda_kernel.cu` for 128-float blocks compressed into 56-byte TQ3 blocks, with scalar fallback support in the VSX header and llama.cpp-style quantization entry points. Production-ready implementations of Google's [TurboQuant](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) (ICLR 2026) 3-bit KV cache quantization for **IBM POWER8 VSX** and **NVIDIA CUDA** GPUs. Built clean-room by [Elyan Labs](https://github.com/Scottcjn) from the published paper. No official Google code was used. For LLM, search, and answer-engine context, see [`llms.txt`](llms.txt). It summarizes the repository scope, canonical files, supported platforms, benchmark caveats, and citation guidance without changing the implementation. ## What Is TurboQuant? TurboQuant compresses the Key-Value cache in transformer attention from FP16/FP32 down to **3 bits per value** with near-zero quality loss. This is the biggest memory bottleneck during LLM and video model inference. | Metric | Before | After TQ3 | |--------|--------|-----------| | KV cache per 128 values | 512 bytes (FP32) / 256 bytes (FP16) | **56 bytes** | | Compression ratio | 1x | **9.1x vs FP32 / 4.6x vs FP16** | | Quality loss (NMSE) | 0% | **~4.1%** (cosine sim > 0.97) | | Retraining required | - | **None** (training-free) | ## Platforms | Platform | File | Status | Throughput | |----------|------|--------|------------| | **IBM POWER8 VSX** | `ggml-turboquant-vsx.h` | Tested on POWER8 S824 | 474K vec/s dequant | | **NVIDIA CUDA** | `turboquant_cuda_kernel.cu` | Tested on V100-SXM2-32GB, M40, RTX 4070 | **97.5M vec/s** dequant | Cross-platform **bit-exact** compatibility: identical norm, scale and packed indices across backends. This is a property of the reduction *order*, not just of the arithmetic. Float addition is not associative, so a sequential accumulator, a 4-lane `vec_madd` loop and a warp-shuffle tree give three different L2 norms in the last bit — and `norm` scales the vector the codebook indices are derived from. All three backends therefore use the same lane layout (accumulator `j` takes elements `j, j+4, j+8, ...` with a fused multiply-add, combined left to right). `tests/test_turboquant_vsx.c` pins a golden block; run the suite on each platform you ship to and the claim stays honest. ## Algorithm 1. **L2 normalize** the 128-float KV vector 2. **Deterministic sign flips** (xoshiro256** PRNG) — the `D` of the rotation 3. **Fast Walsh-Hadamard Transform** (7 butterfly stages) to Gaussianize the distribution 4. **Absmax scale** to [-1, +1] range 5. **Lloyd-Max 8-level codebook** quantization (3 bits per coordinate) 6. **3-bit packing** into 48 bytes + 4-byte norm + 4-byte scale = 56 bytes Dequantize is the exact reverse. Steps 2 and 3 are the randomized Hadamard transform `H*D`, and the order is not free: `D` has to be applied **before** `H`. Applied afterwards the flips are inert — absmax is invariant under a sign flip and the codebook is symmetric about zero, so quantizing `s*v` and multiplying the centroid by `s` again reproduces exactly what quantizing `v` produces. Blocks that `H` alone concentrates (a DC offset, a low-frequency ramp, a Hadamard row — all ordinary KV cache shapes) then round-trip at cosine ~0.33 instead of ~0.96. ### Lloyd-Max Centroids (d=128 Beta distribution) ```c // 8 centroids, MSE-optimal for transformed distribution {-0.9816f, -0.6168f, -0.3479f, -0.1129f, 0.1129f, 0.3479f, 0.6168f, 0.9816f} ``` These are the levels the code ships, in `TQ3_CENTROIDS` and `d_TQ3_CENTROIDS`. They are deliberately *not* the Lloyd-Max table for a plain `N(0,1)` source — that one is `{+/-0.2451, +/-0.7559, +/-1.3438, +/-2.1519}`, whose outer levels do not fit in `[-1,+1]`. The quantizer input here is a Hadamard coefficient divided by the block absmax, which concentrates it near zero, so the levels are denser there. `tests/check_backends_agree.sh` checks that this table, the header and the CUDA kernel still say the same thing. ## Quick Start ### POWER8 VSX ```bash # Self-test (on POWER8) cat > test.c << 'EOF' #include "ggml-turboquant-vsx.h" int main(void) { return tq3_self_test(); } EOF gcc -mcpu=power8 -mvsx -maltivec -O2 -std=c11 -o tq3_test test.c -lm ./tq3_test ``` ### CUDA (V100/5070/3060/M40) ```bash # Self-test nvcc -arch=sm_70 -O2 -o tq3_cuda_test turboquant_cuda_kernel.cu -DTURBOQUANT_SELF_TEST ./tq3_cuda_test ``` ### x86 Scalar Fallback The VSX header includes a scalar fallback that works on any platform: ```bash gcc -O2 -std=c11 -o tq3_test test.c -lm # no -mvsx needed ./tq3_test ``` ### Test suite ```bash make test # full suite for this machine make test-vsx # cross-build for POWER8 (run on hardware or under qemu) sh tests/check_backends_agree.sh # header, CUDA kernel and README agree ``` ## Benchmark Results ### POWER8 S824 (512GB RAM, 128 threads) | Operation | Throughput | Bandwidth | |-----------|-----------|-----------| | Quantize | 249,247 vec/s | 127.6 MB/s | | Dequantize | 474,182 vec/s | 242.8 MB/s | ### V100-SXM2-32GB (CUDA SM_70) | Operation | Throughput | Bandwidth | |-----------|-----------|-----------| | Quantize | 87,576,227 vec/s | 44.8 GB/s | | Dequantize | 97,539,209 vec/s | 49.9 GB/s | The V100 dequant bandwidth (49.9 GB/s) means TQ3 decompression is **hidden by HBM2 latency** -- effectively free. ## Use Cases - **LLM inference**: Fit 6x longer contexts in the same VRAM. A 70B model's KV cache at 128K context drops from ~40GB to ~7GB. - **Video generation (DiT)**: LTX-2, HunyuanVideo, Wan2.1 use transformer attention. TQ3 enables 6x longer video clips or higher resolution on the same GPU. - **POWER8 inference**: Combined with PSE vec_perm collapse, enables 512GB RAM to serve as ~4.5TB effective KV cache. - **Heterogeneous fleets**: Bit-exact cross-platform means a model can quantize KV on GPU and serve from CPU (or vice versa). ## Integration with llama.cpp The headers follow the ggml quantization convention: ```c // Quantize k floats into TQ3 blocks void quantize_row_tq3_vsx(const float *x, void *y, int64_t k); // Dequantize TQ3 blocks back to floats void dequantize_row_tq3_vsx(const void *vx, float *y, int64_t k); ``` To add to an existing llama.cpp POWER8 build, include the header in `quants.c`: ```c #include "ggml-turboquant-vsx.h" ``` ## Related Work - [ram-coffers](https://github.com/Scottcjn/ram-coffers) -- NUMA-aware weight banking and neuromorphic cognitive routing for POWER8 - [Google TurboQuant Paper](https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/) (ICLR 2026) - [llama.cpp TurboQuant Discussion](https://github.com/ggml-org/llama.cpp/discussions/20969) ## License MIT License. (c) 2026 Elyan Labs. ## Citation If you use this implementation in research or production, please cite: ``` @software{turboquant_elyan_2026, title={TurboQuant-Elyan: POWER8 VSX + CUDA TQ3 Implementation}, author={Boudreaux, Scott and Elyan Labs}, year={2026}, url={https://github.com/Scottcjn/turboquant-elyan} } ```