fraQtl/qwen3-4b-instruct-2507-kv-sidecars
Updated • 21 • 1
KV cache · more users per GPU
Compressed KV cache for vLLM and llama.cpp. Three-arm, needle-gated receipts, reproducible with one command.
Note 9 users at 128K on one A100 where FP8 fits 4. 134.1 tok/s aggregate, 9/9 needles.
fraQtl/mistral-nemo-instruct-2407-kv-sidecars
UpdatedNote 32 users at 32K each on one A100 vs 24 for q8_0 KV. 32/32 needles.
fraQtl/mistral-7b-instruct-v0.3-kv-sidecars
Updated • 14Note 2.4x the fp16 KV pool at 95 to 100% decode speed, 8K and 32K.
fraQtl/fraqtl-sm80-runtime
Updated • 15Note Prebuilt A100 wheel plus a one-command repro script.
fraQtl/fraqtl-membrane-llamacpp-runtime
UpdatedNote llama.cpp runtime: 128K decode 1.79x faster than q8_0 KV on Qwen3-4B.