Relativ3pa1n/Laguna-S-2.1-W4A16

Laguna S 2.1 — W4A16 (group_size=32) — tested on 8× RTX 3090

A W4A16 quantization of poolside/Laguna-S-2.1 (118B total / 8B active, 256-expert MoE, OpenMDW-1.1). Same model, compressed to fit consumer Ampere — see the parent for the full model card, capabilities, and license.

A clean W4A16 quantization of poolside/Laguna-S-2.1 (118B total / 8B active, 256-expert MoE) that serves fast on consumer Ampere GPUs — the cards with no native FP8/FP4. No official W4A16 for Ampere exists (poolside ships QAT-INT4 that loads poorly on Ampere vLLM, and FP8/NVFP4 which are Blackwell-only). This fills that gap.

~92 tok/s decode · 69 GB · 256K context (7.33× concurrent) · clean output with thinking on · tested on 8× RTX 3090 (Ampere, 192 GB VRAM)

TL;DR

Size	69 GB (from 219 GB BF16)
Format	compressed-tensors W4A16, group_size=32, Marlin kernels
Serves on	8× RTX 3090 (192 GB VRAM), TP=8
Throughput	~92 tok/s single-stream (CUDA graphs on)
Context / concurrency	256K, 7.33× concurrent
Output	Clean — including technical/markdown content, thinking on
Terminal-Bench (core 0.1.1, 18-task subset, terminus agent, pass@1)	4/18 = 22.2%

Why this quant exists

Laguna S 2.1 is a top open agentic-coding MoE that fits consumer hardware — but only if you can quantize it without OOMing, pick the right group_size, and serve it without leaving throughput on the floor. The full story (the llmcompressor OOM wall, the per-layer fix, the stale-base red herring, the group_size correction, the cudagraph win) is in LAGUNA-W4A16-BLOG.md.

The short version of the two non-obvious wins:

1. group_size=32, not 128. The Qwen3.5-style default group_size=128 is too coarse for Laguna's 256-expert MoE — large per-group error flips logits at token boundaries and splices stray number/special tokens into technical + markdown output. Poolside's own reference INT4 uses g32. At g32 the corruption disappears. (Set via QuantizationModifier's config_groups, not scheme — the scheme field only accepts preset names.)
2. CUDA graphs on (--enforce-eager removed). With 8B active params, per-step Python + MoE-routing-dispatch overhead dominates the tiny compute in eager mode. Cudagraphs fuse it out: **7 tok/s → ~92 tok/s (12.5×)** at the cost of a hair less KV (7.65× → 7.33× concurrency @ 256K).

Serve (vLLM)

vLLM 0.25.1+, 8× 24 GB Ampere. The cudagraph config is the whole game:

python -m vllm.entrypoints.openai.api_server \
  --model . --served-model-name laguna \
  --tensor-parallel-size 8 --device-ids 0,1,2,3,4,5,6,7 \
  --quantization compressed-tensors \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 --trust-remote-code \
  --host 0.0.0.0 --port 8085

Key flags: --quantization compressed-tensors (required), no --enforce-eager (cudagraph = the 12× lever), --gpu-memory-utilization 0.90 (higher over-reserves KV and OOMs sampler warmup), poolside_v1 tool/reasoning parsers, enable_thinking. Full runnable script: Laguna-S-2.1-W4A16-vLLM.sh.

Terminal-Bench results

18-task random subset of terminal-bench-core==0.1.1, terminus agent, pass@1, default turn budget, 900s agent cap. Honest sample — not the full suite.

Task	Result	Failed sub-tests	Reason
crack-7z-hash.easy	✅ PASS	—	solved (2 sub-tests)
sqlite-with-gcov	✅ PASS	—	solved (3 sub-tests)
prove-plus-comm	✅ PASS	—	solved (4 sub-tests)
sanitize-git-repo	✅ PASS	—	solved (3 sub-tests)
swe-bench-astropy-2	❌ FAIL	test_roundtrip[True]	near-solve — 8/9 sub-tests passed
security-vulhub-minio	❌ FAIL	test_command_output_content	sub-test failed (1/1)
download-youtube	❌ FAIL	test_output_file_exists, test_correct_video	sub-tests failed (2/2)
path-tracing	❌ FAIL	image compile/run/similarity (4)	sub-tests failed (4/4)
blind-maze-explorer-5x5	❌ FAIL	test_maze_map_contents	agent 900s timeout
qemu-alpine-ssh	❌ FAIL	test_sshpass	agent 900s timeout (hard infra)
build-linux-kernel-qemu	❌ FAIL	TypeError in test	agent 900s timeout (hard infra)
sqlite-db-truncate	❌ FAIL	test_json_data	agent 900s timeout
polyglot-c-py	❌ FAIL	—	agent 900s timeout
extract-moves-from-video	❌ FAIL	—	agent 900s timeout (video processing)
chess-best-move	❌ FAIL	test_move_correct	agent 900s timeout
password-recovery	❌ FAIL	test_recovery_file_exists, test_password_match	agent 900s timeout
pytorch-model-cli.hard	❌ FAIL	—	test-harness 120s timeout
cron-broken-network	❌ FAIL	—	test output unparseable (tests didn't run)

Notable: several fails were near-solves (e.g. one task passed 8/9 sub-tests), and the hard infrastructure tasks (QEMU / kernel-build / maze / video-extraction) hit the agent timeout — expected for an 8B-active model on a frontier-calibrated benchmark (the TB paper has frontier models <65% on the full suite).

Sample agent reasoning trace

Laguna drives the terminus agent with structured per-episode reasoning. On the hf-model-inference task (build a Flask sentiment API around a HF model), episode 1:

state_analysis: "Packages installed successfully (torch, transformers, flask). Now need to download the model to /app/model_cache/sentiment_model and create the Flask API." explanation: "Will download the Hugging Face model using the transformers library and save it to the specified cache directory. Then create the Flask API application." commands: mkdir -p /app/model_cache/sentiment_model → python3 -c "from transformers import AutoTokenizer, AutoModelForSequenceClassification; … model.save_pretrained('/app/model_cache/sentiment_model')"

Concrete evidence the quant retains agentic reasoning, not just chat fluency.

License & attribution

• Model weights: OpenMDW-1.1 (poolside). Derivative quant — same license.
• Quant scripts + recipe + blog: MIT-adjacent, use freely.
• Built on an 8×3090 rig (192 GB VRAM, ~251 GB RAM) with llmcompressor 0.12 / transformers 5.10.1 / vLLM 0.25.1.

Files

file	what
model-*.safetensors (4 shards)	the W4A16-g32 weights (69 GB)
config.json	includes quantization_config (group_size=32, Marlin)
Laguna-S-2.1-W4A16-vLLM.sh	the vLLM serve recipe (cudagraph config)
LAGUNA-W4A16-BLOG.md	full quant writeup (5 walls)
w4a16-quant-laguna-s-2.1-perlayer.py	the per-layer quantizer (restricted-RAM)