Instructions to use jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality

SGLang

How to use jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality with Docker Model Runner:
```
docker model run hf.co/jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality
```

Laguna-S-2.1-NVFP4-W4A16-Quality

Quality-first mixed-precision ModelOpt quantization of poolside/Laguna-S-2.1, exported directly from Poolside's BF16 checkpoint. This is a serving-oriented checkpoint, not a smallest-possible or single-GPU quant.

The guiding rule was to leave the model's signal path intact and quantize only the routed experts. All attention projections, shared experts, MoE routers, embeddings, dense layer 0, and the LM head remain BF16. In particular, no shared expert in layer 47 was quantized.

Terminal-Bench 2.1 with Poolside's PoolCLI harness is running now. Results will be added when the matched BF16, Poolside NVFP4, and this checkpoint campaigns finish. The KLD results below are complete five-run gates; they are not a substitute for downstream benchmark results.

Precision placement

Module	Precision
Dense layer 0	BF16
Routed experts, layers 1-5	BF16
Routed experts, layers 6-8	W8A8 FP8
Routed experts, layers 9-47	static W4A16 NVFP4, group size 16
Attention, shared experts, routers, embeddings, LM head	BF16

39 routed-expert layers use W4A16 NVFP4.
3 routed-expert layers use FP8.
5 routed-expert layers remain BF16.
Effective routed-expert weight precision: 5.9468 bits.
Audited export: 92,469,054,276 bytes, 10 shards, 101,377 tensors.
ModelOpt producer: 0.46.0.dev184+g8813b7001.

Placement was selected with KL-guided AutoQuantize over 512 samples from nemotron-post-training-v3, 512 tokens per sample. The chosen placement was then rebuilt directly from BF16 using FP8 scale sweep and static MSE-scaled W4A16 NVFP4. The exact replay recipe is included as quantization_recipe.yaml.

Five-run KLD

KL direction is KL(BF16 reference || candidate). Every row uses the same hash-pinned 2,048-token prompt, 2,047 next-token positions, and 100,352 logits per position. Runs used TP4, BF16 compute, eager execution, seed 0, no prefix cache, and a fresh engine per run. Quant candidates used the same B12X A16 expert path. Lower is better.

Weights	KV cache	Mean KLD	Sample SD	Range
This checkpoint	BF16	0.237254	0.004783	0.230693-0.242978
Poolside NVFP4 `216d1f13`	BF16	0.338808	0.009480	0.325988-0.349905
BF16 `a50e85e7`	dynamic FP8 per-token/head	0.247317	0.003392	0.242902-0.251436
This checkpoint	dynamic FP8 per-token/head	0.269351	0.012157	0.254391-0.282603
Poolside NVFP4 `216d1f13`	dynamic FP8 per-token/head	0.337315	0.012306	0.322523-0.349913

Against the pinned Poolside NVFP4 checkpoint, this export reduced mean KLD by 29.97% with BF16 KV and 20.15% with matched dynamic FP8 KV. KLD values are specific to this model and protocol and should not be compared directly to numbers from other model families.

Reference provenance:

BF16 weights: poolside/Laguna-S-2.1@a50e85e7e0aae7b0a504d156bd36a616ec9fea38
Poolside NVFP4 comparison: poolside/Laguna-S-2.1-NVFP4@216d1f13878dd4e715bc7412848d0f330e95bba6
Prompt token IDs SHA-256: 1f4a417c5159d1ac1202e38c0cb4573955d8ceb2037d8e316b7c3344267959d6
BF16 reference logits SHA-256: beaf167ad42de75af7a0f33554f37c2672092eed175ccaa76cb726e684bca109
Quant config SHA-256: 2102d1290212d218211efbaf0e6b670666a7ea47a11bf255c4980e021899f6af
Safetensors index SHA-256: 0a12b7fa0676018cefe4f4c518f60ee6550d94388273bc7e4cfef027604ae68a

The five aggregate JSON files, placement audit, and checkpoint audit are shipped in this repository.

Serving

This checkpoint is intended for Blackwell GPUs and a W4A16-capable vLLM build. Start with BF16 KV when evaluating quality, then test FP8 KV separately for capacity.

vllm serve jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality \
  --trust-remote-code \
  --dtype bfloat16 \
  --quantization auto \
  --tensor-parallel-size 2 \
  --kv-cache-dtype bfloat16

The repository includes Poolside's updated tokenizer from BF16 revision 179ee67cf0fff5391c67fe1a392ea849fa6d643f: thinking is enabled by default, reasoning is preserved across turns, </assistant> is special, and the old 32K generation cap is removed.

Status and limitations

Terminal-Bench 2.1 is in progress using PoolCLI 1.0.13, four attempts per task, and six-hour agent timeouts.
Estonia and LAVD smoke characterization indicated that this release family is sensitive to reasoning behavior; those observations motivated the matched PoolCLI benchmark rather than ad-hoc prompt scoring.
This is an early public release. Please report loops, tool-call failures, and runtime details with the exact KV-cache dtype and serving commit.
Use is governed by the upstream OpenMDW 1.1 license.

Quantized by Jack Zampolin with the local-inference community's quality-first serving workflow.

Downloads last month: 606

Safetensors

Model size

70B params

Tensor type

BF16

F8_E4M3

Model tree for jackzampolin/Laguna-S-2.1-NVFP4-W4A16-Quality

Base model

poolside/Laguna-S-2.1

Quantized

(70)

this model