kimrinking/pplx-decider-flash-v1.2-nvfp4

PPLX Decider Flash v1.2 — NVFP4 / Attention·Gate BF16

An NVFP4 quantization of perplexity-ai/pplx-decider-flash-v1.2. Selected full-attention and linear-attention gate modules retain their original BF16 weights.

Quantization

Component	Configuration
Quantization	NVIDIA ModelOpt NVFP4, Local-Hessian
Calibration requests	160
Retained in BF16	32 full-attention Q/K/V/O modules
Retained in BF16	48 linear-attention in_proj_a / in_proj_b modules
Decision readout	Original BF16
Weight file size	4.38 GB

The retained gate modules are linear-attention in_proj_a and in_proj_b; MLP gate_proj is not included.

Performance

Measured on DGX Spark GB10 over 5 independent server starts per model. The machine was rebooted when switching models, and the same cache was restored for each run. Throughput is reported as mean ± sample standard deviation.

Model	Accuracy	Throughput, concurrency 1	Throughput, concurrency 8
NVFP4 / Attention·Gate BF16	636/838 · 75.895%	24.742 ± 0.096 req/s	38.221 ± 0.316 req/s
FP8	648/838 · 77.327%	20.384 ± 0.167 req/s	30.125 ± 0.141 req/s
BF16	647/838 · 77.208%	13.340 ± 0.077 req/s	20.225 ± 0.302 req/s

At concurrency 8, throughput was 26.9% higher than FP8, with accuracy 1.432 percentage points lower. The FP8 baseline was quantized by vLLM at load time from the same BF16 source.

Accuracy was measured with the custom DecisionCore evaluation, not the official Decision Index. The 2 shared over-length items were excluded from the full set of 840 questions. On the 695 questions remaining after excluding known calibration-data overlap, accuracy was 539/695 · 77.554%.

Throughput was measured after warm-up using 1,024 requests per concurrency setting. Input length averaged 370.1 tokens, with a range of 95–3,576 tokens. Per-run results are included in provenance/benchmark-summary.json.

Reproducibility

Across 5 starts with the same cache, all selected answers and probabilities were identical for the 838 questions.

In a separate fresh-cache check, the number of correct answers was unchanged, but 2 selected answers differed. A control run using the original tuning files reproduced all original selected answers.

Running the model

Validated environment: DGX Spark GB10, Linux aarch64, CUDA 13.0, vLLM 0.32.0, PyTorch 2.13.0+cu130, Transformers 5.19.0.

Run the included vLLM adapter in a compatible environment. runtime/requirements-validated.txt records the versions observed in the validated image; it is not a lock file validated by a fresh installation. Prepare CUDA builds of vLLM and PyTorch appropriate for your GPU and platform.

source/pyproject.toml preserves the upstream reference implementation and lists Torch and Transformers versions that differ from the runtime above. The included server imports source/src directly. Do not overwrite the serving environment's dependencies with pip install ./source; use runtime/requirements-validated.txt to check runtime versions.

The container image used for validation is private. A fresh dependency installation in a public image, or execution from a newly downloaded Hub snapshot, has not been validated.

bash runtime/start.sh

Once the engine is ready, run the example:

python3 runtime/example.py --base-url http://127.0.0.1:8140

The API exposes POST /v1/decisions and POST /v1/systemone. Both accept state and questions and return per-option probabilities and decisions. The current adapter supports text input.

License

Apache-2.0. The package includes the upstream LICENSE and NOTICE, quantization settings, and records of the modules retained in BF16.