hoja-solutions/humanizer-NVFP4

humanizer 12B, NVFP4 (W4A4)

This is jialinyyzz/humanizer with both the weights and the input activations of every decoder projection in NVFP4: FP4 (E2M1) values with one FP8 scale per group of 16. On Blackwell GPUs vLLM runs it on the native FP4 tensor cores. The checkpoint is 8.25 GB instead of 23.9 GB.

The NVFP4A16 build, which keeps activations in bf16, is the same size, decodes a single request just as fast, and stays much closer to bf16. Use this W4A4 build only when batched throughput matters more than fidelity.

Quality

The comparison is against bf16 in vLLM 0.30.0 on the authors' held-out evaluation set: 209 English and 102 Chinese drafts, each followed by the authors' first bf16 rewrite, with every draft and rewrite token scored (187,551 positions in English, 99,166 in Chinese).

Build	Top-1 vs. bf16, EN / ZH (avg)	KL to bf16, EN / ZH
bf16, second run in vLLM	99.20% / 99.10% (99.15%)	0.0000 / 0.0001
NVFP4A16	95.03% / 94.51% (94.77%)	0.0120 / 0.0121
NVFP4 W4A4 (this build)	90.86% / 89.25% (90.05%)	0.0427 / 0.0500

KL is computed on bf16's 20 most likely tokens plus one bucket for the rest of the vocabulary, so it reads slightly lower than a full-vocabulary KL. On the authors' own scale (llama.cpp, same evaluation set, different tokens), 90% top-1 is where a standard 3-bit IQ3_M GGUF sits, and their quantization-aware Q4_K_M reaches 95.2%.

Over one rewrite of each of the 311 drafts, this build left a number out of 37 of the 102 Chinese rewrites. Two bf16 samples left one out of 28 and 31. In English it was within the bf16 range. Many of these are dates written another way, not wrong facts.

Why it drifts

Most of the loss comes from the 4-bit activations. Simulating them on top of bf16 weights reproduces most of the drift (full-vocabulary KL 0.038 on 24 of the drafts). It comes from every projection type (attention 0.017, gate/up 0.014, down 0.009) and from every layer, with the later layers worst, so keeping a few layers at 16 bits does not fix it. Gemma 4's hidden states include a few values near 3,000 at the inputs of the first layer. A likely cause is that a group of 16 activations holding one of them shares its single scale with that value, so the other 15 round to zero.

All builds measured the same way, by top-1 average and KL (EN / ZH):

Build	Top-1	KL
W4A4, round to nearest	86.89%	0.0751 / 0.0950
W4A4, importance-weighted scales (imatrix)	88.25%	0.0598 / 0.0739
W4A4, GPTQ (this build)	90.05%	0.0427 / 0.0500
W4A4, GPTQ, down_proj activations in bf16	90.70%	0.0367 / 0.0421
W4A4, GPTQ, attention activations in bf16	91.96%	0.0275 / 0.0325
W4A16, GPTQ with imatrix scales	94.30%	0.0146 / 0.0142
W4A16, GPTQ (the NVFP4A16 build)	94.77%	0.0120 / 0.0121

Speed

RTX 5090, vLLM 0.30.0, 311 drafts, about 144,000 generated tokens:

Build	One request at a time	311 requests, up to 32 at once
bf16	59 tok/s	501 tok/s
NVFP4A16	136 tok/s	1,585 tok/s
NVFP4 W4A4 (this build)	134 tok/s	1,930 tok/s

Usage

The usage is the same as for the NVFP4A16 build and the original model. Use the text-completion prompt from prompt_format.json on /v1/completions, or one draft per message on /v1/chat/completions with the bundled chat_template.jinja, which adds BOS to the authors' GGUF template. Sample with temperature 1.0, top-p 0.95 and top-k off, and no stop strings. generation_config.json sets those defaults (top_k: 0 instead of the original's 64).

vllm serve hoja-solutions/humanizer-NVFP4 --served-model-name humanizer --max-model-len 8192 \
  --limit-mm-per-prompt '{"image":0,"audio":0,"video":0}'

How it was made

GPTQ with llm-compressor 0.14.0 (static activation ordering, block size 128, dampening 0.01) and the NVFP4 scheme on all 328 decoder projections. The activation group scales are computed at run time, under a per-tensor global scale fixed from the calibration data. The token embeddings (tied to the output layer) and the vision and audio projections stay in bf16. The calibration set (384 sequences, 365,486 tokens) is described in the NVFP4A16 card. It comes from train rows of jialinyyzz/humanizer-data, whose DATA.md lists source licenses and the terms of the models that wrote the drafts. The run took 8 minutes on an RTX 5090 and peaked at 20.9 GB of RSS.

Only tested on an RTX 5090 with vLLM 0.30.0, for text input.

Credits

The model, its training data and the evaluation set are by the humanizer authors (jialinyyzz/humanizer, sgaofen/humanizer-local-model), released under Apache-2.0. It is a finetune of google/gemma-4-12B.