Ttimms commited on
Commit
8ccc63f
·
verified ·
1 Parent(s): a626636

docs: full quant comparison table + serving notes + writeup link

Browse files
Files changed (1) hide show
  1. README.md +26 -18
README.md CHANGED
@@ -36,25 +36,32 @@ activations), `lm_head` and embeddings kept in bf16. Produced with
36
  This is the recommended quant for MiniCPM5-2B when you want maximum quality
37
  retention. If you need to fit closer to 2 GB, see the
38
  [NVFP4 W4A16 build](https://huggingface.co/Ttimms/MiniCPM5-2B-NVFP4)
39
- (smaller, ~2–5 pp coding cost).
40
 
41
  ## Evaluation
42
 
43
- `lm-evaluation-harness`, vLLM backend, greedy decoding, **3 draws** each (the
44
- harness is non-deterministic run-to-run even at greedy — median and range are
45
- reported; a single draw is not a reproducible score).
46
-
47
- | build | HumanEval-instruct | MBPP (3-shot) | size |
48
- |---|---:|---:|---:|
49
- | **bf16 base** | 86.59 % (85.98–86.59) | 50.60 % (50.40–51.00) | 4.68 GiB |
50
- | **FP8 (this)** | **84.76 %** (84.15–85.37) | **48.80 %** (48.80–49.00) | **2.84 GiB** |
51
- | Δ vs bf16 | **−1.8 pp** | **−1.8 pp** | −39 % |
52
-
53
- Both deltas sit inside the eval's own run-to-run spread — FP8 is effectively
54
- lossless here. (For contrast, plain RTN NVFP4-W4A16 of the same model loses
55
- 6.7 / 9.4 pp; GPTQ-recovered NVFP4-W4A16 loses 2.4 / 4.8 pp.)
56
-
57
- *v1 — updated as more evals land (RULER long-context, agentic SWE-style, IFEval).*
 
 
 
 
 
 
 
58
 
59
  ## Usage
60
 
@@ -69,14 +76,15 @@ print(llm.chat([{"role": "user", "content": "Write a Python LRU cache."}],
69
  SamplingParams(temperature=0.6, max_tokens=512))[0].outputs[0].text)
70
  ```
71
 
 
 
 
72
  ## Method & provenance
73
 
74
  - Quantizer: `llm-compressor` 0.13, `QuantizationModifier(scheme="FP8_DYNAMIC")`,
75
  ignore `lm_head` + `embed_tokens`. No calibration data required (dynamic activations).
76
  - Base: `openbmb/MiniCPM5-2B` (`LlamaForCausalLM`, 2.5 B, Apache-2.0).
77
  - Built and evaluated on an RTX 5070 Ti (Blackwell, SM120), vLLM 0.26.
78
- - Full quant-format comparison + methodology:
79
- <https://github.com/t-timms/blackwell-16gb-moe>
80
 
81
  ## License
82
 
 
36
  This is the recommended quant for MiniCPM5-2B when you want maximum quality
37
  retention. If you need to fit closer to 2 GB, see the
38
  [NVFP4 W4A16 build](https://huggingface.co/Ttimms/MiniCPM5-2B-NVFP4)
39
+ (smaller, ~2.4 pp HumanEval / ~4.8 pp MBPP cost).
40
 
41
  ## Evaluation
42
 
43
+ `lm-evaluation-harness`, vLLM 0.26 backend, greedy decoding, **median of 3 draws**
44
+ with the range (the harness is non-deterministic run-to-run even at greedy — a
45
+ single draw is not a reproducible score). All arms measured in one session against
46
+ the released checkpoint. HumanEval-instruct `pass@1`/`create_test` (n = 164);
47
+ MBPP base 3-shot (n = 500).
48
+
49
+ | build | HumanEval-inst | MBPP (3-shot) | size | Δ HE / MBPP |
50
+ |---|---:|---:|---:|---:|
51
+ | bf16 base | 86.59 % (85.98–86.59) | 50.60 % (50.40–51.00) | 4.68 GiB | — |
52
+ | **FP8-dynamic (this)** | **84.76 %** (84.15–85.37) | **48.80 %** (48.80–49.00) | **2.84 GiB** | **−1.8 / −1.8 pp** |
53
+ | NVFP4-W4A16 GPTQ | 84.15 % (81.71–84.15) | 45.80 % (45.60–46.40) | 2.03 GiB | −2.4 / −4.8 pp |
54
+ | NVFP4-W4A16 RTN | 79.88 % (77.44–79.88) | 41.20 % (41.20–41.80) | 2.03 GiB | −6.7 / −9.4 pp |
55
+ | mixed (MLP-NVFP4 + attn-FP8) | 84.15 % (82.93–85.98) | 46.60 % (46.00–46.80) | 2.20 GiB | −2.4 / −4.0 pp |
56
+
57
+ Both FP8 deltas sit inside the eval's own run-to-run spread — FP8 is effectively
58
+ lossless here. NVFP4 costs real accuracy on a 2.5 B dense model (little redundancy
59
+ to absorb 4-bit weights); pick it only if the extra ~0.8 GiB matters for your KV
60
+ budget.
61
+
62
+ Full comparison, serving notes, and a fine-tune experiment that regressed the
63
+ base's coding (so these quants target the released checkpoint):
64
+ **<https://github.com/t-timms/minicpm5-2b-quants>**
65
 
66
  ## Usage
67
 
 
76
  SamplingParams(temperature=0.6, max_tokens=512))[0].outputs[0].text)
77
  ```
78
 
79
+ On WSL, set `VLLM_USE_V2_MODEL_RUNNER=0` (the V2 runner needs UVA, which WSL
80
+ disables).
81
+
82
  ## Method & provenance
83
 
84
  - Quantizer: `llm-compressor` 0.13, `QuantizationModifier(scheme="FP8_DYNAMIC")`,
85
  ignore `lm_head` + `embed_tokens`. No calibration data required (dynamic activations).
86
  - Base: `openbmb/MiniCPM5-2B` (`LlamaForCausalLM`, 2.5 B, Apache-2.0).
87
  - Built and evaluated on an RTX 5070 Ti (Blackwell, SM120), vLLM 0.26.
 
 
88
 
89
  ## License
90