Myric commited on
Commit
5762307
·
verified ·
1 Parent(s): c1f08c3

Upload ADDING_RESULTS.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. ADDING_RESULTS.md +18 -0
ADDING_RESULTS.md CHANGED
@@ -12,6 +12,24 @@ This repo is a report, not a model. Its source of truth has two files:
12
  `results.json` and the CSV are **generated** — do not hand-edit them. Edit `models.json` and
13
  re-run the scripts.
14
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
  ## 1. Run the benchmark (private harness)
16
 
17
  In `brywil/quantkit/bench`, run all four suites — the canonical 42 tasks:
 
12
  `results.json` and the CSV are **generated** — do not hand-edit them. Edit `models.json` and
13
  re-run the scripts.
14
 
15
+ ## 0. Quality gate (required for every uploaded model)
16
+
17
+ Every model we upload must carry all five checks — see `QUALITY_STANDARD.md` in the lab repo:
18
+
19
+ | check | minimum |
20
+ |---|---|
21
+ | **Gen check** | fixed prompt set + chained tool-call gate, at the shipped context |
22
+ | **Perplexity** | `llama-perplexity`, `wiki.test.raw`, `-c 512`, recorded `--chunks` |
23
+ | **KL divergence** | mean + percentiles vs BF16 reference logits |
24
+ | **Top-token agreement** | same-top % (+ RMS Δp) from the same KLD run |
25
+ | **Task subset** | at least a representative subset of the 42; full 42 for a headline release |
26
+
27
+ Gen check is the bare minimum — a quant that has generated nothing is not validated. PPL
28
+ alone is not sufficient (it has ranked quants in the opposite order to KLD). While a repo is
29
+ staged, the card may say `validation pending`; do not announce the artifact as usable until
30
+ the checks land. The result row links to the model card, which must state pass/fail and link
31
+ the logs.
32
+
33
  ## 1. Run the benchmark (private harness)
34
 
35
  In `brywil/quantkit/bench`, run all four suites — the canonical 42 tasks: