Upload ADDING_RESULTS.md with huggingface_hub
Browse files- ADDING_RESULTS.md +18 -0
ADDING_RESULTS.md
CHANGED
|
@@ -12,6 +12,24 @@ This repo is a report, not a model. Its source of truth has two files:
|
|
| 12 |
`results.json` and the CSV are **generated** — do not hand-edit them. Edit `models.json` and
|
| 13 |
re-run the scripts.
|
| 14 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
## 1. Run the benchmark (private harness)
|
| 16 |
|
| 17 |
In `brywil/quantkit/bench`, run all four suites — the canonical 42 tasks:
|
|
|
|
| 12 |
`results.json` and the CSV are **generated** — do not hand-edit them. Edit `models.json` and
|
| 13 |
re-run the scripts.
|
| 14 |
|
| 15 |
+
## 0. Quality gate (required for every uploaded model)
|
| 16 |
+
|
| 17 |
+
Every model we upload must carry all five checks — see `QUALITY_STANDARD.md` in the lab repo:
|
| 18 |
+
|
| 19 |
+
| check | minimum |
|
| 20 |
+
|---|---|
|
| 21 |
+
| **Gen check** | fixed prompt set + chained tool-call gate, at the shipped context |
|
| 22 |
+
| **Perplexity** | `llama-perplexity`, `wiki.test.raw`, `-c 512`, recorded `--chunks` |
|
| 23 |
+
| **KL divergence** | mean + percentiles vs BF16 reference logits |
|
| 24 |
+
| **Top-token agreement** | same-top % (+ RMS Δp) from the same KLD run |
|
| 25 |
+
| **Task subset** | at least a representative subset of the 42; full 42 for a headline release |
|
| 26 |
+
|
| 27 |
+
Gen check is the bare minimum — a quant that has generated nothing is not validated. PPL
|
| 28 |
+
alone is not sufficient (it has ranked quants in the opposite order to KLD). While a repo is
|
| 29 |
+
staged, the card may say `validation pending`; do not announce the artifact as usable until
|
| 30 |
+
the checks land. The result row links to the model card, which must state pass/fail and link
|
| 31 |
+
the logs.
|
| 32 |
+
|
| 33 |
## 1. Run the benchmark (private harness)
|
| 34 |
|
| 35 |
In `brywil/quantkit/bench`, run all four suites — the canonical 42 tasks:
|