6 names the non-rebuildable artifact; 2.6 names the 27B ppl corpus
Browse files- index.html +10 -5
index.html
CHANGED
|
@@ -288,7 +288,8 @@ corpora, first 8192 tokens. Deterministic — an artifact reproduces its
|
|
| 288 |
total negative log-likelihood to all printed decimals across launches
|
| 289 |
and machines. <strong>35B and 27B:</strong> KL divergence to the bf16 model's cached
|
| 290 |
logits, in millinats, with top-1 agreement, plus referee perplexity on
|
| 291 |
-
the 27B
|
|
|
|
| 292 |
to every reported digit. The gemma-4 family, where we observed similar
|
| 293 |
size-quality behaviour, is excluded throughout: raw likelihood is
|
| 294 |
invalid on those instruction-tuned models as a property of the model
|
|
@@ -945,10 +946,14 @@ GiB residency on our 128 GB machine and is uncharacterized at other
|
|
| 945 |
sizes; we publish ratios within a session, never absolutes.</p>
|
| 946 |
<p><strong>Costs we pay.</strong> Prefill remains ~0.5x affine at 35B scale even after
|
| 947 |
the shipped lever. Codebooks beyond threadgroup capacity pay ~19%
|
| 948 |
-
decode. One published artifact cannot be rebuilt at all
|
| 949 |
-
|
| 950 |
-
|
| 951 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 952 |
<h2 id="7-reproducibility">7. Reproducibility</h2>
|
| 953 |
<p>All artifacts are published under <code>TheDrainFlorist</code> on Hugging Face
|
| 954 |
with their VQ runtimes bundled in-checkpoint (stock mlx-lm, no
|
|
|
|
| 288 |
total negative log-likelihood to all printed decimals across launches
|
| 289 |
and machines. <strong>35B and 27B:</strong> KL divergence to the bf16 model's cached
|
| 290 |
logits, in millinats, with top-1 agreement, plus referee perplexity on
|
| 291 |
+
the 27B — the same wikitext-style prose corpus at its first 2048 tokens,
|
| 292 |
+
a single corpus where the 397B uses two. Comparator rows were re-verified on a second machine and agree
|
| 293 |
to every reported digit. The gemma-4 family, where we observed similar
|
| 294 |
size-quality behaviour, is excluded throughout: raw likelihood is
|
| 295 |
invalid on those instruction-tuned models as a property of the model
|
|
|
|
| 946 |
sizes; we publish ratios within a session, never absolutes.</p>
|
| 947 |
<p><strong>Costs we pay.</strong> Prefill remains ~0.5x affine at 35B scale even after
|
| 948 |
the shipped lever. Codebooks beyond threadgroup capacity pay ~19%
|
| 949 |
+
decode. One published artifact cannot be rebuilt at all: the 111.6 GiB
|
| 950 |
+
d4/K256 build (<code>VQ-2.4bpw</code>) predates the manifest system, and the log
|
| 951 |
+
recording its exact fit invocation was overwritten four days after the
|
| 952 |
+
fit, so its remaining unrecoverable inputs are the fit flags themselves.
|
| 953 |
+
It remains downloadable, its scores reproduce exactly on the artifact,
|
| 954 |
+
and its quality sits inside the measured draw distribution of its
|
| 955 |
+
geometry — a favourable but unexceptional draw, not a mystery (§2.6's
|
| 956 |
+
floors are how we know).</p>
|
| 957 |
<h2 id="7-reproducibility">7. Reproducibility</h2>
|
| 958 |
<p>All artifacts are published under <code>TheDrainFlorist</code> on Hugging Face
|
| 959 |
with their VQ runtimes bundled in-checkpoint (stock mlx-lm, no
|