List the Gate V shakedown checkpoint in the index
Browse files
README.md
CHANGED
|
@@ -54,6 +54,13 @@ not uploaded here. Regenerate it by running `v3_cli.py --real-corpus` once, whic
|
|
| 54 |
| File | Gate | Result | Doc |
|
| 55 |
|---|---|---|---|
|
| 56 |
| `compact_v3_wikitext103_2ep.pt` | W | 2 epochs, 229.4M tokens, **41.35 PPL** | `experiments/GATE_W_WIKITEXT103_2EPOCH.md` |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
|
| 58 |
Trained at batch 7, sequence 512, context 520, with MTP enabled and the loss weight annealed 0.3 to 0.1. The
|
| 59 |
earlier 292.54 figure quoted for Gate Q is not comparable: it used WikiText-2 validation with a WikiText-2
|
|
|
|
| 54 |
| File | Gate | Result | Doc |
|
| 55 |
|---|---|---|---|
|
| 56 |
| `compact_v3_wikitext103_2ep.pt` | W | 2 epochs, 229.4M tokens, **41.35 PPL** | `experiments/GATE_W_WIKITEXT103_2EPOCH.md` |
|
| 57 |
+
| `compact_v3_wikitext103_shakedown.pt` | V | Shakedown, step 8100, 29.0M tokens, 87.57 PPL | `experiments/GATE_V_WIKITEXT103_SHAKEDOWN.md` |
|
| 58 |
+
|
| 59 |
+
Both use `tokenizer_wikitext103.json`. Gate V's checkpoint is the shakedown run that Gate W continued from; it is
|
| 60 |
+
kept because `serve.py` still names it as its default `--checkpoint`. Its 87.57 was re-measured over all 470
|
| 61 |
+
validation windows, against the 87.85 recorded in the gate document: the difference is the evaluation batch
|
| 62 |
+
size, not the weights, because `evaluate_tokens` averages per-batch means and so weights the final partial batch
|
| 63 |
+
like a full one.
|
| 64 |
|
| 65 |
Trained at batch 7, sequence 512, context 520, with MTP enabled and the loss weight annealed 0.3 to 0.1. The
|
| 66 |
earlier 292.54 figure quoted for Gate Q is not comparable: it used WikiText-2 validation with a WikiText-2
|