syedazeez commited on
Commit
da8ff41
·
verified ·
1 Parent(s): a435c67

List the Gate V shakedown checkpoint in the index

Browse files
Files changed (1) hide show
  1. README.md +7 -0
README.md CHANGED
@@ -54,6 +54,13 @@ not uploaded here. Regenerate it by running `v3_cli.py --real-corpus` once, whic
54
  | File | Gate | Result | Doc |
55
  |---|---|---|---|
56
  | `compact_v3_wikitext103_2ep.pt` | W | 2 epochs, 229.4M tokens, **41.35 PPL** | `experiments/GATE_W_WIKITEXT103_2EPOCH.md` |
 
 
 
 
 
 
 
57
 
58
  Trained at batch 7, sequence 512, context 520, with MTP enabled and the loss weight annealed 0.3 to 0.1. The
59
  earlier 292.54 figure quoted for Gate Q is not comparable: it used WikiText-2 validation with a WikiText-2
 
54
  | File | Gate | Result | Doc |
55
  |---|---|---|---|
56
  | `compact_v3_wikitext103_2ep.pt` | W | 2 epochs, 229.4M tokens, **41.35 PPL** | `experiments/GATE_W_WIKITEXT103_2EPOCH.md` |
57
+ | `compact_v3_wikitext103_shakedown.pt` | V | Shakedown, step 8100, 29.0M tokens, 87.57 PPL | `experiments/GATE_V_WIKITEXT103_SHAKEDOWN.md` |
58
+
59
+ Both use `tokenizer_wikitext103.json`. Gate V's checkpoint is the shakedown run that Gate W continued from; it is
60
+ kept because `serve.py` still names it as its default `--checkpoint`. Its 87.57 was re-measured over all 470
61
+ validation windows, against the 87.85 recorded in the gate document: the difference is the evaluation batch
62
+ size, not the weights, because `evaluate_tokens` averages per-batch means and so weights the final partial batch
63
+ like a full one.
64
 
65
  Trained at batch 7, sequence 512, context 520, with MTP enabled and the loss weight annealed 0.3 to 0.1. The
66
  earlier 292.54 figure quoted for Gate Q is not comparable: it used WikiText-2 validation with a WikiText-2