EverestAn commited on
Commit
aca3407
·
verified ·
1 Parent(s): 3a59b32

M2-2B: 30K-step byte-level checkpoint (val PPL 2.73, bf16)

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -16,16 +16,18 @@ pipeline_tag: text-generation
16
  the M2 experimental line's 2b configuration trained via `m2_training`.
17
  Byte-level tokenizer (UTF-8 bytes, vocab 256).
18
 
19
- **Status at upload:** research milestone — first complete 30,000-step run.
20
 
21
  ## Measured
22
 
23
  | Milestone | val PPL (byte-level, WikiText-103) |
24
  |---|---|
25
  | 10K steps (feasibility) | 4.53 |
26
- | **30K steps (this checkpoint)** | **2.73** |
 
27
 
28
- ≈ 1.45 BPB. Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
 
29
  batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
30
 
31
  ## Boundaries (honest)
 
16
  the M2 experimental line's 2b configuration trained via `m2_training`.
17
  Byte-level tokenizer (UTF-8 bytes, vocab 256).
18
 
19
+ **Status at upload:** research milestone — 60,000-step checkpoint.
20
 
21
  ## Measured
22
 
23
  | Milestone | val PPL (byte-level, WikiText-103) |
24
  |---|---|
25
  | 10K steps (feasibility) | 4.53 |
26
+ | 30K steps | 2.73 |
27
+ | **60K steps (this checkpoint)** | **2.556** |
28
 
29
+ Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation).
30
+ Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
31
  batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
32
 
33
  ## Boundaries (honest)