M2-2B: 30K-step byte-level checkpoint (val PPL 2.73, bf16)
Browse files
README.md
CHANGED
|
@@ -16,16 +16,18 @@ pipeline_tag: text-generation
|
|
| 16 |
the M2 experimental line's 2b configuration trained via `m2_training`.
|
| 17 |
Byte-level tokenizer (UTF-8 bytes, vocab 256).
|
| 18 |
|
| 19 |
-
**Status at upload:** research milestone —
|
| 20 |
|
| 21 |
## Measured
|
| 22 |
|
| 23 |
| Milestone | val PPL (byte-level, WikiText-103) |
|
| 24 |
|---|---|
|
| 25 |
| 10K steps (feasibility) | 4.53 |
|
| 26 |
-
|
|
|
|
|
| 27 |
|
| 28 |
-
|
|
|
|
| 29 |
batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
|
| 30 |
|
| 31 |
## Boundaries (honest)
|
|
|
|
| 16 |
the M2 experimental line's 2b configuration trained via `m2_training`.
|
| 17 |
Byte-level tokenizer (UTF-8 bytes, vocab 256).
|
| 18 |
|
| 19 |
+
**Status at upload:** research milestone — 60,000-step checkpoint.
|
| 20 |
|
| 21 |
## Measured
|
| 22 |
|
| 23 |
| Milestone | val PPL (byte-level, WikiText-103) |
|
| 24 |
|---|---|
|
| 25 |
| 10K steps (feasibility) | 4.53 |
|
| 26 |
+
| 30K steps | 2.73 |
|
| 27 |
+
| **60K steps (this checkpoint)** | **2.556** |
|
| 28 |
|
| 29 |
+
Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation).
|
| 30 |
+
Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
|
| 31 |
batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
|
| 32 |
|
| 33 |
## Boundaries (honest)
|