| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - awareliquid |
| - mt-lnn |
| - liquid-neural-network |
| - research |
| pipeline_tag: text-generation |
| --- |
| |
| # M2-2B |
|
|
| **1.93B-parameter AwareLiquid M2 preset** (2080 width × 34 layers × 16 heads), |
| the M2 experimental line's 2b configuration trained via `m2_training`. |
| Byte-level tokenizer (UTF-8 bytes, vocab 256). |
|
|
| **Status at upload:** research milestone — 200,000-step FINAL checkpoint (val PPL 2.4106). `model.safetensors` holds the 200K weights; the raw resumable PyTorch checkpoint is `t2b_200k.pt`. |
| **Training status (2026-09-28):** the baseline run COMPLETED at 200,000 steps, |
| final val PPL 2.4106. Full milestone table below; the 200K weights live in the |
| M2 repo (docs/RESULTS_2B_30K.md) and on the training server |
| (/root/M2/runs/t2b_30k.pt, 3.86GB bf16). **The 200K raw checkpoint |
| is now in this repo as `t2b_200k.pt`** (PyTorch bf16, weight-tied; the matching 200K-weight `model.safetensors` export is in this repo since 2026-09-29). |
| |
| ## Measured |
| |
| | Milestone | val PPL (byte-level, WikiText-103) | |
| |---|---| |
| | 10K steps (feasibility) | 4.53 | |
| | 30K steps | 2.73 | |
| | 60K steps | 2.556 | |
| | 90K steps | 2.482 | |
| | 150K steps | 2.413 | |
| | **200K steps (final, 09-28)** | **2.4106** | |
| |
| Convergence: 30K→60K −6.4%, 60K→90K −2.9%, 90K→150K −2.8%, 150K→200K |
| −0.1% (saturated). |
| |
| Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation). |
| Training: bf16 model, AdamW8bit, lr 1e-4, seq 128, |
| batch 8 × grad_accum 4, on a single A100-PCIE-40GB. |
|
|
| ## Boundaries (honest) |
|
|
| - **Byte-level, not a subword model** — PPL is on UTF-8 byte tokens; NOT |
| comparable to gpt2-vocab models. Serve with byte tokenization. |
| - **Not an instruct/chat model** — raw next-token pretraining only. |
| - **Research checkpoint** — bf16 safetensors (weight-tied embedding; the tied |
| `lm_head.weight` is omitted, standard HF re-tie on load). No optimizer state: |
| not exactly resumable. |
| - The M2 line's verdict so far is honest-null on reasoning mechanisms; this |
| checkpoint is the language-line scaling artifact, not a capability claim. |
|
|
| ## SFT checkpoints (post-training experiments, 2026-10) |
|
|
| | file | recipe | val PPL | JevBench public (official harness) | |
| |---|---|---|---| |
| | `t2b_sft_v1.pt` | alpaca-short 5.4k pairs (<=128B), 3 epochs, lr 1e-4 | 3.95 | - | |
| | `t2b_sft_v2.pt` | alpaca-short512 19.6k pairs (<=512B), 1+5k steps, lr 2e-5, rope_scale 4 | **3.20** | raw 31.3% (48 easy + 72 original decisions) | |
| |
| Boundaries: instruction-SFT on **short** pairs; held-out generation went from |
| 0/8 (base) to 4/8 format-correct, but decision-task accuracy remains the open |
| problem (best family configuration: 2B frozen core + decision head, 39.6% |
| easy-tier raw / ECE 0.20 / 0.168 s - see github.com/AwareLiquid/O1-Flash). |
| Serving interface for all of the above: byte-level, `max_seq_len` per config. |
| |
| ## Config |
| |
| `config.json` carries the full MTLNNConfig (d_model 2080, 34 layers, 16 heads, |
| gwtb_n_heads=1, vocab 256, max_seq_len 129). |
|
|
