Add paper/demo links to model card
Browse files
README.md
CHANGED
|
@@ -1,47 +1,52 @@
|
|
| 1 |
-
---
|
| 2 |
-
language:
|
| 3 |
-
- en
|
| 4 |
-
license: mit
|
| 5 |
-
tags:
|
| 6 |
-
- awareliquid
|
| 7 |
-
- mt-lnn
|
| 8 |
-
- liquid-neural-network
|
| 9 |
-
- research
|
| 10 |
-
pipeline_tag: text-generation
|
| 11 |
-
---
|
| 12 |
-
|
| 13 |
-
# M2-2B
|
| 14 |
-
|
| 15 |
-
**1.93B-parameter AwareLiquid M2 preset** (2080 width × 34 layers × 16 heads),
|
| 16 |
-
the M2 experimental line's 2b configuration trained via `m2_training`.
|
| 17 |
-
Byte-level tokenizer (UTF-8 bytes, vocab 256).
|
| 18 |
-
|
| 19 |
-
**Status at upload:** research milestone — 60,000-step checkpoint.
|
| 20 |
-
|
| 21 |
-
## Measured
|
| 22 |
-
|
| 23 |
-
| Milestone | val PPL (byte-level, WikiText-103) |
|
| 24 |
-
|---|---|
|
| 25 |
-
| 10K steps (feasibility) | 4.53 |
|
| 26 |
-
| 30K steps | 2.73 |
|
| 27 |
-
| **60K steps (this checkpoint)** | **2.556** |
|
| 28 |
-
|
| 29 |
-
Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation).
|
| 30 |
-
Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
|
| 31 |
-
batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
|
| 32 |
-
|
| 33 |
-
## Boundaries (honest)
|
| 34 |
-
|
| 35 |
-
- **Byte-level, not a subword model** — PPL is on UTF-8 byte tokens; NOT
|
| 36 |
-
comparable to gpt2-vocab models. Serve with byte tokenization.
|
| 37 |
-
- **Not an instruct/chat model** — raw next-token pretraining only.
|
| 38 |
-
- **Research checkpoint** — bf16 safetensors (weight-tied embedding; the tied
|
| 39 |
-
`lm_head.weight` is omitted, standard HF re-tie on load). No optimizer state:
|
| 40 |
-
not exactly resumable.
|
| 41 |
-
- The M2 line's verdict so far is honest-null on reasoning mechanisms; this
|
| 42 |
-
checkpoint is the language-line scaling artifact, not a capability claim.
|
| 43 |
-
|
| 44 |
-
## Config
|
| 45 |
-
|
| 46 |
-
`config.json` carries the full MTLNNConfig (d_model 2080, 34 layers, 16 heads,
|
| 47 |
-
gwtb_n_heads=1, vocab 256, max_seq_len 129).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
license: mit
|
| 5 |
+
tags:
|
| 6 |
+
- awareliquid
|
| 7 |
+
- mt-lnn
|
| 8 |
+
- liquid-neural-network
|
| 9 |
+
- research
|
| 10 |
+
pipeline_tag: text-generation
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# M2-2B
|
| 14 |
+
|
| 15 |
+
**1.93B-parameter AwareLiquid M2 preset** (2080 width × 34 layers × 16 heads),
|
| 16 |
+
the M2 experimental line's 2b configuration trained via `m2_training`.
|
| 17 |
+
Byte-level tokenizer (UTF-8 bytes, vocab 256).
|
| 18 |
+
|
| 19 |
+
**Status at upload:** research milestone — 60,000-step checkpoint.
|
| 20 |
+
|
| 21 |
+
## Measured
|
| 22 |
+
|
| 23 |
+
| Milestone | val PPL (byte-level, WikiText-103) |
|
| 24 |
+
|---|---|
|
| 25 |
+
| 10K steps (feasibility) | 4.53 |
|
| 26 |
+
| 30K steps | 2.73 |
|
| 27 |
+
| **60K steps (this checkpoint)** | **2.556** |
|
| 28 |
+
|
| 29 |
+
Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation).
|
| 30 |
+
Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
|
| 31 |
+
batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
|
| 32 |
+
|
| 33 |
+
## Boundaries (honest)
|
| 34 |
+
|
| 35 |
+
- **Byte-level, not a subword model** — PPL is on UTF-8 byte tokens; NOT
|
| 36 |
+
comparable to gpt2-vocab models. Serve with byte tokenization.
|
| 37 |
+
- **Not an instruct/chat model** — raw next-token pretraining only.
|
| 38 |
+
- **Research checkpoint** — bf16 safetensors (weight-tied embedding; the tied
|
| 39 |
+
`lm_head.weight` is omitted, standard HF re-tie on load). No optimizer state:
|
| 40 |
+
not exactly resumable.
|
| 41 |
+
- The M2 line's verdict so far is honest-null on reasoning mechanisms; this
|
| 42 |
+
checkpoint is the language-line scaling artifact, not a capability claim.
|
| 43 |
+
|
| 44 |
+
## Config
|
| 45 |
+
|
| 46 |
+
`config.json` carries the full MTLNNConfig (d_model 2080, 34 layers, 16 heads,
|
| 47 |
+
gwtb_n_heads=1, vocab 256, max_seq_len 129).
|
| 48 |
+
|
| 49 |
+
## Papers
|
| 50 |
+
|
| 51 |
+
- MT-LNN architecture paper (PDF): https://huggingface.co/EverestAn/MT-LNN/resolve/main/mt_lnn_arxiv.pdf
|
| 52 |
+
- 中文版: https://huggingface.co/EverestAn/MT-LNN/resolve/main/mt_lnn_arxiv_zh.pdf
|