EverestAn commited on
Commit
96fd312
·
verified ·
1 Parent(s): edd793c

Add paper/demo links to model card

Browse files
Files changed (1) hide show
  1. README.md +52 -47
README.md CHANGED
@@ -1,47 +1,52 @@
1
- ---
2
- language:
3
- - en
4
- license: mit
5
- tags:
6
- - awareliquid
7
- - mt-lnn
8
- - liquid-neural-network
9
- - research
10
- pipeline_tag: text-generation
11
- ---
12
-
13
- # M2-2B
14
-
15
- **1.93B-parameter AwareLiquid M2 preset** (2080 width × 34 layers × 16 heads),
16
- the M2 experimental line's 2b configuration trained via `m2_training`.
17
- Byte-level tokenizer (UTF-8 bytes, vocab 256).
18
-
19
- **Status at upload:** research milestone — 60,000-step checkpoint.
20
-
21
- ## Measured
22
-
23
- | Milestone | val PPL (byte-level, WikiText-103) |
24
- |---|---|
25
- | 10K steps (feasibility) | 4.53 |
26
- | 30K steps | 2.73 |
27
- | **60K steps (this checkpoint)** | **2.556** |
28
-
29
- Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation).
30
- Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
31
- batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
32
-
33
- ## Boundaries (honest)
34
-
35
- - **Byte-level, not a subword model** — PPL is on UTF-8 byte tokens; NOT
36
- comparable to gpt2-vocab models. Serve with byte tokenization.
37
- - **Not an instruct/chat model** — raw next-token pretraining only.
38
- - **Research checkpoint** — bf16 safetensors (weight-tied embedding; the tied
39
- `lm_head.weight` is omitted, standard HF re-tie on load). No optimizer state:
40
- not exactly resumable.
41
- - The M2 line's verdict so far is honest-null on reasoning mechanisms; this
42
- checkpoint is the language-line scaling artifact, not a capability claim.
43
-
44
- ## Config
45
-
46
- `config.json` carries the full MTLNNConfig (d_model 2080, 34 layers, 16 heads,
47
- gwtb_n_heads=1, vocab 256, max_seq_len 129).
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: mit
5
+ tags:
6
+ - awareliquid
7
+ - mt-lnn
8
+ - liquid-neural-network
9
+ - research
10
+ pipeline_tag: text-generation
11
+ ---
12
+
13
+ # M2-2B
14
+
15
+ **1.93B-parameter AwareLiquid M2 preset** (2080 width × 34 layers × 16 heads),
16
+ the M2 experimental line's 2b configuration trained via `m2_training`.
17
+ Byte-level tokenizer (UTF-8 bytes, vocab 256).
18
+
19
+ **Status at upload:** research milestone — 60,000-step checkpoint.
20
+
21
+ ## Measured
22
+
23
+ | Milestone | val PPL (byte-level, WikiText-103) |
24
+ |---|---|
25
+ | 10K steps (feasibility) | 4.53 |
26
+ | 30K steps | 2.73 |
27
+ | **60K steps (this checkpoint)** | **2.556** |
28
+
29
+ Context-flat: PPL 2.72 / 2.72 / 2.68 at seq 128 / 256 / 512 (no length degradation).
30
+ Training: bf16 model, AdamW8bit, lr 1e-4, seq 128,
31
+ batch 8 × grad_accum 4, on a single A100-PCIE-40GB.
32
+
33
+ ## Boundaries (honest)
34
+
35
+ - **Byte-level, not a subword model** — PPL is on UTF-8 byte tokens; NOT
36
+ comparable to gpt2-vocab models. Serve with byte tokenization.
37
+ - **Not an instruct/chat model** — raw next-token pretraining only.
38
+ - **Research checkpoint** — bf16 safetensors (weight-tied embedding; the tied
39
+ `lm_head.weight` is omitted, standard HF re-tie on load). No optimizer state:
40
+ not exactly resumable.
41
+ - The M2 line's verdict so far is honest-null on reasoning mechanisms; this
42
+ checkpoint is the language-line scaling artifact, not a capability claim.
43
+
44
+ ## Config
45
+
46
+ `config.json` carries the full MTLNNConfig (d_model 2080, 34 layers, 16 heads,
47
+ gwtb_n_heads=1, vocab 256, max_seq_len 129).
48
+
49
+ ## Papers
50
+
51
+ - MT-LNN architecture paper (PDF): https://huggingface.co/EverestAn/MT-LNN/resolve/main/mt_lnn_arxiv.pdf
52
+ - 中文版: https://huggingface.co/EverestAn/MT-LNN/resolve/main/mt_lnn_arxiv_zh.pdf