User01110 commited on
Commit
ba5e431
·
verified ·
1 Parent(s): d95a00a

Add verified benchmark results to model card

Browse files
Files changed (1) hide show
  1. README.md +129 -105
README.md CHANGED
@@ -1,105 +1,129 @@
1
- ---
2
- library_name: transformers
3
- base_model: Qwen/Qwen3-0.6B
4
- tags:
5
- - qwen3
6
- - causal-lm
7
- - tiny-language-model
8
- - novelty-gated-attention
9
- - trust-remote-code
10
- ---
11
-
12
- # tinyLM-8M-exp
13
-
14
- Tiny 8M-class Qwen3-config causal LM with math-only novelty-gated GQA.
15
-
16
- ## Architecture
17
-
18
- | Item | Value |
19
- | --- | ---: |
20
- | Config type | `qwen3` |
21
- | Parameters | 8.132M |
22
- | Layers | 8 |
23
- | Hidden size | 256 |
24
- | MLP size | 896 |
25
- | Query heads | 8 |
26
- | KV heads | 4 |
27
- | Head dim | 32 |
28
- | RoPE theta | 2500 |
29
- | Tied embeddings | yes |
30
-
31
- | Attention | Value |
32
- | --- | --- |
33
- | Type | GQA |
34
- | Novelty gate | math-only element-wise RMS-normalized abs-delta |
35
- | Gate floor | 0.05 |
36
-
37
- ## Training
38
-
39
- | Item | Value |
40
- | --- | --- |
41
- | Tokenizer | `AxiomicLabs/GPT-S2-5M` |
42
- | Sequence length | 512 |
43
- | Microbatch size | 1024 |
44
- | Gradient accumulation | 4 |
45
- | Effective batch size | 4096 |
46
- | Steps | 10,000 |
47
- | Validation cadence | every 1,000 steps |
48
- | Official lm-eval | after final Hub upload on ARC-Easy, ARC-Challenge, PIQA, HellaSwag |
49
- | LR schedule | warmup, cosine to min by 10,000 |
50
- | Optimizer | Muon for middle 2D weights, AdamW for the rest |
51
- | Special-token policy | BOS/EOS are document-level; `<|im_start|>`/`<|im_end|>` are sequence-level |
52
-
53
- | Dataset | Share | Config |
54
- | --- | ---: | --- |
55
- | `HuggingFaceFW/fineweb-edu` | 60.0% | `sample-100BT` |
56
- | `HuggingFaceTB/smollm-corpus` | 30.0% | `cosmopedia-v2` only |
57
- | `epfml/FineWeb-HQ` | 10.0% | `default` |
58
-
59
- ## Validation
60
-
61
- | Metric | Value |
62
- | --- | ---: |
63
- | Dataset | `Salesforce/wikitext`, `wikitext-103-raw-v1`, validation |
64
- | Context / stride | 512 / 256 |
65
- | Loss | 3.2769 |
66
- | Perplexity | 26.49 |
67
- | UTF-8 BPB | 1.4992 |
68
- | Scored tokens | 365,258 |
69
- | UTF-8 bytes | 1,151,766 |
70
-
71
- ## Load And Generate
72
-
73
- ```python
74
- from transformers import AutoModelForCausalLM, AutoTokenizer
75
- import torch
76
-
77
- repo = "User01110/tinyLM-8M-exp"
78
- tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
79
- model = AutoModelForCausalLM.from_pretrained(
80
- repo,
81
- trust_remote_code=True,
82
- torch_dtype="auto",
83
- device_map="auto",
84
- )
85
-
86
- prompt = "The future of AI is"
87
- inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
88
- print(inputs.input_ids[0][:2].tolist()) # auto-prefix: [<|im_start|>, <bos>]
89
-
90
- with torch.no_grad():
91
- output = model.generate(
92
- **inputs,
93
- max_new_tokens=512,
94
- do_sample=True,
95
- temperature=0.65,
96
- top_k=30,
97
- repetition_penalty=1.2,
98
- pad_token_id=tokenizer.pad_token_id,
99
- eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
100
- )
101
-
102
- print(tokenizer.decode(output[0], skip_special_tokens=True))
103
- ```
104
-
105
- This repo uses a native `Qwen3Config` plus remote model code for the math-only novelty-gated attention block.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ base_model: Qwen/Qwen3-0.6B
4
+ tags:
5
+ - qwen3
6
+ - causal-lm
7
+ - tiny-language-model
8
+ - novelty-gated-attention
9
+ - trust-remote-code
10
+ ---
11
+
12
+ # tinyLM-8M-exp
13
+
14
+ Tiny 8M-class Qwen3-config causal LM with math-only novelty-gated GQA.
15
+
16
+ ## Architecture
17
+
18
+ | Item | Value |
19
+ | --- | ---: |
20
+ | Config type | `qwen3` |
21
+ | Parameters | 8.132M |
22
+ | Layers | 8 |
23
+ | Hidden size | 256 |
24
+ | MLP size | 896 |
25
+ | Query heads | 8 |
26
+ | KV heads | 4 |
27
+ | Head dim | 32 |
28
+ | RoPE theta | 2500 |
29
+ | Tied embeddings | yes |
30
+
31
+ | Attention | Value |
32
+ | --- | --- |
33
+ | Type | GQA |
34
+ | Novelty gate | math-only element-wise RMS-normalized abs-delta |
35
+ | Gate floor | 0.05 |
36
+
37
+ ## Training
38
+
39
+ | Item | Value |
40
+ | --- | --- |
41
+ | Tokenizer | `AxiomicLabs/GPT-S2-5M` |
42
+ | Sequence length | 512 |
43
+ | Microbatch size | 1024 |
44
+ | Gradient accumulation | 4 |
45
+ | Effective batch size | 4096 |
46
+ | Steps | 10,000 |
47
+ | Validation cadence | every 1,000 steps |
48
+ | Official lm-eval | after final Hub upload on ARC-Easy, ARC-Challenge, PIQA, HellaSwag |
49
+ | LR schedule | warmup, cosine to min by 10,000 |
50
+ | Optimizer | Muon for middle 2D weights, AdamW for the rest |
51
+ | Special-token policy | BOS/EOS are document-level; `<|im_start|>`/`<|im_end|>` are sequence-level |
52
+
53
+ | Dataset | Share | Config |
54
+ | --- | ---: | --- |
55
+ | `HuggingFaceFW/fineweb-edu` | 60.0% | `sample-100BT` |
56
+ | `HuggingFaceTB/smollm-corpus` | 30.0% | `cosmopedia-v2` only |
57
+ | `epfml/FineWeb-HQ` | 10.0% | `default` |
58
+
59
+ ## Validation
60
+
61
+ | Metric | Value |
62
+ | --- | ---: |
63
+ | Dataset | `Salesforce/wikitext`, `wikitext-103-raw-v1`, validation |
64
+ | Context / stride | 512 / 256 |
65
+ | Loss | 3.2769 |
66
+ | Perplexity | 26.49 |
67
+ | UTF-8 BPB | 1.4992 |
68
+ | Scored tokens | 365,258 |
69
+ | UTF-8 bytes | 1,151,766 |
70
+
71
+ ## Evaluation
72
+
73
+ Scores were run after Hub upload against revision
74
+ `d95a00a6edafab4bc2d6b60a28e6893b00f52699`.
75
+ ARC-Easy, ARC-Challenge, PIQA, and HellaSwag use official `lm_eval`
76
+ 0-shot log-likelihood scoring. ArithMark-2.0 uses the same continuation
77
+ NLL scoring style with a custom scorer because it is not available in
78
+ `lm_eval`.
79
+
80
+ | Task | n | acc | acc stderr | acc_norm | acc_norm stderr |
81
+ | --- | ---: | ---: | ---: | ---: | ---: |
82
+ | ARC-Easy | 2,376 | 37.04% | 0.99% | 35.86% | 0.98% |
83
+ | ARC-Challenge | 1,172 | 18.77% | 1.14% | 22.87% | 1.23% |
84
+ | PIQA | 1,838 | 57.67% | 1.15% | 57.89% | 1.15% |
85
+ | HellaSwag | 10,042 | 26.88% | 0.44% | 27.88% | 0.45% |
86
+ | ArithMark-2.0 | 2,500 | 25.12% | 0.87% | 24.44% | 0.86% |
87
+
88
+ Leaderboard-style grouped average using `acc_norm`:
89
+
90
+ | Metric | Value |
91
+ | --- | ---: |
92
+ | ARC average | 29.36% |
93
+ | Grouped average | 34.89% |
94
+
95
+ ## Load And Generate
96
+
97
+ ```python
98
+ from transformers import AutoModelForCausalLM, AutoTokenizer
99
+ import torch
100
+
101
+ repo = "User01110/tinyLM-8M-exp"
102
+ tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
103
+ model = AutoModelForCausalLM.from_pretrained(
104
+ repo,
105
+ trust_remote_code=True,
106
+ torch_dtype="auto",
107
+ device_map="auto",
108
+ )
109
+
110
+ prompt = "The future of AI is"
111
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
112
+ print(inputs.input_ids[0][:2].tolist()) # auto-prefix: [<|im_start|>, <bos>]
113
+
114
+ with torch.no_grad():
115
+ output = model.generate(
116
+ **inputs,
117
+ max_new_tokens=512,
118
+ do_sample=True,
119
+ temperature=0.65,
120
+ top_k=30,
121
+ repetition_penalty=1.2,
122
+ pad_token_id=tokenizer.pad_token_id,
123
+ eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>"),
124
+ )
125
+
126
+ print(tokenizer.decode(output[0], skip_special_tokens=True))
127
+ ```
128
+
129
+ This repo uses a native `Qwen3Config` plus remote model code for the math-only novelty-gated attention block.