User01110 commited on
Commit
080798f
·
verified ·
1 Parent(s): 80088fc

Remove leaderboard framing from model card

Browse files
Files changed (2) hide show
  1. README.md +9 -22
  2. assets/training-trajectory.svg +29 -27
README.md CHANGED
@@ -25,7 +25,7 @@ A compact generalist language model built to study channel mixing at small scale
25
 
26
  **7.85M parameters | 1,024-token context | 4,096-token vocabulary | Safetensors**
27
 
28
- **Benchmark-selected at step 40,000 | Open SLM Leaderboard-style average: 36.21%**
29
 
30
  </div>
31
 
@@ -33,9 +33,9 @@ A compact generalist language model built to study channel mixing at small scale
33
 
34
  ## Overview
35
 
36
- CMA 8M is the benchmark-selected checkpoint of a **7,849,161-parameter**
37
- Channel-Mixing Attention generalist causal language model. It pairs a compact
38
- nine-layer architecture with the unmodified native GPT-S 4,096-token tokenizer.
39
 
40
  | Detail | Value |
41
  |---|---|
@@ -54,7 +54,7 @@ CMA in the feed-forward position. For every token, CMA reshapes the 288-channel
54
  hidden state into **12 chunks of 24**, routes between those chunks with **3
55
  heads**, applies a signed blend and SiLU gate, then projects back to 288 channels.
56
 
57
- ![CMA 8M architecture](https://huggingface.co/User01110/cma-8M/resolve/main/assets/cma-architecture.svg?v=4)
58
 
59
  ## Quick start
60
 
@@ -133,11 +133,11 @@ ordinary causal-language-model text.
133
 
134
  ## Training trajectory
135
 
136
- Across the logged evaluation checkpoints, normalized bits per byte (BPB)
137
- decreased from **3.7838** to **1.3520**, while the Open SLM Leaderboard-style
138
- average increased from **30.66%** to **36.21%**.
139
 
140
- ![CMA 8M training trajectory](https://huggingface.co/User01110/cma-8M/resolve/main/assets/training-trajectory.svg?v=4)
141
 
142
  ## Evaluation
143
 
@@ -154,17 +154,6 @@ official raw continuation log-likelihood-sum rule.
154
  | ARC-Challenge | **23.29%** |
155
  | PIQA | **58.22%** |
156
  | ArithMark-2 | **29.12%** |
157
- | ARC mean | **29.32%** |
158
- | **Open SLM Leaderboard-style average** | **36.21%** |
159
-
160
- The selection average is:
161
-
162
- ```text
163
- (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4
164
- ```
165
-
166
- This matches the Open SLM Leaderboard formula. The checkpoint was selected at
167
- step 40,000 for its **36.21%** average.
168
 
169
  ### WikiText-103 validation
170
 
@@ -176,5 +165,3 @@ step 40,000 for its **36.21%** average.
176
  | Scored tokens | 358,911 |
177
  | Normalized UTF-8 bytes | 1,145,226 |
178
 
179
- WikiText normalized BPB is reported separately and was **not** used for
180
- checkpoint selection.
 
25
 
26
  **7.85M parameters | 1,024-token context | 4,096-token vocabulary | Safetensors**
27
 
28
+ **40,000 training steps | 20.97B trained tokens**
29
 
30
  </div>
31
 
 
33
 
34
  ## Overview
35
 
36
+ CMA 8M is a **7,849,161-parameter** Channel-Mixing Attention generalist
37
+ causal language model trained for 40,000 steps. It pairs a compact nine-layer
38
+ architecture with the unmodified native GPT-S 4,096-token tokenizer.
39
 
40
  | Detail | Value |
41
  |---|---|
 
54
  hidden state into **12 chunks of 24**, routes between those chunks with **3
55
  heads**, applies a signed blend and SiLU gate, then projects back to 288 channels.
56
 
57
+ ![CMA 8M architecture](https://huggingface.co/SupraLabs/cma-8M/resolve/main/assets/cma-architecture.svg?v=5)
58
 
59
  ## Quick start
60
 
 
133
 
134
  ## Training trajectory
135
 
136
+ Across the logged validation checkpoints, normalized bits per byte (BPB)
137
+ decreased from **3.7838** to **1.3520**, while validation loss decreased from
138
+ **8.3686** to **2.9903**.
139
 
140
+ ![CMA 8M training trajectory](https://huggingface.co/SupraLabs/cma-8M/resolve/main/assets/training-trajectory.svg?v=5)
141
 
142
  ## Evaluation
143
 
 
154
  | ARC-Challenge | **23.29%** |
155
  | PIQA | **58.22%** |
156
  | ArithMark-2 | **29.12%** |
 
 
 
 
 
 
 
 
 
 
 
157
 
158
  ### WikiText-103 validation
159
 
 
165
  | Scored tokens | 358,911 |
166
  | Normalized UTF-8 bytes | 1,145,226 |
167
 
 
 
assets/training-trajectory.svg CHANGED