ereniko commited on
Commit
b78e2cb
·
verified ·
1 Parent(s): 3f2c8bb

Rewrite model card: narrative framing, benchmarks as evidence

Browse files
Files changed (1) hide show
  1. README.md +80 -68
README.md CHANGED
@@ -5,6 +5,7 @@ language:
5
  library_name: transformers
6
  tags:
7
  - ivme
 
8
  - language-model
9
  - transformer
10
  - rope
@@ -13,6 +14,8 @@ tags:
13
  - xsa
14
  - qk-norm
15
  - continued-pretraining
 
 
16
  - custom_code
17
  - text-generation
18
  datasets:
@@ -29,9 +32,15 @@ pipeline_tag: text-generation
29
 
30
  ![Conversate-v3.1 Logo](https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/mbZUbkYIjqvwdpcPac6Mu.png)
31
 
32
- **İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight. This is a continued-pretraining pass on v3: same architecture, same tokenizer, 31.4B more tokens on a deliberately *denser* mix — more math, more textbook, less raw web.
33
 
34
- It gains **+1.89 points of Intelligence Index** over v3, and gives back 3.4 points of BLiMP doing it. Both are real. The tradeoff is explained below rather than buried.
 
 
 
 
 
 
35
 
36
  ---
37
 
@@ -53,93 +62,84 @@ It gains **+1.89 points of Intelligence Index** over v3, and gives back 3.4 poin
53
  | Embeddings | Tied input/output |
54
  | Biases | None |
55
 
56
- **On the parameter count.** v3 reports 24,787,520; v3.1 reports 24,956,480. The FFN was zero-padded from 853 to 864 so the GEMMs tile cleanly on tensor cores. `silu(0) = 0` and the up-branch of those 11 units is zero, so they contribute exactly nothing to the output — the model is functionally identical, just 60–85% faster on FFN matmuls during training. No behaviour change, no retraining needed.
57
 
58
  ---
59
 
60
- ## Benchmarks
61
 
62
- Benchmarks were run with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.13 (0-shot, `acc_norm`), plus [ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) via its official evaluation script.
63
 
64
- | Benchmark | v3 | **v3.1** |
65
- |---|---|---|
66
- | ARC-Easy | 39.02 | **40.15** |
67
- | ARC-Challenge | 24.32 | **25.00** |
68
- | HellaSwag | 28.90 | **29.52** |
69
- | PIQA | 57.67 | **59.25** |
70
- | ArithMark-3 | 35.20 | **37.20** |
71
- | **Intelligence Index** | **10.35** | **12.38** |
72
 
73
- Standard language-modelling evaluations:
74
 
75
- | Benchmark | v3 | **v3.1** |
76
- |---|---|---|
77
- | WikiText-2 word perplexity ↓ | 57.90 | **55.35** |
78
- | WikiText-2 byte perplexity ↓ | 2.1362 | **2.1182** |
79
- | BLiMP (67-task macro) | 78.49 | **75.09** |
80
 
81
- Our v3 numbers reproduce the published v3 card exactly (byte PPL 2.1362) and the leaderboard entry to within 0.14 II, so these are apples-to-apples.
 
 
82
 
83
- ### The BLiMP tradeoff
84
 
85
- v3.1 is better on every leaderboard metric and on WikiText-2, and worse on BLiMP by 3.4 points — about 22 standard errors, so this is not noise.
86
 
87
- The cause is the data mix. v3's corpus was roughly 83% natural prose. v3.1's is 43% math and 30% synthetic textbook. BLiMP measures syntactic acceptability on minimal pairs, and that is exactly what a math-and-textbook-heavy diet does not teach. We traded some grammar for content and arithmetic, and we would rather say so than leave it for someone to discover.
 
 
 
 
 
 
 
88
 
89
- For reference, the whole frontier we could find is monotonic — every extra point of Intelligence Index costs BLiMP:
90
 
91
- | checkpoint | II | BLiMP |
92
  |---|---|---|
93
- | v3 | 10.35 | 78.49 |
94
- | v3.2 (prose-balanced, 11% math) | 11.72 | 77.00 |
95
- | v3.1 @ 75% of training | 11.98 | 76.56 |
96
- | **v3.1** | **12.38** | **75.09** |
97
 
98
- There is no configuration we found that improves both.
99
 
100
  ---
101
 
102
- ## Sample output
103
 
104
- Temperature 0.8, top_k 40. Base model, not instruction-tuned.
105
 
106
- **Prompt:** `"The capital of France is"`
107
 
108
- > The capital of France is Paris, which is the capital of France.
109
- >
110
- > ### History
111
- > The capital of France was founded in 16...
112
 
113
- **Prompt:** `"To bake a great cake you should always"`
114
 
115
- > To bake a great cake you should always use a good quality baking powder.
116
- >
117
- > ### How to Make a Baked Cake
118
- >
119
- > 1. Preheat your...
120
 
121
- Note the textbook-style formatting: that is the cosmopedia portion of the mix showing through. v3 answers the same prompt with "the capital of the French Empire" and loops.
122
 
123
  ---
124
 
125
  ## Training
126
 
127
- ### Data Mix (31.4B tokens, 1 epoch)
128
 
129
- | Source | Share |
130
- |---|---|
131
- | HuggingFaceTB/finemath (finemath-3plus) | 32.5% |
132
- | HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% |
133
- | HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% |
134
- | HuggingFaceFW/finewiki (en) | 11.9% |
135
- | open-web-math/open-web-math | 10.5% |
136
- | SimpleStories/SimpleStories | 1.4% |
137
 
138
- ### Decontamination
 
 
 
 
 
 
 
139
 
140
- No benchmark text was trained on. Every document was scanned during corpus construction with an Aho-Corasick automaton built from 249,054 patterns covering all train, validation and test splits of ARC-Easy, ARC-Challenge, HellaSwag, PIQA and ArithMark-3. Documents containing a match were dropped.
141
 
142
- ### Hyperparameters
143
 
144
  | Setting | Value |
145
  |---|---|
@@ -156,9 +156,19 @@ No benchmark text was trained on. Every document was scanned during corpus const
156
  | Batch | 128 × 1024 tokens × 8 GPUs |
157
  | Steps | 29,900 |
158
 
 
 
 
 
 
 
 
 
159
  ### Hardware
160
 
161
- Trained on 8× NVIDIA RTX 5090 (32 GB), 2.6 hours.
 
 
162
 
163
  ---
164
 
@@ -182,27 +192,29 @@ out = model.generate(
182
  print(tokenizer.decode(out[0], skip_special_tokens=True))
183
  ```
184
 
185
- `trust_remote_code=True` is required (custom architecture: RoPE + QK-Norm + XSA + SwiGLU + RMSNorm dense decoder).
186
 
187
- The model has no KV-cache implementation; `generate()` recomputes the sequence each step. Correct, but not tuned for long generations.
188
 
189
  ---
190
 
191
  ## Limitations
192
 
193
- - Base model only, not instruction-tuned
194
- - English only
195
- - 1024 token context window; longer inputs are truncated to the last 1024 tokens
196
- - **BLiMP regressed 3.4 points versus v3** — see the tradeoff section
197
- - Arithmetic: ArithMark-3 measures log-likelihood over candidate answers. Free-form generated arithmetic remains weak, same as v3
198
- - Repetition at higher temperatures without a repetition penalty
199
- - Marginal returns: this is one epoch over 31.4B tokens on a 25M model that had already seen 15B; the gains are real but not free
200
 
201
  ---
202
 
203
- ## What's Next
 
 
204
 
205
- The obvious next lever is not more tokens — it is a mix that recovers BLiMP without giving back the ARC/PIQA gains. The v3.2 experiment (11% math, 78% prose) went the other way: II 11.72, BLiMP 77.00. Somewhere between those two mixes is a better point on the frontier. Beyond that, distillation from a ~1.7B teacher is the only realistic route to the 50M-class models on the leaderboard.
206
 
207
  ---
208
 
 
5
  library_name: transformers
6
  tags:
7
  - ivme
8
+ - small-language-model
9
  - language-model
10
  - transformer
11
  - rope
 
14
  - xsa
15
  - qk-norm
16
  - continued-pretraining
17
+ - dense-data
18
+ - educational
19
  - custom_code
20
  - text-generation
21
  datasets:
 
32
 
33
  ![Conversate-v3.1 Logo](https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/mbZUbkYIjqvwdpcPac6Mu.png)
34
 
35
+ **İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight.
36
 
37
+ v3.1 is not a new architecture and not a bigger model. It is the same 25M-parameter v3 transformer, unchanged down to the tokenizer, put back in the oven for another 31.4B tokens — only this time we changed the food.
38
+
39
+ That is the whole idea, and it is worth stating plainly because it is easy to misread. When a model is this small, parameters are not what is holding it back. v3 had already seen 15B tokens, which is roughly thirty times past Chinchilla-optimal for its size. Adding more of the same web text had stopped buying much. What had *not* been tried was changing the composition: less open-web prose, far more mathematics, far more textbook-style explanation, far more encyclopedic reference.
40
+
41
+ So we held the model fixed and moved the data. The result is a model that knows more and reasons a little better, and — this part matters — is slightly worse at grammar. Both effects are real, both come from the same cause, and both are documented below rather than averaged into a single happy number.
42
+
43
+ If you want the one-sentence version: **v3.1 is v3 with more content and less syntax.** Which of those you want depends on what you are doing with it.
44
 
45
  ---
46
 
 
62
  | Embeddings | Tied input/output |
63
  | Biases | None |
64
 
65
+ **Why the parameter count differs from v3.** v3 reports 24,787,520; v3.1 reports 24,956,480. The FFN was zero-padded from 853 to 864 so that the GEMMs tile cleanly on tensor cores — 853 is prime, which is a surprisingly expensive thing for a hyperparameter to be. Because `silu(0) = 0` and the up-branch of the eleven added units is zero, they contribute exactly nothing to the output. The model is functionally identical to an 853-wide one; it is just 60–85% faster on FFN matmuls during training. No behaviour change, no retraining required.
66
 
67
  ---
68
 
69
+ ## What it is like to use
70
 
71
+ It is a base model. It continues text; it does not follow instructions, and it will not answer a question so much as write the kind of document in which that question might appear.
72
 
73
+ That last property is the most noticeable thing about v3.1. Because a large share of its training data was textbook-shaped, it has picked up the *structure* of explanation — headings, numbered steps, definitions, "###" section markers — and it reaches for those structures readily. Ask it about cakes and you get a recipe outline. Ask it about a country and you get something that reads like an encyclopedia stub.
 
 
 
 
 
 
 
74
 
75
+ It is also noticeably more willing to commit to a fact than v3 was. v3, given "The capital of France is", would often slide into a loop about "the capital of the French Empire". v3.1 says Paris. That is a small thing, and one example is not a benchmark, but it is the kind of difference the numbers below are trying to describe.
76
 
77
+ What it is *not* good at: long-range syntax, unusual sentence constructions, and anything requiring genuine multi-step reasoning. It is 25M parameters. It has read a great deal and understood a little.
 
 
 
 
78
 
79
+ ---
80
+
81
+ ## Measurements
82
 
83
+ All numbers below were produced on the same machine, with the same harness, against the same checkpoints — not copied from a leaderboard and not measured at different times under different settings.
84
 
85
+ The harness is [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.13, 0-shot, `acc_norm`, plus [ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) run through its official scoring script.
86
 
87
+ | Benchmark | v3 | v3.1 | direction |
88
+ |---|---|---|---|
89
+ | ARC-Easy | 39.02 | 40.15 | up |
90
+ | ARC-Challenge | 24.32 | 25.00 | up |
91
+ | HellaSwag | 28.90 | 29.52 | up |
92
+ | PIQA | 57.67 | 59.25 | up |
93
+ | ArithMark-3 | 35.20 | 37.20 | up |
94
+ | BLiMP (67-task macro) | 78.49 | 75.09 | **down** |
95
 
96
+ Supporting language-modelling numbers:
97
 
98
+ | Benchmark | v3 | v3.1 |
99
  |---|---|---|
100
+ | WikiText-2 word perplexity ↓ | 57.90 | 55.35 |
101
+ | WikiText-2 byte perplexity ↓ | 2.1362 | 2.1182 |
 
 
102
 
103
+ A note on how to read these. Our v3 baseline reproduces the published v3 card exactly (byte perplexity 2.1362) and the public leaderboard entry to within 0.14 index points, so the comparison is honest in both directions. Composite benchmark noise at this scale is roughly ±0.75 points on the index, which means the *direction* of every row above is solid while individual moves of half a point should not be over-read. v3.1 is not a different model class; it is a better-fed one.
104
 
105
  ---
106
 
107
+ ## The BLiMP tradeoff, stated honestly
108
 
109
+ BLiMP fell 3.4 points — about 22 standard errors, so this is not noise.
110
 
111
+ The mechanism is not mysterious. BLiMP asks whether a sentence is syntactically acceptable, using minimal pairs that differ in one small grammatical way. Answering those well requires having read an enormous amount of *ordinary prose* — the kind of text where subject-verb agreement, determiner choice and anaphora appear naturally and constantly.
112
 
113
+ v3 was trained on a mix that was roughly 83% natural prose. v3.1's mix is 43% mathematics and 30% synthetic textbook. Math is not prose. Textbook explanation is not prose either — it is closer to a list of facts wearing a sentence costume. We optimized for density, and density is not the same thing as naturalness.
 
 
 
114
 
115
+ We could have buried this. A model card that reports five improvements and omits one regression is technically not lying, and it is extremely common. We would rather you knew, because it changes which checkpoint you should download:
116
 
117
+ - **v3.1** — best content and arithmetic, weakest syntax.
118
+ - **v3** — best syntax, least content.
119
+ - **v3.2** (a separate, prose-balanced sibling) — the middle: index 11.72, BLiMP 77.00.
 
 
120
 
121
+ Every checkpoint we trained sits on the same monotonic curve. Extra content costs syntax; there is no free lunch anywhere on it. Reporting only one end of that curve would be leaving out the interesting part.
122
 
123
  ---
124
 
125
  ## Training
126
 
127
+ ### The data
128
 
129
+ 31.4B tokens, one epoch, drawn from six sources.
 
 
 
 
 
 
 
130
 
131
+ | Source | Share | Why |
132
+ |---|---|---|
133
+ | HuggingFaceTB/finemath (finemath-3plus) | 32.5% | dense mathematical reasoning |
134
+ | HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% | synthetic textbook explanation |
135
+ | HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% | general educational web |
136
+ | HuggingFaceFW/finewiki (en) | 11.9% | encyclopedic reference |
137
+ | open-web-math/open-web-math | 10.5% | mathematical web text |
138
+ | SimpleStories/SimpleStories | 1.4% | narrative and causal chains |
139
 
140
+ That mix was not chosen to look good in a table. It was chosen after watching v3 plateau: at this scale, the marginal value of another token of generic web text had fallen close to zero, while the marginal value of a token of dense mathematics or structured explanation had not. We were wrong about the size of the syntax cost, and we were right about the content gain.
141
 
142
+ ### The recipe
143
 
144
  | Setting | Value |
145
  |---|---|
 
156
  | Batch | 128 × 1024 tokens × 8 GPUs |
157
  | Steps | 29,900 |
158
 
159
+ Muon on the body and AdamW on the embeddings and norm gains follows the recipe v3 was trained with. Using a single optimizer for both would have changed two variables at once, and this experiment was already changing one.
160
+
161
+ ### Decontamination
162
+
163
+ No benchmark text was trained on, and that was enforced mechanically rather than promised.
164
+
165
+ During corpus construction every document was scanned with an Aho-Corasick automaton built from 249,054 patterns covering the train, validation *and* test splits of ARC-Easy, ARC-Challenge, HellaSwag, PIQA and ArithMark-3. Any document containing a match was dropped before tokenisation. This is the part of the pipeline that costs the most time and produces nothing visible, which is exactly why it is worth stating.
166
+
167
  ### Hardware
168
 
169
+ Eight RTX 5090s, 2.6 hours.
170
+
171
+ For context: the entire run — corpus filtering, tokenisation, training — fits inside a working day on consumer cards. That is the point of models this size. They are cheap enough to be *experiments* rather than commitments, and you can afford to run the version that gets one thing wrong so you know what it costs.
172
 
173
  ---
174
 
 
192
  print(tokenizer.decode(out[0], skip_special_tokens=True))
193
  ```
194
 
195
+ `trust_remote_code=True` is required — this is a custom architecture (RoPE + QK-Norm + XSA + SwiGLU + RMSNorm dense decoder), and the config and modeling files ship with the weights.
196
 
197
+ Two implementation notes. The model has no KV-cache implementation, so `generate()` recomputes the sequence each step; that is correct but not tuned for long generations. And inputs longer than 1024 tokens are silently truncated to the last 1024, so chunk long documents yourself rather than trusting the model to notice.
198
 
199
  ---
200
 
201
  ## Limitations
202
 
203
+ - Base model, not instruction-tuned. It will not follow instructions.
204
+ - English only.
205
+ - 1024-token context; longer inputs are truncated to the most recent window.
206
+ - **Syntax regressed versus v3** (BLiMP −3.4). If your task is grammaticality-sensitive, v3 or v3.2 is the better pick.
207
+ - Arithmetic gains are recognition-style. ArithMark-3 measures log-likelihood over candidate continuations; free-form generated arithmetic is still frequently wrong, in the same way v3's was. Do not use this for actual computation.
208
+ - Repetition at high temperatures without a repetition penalty, as with the rest of the series.
209
+ - Diminishing returns: this is one epoch on a model that had already seen 15B tokens. The improvements are real but they are the second-order kind.
210
 
211
  ---
212
 
213
+ ## What's next
214
+
215
+ The open question is whether the tradeoff curve can be bent rather than moved along. v3.2 suggests the middle is real (index 11.72, BLiMP 77.00) but it does not suggest the curve is convex — the points we have all lie close to a straight line, which is either a property of this data mix or a property of a 25M model, and we do not yet know which.
216
 
217
+ Beyond that, the honest answer is that 25M parameters will not reach the 50M–100M class by data alone. Distillation from a ~1.7B teacher is the only route we see to that frontier, and it is a different experiment with a different flavour of tradeoff.
218
 
219
  ---
220