ereniko commited on
Commit
dabc9bb
·
verified ·
1 Parent(s): b78e2cb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +50 -83
README.md CHANGED
@@ -32,29 +32,21 @@ pipeline_tag: text-generation
32
 
33
  ![Conversate-v3.1 Logo](https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/mbZUbkYIjqvwdpcPac6Mu.png)
34
 
35
- **İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight.
36
 
37
- v3.1 is not a new architecture and not a bigger model. It is the same 25M-parameter v3 transformer, unchanged down to the tokenizer, put back in the oven for another 31.4B tokens — only this time we changed the food.
38
 
39
- That is the whole idea, and it is worth stating plainly because it is easy to misread. When a model is this small, parameters are not what is holding it back. v3 had already seen 15B tokens, which is roughly thirty times past Chinchilla-optimal for its size. Adding more of the same web text had stopped buying much. What had *not* been tried was changing the composition: less open-web prose, far more mathematics, far more textbook-style explanation, far more encyclopedic reference.
40
-
41
- So we held the model fixed and moved the data. The result is a model that knows more and reasons a little better, and — this part matters — is slightly worse at grammar. Both effects are real, both come from the same cause, and both are documented below rather than averaged into a single happy number.
42
-
43
- If you want the one-sentence version: **v3.1 is v3 with more content and less syntax.** Which of those you want depends on what you are doing with it.
44
-
45
- ---
46
-
47
- ## Model Details
48
 
49
  | Parameter | Value |
50
  |---|---|
51
- | Architecture | Decoder-only transformer, dense (no loops, no exotic recurrence) |
52
  | Parameters | 24,956,480 |
53
  | Layers | 16 |
54
  | Hidden dim | 320 |
55
  | FFN dim | SwiGLU, 864 |
56
- | Attention heads | 8, full attention (no GQA), head_dim 40 |
57
- | Attention variant | QK-Norm + XSA (Exclusive Self Attention, arXiv:2603.09078) |
58
  | Context length | 1024 tokens |
59
  | Vocab size | 16,000 (custom digit-atomic BPE) |
60
  | Positional encoding | RoPE (θ=10,000) |
@@ -62,29 +54,33 @@ If you want the one-sentence version: **v3.1 is v3 with more content and less sy
62
  | Embeddings | Tied input/output |
63
  | Biases | None |
64
 
65
- **Why the parameter count differs from v3.** v3 reports 24,787,520; v3.1 reports 24,956,480. The FFN was zero-padded from 853 to 864 so that the GEMMs tile cleanly on tensor cores — 853 is prime, which is a surprisingly expensive thing for a hyperparameter to be. Because `silu(0) = 0` and the up-branch of the eleven added units is zero, they contribute exactly nothing to the output. The model is functionally identical to an 853-wide one; it is just 60–85% faster on FFN matmuls during training. No behaviour change, no retraining required.
66
 
67
- ---
68
 
69
- ## What it is like to use
70
 
71
- It is a base model. It continues text; it does not follow instructions, and it will not answer a question so much as write the kind of document in which that question might appear.
72
 
73
- That last property is the most noticeable thing about v3.1. Because a large share of its training data was textbook-shaped, it has picked up the *structure* of explanation — headings, numbered steps, definitions, "###" section markers — and it reaches for those structures readily. Ask it about cakes and you get a recipe outline. Ask it about a country and you get something that reads like an encyclopedia stub.
74
 
75
- It is also noticeably more willing to commit to a fact than v3 was. v3, given "The capital of France is", would often slide into a loop about "the capital of the French Empire". v3.1 says Paris. That is a small thing, and one example is not a benchmark, but it is the kind of difference the numbers below are trying to describe.
76
 
77
- What it is *not* good at: long-range syntax, unusual sentence constructions, and anything requiring genuine multi-step reasoning. It is 25M parameters. It has read a great deal and understood a little.
78
 
79
- ---
 
 
 
 
80
 
81
  ## Measurements
82
 
83
- All numbers below were produced on the same machine, with the same harness, against the same checkpoints — not copied from a leaderboard and not measured at different times under different settings.
84
 
85
- The harness is [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.13, 0-shot, `acc_norm`, plus [ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) run through its official scoring script.
86
 
87
- | Benchmark | v3 | v3.1 | direction |
88
  |---|---|---|---|
89
  | ARC-Easy | 39.02 | 40.15 | up |
90
  | ARC-Challenge | 24.32 | 25.00 | up |
@@ -93,53 +89,39 @@ The harness is [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluati
93
  | ArithMark-3 | 35.20 | 37.20 | up |
94
  | BLiMP (67-task macro) | 78.49 | 75.09 | **down** |
95
 
96
- Supporting language-modelling numbers:
97
 
98
  | Benchmark | v3 | v3.1 |
99
  |---|---|---|
100
  | WikiText-2 word perplexity ↓ | 57.90 | 55.35 |
101
  | WikiText-2 byte perplexity ↓ | 2.1362 | 2.1182 |
102
 
103
- A note on how to read these. Our v3 baseline reproduces the published v3 card exactly (byte perplexity 2.1362) and the public leaderboard entry to within 0.14 index points, so the comparison is honest in both directions. Composite benchmark noise at this scale is roughly ±0.75 points on the index, which means the *direction* of every row above is solid while individual moves of half a point should not be over-read. v3.1 is not a different model class; it is a better-fed one.
104
-
105
- ---
106
 
107
- ## The BLiMP tradeoff, stated honestly
108
 
109
- BLiMP fell 3.4 points — about 22 standard errors, so this is not noise.
110
 
111
- The mechanism is not mysterious. BLiMP asks whether a sentence is syntactically acceptable, using minimal pairs that differ in one small grammatical way. Answering those well requires having read an enormous amount of *ordinary prose* — the kind of text where subject-verb agreement, determiner choice and anaphora appear naturally and constantly.
112
 
113
- v3 was trained on a mix that was roughly 83% natural prose. v3.1's mix is 43% mathematics and 30% synthetic textbook. Math is not prose. Textbook explanation is not prose either — it is closer to a list of facts wearing a sentence costume. We optimized for density, and density is not the same thing as naturalness.
114
 
115
- We could have buried this. A model card that reports five improvements and omits one regression is technically not lying, and it is extremely common. We would rather you knew, because it changes which checkpoint you should download:
116
 
117
- - **v3.1** — best content and arithmetic, weakest syntax.
118
- - **v3** — best syntax, least content.
119
- - **v3.2** (a separate, prose-balanced sibling) — the middle: index 11.72, BLiMP 77.00.
120
-
121
- Every checkpoint we trained sits on the same monotonic curve. Extra content costs syntax; there is no free lunch anywhere on it. Reporting only one end of that curve would be leaving out the interesting part.
122
-
123
- ---
124
-
125
- ## Training
126
-
127
- ### The data
128
-
129
- 31.4B tokens, one epoch, drawn from six sources.
130
 
131
  | Source | Share | Why |
132
  |---|---|---|
133
- | HuggingFaceTB/finemath (finemath-3plus) | 32.5% | dense mathematical reasoning |
134
- | HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% | synthetic textbook explanation |
135
- | HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% | general educational web |
136
- | HuggingFaceFW/finewiki (en) | 11.9% | encyclopedic reference |
137
- | open-web-math/open-web-math | 10.5% | mathematical web text |
138
- | SimpleStories/SimpleStories | 1.4% | narrative and causal chains |
139
 
140
- That mix was not chosen to look good in a table. It was chosen after watching v3 plateau: at this scale, the marginal value of another token of generic web text had fallen close to zero, while the marginal value of a token of dense mathematics or structured explanation had not. We were wrong about the size of the syntax cost, and we were right about the content gain.
141
 
142
- ### The recipe
143
 
144
  | Setting | Value |
145
  |---|---|
@@ -156,21 +138,17 @@ That mix was not chosen to look good in a table. It was chosen after watching v3
156
  | Batch | 128 × 1024 tokens × 8 GPUs |
157
  | Steps | 29,900 |
158
 
159
- Muon on the body and AdamW on the embeddings and norm gains follows the recipe v3 was trained with. Using a single optimizer for both would have changed two variables at once, and this experiment was already changing one.
160
 
161
  ### Decontamination
162
 
163
- No benchmark text was trained on, and that was enforced mechanically rather than promised.
164
 
165
- During corpus construction every document was scanned with an Aho-Corasick automaton built from 249,054 patterns covering the train, validation *and* test splits of ARC-Easy, ARC-Challenge, HellaSwag, PIQA and ArithMark-3. Any document containing a match was dropped before tokenisation. This is the part of the pipeline that costs the most time and produces nothing visible, which is exactly why it is worth stating.
166
 
167
  ### Hardware
168
 
169
- Eight RTX 5090s, 2.6 hours.
170
-
171
- For context: the entire run — corpus filtering, tokenisation, training — fits inside a working day on consumer cards. That is the point of models this size. They are cheap enough to be *experiments* rather than commitments, and you can afford to run the version that gets one thing wrong so you know what it costs.
172
-
173
- ---
174
 
175
  ## Inference
176
 
@@ -192,31 +170,20 @@ out = model.generate(
192
  print(tokenizer.decode(out[0], skip_special_tokens=True))
193
  ```
194
 
195
- `trust_remote_code=True` is required — this is a custom architecture (RoPE + QK-Norm + XSA + SwiGLU + RMSNorm dense decoder), and the config and modeling files ship with the weights.
196
 
197
- Two implementation notes. The model has no KV-cache implementation, so `generate()` recomputes the sequence each step; that is correct but not tuned for long generations. And inputs longer than 1024 tokens are silently truncated to the last 1024, so chunk long documents yourself rather than trusting the model to notice.
198
-
199
- ---
200
 
201
  ## Limitations
202
 
203
- - Base model, not instruction-tuned. It will not follow instructions.
204
- - English only.
205
- - 1024-token context; longer inputs are truncated to the most recent window.
206
- - **Syntax regressed versus v3** (BLiMP −3.4). If your task is grammaticality-sensitive, v3 or v3.2 is the better pick.
207
- - Arithmetic gains are recognition-style. ArithMark-3 measures log-likelihood over candidate continuations; free-form generated arithmetic is still frequently wrong, in the same way v3's was. Do not use this for actual computation.
208
- - Repetition at high temperatures without a repetition penalty, as with the rest of the series.
209
- - Diminishing returns: this is one epoch on a model that had already seen 15B tokens. The improvements are real but they are the second-order kind.
210
-
211
- ---
212
-
213
- ## What's next
214
-
215
- The open question is whether the tradeoff curve can be bent rather than moved along. v3.2 suggests the middle is real (index 11.72, BLiMP 77.00) but it does not suggest the curve is convex — the points we have all lie close to a straight line, which is either a property of this data mix or a property of a 25M model, and we do not yet know which.
216
 
217
- Beyond that, the honest answer is that 25M parameters will not reach the 50M–100M class by data alone. Distillation from a ~1.7B teacher is the only route we see to that frontier, and it is a different experiment with a different flavour of tradeoff.
218
 
219
- ---
220
 
221
  ## Citation
222
 
@@ -232,4 +199,4 @@ Beyond that, the honest answer is that 25M parameters will not reach the 50M–1
232
 
233
  ---
234
 
235
- *Built by IvmeLabs. Small models, deliberate choices.*
 
32
 
33
  ![Conversate-v3.1 Logo](https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/mbZUbkYIjqvwdpcPac6Mu.png)
34
 
35
+ İvme-Conversate-v3.1-Base is a 25M-parameter AI model based on [İvme-Conversate-v3-Base](https://huggingface.co/IvmeLabs/Ivme-Conversate-v3-Base). It was trained through continued pretraining to fix v3's problem of having good grammar but weak knowledge. Though even with the increased knowledge and capabilities in v3.1, benchmarks show a decrease in grammar performance.
36
 
37
+ Conversate-v3 really wasn't the update I was expecting. It showed me that yeah, architecture can help at this size, but data matters just as much when your model is this small. Conversate-v3.1 basically just solves that. Instead of another architecture change, I decided to go with the v2 playbook. Just feed it more data. To my surprise, even though I expected the model to be oversaturated by now, it worked.
38
 
39
+ ## Architecture
 
 
 
 
 
 
 
 
40
 
41
  | Parameter | Value |
42
  |---|---|
43
+ | Architecture | Decoder-only transformer, dense (Same as v3, nothing magic.) |
44
  | Parameters | 24,956,480 |
45
  | Layers | 16 |
46
  | Hidden dim | 320 |
47
  | FFN dim | SwiGLU, 864 |
48
+ | Attention heads | 8, full attention (no GQA), head_dim 40 (It's a bit of an odd number for GPUs, which makes v3 and v3.1 less efficient on GPUs) |
49
+ | Attention variant | QK-Norm + XSA (Exclusive Self-Attention, arXiv:2603.09078) |
50
  | Context length | 1024 tokens |
51
  | Vocab size | 16,000 (custom digit-atomic BPE) |
52
  | Positional encoding | RoPE (θ=10,000) |
 
54
  | Embeddings | Tied input/output |
55
  | Biases | None |
56
 
57
+ Its parameter count isn't exactly identical to v3. The FFN got padded from 853 to 864 so it makes better use of GEMMs. This didn't need retraining or anything. v3.1 just got a little chunkier for higher speeds. Still sub-25M.
58
 
59
+ ### Explanation for a Few Things
60
 
61
+ Most of these are pretty standard, so I'll only explain the special parts.
62
 
63
+ XSA (Exclusive Self-Attention) is a little attention mechanism discovered by Apple. It makes it so that each token looks at its neighboring tokens instead of focusing on itself. This allows the model to spend more effort on actual token-to-token relationships instead of spending compute on the token it already sees and doesn't need to process as much.
64
 
65
+ QK-Norm makes it so that the query and key values don't go crazy during long training runs and start drowning out other values and messing things up.
66
 
67
+ Digit-atomic BPE makes the tokenizer give every digit its own unique token (so instead of clumping 123 together, it separates them into 1 2 3, even if the data suggests clumping them). This makes math easier since the model can see and understand the relationship between individual digits. Still, like v3 showed, it's not a magic gateway. You still need good training to actually utilize it well. Let's just say Conversate models aren't really warm with math.
68
 
69
+ ## So What Does This Model Output?
70
 
71
+ It's kinda like a very intelligent autocomplete. It has no instruction training (the type of training where you feed an AI model user queries and AI model answers so it can chat), so it won't actually answer your questions. Though you can try formatting your text to mimic questions (like "The capital of France is". In that case, the model completes it as "The capital of France is Paris").
72
+
73
+ v3.1 is more stubborn than v3, in a good way. It tends to stick to its facts instead of constantly going into loops.
74
+
75
+ But this won't compete with, say, a 100M model. v3.1 is very good, but it can't do complex multi-step jobs or actually be ready for real-life usage.
76
 
77
  ## Measurements
78
 
79
+ ### Benchmarks
80
 
81
+ All benchmarks were run using lm-eval-harness. To make sure the readings were correct, v3 was also benchmarked in the same training environment. So expect these v3 scores to differ slightly from what might be on the model card or leaderboards.
82
 
83
+ | Benchmark | v3 | v3.1 | Direction |
84
  |---|---|---|---|
85
  | ARC-Easy | 39.02 | 40.15 | up |
86
  | ARC-Challenge | 24.32 | 25.00 | up |
 
89
  | ArithMark-3 | 35.20 | 37.20 | up |
90
  | BLiMP (67-task macro) | 78.49 | 75.09 | **down** |
91
 
92
+ Supporting language-modeling numbers:
93
 
94
  | Benchmark | v3 | v3.1 |
95
  |---|---|---|
96
  | WikiText-2 word perplexity ↓ | 57.90 | 55.35 |
97
  | WikiText-2 byte perplexity ↓ | 2.1362 | 2.1182 |
98
 
99
+ (You'll mostly only see byte perplexity in actual benchmarks, but just in case, I still included word perplexity.)
 
 
100
 
101
+ ### But BLiMP Is Down?
102
 
103
+ Yes, it fell by 3.4 points. Conversate-v3.1 genuinely has worse grammar than v3. The reason is pretty simple: I stopped prioritizing BLiMP. Older Conversate releases had pretty high BLiMP scores, but everything else came out disappointing. And trust me, in real usage, BLiMP shouldn't matter this much. Conversate-v3.1 is also still very close to its competitors, even with the lower BLiMP score.
104
 
105
+ Actually, there's a secret v3.2 I trained to fix this. But the BLiMP fix brought down the benchmarks I actually cared about (II of 11.72). So I decided to go with v3.1.
106
 
107
+ ## Training Thingies
108
 
109
+ ### Data
110
 
111
+ 31.4B tokens in total, just over a single epoch.
 
 
 
 
 
 
 
 
 
 
 
 
112
 
113
  | Source | Share | Why |
114
  |---|---|---|
115
+ | HuggingFaceTB/finemath (finemath-3plus) | 32.5% | mathematical reasoning to help math performance |
116
+ | HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% | synthetic textbook explanations to boost knowledge |
117
+ | HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% | general educational web text, added just in case. Weirdly enough, it helps |
118
+ | HuggingFaceFW/finewiki (en) | 11.9% | encyclopedic references for boring but valuable information |
119
+ | open-web-math/open-web-math | 10.5% | mathematical web text for even more math |
120
+ | SimpleStories/SimpleStories | 1.4% | narrative and causal chains so it's ready for conversation in any future instruct fine-tune |
121
 
122
+ Generic web text wasn't really moving v3 anymore. So I decided to go with this mix, designed specifically to patch v3's weak points. As you can see, it lines up pretty cleanly with the benchmark scores. Less generic text likely caused the worse BLiMP score, but the higher-knowledge sources brought everything else up.
123
 
124
+ ### The Recipe
125
 
126
  | Setting | Value |
127
  |---|---|
 
138
  | Batch | 128 × 1024 tokens × 8 GPUs |
139
  | Steps | 29,900 |
140
 
141
+ Using Muon on the body weights and AdamW on the embeddings and norm gains follows the same recipe v3 was trained with. Muon tends to help with training costs and stuff, plus it's just safer to stick with it since v3 was trained with Muon too.
142
 
143
  ### Decontamination
144
 
145
+ No benchmark text was trained on, and that was enforced mechanically rather than just promised.
146
 
147
+ The pipeline checked all the training data against the benchmark splits from every benchmark I used. It took a long time to finish, but it's a good way to show that v3.1 isn't benchmaxxed.
148
 
149
  ### Hardware
150
 
151
+ Eight RTX 5090s. Training alone took about 2.6 hours. It was running at around 3.4M tok/s, if I remember correctly.
 
 
 
 
152
 
153
  ## Inference
154
 
 
170
  print(tokenizer.decode(out[0], skip_special_tokens=True))
171
  ```
172
 
173
+ It needs `trust_remote_code=True` because of the custom architecture. Also, a few warnings: it has no KV cache, so it has to reprocess the context for every new token during generation. I don't think it'll matter too much for a 25M model, but just be aware.
174
 
175
+ And after 1024 tokens, your prompts will be silently truncated, so don't get confused if it suddenly starts missing details. It can't even keep 256 tokens cleanly, but still :P
 
 
176
 
177
  ## Limitations
178
 
179
+ - It's base only, so it can't chat
180
+ - Definitely not AGI
181
+ - BLiMP regressed
182
+ - English only (I know the org is called İvme, but I'm still not sure how I'm supposed to fit Turkish into an SLM)
 
 
 
 
 
 
 
 
 
183
 
184
+ ## What's Next
185
 
186
+ To be honest? Definitely Conversate-v4 after this. Conversate-v3 and v3.1 have too many architectural inefficiencies and problems that gave me way too many headaches. I'll start from scratch.
187
 
188
  ## Citation
189
 
 
199
 
200
  ---
201
 
202
+ *By Eren Ekşi*