File size: 9,623 Bytes
07d8f8e
 
 
 
 
 
 
b78e2cb
07d8f8e
 
 
 
 
 
 
 
b78e2cb
 
07d8f8e
 
 
 
 
 
 
 
 
 
dcde018
 
07d8f8e
 
 
 
5e6b9f5
07d8f8e
dabc9bb
07d8f8e
dabc9bb
b78e2cb
dabc9bb
07d8f8e
 
 
dabc9bb
07d8f8e
 
 
 
dabc9bb
 
07d8f8e
 
 
 
 
 
 
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
 
 
 
 
b78e2cb
 
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
b78e2cb
 
 
 
 
 
 
07d8f8e
dabc9bb
07d8f8e
b78e2cb
07d8f8e
b78e2cb
 
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
b78e2cb
 
dabc9bb
 
 
 
 
 
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dabc9bb
b78e2cb
 
 
dabc9bb
b78e2cb
dabc9bb
b78e2cb
07d8f8e
 
dabc9bb
07d8f8e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dabc9bb
07d8f8e
dabc9bb
07d8f8e
 
 
dabc9bb
 
 
 
07d8f8e
dabc9bb
07d8f8e
dabc9bb
07d8f8e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dabc9bb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- ivme
- small-language-model
- language-model
- transformer
- rope
- swiglu
- muon
- xsa
- qk-norm
- continued-pretraining
- dense-data
- educational
- custom_code
- text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/smollm-corpus
- HuggingFaceFW/finewiki
- HuggingFaceTB/finemath
- open-web-math/open-web-math
- SimpleStories/SimpleStories
pipeline_tag: text-generation
thumbnail: https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/j5dtGRhvqaQPIjyUK5e0Q.png

---

# İvme-Conversate-v3.1-Base

![Conversate-v3.1 Logo](https://files.catbox.moe/kia91k.apng)

İvme-Conversate-v3.1-Base is a 25M-parameter AI model based on [İvme-Conversate-v3-Base](https://huggingface.co/IvmeLabs/Ivme-Conversate-v3-Base). It was trained through continued pretraining to fix v3's problem of having good grammar but weak knowledge. Though even with the increased knowledge and capabilities in v3.1, benchmarks show a decrease in grammar performance.

Conversate-v3 really wasn't the update I was expecting. It showed me that yeah, architecture can help at this size, but data matters just as much when your model is this small. Conversate-v3.1 basically just solves that. Instead of another architecture change, I decided to go with the v2 playbook. Just feed it more data. To my surprise, even though I expected the model to be oversaturated by now, it worked.

## Architecture

| Parameter | Value |
|---|---|
| Architecture | Decoder-only transformer, dense (Same as v3, nothing magic.) |
| Parameters | 24,956,480 |
| Layers | 16 |
| Hidden dim | 320 |
| FFN dim | SwiGLU, 864 |
| Attention heads | 8, full attention (no GQA), head_dim 40 (It's a bit of an odd number for GPUs, which makes v3 and v3.1 less efficient on GPUs) |
| Attention variant | QK-Norm + XSA (Exclusive Self-Attention, arXiv:2603.09078) |
| Context length | 1024 tokens |
| Vocab size | 16,000 (custom digit-atomic BPE) |
| Positional encoding | RoPE (θ=10,000) |
| Normalization | RMSNorm (pre-norm) |
| Embeddings | Tied input/output |
| Biases | None |

Its parameter count isn't exactly identical to v3. The FFN got padded from 853 to 864 so it makes better use of GEMMs. This didn't need retraining or anything. v3.1 just got a little chunkier for higher speeds. Still sub-25M.

### Explanation for a Few Things

Most of these are pretty standard, so I'll only explain the special parts.

XSA (Exclusive Self-Attention) is a little attention mechanism discovered by Apple. It makes it so that each token looks at its neighboring tokens instead of focusing on itself. This allows the model to spend more effort on actual token-to-token relationships instead of spending compute on the token it already sees and doesn't need to process as much.

QK-Norm makes it so that the query and key values don't go crazy during long training runs and start drowning out other values and messing things up.

Digit-atomic BPE makes the tokenizer give every digit its own unique token (so instead of clumping 123 together, it separates them into 1 2 3, even if the data suggests clumping them). This makes math easier since the model can see and understand the relationship between individual digits. Still, like v3 showed, it's not a magic gateway. You still need good training to actually utilize it well. Let's just say Conversate models aren't really warm with math.

## So What Does This Model Output?

It's kinda like a very intelligent autocomplete. It has no instruction training (the type of training where you feed an AI model user queries and AI model answers so it can chat), so it won't actually answer your questions. Though you can try formatting your text to mimic questions (like "The capital of France is". In that case, the model completes it as "The capital of France is Paris").

v3.1 is more stubborn than v3, in a good way. It tends to stick to its facts instead of constantly going into loops.

But this won't compete with, say, a 100M model. v3.1 is very good, but it can't do complex multi-step jobs or actually be ready for real-life usage.

## Measurements

### Benchmarks

All benchmarks were run using lm-eval-harness. To make sure the readings were correct, v3 was also benchmarked in the same training environment. So expect these v3 scores to differ slightly from what might be on the model card or leaderboards.

| Benchmark | v3 | v3.1 | Direction |
|---|---|---|---|
| ARC-Easy | 39.02 | 40.15 | up |
| ARC-Challenge | 24.32 | 25.00 | up |
| HellaSwag | 28.90 | 29.52 | up |
| PIQA | 57.67 | 59.25 | up |
| ArithMark-3 | 35.20 | 37.20 | up |
| BLiMP (67-task macro) | 78.49 | 75.09 | **down** |

Supporting language-modeling numbers:

| Benchmark | v3 | v3.1 |
|---|---|---|
| WikiText-2 word perplexity ↓ | 57.90 | 55.35 |
| WikiText-2 byte perplexity ↓ | 2.1362 | 2.1182 |

(You'll mostly only see byte perplexity in actual benchmarks, but just in case, I still included word perplexity.)

### But BLiMP Is Down?

Yes, it fell by 3.4 points. Conversate-v3.1 genuinely has worse grammar than v3. The reason is pretty simple: I stopped prioritizing BLiMP. Older Conversate releases had pretty high BLiMP scores, but everything else came out disappointing. And trust me, in real usage, BLiMP shouldn't matter this much. Conversate-v3.1 is also still very close to its competitors, even with the lower BLiMP score.

Actually, there's a secret v3.2 I trained to fix this. But the BLiMP fix brought down the benchmarks I actually cared about (II of 11.72). So I decided to go with v3.1.

## Training Thingies

### Data

31.4B tokens in total, just over a single epoch.

| Source | Share | Why |
|---|---|---|
| HuggingFaceTB/finemath (finemath-3plus) | 32.5% | mathematical reasoning to help math performance |
| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% | synthetic textbook explanations to boost knowledge |
| HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% | general educational web text, added just in case. Weirdly enough, it helps |
| HuggingFaceFW/finewiki (en) | 11.9% | encyclopedic references for boring but valuable information |
| open-web-math/open-web-math | 10.5% | mathematical web text for even more math |
| SimpleStories/SimpleStories | 1.4% | narrative and causal chains so it's ready for conversation in any future instruct fine-tune |

Generic web text wasn't really moving v3 anymore. So I decided to go with this mix, designed specifically to patch v3's weak points. As you can see, it lines up pretty cleanly with the benchmark scores. Less generic text likely caused the worse BLiMP score, but the higher-knowledge sources brought everything else up.

### The Recipe

| Setting | Value |
|---|---|
| Optimizer | Muon (body weights) + AdamW (embeddings, norms) |
| Muon lr | 5e-4 |
| AdamW lr | 1e-5 |
| LR schedule | Warmup-Stable-Decay (WSD) |
| Warmup steps | 500 |
| Decay fraction | 10% of training |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | bfloat16 (fp32 master weights) |
| Attention | PyTorch `scaled_dot_product_attention` |
| Batch | 128 × 1024 tokens × 8 GPUs |
| Steps | 29,900 |

Using Muon on the body weights and AdamW on the embeddings and norm gains follows the same recipe v3 was trained with. Muon tends to help with training costs and stuff, plus it's just safer to stick with it since v3 was trained with Muon too.

### Decontamination

No benchmark text was trained on, and that was enforced mechanically rather than just promised.

The pipeline checked all the training data against the benchmark splits from every benchmark I used. It took a long time to finish, but it's a good way to show that v3.1 isn't benchmaxxed.

### Hardware

Eight RTX 5090s. Training alone took about 2.6 hours. It was running at around 3.4M tok/s, if I remember correctly.

## Inference

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-v3.1-Base", trust_remote_code=True, dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-v3.1-Base", trust_remote_code=True)
model.eval()

inputs = tokenizer("Once upon a time, there was a", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=40, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

It needs `trust_remote_code=True` because of the custom architecture. Also, a few warnings: it has no KV cache, so it has to reprocess the context for every new token during generation. I don't think it'll matter too much for a 25M model, but just be aware.

And after 1024 tokens, your prompts will be silently truncated, so don't get confused if it suddenly starts missing details. It can't even keep 256 tokens cleanly, but still :P

## Limitations

- It's base only, so it can't chat
- Definitely not AGI
- BLiMP regressed
- English only (I know the org is called İvme, but I'm still not sure how I'm supposed to fit Turkish into an SLM)

## What's Next

To be honest? Definitely Conversate-v4 after this. Conversate-v3 and v3.1 have too many architectural inefficiencies and problems that gave me way too many headaches. I'll start from scratch.

## Citation

```bibtex
@misc{ivme-conversate-v3.1-base,
  author       = {IvmeLabs},
  title        = {İvme-Conversate-v3.1-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-v3.1-Base}
}
```

---

*By Eren Ekşi*