Text Generation
Transformers
Safetensors
English
ivme_conversate
ivme
small-language-model
language-model
transformer
rope
swiglu
muon
xsa
qk-norm
continued-pretraining
dense-data
educational
custom_code
Instructions to use IvmeLabs/Ivme-Conversate-v3.1-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use IvmeLabs/Ivme-Conversate-v3.1-Base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="IvmeLabs/Ivme-Conversate-v3.1-Base", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("IvmeLabs/Ivme-Conversate-v3.1-Base", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use IvmeLabs/Ivme-Conversate-v3.1-Base with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "IvmeLabs/Ivme-Conversate-v3.1-Base" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "IvmeLabs/Ivme-Conversate-v3.1-Base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/IvmeLabs/Ivme-Conversate-v3.1-Base
- SGLang
How to use IvmeLabs/Ivme-Conversate-v3.1-Base with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "IvmeLabs/Ivme-Conversate-v3.1-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "IvmeLabs/Ivme-Conversate-v3.1-Base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "IvmeLabs/Ivme-Conversate-v3.1-Base" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "IvmeLabs/Ivme-Conversate-v3.1-Base", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use IvmeLabs/Ivme-Conversate-v3.1-Base with Docker Model Runner:
docker model run hf.co/IvmeLabs/Ivme-Conversate-v3.1-Base
Rewrite model card: narrative framing, benchmarks as evidence
Browse files
README.md
CHANGED
|
@@ -5,6 +5,7 @@ language:
|
|
| 5 |
library_name: transformers
|
| 6 |
tags:
|
| 7 |
- ivme
|
|
|
|
| 8 |
- language-model
|
| 9 |
- transformer
|
| 10 |
- rope
|
|
@@ -13,6 +14,8 @@ tags:
|
|
| 13 |
- xsa
|
| 14 |
- qk-norm
|
| 15 |
- continued-pretraining
|
|
|
|
|
|
|
| 16 |
- custom_code
|
| 17 |
- text-generation
|
| 18 |
datasets:
|
|
@@ -29,9 +32,15 @@ pipeline_tag: text-generation
|
|
| 29 |
|
| 30 |

|
| 31 |
|
| 32 |
-
**İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight.
|
| 33 |
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
---
|
| 37 |
|
|
@@ -53,93 +62,84 @@ It gains **+1.89 points of Intelligence Index** over v3, and gives back 3.4 poin
|
|
| 53 |
| Embeddings | Tied input/output |
|
| 54 |
| Biases | None |
|
| 55 |
|
| 56 |
-
**
|
| 57 |
|
| 58 |
---
|
| 59 |
|
| 60 |
-
##
|
| 61 |
|
| 62 |
-
|
| 63 |
|
| 64 |
-
|
| 65 |
-
|---|---|---|
|
| 66 |
-
| ARC-Easy | 39.02 | **40.15** |
|
| 67 |
-
| ARC-Challenge | 24.32 | **25.00** |
|
| 68 |
-
| HellaSwag | 28.90 | **29.52** |
|
| 69 |
-
| PIQA | 57.67 | **59.25** |
|
| 70 |
-
| ArithMark-3 | 35.20 | **37.20** |
|
| 71 |
-
| **Intelligence Index** | **10.35** | **12.38** |
|
| 72 |
|
| 73 |
-
|
| 74 |
|
| 75 |
-
|
| 76 |
-
|---|---|---|
|
| 77 |
-
| WikiText-2 word perplexity ↓ | 57.90 | **55.35** |
|
| 78 |
-
| WikiText-2 byte perplexity ↓ | 2.1362 | **2.1182** |
|
| 79 |
-
| BLiMP (67-task macro) | 78.49 | **75.09** |
|
| 80 |
|
| 81 |
-
|
|
|
|
|
|
|
| 82 |
|
| 83 |
-
|
| 84 |
|
| 85 |
-
|
| 86 |
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
-
|
| 90 |
|
| 91 |
-
|
|
| 92 |
|---|---|---|
|
| 93 |
-
|
|
| 94 |
-
|
|
| 95 |
-
| v3.1 @ 75% of training | 11.98 | 76.56 |
|
| 96 |
-
| **v3.1** | **12.38** | **75.09** |
|
| 97 |
|
| 98 |
-
|
| 99 |
|
| 100 |
---
|
| 101 |
|
| 102 |
-
##
|
| 103 |
|
| 104 |
-
|
| 105 |
|
| 106 |
-
**
|
| 107 |
|
| 108 |
-
|
| 109 |
-
>
|
| 110 |
-
> ### History
|
| 111 |
-
> The capital of France was founded in 16...
|
| 112 |
|
| 113 |
-
|
| 114 |
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
>
|
| 119 |
-
> 1. Preheat your...
|
| 120 |
|
| 121 |
-
|
| 122 |
|
| 123 |
---
|
| 124 |
|
| 125 |
## Training
|
| 126 |
|
| 127 |
-
###
|
| 128 |
|
| 129 |
-
|
| 130 |
-
|---|---|
|
| 131 |
-
| HuggingFaceTB/finemath (finemath-3plus) | 32.5% |
|
| 132 |
-
| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% |
|
| 133 |
-
| HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% |
|
| 134 |
-
| HuggingFaceFW/finewiki (en) | 11.9% |
|
| 135 |
-
| open-web-math/open-web-math | 10.5% |
|
| 136 |
-
| SimpleStories/SimpleStories | 1.4% |
|
| 137 |
|
| 138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
|
| 140 |
-
|
| 141 |
|
| 142 |
-
###
|
| 143 |
|
| 144 |
| Setting | Value |
|
| 145 |
|---|---|
|
|
@@ -156,9 +156,19 @@ No benchmark text was trained on. Every document was scanned during corpus const
|
|
| 156 |
| Batch | 128 × 1024 tokens × 8 GPUs |
|
| 157 |
| Steps | 29,900 |
|
| 158 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
### Hardware
|
| 160 |
|
| 161 |
-
|
|
|
|
|
|
|
| 162 |
|
| 163 |
---
|
| 164 |
|
|
@@ -182,27 +192,29 @@ out = model.generate(
|
|
| 182 |
print(tokenizer.decode(out[0], skip_special_tokens=True))
|
| 183 |
```
|
| 184 |
|
| 185 |
-
`trust_remote_code=True` is required
|
| 186 |
|
| 187 |
-
The model has no KV-cache implementation
|
| 188 |
|
| 189 |
---
|
| 190 |
|
| 191 |
## Limitations
|
| 192 |
|
| 193 |
-
- Base model
|
| 194 |
-
- English only
|
| 195 |
-
- 1024
|
| 196 |
-
- **
|
| 197 |
-
- Arithmetic
|
| 198 |
-
- Repetition at
|
| 199 |
-
-
|
| 200 |
|
| 201 |
---
|
| 202 |
|
| 203 |
-
## What's
|
|
|
|
|
|
|
| 204 |
|
| 205 |
-
|
| 206 |
|
| 207 |
---
|
| 208 |
|
|
|
|
| 5 |
library_name: transformers
|
| 6 |
tags:
|
| 7 |
- ivme
|
| 8 |
+
- small-language-model
|
| 9 |
- language-model
|
| 10 |
- transformer
|
| 11 |
- rope
|
|
|
|
| 14 |
- xsa
|
| 15 |
- qk-norm
|
| 16 |
- continued-pretraining
|
| 17 |
+
- dense-data
|
| 18 |
+
- educational
|
| 19 |
- custom_code
|
| 20 |
- text-generation
|
| 21 |
datasets:
|
|
|
|
| 32 |
|
| 33 |

|
| 34 |
|
| 35 |
+
**İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight.
|
| 36 |
|
| 37 |
+
v3.1 is not a new architecture and not a bigger model. It is the same 25M-parameter v3 transformer, unchanged down to the tokenizer, put back in the oven for another 31.4B tokens — only this time we changed the food.
|
| 38 |
+
|
| 39 |
+
That is the whole idea, and it is worth stating plainly because it is easy to misread. When a model is this small, parameters are not what is holding it back. v3 had already seen 15B tokens, which is roughly thirty times past Chinchilla-optimal for its size. Adding more of the same web text had stopped buying much. What had *not* been tried was changing the composition: less open-web prose, far more mathematics, far more textbook-style explanation, far more encyclopedic reference.
|
| 40 |
+
|
| 41 |
+
So we held the model fixed and moved the data. The result is a model that knows more and reasons a little better, and — this part matters — is slightly worse at grammar. Both effects are real, both come from the same cause, and both are documented below rather than averaged into a single happy number.
|
| 42 |
+
|
| 43 |
+
If you want the one-sentence version: **v3.1 is v3 with more content and less syntax.** Which of those you want depends on what you are doing with it.
|
| 44 |
|
| 45 |
---
|
| 46 |
|
|
|
|
| 62 |
| Embeddings | Tied input/output |
|
| 63 |
| Biases | None |
|
| 64 |
|
| 65 |
+
**Why the parameter count differs from v3.** v3 reports 24,787,520; v3.1 reports 24,956,480. The FFN was zero-padded from 853 to 864 so that the GEMMs tile cleanly on tensor cores — 853 is prime, which is a surprisingly expensive thing for a hyperparameter to be. Because `silu(0) = 0` and the up-branch of the eleven added units is zero, they contribute exactly nothing to the output. The model is functionally identical to an 853-wide one; it is just 60–85% faster on FFN matmuls during training. No behaviour change, no retraining required.
|
| 66 |
|
| 67 |
---
|
| 68 |
|
| 69 |
+
## What it is like to use
|
| 70 |
|
| 71 |
+
It is a base model. It continues text; it does not follow instructions, and it will not answer a question so much as write the kind of document in which that question might appear.
|
| 72 |
|
| 73 |
+
That last property is the most noticeable thing about v3.1. Because a large share of its training data was textbook-shaped, it has picked up the *structure* of explanation — headings, numbered steps, definitions, "###" section markers — and it reaches for those structures readily. Ask it about cakes and you get a recipe outline. Ask it about a country and you get something that reads like an encyclopedia stub.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
+
It is also noticeably more willing to commit to a fact than v3 was. v3, given "The capital of France is", would often slide into a loop about "the capital of the French Empire". v3.1 says Paris. That is a small thing, and one example is not a benchmark, but it is the kind of difference the numbers below are trying to describe.
|
| 76 |
|
| 77 |
+
What it is *not* good at: long-range syntax, unusual sentence constructions, and anything requiring genuine multi-step reasoning. It is 25M parameters. It has read a great deal and understood a little.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
+
---
|
| 80 |
+
|
| 81 |
+
## Measurements
|
| 82 |
|
| 83 |
+
All numbers below were produced on the same machine, with the same harness, against the same checkpoints — not copied from a leaderboard and not measured at different times under different settings.
|
| 84 |
|
| 85 |
+
The harness is [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.13, 0-shot, `acc_norm`, plus [ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) run through its official scoring script.
|
| 86 |
|
| 87 |
+
| Benchmark | v3 | v3.1 | direction |
|
| 88 |
+
|---|---|---|---|
|
| 89 |
+
| ARC-Easy | 39.02 | 40.15 | up |
|
| 90 |
+
| ARC-Challenge | 24.32 | 25.00 | up |
|
| 91 |
+
| HellaSwag | 28.90 | 29.52 | up |
|
| 92 |
+
| PIQA | 57.67 | 59.25 | up |
|
| 93 |
+
| ArithMark-3 | 35.20 | 37.20 | up |
|
| 94 |
+
| BLiMP (67-task macro) | 78.49 | 75.09 | **down** |
|
| 95 |
|
| 96 |
+
Supporting language-modelling numbers:
|
| 97 |
|
| 98 |
+
| Benchmark | v3 | v3.1 |
|
| 99 |
|---|---|---|
|
| 100 |
+
| WikiText-2 word perplexity ↓ | 57.90 | 55.35 |
|
| 101 |
+
| WikiText-2 byte perplexity ↓ | 2.1362 | 2.1182 |
|
|
|
|
|
|
|
| 102 |
|
| 103 |
+
A note on how to read these. Our v3 baseline reproduces the published v3 card exactly (byte perplexity 2.1362) and the public leaderboard entry to within 0.14 index points, so the comparison is honest in both directions. Composite benchmark noise at this scale is roughly ±0.75 points on the index, which means the *direction* of every row above is solid while individual moves of half a point should not be over-read. v3.1 is not a different model class; it is a better-fed one.
|
| 104 |
|
| 105 |
---
|
| 106 |
|
| 107 |
+
## The BLiMP tradeoff, stated honestly
|
| 108 |
|
| 109 |
+
BLiMP fell 3.4 points — about 22 standard errors, so this is not noise.
|
| 110 |
|
| 111 |
+
The mechanism is not mysterious. BLiMP asks whether a sentence is syntactically acceptable, using minimal pairs that differ in one small grammatical way. Answering those well requires having read an enormous amount of *ordinary prose* — the kind of text where subject-verb agreement, determiner choice and anaphora appear naturally and constantly.
|
| 112 |
|
| 113 |
+
v3 was trained on a mix that was roughly 83% natural prose. v3.1's mix is 43% mathematics and 30% synthetic textbook. Math is not prose. Textbook explanation is not prose either — it is closer to a list of facts wearing a sentence costume. We optimized for density, and density is not the same thing as naturalness.
|
|
|
|
|
|
|
|
|
|
| 114 |
|
| 115 |
+
We could have buried this. A model card that reports five improvements and omits one regression is technically not lying, and it is extremely common. We would rather you knew, because it changes which checkpoint you should download:
|
| 116 |
|
| 117 |
+
- **v3.1** — best content and arithmetic, weakest syntax.
|
| 118 |
+
- **v3** — best syntax, least content.
|
| 119 |
+
- **v3.2** (a separate, prose-balanced sibling) — the middle: index 11.72, BLiMP 77.00.
|
|
|
|
|
|
|
| 120 |
|
| 121 |
+
Every checkpoint we trained sits on the same monotonic curve. Extra content costs syntax; there is no free lunch anywhere on it. Reporting only one end of that curve would be leaving out the interesting part.
|
| 122 |
|
| 123 |
---
|
| 124 |
|
| 125 |
## Training
|
| 126 |
|
| 127 |
+
### The data
|
| 128 |
|
| 129 |
+
31.4B tokens, one epoch, drawn from six sources.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
+
| Source | Share | Why |
|
| 132 |
+
|---|---|---|
|
| 133 |
+
| HuggingFaceTB/finemath (finemath-3plus) | 32.5% | dense mathematical reasoning |
|
| 134 |
+
| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 29.7% | synthetic textbook explanation |
|
| 135 |
+
| HuggingFaceFW/fineweb-edu (score ≥ 3.5) | 14.0% | general educational web |
|
| 136 |
+
| HuggingFaceFW/finewiki (en) | 11.9% | encyclopedic reference |
|
| 137 |
+
| open-web-math/open-web-math | 10.5% | mathematical web text |
|
| 138 |
+
| SimpleStories/SimpleStories | 1.4% | narrative and causal chains |
|
| 139 |
|
| 140 |
+
That mix was not chosen to look good in a table. It was chosen after watching v3 plateau: at this scale, the marginal value of another token of generic web text had fallen close to zero, while the marginal value of a token of dense mathematics or structured explanation had not. We were wrong about the size of the syntax cost, and we were right about the content gain.
|
| 141 |
|
| 142 |
+
### The recipe
|
| 143 |
|
| 144 |
| Setting | Value |
|
| 145 |
|---|---|
|
|
|
|
| 156 |
| Batch | 128 × 1024 tokens × 8 GPUs |
|
| 157 |
| Steps | 29,900 |
|
| 158 |
|
| 159 |
+
Muon on the body and AdamW on the embeddings and norm gains follows the recipe v3 was trained with. Using a single optimizer for both would have changed two variables at once, and this experiment was already changing one.
|
| 160 |
+
|
| 161 |
+
### Decontamination
|
| 162 |
+
|
| 163 |
+
No benchmark text was trained on, and that was enforced mechanically rather than promised.
|
| 164 |
+
|
| 165 |
+
During corpus construction every document was scanned with an Aho-Corasick automaton built from 249,054 patterns covering the train, validation *and* test splits of ARC-Easy, ARC-Challenge, HellaSwag, PIQA and ArithMark-3. Any document containing a match was dropped before tokenisation. This is the part of the pipeline that costs the most time and produces nothing visible, which is exactly why it is worth stating.
|
| 166 |
+
|
| 167 |
### Hardware
|
| 168 |
|
| 169 |
+
Eight RTX 5090s, 2.6 hours.
|
| 170 |
+
|
| 171 |
+
For context: the entire run — corpus filtering, tokenisation, training — fits inside a working day on consumer cards. That is the point of models this size. They are cheap enough to be *experiments* rather than commitments, and you can afford to run the version that gets one thing wrong so you know what it costs.
|
| 172 |
|
| 173 |
---
|
| 174 |
|
|
|
|
| 192 |
print(tokenizer.decode(out[0], skip_special_tokens=True))
|
| 193 |
```
|
| 194 |
|
| 195 |
+
`trust_remote_code=True` is required — this is a custom architecture (RoPE + QK-Norm + XSA + SwiGLU + RMSNorm dense decoder), and the config and modeling files ship with the weights.
|
| 196 |
|
| 197 |
+
Two implementation notes. The model has no KV-cache implementation, so `generate()` recomputes the sequence each step; that is correct but not tuned for long generations. And inputs longer than 1024 tokens are silently truncated to the last 1024, so chunk long documents yourself rather than trusting the model to notice.
|
| 198 |
|
| 199 |
---
|
| 200 |
|
| 201 |
## Limitations
|
| 202 |
|
| 203 |
+
- Base model, not instruction-tuned. It will not follow instructions.
|
| 204 |
+
- English only.
|
| 205 |
+
- 1024-token context; longer inputs are truncated to the most recent window.
|
| 206 |
+
- **Syntax regressed versus v3** (BLiMP −3.4). If your task is grammaticality-sensitive, v3 or v3.2 is the better pick.
|
| 207 |
+
- Arithmetic gains are recognition-style. ArithMark-3 measures log-likelihood over candidate continuations; free-form generated arithmetic is still frequently wrong, in the same way v3's was. Do not use this for actual computation.
|
| 208 |
+
- Repetition at high temperatures without a repetition penalty, as with the rest of the series.
|
| 209 |
+
- Diminishing returns: this is one epoch on a model that had already seen 15B tokens. The improvements are real but they are the second-order kind.
|
| 210 |
|
| 211 |
---
|
| 212 |
|
| 213 |
+
## What's next
|
| 214 |
+
|
| 215 |
+
The open question is whether the tradeoff curve can be bent rather than moved along. v3.2 suggests the middle is real (index 11.72, BLiMP 77.00) but it does not suggest the curve is convex — the points we have all lie close to a straight line, which is either a property of this data mix or a property of a 25M model, and we do not yet know which.
|
| 216 |
|
| 217 |
+
Beyond that, the honest answer is that 25M parameters will not reach the 50M–100M class by data alone. Distillation from a ~1.7B teacher is the only route we see to that frontier, and it is a different experiment with a different flavour of tradeoff.
|
| 218 |
|
| 219 |
---
|
| 220 |
|