README.md

---
license: apache-2.0
language:
- en
library_name: transformers
tags:
- ivme
- language-model
- transformer
- rope
- swiglu
- muon
- from-scratch
- tiny
- small
- decoder-only
- custom_code
datasets:
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/smollm-corpus
- HuggingFaceFW/finewiki
- HuggingFaceTB/finemath
- SimpleStories/SimpleStories
pipeline_tag: text-generation
---

# İvme-Conversate-v3-Base

![Conversate-v3 Logo](https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/vmQhCHaAHm5jHHXB2IqZX.png)

**İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight. This is the third release: a 24.79M parameter decoder-only base model trained from scratch, with a deeper/narrower architecture, two new attention-level changes, a digit-atomic tokenizer, and a revised data mix.

v2 fixed v1's biggest problem (coherence) mostly through data — more tokens, better weighted toward staying on topic. v3 keeps that lesson but changes the architecture itself for the first time since v1: more layers, a narrower hidden dimension, and two specific attention modifications (QK-Norm and XSA) chosen because they target coherence directly rather than just adding parameters. The tokenizer was also rebuilt to keep every digit as its own token, aimed squarely at v2's other well-known weak spot: arithmetic.

---

## Model Details

| Parameter | Value |
|---|---|
| Architecture | Decoder-only transformer, dense (no loops, no exotic recurrence) |
| Parameters | 24,787,520 |
| Layers | 16 |
| Hidden dim | 320 |
| FFN dim | SwiGLU (~853) |
| Attention heads | 8, full attention (no GQA), head_dim 40 |
| Attention variant | QK-Norm + XSA (Exclusive Self Attention, arXiv:2603.09078) |
| Context length | 1024 tokens |
| Vocab size | 16,000 (custom digit-atomic BPE) |
| Positional encoding | RoPE (θ=10,000) |
| Normalization | RMSNorm (pre-norm) |
| Embeddings | Tied input/output |
| Biases | None |

v2 was 10 layers × 384 hidden dim (depth:width ≈ 0.026). v3 deliberately goes deeper and narrower — 16 layers × 320 hidden dim (depth:width ≈ 0.050) — while landing in almost exactly the same parameter budget. Depth tends to help multi-step reasoning and cross-sentence coherence more than raw width does; width tends to help how much can be memorized per token. This is a real rebalancing of the budget, not a token gesture: the embedding table's share of total parameters drops from v2's 25.8% to v3's 20.7%, with the difference going into deeper transformer layers.

**QK-Norm**: per-head RMSNorm applied to queries and keys before the RoPE rotation and the dot product, for training stability at higher depth and a higher effective learning rate.

**XSA (Exclusive Self Attention)**: after standard scaled-dot-product attention produces output yᵢ for token i, subtract the projection of yᵢ onto that token's own value vector vᵢ:

```
zᵢ = yᵢ − (yᵢ · vᵢ) / (‖vᵢ‖² + ε) · vᵢ
```

The idea, from the source paper: standard attention lets a token attend heavily to itself, which can let a model get lazy and just echo its own representation forward instead of pulling in genuinely new context from other tokens. XSA removes that redundant self-component. Zero extra parameters, negligible extra compute.

---

## Benchmarks

Benchmarks were run with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) via a `transformers`-compatible custom model wrapper, plus [ArithMark 3.0](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0) via its official evaluation script.

| Benchmark | v2 | v3 |
|---|---|---|
| WikiText-2 (byte perplexity) ↓ | 2.2250 | **2.1362** |
| BLiMP (macro-average) ↑ | 75.09% | **78.49%** |
| ARC-Easy (acc) ↑ | 43.56% | 43.60% |
| ARC-Easy (acc_norm) ↑ | 39.98% | 39.02% |
| ARC-Challenge (acc_norm) ↑ | — | 24.06% |
| HellaSwag (acc_norm) ↑ | — | 28.82% |
| PIQA (acc_norm) ↑ | — | 57.78% |
| ArithMark-3 (acc_norm) ↑ | — | 34.70% |

BLiMP moved up a real 3.4 points — the clearest confirmation that the depth/width rebalancing and the two attention changes did what they were meant to do for grammaticality and coherence. WikiText-2 perplexity also improved.

ARC-Easy is flat to slightly down (acc_norm: 39.98% → 39.02%), not up. Worth being straightforward about this rather than spinning it: ARC-Easy tests science-fact recall, which leans more on raw embedding/knowledge capacity than on the kind of cross-sentence reasoning BLiMP measures. v3 deliberately shrank the embedding table's share of the parameter budget (25.8% → 20.7%) to buy depth, and also cut Cosmopedia's weight hard (27% → 5.6%, it was stale synthetic data) in favor of more naturalistic web text. Both are plausible, specific mechanisms for a small knowledge-recall cost alongside the coherence gain — this looks like a real tradeoff from the choices made, not noise.

### BLiMP paradigm breakdown

Strong on core agreement paradigms, several near-ceiling:

| Paradigm | Accuracy |
|---|---|
| blimp_principle_A_case_1 | 100.00% |
| blimp_anaphor_number_agreement | 99.30% |
| blimp_determiner_noun_agreement_1 | 98.40% |
| blimp_sentential_negation_npi_licensor_present | 97.60% |
| blimp_anaphor_gender_agreement | 95.70% |

Weaker on long-distance dependencies and island constraints, same pattern as v2 and typical for small models:

| Paradigm | Accuracy |
|---|---|
| blimp_wh_vs_that_with_gap_long_distance | 37.90% |
| blimp_left_branch_island_echo_question | 26.20% |
| blimp_sentential_subject_island | 41.50% |
| blimp_coordinate_structure_constraint_complex_left_branch | 42.10% |
| blimp_principle_A_reconstruction | 35.40% |

---

## Does the arithmetic tokenizer fix actually work?

Short answer: it fixes the representation, not the reasoning. ArithMark-3 lands at 34.70% (acc_norm), well above the benchmark's 25% random-choice floor and ahead of most other small models in its published leaderboard cohort — a real, positive signal that digit-atomic tokenization helped. But free-form arithmetic generation is still weak: asked directly (`1 + 1 =`), the model reliably produces plausible-*looking* numbers that are frequently wrong (`1 + 1 = 20`).

This split result makes sense once you separate what digit-atomic tokenization actually fixes from what it doesn't. Before v3, multi-digit numbers collapsed into single opaque BPE tokens, so the model never even saw place-value structure — "127" was one indivisible symbol, not 1 hundred + 2 tens + 7 ones. Splitting every digit into its own token fixes that representation problem, and ArithMark's log-likelihood scoring (pick the more probable of four candidate completions) rewards exactly that: the model can lean on plausible-looking digit sequences without needing to actually execute the carry algorithm. Free-form generation has no such safety net — it has to get the arithmetic right, not just recognize which answer looks more likely, and that remains genuinely hard for a model this size. Digit-atomic tokenization was necessary for arithmetic to be learnable at all; it was never going to be sufficient on its own.

---

## Sample output

Temperature 0.8, top_k 40. This is a base model, not instruction-tuned — it continues text naturally rather than answering like a chat assistant.

**Prompt:** `"To bake a great cake you should always"`

> To bake a great cake you should always make sure that there are plenty of available ingredients. Some ingredients can be used in the pasting and baking process but you need to make sure that you use these at all times to create the perfect cake. If you don't have the ingredients, you may want to use more of the ingredients from another recipe. Also, it's essential to keep your ingredients for a long time so that you can get the most out of the recipe.

Coherent, on-topic, grammatically clean across multiple sentences — the thing v2 was built to fix, and v3 pushes further. Some repetition is visible at this temperature; that's a decoding-time property of small models' flatter output distributions, not primarily a training-quality signal. Raising temperature slightly (0.9–1.0) or lowering top_k typically reduces it.

---

## Training

### Data Mix (~15B tokens)

v2 was already overtrained well past Chinchilla-optimal on purpose, since the model is small and cheap to run regardless. v3 continues that, with a revised mix: Cosmopedia cut hard (it's synthetic data from an older, weaker generator model and was being over-weighted), DCLM-baseline raised substantially (better conversational/coherent text), and a new dense-knowledge source (FineWiki) added.

| Source | Share |
|---|---|
| HuggingFaceFW/fineweb-edu | 46.67% |
| mlfoundations/dclm-baseline-1.0 | 27.78% |
| HuggingFaceFW/finewiki (en) | 8.89% |
| HuggingFaceTB/finemath (finemath-3plus) | 7.78% |
| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 5.56% |
| SimpleStories/SimpleStories | 3.32% |

### Tokenizer

Custom byte-level BPE, vocab size 16,000, trained from scratch on a 2M-document sample of the pretraining mix — same as v2, but with one deliberate constraint added: the pre-tokenizer splits every digit 0–9 into its own token before BPE merging ever runs, so no digit sequence can be merged into a multi-digit chunk. `'48291'` tokenizes as five separate single-digit tokens, not one opaque symbol. This is the fix aimed at v2's flagged weak spot ("the model has learned syntax and grammar very well, but falls short on knowledge and easily hallucinates" — arithmetic specifically was part of that gap).

### Hyperparameters

| Setting | Value |
|---|---|
| Optimizer | Muon (body weights) + AdamW (embeddings, norms) |
| Muon lr | 0.02 |
| AdamW lr | 3e-4 |
| LR schedule | Warmup-Stable-Decay (WSD) |
| Warmup steps | 1,000 |
| Decay fraction | 10% of training |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | bfloat16 |
| Attention | PyTorch `scaled_dot_product_attention` |

### Hardware

Trained on a single NVIDIA RTX PRO 6000 Blackwell (96GB).

---

## Inference

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "IvmeLabs/Ivme-Conversate-v3-Base", trust_remote_code=True, dtype=torch.float32,
)
tokenizer = AutoTokenizer.from_pretrained("IvmeLabs/Ivme-Conversate-v3-Base", trust_remote_code=True)
model.eval()

inputs = tokenizer("Once upon a time, there was a", return_tensors="pt")
out = model.generate(
    **inputs, max_new_tokens=200, do_sample=True,
    temperature=0.8, top_k=40, pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

`trust_remote_code=True` is required (custom architecture: RoPE + QK-Norm + XSA + SwiGLU + RMSNorm dense decoder).

---

## Limitations

- Base model only, not instruction-tuned, will not follow instructions or answer questions
- English only
- 1024 token context window
- Weaker on long-distance syntactic dependencies than on local agreement, see BLiMP breakdown above
- **Arithmetic**: digit-atomic tokenization measurably helps recognition-style scoring (ArithMark-3, log-likelihood over candidate answers), but free-form generated arithmetic is still frequently wrong. Don't rely on this model for actual computation.
- ARC-Easy knowledge recall is flat versus v2, likely a real tradeoff from the deeper/narrower architecture and the reduced Cosmopedia weight, not a bug
- Repetition at higher temperatures without a repetition penalty

---

## What's Next

A math-focused data pass to give the digit-atomic tokenizer's representational fix an actual chance at teaching the underlying algorithm, not just recognition. Continued exploration of the architecture changes introduced here (QK-Norm, XSA) at larger scale.

You can check our other models on our [organization card](https://huggingface.co/IvmeLabs)!

---

## Citation

```bibtex
@misc{ivme-conversate-v3-base,
  author       = {IvmeLabs},
  title        = {İvme-Conversate-v3-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-v3-Base}
}
```

---

*Built by IvmeLabs. Small models, deliberate choices.*