README.md

---
language:
- hu
license: mit
library_name: mlx
tags:
- hungarian
- emese
- from-scratch
- llama
- base-model
- mlx
- apple-silicon
pipeline_tag: text-generation
---

# Emese-Ér (517M)

**Ér** ("brook/rill" — the smallest flowing water in the family name scheme) is the edge/embedded/research
tier of the [Emese](https://emese.tech) Hungarian model family. Unlike the other three tiers, Ér is
**not** built on EuroLLM — it's a from-scratch Hungarian foundation model, pretrained on ~4.5B tokens of
Hungarian text with a custom 32k SentencePiece tokenizer.

| | |
|---|---|
| **Parameters** | 516.8M |
| **Base** | none — from-scratch pretraining |
| **Architecture** | LLaMA-style decoder (RoPE + YaRN, RMSNorm, SwiGLU, tied embeddings) |
| **Hidden / layers / heads** | 1280 / 24 / 20 |
| **FFN size** | 3456 |
| **Vocabulary** | 32,000 (custom Hungarian SentencePiece Unigram) |
| **Max context length** | **32,768 tokens** (trained at 1,024, YaRN-extended to 8,192 during CPT, RoPE scale 4.0 rated to 32,768) |
| **Precision** | bfloat16 |
| **License** | MIT |

## Formats in this release

| Folder | Format | Notes |
|---|---|---|
| `er/` (this repo) | bf16, MLX-native safetensors | **not** a standard `transformers` repo — see below |
| `er-mlx/` | MLX q8 | `mlx_lm`-loadable, quantized |

## ⚠️ Not `transformers`-compatible out of the box

Ér predates the EuroLLM/Llama-arch pipeline used for Csermely/Patak/Folyó. Its `config.json` uses a
custom schema (`d_model`, `n_layers`, `n_heads`, `d_ff` — not HF's `hidden_size`/`num_hidden_layers`/...)
and its weight tensor names are custom (`embed.weight`, `layers.N.attn.wq.weight`, ...), not the standard
`model.embed_tokens.weight` / `model.layers.N.self_attn.q_proj.weight` naming. **`AutoModelForCausalLM.
from_pretrained` will fail on this repo.** Use the project's own `model.py` (an MLX `Emese` class) or load
`er-mlx/` via `mlx_lm`, which does understand this checkpoint despite the non-standard config.

## Usage (MLX, via mlx_lm)

```python
from mlx_lm import load, generate
model, tok = load("er-mlx")  # or "er" — both load via mlx_lm
print(generate(model, tok, prompt="A magyar nyelv", max_tokens=100))
```

## Usage (project's own loader)

```bash
python generate.py --model models-release/er/ --prompt "A magyar nyelv"
```

## Training

- **Pretraining (from scratch):** ~4.5B tokens of Hungarian text (the same corpus family documented in
  `corpus/cpt/README.md`: Wikipedia + HPLT web text, quality-filtered). No separate "CPT" stage — this
  *is* the base pretraining, there is no upstream foundation model underneath it.

## Benchmarks

**Not applicable** — `emese-bench` (and its Ultimate/BlindSpot predecessors) are chat-instruction
benchmarks (ChatML turns, persona/safety/instruction-following categories); Ér has no chat template or
instruction-tuning, so scoring it on those benchmarks would not be a meaningful comparison against the
other three tiers. During development it was evaluated on HuCoLA (Hungarian grammatical-acceptability
classification) instead, not on chat-style benchmarks.
- **No SFT / no DPO.** Ér ships as a **base/research model only** — it has no chat template, no persona
  training, no instruction-tuning. It is not conversational; treat it as a research/embedding artifact for
  downstream fine-tuning experiments, not an assistant.

## Limitations

- Base model only — expects raw text continuation, not chat-formatted prompts.
- Smallest tier by a wide margin (517M vs. 1.7B+ for the rest of the family) — limited world knowledge and
  reasoning capacity even relative to Csermely.
- Requires custom loading code (`model.py`) for anything beyond `mlx_lm`.