| --- |
| language: |
| - hu |
| license: mit |
| library_name: mlx |
| tags: |
| - hungarian |
| - emese |
| - from-scratch |
| - llama |
| - base-model |
| - mlx |
| - apple-silicon |
| pipeline_tag: text-generation |
| --- |
| |
| # Emese-Ér (517M) |
|
|
| **Ér** ("brook/rill" — the smallest flowing water in the family name scheme) is the edge/embedded/research |
| tier of the [Emese](https://emese.tech) Hungarian model family. Unlike the other three tiers, Ér is |
| **not** built on EuroLLM — it's a from-scratch Hungarian foundation model, pretrained on ~4.5B tokens of |
| Hungarian text with a custom 32k SentencePiece tokenizer. |
|
|
| | | | |
| |---|---| |
| | **Parameters** | 516.8M | |
| | **Base** | none — from-scratch pretraining | |
| | **Architecture** | LLaMA-style decoder (RoPE + YaRN, RMSNorm, SwiGLU, tied embeddings) | |
| | **Hidden / layers / heads** | 1280 / 24 / 20 | |
| | **FFN size** | 3456 | |
| | **Vocabulary** | 32,000 (custom Hungarian SentencePiece Unigram) | |
| | **Max context length** | **32,768 tokens** (trained at 1,024, YaRN-extended to 8,192 during CPT, RoPE scale 4.0 rated to 32,768) | |
| | **Precision** | bfloat16 | |
| | **License** | MIT | |
|
|
| ## Formats in this release |
|
|
| | Folder | Format | Notes | |
| |---|---|---| |
| | `er/` (this repo) | bf16, MLX-native safetensors | **not** a standard `transformers` repo — see below | |
| | `er-mlx/` | MLX q8 | `mlx_lm`-loadable, quantized | |
|
|
| ## ⚠️ Not `transformers`-compatible out of the box |
|
|
| Ér predates the EuroLLM/Llama-arch pipeline used for Csermely/Patak/Folyó. Its `config.json` uses a |
| custom schema (`d_model`, `n_layers`, `n_heads`, `d_ff` — not HF's `hidden_size`/`num_hidden_layers`/...) |
| and its weight tensor names are custom (`embed.weight`, `layers.N.attn.wq.weight`, ...), not the standard |
| `model.embed_tokens.weight` / `model.layers.N.self_attn.q_proj.weight` naming. **`AutoModelForCausalLM. |
| from_pretrained` will fail on this repo.** Use the project's own `model.py` (an MLX `Emese` class) or load |
| `er-mlx/` via `mlx_lm`, which does understand this checkpoint despite the non-standard config. |
| |
| ## Usage (MLX, via mlx_lm) |
| |
| ```python |
| from mlx_lm import load, generate |
| model, tok = load("er-mlx") # or "er" — both load via mlx_lm |
| print(generate(model, tok, prompt="A magyar nyelv", max_tokens=100)) |
| ``` |
| |
| ## Usage (project's own loader) |
| |
| ```bash |
| python generate.py --model models-release/er/ --prompt "A magyar nyelv" |
| ``` |
| |
| ## Training |
| |
| - **Pretraining (from scratch):** ~4.5B tokens of Hungarian text (the same corpus family documented in |
| `corpus/cpt/README.md`: Wikipedia + HPLT web text, quality-filtered). No separate "CPT" stage — this |
| *is* the base pretraining, there is no upstream foundation model underneath it. |
|
|
| ## Benchmarks |
|
|
| **Not applicable** — `emese-bench` (and its Ultimate/BlindSpot predecessors) are chat-instruction |
| benchmarks (ChatML turns, persona/safety/instruction-following categories); Ér has no chat template or |
| instruction-tuning, so scoring it on those benchmarks would not be a meaningful comparison against the |
| other three tiers. During development it was evaluated on HuCoLA (Hungarian grammatical-acceptability |
| classification) instead, not on chat-style benchmarks. |
| - **No SFT / no DPO.** Ér ships as a **base/research model only** — it has no chat template, no persona |
| training, no instruction-tuning. It is not conversational; treat it as a research/embedding artifact for |
| downstream fine-tuning experiments, not an assistant. |
|
|
| ## Limitations |
|
|
| - Base model only — expects raw text continuation, not chat-formatted prompts. |
| - Smallest tier by a wide margin (517M vs. 1.7B+ for the rest of the family) — limited world knowledge and |
| reasoning capacity even relative to Csermely. |
| - Requires custom loading code (`model.py`) for anything beyond `mlx_lm`. |
|
|
