| --- |
| language: |
| - hu |
| license: apache-2.0 |
| library_name: mlx |
| base_model: utter-project/EuroLLM-9B |
| pipeline_tag: text-generation |
| tags: |
| - hungarian |
| - emese |
| - eurollm |
| - instruct |
| - chatml |
| - mlx |
| - apple-silicon |
| - quantized |
| --- |
| |
| # Emese-Patak (9.15B) — MLX q8 |
|
|
| **MLX q8** — the **native training/serving precision** for Patak (SFT and DPO were both trained directly |
| on top of a q8-quantized base, not fused-then-quantized after the fact). See the `patak/` repo's README |
| for full architecture, CPT/SFT/DPO training details, and benchmarks — this file covers only the |
| q8-specific notes. |
|
|
| | | | |
| |---|---| |
| | **Quantization** | q8, group size 64 | |
| | **Size on disk** | ~9.1 GB (vs. ~17 GB bf16) | |
| | **Quality** | This *is* the model's native precision — the `patak/` bf16 repo is dequantized *from* this, not the other way around. Own benchmark testing found bf16 and q8 score within 1 point of each other (218 vs. 217/250). | |
| | **Max context length** | 32,768 tokens (EuroLLM-9B's native context) | |
|
|
| ## Usage |
|
|
| ```python |
| from mlx_lm import load, generate |
| from mlx_lm.sample_utils import make_sampler |
| model, tok = load("emese-tech/patak-mlx") |
| p = tok.apply_chat_template([{"role": "user", "content": "Mi Magyarország fővárosa?"}], |
| tokenize=False, add_generation_prompt=True) |
| print(generate(model, tok, prompt=p, max_tokens=256, sampler=make_sampler(temp=0.2))) |
| ``` |
|
|
| **Decode:** temperature `0.2`, no repetition penalty, eos `{2, 4}`, ChatML template. |
|
|
| ⚠️ **This repo is `mlx_lm`-only** — MLX's q8 quantization packs weights into `uint32` + per-group |
| `scales`/`biases` tensors with a `quantization` block in `config.json` that plain `transformers` does not |
| understand. Use the `patak/` (bf16) repo for `transformers`/vLLM/TGI. |
| |
| ## Training |
| |
| This is the primary artifact of the SFT+DPO training chain — see `patak/README.md` for the full CPT |
| (5.1M tokens/5,000 iters), SFT (`instruct_v18b`, 1 epoch, rank16/scale32/lr1.5e-5), and DPO (36 alfa |
| pairs, 120 iters, rank16/scale32/lr5e-6) recipe, plus the 218/250 Ultimate · 302/376 BlindSpot benchmark |
| results. |
| |
| ## Benchmarks |
| |
| **This exact q8 artifact scored 413/500 (83%) on emese-bench v1** (the consolidated 500-pt |
| Ultimate+BlindSpot benchmark) — by far the family's strongest result, near-perfect on longform, |
| reading, code, safety, honesty, and English. See `emese-bench/results/patak-mlx.md` for the full |
| category-by-category transcript and `emese-bench/README.md` for the benchmark's design. |
|
|
