| --- |
| language: |
| - hu |
| license: apache-2.0 |
| library_name: mlx |
| base_model: utter-project/EuroLLM-22B |
| pipeline_tag: text-generation |
| tags: |
| - hungarian |
| - emese |
| - eurollm |
| - instruct |
| - chatml |
| - mlx |
| - apple-silicon |
| - quantized |
| --- |
| |
| # Emese-Folyó (22B) — MLX q8 |
|
|
| **MLX q8** — the **native training/serving precision** for Folyó (CPT, SFT, and DPO were all trained on |
| top of a q8-quantized base). See the `folyo/` repo's README for full architecture, CPT/SFT/DPO training |
| details, and benchmarks — this file covers only the q8-specific notes. |
|
|
| | | | |
| |---|---| |
| | **Quantization** | q8, group size 64 | |
| | **Size on disk** | ~22 GB (vs. ~42 GB bf16) | |
| | **Quality** | This *is* the model's native precision — the `folyo/` bf16 repo is dequantized *from* this, not the other way around. | |
| | **Max context length** | 32,768 tokens (EuroLLM-22B's native context) | |
|
|
| ## Usage |
|
|
| ```python |
| from mlx_lm import load, generate |
| from mlx_lm.sample_utils import make_sampler |
| model, tok = load("emese-tech/folyo-mlx") |
| p = tok.apply_chat_template([{"role": "user", "content": "Mi Magyarország fővárosa?"}], |
| tokenize=False, add_generation_prompt=True) |
| print(generate(model, tok, prompt=p, max_tokens=256, sampler=make_sampler(temp=0.2))) |
| ``` |
|
|
| **Decode:** temperature `0.2`, no repetition penalty, eos `{2, 4}`, ChatML template. |
|
|
| ⚠️ **This repo is `mlx_lm`-only** — MLX's q8 quantization packs weights into `uint32` + per-group |
| `scales`/`biases` tensors with a `quantization` block in `config.json` that plain `transformers` does not |
| understand. Use the `folyo/` (bf16) repo for `transformers`/vLLM/TGI. |
| |
| ## Training |
| |
| This is the primary artifact of the CPT+SFT+DPO training chain — see `folyo/README.md` for the full |
| q8-native CPT (~6M tokens/6,000 iters, rank64), SFT (`instruct_v18b`, 1 epoch, rank16/scale16/lr5e-6, |
| iteration 3,600 selected), and DPO (36 alfa pairs, 120 iters, rank16/scale16/lr2e-6) recipe, plus the |
| 211/250 Ultimate · 310/376 BlindSpot benchmark results. |
| |
| ## Benchmarks |
| |
| **This exact q8 artifact scored 410/500 (82%) on emese-bench v1** (200 questions + 10 chat sessions, |
| consolidated Ultimate+BlindSpot) — a close second to Patak (413/500) and far ahead of Csermely |
| (211/500). Strongest on safety, reading, translation, and code; weakest on multi-step math and logic |
| puzzles. Notably the best of the three models on in-context memory recall and roleplay (both 10/10), |
| but it failed to stay in English during the English-chat session, answering in Hungarian throughout. |
| See `emese-bench/results/folyo-mlx.md` for the full transcript and category breakdown. |
|
|
