File size: 2,598 Bytes
e26e804 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 | ---
language:
- hu
license: apache-2.0
library_name: mlx
base_model: utter-project/EuroLLM-22B
pipeline_tag: text-generation
tags:
- hungarian
- emese
- eurollm
- instruct
- chatml
- mlx
- apple-silicon
- quantized
---
# Emese-Folyó (22B) — MLX q8
**MLX q8** — the **native training/serving precision** for Folyó (CPT, SFT, and DPO were all trained on
top of a q8-quantized base). See the `folyo/` repo's README for full architecture, CPT/SFT/DPO training
details, and benchmarks — this file covers only the q8-specific notes.
| | |
|---|---|
| **Quantization** | q8, group size 64 |
| **Size on disk** | ~22 GB (vs. ~42 GB bf16) |
| **Quality** | This *is* the model's native precision — the `folyo/` bf16 repo is dequantized *from* this, not the other way around. |
| **Max context length** | 32,768 tokens (EuroLLM-22B's native context) |
## Usage
```python
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler
model, tok = load("emese-tech/folyo-mlx")
p = tok.apply_chat_template([{"role": "user", "content": "Mi Magyarország fővárosa?"}],
tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt=p, max_tokens=256, sampler=make_sampler(temp=0.2)))
```
**Decode:** temperature `0.2`, no repetition penalty, eos `{2, 4}`, ChatML template.
⚠️ **This repo is `mlx_lm`-only** — MLX's q8 quantization packs weights into `uint32` + per-group
`scales`/`biases` tensors with a `quantization` block in `config.json` that plain `transformers` does not
understand. Use the `folyo/` (bf16) repo for `transformers`/vLLM/TGI.
## Training
This is the primary artifact of the CPT+SFT+DPO training chain — see `folyo/README.md` for the full
q8-native CPT (~6M tokens/6,000 iters, rank64), SFT (`instruct_v18b`, 1 epoch, rank16/scale16/lr5e-6,
iteration 3,600 selected), and DPO (36 alfa pairs, 120 iters, rank16/scale16/lr2e-6) recipe, plus the
211/250 Ultimate · 310/376 BlindSpot benchmark results.
## Benchmarks
**This exact q8 artifact scored 410/500 (82%) on emese-bench v1** (200 questions + 10 chat sessions,
consolidated Ultimate+BlindSpot) — a close second to Patak (413/500) and far ahead of Csermely
(211/500). Strongest on safety, reading, translation, and code; weakest on multi-step math and logic
puzzles. Notably the best of the three models on in-context memory recall and roleplay (both 10/10),
but it failed to stay in English during the English-chat session, answering in Hungarian throughout.
See `emese-bench/results/folyo-mlx.md` for the full transcript and category breakdown.
|