Two arms pretrained on the same data mix (ko_web .47, ko_pdf .10, en_edu .30, en_edu_hi .07, math .06), 1.0B tokens each,
| arm | preset | tok/s (5090) | val ko | val en |
|---|---|---|---|---|
| arm1_moe10 | max-1B-moe10 (1124M total, MoE top-2, capacity 1.5) | 73.2K | 3.1598 | 2.9595 |
| arm2_dense05 | max-0.5B-dense (502M) | 45.9K | 3.1445 | 2.9298 |
Same tokens: dense0.5B slightly better (ko -0.015, en -0.030). Same wall time: moe10 sees ~1.6x tokens.
Files are final weights only, bf16 safetensors (no optimizer state; the private storage quota did not fit full checkpoints).
Load with the repo's src/model.py: MaxLM(Config(**cfg)) then load_state_dict(load_file(...)) (tied embeddings).
| file | what |
|---|---|
<arm>.safetensors |
final model weights (after cooldown), bf16; cfg also in safetensors metadata |
<arm>.cfg.json |
model Config |
<arm>/log.jsonl |
training + val log |
tokenizer/ |
32K byte-level BPE |
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support