Two arms pretrained on the same data mix (ko_web .47, ko_pdf .10, en_edu .30, en_edu_hi .07, math .06), 1.0B tokens each,

arm preset tok/s (5090) val ko val en
arm1_moe10 max-1B-moe10 (1124M total, MoE top-2, capacity 1.5) 73.2K 3.1598 2.9595
arm2_dense05 max-0.5B-dense (502M) 45.9K 3.1445 2.9298

Same tokens: dense0.5B slightly better (ko -0.015, en -0.030). Same wall time: moe10 sees ~1.6x tokens.

Files are final weights only, bf16 safetensors (no optimizer state; the private storage quota did not fit full checkpoints). Load with the repo's src/model.py: MaxLM(Config(**cfg)) then load_state_dict(load_file(...)) (tied embeddings).

file what
<arm>.safetensors final model weights (after cooldown), bf16; cfg also in safetensors metadata
<arm>.cfg.json model Config
<arm>/log.jsonl training + val log
tokenizer/ 32K byte-level BPE
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support