max-1B-moe10 · r1 (early experiment)
Status: early research checkpoint, not a usable model yet. First from-scratch run of the max 1B mixture-of-experts
shape, trained for only ~0.9B tokens. On held-out loss it is still behind the smaller
max-0.1b-v1. Continued pretraining runs resume from
stable_pre_decay_1b.pt; later rounds will be published as separate repos.
| parameters | 1.12B total, 149M active per token |
| shape | 16 layers, hidden 1024, 16 heads / 4 KV heads (head_dim 128), 64 experts top-2, expert FFN 320, vocab 32000, context 1024 |
| tokens | 0.90B (WSD schedule, Muon optimizer, batch 0.5M tokens) |
| data mix (stable) | Korean web 40%, Korean PDF 20%, English edu 25%, English edu-hi 8%, math 7% |
Final held-out loss / bits-per-byte (step 1795):
| ko_val | en_val | kopdf_val | math_val | gov_val |
|---|---|---|---|---|
| 3.195 / 0.944 | 2.972 / 1.063 | 2.337 / 0.837 | 2.159 / 1.000 | 2.296 / 0.880 |
Files
| file | what |
|---|---|
final.pt |
end of cooldown (step 1795, 0.90B tokens): model + optimizer state |
stable_pre_decay_1b.pt |
step 1455, 0.72B tokens, saved at the start of the LR decay: resume point for continued pretraining |
config.json |
model config |
tokenizer.json |
32k BPE tokenizer |
log.jsonl |
training log (loss, throughput, evals) |
These are raw PyTorch training checkpoints (torch.load(..., weights_only=False), weights under the model key), not a
transformers model. Verify downloads with SHA256SUMS.
max 1B MoE(총 1.12B, 활성 149M) 첫 사전학습 실험 체크포인트입니다. 약 0.9B 토큰만 학습해서 아직 실사용 수준이 아니며,
검증 손실은 max-0.1b-v1보다 낮지 않습니다. 추가학습은 stable_pre_decay_1b.pt에서 이어 가고, 이후 결과는 별도 저장소로 공개합니다.
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support