max-1B-moe10 · r1 (early experiment)

Status: early research checkpoint, not a usable model yet. First from-scratch run of the max 1B mixture-of-experts shape, trained for only ~0.9B tokens. On held-out loss it is still behind the smaller max-0.1b-v1. Continued pretraining runs resume from stable_pre_decay_1b.pt; later rounds will be published as separate repos.

parameters 1.12B total, 149M active per token
shape 16 layers, hidden 1024, 16 heads / 4 KV heads (head_dim 128), 64 experts top-2, expert FFN 320, vocab 32000, context 1024
tokens 0.90B (WSD schedule, Muon optimizer, batch 0.5M tokens)
data mix (stable) Korean web 40%, Korean PDF 20%, English edu 25%, English edu-hi 8%, math 7%

Final held-out loss / bits-per-byte (step 1795):

ko_val en_val kopdf_val math_val gov_val
3.195 / 0.944 2.972 / 1.063 2.337 / 0.837 2.159 / 1.000 2.296 / 0.880

Files

file what
final.pt end of cooldown (step 1795, 0.90B tokens): model + optimizer state
stable_pre_decay_1b.pt step 1455, 0.72B tokens, saved at the start of the LR decay: resume point for continued pretraining
config.json model config
tokenizer.json 32k BPE tokenizer
log.jsonl training log (loss, throughput, evals)

These are raw PyTorch training checkpoints (torch.load(..., weights_only=False), weights under the model key), not a transformers model. Verify downloads with SHA256SUMS.


max 1B MoE(총 1.12B, 활성 149M) 첫 사전학습 실험 체크포인트입니다. 약 0.9B 토큰만 학습해서 아직 실사용 수준이 아니며, 검증 손실은 max-0.1b-v1보다 낮지 않습니다. 추가학습은 stable_pre_decay_1b.pt에서 이어 가고, 이후 결과는 별도 저장소로 공개합니다.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support