L16-1.2B-400B (training in progress)

A 1.16B-parameter base language model, pretrained from scratch on 16 languages with no English. The run targets 400B tokens. Checkpoints are published as training goes on.

This is an intermediate checkpoint. main currently holds step 96,880 = 200.0B tokens, taken while the learning rate is still at its peak, before the final cooldown. Expect it to improve a lot, especially after the cooldown that starts at 360B tokens.

revision	step	tokens	held-out val loss*
step096880-200B (= main)	96,880	200.0B	2.232 (eval at step 96,500)
step072660-150B	72,660	150.0B	2.245 (eval at step 72,500)
step048440-100B	48,440	100.0B	2.264 (eval at step 48,500)
step024220-50B	24,220	50.0B	2.318 (eval at step 24,000)

*The val loss is per token of this model's own tokenizer, measured on a held-out 10M-token validation set (625k tokens per language) with document masking. It is not comparable with other tokenizers.

Languages

Russian, Spanish, Chinese (Simplified), Indonesian, Korean, Japanese, Turkish, Persian, German, Hungarian, Arabic, Vietnamese, Finnish, Greek, Thai and Hindi. The training mixture has 25B tokens per language and contains no English.

Model

• Architecture: Qwen3 decoder, the Marin delphi 1.4B configuration: 18 layers, hidden 1792, 14 heads (MHA, head_dim 128), MLP 7168, QK-norm, RoPE θ = 500k, context 4096, untied embeddings.
• Tokenizer: amphora/L16-V65536-chat-tokenizer.
  • 65,536-token SentencePiece BPE for the 16 languages.
  • Includes the Llama-3 chat special tokens.
  • The vocab size gives 1.16B params, against 1.4B with the Llama-3 vocab.

Training

Optimizer	MuonH (Muon, Polar Express orthogonalization, hyperball weight norm) at lr 3.5e-3 for weight matrices; AdamW at lr 8.4e-4 for embeddings, head and norms; no weight decay
Lookahead	EMA-Nesterov (β = 0.6) during the stable phase
Schedule	warmup-stable-decay: 500 warmup steps, constant LR, then a 1-sqrt cooldown over the last 10% (from 360B tokens)
Batch	504 × 4096 = 2.06M tokens per step, 193,752 steps (399.98B tokens)
Precision	bf16 autocast, FP8 (tensorwise) for MLP and attention projections, fp32 master weights
Data handling	document-boundary attention masking; a single shuffled pass, so no token is repeated
Hardware	6 × H200; 8 × H200 from step 92,000 (same global batch)

The peak LR comes from a sweep on the same 400B schedule, early-stopped at 9B tokens with a 1-sqrt decay to 10B. The sweep's best, 1e-2, was then lowered for the 400B horizon, following LR ∝ tokens^-0.3.

Training code: github.com/guijinSON/delphi-pretrain.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "amphora/L16-1.2B-400B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, revision="step096880-200B", dtype="bfloat16")
ids = tok("대한민국의 수도는", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=20)[0]))

The tokenizer adds no BOS token. During training each document was its tokens followed by </s>.

Limitations

• This is a base model: no instruction tuning and no safety tuning.
• It is not trained on English.
• Intermediate checkpoints come from the constant-LR phase and are not annealed.