Dodecahedron-67M

A 67M parameter language model from the Dodecahedron family, trained for singular learning theory (SLT) research.

Model Details

Parameter Value
Parameters 67M
Architecture LlamaForCausalLM
Hidden size 512
Layers 20
Attention heads 8 (4 KV heads, GQA 2:1)
Head dim 64
MLP intermediate 1408
Vocab size 16,384
Context length 2,048
Tied embeddings Yes

Training

  • Data: 40B tokens of FineWeb
  • Tokenizer: Custom 16k vocab BPE trained on FineWeb (same as Dodecahedron-32M)
  • Optimizer: AdamW (betas 0.9/0.95, weight decay 0.01, grad clip 1.0)
  • Learning rate: 3.5e-3 with cosine decay, 1000 warmup steps
  • Batch size: 524k tokens/step (64 micro Γ— 4 grad accum Γ— 2048 seq)
  • Precision: bfloat16
  • Hardware: 8Γ— H100

Checkpoints

53 checkpoints are available as branches, log-spaced from step 1 to step 9536 (~40B tokens):

from transformers import AutoModelForCausalLM

# Final checkpoint
model = AutoModelForCausalLM.from_pretrained("timaeus/dodecahedron-67m-v2")

# Intermediate checkpoint
model = AutoModelForCausalLM.from_pretrained("timaeus/dodecahedron-67m-v2", revision="step5108")

Available steps: 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 19, 22, 25, 30, 35, 40, 47, 55, 65, 75, 88, 103, 121, 141, 165, 193, 225, 263, 308, 359, 420, 491, 574, 671, 785, 917, 1072, 1253, 1465, 1713, 2002, 2340, 2735, 3198, 3738, 4369, 5108, 5970, 6979, 8158, 9536

Purpose

The Dodecahedron family is designed as small, well-characterized reference models for developmental interpretability and SLT research. Having densely-checkpointed models enables studying learning dynamics, phase transitions, and the geometry of the loss landscape throughout training.

Related

Downloads last month
83
Safetensors
Model size
67.4M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train timaeus/dodecahedron-67m