{
  "size": "15M",
  "params": 15531840,
  "config": {
    "vocab_size": 8192,
    "hidden_size": 320,
    "intermediate_size": 864,
    "num_hidden_layers": 12,
    "num_attention_heads": 10,
    "num_key_value_heads": 2,
    "max_position_embeddings": 2048,
    "rope_theta": 10000.0,
    "rms_norm_eps": 1e-05,
    "initializer_range": 0.02
  },
  "tokens": 2999975936,
  "steps": 11444,
  "batch_tokens": 262144,
  "seq_len": 2048,
  "opt": "muon",
  "lr": 0.007,
  "wd": 0.1,
  "warmup_frac": 0.01,
  "decay_frac": 0.2,
  "gpu": "NVIDIA L4",
  "train_minutes": 372.6,
  "final_val": {
    "dclm": 3.1669,
    "finemath": 2.2753,
    "fineweb": 2.9144,
    "stack": 1.4606,
    "avg": 2.4543
  },
  "samples": [
    "The meaning of life is that the person who lives in a certain place will be able to live with his or her own needs. The person who lives in an area where he/she has no need for food and drinks, can live without food.\nIn addition",
    "Once upon a time, the people of the country were in their own hands. The people of the country were not able to get any help from them. They had no idea what they wanted and could not find anything else.\nThe people of the country were not able",
    "def is_prime(n): print(prime)\nprint(\"Prime number: \", n)\n\n# Prime numbers are not prime\nprint(prime)\nprint(prime)\nprint(prime)\n\n# Prime numbers are not prime\nprint(",
    "The derivative of x^2 is 1/x^3. The derivative of y^2 is 0, which means that the derivative of y is 0.\n\n## What is the derivative of x?\n\nThe derivative of a function is defined as"
  ],
  "parity_max_diff": 2.193450927734375e-05
}