mathcode-llm

A decoder-only transformer language model built from scratch -- no pretrained weights, no transformers model classes. Every layer (RMSNorm, RoPE, grouped-query attention, SwiGLU, KV-cached decoding) is hand-written PyTorch in model/transformer.py, and the BPE tokenizer was trained from scratch on our own corpus.

Training data

Source	Share
open-web-math (arXiv papers, Math StackExchange)	50.2%
codeparrot-clean (Python from GitHub)	37.7%
code-search-net (Python / JS / Go / Java functions)	5.8%
verifiable-coding-problems-python	4.8%
The Nature of Code (Daniel Shiffman, full 13 chapters)	1.4%

600 M training tokens, interleaved at 4096-token granularity so the domain mixture stays stable throughout training.

Architecture

Parameters	~110 M (85 M non-embedding)
Layers / heads / width	12 / 12 / 768
Context length	1024 tokens
Vocabulary	32,768 (byte-level BPE, trained from scratch)
Normalisation	RMSNorm (pre-norm)
Position	RoPE
FFN	SwiGLU
Attention	multi-head, KV cache for fast sampling
Embeddings	tied

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("teddyoas/mathcode-llm")
model = AutoModelForCausalLM.from_pretrained("teddyoas/mathcode-llm")

The tokenizer is a PreTrainedTokenizerFast built with the tokenizers library, so AutoTokenizer loads it directly.

Resuming training

checkpoint/latest.pt holds the model, the optimiser state, the AMP scaler, the step counter and the token counter:

python train.py --config configs/mathcode.yaml \
    --resume checkpoint/latest.pt

Status

Training checkpoint: step 100 / 4577 (13,107,200 tokens seen, val loss inf)