metadata
library_name: transformers
pipeline_tag: text-generation
tags:
- llm
- from-scratch
- mathematics
- code
- nature-of-code
license: mit
language:
- en
- code
mathcode-llm
A decoder-only transformer language model built from scratch -- no
pretrained weights, no transformers model classes. Every layer (RMSNorm,
RoPE, grouped-query attention, SwiGLU, KV-cached decoding) is hand-written
PyTorch in model/transformer.py, and the BPE tokenizer was trained from
scratch on our own corpus.
Training data
| Source | Share |
|---|---|
open-web-math (arXiv papers, Math StackExchange) |
50.2% |
codeparrot-clean (Python from GitHub) |
37.7% |
code-search-net (Python / JS / Go / Java functions) |
5.8% |
verifiable-coding-problems-python |
4.8% |
| The Nature of Code (Daniel Shiffman, full 13 chapters) | 1.4% |
600 M training tokens, interleaved at 4096-token granularity so the domain mixture stays stable throughout training.
Architecture
| Parameters | ~110 M (85 M non-embedding) |
| Layers / heads / width | 12 / 12 / 768 |
| Context length | 1024 tokens |
| Vocabulary | 32,768 (byte-level BPE, trained from scratch) |
| Normalisation | RMSNorm (pre-norm) |
| Position | RoPE |
| FFN | SwiGLU |
| Attention | multi-head, KV cache for fast sampling |
| Embeddings | tied |
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("teddyoas/mathcode-llm")
model = AutoModelForCausalLM.from_pretrained("teddyoas/mathcode-llm")
The tokenizer is a PreTrainedTokenizerFast built with the tokenizers
library, so AutoTokenizer loads it directly.
Resuming training
checkpoint/latest.pt holds the model, the optimiser state, the AMP scaler,
the step counter and the token counter:
python train.py --config configs/mathcode.yaml \
--resume checkpoint/latest.pt
Status
Training checkpoint: step 100 / 4577 (13,107,200 tokens seen, val loss inf)