| --- |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - llm |
| - from-scratch |
| - mathematics |
| - code |
| - nature-of-code |
| license: mit |
| language: |
| - en |
| - code |
| --- |
| |
| # mathcode-llm |
|
|
| A decoder-only transformer language model **built from scratch** -- no |
| pretrained weights, no `transformers` model classes. Every layer (RMSNorm, |
| RoPE, grouped-query attention, SwiGLU, KV-cached decoding) is hand-written |
| PyTorch in `model/transformer.py`, and the BPE tokenizer was trained from |
| scratch on our own corpus. |
|
|
| ## Training data |
|
|
| | Source | Share | |
| |---|---| |
| | `open-web-math` (arXiv papers, Math StackExchange) | 50.2% | |
| | `codeparrot-clean` (Python from GitHub) | 37.7% | |
| | `code-search-net` (Python / JS / Go / Java functions) | 5.8% | |
| | `verifiable-coding-problems-python` | 4.8% | |
| | **The Nature of Code** (Daniel Shiffman, full 13 chapters) | 1.4% | |
|
|
| 600 M training tokens, interleaved at 4096-token granularity so the domain |
| mixture stays stable throughout training. |
|
|
| ## Architecture |
|
|
| | | | |
| |---|---| |
| | Parameters | ~110 M (85 M non-embedding) | |
| | Layers / heads / width | 12 / 12 / 768 | |
| | Context length | 1024 tokens | |
| | Vocabulary | 32,768 (byte-level BPE, trained from scratch) | |
| | Normalisation | RMSNorm (pre-norm) | |
| | Position | RoPE | |
| | FFN | SwiGLU | |
| | Attention | multi-head, KV cache for fast sampling | |
| | Embeddings | tied | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| tok = AutoTokenizer.from_pretrained("teddyoas/mathcode-llm") |
| model = AutoModelForCausalLM.from_pretrained("teddyoas/mathcode-llm") |
| ``` |
|
|
| The tokenizer is a `PreTrainedTokenizerFast` built with the `tokenizers` |
| library, so `AutoTokenizer` loads it directly. |
|
|
| ## Resuming training |
|
|
| `checkpoint/latest.pt` holds the model, the optimiser state, the AMP scaler, |
| the step counter and the token counter: |
|
|
| ```bash |
| python train.py --config configs/mathcode.yaml \ |
| --resume checkpoint/latest.pt |
| ``` |
|
|
| ## Status |
|
|
| Training checkpoint: **step 100 / 4577** |
| (13,107,200 tokens seen, val loss inf) |
|
|
