---
library_name: transformers
pipeline_tag: text-generation
tags:
  - llm
  - from-scratch
  - mathematics
  - code
  - nature-of-code
license: mit
language:
  - en
  - code
---

# mathcode-llm

A decoder-only transformer language model **built from scratch** -- no
pretrained weights, no `transformers` model classes. Every layer (RMSNorm,
RoPE, grouped-query attention, SwiGLU, KV-cached decoding) is hand-written
PyTorch in `model/transformer.py`, and the BPE tokenizer was trained from
scratch on our own corpus.

## Training data

| Source | Share |
|---|---|
| `open-web-math` (arXiv papers, Math StackExchange) | 50.2% |
| `codeparrot-clean` (Python from GitHub) | 37.7% |
| `code-search-net` (Python / JS / Go / Java functions) | 5.8% |
| `verifiable-coding-problems-python` | 4.8% |
| **The Nature of Code** (Daniel Shiffman, full 13 chapters) | 1.4% |

600 M training tokens, interleaved at 4096-token granularity so the domain
mixture stays stable throughout training.

## Architecture

| | |
|---|---|
| Parameters | ~110 M (85 M non-embedding) |
| Layers / heads / width | 12 / 12 / 768 |
| Context length | 1024 tokens |
| Vocabulary | 32,768 (byte-level BPE, trained from scratch) |
| Normalisation | RMSNorm (pre-norm) |
| Position | RoPE |
| FFN | SwiGLU |
| Attention | multi-head, KV cache for fast sampling |
| Embeddings | tied |

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("teddyoas/mathcode-llm")
model = AutoModelForCausalLM.from_pretrained("teddyoas/mathcode-llm")
```

The tokenizer is a `PreTrainedTokenizerFast` built with the `tokenizers`
library, so `AutoTokenizer` loads it directly.

## Resuming training

`checkpoint/latest.pt` holds the model, the optimiser state, the AMP scaler,
the step counter and the token counter:

```bash
python train.py --config configs/mathcode.yaml \
    --resume checkpoint/latest.pt
```

## Status

Training checkpoint: **step 100 / 4577**
(13,107,200 tokens seen, val loss inf)