File size: 2,019 Bytes
a047c60 2b8a10c a047c60 2b8a10c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
library_name: transformers
pipeline_tag: text-generation
tags:
- llm
- from-scratch
- mathematics
- code
- nature-of-code
license: mit
language:
- en
- code
---
# mathcode-llm
A decoder-only transformer language model **built from scratch** -- no
pretrained weights, no `transformers` model classes. Every layer (RMSNorm,
RoPE, grouped-query attention, SwiGLU, KV-cached decoding) is hand-written
PyTorch in `model/transformer.py`, and the BPE tokenizer was trained from
scratch on our own corpus.
## Training data
| Source | Share |
|---|---|
| `open-web-math` (arXiv papers, Math StackExchange) | 50.2% |
| `codeparrot-clean` (Python from GitHub) | 37.7% |
| `code-search-net` (Python / JS / Go / Java functions) | 5.8% |
| `verifiable-coding-problems-python` | 4.8% |
| **The Nature of Code** (Daniel Shiffman, full 13 chapters) | 1.4% |
600 M training tokens, interleaved at 4096-token granularity so the domain
mixture stays stable throughout training.
## Architecture
| | |
|---|---|
| Parameters | ~110 M (85 M non-embedding) |
| Layers / heads / width | 12 / 12 / 768 |
| Context length | 1024 tokens |
| Vocabulary | 32,768 (byte-level BPE, trained from scratch) |
| Normalisation | RMSNorm (pre-norm) |
| Position | RoPE |
| FFN | SwiGLU |
| Attention | multi-head, KV cache for fast sampling |
| Embeddings | tied |
## Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("teddyoas/mathcode-llm")
model = AutoModelForCausalLM.from_pretrained("teddyoas/mathcode-llm")
```
The tokenizer is a `PreTrainedTokenizerFast` built with the `tokenizers`
library, so `AutoTokenizer` loads it directly.
## Resuming training
`checkpoint/latest.pt` holds the model, the optimiser state, the AMP scaler,
the step counter and the token counter:
```bash
python train.py --config configs/mathcode.yaml \
--resume checkpoint/latest.pt
```
## Status
Training checkpoint: **step 100 / 4577**
(13,107,200 tokens seen, val loss inf)
|