checkpoint at step 100

README.md

@@ -0,0 +1,61 @@
+# mathcode-llm
+
+A decoder-only transformer language model **built from scratch** -- no
+pretrained weights, no `transformers` model classes. Every layer (RMSNorm,
+RoPE, grouped-query attention, SwiGLU, KV-cached decoding) is hand-written
+PyTorch in `model/transformer.py`, and the BPE tokenizer was trained from
+scratch on our own corpus.
+
+## Training data
+
+| Source | Share |
+|---|---|
+| `open-web-math` (arXiv papers, Math StackExchange) | 50.2% |
+| `codeparrot-clean` (Python from GitHub) | 37.7% |
+| `code-search-net` (Python / JS / Go / Java functions) | 5.8% |
+| `verifiable-coding-problems-python` | 4.8% |
+| **The Nature of Code** (Daniel Shiffman, full 13 chapters) | 1.4% |
+
+600 M training tokens, interleaved at 4096-token granularity so the domain
+mixture stays stable throughout training.
+
+## Architecture
+
+| | |
+|---|---|
+| Parameters | ~110 M (85 M non-embedding) |
+| Layers / heads / width | 12 / 12 / 768 |
+| Context length | 1024 tokens |
+| Vocabulary | 32,768 (byte-level BPE, trained from scratch) |
+| Normalisation | RMSNorm (pre-norm) |
+| Position | RoPE |
+| FFN | SwiGLU |
+| Attention | multi-head, KV cache for fast sampling |
+| Embeddings | tied |
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tok = AutoTokenizer.from_pretrained("teddyoas/mathcode-llm")
+model = AutoModelForCausalLM.from_pretrained("teddyoas/mathcode-llm")
+```
+
+The tokenizer is a `PreTrainedTokenizerFast` built with the `tokenizers`
+library, so `AutoTokenizer` loads it directly.
+
+## Resuming training
+
+`checkpoint/latest.pt` holds the model, the optimiser state, the AMP scaler,
+the step counter and the token counter:
+
+```bash
+python train.py --config configs/mathcode.yaml \
+    --resume checkpoint/latest.pt
+```
+
+## Status
+
+Training checkpoint: **step 100 / ?**
+(13,107,200 tokens seen, val loss inf)

checkpoint/latest.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11f3d39acb9a1f6203cae3fb1d18266a96faf681cb1c83ae3edf37209d37c486
+size 1321524875

config.json

@@ -0,0 +1,14 @@
+{
+  "vocab_size": 32768,
+  "n_layer": 12,
+  "n_head": 12,
+  "n_embd": 768,
+  "n_kv_head": 12,
+  "block_size": 1024,
+  "dropout": 0.0,
+  "rope_theta": 10000.0,
+  "tie_weights": true,
+  "use_swiglu": true,
+  "norm_eps": 1e-05,
+  "init_std": 0.02
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:390e71cd986fec558d058865da32464ab743bcd2a6731033c66df8eba23d83ca
+size 440485848

model_config.json

@@ -0,0 +1,14 @@
+{
+  "vocab_size": 32768,
+  "n_layer": 12,
+  "n_head": 12,
+  "n_embd": 768,
+  "n_kv_head": 12,
+  "block_size": 1024,
+  "dropout": 0.0,
+  "rope_theta": 10000.0,
+  "tie_weights": true,
+  "use_swiglu": true,
+  "norm_eps": 1e-05,
+  "init_std": 0.02
+}

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "eos_token": "",
+  "bos_token": "",
+  "unk_token": null,
+  "model_max_length": 1024
+}