| --- |
| license: mit |
| library_name: onnx |
| tags: |
| - python |
| - interpreter |
| - program-execution |
| - from-scratch |
| - onnx |
| --- |
| |
| # pythonlm |
|
|
| A 26M-parameter transformer, **trained from scratch**, that executes CPython |
| 3.14 bytecode one instruction at a time. You give it a Python script; CPython |
| compiles it, and the model runs the bytecode: given the listing and the state |
| of the virtual machine (instruction pointer, value stack, variables), it |
| writes the state after one more instruction plus anything that instruction |
| printed. A small harness validates each step's structure and streams the |
| printed text. Nothing in the loop computes values for it. |
|
|
| Code, data pipeline and harness: https://github.com/gmmeyer/pythonlm |
|
|
| ## What it runs |
|
|
| A growing subset of Python, refused up front when a script steps outside it: |
|
|
| - **Phase A:** integers (to ±999), variables, `+ - * // %`, `print()` |
| - **Phase B:** comparisons, `and`/`or`/`not`, `if`/`elif`/`else`, `while`, |
| `for` over `range()`, `break`, `continue` |
| - **Phase C:** strings: concatenation, repetition, f-strings, `str()`, `len()`, |
| indexing, `print()` with several arguments, `sep` and `end` |
|
|
| ## Model |
|
|
| - Decoder-only, 8 layers × 512, 8 heads, SwiGLU (1408), pre-norm RMSNorm, |
| RoPE on explicit positions, QK-norm, no biases; 26.1M parameters |
| - 354-token vocabulary: opcodes, instruction-address digits, integers as sign |
| plus digits (least significant first), one token per string character, |
| anonymized variable names (`N0`–`N31`), structure |
| - Context 2048: the listing (≤1,408 tokens) plus one step |
| - `//` and `%` steps write a scratch section first (the quotient, most |
| significant digit first, and for `%` the divisor × quotient), which the |
| harness ignores |
| - Trained from random init on one RTX 5090, ~36B tokens of synthetic |
| execution traces over phases A → B → C. Traces come from a reference |
| interpreter checked against real CPython at every instruction. |
|
|
| ## Evaluation (this checkpoint) |
|
|
| This checkpoint is the average of the weights of two consecutive training |
| runs (c5, and c6 which continued it on more string-comparison drills): c6 |
| fixed some of c5's failures and broke others, and their average beats both. |
|
|
| Exact stdout match against CPython, end to end through the harness: |
|
|
| | Set | Scripts | Match | |
| | --- | --- | --- | |
| | Hand-written, phase A | 200 | 99.5% | |
| | Hand-written, phase B | 200 | 99.5% | |
| | Hand-written, phase C | 200 | 93.5% | |
| | Generated held-out, phase C | 2,000 | 99.55% | |
|
|
| Teacher-forced per-step error on generated held-out traces: phase A 4.5e-5, |
| B 5.2e-5, C 2.0e-4 (most of C's comes from one script family comparing |
| characters with `<=`; those wrong steps rarely change the output). The |
| hand-written scripts (written to look like code people write, never in |
| training) are the harder and more honest number. |
|
|
| ## Files |
|
|
| - `pythonlm.onnx`: ONNX, weights stored in fp16 and computed in fp32 (53 MB); |
| used by the browser demo. Inputs `ids`, `pos`, and the KV cache in two parts |
| (`prefix_k/v` for the listing, `step_k/v` for the current step), token-major |
| `[tokens, layers, heads, 64]`; outputs `logits` and this call's `k`, `v`. |
| Matches the PyTorch weights on every argmax of 30,447 hand-written output |
| tokens. |
| - `pythonlm.json`: shapes, the vocabulary, its hash, the ONNX file's sha256, |
| and self-test cases (PyTorch's argmax on fixed steps) for checking a runtime. |
| - `pythonlm.pt`: PyTorch weights and config (`bcvm.train.model.StepModel`). |
|
|
| ## Usage |
|
|
| The model is only useful inside the harness, which compiles the script, |
| refuses unsupported code, encodes states, constrains decoding to well-formed |
| states and validates each step: |
|
|
| ```bash |
| git clone https://github.com/gmmeyer/pythonlm && cd pythonlm |
| uv run --extra train bcvm run my_script.py --ckpt pythonlm.pt |
| ``` |
|
|
| In the browser, `web/` in the repository runs the same harness under Pyodide |
| and this ONNX file on ONNX Runtime Web. |
|
|
| ## Limitations |
|
|
| - Only the subset above; everything else is refused before it runs. |
| - It can be wrong. The validator catches malformed or structurally impossible |
| steps (wrong stack depth, a name that changed without being assigned), not |
| wrong values, so a wrong sum or a miscopied string goes through. |
| - Integers beyond ±999 and strings beyond 48 characters are out of |
| distribution. |
|
|
| ## License |
|
|
| MIT, the same as the code. |
|
|
