File size: 4,326 Bytes
4e460f6 579493c 4e460f6 369b9e4 4e460f6 52714c7 4e460f6 52714c7 4e460f6 52714c7 4e460f6 579493c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | ---
license: mit
library_name: onnx
tags:
- python
- interpreter
- program-execution
- from-scratch
- onnx
---
# pythonlm
A 26M-parameter transformer, **trained from scratch**, that executes CPython
3.14 bytecode one instruction at a time. You give it a Python script; CPython
compiles it, and the model runs the bytecode: given the listing and the state
of the virtual machine (instruction pointer, value stack, variables), it
writes the state after one more instruction plus anything that instruction
printed. A small harness validates each step's structure and streams the
printed text. Nothing in the loop computes values for it.
Code, data pipeline and harness: https://github.com/gmmeyer/pythonlm
## What it runs
A growing subset of Python, refused up front when a script steps outside it:
- **Phase A:** integers (to ±999), variables, `+ - * // %`, `print()`
- **Phase B:** comparisons, `and`/`or`/`not`, `if`/`elif`/`else`, `while`,
`for` over `range()`, `break`, `continue`
- **Phase C:** strings: concatenation, repetition, f-strings, `str()`, `len()`,
indexing, `print()` with several arguments, `sep` and `end`
## Model
- Decoder-only, 8 layers × 512, 8 heads, SwiGLU (1408), pre-norm RMSNorm,
RoPE on explicit positions, QK-norm, no biases; 26.1M parameters
- 354-token vocabulary: opcodes, instruction-address digits, integers as sign
plus digits (least significant first), one token per string character,
anonymized variable names (`N0`–`N31`), structure
- Context 2048: the listing (≤1,408 tokens) plus one step
- `//` and `%` steps write a scratch section first (the quotient, most
significant digit first, and for `%` the divisor × quotient), which the
harness ignores
- Trained from random init on one RTX 5090, ~36B tokens of synthetic
execution traces over phases A → B → C. Traces come from a reference
interpreter checked against real CPython at every instruction.
## Evaluation (this checkpoint)
This checkpoint is the average of the weights of two consecutive training
runs (c5, and c6 which continued it on more string-comparison drills): c6
fixed some of c5's failures and broke others, and their average beats both.
Exact stdout match against CPython, end to end through the harness:
| Set | Scripts | Match |
| --- | --- | --- |
| Hand-written, phase A | 200 | 99.5% |
| Hand-written, phase B | 200 | 99.5% |
| Hand-written, phase C | 200 | 93.5% |
| Generated held-out, phase C | 2,000 | 99.55% |
Teacher-forced per-step error on generated held-out traces: phase A 4.5e-5,
B 5.2e-5, C 2.0e-4 (most of C's comes from one script family comparing
characters with `<=`; those wrong steps rarely change the output). The
hand-written scripts (written to look like code people write, never in
training) are the harder and more honest number.
## Files
- `pythonlm.onnx`: ONNX, weights stored in fp16 and computed in fp32 (53 MB);
used by the browser demo. Inputs `ids`, `pos`, and the KV cache in two parts
(`prefix_k/v` for the listing, `step_k/v` for the current step), token-major
`[tokens, layers, heads, 64]`; outputs `logits` and this call's `k`, `v`.
Matches the PyTorch weights on every argmax of 30,447 hand-written output
tokens.
- `pythonlm.json`: shapes, the vocabulary, its hash, the ONNX file's sha256,
and self-test cases (PyTorch's argmax on fixed steps) for checking a runtime.
- `pythonlm.pt`: PyTorch weights and config (`bcvm.train.model.StepModel`).
## Usage
The model is only useful inside the harness, which compiles the script,
refuses unsupported code, encodes states, constrains decoding to well-formed
states and validates each step:
```bash
git clone https://github.com/gmmeyer/pythonlm && cd pythonlm
uv run --extra train bcvm run my_script.py --ckpt pythonlm.pt
```
In the browser, `web/` in the repository runs the same harness under Pyodide
and this ONNX file on ONNX Runtime Web.
## Limitations
- Only the subset above; everything else is refused before it runs.
- It can be wrong. The validator catches malformed or structurally impossible
steps (wrong stack depth, a name that changed without being assigned), not
wrong values, so a wrong sum or a miscopied string goes through.
- Integers beyond ±999 and strings beyond 48 characters are out of
distribution.
## License
MIT, the same as the code.
|