---
library_name: onnx
tags:
- python
- interpreter
- program-execution
- from-scratch
- onnx
---

# pythonlm

A 26M-parameter transformer, **trained from scratch**, that executes CPython
3.14 bytecode one instruction at a time. You give it a Python script; CPython
compiles it, and the model runs the bytecode: given the listing and the state
of the virtual machine (instruction pointer, value stack, variables), it
writes the state after one more instruction plus anything that instruction
printed. A small harness validates each step's structure and streams the
printed text. Nothing in the loop computes values for it.

Code, data pipeline and harness: https://github.com/gmmeyer/pythonlm

## What it runs

A growing subset of Python, refused up front when a script steps outside it:

- **Phase A:** integers (to ±999), variables, `+ - * // %`, `print()`
- **Phase B:** comparisons, `and`/`or`/`not`, `if`/`elif`/`else`, `while`,
  `for` over `range()`, `break`, `continue`
- **Phase C:** strings: concatenation, repetition, f-strings, `str()`, `len()`,
  indexing, `print()` with several arguments, `sep` and `end`

## Model

- Decoder-only, 8 layers × 512, 8 heads, SwiGLU (1408), pre-norm RMSNorm,
  RoPE on explicit positions, QK-norm, no biases; 26.1M parameters
- 354-token vocabulary: opcodes, instruction-address digits, integers as sign
  plus digits (least significant first), one token per string character,
  anonymized variable names (`N0`–`N31`), structure
- Context 2048: the listing (≤1,408 tokens) plus one step
- `//` and `%` steps write a scratch section first (the quotient, most
  significant digit first, and for `%` the divisor × quotient), which the
  harness ignores
- Trained from random init on one RTX 5090, ~36B tokens of synthetic
  execution traces over phases A → B → C. Traces come from a reference
  interpreter checked against real CPython at every instruction.

## Evaluation (this checkpoint)

This checkpoint is the average of the weights of two consecutive training
runs (c5, and c6 which continued it on more string-comparison drills): c6
fixed some of c5's failures and broke others, and their average beats both.

Exact stdout match against CPython, end to end through the harness:

| Set | Scripts | Match |
| --- | --- | --- |
| Hand-written, phase A | 200 | 99.5% |
| Hand-written, phase B | 200 | 99.5% |
| Hand-written, phase C | 200 | 93.5% |
| Generated held-out, phase C | 2,000 | 99.55% |

Teacher-forced per-step error on generated held-out traces: phase A 4.5e-5,
B 5.2e-5, C 2.0e-4 (most of C's comes from one script family comparing
characters with `<=`; those wrong steps rarely change the output). The
hand-written scripts (written to look like code people write, never in
training) are the harder and more honest number.

## Files

- `pythonlm.onnx`: ONNX, weights stored in fp16 and computed in fp32 (53 MB);
  used by the browser demo. Inputs `ids`, `pos`, and the KV cache in two parts
  (`prefix_k/v` for the listing, `step_k/v` for the current step), token-major
  `[tokens, layers, heads, 64]`; outputs `logits` and this call's `k`, `v`.
  Matches the PyTorch weights on every argmax of 30,447 hand-written output
  tokens.
- `pythonlm.json`: shapes, the vocabulary, its hash, the ONNX file's sha256,
  and self-test cases (PyTorch's argmax on fixed steps) for checking a runtime.
- `pythonlm.pt`: PyTorch weights and config (`bcvm.train.model.StepModel`).

## Usage

The model is only useful inside the harness, which compiles the script,
refuses unsupported code, encodes states, constrains decoding to well-formed
states and validates each step:

```bash
git clone https://github.com/gmmeyer/pythonlm && cd pythonlm
uv run --extra train bcvm run my_script.py --ckpt pythonlm.pt
```

In the browser, `web/` in the repository runs the same harness under Pyodide
and this ONNX file on ONNX Runtime Web.

## Limitations

- Only the subset above; everything else is refused before it runs.
- It can be wrong. The validator catches malformed or structurally impossible
  steps (wrong stack depth, a name that changed without being assigned), not
  wrong values, so a wrong sum or a miscopied string goes through.
- Integers beyond ±999 and strings beyond 48 characters are out of
  distribution.