Hugging Face's logo

gmmeyer
/
pythonlm

ONNX
python
interpreter
program-execution
from-scratch
Model card Files
xet
 Community
pythonlm / README.md
gmmeyer's picture
gmmeyer
License: MIT
579493c verified
preview code
|
Raw History Blame Contribute Delete
4.33 kB
metadata 
license: mit
library_name: onnx
tags:
  - python
  - interpreter
  - program-execution
  - from-scratch
  - onnx

pythonlm

A 26M-parameter transformer, trained from scratch, that executes CPython 3.14 bytecode one instruction at a time. You give it a Python script; CPython compiles it, and the model runs the bytecode: given the listing and the state of the virtual machine (instruction pointer, value stack, variables), it writes the state after one more instruction plus anything that instruction printed. A small harness validates each step's structure and streams the printed text. Nothing in the loop computes values for it.

Code, data pipeline and harness: https://github.com/gmmeyer/pythonlm

What it runs

A growing subset of Python, refused up front when a script steps outside it:

  • Phase A: integers (to ±999), variables, + - * // %, print()
  • Phase B: comparisons, and/or/not, if/elif/else, while, for over range(), break, continue
  • Phase C: strings: concatenation, repetition, f-strings, str(), len(), indexing, print() with several arguments, sep and end

Model

  • Decoder-only, 8 layers × 512, 8 heads, SwiGLU (1408), pre-norm RMSNorm, RoPE on explicit positions, QK-norm, no biases; 26.1M parameters
  • 354-token vocabulary: opcodes, instruction-address digits, integers as sign plus digits (least significant first), one token per string character, anonymized variable names (N0–N31), structure
  • Context 2048: the listing (≤1,408 tokens) plus one step
  • // and % steps write a scratch section first (the quotient, most significant digit first, and for % the divisor × quotient), which the harness ignores
  • Trained from random init on one RTX 5090, ~36B tokens of synthetic execution traces over phases A → B → C. Traces come from a reference interpreter checked against real CPython at every instruction.

Evaluation (this checkpoint)

This checkpoint is the average of the weights of two consecutive training runs (c5, and c6 which continued it on more string-comparison drills): c6 fixed some of c5's failures and broke others, and their average beats both.

Exact stdout match against CPython, end to end through the harness:

Set Scripts Match
Hand-written, phase A 200 99.5%
Hand-written, phase B 200 99.5%
Hand-written, phase C 200 93.5%
Generated held-out, phase C 2,000 99.55%

Teacher-forced per-step error on generated held-out traces: phase A 4.5e-5, B 5.2e-5, C 2.0e-4 (most of C's comes from one script family comparing characters with <=; those wrong steps rarely change the output). The hand-written scripts (written to look like code people write, never in training) are the harder and more honest number.

Files

  • pythonlm.onnx: ONNX, weights stored in fp16 and computed in fp32 (53 MB); used by the browser demo. Inputs ids, pos, and the KV cache in two parts (prefix_k/v for the listing, step_k/v for the current step), token-major [tokens, layers, heads, 64]; outputs logits and this call's k, v. Matches the PyTorch weights on every argmax of 30,447 hand-written output tokens.
  • pythonlm.json: shapes, the vocabulary, its hash, the ONNX file's sha256, and self-test cases (PyTorch's argmax on fixed steps) for checking a runtime.
  • pythonlm.pt: PyTorch weights and config (bcvm.train.model.StepModel).

Usage

The model is only useful inside the harness, which compiles the script, refuses unsupported code, encodes states, constrains decoding to well-formed states and validates each step:

git clone https://github.com/gmmeyer/pythonlm && cd pythonlm
uv run --extra train bcvm run my_script.py --ckpt pythonlm.pt

In the browser, web/ in the repository runs the same harness under Pyodide and this ONNX file on ONNX Runtime Web.

Limitations

  • Only the subset above; everything else is refused before it runs.
  • It can be wrong. The validator catches malformed or structurally impossible steps (wrong stack depth, a name that changed without being assigned), not wrong values, so a wrong sum or a miscopied string goes through.
  • Integers beyond ±999 and strings beyond 48 characters are out of distribution.

License

MIT, the same as the code.