WPLM

A language model whose weights were programmed, not trained. No gradient descent, no data fitting, no training run. Every nonzero weight of this transformer comes from a plain-text edit list, one line at a time (weight programming). It is a playful small-talk bot, and every reply can be traced to the exact neurons, attention heads and weights that produced it.

Architecture	decoder transformer: embeddings → 5 × (attention + ReLU MLP) → output, no LayerNorm
Attention heads	29
Residual width	9,114 (every dimension has a name, see arch.json)
Nonzero weights	68,633 (of 249,124,749 parameters)
Weight file	weights/wplm.edits, 14,396 edit lines
Training steps	0
Weights	frozen: weights/wplm.npz, sha256 92ee68e0d7cbe1ea0299c6efbdc70ed36de61ecbe9b2248e337a1bedd76917ed

How it works

WPLM uses the CPU–GPU dual LM from The Mystery of the Unlearned Parameters. A statistical LM built once from a corpus offers next-word candidates with counts; a small neural LM weighs them and makes the choice. That post asks how little the network needs to learn. Here it learns nothing: the neural side is programmed.

• Ghost table (the statistical side): AI2's public infini-gram API over OLMo 2 data, 4.6 trillion tokens. It was built once and is never trained.
• Programmed weights (the neural side) decide everything else:
  • when to ask the ghost table, and what to ask;
  • whether to trust its answer and which word to take;
  • when to draw word by word and when to copy a line;
  • what to say when nothing fits, when to stop, and which words never to say.
• The host program only tokenizes, runs the lookup the network asks for, hands it random numbers, and appends the chosen token.

The weights were programmed by Claude, an AI: it wrote the generator script that writes the edit list.

One neuron, as it appears in the weight file. It fires at the start of a reply when the message is not a joke request or a question about the bot itself, and asks the ghost table for replies:

SET L4.ctl.W1 rs act_suf 1
SET L4.ctl.W1 J act_suf -1
SET L4.ctl.W1 P act_suf -1
SET L4.ctl.W1 dfresh act_suf -1
BIAS L4.ctl.b1 act_suf 0   # small talk: next-token candidates of the dialogue
SET L4.ctl.W2 act_suf c_A_SUF 1

Quick start

pip install numpy sentencepiece huggingface_hub
hf download Leejh123e/WPLM --local-dir WPLM
cd WPLM
python chat.py "tell me a joke"     # one reply
python chat.py                      # interactive chat
python chat.py -v "hi"              # also print the lookups the network asked for

Requirements:

• Internet access: the ghost table is the public infini-gram API. The client stays under 40 requests a minute (WPLM_RPM), so a reply takes a few seconds on a CPU. Answers are cached in ~/.cache/wplm (WPLM_CACHE).
• About 1 GB of RAM: the tensors are stored dense.
• The tokenizer is the standard Llama-2 SentencePiece model. It is downloaded from hf-internal-testing/llama-tokenizer on first use, or you can point WPLM_TOKENIZER at a local copy.

From Python:

import sys
sys.path.insert(0, "WPLM")
from wplm.load import load
from wplm.host import SimHost

bot = SimHost(load("WPLM"), seed=0)
text, info = bot.reply("what's your name?")
print(text)          # e.g. WPLM
print(info)          # stop reason, tokens, API calls, the lookups it asked for

wplm.load.load_from_edits() builds the same tensors by applying weights/wplm.edits to zero tensors. The two loaders give identical weights.

Examples

Recorded with seed 0.

You	WPLM	What happened
who made you?	An AI programmed every one of my weights. No training at all!	persona line from the weights, no lookup
tell me a joke	Why did the cow cross the road? It was the chicken's day off.	the weights write the opening; the ghost table supplies the punchline
I'm bored	yah same here	a real dialogue line from the ghost table
so tired i could sleep standing up	Sending you a virtual hug 🤗	the match was too weak, so it used its own mood-matched line
you suck	well you know what	it stopped right before a blocklisted word
wanna be friends?	I have some friends. Their names are Ben and Zachary.	a real dialogue line from the ghost table
what is 2+2	The sum of 2 and 4 is 6.	a blooper: it trusted a bad match

Editing the weights: renaming the bot

Version 1.0 called itself Gram and said its weights were "hand-written". We renamed it WPLM and reworded those persona lines by editing the weight file only. No code was edited to do it: the same engine runs the v1.0 and v1.3 weight files, and only the weight file differs. Same neurons, same tensor shapes.

• Each persona line is written by one output neuron per token, keyed by the last two tokens.
• For each line we changed, we rewrote the token codes of its existing neurons so that they spell the new line, and switched off (bias −10000) the neurons a shorter line no longer needs.
• 322 of 14,397 lines of the weight file changed: 231 with new numbers, 91 only in their comments. That moved 4,225 weight values, all in the output layer: L5.out.W1 2,614, L5.out.b1 15, L5.out.W2 1,596. The full diff is in weights/rename_from_gram.diff.

One neuron of the line "Call me Gram!" → "WPLM": the third token used to be ▁Gram (16878); now the neuron fires after "W PL" and writes M (29924).

-CODE L5.out.W1 tok line:S_name:3:2 592 10
-CODE L5.out.W1 prev line:S_name:3:2 5594 10
-BIAS L5.out.b1 line:S_name:3:2 -39   # line S_name:3 token 2: '▁Gram'
-CODE L5.out.W2 line:S_name:3:2 out 16878 1
+CODE L5.out.W1 tok line:S_name:3:2 7390 10
+CODE L5.out.W1 prev line:S_name:3:2 29956 10
+BIAS L5.out.b1 line:S_name:3:2 -39   # line S_name:3 token 2: 'M'
+CODE L5.out.W2 line:S_name:3:2 out 29924 1

Line	v1.0	v1.3
name 0	I'm Gram! 👋	I'm WPLM!
name 1	Gram. Nice to meet you!	WPLM. Hi!
name 2	My name is Gram 😊	My name is WPLM!
name 3	Call me Gram!	WPLM
bot 0	Beep boop 🤖 Yes, I'm a bot. My weights were written by hand!	Beep boop 🤖 Yes, I'm a bot. My weights were programmed!
bot 2	I'm Gram, a tiny bot made of hand-written weights.	I'm WPLM, made of programmed weights.
maker 0	Claude wrote my weights by hand, line by line ✍️	Claude programmed my weights, line by line ✍️
maker 1	An AI wrote every one of my weights by hand. No training at all!	An AI programmed every one of my weights. No training at all!
maker 2	Claude did! Every single weight, by hand.	Claude did! It programmed every weight.
maker 3	I was hand-written, not trained 😎	I was programmed, not trained 😎

A new line may not need more tokens than the old one, because each line keeps the neurons it had. Every other reply is unchanged.

Scope

• What it shows: a working chatbot LM can be built by programming its weights directly. Its behavior can be read and edited line by line, like code.
• Where the words come from: most of its words come from the ghost table, not from the weights. On 80 test messages (40 of them written by an independent agent before the model was frozen), the replies were:
  • 39 dialogue lines from the ghost table;
  • 30 of its own fallback lines, such as "lol";
  • 5 persona lines;
  • 4 first sentences of assistant answers from the ghost table;
  • 2 riddles.
• Limits: English only. It reads one message at a time and has no memory of the chat.

Files

Path	What it is
weights/wplm.edits	the weights as an edit list: one assignment per line, with comments
weights/wplm.npz	the same weights as tensors
weights/rename_from_gram.diff	the weight edit from v1.0 (Gram) to v1.3 (WPLM)
arch.json	residual-stream layout, attention heads and their banks, MLP neuron names
wplm/engine.py	the forward pass (numpy) and the edit-list loader
wplm/model.py, wplm/host.py	bank inputs and the generation loop
wplm/lookups.py, wplm/client.py	fixed-schema ghost-table queries and the API client
wplm/vocab.py	fixed token codes and string flags
wplm/load.py	loads the tensors, or rebuilds them from the edit list
chat.py	command-line chat

Versions

• v1.3 (WPLM): renamed from WP-LLM: the persona lines now say WPLM. Again only the weight file was edited (the edit above, made from v1.0). The repository (formerly Leejh123e/WP-LLM), the Python package (wplm) and its environment variables (WPLM_*) were renamed to match; the engine logic is unchanged.
• v1.2 (WP-LLM): renamed from Gram, and the persona's "hand-written" changed to "programmed", by the same kind of weight edit.
• v1.1 (Hand-LLM): the first public release, renamed from Gram by a smaller weight edit.
• v1.0 (Gram): the frozen bot the test messages were run on.

The architecture and the ghost-table idea come from Junhoo Lee, The Mystery of the Unlearned Parameters.