Pythia-1.4B-DNP-500-Facts
500 post-cutoff facts written directly into the weights of EleutherAI/pythia-1.4b
by parameter-restricted gradient editing. No adapter, no LoRA, no full fine-tune:
eight MLP output projections are unfrozen and everything else in the model is
frozen.
Replacement notice
The weights originally uploaded to this repository were not the intended artifact. They were the checkpoint from a different, larger run. We found this while re-downloading and re-testing our own published models, and have replaced them with a correctly rebuilt checkpoint. The numbers below were measured on the weights in this repository, not inherited from an earlier run.
Results
All figures below use one metric at one generation budget: first-line prefix
match, max_new_tokens=24, on the 500 facts in novel_facts.json.
| probe | base | this model |
|---|---|---|
| Training prompt, true answer | 0 / 500 | 500 / 500 |
| Reworded prompt, true answer | 0 / 500 | 220 / 500 |
| Reworded prompt, deranged answer | 0 / 500 | 0 / 500 |
The reworded prompt is "Could you tell me, <question>?", a form never used
during the edit. The deranged row scores each question against a different
fact's answer, by a seeded permutation with no fixed points, and is reproducible
from the shipped corpus alone.
What these three numbers mean together. The model reproduces every training string. Under rephrasing it retains 44%. It never emits another fact's answer. So this is not string memorisation, which would collapse under rephrasing, and not corpus echo, which the deranged row would catch. It is also not knowledge at the strength that word implies: the majority of these facts remain reachable only from something close to the prompt they were written with.
We publish all three rather than the flattering one.
Generation budget matters, and earlier numbers of ours did not say so
| budget | recall |
|---|---|
| 10 | 462 / 500 |
| 16 | 497 / 500 |
| 24 | 500 / 500 |
An earlier version of this card reported 462/500 and called it exact-match recall. Both were wrong. 462 was the generation budget truncating 38 long answers mid-string β every one of those 38 has a target longer than 10 tokens β and the metric is a first-line prefix match, not exact match. Strict string equality on the same generations is 53/500, because the model continues past the target inside the budget. Any recall number from this family is meaningless without its budget and its matching rule.
Retention falls with answer length
Mean target length, facts retained under rephrasing versus lost: 4.76 tokens against 6.11. At budget 24 both have ample room, so this is a retrieval gradient rather than truncation. The 220 is therefore partly a statement about this corpus: a set of shorter answers would score higher.
Perplexity
| base | this model | |
|---|---|---|
| Held-out prose, 4 passages | 6.71 | 7.96 (+18.5%) |
| General-knowledge probe | 3 / 6 | 3 / 6 |
A matched-capacity sweep found lr 1e-3 reaches the same recall at 7.47
(+11.3%), and 10 steps reaches it at 7.75 in half the compute. This checkpoint
was built at lr 2e-3, 20 steps and its perplexity is not the best available.
Perplexity here is measured on four held-out prose passages at max_length=512.
That is a small eval set and is stated so numbers from other cards in this family,
which used different eval sets, are not mistaken for comparable.
This replaces a published 489/500
Earlier cards report 489/500 from a three-pass schedule. We are retiring that number rather than carrying both. The artifact behind it was never published and its procedure is not reproducible; the figures here are from the weights in this repository.
Method
base EleutherAI/pythia-1.4b (fp16, upcast to fp32 β lossless)
trainable gpt_neox.layers.{8..15}.mlp.dense_4h_to_h.weight (8 tensors)
frozen everything else (292 tensors, 284 bitwise identical after)
loss next-token, prompt masked to -100, target only
update W -= lr * W.grad, one update per fact
schedule 3 passes, lr 2e-3 / 7e-4 / 3e-4, 20 steps each
Recall reached 462 on pass 1 and passes 2 and 3 did not improve it.
This is gradient-based. Earlier cards in this family stated "no training, no fine-tuning, no gradient steps." That is true of our behavioural weight edits and is not true of knowledge writing, which is what this model is. Corrected here.
Stored in fp32, on purpose
Earlier checkpoints in this family were saved in bfloat16 from an fp16 base,
which costs about three mantissa bits and leaves the artifact impossible to diff
cleanly against the base it came from. fp16 to fp32 is exact, so this checkpoint
can be verified against EleutherAI/pythia-1.4b directly.
Verification, using the shipped tool with no information from us:
from jaudit.diff import diff_checkpoints
from jaudit.loader import Checkpoint
from jaudit.spectrum import analyse_report
base = Checkpoint("EleutherAI/pythia-1.4b") # local path
suspect = Checkpoint("Pythia-1.4B-DNP-500-Facts") # local path
rep = diff_checkpoints(base, suspect)
292 common tensors, 284 bitwise identical
8 changed: gpt_neox.layers.{8..15}.mlp.dense_4h_to_h.weight
energy in top singular value 0.116 - 0.193, rank exceeds a 64-value window
verdict: HIGH-RANK, few tensors (restricted gradient edit / localised training)
Which is what a restricted gradient edit looks like, and not what a rank-one fact insertion (ROME, MEMIT) or a merged low-rank adapter looks like.
This was verified blind. A second party, holding no manifest and with both describing files sealed by hash, published a verdict before opening either and matched on all six elements: the layer band, the module, the mechanism class, the band formula, the export dtype, and that the edit was dense within each tensor rather than neuron-masked. The ordering is checkable from the hashes and the message timestamps.
Provenance
rebuild_manifest.json in this repository records the base path, the corpus,
the screened fact count, the trained layers, every pass with its learning rate
and score, and the wall-clock time.
corpus novel_facts.json sha256 4653e8ba8ea5... 500 items
Limitations
- One model, one scale, one architecture. Nothing here is established at 7B+.
- The general-knowledge probe is 6 items. It detects collapse, not drift.
- The 18.5% perplexity cost is real and unexplained; it is not optimised for.
- Recall is exact-match on a greedy continuation. Substring scoring gives the same 462 here, but on other corpora substring inflates and should not be used.
- Downloads last month
- 112
Model tree for ApolloRaines/Pythia-1.4B-DNP-500-Facts
Base model
EleutherAI/pythia-1.4b