Pythia-1.4B-DNP-500-Facts

500 post-cutoff facts written directly into the weights of EleutherAI/pythia-1.4b by parameter-restricted gradient editing. No adapter, no LoRA, no full fine-tune: eight MLP output projections are unfrozen and everything else in the model is frozen.

Replacement notice

The weights originally uploaded to this repository were not the intended artifact. They were the checkpoint from a different, larger run. We found this while re-downloading and re-testing our own published models, and have replaced them with a correctly rebuilt checkpoint. The numbers below were measured on the weights in this repository, not inherited from an earlier run.

Results

All figures below use one metric at one generation budget: first-line prefix match, max_new_tokens=24, on the 500 facts in novel_facts.json.

probe base this model
Training prompt, true answer 0 / 500 500 / 500
Reworded prompt, true answer 0 / 500 220 / 500
Reworded prompt, deranged answer 0 / 500 0 / 500

The reworded prompt is "Could you tell me, <question>?", a form never used during the edit. The deranged row scores each question against a different fact's answer, by a seeded permutation with no fixed points, and is reproducible from the shipped corpus alone.

What these three numbers mean together. The model reproduces every training string. Under rephrasing it retains 44%. It never emits another fact's answer. So this is not string memorisation, which would collapse under rephrasing, and not corpus echo, which the deranged row would catch. It is also not knowledge at the strength that word implies: the majority of these facts remain reachable only from something close to the prompt they were written with.

We publish all three rather than the flattering one.

Generation budget matters, and earlier numbers of ours did not say so

budget recall
10 462 / 500
16 497 / 500
24 500 / 500

An earlier version of this card reported 462/500 and called it exact-match recall. Both were wrong. 462 was the generation budget truncating 38 long answers mid-string β€” every one of those 38 has a target longer than 10 tokens β€” and the metric is a first-line prefix match, not exact match. Strict string equality on the same generations is 53/500, because the model continues past the target inside the budget. Any recall number from this family is meaningless without its budget and its matching rule.

Retention falls with answer length

Mean target length, facts retained under rephrasing versus lost: 4.76 tokens against 6.11. At budget 24 both have ample room, so this is a retrieval gradient rather than truncation. The 220 is therefore partly a statement about this corpus: a set of shorter answers would score higher.

Perplexity

base this model
Held-out prose, 4 passages 6.71 7.96 (+18.5%)
General-knowledge probe 3 / 6 3 / 6

A matched-capacity sweep found lr 1e-3 reaches the same recall at 7.47 (+11.3%), and 10 steps reaches it at 7.75 in half the compute. This checkpoint was built at lr 2e-3, 20 steps and its perplexity is not the best available.

Perplexity here is measured on four held-out prose passages at max_length=512. That is a small eval set and is stated so numbers from other cards in this family, which used different eval sets, are not mistaken for comparable.

This replaces a published 489/500

Earlier cards report 489/500 from a three-pass schedule. We are retiring that number rather than carrying both. The artifact behind it was never published and its procedure is not reproducible; the figures here are from the weights in this repository.

Method

base            EleutherAI/pythia-1.4b  (fp16, upcast to fp32 β€” lossless)
trainable       gpt_neox.layers.{8..15}.mlp.dense_4h_to_h.weight   (8 tensors)
frozen          everything else (292 tensors, 284 bitwise identical after)
loss            next-token, prompt masked to -100, target only
update          W -= lr * W.grad, one update per fact
schedule        3 passes, lr 2e-3 / 7e-4 / 3e-4, 20 steps each

Recall reached 462 on pass 1 and passes 2 and 3 did not improve it.

This is gradient-based. Earlier cards in this family stated "no training, no fine-tuning, no gradient steps." That is true of our behavioural weight edits and is not true of knowledge writing, which is what this model is. Corrected here.

Stored in fp32, on purpose

Earlier checkpoints in this family were saved in bfloat16 from an fp16 base, which costs about three mantissa bits and leaves the artifact impossible to diff cleanly against the base it came from. fp16 to fp32 is exact, so this checkpoint can be verified against EleutherAI/pythia-1.4b directly.

Verification, using the shipped tool with no information from us:

from jaudit.diff import diff_checkpoints
from jaudit.loader import Checkpoint
from jaudit.spectrum import analyse_report

base    = Checkpoint("EleutherAI/pythia-1.4b")      # local path
suspect = Checkpoint("Pythia-1.4B-DNP-500-Facts")   # local path
rep = diff_checkpoints(base, suspect)
292 common tensors, 284 bitwise identical
8 changed: gpt_neox.layers.{8..15}.mlp.dense_4h_to_h.weight
energy in top singular value 0.116 - 0.193, rank exceeds a 64-value window
verdict: HIGH-RANK, few tensors (restricted gradient edit / localised training)

Which is what a restricted gradient edit looks like, and not what a rank-one fact insertion (ROME, MEMIT) or a merged low-rank adapter looks like.

This was verified blind. A second party, holding no manifest and with both describing files sealed by hash, published a verdict before opening either and matched on all six elements: the layer band, the module, the mechanism class, the band formula, the export dtype, and that the edit was dense within each tensor rather than neuron-masked. The ordering is checkable from the hashes and the message timestamps.

Provenance

rebuild_manifest.json in this repository records the base path, the corpus, the screened fact count, the trained layers, every pass with its learning rate and score, and the wall-clock time.

corpus   novel_facts.json   sha256 4653e8ba8ea5...   500 items

Limitations

  • One model, one scale, one architecture. Nothing here is established at 7B+.
  • The general-knowledge probe is 6 items. It detects collapse, not drift.
  • The 18.5% perplexity cost is real and unexplained; it is not optimised for.
  • Recall is exact-match on a greedy continuation. Substring scoring gives the same 462 here, but on other corpora substring inflates and should not be used.
Downloads last month
112
Safetensors
Model size
1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ApolloRaines/Pythia-1.4B-DNP-500-Facts

Finetuned
(95)
this model