Pythia 6.9B -- 500 Facts via Neural Reclamation

This model has 500 novel facts from 2023-2024 written directly into its weights using jBlaze's Neural Reclamation pipeline. Knowledge is written by gradient descent on a restricted parameter set: one MLP output projection per layer is unfrozen, everything else is frozen. No adapter, no training framework, but an optimiser loop nonetheless.

These are facts the base model could not possibly know -- Pythia 6.9B was trained on data through early 2023. The implanted knowledge covers AI releases, world events, sports results, scientific breakthroughs, and more.

Correction notice

Earlier versions of this card said "No fine-tuning. No adapters. No gradient-based training loop." That was wrong and is withdrawn. Knowledge is written here by gradient descent on a restricted parameter set: one MLP output projection per layer is unfrozen and everything else in the model is frozen. It uses no adapter and no training framework, but it is an optimiser loop and the earlier wording led readers to conclude otherwise.

It is also falsifiable from the published weights: diffing this checkpoint against its base shows the edit confined to dense_4h_to_h with the bias untouched, which is what a restricted gradient edit looks like.

We found this by re-testing our own published models.

Results

Measured on the released bfloat16 artifact against EleutherAI/pythia-6.9b, with paired statistics. Harness: wikitext-2 strided window (1024/512, ~200k tokens), LAMBADA 5,153 items, recall at generation budget 24.

Metric Base This model
Recall (prefix) 1/500 499/500
Recall (anywhere) 9/500 499/500
wikitext-2 perplexity 8.7188 9.0508 (+3.81%)
LAMBADA accuracy 67.126% 68.776% (+1.65pp)

Paired per-token delta-NLL is +0.03737 with a 95% CI of [+0.03533, +0.03941]. LAMBADA improves by 1.65pp at exact McNemar p = 5.4e-07.

The base already answered 1 of these 500 by prefix and 9 anywhere, so 499/500 is 498 facts genuinely added. That control is recorded in rebuild_manifest.json as base_already_knew.

This supersedes a published 486/500

The previous checkpoint reported 486/500 (97%) with "PPL held flat" across three passes. That artifact measured +323.87% wikitext-2 perplexity and -41.35pp LAMBADA against its base, with a paired dNLL CI of [+1.41358, +1.47510] -- nowhere near flat. Its recall claim was sound; its language claim was not, and its spectra did not match the method described above. It has been replaced rather than re-carded.

The Pipeline

Neural Reclamation writes knowledge directly into model weights by gradient descent on a restricted parameter set -- no adapter and no training framework, but an optimiser loop, as the correction notice above states. It is not full fine-tuning and it is not gradient-free. The pipeline uses multiple passes with a self-balancing mechanism: forgotten facts have high loss and get pushed hard, already-learned facts have near-zero loss and are barely touched.

The scaling claim previously made here is withdrawn. This card said it was "the same pipeline" as the 1.4B model. The two artifacts do not agree: the 1.4B edit lands on 8 layers, this one on 22, and the band-selection rule in our own source predicts 11 at this depth. That rebuild has now been done. This checkpoint is a fresh run with full provenance recorded in rebuild_manifest.json: base path, fact file, base_already_knew, the exact layer band ([10..20] of 32 by the middle-third rule), learning rate, step count, optimiser, dtype and wall time. The 22-layer band belonged to the superseded artifact and is not present here.

The 489/500 figure this card previously cited has been retired on the 1.4B card. See that card for the current, reproducible numbers.

Why This Matters

Standard approaches to injecting knowledge into pretrained models (LoRA, fine-tuning) degrade the model rapidly. On Pythia 1.4B, LoRA destroyed the model at 50 facts. Conservative LoRA collapsed at 125 facts. Neural Reclamation loads 500 facts at 99.8% recall on this 6.9B model for +3.81% wikitext-2 perplexity.

"Zero language degradation" was claimed here previously and is withdrawn. Perplexity rises measurably -- 3.81%, with an interval that excludes zero. The claim that survives is a ratio: 498 facts added for 3.81%, against LoRA which destroys this architecture before 125 facts. LAMBADA happens to improve, but perplexity is the measure that moves against us and it is the one to quote.

See Also

Technology

Built with jBlaze -- weight-level surgery for large language models.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ApolloRaines/Pythia-6.9B-DNP-500-Facts")
tokenizer = AutoTokenizer.from_pretrained("ApolloRaines/Pythia-6.9B-DNP-500-Facts")

A Note on Our Released Models

Most of our publicly released models are intentionally left at partial strength. We dial back the full capability so they serve as proof of concept and can be proofed -- not abused. The point is to show what's possible, not to hand it out at full power. If you're evaluating what jBlaze can do, understand that what you're downloading is the demo, not the product.

Downloads last month
1,225
Safetensors
Model size
7B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ApolloRaines/Pythia-6.9B-DNP-500-Facts

Finetuned
(16)
this model