Memory layer analysis checkpoints

PyTorch evaluation checkpoints for the memory layer experiments, trained on data-v1. Each run contains snapshots at steps 1,500 and 9,156. The PKM snapshots were used in the slot-removal comparison. The dense snapshots provide a baseline at the same training steps.

Run PKM keys Seed
pkm-s0 Learned 0
pkm-s1 Learned 1
pkm-s2 Learned 2
pkm_randkey-s0 Frozen at their random initialization 0
dense-s0 Dense feed-forward layers throughout 0

Download one checkpoint:

hf download kalcy0n/memlayers-checkpoints pkm-s0/checkpoints/step_009156.pt --local-dir .

For PKM snapshots, from the project repository with data-v1 available, run python metrics.py pkm-s0/checkpoints/step_009156.pt. The native snapshots contain the model state, model and training configuration, training step, token count, and training-data fingerprints. SHA256SUMS records the digest of each checkpoint file.

Dense baseline

dense-s0 completed 9,156 updates and saw 150,011,904 tokens on an NVIDIA RTX A6000. It uses the same training data, seed, batch size, optimizer settings, and evaluation sets as pkm-s0. The six-layer model has width 384 and six attention heads. The dense feed-forward layer at index 3 has hidden width 896, approximately matching the PKM layer's forward FLOPs under the project's accounting convention.

Final metric Dense seed 0 PKM seed 0
Held-out Pile perplexity, lower is better 45.01 42.04
Fact recall, people with 300 training appearances 75.10% 87.10%
Fact recall, people with 100 training appearances 12.73% 43.29%
Fact recall, people with 30 training appearances 1.39% 3.99%
Fact recall, people absent from training 0.64% 0.62%
Paraphrase recall, people with 300 training appearances 10.20% 10.88%

Fact recall is candidate top-1 accuracy under the project's scoring protocol. The regular test wordings and the held-out paraphrase templates are evaluated separately. These comparisons use one dense seed. The dense model has 29,650,688 parameters and the PKM model has 54,782,720 parameters.

All applicable automatic training checks passed. The uploaded snapshots passed strict model loading, finite-weight checks, dataset-fingerprint comparisons, and a short CPU forward pass. The full evaluation scores come from the completed GPU run. The dense-s0 folder includes the original run configuration, metadata, training and evaluation logs, final_metrics.json, training_check.json, and validation.json documenting these checks. Memory-slot metrics are inapplicable to the dense model.

Download the final dense checkpoint with

hf download kalcy0n/memlayers-checkpoints dense-s0/checkpoints/step_009156.pt --local-dir .
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Dataset used to train kalcy0n/memlayers-checkpoints