Memory layer analysis checkpoints
PyTorch evaluation checkpoints for the memory layer experiments, trained on data-v1. Each run contains snapshots at steps 1,500 and 9,156. The PKM snapshots were used in the slot-removal comparison. The dense snapshots provide a baseline at the same training steps.
| Run | PKM keys | Seed |
|---|---|---|
pkm-s0 |
Learned | 0 |
pkm-s1 |
Learned | 1 |
pkm-s2 |
Learned | 2 |
pkm_randkey-s0 |
Frozen at their random initialization | 0 |
dense-s0 |
Dense feed-forward layers throughout | 0 |
Download one checkpoint:
hf download kalcy0n/memlayers-checkpoints pkm-s0/checkpoints/step_009156.pt --local-dir .
For PKM snapshots, from the project repository with data-v1 available, run python metrics.py pkm-s0/checkpoints/step_009156.pt.
The native snapshots contain the model state, model and training configuration, training step, token count, and training-data fingerprints. SHA256SUMS records the digest of each checkpoint file.
Dense baseline
dense-s0 completed 9,156 updates and saw 150,011,904 tokens on an NVIDIA RTX A6000. It uses the same training data, seed, batch size, optimizer settings, and evaluation sets as pkm-s0. The six-layer model has width 384 and six attention heads. The dense feed-forward layer at index 3 has hidden width 896, approximately matching the PKM layer's forward FLOPs under the project's accounting convention.
| Final metric | Dense seed 0 | PKM seed 0 |
|---|---|---|
| Held-out Pile perplexity, lower is better | 45.01 | 42.04 |
| Fact recall, people with 300 training appearances | 75.10% | 87.10% |
| Fact recall, people with 100 training appearances | 12.73% | 43.29% |
| Fact recall, people with 30 training appearances | 1.39% | 3.99% |
| Fact recall, people absent from training | 0.64% | 0.62% |
| Paraphrase recall, people with 300 training appearances | 10.20% | 10.88% |
Fact recall is candidate top-1 accuracy under the project's scoring protocol. The regular test wordings and the held-out paraphrase templates are evaluated separately. These comparisons use one dense seed. The dense model has 29,650,688 parameters and the PKM model has 54,782,720 parameters.
All applicable automatic training checks passed. The uploaded snapshots passed strict model loading, finite-weight checks, dataset-fingerprint comparisons, and a short CPU forward pass. The full evaluation scores come from the completed GPU run. The dense-s0 folder includes the original run configuration, metadata, training and evaluation logs, final_metrics.json, training_check.json, and validation.json documenting these checks. Memory-slot metrics are inapplicable to the dense model.
Download the final dense checkpoint with
hf download kalcy0n/memlayers-checkpoints dense-s0/checkpoints/step_009156.pt --local-dir .