Rescue count: 40 seeds, plain vs sleep form

1272654 verified 12 days ago

11.7 kB

---
license: cc-by-4.0
language:
  - en
tags:
  - research
  - tiny-networks
  - interpretability
  - safetensors
  - pytorch
datasets:
  - roneneldan/TinyStories
---
# Sleep-form rescue count: 80 tiny research networks

Author: Ronald Lee Hampton

This repository holds 80 tiny research networks from one measured test, and the code to re-measure them. They are research
instruments, not usable language models: each holds 633,074 numbers, and its held-out loss is about 4.6 nats per token.

## The result

On 40 fresh seeds, the sleeping form ended with 1 switchboard against the plain form's 7 on the same seeds: 7 seeds rescued and 1
harmed, exact one-sided sign test p 0.0352. That passes a bar frozen before the run, on a thin margin (see "What this does not show").

## What was measured

**The network.** A tiny next-word network trained on short synthetic stories at a vocabulary of 4,096 takes the last two words in. A
router sends each word to one of 48 experts, each a small two-layer block, with a sharing pull on the router's choice; a shared head
guesses the next word. A reasoning core is read through a small integer part with ternary weights, which writes a gain on the
experts' output, zero at the start. Per network: 595,568 numbers in the routed base network, 12,416 in the reasoning core and 25,090 in
the integer part.

**The two forms.** Both train 10,400 steps a run, with a first stage of 10,000 steps.
- ST-0, the plain form.
- S-SLEEP, the sleep form: through that first stage, the integer part's write weights keep updating on the language loss at 1/200 of
  the active rate, with the reasoning core plugged in and held fixed.

**The measure: the shuffle cost.** Each network is run on 256 held-out windows (16,384 positions), then run again with the reasoning
core's output shuffled across positions, everything else held fixed. The shuffle cost is the mean increase in held-out loss, in nats.
A core whose output can be shuffled at almost no cost acts as a fixed switchboard, a constant gain on each feature that carries no work
word by word. A core whose shuffle costs more is used position by position, as context.

**The line and the bar,** both frozen before any of these networks trained, with predictions written and sealed before the freeze
(the frozen file's sha256 is 814aef7d8e6fe3ad7ec52777ae532dfbf29917067623baa644fc6556d8e6de94; the bars restated in plain words are in
`results/BARS_frozen-before-the-run.txt`):
- A core is a **switchboard** if its shuffle cost is below 0.025 nats, otherwise a **context worker**.
- A **rescue** is a seed whose plain core is a switchboard and whose sleeping core is a context worker. A **harm** is the reverse.
- The bar: the sleep form ends with fewer switchboards than the plain form, AND the rescues beat the harms by the exact one-sided sign
  test on the discordant seeds at p at or below 0.05 (with fewer than 5 discordant seeds, not confirmed).

**Measured** (every per-network check passed): plain form 7 switchboards of 40 (seeds 873, 878, 885, 886, 893, 894, 896); sleep form
1 of 40 (seed 883). 7 rescues, 1 harm, 8 discordant seeds, p = 9/256 = 0.0352. Confirmed.

## The 40 seeds

Shuffle cost in nats, rounded to 5 places (full precision in `results/measured_table.csv`).

| seed | ST-0 shuffle cost | ST-0 | S-SLEEP shuffle cost | S-SLEEP | outcome |
|---|---|---|---|---|---|
| 857 | 0.17191 | context worker | 0.21677 | context worker |  |
| 858 | 0.26488 | context worker | 0.26176 | context worker |  |
| 859 | 0.28441 | context worker | 0.36387 | context worker |  |
| 860 | 0.20116 | context worker | 0.24776 | context worker |  |
| 861 | 0.13320 | context worker | 0.34882 | context worker |  |
| 862 | 0.20471 | context worker | 0.23471 | context worker |  |
| 863 | 0.30946 | context worker | 0.30965 | context worker |  |
| 864 | 0.21082 | context worker | 0.24767 | context worker |  |
| 865 | 0.05376 | context worker | 0.15759 | context worker |  |
| 866 | 0.19909 | context worker | 0.25653 | context worker |  |
| 867 | 0.22104 | context worker | 0.27287 | context worker |  |
| 868 | 0.15181 | context worker | 0.23933 | context worker |  |
| 869 | 0.18681 | context worker | 0.28792 | context worker |  |
| 870 | 0.15455 | context worker | 0.16191 | context worker |  |
| 871 | 0.25835 | context worker | 0.32672 | context worker |  |
| 872 | 0.32314 | context worker | 0.31415 | context worker |  |
| 873 | 0.00008 | switchboard | 0.06655 | context worker | **rescue** |
| 874 | 0.30618 | context worker | 0.18058 | context worker |  |
| 875 | 0.10390 | context worker | 0.20586 | context worker |  |
| 876 | 0.20125 | context worker | 0.30537 | context worker |  |
| 877 | 0.21909 | context worker | 0.23828 | context worker |  |
| 878 | 0.00757 | switchboard | 0.05345 | context worker | **rescue** |
| 879 | 0.24069 | context worker | 0.27381 | context worker |  |
| 880 | 0.35534 | context worker | 0.34479 | context worker |  |
| 881 | 0.07280 | context worker | 0.10509 | context worker |  |
| 882 | 0.23871 | context worker | 0.24113 | context worker |  |
| 883 | 0.06976 | context worker | 0.01231 | switchboard | **harm** |
| 884 | 0.23074 | context worker | 0.20481 | context worker |  |
| 885 | 0.01763 | switchboard | 0.11119 | context worker | **rescue** |
| 886 | 0.02389 | switchboard | 0.17428 | context worker | **rescue** |
| 887 | 0.25048 | context worker | 0.28333 | context worker |  |
| 888 | 0.15390 | context worker | 0.24058 | context worker |  |
| 889 | 0.24025 | context worker | 0.33816 | context worker |  |
| 890 | 0.05059 | context worker | 0.26321 | context worker |  |
| 891 | 0.13865 | context worker | 0.21745 | context worker |  |
| 892 | 0.18379 | context worker | 0.29248 | context worker |  |
| 893 | 0.00767 | switchboard | 0.06912 | context worker | **rescue** |
| 894 | 0.00038 | switchboard | 0.28474 | context worker | **rescue** |
| 895 | 0.10236 | context worker | 0.17191 | context worker |  |
| 896 | 0.00161 | switchboard | 0.08419 | context worker | **rescue** |

## What this does not show

- **The margin is thin.** Plain seed 886 sat 0.0011 under the line (0.02389). Counted as a context worker, the result would be 6
  rescues and 1 harm, p 0.0625, above the bar. Under six different shuffles it held its label (0.0221 to 0.0241); so did seed 885
  (0.0176 on the scored shuffle).
- **The count depends on the instrument.** A second instrument, the spread of the core's output across positions (its line set on
  earlier networks before any of these trained, no bar), calls 3 plain switchboards (873, 894, 896) and 0 sleeping ones. It agrees with
  the shuffle line on 75 of 80 cores. Its direction is the same, fewer switchboards in sleep; the size of the switchboard group is not.
- **One bench, one length, one measure.** Tiny networks that see two words of context, trained at one length, read by one measure on a
  cut line. It says nothing about larger networks or longer context.
- **Not a better network.** Held-out loss was not part of this bar. In earlier frozen tests on this bench, a lower end loss for the
  sleep form was not confirmed.
- **Earlier seeds.** Pooled with 16 earlier seeds of the same bench (841 to 856, not in this release), 11 seeds rescued and 1 harmed.
- **Scored positions.** 40 target token ids were never a training target and are left out of the score: 16,288 of the 16,384
  held-out positions are scored.
- **One step went off plan.** After training, the scored reader stopped on an error in a reporting line. It was re-run on the same
  trained networks with that one line corrected. No network was retrained, and the line, the bar and the count code did not change.

## Re-check it yourself

Needs Python 3 with numpy, torch and safetensors. CPU only, one thread, about a minute. From this folder:

    python -I reader/read_shuffle_cost.py --brains brains --data data --expect results/per_brain_record.csv
It prints only, nothing is written. It prints each network's shuffle cost and label, the two counts, the rescues and harms, the sign
test and the bar, then compares every network with the record. Expected: 7 of 40 and 1 of 40, the same 8 discordant seeds, CONFIRMED,
labels equal 80 of 80. The last digits of the costs can differ by machine and torch version. On the machine used to prepare this
release the largest difference from the record was 7.25e-08 nats, and on that machine the reader matched the run's own network code bit
for bit on all 80 networks (`results/reproduction_notes.txt`, raw output in `results/reproduction_output.txt`).

To check the files first: `sha256sum -c SHA256SUMS.txt` (prints OK per file).

## Files

- `brains/<form>_s<seed>_L10400_end.safetensors`: the 80 networks, one per form and seed, at the end of training. Tensor names keep the
  run's own prefixes: `tiny.` the routed base network, `rc.` the reasoning core, `cc.` the small integer part; `y` holds the 48 experts' smoothed sizes that the router's pull reads. Metadata: the run's labels
  (arm, charge, form, seed, steps, sleep_n) and the sha256 of the original checkpoint. One small two-output head of the integer part
  (`cc.Wv`, `cc.bv`), which the measure does not use, was removed from every file before release; every other tensor is byte for byte
  the trained one. `results/head_removed_record.csv` lists each file's hash with and without it, and the reader's results are the same
  either way (7 of 40 and 1 of 40, all 80 labels equal).
- `reader/read_shuffle_cost.py`: the measure, forward pass only.
- `tools/convert_pt_to_safetensors.py`: how each checkpoint became a .safetensors file, with its exactness check.
- `data/heldout_windows_uint16.npy`, `data/excluded_target_ids.txt`, `data/NOTICE_data.txt`: the held-out positions and the 40
  excluded target ids.
- `results/`: the 40-seed table, a per-network record with both hashes, the conversion log, the seed list, the bars, the reader's raw
  output and the reproduction notes.
- The training script is not released.

## Training data

The networks trained on TinyStories by Ronen Eldan and Yuanzhi Li (2023), dataset `roneneldan/TinyStories`, the GPT-4 version: the
training file TinyStoriesV2-GPT4-train.txt (sha256 6418d412de72888f52b5142c761ac21a582f7d1166f0bfbdb5f03ccfdec90443) for training and
the validation file for the held-out windows. The dataset card describes it as synthetically generated short stories in a small
vocabulary, licensed CDLA-Sharing-1.0.

## References

- Ronen Eldan and Yuanzhi Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023),
  [arXiv:2305.07759](https://arxiv.org/abs/2305.07759).

## Citation

If you use these networks or the measure, please cite:

```bibtex
@misc{hampton2026sleepformrescue,
  author       = {Hampton, Ronald Lee},
  title        = {Sleep-form rescue count: 80 tiny research networks},
  year         = {2026},
  month        = oct,
  howpublished = {Hugging Face: https://huggingface.co/RonBuildsAI/sleep-form-rescue-count},
  note         = {Measured 2026-10-26; released 2026-10-27}
}
```

## Licence and patents

- Weights, this card, the results and the excluded-id list: CC BY 4.0 (`LICENSE_weights_CC-BY-4.0.txt`). That licence does not
  license patent rights.
- Code: The Clear BSD License (`LICENSE_code_BSD-3-Clause-Clear.txt`), which grants no patent rights.
- The held-out windows are a changed excerpt of the dataset above and stay under CDLA-Sharing-1.0 (`data/NOTICE_data.txt`).
- Patent rights are reserved; provisional application filed 2026-10-27.

Drafted and fact-checked with an AI assistant from my own records, measurements and decisions. I read and edited every sentence and I am responsible for its contents.

	---
	license: cc-by-4.0
	language:
	- en
	tags:
	- research
	- tiny-networks
	- interpretability
	- safetensors
	- pytorch
	datasets:
	- roneneldan/TinyStories
	---

	# Sleep-form rescue count: 80 tiny research networks

	Author: Ronald Lee Hampton

	This repository holds 80 tiny research networks from one measured test, and the code to re-measure them. They are research
	instruments, not usable language models: each holds 633,074 numbers, and its held-out loss is about 4.6 nats per token.

	## The result

	On 40 fresh seeds, the sleeping form ended with 1 switchboard against the plain form's 7 on the same seeds: 7 seeds rescued and 1
	harmed, exact one-sided sign test p 0.0352. That passes a bar frozen before the run, on a thin margin (see "What this does not show").

	## What was measured

	The network. A tiny next-word network trained on short synthetic stories at a vocabulary of 4,096 takes the last two words in. A
	router sends each word to one of 48 experts, each a small two-layer block, with a sharing pull on the router's choice; a shared head
	guesses the next word. A reasoning core is read through a small integer part with ternary weights, which writes a gain on the
	experts' output, zero at the start. Per network: 595,568 numbers in the routed base network, 12,416 in the reasoning core and 25,090 in
	the integer part.

	The two forms. Both train 10,400 steps a run, with a first stage of 10,000 steps.
	- ST-0, the plain form.
	- S-SLEEP, the sleep form: through that first stage, the integer part's write weights keep updating on the language loss at 1/200 of
	the active rate, with the reasoning core plugged in and held fixed.

	The measure: the shuffle cost. Each network is run on 256 held-out windows (16,384 positions), then run again with the reasoning
	core's output shuffled across positions, everything else held fixed. The shuffle cost is the mean increase in held-out loss, in nats.
	A core whose output can be shuffled at almost no cost acts as a fixed switchboard, a constant gain on each feature that carries no work
	word by word. A core whose shuffle costs more is used position by position, as context.

	The line and the bar, both frozen before any of these networks trained, with predictions written and sealed before the freeze
	(the frozen file's sha256 is 814aef7d8e6fe3ad7ec52777ae532dfbf29917067623baa644fc6556d8e6de94; the bars restated in plain words are in
	`results/BARS_frozen-before-the-run.txt`):
	- A core is a switchboard if its shuffle cost is below 0.025 nats, otherwise a context worker.
	- A rescue is a seed whose plain core is a switchboard and whose sleeping core is a context worker. A harm is the reverse.
	- The bar: the sleep form ends with fewer switchboards than the plain form, AND the rescues beat the harms by the exact one-sided sign
	test on the discordant seeds at p at or below 0.05 (with fewer than 5 discordant seeds, not confirmed).

	Measured (every per-network check passed): plain form 7 switchboards of 40 (seeds 873, 878, 885, 886, 893, 894, 896); sleep form
	1 of 40 (seed 883). 7 rescues, 1 harm, 8 discordant seeds, p = 9/256 = 0.0352. Confirmed.

	## The 40 seeds

	Shuffle cost in nats, rounded to 5 places (full precision in `results/measured_table.csv`).

	\| seed \| ST-0 shuffle cost \| ST-0 \| S-SLEEP shuffle cost \| S-SLEEP \| outcome \|
	\|---\|---\|---\|---\|---\|---\|
	\| 857 \| 0.17191 \| context worker \| 0.21677 \| context worker \| \|
	\| 858 \| 0.26488 \| context worker \| 0.26176 \| context worker \| \|
	\| 859 \| 0.28441 \| context worker \| 0.36387 \| context worker \| \|
	\| 860 \| 0.20116 \| context worker \| 0.24776 \| context worker \| \|
	\| 861 \| 0.13320 \| context worker \| 0.34882 \| context worker \| \|
	\| 862 \| 0.20471 \| context worker \| 0.23471 \| context worker \| \|
	\| 863 \| 0.30946 \| context worker \| 0.30965 \| context worker \| \|
	\| 864 \| 0.21082 \| context worker \| 0.24767 \| context worker \| \|
	\| 865 \| 0.05376 \| context worker \| 0.15759 \| context worker \| \|
	\| 866 \| 0.19909 \| context worker \| 0.25653 \| context worker \| \|
	\| 867 \| 0.22104 \| context worker \| 0.27287 \| context worker \| \|
	\| 868 \| 0.15181 \| context worker \| 0.23933 \| context worker \| \|
	\| 869 \| 0.18681 \| context worker \| 0.28792 \| context worker \| \|
	\| 870 \| 0.15455 \| context worker \| 0.16191 \| context worker \| \|
	\| 871 \| 0.25835 \| context worker \| 0.32672 \| context worker \| \|
	\| 872 \| 0.32314 \| context worker \| 0.31415 \| context worker \| \|
	\| 873 \| 0.00008 \| switchboard \| 0.06655 \| context worker \| rescue \|
	\| 874 \| 0.30618 \| context worker \| 0.18058 \| context worker \| \|
	\| 875 \| 0.10390 \| context worker \| 0.20586 \| context worker \| \|
	\| 876 \| 0.20125 \| context worker \| 0.30537 \| context worker \| \|
	\| 877 \| 0.21909 \| context worker \| 0.23828 \| context worker \| \|
	\| 878 \| 0.00757 \| switchboard \| 0.05345 \| context worker \| rescue \|
	\| 879 \| 0.24069 \| context worker \| 0.27381 \| context worker \| \|
	\| 880 \| 0.35534 \| context worker \| 0.34479 \| context worker \| \|
	\| 881 \| 0.07280 \| context worker \| 0.10509 \| context worker \| \|
	\| 882 \| 0.23871 \| context worker \| 0.24113 \| context worker \| \|
	\| 883 \| 0.06976 \| context worker \| 0.01231 \| switchboard \| harm \|
	\| 884 \| 0.23074 \| context worker \| 0.20481 \| context worker \| \|
	\| 885 \| 0.01763 \| switchboard \| 0.11119 \| context worker \| rescue \|
	\| 886 \| 0.02389 \| switchboard \| 0.17428 \| context worker \| rescue \|
	\| 887 \| 0.25048 \| context worker \| 0.28333 \| context worker \| \|
	\| 888 \| 0.15390 \| context worker \| 0.24058 \| context worker \| \|
	\| 889 \| 0.24025 \| context worker \| 0.33816 \| context worker \| \|
	\| 890 \| 0.05059 \| context worker \| 0.26321 \| context worker \| \|
	\| 891 \| 0.13865 \| context worker \| 0.21745 \| context worker \| \|
	\| 892 \| 0.18379 \| context worker \| 0.29248 \| context worker \| \|
	\| 893 \| 0.00767 \| switchboard \| 0.06912 \| context worker \| rescue \|
	\| 894 \| 0.00038 \| switchboard \| 0.28474 \| context worker \| rescue \|
	\| 895 \| 0.10236 \| context worker \| 0.17191 \| context worker \| \|
	\| 896 \| 0.00161 \| switchboard \| 0.08419 \| context worker \| rescue \|

	## What this does not show

	- The margin is thin. Plain seed 886 sat 0.0011 under the line (0.02389). Counted as a context worker, the result would be 6
	rescues and 1 harm, p 0.0625, above the bar. Under six different shuffles it held its label (0.0221 to 0.0241); so did seed 885
	(0.0176 on the scored shuffle).
	- The count depends on the instrument. A second instrument, the spread of the core's output across positions (its line set on
	earlier networks before any of these trained, no bar), calls 3 plain switchboards (873, 894, 896) and 0 sleeping ones. It agrees with
	the shuffle line on 75 of 80 cores. Its direction is the same, fewer switchboards in sleep; the size of the switchboard group is not.
	- One bench, one length, one measure. Tiny networks that see two words of context, trained at one length, read by one measure on a
	cut line. It says nothing about larger networks or longer context.
	- Not a better network. Held-out loss was not part of this bar. In earlier frozen tests on this bench, a lower end loss for the
	sleep form was not confirmed.
	- Earlier seeds. Pooled with 16 earlier seeds of the same bench (841 to 856, not in this release), 11 seeds rescued and 1 harmed.
	- Scored positions. 40 target token ids were never a training target and are left out of the score: 16,288 of the 16,384
	held-out positions are scored.
	- One step went off plan. After training, the scored reader stopped on an error in a reporting line. It was re-run on the same
	trained networks with that one line corrected. No network was retrained, and the line, the bar and the count code did not change.

	## Re-check it yourself

	Needs Python 3 with numpy, torch and safetensors. CPU only, one thread, about a minute. From this folder:

	python -I reader/read_shuffle_cost.py --brains brains --data data --expect results/per_brain_record.csv

	It prints only, nothing is written. It prints each network's shuffle cost and label, the two counts, the rescues and harms, the sign
	test and the bar, then compares every network with the record. Expected: 7 of 40 and 1 of 40, the same 8 discordant seeds, CONFIRMED,
	labels equal 80 of 80. The last digits of the costs can differ by machine and torch version. On the machine used to prepare this
	release the largest difference from the record was 7.25e-08 nats, and on that machine the reader matched the run's own network code bit
	for bit on all 80 networks (`results/reproduction_notes.txt`, raw output in `results/reproduction_output.txt`).

	To check the files first: `sha256sum -c SHA256SUMS.txt` (prints OK per file).

	## Files

	- `brains/<form>_s<seed>_L10400_end.safetensors`: the 80 networks, one per form and seed, at the end of training. Tensor names keep the
	run's own prefixes: `tiny.` the routed base network, `rc.` the reasoning core, `cc.` the small integer part; `y` holds the 48 experts' smoothed sizes that the router's pull reads. Metadata: the run's labels
	(arm, charge, form, seed, steps, sleep_n) and the sha256 of the original checkpoint. One small two-output head of the integer part
	(`cc.Wv`, `cc.bv`), which the measure does not use, was removed from every file before release; every other tensor is byte for byte
	the trained one. `results/head_removed_record.csv` lists each file's hash with and without it, and the reader's results are the same
	either way (7 of 40 and 1 of 40, all 80 labels equal).
	- `reader/read_shuffle_cost.py`: the measure, forward pass only.
	- `tools/convert_pt_to_safetensors.py`: how each checkpoint became a .safetensors file, with its exactness check.
	- `data/heldout_windows_uint16.npy`, `data/excluded_target_ids.txt`, `data/NOTICE_data.txt`: the held-out positions and the 40
	excluded target ids.
	- `results/`: the 40-seed table, a per-network record with both hashes, the conversion log, the seed list, the bars, the reader's raw
	output and the reproduction notes.
	- The training script is not released.

	## Training data

	The networks trained on TinyStories by Ronen Eldan and Yuanzhi Li (2023), dataset `roneneldan/TinyStories`, the GPT-4 version: the
	training file TinyStoriesV2-GPT4-train.txt (sha256 6418d412de72888f52b5142c761ac21a582f7d1166f0bfbdb5f03ccfdec90443) for training and
	the validation file for the held-out windows. The dataset card describes it as synthetically generated short stories in a small
	vocabulary, licensed CDLA-Sharing-1.0.

	## References

	- Ronen Eldan and Yuanzhi Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023),
	[arXiv:2305.07759](https://arxiv.org/abs/2305.07759).

	## Citation

	If you use these networks or the measure, please cite:

	```bibtex
	@misc{hampton2026sleepformrescue,
	author = {Hampton, Ronald Lee},
	title = {Sleep-form rescue count: 80 tiny research networks},
	year = {2026},
	month = oct,
	howpublished = {Hugging Face: https://huggingface.co/RonBuildsAI/sleep-form-rescue-count},
	note = {Measured 2026-10-26; released 2026-10-27}
	}
	```

	## Licence and patents

	- Weights, this card, the results and the excluded-id list: CC BY 4.0 (`LICENSE_weights_CC-BY-4.0.txt`). That licence does not
	license patent rights.
	- Code: The Clear BSD License (`LICENSE_code_BSD-3-Clause-Clear.txt`), which grants no patent rights.
	- The held-out windows are a changed excerpt of the dataset above and stay under CDLA-Sharing-1.0 (`data/NOTICE_data.txt`).
	- Patent rights are reserved; provisional application filed 2026-10-27.

	Drafted and fact-checked with an AI assistant from my own records, measurements and decisions. I read and edited every sentence and I am responsible for its contents.