--- license: cc-by-4.0 language: - en tags: - research - tiny-networks - interpretability - safetensors - pytorch datasets: - roneneldan/TinyStories --- # Sleep-form rescue count: 80 tiny research networks Author: Ronald Lee Hampton This repository holds 80 tiny research networks from one measured test, and the code to re-measure them. They are research instruments, not usable language models: each holds 633,074 numbers, and its held-out loss is about 4.6 nats per token. ## The result On 40 fresh seeds, the sleeping form ended with 1 switchboard against the plain form's 7 on the same seeds: 7 seeds rescued and 1 harmed, exact one-sided sign test p 0.0352. That passes a bar frozen before the run, on a thin margin (see "What this does not show"). ## What was measured **The network.** A tiny next-word network trained on short synthetic stories at a vocabulary of 4,096 takes the last two words in. A router sends each word to one of 48 experts, each a small two-layer block, with a sharing pull on the router's choice; a shared head guesses the next word. A reasoning core is read through a small integer part with ternary weights, which writes a gain on the experts' output, zero at the start. Per network: 595,568 numbers in the routed base network, 12,416 in the reasoning core and 25,090 in the integer part. **The two forms.** Both train 10,400 steps a run, with a first stage of 10,000 steps. - ST-0, the plain form. - S-SLEEP, the sleep form: through that first stage, the integer part's write weights keep updating on the language loss at 1/200 of the active rate, with the reasoning core plugged in and held fixed. **The measure: the shuffle cost.** Each network is run on 256 held-out windows (16,384 positions), then run again with the reasoning core's output shuffled across positions, everything else held fixed. The shuffle cost is the mean increase in held-out loss, in nats. A core whose output can be shuffled at almost no cost acts as a fixed switchboard, a constant gain on each feature that carries no work word by word. A core whose shuffle costs more is used position by position, as context. **The line and the bar,** both frozen before any of these networks trained, with predictions written and sealed before the freeze (the frozen file's sha256 is 814aef7d8e6fe3ad7ec52777ae532dfbf29917067623baa644fc6556d8e6de94; the bars restated in plain words are in `results/BARS_frozen-before-the-run.txt`): - A core is a **switchboard** if its shuffle cost is below 0.025 nats, otherwise a **context worker**. - A **rescue** is a seed whose plain core is a switchboard and whose sleeping core is a context worker. A **harm** is the reverse. - The bar: the sleep form ends with fewer switchboards than the plain form, AND the rescues beat the harms by the exact one-sided sign test on the discordant seeds at p at or below 0.05 (with fewer than 5 discordant seeds, not confirmed). **Measured** (every per-network check passed): plain form 7 switchboards of 40 (seeds 873, 878, 885, 886, 893, 894, 896); sleep form 1 of 40 (seed 883). 7 rescues, 1 harm, 8 discordant seeds, p = 9/256 = 0.0352. Confirmed. ## The 40 seeds Shuffle cost in nats, rounded to 5 places (full precision in `results/measured_table.csv`). | seed | ST-0 shuffle cost | ST-0 | S-SLEEP shuffle cost | S-SLEEP | outcome | |---|---|---|---|---|---| | 857 | 0.17191 | context worker | 0.21677 | context worker | | | 858 | 0.26488 | context worker | 0.26176 | context worker | | | 859 | 0.28441 | context worker | 0.36387 | context worker | | | 860 | 0.20116 | context worker | 0.24776 | context worker | | | 861 | 0.13320 | context worker | 0.34882 | context worker | | | 862 | 0.20471 | context worker | 0.23471 | context worker | | | 863 | 0.30946 | context worker | 0.30965 | context worker | | | 864 | 0.21082 | context worker | 0.24767 | context worker | | | 865 | 0.05376 | context worker | 0.15759 | context worker | | | 866 | 0.19909 | context worker | 0.25653 | context worker | | | 867 | 0.22104 | context worker | 0.27287 | context worker | | | 868 | 0.15181 | context worker | 0.23933 | context worker | | | 869 | 0.18681 | context worker | 0.28792 | context worker | | | 870 | 0.15455 | context worker | 0.16191 | context worker | | | 871 | 0.25835 | context worker | 0.32672 | context worker | | | 872 | 0.32314 | context worker | 0.31415 | context worker | | | 873 | 0.00008 | switchboard | 0.06655 | context worker | **rescue** | | 874 | 0.30618 | context worker | 0.18058 | context worker | | | 875 | 0.10390 | context worker | 0.20586 | context worker | | | 876 | 0.20125 | context worker | 0.30537 | context worker | | | 877 | 0.21909 | context worker | 0.23828 | context worker | | | 878 | 0.00757 | switchboard | 0.05345 | context worker | **rescue** | | 879 | 0.24069 | context worker | 0.27381 | context worker | | | 880 | 0.35534 | context worker | 0.34479 | context worker | | | 881 | 0.07280 | context worker | 0.10509 | context worker | | | 882 | 0.23871 | context worker | 0.24113 | context worker | | | 883 | 0.06976 | context worker | 0.01231 | switchboard | **harm** | | 884 | 0.23074 | context worker | 0.20481 | context worker | | | 885 | 0.01763 | switchboard | 0.11119 | context worker | **rescue** | | 886 | 0.02389 | switchboard | 0.17428 | context worker | **rescue** | | 887 | 0.25048 | context worker | 0.28333 | context worker | | | 888 | 0.15390 | context worker | 0.24058 | context worker | | | 889 | 0.24025 | context worker | 0.33816 | context worker | | | 890 | 0.05059 | context worker | 0.26321 | context worker | | | 891 | 0.13865 | context worker | 0.21745 | context worker | | | 892 | 0.18379 | context worker | 0.29248 | context worker | | | 893 | 0.00767 | switchboard | 0.06912 | context worker | **rescue** | | 894 | 0.00038 | switchboard | 0.28474 | context worker | **rescue** | | 895 | 0.10236 | context worker | 0.17191 | context worker | | | 896 | 0.00161 | switchboard | 0.08419 | context worker | **rescue** | ## What this does not show - **The margin is thin.** Plain seed 886 sat 0.0011 under the line (0.02389). Counted as a context worker, the result would be 6 rescues and 1 harm, p 0.0625, above the bar. Under six different shuffles it held its label (0.0221 to 0.0241); so did seed 885 (0.0176 on the scored shuffle). - **The count depends on the instrument.** A second instrument, the spread of the core's output across positions (its line set on earlier networks before any of these trained, no bar), calls 3 plain switchboards (873, 894, 896) and 0 sleeping ones. It agrees with the shuffle line on 75 of 80 cores. Its direction is the same, fewer switchboards in sleep; the size of the switchboard group is not. - **One bench, one length, one measure.** Tiny networks that see two words of context, trained at one length, read by one measure on a cut line. It says nothing about larger networks or longer context. - **Not a better network.** Held-out loss was not part of this bar. In earlier frozen tests on this bench, a lower end loss for the sleep form was not confirmed. - **Earlier seeds.** Pooled with 16 earlier seeds of the same bench (841 to 856, not in this release), 11 seeds rescued and 1 harmed. - **Scored positions.** 40 target token ids were never a training target and are left out of the score: 16,288 of the 16,384 held-out positions are scored. - **One step went off plan.** After training, the scored reader stopped on an error in a reporting line. It was re-run on the same trained networks with that one line corrected. No network was retrained, and the line, the bar and the count code did not change. ## Re-check it yourself Needs Python 3 with numpy, torch and safetensors. CPU only, one thread, about a minute. From this folder: python -I reader/read_shuffle_cost.py --brains brains --data data --expect results/per_brain_record.csv It prints only, nothing is written. It prints each network's shuffle cost and label, the two counts, the rescues and harms, the sign test and the bar, then compares every network with the record. Expected: 7 of 40 and 1 of 40, the same 8 discordant seeds, CONFIRMED, labels equal 80 of 80. The last digits of the costs can differ by machine and torch version. On the machine used to prepare this release the largest difference from the record was 7.25e-08 nats, and on that machine the reader matched the run's own network code bit for bit on all 80 networks (`results/reproduction_notes.txt`, raw output in `results/reproduction_output.txt`). To check the files first: `sha256sum -c SHA256SUMS.txt` (prints OK per file). ## Files - `brains/
_s_L10400_end.safetensors`: the 80 networks, one per form and seed, at the end of training. Tensor names keep the run's own prefixes: `tiny.` the routed base network, `rc.` the reasoning core, `cc.` the small integer part; `y` holds the 48 experts' smoothed sizes that the router's pull reads. Metadata: the run's labels (arm, charge, form, seed, steps, sleep_n) and the sha256 of the original checkpoint. One small two-output head of the integer part (`cc.Wv`, `cc.bv`), which the measure does not use, was removed from every file before release; every other tensor is byte for byte the trained one. `results/head_removed_record.csv` lists each file's hash with and without it, and the reader's results are the same either way (7 of 40 and 1 of 40, all 80 labels equal). - `reader/read_shuffle_cost.py`: the measure, forward pass only. - `tools/convert_pt_to_safetensors.py`: how each checkpoint became a .safetensors file, with its exactness check. - `data/heldout_windows_uint16.npy`, `data/excluded_target_ids.txt`, `data/NOTICE_data.txt`: the held-out positions and the 40 excluded target ids. - `results/`: the 40-seed table, a per-network record with both hashes, the conversion log, the seed list, the bars, the reader's raw output and the reproduction notes. - The training script is not released. ## Training data The networks trained on TinyStories by Ronen Eldan and Yuanzhi Li (2023), dataset `roneneldan/TinyStories`, the GPT-4 version: the training file TinyStoriesV2-GPT4-train.txt (sha256 6418d412de72888f52b5142c761ac21a582f7d1166f0bfbdb5f03ccfdec90443) for training and the validation file for the held-out windows. The dataset card describes it as synthetically generated short stories in a small vocabulary, licensed CDLA-Sharing-1.0. ## References - Ronen Eldan and Yuanzhi Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023), [arXiv:2305.07759](https://arxiv.org/abs/2305.07759). ## Citation If you use these networks or the measure, please cite: ```bibtex @misc{hampton2026sleepformrescue, author = {Hampton, Ronald Lee}, title = {Sleep-form rescue count: 80 tiny research networks}, year = {2026}, month = oct, howpublished = {Hugging Face: https://huggingface.co/RonBuildsAI/sleep-form-rescue-count}, note = {Measured 2026-10-26; released 2026-10-27} } ``` ## Licence and patents - Weights, this card, the results and the excluded-id list: CC BY 4.0 (`LICENSE_weights_CC-BY-4.0.txt`). That licence does not license patent rights. - Code: The Clear BSD License (`LICENSE_code_BSD-3-Clause-Clear.txt`), which grants no patent rights. - The held-out windows are a changed excerpt of the dataset above and stay under CDLA-Sharing-1.0 (`data/NOTICE_data.txt`). - Patent rights are reserved; provisional application filed 2026-10-27. Drafted and fact-checked with an AI assistant from my own records, measurements and decisions. I read and edited every sentence and I am responsible for its contents.