Card: the higher floor at 48 records a row (2026-10-28)
Browse files
README.md
CHANGED
|
@@ -813,6 +813,44 @@ Each model's run record is `bed/reports/bed_<model>.json`, its look `bed/reports
|
|
| 813 |
[AbstractEyes/geolip-nano-beatrix-trainer](https://github.com/AbstractEyes/geolip-nano-beatrix-trainer) 0.1.100, run by its notebooks
|
| 814 |
`colab/nano_slow_anneal_lane{1,2,3}.ipynb`.
|
| 815 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 816 |
## Caveats
|
| 817 |
|
| 818 |
- Small synthetic beds: a 4-layer width-128 core, full attention at context 256, 64 value tokens, at most 512 compartments.
|
|
|
|
| 813 |
[AbstractEyes/geolip-nano-beatrix-trainer](https://github.com/AbstractEyes/geolip-nano-beatrix-trainer) 0.1.100, run by its notebooks
|
| 814 |
`colab/nano_slow_anneal_lane{1,2,3}.ipynb`.
|
| 815 |
|
| 816 |
+
## The higher floor at 48 records a row (2026-10-28)
|
| 817 |
+
|
| 818 |
+
The same higher floor (the cosine from 1e-3 ending at half the peak, `--lr_floor=0.5`) on the four-digit address at 48 records a row,
|
| 819 |
+
whose index had not formed by 24,000 steps on four seeds of six: the six stored seeds retrained for the stored 24,000 steps, on the same
|
| 820 |
+
rows at the same steps, each beside the stored model of its seed (the names carry `_a48sa`):
|
| 821 |
+
|
| 822 |
+
| seed | the index at 24,000 steps before | held-out bits a byte: stored / this | written index at 24,000 steps: stored / this | first step the index read >= .99: stored / this | look at the question |
|
| 823 |
+
|---|---|---|---|---|---|
|
| 824 |
+
| 1 | on the fourth digit | 1.4073 / 1.4025 | 0.7938 / 0.8088 | never / never | 1.0000 |
|
| 825 |
+
| 2 | on the fourth digit | 1.4029 / 1.4037 | 0.8160 / 0.8070 | never / never | 1.0000 |
|
| 826 |
+
| 3 | near the first digit | 1.4156 / 1.4126 | 0.1108 / 0.9998 | never / 14,000 | 1.0000 |
|
| 827 |
+
| 5 | in the second digit | 1.4027 / 1.3994 | 0.2952 / 0.4903 | never / never | 1.0000 |
|
| 828 |
+
| 4 | formed | 1.4131 / 1.4117 | 0.9989 / 0.7515 | 18,000 / never | 1.0000 |
|
| 829 |
+
| 6 | formed | 1.3970 / 1.3998 | 1.0000 / 0.9999 | 4,000 / 8,000 | 1.0000 |
|
| 830 |
+
|
| 831 |
+
What the numbers say:
|
| 832 |
+
|
| 833 |
+
- **Here the higher floor formed one of the four.** Seed 3 formed at 14,000 steps (the step at which the floor formed the 96-record
|
| 834 |
+
seed above); seeds 1 and 2 stayed where they were (0.8088 / 0.8070); seed 5 rose from 0.2952 to 0.4903 without forming.
|
| 835 |
+
- **A seed whose index had formed stayed on the plateau.** Seed 4, whose stored model formed its index at 18,000 steps, reached the
|
| 836 |
+
same plateau at 8,000 steps under the higher floor and stayed on it to the end (0.7515); its stored model had climbed off it between
|
| 837 |
+
12,000 and 18,000 steps, while that schedule's rate fell from about half the peak to about a quarter. Seed 6 formed again, at 8,000
|
| 838 |
+
steps instead of 4,000.
|
| 839 |
+
- **Where each index stops, digit by digit.** Seeds 1, 2 and 4 stop at the fourth digit: the first three right on every entry (seed 4:
|
| 840 |
+
the third on 0.9831), the fourth missed on one entry in four or five (0.8164 / 0.8125 / 0.7552 right), almost always naming another record
|
| 841 |
+
of the row, usually the next or the previous slot - the stop of the stored models of seeds 1 and 2. Seed 5 now writes the first two
|
| 842 |
+
digits right on every entry (its stored model missed the second on 22 % of them) and the third on 0.6628 of them.
|
| 843 |
+
- **Why 48 records a row is harder to leave.** With 48 records among 256 addresses, more than half the entries have no other record
|
| 844 |
+
under their first three digits, so a model that knows three digits and picks the fourth among the few records sharing them is right on
|
| 845 |
+
about three entries in four by chance (.7574; at 96 records a row .5687). A seed that reached three digits has little left to gain;
|
| 846 |
+
the seeds the floor moved were the ones stopped earlier.
|
| 847 |
+
- **The text and the lookups held.** The held-out text -0.0048 to +0.0027 bits a byte against the stored models; every look exact (the look at the
|
| 848 |
+
question and on the hard-key rows 1.0000 on all six); the answer 0.9970-0.9989 at the true request and 0 at the null.
|
| 849 |
+
|
| 850 |
+
The next stage trains the same recipe for 48,000 steps on the six seeds (the names carry `_a48long`), the length that formed every
|
| 851 |
+
stalled index at 96 records a row and on the plain-rows recipe; the notebooks are `colab/nano_a48long_lane{1,2,3}.ipynb` in
|
| 852 |
+
[AbstractEyes/geolip-nano-beatrix-trainer](https://github.com/AbstractEyes/geolip-nano-beatrix-trainer) (0.1.100, as the stage above).
|
| 853 |
+
|
| 854 |
## Caveats
|
| 855 |
|
| 856 |
- Small synthetic beds: a 4-layer width-128 core, full attention at context 256, 64 value tokens, at most 512 compartments.
|