AbstractPhil commited on
Commit
56a5dc9
·
verified ·
1 Parent(s): 8564f76

Card: the higher floor at 48 records a row (2026-10-28)

Browse files
Files changed (1) hide show
  1. README.md +38 -0
README.md CHANGED
@@ -813,6 +813,44 @@ Each model's run record is `bed/reports/bed_<model>.json`, its look `bed/reports
813
  [AbstractEyes/geolip-nano-beatrix-trainer](https://github.com/AbstractEyes/geolip-nano-beatrix-trainer) 0.1.100, run by its notebooks
814
  `colab/nano_slow_anneal_lane{1,2,3}.ipynb`.
815
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
816
  ## Caveats
817
 
818
  - Small synthetic beds: a 4-layer width-128 core, full attention at context 256, 64 value tokens, at most 512 compartments.
 
813
  [AbstractEyes/geolip-nano-beatrix-trainer](https://github.com/AbstractEyes/geolip-nano-beatrix-trainer) 0.1.100, run by its notebooks
814
  `colab/nano_slow_anneal_lane{1,2,3}.ipynb`.
815
 
816
+ ## The higher floor at 48 records a row (2026-10-28)
817
+
818
+ The same higher floor (the cosine from 1e-3 ending at half the peak, `--lr_floor=0.5`) on the four-digit address at 48 records a row,
819
+ whose index had not formed by 24,000 steps on four seeds of six: the six stored seeds retrained for the stored 24,000 steps, on the same
820
+ rows at the same steps, each beside the stored model of its seed (the names carry `_a48sa`):
821
+
822
+ | seed | the index at 24,000 steps before | held-out bits a byte: stored / this | written index at 24,000 steps: stored / this | first step the index read >= .99: stored / this | look at the question |
823
+ |---|---|---|---|---|---|
824
+ | 1 | on the fourth digit | 1.4073 / 1.4025 | 0.7938 / 0.8088 | never / never | 1.0000 |
825
+ | 2 | on the fourth digit | 1.4029 / 1.4037 | 0.8160 / 0.8070 | never / never | 1.0000 |
826
+ | 3 | near the first digit | 1.4156 / 1.4126 | 0.1108 / 0.9998 | never / 14,000 | 1.0000 |
827
+ | 5 | in the second digit | 1.4027 / 1.3994 | 0.2952 / 0.4903 | never / never | 1.0000 |
828
+ | 4 | formed | 1.4131 / 1.4117 | 0.9989 / 0.7515 | 18,000 / never | 1.0000 |
829
+ | 6 | formed | 1.3970 / 1.3998 | 1.0000 / 0.9999 | 4,000 / 8,000 | 1.0000 |
830
+
831
+ What the numbers say:
832
+
833
+ - **Here the higher floor formed one of the four.** Seed 3 formed at 14,000 steps (the step at which the floor formed the 96-record
834
+ seed above); seeds 1 and 2 stayed where they were (0.8088 / 0.8070); seed 5 rose from 0.2952 to 0.4903 without forming.
835
+ - **A seed whose index had formed stayed on the plateau.** Seed 4, whose stored model formed its index at 18,000 steps, reached the
836
+ same plateau at 8,000 steps under the higher floor and stayed on it to the end (0.7515); its stored model had climbed off it between
837
+ 12,000 and 18,000 steps, while that schedule's rate fell from about half the peak to about a quarter. Seed 6 formed again, at 8,000
838
+ steps instead of 4,000.
839
+ - **Where each index stops, digit by digit.** Seeds 1, 2 and 4 stop at the fourth digit: the first three right on every entry (seed 4:
840
+ the third on 0.9831), the fourth missed on one entry in four or five (0.8164 / 0.8125 / 0.7552 right), almost always naming another record
841
+ of the row, usually the next or the previous slot - the stop of the stored models of seeds 1 and 2. Seed 5 now writes the first two
842
+ digits right on every entry (its stored model missed the second on 22 % of them) and the third on 0.6628 of them.
843
+ - **Why 48 records a row is harder to leave.** With 48 records among 256 addresses, more than half the entries have no other record
844
+ under their first three digits, so a model that knows three digits and picks the fourth among the few records sharing them is right on
845
+ about three entries in four by chance (.7574; at 96 records a row .5687). A seed that reached three digits has little left to gain;
846
+ the seeds the floor moved were the ones stopped earlier.
847
+ - **The text and the lookups held.** The held-out text -0.0048 to +0.0027 bits a byte against the stored models; every look exact (the look at the
848
+ question and on the hard-key rows 1.0000 on all six); the answer 0.9970-0.9989 at the true request and 0 at the null.
849
+
850
+ The next stage trains the same recipe for 48,000 steps on the six seeds (the names carry `_a48long`), the length that formed every
851
+ stalled index at 96 records a row and on the plain-rows recipe; the notebooks are `colab/nano_a48long_lane{1,2,3}.ipynb` in
852
+ [AbstractEyes/geolip-nano-beatrix-trainer](https://github.com/AbstractEyes/geolip-nano-beatrix-trainer) (0.1.100, as the stage above).
853
+
854
  ## Caveats
855
 
856
  - Small synthetic beds: a 4-layer width-128 core, full attention at context 256, 64 value tokens, at most 512 compartments.