Upload docs/logline_2026-09-16.txt with huggingface_hub

docs/logline_2026-09-16.txt

@@ -0,0 +1 @@
+2026-09-16 - Re-ran the 680M phase-1 question on a fresh 2x H100 and found the whole extension programme has been optimising the wrong variable: retrieval accuracy is set by the YaRN CONFIGURATION, not by extension training. Swept YARN_SCALE 4/8/16/32 (ORIG held at the 4096 pretrain length) on the UNTRAINED base680 and on p1final, N=100 niah_single, byte-identical haystack (199,754,694 B) to every prior eval. Same weights at 16,360 score 40 -> 67 -> 92 as scale goes 4 -> 16; at 32,744 34 -> 54 -> 79 and at 65,512 32 -> 46. The untrained 680M at its best config beats the 1.06B's entire three-stage ladder at every length: 92 vs 70 @16k, 79 vs 50 @32k, 46 vs 31 @64k, with ZERO long-context training. Built a two-factor model from the full grid: score depends on (a) VIRTUAL POSITION = real_pos/scale, i.e. where content lands inside the 4,096 range the model was pretrained on, and (b) distractor load = token count; reading across a row (same content, only scale changed) isolates (a), reading down a column isolates (b). This reconciles the apparent contradiction with 2026-09-09, where l3long gained +7pp from doubling steps 1000->2000: training moves a model TOWARD its config's ceiling but never past it - the 1.06B climbed 30/35/36/43 to the scale-4 16k ceiling of ~43 while the 680M started at 40 and 1200 steps moved it to 42. Both models, 36 and 24 layers, plateau at the same ~40-43, consistent with the 2026-09-09 peer result that 0.6B-8B Qwen3 all score 100% with "zero variation" across 13x params. The ceiling bend point (virtual ~2,000-3,000) matches the independently measured 2026-09-02 finding that reach saturates at 2,700-3,500 tokens ~= flux p90 document length 2,721, so the mechanism is most likely pretraining position occupancy - INFERRED from correlation, not measured, and that p90 is the 1.06B's corpus since varybatch's pretraining data is unknown to me. Pre-registered three predictions before the x32 run and got 1 hit / 1 near / 1 miss: the two derived from virtual position alone were wrong (32,744 predicted 85-92, actual 79) and the one that also accounted for distractor load was right (65,512 predicted 45-55, actual 46) - the failure mode is what established the second factor. Own errors: I named LEARNING RATE as the likely cause of the phase-1 null when CP_MAX_LR/CP_MIN_LR appear ZERO times in bluscriptCP.cpp (max_lr hardcoded 6e-4:151, so every run on both models trained at 1.8e-4) - the SAME error I logged on 2026-09-09; claimed phase 1 saw too few long tokens when it saw 1.67x MORE b3 than mix64k; worried about b3 truncation when b3 >= window is the good case; compared phase 1 against a 3-stage model instead of the stage analogue; built a comparison table across eval dirs with DIFFERENT haystacks (same ckpt run2_step_400 scores 78 under lenb/b3val and 62 under decide/b2val); presented a both-models table showing only the base column so it looked like p1final was never evaluated; and ran make with no target in Tensor-Implementations then reported BUILD_EXIT=0 from a trailing ls - Context_Parallelism / p1_eval32k.sh, e_short.sh, e64k.sh, e128k.sh, ctl_106b.sh, dl_eval32k.sh, run_longeval.sh