Upload docs/logline_2026-09-16.txt with huggingface_hub

docs/logline_2026-09-16.txt

@@ -1 +1 @@
-2026-09-16 - Re-ran the 680M phase-1 question on a fresh 2x H100 and found the whole extension programme has been optimising the wrong variable: retrieval accuracy is set by the YaRN CONFIGURATION, not by extension training. Swept YARN_SCALE 4/8/16/32 (ORIG held at the 4096 pretrain length) on the UNTRAINED base680 and on p1final, N=100 niah_single, byte-identical haystack (199,754,694 B) to every prior eval. Same weights at 16,360 score 40 -> 67 -> 92 as scale goes 4 -> 16; at 32,744 34 -> 54 -> 79 and at 65,512 32 -> 46. The untrained 680M at its best config beats the 1.06B's entire three-stage ladder at every length: 92 vs 70 @16k, 79 vs 50 @32k, 46 vs 31 @64k, with ZERO long-context training. Built a two-factor model from the full grid: score depends on (a) VIRTUAL POSITION = real_pos/scale, i.e. where content lands inside the 4,096 range the model was pretrained on, and (b) distractor load = token count; reading across a row (same content, only scale changed) isolates (a), reading down a column isolates (b). This reconciles the apparent contradiction with 2026-09-09, where l3long gained +7pp from doubling steps 1000->2000: training moves a model TOWARD its config's ceiling but never past it - the 1.06B climbed 30/35/36/43 to the scale-4 16k ceiling of ~43 while the 680M started at 40 and 1200 steps moved it to 42. Both models, 36 and 24 layers, plateau at the same ~40-43, consistent with the 2026-09-09 peer result that 0.6B-8B Qwen3 all score 100% with "zero variation" across 13x params. The ceiling bend point (virtual ~2,000-3,000) matches the independently measured 2026-09-02 finding that reach saturates at 2,700-3,500 tokens ~= flux p90 document length 2,721, so the mechanism is most likely pretraining position occupancy - INFERRED from correlation, not measured, and that p90 is the 1.06B's corpus since varybatch's pretraining data is unknown to me. Pre-registered three predictions before the x32 run and got 1 hit / 1 near / 1 miss: the two derived from virtual position alone were wrong (32,744 predicted 85-92, actual 79) and the one that also accounted for distractor load was right (65,512 predicted 45-55, actual 46) - the failure mode is what established the second factor. Own errors: I named LEARNING RATE as the likely cause of the phase-1 null when CP_MAX_LR/CP_MIN_LR appear ZERO times in bluscriptCP.cpp (max_lr hardcoded 6e-4:151, so every run on both models trained at 1.8e-4) - the SAME error I logged on 2026-09-09; claimed phase 1 saw too few long tokens when it saw 1.67x MORE b3 than mix64k; worried about b3 truncation when b3 >= window is the good case; compared phase 1 against a 3-stage model instead of the stage analogue; built a comparison table across eval dirs with DIFFERENT haystacks (same ckpt run2_step_400 scores 78 under lenb/b3val and 62 under decide/b2val); presented a both-models table showing only the base column so it looked like p1final was never evaluated; and ran make with no target in Tensor-Implementations then reported BUILD_EXIT=0 from a trailing ls - Context_Parallelism / p1_eval32k.sh, e_short.sh, e64k.sh, e128k.sh, ctl_106b.sh, dl_eval32k.sh, run_longeval.sh
+2026-09-16 - Re-ran the 680M phase-1 question on a fresh 2x H100 and found YaRN CONFIGURATION is a large free lever the project had left on the table, but a 1.06B control then REVERSED two of my conclusions. Swept YARN_SCALE 4/8/16/32 (ORIG held at the 4096 pretrain length) on the UNTRAINED base680 and on p1final, N=100 niah_single, byte-identical haystack (199,754,694 B) to every prior eval. Same weights at 16,360 score 40 -> 67 -> 92 as scale goes 4 -> 16; at 32,744 34 -> 54 -> 79 and at 65,512 32 -> 46. The untrained 680M at its best config beats the 1.06B's entire three-stage ladder at every length: 92 vs 70 @16k, 79 vs 50 @32k, 46 vs 31 @64k, with ZERO long-context training. Built a two-factor model from the full grid: score depends on (a) VIRTUAL POSITION = real_pos/scale, i.e. where content lands inside the 4,096 range the model was pretrained on, and (b) distractor load = token count; reading across a row (same content, only scale changed) isolates (a), reading down a column isolates (b). This reconciles the apparent contradiction with 2026-09-09, where l3long gained +7pp from doubling steps 1000->2000: training moves a model TOWARD its config's ceiling but never past it - the 1.06B climbed 30/35/36/43 to the scale-4 16k ceiling of ~43 while the 680M started at 40 and 1200 steps moved it to 42. Both models, 36 and 24 layers, plateau at the same ~40-43, consistent with the 2026-09-09 peer result that 0.6B-8B Qwen3 all score 100% with "zero variation" across 13x params. The ceiling bend point (virtual ~2,000-3,000) matches the independently measured 2026-09-02 finding that reach saturates at 2,700-3,500 tokens ~= flux p90 document length 2,721, so the mechanism is most likely pretraining position occupancy - INFERRED from correlation, not measured, and that p90 is the 1.06B's corpus since varybatch's pretraining data is unknown to me. Pre-registered three predictions before the x32 run and got 1 hit / 1 near / 1 miss: the two derived from virtual position alone were wrong (32,744 predicted 85-92, actual 79) and the one that also accounted for distractor load was right (65,512 predicted 45-55, actual 46) - the failure mode is what established the second factor. Own errors: I named LEARNING RATE as the likely cause of the phase-1 null when CP_MAX_LR/CP_MIN_LR appear ZERO times in bluscriptCP.cpp (max_lr hardcoded 6e-4:151, so every run on both models trained at 1.8e-4) - the SAME error I logged on 2026-09-09; claimed phase 1 saw too few long tokens when it saw 1.67x MORE b3 than mix64k; worried about b3 truncation when b3 >= window is the good case; compared phase 1 against a 3-stage model instead of the stage analogue; built a comparison table across eval dirs with DIFFERENT haystacks (same ckpt run2_step_400 scores 78 under lenb/b3val and 62 under decide/b2val); presented a both-models table showing only the base column so it looked like p1final was never evaluated; and ran make with no target in Tensor-Implementations then reported BUILD_EXIT=0 from a trailing ls. THE 1.06B CONTROL (run after the above): reproduction check EXACT (mix64k_step1200 at 16,360 under its as-trained ORIG=32768 SCALE=2 returned 70.0 vs the prior session's 70, so the harness is sound). Under the corrected parameterisation the 1.06B gains +8/+23/+34pp at 16k/32k/64k and the COLLAPSED cliff1400 gains +12/+39/+47, recovering 18 -> 65 at 65,512 and becoming IDENTICAL to the healthy checkpoint (collapse gap +13pp -> 0pp), so most of the collapse we spent days diagnosing via copying analysis and per-position NLL was a CONFIG ARTEFACT, not destroyed retrieval (residual 12pp damage remains at 16k). REVERSAL: at 65,512 the 1.06B scores 65 against the 680M base's 46 - training wins by 19pp at exactly the length it was trained - so 'training adds nothing' was an artefact of only ever training at the WRONG scale, on a model already at that config's ceiling, evaluated below the length where training pays. I pre-registered 3 control predictions and went 1 hit / 2 misses with BOTH misses under-weighting training: having spent the session correcting an over-claim about training I over-corrected into dismissing it. Literature check: Llama 3 (2407.21783) does NOT use position interpolation - it raises rope_theta to 500,000 (the value we already use) and TRAINS DIRECTLY at each target length, 8K->128K in six stages on ~800B tokens, train config == eval config throughout, with NIAH as a GATE passed only alongside full short-context recovery; YaRN (2309.00071) claims only >2x extension without fine-tuning; RULER (2404.06654) calls vanilla NIAH 'indicative of only a superficial form of long-context understanding'. So the train/eval cache mismatch I exploited is NOT a frontier technique - it was an inconsistency in my own experiment (phase 1 trained x4, evaluated x8/16/32) that I discovered and reported as a finding. Everything here is niah_single, the benchmark our own 2026-09-09 peer sweep already showed is non-discriminating (Qwen3 0.6B-8B all 100%, zero variation over 13x params, while val_backward separates them and ours is at chance). NEVER TESTED: training AT the good scale - Context_Parallelism / p1_eval32k.sh, e_short.sh, e64k.sh, e128k.sh, ctl_106b.sh, dl_eval32k.sh, run_longeval.sh