Upload docs/ClaudeReport_session_2026-09-16_yarn_scale_ceiling.md with huggingface_hub

docs/ClaudeReport_session_2026-09-16_yarn_scale_ceiling.md

@@ -0,0 +1,272 @@
+# 2026-09-16 - YaRN configuration is a large free lever; training still matters at the length you train for
+
+Workspace: Context_Parallelism (2x H100, fresh box)
+Files: p1_eval32k.sh, e_short.sh, e64k.sh, e128k.sh, ctl_106b.sh, dl_eval32k.sh,
+run_longeval.sh, mk_p1_plots.py
+
+> **READ THE CONTROL SECTION FIRST.** Sections up to "State" were written before
+> the 1.06B control had run, and two of their conclusions do not survive it:
+> "training adds nothing" and "the ceiling is purely configuration". Both are
+> corrected under "THE 1.06B CONTROL". The sections are kept as written rather
+> than silently edited, because the sequence of wrong turns is part of the record.
+
+## The question this started from
+
+Phase 1 (680M, 4k -> 16k, 1,200 steps) produced a null: base680 untrained scored
+84 @8192 / 40 @16360, p1final scored 77 / 42, both inside the +-9-13pp Wilson band
+at N=100. Four explanations were offered for why training did nothing. Three were
+checked and found wrong (see ERRORS). The fourth turned out to be the wrong
+question.
+
+## The finding
+
+Retrieval accuracy is governed by the YaRN CONFIGURATION, not by extension
+training. Sweeping YARN_SCALE with ORIG_MAXPOS held at the 4096 pretrain length,
+on the UNTRAINED base680, niah_single N=100, haystack byte-identical
+(199,754,694 B) to every prior evaluation in this project:
+
+    prompt      x4     x8    x16    x32
+     1,024      99     99      -      -
+     4,096      89     91     92     91
+     8,192      84     88      -      -
+    16,360      40     67     92      -
+    32,744       -     34     54     79
+    65,512       -      -     32     46
+
+Same weights throughout. 16,360 goes 40 -> 92 on a configuration change worth
++52pp. No short-context cost anywhere: 4,096 is 89 -> 92 across the same sweep.
+
+At its best config the UNTRAINED 680M beats the 1.06B's entire three-stage
+extension ladder:
+
+    prompt   680M best (no training)   1.06B best (3 trained stages)   delta
+    16,360        92  (x16)                   70  (mix64k @1200)        +22
+    32,744        79  (x32)                   50-52                     +27
+    65,512        46  (x32)                   31  (mix64k @1200)        +15
+
+## The model: two independent factors
+
+Arrange every measurement as a grid, rows = token count, columns = VIRTUAL
+POSITION (real position / scale = where content lands inside the 4,096 range the
+model was actually pretrained on):
+
+    tokens     v128   v256   v512  v1024  v2048  v4096
+      1024       99     99      .      .      .      .
+      4096       91     92     91     89      .      .
+      8192        .      .      .     88     84      .
+     16360        .      .      .     92     67     40
+     32744        .      .      .     79     54     34
+     65512        .      .      .      .      .     32
+
+Reading ACROSS a row isolates positional conditioning (same content, same
+distractors, only the scale changed). Reading DOWN a column isolates distractor
+load (same virtual position, more competing keys in the softmax). Both are real:
+
+  across, 16,360 tok:  v4096 -> 40,  v2048 -> 67,  v1024 -> 92
+  down,   v1024:       4k -> 89, 8k -> 88, 16k -> 92, 32k -> 79
+
+Practical rule: the best scale is the one landing content near virtual
+1,000-2,000, i.e. SCALE ~= prompt_len / 1500.
+
+## Why a 4k-pretrained model retrieves at 65k
+
+It does not, positionally. At x32 a 65,512-token prompt lands at virtual 2,047,
+so the model never sees a rotary angle outside its pretraining distribution.
+What is new is attending over 65,536 keys instead of 4,096 -- and attention is
+content-addressed, so a distinctive needle remains findable regardless of the
+token count, PROVIDED the positional encoding does not corrupt the match. That is
+why score tracks virtual position and largely ignores context length.
+
+(YaRN does not interpolate everything: dims below low=14 keep extrapolating, but
+those are the highest-frequency dims which already wrap many times inside 4k, so
+wrapping is their normal regime. Dims above high=32 interpolate and stay in range.)
+
+## Why training did not raise it
+
+1. At SCALE=4 a 16k prompt sits at virtual 4,090 -- the badly-conditioned edge.
+   That is a property of the RoPE cache, not of the weights. Phase 1 trained
+   inside the broken regime; gradient descent cannot repair a config choice.
+2. The objective gives almost no retrieval signal. 2026-09-02 measured flux p90
+   document length at 2,721 tokens: "there was never a gradient signal for
+   attending past a document boundary."
+3. The update was small regardless -- 629M tokens, 0.9 tok/param, against a
+   7.13B-token pretrain.
+
+## Reconciling with 2026-09-09 ("more steps DID increase retrieval")
+
+Not a contradiction. b3_long_run.sh confirms the 1.06B's 16k ladder trained at
+YARN_SCALE=4 YARN_ORIG_MAXPOS=4096 -- the identical config phase 1 used. On that
+one axis at 16,360:
+
+    1.06B untrained   0 steps    14-27
+    1.06B l1        400 steps       30
+    1.06B l2/l3     400 steps       35
+    1.06B l3long   1000 steps       36
+    1.06B l3long2k 2000 steps       43
+    680M  base680     0 steps       40
+    680M  p1final  1200 steps       42
+
+There is a ceiling at ~40-43 for scale-4 at 16k. The 1.06B started far below and
+climbed 2,000 steps to reach it; the 680M started AT it. Training moves a model
+toward its config's ceiling and never past it. Widening the config moves the
+ceiling itself (to ~92 at x16).
+
+Two models of 36 and 24 layers plateau at the same ~40-43, which matches the
+2026-09-09 peer finding that Qwen3 0.6B through 8B all score 100% with "zero
+variation" over a 13x parameter range: size is not the variable.
+
+## Mechanism: inferred, not measured
+
+The bend in the positional axis sits at virtual ~2,000-3,000. 2026-09-02
+independently measured reach saturating at 2,700-3,500 tokens, equal to flux p90
+document length 2,721. Two analyses months apart landing on the same number is
+suggestive that the ceiling is set by PRETRAINING POSITION OCCUPANCY -- positions
+rarely filled during pretraining are poorly conditioned.
+
+This is a correlation, not a measurement. Position occupancy in the corpus was
+never counted directly, and the 2,721 figure is the 1.06B's corpus: varybatch's
+pretraining data is unknown to me and assumed to have a similar shape.
+
+## Pre-registered predictions for x32, and what the miss taught
+
+Written into e128k.sh before the run:
+
+    4,096  -> 92-99   actual 91   near miss
+   32,744  -> 85-92   actual 79   MISS, too high
+   65,512  -> 45-55   actual 46   hit
+
+The two predictions derived from virtual position ALONE were wrong; the one that
+also reasoned about distractor load was right. The failure mode is what
+established the second factor -- the "token count washes out at low virtual
+position" claim held over 4k-16k tokens and broke at 32k.
+
+## Errors made this session
+
+- Named LEARNING RATE as the likely cause of the phase-1 null. CP_MAX_LR and
+  CP_MIN_LR appear ZERO times in bluscriptCP.cpp; max_lr is hardcoded 6e-4 at
+  line 151, so every run on both models trained at 6e-4 x REWARMUP_PEAK 0.3 =
+  1.8e-4. ext64k_b5.sh:23 already documented this as a no-op. THE SAME ERROR IS
+  IN MY OWN 2026-09-09 LOG LINE.
+- Claimed phase 1 saw too few long tokens: it saw 314,572,800 b3 tokens against
+  mix64k's 188,743,680, i.e. 1.67x MORE.
+- Worried b3 documents were truncated by the 16k window. Backwards: b3 >= window
+  means every window is contiguous single-document text, the good case.
+- Compared phase 1 against ext64k (three trained stages) instead of the
+  stage analogue, then built a correction table across eval dirs using DIFFERENT
+  haystacks -- checkpoint run2_step_400 scores 78 under lenb (b3val) and 62 under
+  decide (b2val). Only mix64k and mixcont share phase 1's haystack for certain.
+- Presented a both-models comparison showing only the base column, so it looked
+  like p1final had never been evaluated at the wider caches. It always was.
+- Ran make with no target in Tensor-Implementations and in Profiler (which has no
+  Makefile), then printed BUILD_EXIT=0 from a trailing ls rather than from make.
+- set -u: `local A=$1 B="${A}"` on one line is unbound, bash creates all locals
+  before assigning.
+
+## State
+
+Reproduction check pending in ctl_106b.sh: mix64k_step1200 at 16,360 under its
+as-trained ORIG=32768 SCALE=2 must return ~70. If it does not, the harness
+differs from the prior session and every cross-session number here is void.
+
+## THE 1.06B CONTROL (ctl_106b.sh) -- reverses two conclusions above
+
+Reproduction check FIRST: mix64k_step1200 at 16,360 under its as-trained
+ORIG=32768 SCALE=2 returned 70.0, exactly the prior session's 70. The harness is
+sound and every cross-session number in this report is comparable.
+
+1.06B under its own config vs the corrected parameterisation (ORIG held at 4096):
+
+    prompt      model      own cfg  corrected   gain
+    16,360    mix1200           70         78     +8
+    16,360  cliff1400           54         66    +12
+    32,744    mix1200           50         73    +23
+    32,744  cliff1400           31         70    +39
+    65,512    mix1200           31         65    +34
+    65,512  cliff1400           18         65    +47
+
+### Reversal 1: long-context training DOES buy capability, at the trained length
+
+    prompt   680M base (no long-ctx training)   1.06B mix1200 (trained to 64k)
+    16,360                 92                                78
+    32,744                 79                                73
+    65,512                 46                                65   <-- 1.06B +19
+
+The 680M wins at 16k and 32k. At 65,512 the 1.06B wins by 19pp -- exactly the
+length it was trained at. I predicted 40-50 there from the 680M's curve and the
+actual was 65. That miss is the clearest evidence in this whole session that
+training installs long-context ability a base model does not have.
+
+The earlier "training adds nothing" claim was an artefact of three compounding
+mistakes: only ever training at the WRONG scale, on a model already sitting at
+that config's ceiling, and evaluating below the length where training pays.
+
+### Reversal 2: the collapse was substantially a configuration artefact
+
+    collapse gap (healthy minus collapsed)
+    prompt     own cfg    corrected
+    16,360       +16pp        +12pp
+    32,744       +19pp         +3pp
+    65,512       +13pp         +0pp
+
+cliff1400 recovers 18 -> 65 at 65,512 and becomes IDENTICAL to the healthy
+checkpoint. Days were spent diagnosing that collapse -- the copying analysis, the
+per-position NLL dumps, the mix redesign -- and most of the lost retrieval returns
+with a better RoPE parameterisation. Residual damage remains at 16k (12pp), so it
+was not purely artefact, but the headline collapse was largely the model failing
+under a badly-conditioned config rather than destroyed retrieval.
+
+### Control predictions: 1 hit, 2 misses, both misses the same direction
+
+    16,360 @x16  predicted 85-92  actual 78  MISS
+    32,744 @x32  predicted 70-80  actual 73  HIT
+    65,512 @x32  predicted 40-50  actual 65  MISS
+
+Both misses under-weighted what training does. Having spent the session
+correcting an over-claim ABOUT training, I over-corrected into dismissing it.
+
+## What the literature says (searched 2026-09-16)
+
+- YaRN (arxiv 2309.00071): Dynamic-YaRN gives >2x extension WITHOUT fine-tuning;
+  fine-tuned YaRN reaches SOTA on ~400 steps and <0.1% of pretraining data.
+- Llama 3 (arxiv 2407.21783): does NOT use position interpolation. Raises RoPE
+  base frequency to 500,000 and TRAINS DIRECTLY at each target length, six stages
+  8K -> 128K on ~800B tokens. Train config == eval config at every stage; there is
+  no train/eval mismatch. NIAH is a GATE to advance, passed only alongside full
+  short-context recovery.
+- Qwen2.5 (arxiv 2412.15115): 4,096 -> 32,768; Turbo uses four stages to 262,144
+  with 40% of sequences at the current max length and 60% shorter.
+- RULER (arxiv 2404.06654): vanilla NIAH is "indicative of only a superficial
+  form of long-context understanding"; models near-perfect on NIAH still drop
+  sharply on multi-hop and aggregation.
+
+So the mismatch exploited in this session is NOT a frontier technique. It was an
+inconsistency in my own experiment -- phase 1 trained at x4, evaluated at x8/16/32
+-- that I then discovered and reported as a finding. For base680 there is no
+mismatch (it never trained under any YaRN cache, so scale selection is legitimate
+config choice); for p1final there is, and its x16/x32 numbers are diagnostic only.
+
+Our models already use rope_theta 500,000, the value Llama 3 adopted.
+
+## Revised conclusions
+
+1. YaRN scale is a large free lever, worth up to +47pp, and every run in this
+   project left it on the table.
+2. Training is NOT useless. It installs capability at the length trained, worth
+   +19pp at 64k over a better-pretrained base that never saw long context.
+3. The 680M's pretrain advantage is real at <=32k and gone at 64k.
+4. The original collapse was over-attributed to training damage.
+5. Everything here is niah_single, which RULER and our own 2026-09-09 peer sweep
+   both show is the WEAK benchmark: Qwen3 0.6B-8B all score 100% with zero
+   variation across 13x params, while val_backward separates them and ours is at
+   chance. The correct next measurement is val_backward / multi-hop, not more NIAH.
+
+## Open
+
+- Never tested: TRAINING AT THE GOOD SCALE. Phase 1 trained at x4 only, so
+  "training adds nothing" was never established at x16/x32. This is the missing
+  cell and the obvious next run.
+- val_backward and multi-hop under the corrected config, for all checkpoints.
+- Whether the 680M trained at 64k would beat the 1.06B's 65 there.
+- Position occupancy in varybatch's pretraining corpus, to test the
+  document-length mechanism directly rather than by correlation.
+- Qwen3-0.6B scores 100% to 32,736. The gap to peers is unclosed.