Upload vault/Reports/ClaudeReport_session_2026-09-19_adamw_const_ablation.md with huggingface_hub

vault/Reports/ClaudeReport_session_2026-09-19_adamw_const_ablation.md

@@ -0,0 +1,80 @@
+# 2026-09-19 - AdamW-only control arm (8e-5 constant), cut at step 400
+
+Workspace: Context_Parallelism (1x H200)
+Files: abl_phase1.sh, go_adamw.sh, cut400.sh, mk_lineage_680m.py, teardown_ship.sh
+
+## Purpose
+
+The 2026-09-18 ablation changed two variables at once: optimizer (Muon vs AdamW)
+and LR (8e-5 vs 1.8e-4). This run holds LR fixed at 8e-5 constant and runs AdamW
+only, so the optimizer is the single free variable.
+
+## Build blocker resolved first
+
+libtensor would not compile on this box. 46 instances of one error, zero .cu
+files compiled:
+
+    mathcalls.h(206): error: exception specification is incompatible with that
+    of previous function "rsqrt" (crt/math_functions.h:629)
+
+Cause: this box ships glibc 2.43 (kernel 7.0.0-31, gcc 15.2). glibc 2.41+ adds
+C23 rsqrt/rsqrtf declared with __THROW; CUDA 13.0 (Aug 2025) declares the same
+names without an exception specification. Every CUDA translation unit failed.
+Earlier boxes had older glibc, so this never appeared before.
+
+The tail of build.log was misleading: it showed a "gathered_idx < 0" line from
+IndexingOps.cu, which is only warning #186-D. The "2 errors" in that file were
+its own rsqrt/rsqrtf pair.
+
+No clean off-switch exists - the feature macro is driven by __USE_GNU and is
+#undef'd first, so -D on the command line is wiped. Fix applied: guarded glibc's
+single declaration in /usr/include/x86_64-linux-gnu/bits/mathcalls.h:206 behind
+#ifndef __CUDACC__. Verified a .cu and a .cpp including cuda_runtime.h both
+compile. libtensor.a (80 MB) and bluscriptCP_exec (28.3 MB) built with 0 errors.
+
+NOTE: this patch lives in /usr/include on an ephemeral box. It dies with the
+instance and must be reapplied on any box with glibc >= 2.41.
+
+## Run configuration, verified not assumed
+
+- [ZeRO-1] optimizer=adamw   (not MUON+adamw-aux)
+- CP_OPTIMIZER absent from the launch env (0 occurrences) - negative control
+- mix b3=320011377 tokens, token-identical to p1mix
+- lr 8.0000e-05 flat after 12 warmup steps (1% of 1200)
+- all four arms start at loss 2.731203 from parent varybatch
+- 43k tok/s, 78.5 of 143.8 GB, grad norm 0.29-0.37 throughout
+
+Cut at step 399 (the 400th step, 0-indexed) after the step-400 checkpoint was
+confirmed written safely. GPU released to 0 MiB.
+
+## Result: matched-step 50-step rolling mean
+
+    arm                          99       199       299       399
+    p1final Muon 1.8e-4      2.5969    2.6673    2.6668    2.6299
+    const   Muon 8e-5        2.5593    2.6083    2.6037    2.5670
+    cosine  Muon 8e-5        2.5593    2.6080    2.6028    2.5655
+    const   AdamW 8e-5       2.5825    2.6317    2.6283    2.5923
+
+At identical LR, identical data order and identical parent, AdamW is worse than
+Muon by +0.0254 nats at step 399. The gap is stable across all four sample
+points (+0.023, +0.023, +0.025, +0.025), so it is not noise.
+
+The two variables the 09-18 ablation confounded now separate cleanly:
+- LR 1.8e-4 -> 8e-5 (Muon held): -0.063 nats  (helps, the larger effect)
+- Muon -> AdamW (LR held at 8e-5): +0.025 nats (hurts)
+
+## Caveat
+
+This is training loss, not retrieval. The 09-18 result was that LR schedule
+shape is a null on retrieval (+1/+1/+2pp). A 0.025 nat training-loss gap is
+small and may not move retrieval at all. No eval was run on this arm - it was
+cut at 400 of 1200 steps, so there is no comparable end-of-phase checkpoint.
+
+## Artifacts on HF (unparallelled/BluScriptCP), all size-verified
+
+- checkpoints/adamw_const_8e5_step_400.ckpt and .zero_g1
+- checkpoints/adamw_const_8e5_step_200.ckpt and .zero_g1
+- logs/2026-09-19_adamw_const_{train,probe,master}.log
+- logs/2026-09-19_libtensor_build_glibc243.log
+- plots/2026-09-19_lineage_680m_phase1.png
+- scripts/{mk_lineage_680m.py,abl_phase1.sh,go_adamw.sh,cut400.sh}