ship logs/small_a41_cbk_gf_wa.log (2026-08-22 4090 queue)

logs/small_a41_cbk_gf_wa.log

@@ -0,0 +1,77 @@
+[alephlm0] disk: 140.8 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a4_splat seed_idx=0 (crc seed 155979836) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a4_splat-cbk-gf-wa-s0/
+[alephlm0] splat geometry recorded: {'M': 128, 'K': 8, 'overlap': 1.5, 'tau': 0.1, 'mchunk': '16'}
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a4_splat-cbk-gf-wa-s0
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A4_SPLAT-CBK-GF-WA-S0 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] -> AbstractPhil/alephlm-0 (private)
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a4_splat: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] a4_splat: M=128 K=8 overlap=1.5 | TRAINABLE 61,479,204 of 61,479,204 total — attention trainable capacity HALVED vs MHA (-525,312/block; frozen fabric is not capacity). A capability LOSS vs a1 is capacity-confounded; a MATCH at -10.8%% trainable is the strong result.
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a4_splat-cbk-gf-wa-s0: 61,479,204 params (0.56x bert-base)
+    12L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 2 = 488 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.8 of 25.8 GB (co-resident/desktop holds 1.9 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 16.5 / reserved 17.4 GB vs 23.8 free -> margin 6.5 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 2 = 488 steps
+    batch plan: 244 batches/epoch | padded length p50 33 p90 117 max 128 (bucketing=True)
+    e1      50/488 loss 7.2830 nce 7.2806 mse 0.00243 acc 0.002 lr 3.03e-04 L64 7m
+    e1     100/488 loss 6.8378 nce 6.8356 mse 0.00221 acc 0.003 lr 6.00e-04 L64 15m
+    aleph: amp mean 0.00542 (b0 0.00511 .. b11 0.00490) | drift mean 0.13052 | gates σ b0 [0.0475, 0.0475, 0.0473]
+      VAL r1 0.0015 cos 0.1541 self_cos +0.1642 erank 2.3 cv 0.8873 | frame r1 0.0050
+    [backup] pushed (step 134) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0/
+    e1     150/488 loss 6.5317 nce 6.5295 mse 0.00212 acc 0.011 lr 5.76e-04 L128 23m
+    e1     200/488 loss 5.4631 nce 5.4614 mse 0.00168 acc 0.031 lr 5.07e-04 L64 30m
+    aleph: amp mean 0.00577 (b0 0.00522 .. b11 0.00478) | drift mean 0.21361 | gates σ b0 [0.0495, 0.0505, 0.0491]
+      VAL r1 0.0330 cos 0.3150 self_cos +0.0286 erank 5.5 cv 0.5455 | frame r1 0.0270
+    e2     250/488 loss 4.9235 nce 4.9219 mse 0.00159 acc 0.077 lr 4.05e-04 L96 38m
+    e2     300/488 loss 4.6453 nce 4.6438 mse 0.00151 acc 0.100 lr 2.86e-04 L96 46m
+    aleph: amp mean 0.00623 (b0 0.00533 .. b11 0.00542) | drift mean 0.23984 | gates σ b0 [0.0509, 0.0526, 0.0515]
+      VAL r1 0.0940 cos 0.4006 self_cos +0.0403 erank 10.1 cv 0.3857 | frame r1 0.1030
+    e2     350/488 loss 4.0234 nce 4.0220 mse 0.00139 acc 0.171 lr 1.69e-04 L64 54m
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 397) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0/
+    e2     400/488 loss 3.8047 nce 3.8034 mse 0.00134 acc 0.225 lr 7.39e-05 L64 62m
+    aleph: amp mean 0.00685 (b0 0.00538 .. b11 0.00683) | drift mean 0.26231 | gates σ b0 [0.0516, 0.0533, 0.0524]
+      VAL r1 0.2065 cos 0.4514 self_cos +0.0091 erank 14.1 cv 0.2321 | frame r1 0.1985
+    e2     450/488 loss 3.7342 nce 3.7328 mse 0.00134 acc 0.240 lr 1.51e-05 L64 70m
+    aleph: amp mean 0.00697 (b0 0.00540 .. b11 0.00708) | drift mean 0.26714 | gates σ b0 [0.0517, 0.0534, 0.0525]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.2255
+    cos to target : 0.4563
+    self_cos      : +0.0111   <- isotropy; teachers .81-.98
+    effective rank: 14.8/768
+    CV            : 0.2412
+    frame-fit R@1 : 0.2185 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (final) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0/