ship logs/small_a41_cbk_gf_wa_e6.log (2026-08-22 4090 queue)

logs/small_a41_cbk_gf_wa_e6.log

@@ -0,0 +1,125 @@
+[alephlm0] disk: 117.7 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a4_splat seed_idx=0 (crc seed 155979836) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+[alephlm0] splat geometry recorded: {'M': 128, 'K': 8, 'overlap': 1.5, 'tau': 0.1, 'mchunk': '16'}
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a4_splat-cbk-gf-wa-s0-e6
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A4_SPLAT-CBK-GF-WA-S0-E6 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] -> AbstractPhil/alephlm-0 (private)
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a4_splat: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] a4_splat: M=128 K=8 overlap=1.5 | TRAINABLE 61,479,204 of 61,479,204 total — attention trainable capacity HALVED vs MHA (-525,312/block; frozen fabric is not capacity). A capability LOSS vs a1 is capacity-confounded; a MATCH at -10.8%% trainable is the strong result.
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a4_splat-cbk-gf-wa-s0-e6: 61,479,204 params (0.56x bert-base)
+    12L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 6 = 1,464 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.8 of 25.8 GB (co-resident/desktop holds 1.9 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 16.5 / reserved 17.4 GB vs 23.8 free -> margin 6.5 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 6 = 1,464 steps
+    batch plan: 244 batches/epoch | padded length p50 33 p90 117 max 128 (bucketing=True)
+    e1      50/1,464 loss 7.2830 nce 7.2806 mse 0.00243 acc 0.002 lr 3.03e-04 L64 8m
+    e1     100/1,464 loss 6.8378 nce 6.8356 mse 0.00221 acc 0.003 lr 6.00e-04 L64 17m
+    aleph: amp mean 0.00542 (b0 0.00511 .. b11 0.00490) | drift mean 0.13052 | gates σ b0 [0.0475, 0.0475, 0.0473]
+      VAL r1 0.0015 cos 0.1541 self_cos +0.1642 erank 2.3 cv 0.8873 | frame r1 0.0050
+    [backup] pushed (step 118) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+    e1     150/1,464 loss 6.5040 nce 6.5019 mse 0.00212 acc 0.011 lr 5.98e-04 L128 26m
+    e1     200/1,464 loss 5.5273 nce 5.5256 mse 0.00174 acc 0.029 lr 5.92e-04 L64 35m
+    aleph: amp mean 0.00573 (b0 0.00525 .. b11 0.00434) | drift mean 0.21761 | gates σ b0 [0.0497, 0.0506, 0.0494]
+      VAL r1 0.0315 cos 0.3011 self_cos +0.0521 erank 5.4 cv 0.5710 | frame r1 0.0220
+    e2     250/1,464 loss 5.2211 nce 5.2194 mse 0.00167 acc 0.049 lr 5.82e-04 L96 45m
+    e2     300/1,464 loss 5.0743 nce 5.0727 mse 0.00161 acc 0.045 lr 5.69e-04 L96 54m
+    aleph: amp mean 0.00602 (b0 0.00538 .. b11 0.00512) | drift mean 0.25270 | gates σ b0 [0.0517, 0.0532, 0.0523]
+      VAL r1 0.0860 cos 0.3998 self_cos +0.0196 erank 10.1 cv 0.3321 | frame r1 0.1025
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 338) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+    e2     350/1,464 loss 4.1543 nce 4.1529 mse 0.00141 acc 0.154 lr 5.52e-04 L64 64m
+    e2     400/1,464 loss 3.6002 nce 3.5989 mse 0.00129 acc 0.279 lr 5.31e-04 L64 73m
+    aleph: amp mean 0.00671 (b0 0.00560 .. b11 0.00578) | drift mean 0.29015 | gates σ b0 [0.0538, 0.0551, 0.055]
+      VAL r1 0.2555 cos 0.4714 self_cos +0.0127 erank 15.2 cv 0.2224 | frame r1 0.2620
+    e2     450/1,464 loss 3.3074 nce 3.3061 mse 0.00125 acc 0.350 lr 5.08e-04 L64 83m
+    e3     500/1,464 loss 3.0270 nce 3.0258 mse 0.00119 acc 0.430 lr 4.82e-04 L64 92m
+    aleph: amp mean 0.00767 (b0 0.00557 .. b11 0.00695) | drift mean 0.34585 | gates σ b0 [0.0552, 0.0561, 0.0576]
+      VAL r1 0.3780 cos 0.5081 self_cos +0.0089 erank 19.5 cv 0.2044 | frame r1 0.3945
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 548) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+    e3     550/1,464 loss 2.6261 nce 2.6250 mse 0.00112 acc 0.556 lr 4.53e-04 L32 103m
+    e3     600/1,464 loss 2.2649 nce 2.2639 mse 0.00105 acc 0.683 lr 4.22e-04 L64 111m
+    aleph: amp mean 0.00834 (b0 0.00555 .. b11 0.00869) | drift mean 0.40763 | gates σ b0 [0.0561, 0.0567, 0.0589]
+      VAL r1 0.5615 cos 0.5552 self_cos +0.0131 erank 25.8 cv 0.1598 | frame r1 0.6165
+    e3     650/1,464 loss 2.2176 nce 2.2166 mse 0.00106 acc 0.701 lr 3.90e-04 L32 119m
+    e3     700/1,464 loss 2.4276 nce 2.4265 mse 0.00107 acc 0.706 lr 3.57e-04 L96 127m
+    aleph: amp mean 0.00881 (b0 0.00559 .. b11 0.00939) | drift mean 0.45153 | gates σ b0 [0.0568, 0.0573, 0.06]
+      VAL r1 0.6120 cos 0.5710 self_cos +0.0336 erank 28.1 cv 0.1686 | frame r1 0.6855
+    e4     750/1,464 loss 2.8266 nce 2.8254 mse 0.00120 acc 0.565 lr 3.23e-04 L32 136m
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 787) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+    e4     800/1,464 loss 2.1217 nce 2.1207 mse 0.00102 acc 0.845 lr 2.88e-04 L128 145m
+    aleph: amp mean 0.00892 (b0 0.00556 .. b11 0.00939) | drift mean 0.48557 | gates σ b0 [0.0571, 0.0577, 0.0606]
+      VAL r1 0.7665 cos 0.6103 self_cos +0.0108 erank 34.8 cv 0.1356 | frame r1 0.8125
+    e4     850/1,464 loss 2.6444 nce 2.6432 mse 0.00119 acc 0.619 lr 2.54e-04 L32 154m
+    e4     900/1,464 loss 1.4496 nce 1.4487 mse 0.00088 acc 0.916 lr 2.20e-04 L64 162m
+    aleph: amp mean 0.00907 (b0 0.00557 .. b11 0.00958) | drift mean 0.50720 | gates σ b0 [0.0573, 0.058, 0.0609]
+      VAL r1 0.8120 cos 0.6200 self_cos +0.0125 erank 37.1 cv 0.1387 | frame r1 0.8585
+    e4     950/1,464 loss 2.2534 nce 2.2523 mse 0.00110 acc 0.733 lr 1.87e-04 L32 170m
+    e5   1,000/1,464 loss 2.2181 nce 2.2170 mse 0.00109 acc 0.816 lr 1.56e-04 L128 179m
+    aleph: amp mean 0.00911 (b0 0.00557 .. b11 0.00958) | drift mean 0.52188 | gates σ b0 [0.0574, 0.0581, 0.0611]
+      VAL r1 0.8435 cos 0.6303 self_cos +0.0111 erank 40.0 cv 0.1417 | frame r1 0.8820
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 1022) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+    e5   1,050/1,464 loss 1.4839 nce 1.4830 mse 0.00087 acc 0.932 lr 1.27e-04 L96 188m
+    e5   1,100/1,464 loss 1.2461 nce 1.2453 mse 0.00082 acc 0.957 lr 1.00e-04 L32 196m
+    aleph: amp mean 0.00916 (b0 0.00558 .. b11 0.00964) | drift mean 0.53161 | gates σ b0 [0.0574, 0.0581, 0.0612]
+      VAL r1 0.8585 cos 0.6336 self_cos +0.0118 erank 41.5 cv 0.1281 | frame r1 0.9000
+    e5   1,150/1,464 loss 1.5764 nce 1.5756 mse 0.00085 acc 0.909 lr 7.60e-05 L96 205m
+    e5   1,200/1,464 loss 1.1935 nce 1.1926 mse 0.00082 acc 0.952 lr 5.47e-05 L32 213m
+    aleph: amp mean 0.00920 (b0 0.00558 .. b11 0.00973) | drift mean 0.53731 | gates σ b0 [0.0575, 0.0582, 0.0612]
+      VAL r1 0.8675 cos 0.6404 self_cos +0.0124 erank 42.6 cv 0.1076 | frame r1 0.9065
+    e6   1,250/1,464 loss 1.1615 nce 1.1607 mse 0.00081 acc 0.967 lr 3.66e-05 L64 221m
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 1269) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/
+    e6   1,300/1,464 loss 1.1344 nce 1.1335 mse 0.00081 acc 0.965 lr 2.21e-05 L64 230m
+    aleph: amp mean 0.00921 (b0 0.00559 .. b11 0.00974) | drift mean 0.54004 | gates σ b0 [0.0575, 0.0582, 0.0613]
+      VAL r1 0.8710 cos 0.6416 self_cos +0.0117 erank 43.1 cv 0.1292 | frame r1 0.9070
+    e6   1,350/1,464 loss 2.2271 nce 2.2260 mse 0.00112 acc 0.757 lr 1.13e-05 L32 239m
+    e6   1,400/1,464 loss 1.6504 nce 1.6494 mse 0.00096 acc 0.952 lr 4.25e-06 L128 248m
+    aleph: amp mean 0.00921 (b0 0.00559 .. b11 0.00972) | drift mean 0.54094 | gates σ b0 [0.0575, 0.0582, 0.0613]
+      VAL r1 0.8690 cos 0.6425 self_cos +0.0112 erank 43.2 cv 0.1324 | frame r1 0.9095
+    e6   1,450/1,464 loss 1.1272 nce 1.1264 mse 0.00081 acc 0.969 lr 1.16e-06 L64 256m
+    aleph: amp mean 0.00921 (b0 0.00559 .. b11 0.00972) | drift mean 0.54104 | gates σ b0 [0.0575, 0.0582, 0.0613]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.8695
+    cos to target : 0.6425
+    self_cos      : +0.0112   <- isotropy; teachers .81-.98
+    effective rank: 43.3/768
+    CV            : 0.1225
+    frame-fit R@1 : 0.9095 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (final) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-e6/