ship logs/small_a41_L8M256_e6_s1.log (2026-08-22 4090 queue)

logs/small_a41_L8M256_e6_s1.log

@@ -0,0 +1,113 @@
+[alephlm0] disk: 98.6 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: n_layers 12 -> 8 (ALEPHLM_N_LAYERS; 2026-08-24 shape probe 256 heads x 8 layers). Use ALEPHLM_RUN_SUFFIX to name the run.
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a4_splat seed_idx=1 (crc seed 46212409) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a4_splat-cbk-gf-wa-s1-e6-L8M256/
+[alephlm0] splat geometry recorded: {'M': 256, 'K': 8, 'overlap': 1.5, 'tau': 0.1, 'mchunk': '16'}
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a4_splat-cbk-gf-wa-s1-e6-L8M256
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A4_SPLAT-CBK-GF-WA-S1-E6-L8M256 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] disabled: ConnectTimeout: [WinError 10060] A connection attempt failed because the connected party did not properly respond af
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a4_splat: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] a4_splat: M=256 K=8 overlap=1.5 | TRAINABLE 52,007,192 of 52,007,192 total — attention trainable capacity HALVED vs MHA (-525,312/block; frozen fabric is not capacity). A capability LOSS vs a1 is capacity-confounded; a MATCH at -10.8%% trainable is the strong result.
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a4_splat-cbk-gf-wa-s1-e6-L8M256: 52,007,192 params (0.48x bert-base)
+    8L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 6 = 1,464 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.9 of 25.8 GB (co-resident/desktop holds 1.9 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 14.3 / reserved 15.2 GB vs 23.9 free -> margin 8.7 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 6 = 1,464 steps
+    batch plan: 244 batches/epoch | padded length p50 34 p90 122 max 128 (bucketing=True)
+    e1      50/1,464 loss 7.3392 nce 7.3369 mse 0.00237 acc 0.001 lr 3.03e-04 L96 17m
+    e1     100/1,464 loss 7.2274 nce 7.2250 mse 0.00241 acc 0.003 lr 6.00e-04 L128 34m
+    aleph: amp mean 0.00488 (b0 0.00598 .. b7 0.00434) | drift mean 0.09092 | gates σ b0 [0.0474, 0.0475, 0.0468]
+      VAL r1 0.0020 cos 0.1362 self_cos +0.1026 erank 2.2 cv 0.9442 | frame r1 0.0030
+    e1     150/1,464 loss 7.1602 nce 7.1579 mse 0.00237 acc 0.004 lr 5.98e-04 L32 52m
+    e1     200/1,464 loss 6.5951 nce 6.5930 mse 0.00206 acc 0.007 lr 5.92e-04 L32 69m
+    aleph: amp mean 0.00586 (b0 0.00543 .. b7 0.00631) | drift mean 0.19872 | gates σ b0 [0.0489, 0.0491, 0.0482]
+      VAL r1 0.0070 cos 0.2192 self_cos +0.0361 erank 4.4 cv 0.6482 | frame r1 0.0120
+    e2     250/1,464 loss 5.9314 nce 5.9295 mse 0.00194 acc 0.013 lr 5.82e-04 L32 88m
+    e2     300/1,464 loss 5.4204 nce 5.4186 mse 0.00179 acc 0.037 lr 5.69e-04 L32 105m
+    aleph: amp mean 0.00602 (b0 0.00565 .. b7 0.00529) | drift mean 0.24204 | gates σ b0 [0.051, 0.0507, 0.0495]
+      VAL r1 0.0250 cos 0.3097 self_cos +0.0228 erank 6.9 cv 0.5072 | frame r1 0.0255
+    e2     350/1,464 loss 5.5172 nce 5.5154 mse 0.00177 acc 0.042 lr 5.52e-04 L96 124m
+    e2     400/1,464 loss 4.8740 nce 4.8724 mse 0.00160 acc 0.065 lr 5.31e-04 L64 146m
+    aleph: amp mean 0.00638 (b0 0.00564 .. b7 0.00555) | drift mean 0.27300 | gates σ b0 [0.0516, 0.0509, 0.0502]
+      VAL r1 0.0620 cos 0.3709 self_cos +0.0228 erank 8.9 cv 0.3384 | frame r1 0.0490
+    e2     450/1,464 loss 4.5133 nce 4.5119 mse 0.00144 acc 0.100 lr 5.08e-04 L64 171m
+    e3     500/1,464 loss 4.1976 nce 4.1961 mse 0.00145 acc 0.141 lr 4.82e-04 L32 191m
+    aleph: amp mean 0.00663 (b0 0.00568 .. b7 0.00632) | drift mean 0.30036 | gates σ b0 [0.0522, 0.0512, 0.0506]
+      VAL r1 0.1265 cos 0.4112 self_cos +0.0312 erank 12.2 cv 0.2837 | frame r1 0.1070
+    e3     550/1,464 loss 5.1066 nce 5.1050 mse 0.00165 acc 0.081 lr 4.53e-04 L32 209m
+    e3     600/1,464 loss 3.8001 nce 3.7988 mse 0.00131 acc 0.226 lr 4.22e-04 L96 229m
+    aleph: amp mean 0.00711 (b0 0.00581 .. b7 0.00739) | drift mean 0.33370 | gates σ b0 [0.0527, 0.0514, 0.0508]
+      VAL r1 0.2295 cos 0.4607 self_cos +0.0187 erank 16.4 cv 0.2440 | frame r1 0.2110
+    e3     650/1,464 loss 3.5647 nce 3.5634 mse 0.00130 acc 0.278 lr 3.90e-04 L32 259m
+    e3     700/1,464 loss 4.1258 nce 4.1244 mse 0.00147 acc 0.211 lr 3.57e-04 L32 291m
+    aleph: amp mean 0.00764 (b0 0.00578 .. b7 0.00800) | drift mean 0.37340 | gates σ b0 [0.0531, 0.0517, 0.051]
+      VAL r1 0.3035 cos 0.4889 self_cos +0.0199 erank 19.0 cv 0.2085 | frame r1 0.3335
+    e4     750/1,464 loss 4.5992 nce 4.5976 mse 0.00158 acc 0.144 lr 3.23e-04 L32 310m
+    e4     800/1,464 loss 3.2292 nce 3.2280 mse 0.00122 acc 0.429 lr 2.88e-04 L128 328m
+    aleph: amp mean 0.00810 (b0 0.00594 .. b7 0.00846) | drift mean 0.41547 | gates σ b0 [0.0536, 0.052, 0.0513]
+      VAL r1 0.4470 cos 0.5264 self_cos +0.0172 erank 23.6 cv 0.1811 | frame r1 0.4600
+    e4     850/1,464 loss 2.9554 nce 2.9543 mse 0.00112 acc 0.492 lr 2.54e-04 L96 347m
+    e4     900/1,464 loss 2.7761 nce 2.7750 mse 0.00108 acc 0.517 lr 2.20e-04 L96 366m
+    aleph: amp mean 0.00851 (b0 0.00602 .. b7 0.00896) | drift mean 0.44646 | gates σ b0 [0.0539, 0.0521, 0.0515]
+      VAL r1 0.5415 cos 0.5488 self_cos +0.0166 erank 27.3 cv 0.1904 | frame r1 0.5675
+    e4     950/1,464 loss 3.7010 nce 3.6997 mse 0.00137 acc 0.343 lr 1.87e-04 L32 385m
+    e5   1,000/1,464 loss 2.7438 nce 2.7426 mse 0.00114 acc 0.645 lr 1.56e-04 L128 404m
+    aleph: amp mean 0.00875 (b0 0.00605 .. b7 0.00922) | drift mean 0.46850 | gates σ b0 [0.054, 0.0523, 0.0516]
+      VAL r1 0.6145 cos 0.5699 self_cos +0.0083 erank 29.2 cv 0.1517 | frame r1 0.6415
+    e5   1,050/1,464 loss 4.2612 nce 4.2596 mse 0.00153 acc 0.248 lr 1.27e-04 L32 423m
+    e5   1,100/1,464 loss 2.4095 nce 2.4085 mse 0.00101 acc 0.658 lr 1.00e-04 L64 442m
+    aleph: amp mean 0.00895 (b0 0.00610 .. b7 0.00946) | drift mean 0.48337 | gates σ b0 [0.0541, 0.0524, 0.0518]
+      VAL r1 0.6430 cos 0.5749 self_cos +0.0134 erank 30.8 cv 0.1787 | frame r1 0.6820
+    e5   1,150/1,464 loss 2.9154 nce 2.9142 mse 0.00121 acc 0.564 lr 7.60e-05 L128 461m
+    e5   1,200/1,464 loss 1.9763 nce 1.9753 mse 0.00097 acc 0.791 lr 5.47e-05 L32 480m
+    aleph: amp mean 0.00902 (b0 0.00612 .. b7 0.00958) | drift mean 0.49205 | gates σ b0 [0.0542, 0.0524, 0.0518]
+      VAL r1 0.6700 cos 0.5840 self_cos +0.0081 erank 31.9 cv 0.1520 | frame r1 0.7065
+    e6   1,250/1,464 loss 2.1743 nce 2.1733 mse 0.00100 acc 0.777 lr 3.66e-05 L96 500m
+    e6   1,300/1,464 loss 2.3669 nce 2.3658 mse 0.00110 acc 0.672 lr 2.21e-05 L32 518m
+    aleph: amp mean 0.00906 (b0 0.00613 .. b7 0.00959) | drift mean 0.49603 | gates σ b0 [0.0542, 0.0525, 0.0519]
+      VAL r1 0.6865 cos 0.5847 self_cos +0.0101 erank 32.3 cv 0.1484 | frame r1 0.7190
+    e6   1,350/1,464 loss 1.9456 nce 1.9446 mse 0.00097 acc 0.784 lr 1.13e-05 L32 537m
+    e6   1,400/1,464 loss 2.2039 nce 2.2029 mse 0.00101 acc 0.798 lr 4.25e-06 L96 556m
+    aleph: amp mean 0.00908 (b0 0.00614 .. b7 0.00962) | drift mean 0.49729 | gates σ b0 [0.0542, 0.0525, 0.0519]
+      VAL r1 0.6840 cos 0.5856 self_cos +0.0099 erank 32.5 cv 0.1274 | frame r1 0.7190
+    e6   1,450/1,464 loss 3.3391 nce 3.3377 mse 0.00134 acc 0.438 lr 1.16e-06 L32 575m
+    aleph: amp mean 0.00908 (b0 0.00614 .. b7 0.00964) | drift mean 0.49743 | gates σ b0 [0.0542, 0.0525, 0.0519]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.6870
+    cos to target : 0.5860
+    self_cos      : +0.0097   <- isotropy; teachers .81-.98
+    effective rank: 32.5/768
+    CV            : 0.1449
+    frame-fit R@1 : 0.7215 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.