ship logs/small_a43_rot_wa_cbk_s1.log (2026-08-22 4090 queue)

logs/small_a43_rot_wa_cbk_s1.log

@@ -0,0 +1,77 @@
+[alephlm0] disk: 123.8 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a4_splat seed_idx=1 (crc seed 46212409) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a4_splat-cbk-rot-wa-s1/
+[alephlm0] splat geometry recorded: {'M': 128, 'K': 8, 'overlap': 1.5, 'tau': 0.1, 'mchunk': '16'}
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a4_splat-cbk-rot-wa-s1
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A4_SPLAT-CBK-ROT-WA-S1 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] -> AbstractPhil/alephlm-0 (private)
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a4_splat: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] a4_splat: M=128 K=8 overlap=1.5 | TRAINABLE 61,479,204 of 61,479,204 total — attention trainable capacity HALVED vs MHA (-525,312/block; frozen fabric is not capacity). A capability LOSS vs a1 is capacity-confounded; a MATCH at -10.8%% trainable is the strong result.
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a4_splat-cbk-rot-wa-s1: 61,479,204 params (0.56x bert-base)
+    12L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 2 = 488 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.8 of 25.8 GB (co-resident/desktop holds 1.9 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 17.0 / reserved 18.0 GB vs 23.8 free -> margin 5.9 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 2 = 488 steps
+    batch plan: 244 batches/epoch | padded length p50 34 p90 122 max 128 (bucketing=True)
+    e1      50/488 loss 7.0995 nce 7.0972 mse 0.00232 acc 0.001 lr 3.03e-04 L96 8m
+    e1     100/488 loss 7.5705 nce 7.5680 mse 0.00254 acc 0.001 lr 6.00e-04 L128 17m
+    aleph: amp mean 0.00507 (b0 0.00576 .. b11 0.00432) | drift mean 0.11460 | gates σ b0 [0.0478, 0.0477, 0.0479]
+      VAL r1 0.0020 cos 0.0889 self_cos +0.5858 erank 1.2 cv 1.2562 | frame r1 0.0015
+    [backup] pushed (step 113) -> runs/alephlm0-small-a4_splat-cbk-rot-wa-s1/
+    e1     150/488 loss 7.1795 nce 7.1772 mse 0.00230 acc 0.004 lr 5.76e-04 L32 28m
+    e1     200/488 loss 6.5124 nce 6.5102 mse 0.00216 acc 0.010 lr 5.07e-04 L32 36m
+    aleph: amp mean 0.00566 (b0 0.00667 .. b11 0.00547) | drift mean 0.17810 | gates σ b0 [0.0492, 0.0481, 0.0485]
+      VAL r1 0.0080 cos 0.2150 self_cos +0.0792 erank 4.2 cv 0.7969 | frame r1 0.0115
+    e2     250/488 loss 5.8780 nce 5.8761 mse 0.00193 acc 0.014 lr 4.05e-04 L32 44m
+    e2     300/488 loss 5.3394 nce 5.3376 mse 0.00176 acc 0.029 lr 2.86e-04 L32 53m
+    aleph: amp mean 0.00575 (b0 0.00662 .. b11 0.00645) | drift mean 0.19325 | gates σ b0 [0.0496, 0.048, 0.0489]
+      VAL r1 0.0325 cos 0.3175 self_cos +0.0434 erank 6.8 cv 0.4737 | frame r1 0.0345
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 348) -> runs/alephlm0-small-a4_splat-cbk-rot-wa-s1/
+    e2     350/488 loss 5.2239 nce 5.2222 mse 0.00174 acc 0.049 lr 1.69e-04 L96 62m
+    e2     400/488 loss 4.8089 nce 4.8073 mse 0.00161 acc 0.062 lr 7.39e-05 L64 71m
+    aleph: amp mean 0.00630 (b0 0.00644 .. b11 0.00847) | drift mean 0.20922 | gates σ b0 [0.05, 0.0479, 0.0492]
+      VAL r1 0.0675 cos 0.3718 self_cos +0.0154 erank 9.4 cv 0.3720 | frame r1 0.0675
+    e2     450/488 loss 4.8514 nce 4.8499 mse 0.00151 acc 0.073 lr 1.51e-05 L64 79m
+    aleph: amp mean 0.00642 (b0 0.00643 .. b11 0.00869) | drift mean 0.21389 | gates σ b0 [0.05, 0.0479, 0.0492]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.0805
+    cos to target : 0.3811
+    self_cos      : +0.0178   <- isotropy; teachers .81-.98
+    effective rank: 9.9/768
+    CV            : 0.3568
+    frame-fit R@1 : 0.0800 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (final) -> runs/alephlm0-small-a4_splat-cbk-rot-wa-s1/