ship logs/small_a41_L8M256_s0.log (2026-08-22 4090 queue)

logs/small_a41_L8M256_s0.log

@@ -0,0 +1,82 @@
+[alephlm0] disk: 107.0 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: n_layers 12 -> 8 (ALEPHLM_N_LAYERS; 2026-08-24 shape probe 256 heads x 8 layers). Use ALEPHLM_RUN_SUFFIX to name the run.
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a4_splat seed_idx=0 (crc seed 155979836) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256/
+[alephlm0] splat geometry recorded: {'M': 256, 'K': 8, 'overlap': 1.5, 'tau': 0.1, 'mchunk': '16'}
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A4_SPLAT-CBK-GF-WA-S0-L8M256 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] -> AbstractPhil/alephlm-0 (private)
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a4_splat: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] a4_splat: M=256 K=8 overlap=1.5 | TRAINABLE 52,007,192 of 52,007,192 total — attention trainable capacity HALVED vs MHA (-525,312/block; frozen fabric is not capacity). A capability LOSS vs a1 is capacity-confounded; a MATCH at -10.8%% trainable is the strong result.
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256: 52,007,192 params (0.48x bert-base)
+    8L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 2 = 488 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.9 of 25.8 GB (co-resident/desktop holds 1.9 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 14.3 / reserved 15.2 GB vs 23.9 free -> margin 8.7 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 2 = 488 steps
+    batch plan: 244 batches/epoch | padded length p50 33 p90 117 max 128 (bucketing=True)
+    e1      50/488 loss 7.3111 nce 7.3086 mse 0.00251 acc 0.001 lr 3.03e-04 L64 17m
+    [backup] pushed (step 58) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256/
+    e1     100/488 loss 6.8890 nce 6.8867 mse 0.00227 acc 0.004 lr 6.00e-04 L64 36m
+    aleph: amp mean 0.00501 (b0 0.00516 .. b7 0.00410) | drift mean 0.11534 | gates σ b0 [0.0472, 0.0473, 0.0476]
+      VAL r1 0.0030 cos 0.1188 self_cos +0.1582 erank 2.3 cv 1.2231 | frame r1 0.0030
+    e1     150/488 loss 6.6665 nce 6.6644 mse 0.00218 acc 0.004 lr 5.76e-04 L128 54m
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 170) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256/
+    e1     200/488 loss 5.8320 nce 5.8301 mse 0.00184 acc 0.019 lr 5.07e-04 L64 73m
+    aleph: amp mean 0.00518 (b0 0.00528 .. b7 0.00429) | drift mean 0.18735 | gates σ b0 [0.0494, 0.0485, 0.0479]
+      VAL r1 0.0125 cos 0.2429 self_cos +0.0327 erank 4.4 cv 0.6433 | frame r1 0.0155
+    e2     250/488 loss 5.4182 nce 5.4164 mse 0.00178 acc 0.039 lr 4.05e-04 L96 92m
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 280) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256/
+    e2     300/488 loss 4.9340 nce 4.9324 mse 0.00160 acc 0.068 lr 2.86e-04 L96 110m
+    aleph: amp mean 0.00591 (b0 0.00530 .. b7 0.00551) | drift mean 0.22520 | gates σ b0 [0.0498, 0.0493, 0.049]
+      VAL r1 0.0605 cos 0.3650 self_cos +0.0258 erank 8.5 cv 0.3869 | frame r1 0.0575
+    e2     350/488 loss 4.5725 nce 4.5710 mse 0.00151 acc 0.091 lr 1.69e-04 L64 129m
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (step 391) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256/
+    e2     400/488 loss 4.3762 nce 4.3747 mse 0.00147 acc 0.123 lr 7.39e-05 L64 147m
+    aleph: amp mean 0.00658 (b0 0.00537 .. b7 0.00711) | drift mean 0.24671 | gates σ b0 [0.0502, 0.0496, 0.0498]
+      VAL r1 0.1035 cos 0.4021 self_cos +0.0120 erank 10.5 cv 0.3292 | frame r1 0.0975
+    e2     450/488 loss 4.3113 nce 4.3098 mse 0.00147 acc 0.123 lr 1.51e-05 L64 166m
+    aleph: amp mean 0.00677 (b0 0.00539 .. b7 0.00744) | drift mean 0.25198 | gates σ b0 [0.0502, 0.0497, 0.0499]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.1130
+    cos to target : 0.4071
+    self_cos      : +0.0133   <- isotropy; teachers .81-.98
+    effective rank: 11.2/768
+    CV            : 0.2968
+    frame-fit R@1 : 0.1035 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (final) -> runs/alephlm0-small-a4_splat-cbk-gf-wa-s0-L8M256/