ship logs/small_a1_anchored_s1.log (2026-08-22 4090 queue)

logs/small_a1_anchored_s1.log

@@ -0,0 +1,73 @@
+[alephlm0] disk: 125.9 GB free on the work drive
+[alephlm0] allocator capped at 22.7 GB (0.880 of 25.8; free was 24.2) — over-commit now FAILS FAST instead of spilling to sysmem
+!! RECIPE DEVIATION: max_tokens 256 -> 128 (ALEPHLM_MAX_TOKENS). Captions beyond the new ceiling are TRUNCATED at tokenization.
+[alephlm0] arm=a1_anchored seed_idx=1 (crc seed 46212409) pooling=cls profile=retrain -> AbstractPhil/alephlm-0 runs/alephlm0-small-a1_anchored-s1/
+── PROFILE 'retrain' ─────────────────────────────────────────────────────────
+   run_name         alephlm0-small-a1_anchored-s1
+   hf_repo          AbstractPhil/alephlm-0
+   artifact_repo    AbstractPhil/captionbert-8192-v2
+   targets_repo     AbstractPhil/captionbert-8192-v2-consensus
+   caption_field    caption_llava
+   fresh_start      True
+   stages          0:False 1:False 2:False 3:True
+   config coherent
+==============================================================================
+ALEPHLM0-SMALL-A1_ANCHORED-S1 — CONSENSUS DISTILLATION, CC12M SCALE
+==============================================================================
+device=cuda  work_dir=E:/mirel/data/alephlm0
+gpu=NVIDIA GeForce RTX 4090 vram=26GB
+chunks: 1 train + 2 holdout | excluded for missing experts: {}
+    [backup] -> AbstractPhil/alephlm-0 (private)
+── ARTIFACTS ─────────────────────────────────────────────────────────────────
+   maps: local E:/mirel/data/alephlm0/maps/alignment_maps.pt
+   (NOT refitting: every consensus target was built with THIS map;
+    a new fit would sit in a different frame with no loss signal.)
+   targets: 3 local + 0 fetched from AbstractPhil/captionbert-8192-v2-consensus
+
+── STAGE 3 — TRAIN ───────────────────────────────────────────────────────────
+    [alephlm0] arm=a1_anchored: dense ff2048 -> trunk ff512 + 3x ff512 aleph-dispatched (tau=0.1, gates σ(-3.0), expert outs zero-init)
+    [alephlm0] C6 null path: bit-exact at init  ✓
+    alephlm0-small-a1_anchored-s1: 58,345,764 params (0.53x bert-base)
+    12L 512d 8h ff2048 pool=cls
+    1 chunks ≈ 500,000 rows | 244 steps/ep x 2 = 488 steps @ batch 2048
+    loss = 1.0*InfoNCE(T=0.07) + 1.0*MSE + 0.0*CV   [champion consensus_nce_mse]
+── VRAM PROBE (co-resident-aware) - worst case before data ───────────────────
+    free 23.9 of 25.8 GB (co-resident/desktop holds 1.8 GB)
+    batch 2048 x L 128 (checkpointing=True) -> peak alloc 13.9 / reserved 14.8 GB vs 23.9 free -> margin 9.2 GB
+    PASS
+    fresh_start=True -- ignoring any local or remote checkpoint
+    building val store...
+    RamStore [val]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_56998379bbd3 — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    building train store (one pass, then zero disk I/O)...
+    RamStore [train]: CACHE HIT E:/mirel/data/alephlm0/store_cache/store_83fdd0bab51e — 500,000 rows, targets+tokens memory-mapped (~0.0 GB resident)
+    500,000 rows resident | 244 steps/ep x 2 = 488 steps
+    batch plan: 244 batches/epoch | padded length p50 34 p90 122 max 128 (bucketing=True)
+    e1      50/488 loss 5.8086 nce 5.8067 mse 0.00195 acc 0.028 lr 3.03e-04 L96 2m
+    e1     100/488 loss 5.3692 nce 5.3673 mse 0.00186 acc 0.064 lr 6.00e-04 L128 5m
+    aleph: amp mean 0.00555 (b0 0.00534 .. b11 0.00551) | drift mean 0.08301 | gates σ b0 [0.0485, 0.0487, 0.0488]
+      VAL r1 0.1380 cos 0.3447 self_cos +0.1040 erank 12.4 cv 0.2949 | frame r1 0.2030
+    e1     150/488 loss 5.1457 nce 5.1440 mse 0.00175 acc 0.121 lr 5.76e-04 L32 8m
+    e1     200/488 loss 4.4786 nce 4.4770 mse 0.00162 acc 0.234 lr 5.07e-04 L32 10m
+    aleph: amp mean 0.00675 (b0 0.00579 .. b11 0.00772) | drift mean 0.24509 | gates σ b0 [0.0525, 0.0523, 0.053]
+      VAL r1 0.5985 cos 0.5168 self_cos +0.0429 erank 25.0 cv 0.1656 | frame r1 0.6645
+    e2     250/488 loss 2.0754 nce 2.0743 mse 0.00108 acc 0.819 lr 4.05e-04 L32 12m
+    e2     300/488 loss 1.7714 nce 1.7704 mse 0.00102 acc 0.853 lr 2.86e-04 L32 14m
+    aleph: amp mean 0.00829 (b0 0.00636 .. b11 0.01143) | drift mean 0.38211 | gates σ b0 [0.0559, 0.0555, 0.0561]
+      VAL r1 0.8100 cos 0.6077 self_cos +0.0078 erank 38.5 cv 0.1427 | frame r1 0.8790
+    e2     350/488 loss 1.6042 nce 1.6032 mse 0.00094 acc 0.954 lr 1.69e-04 L96 17m
+    e2     400/488 loss 1.2386 nce 1.2378 mse 0.00087 acc 0.951 lr 7.39e-05 L64 19m
+    aleph: amp mean 0.00870 (b0 0.00659 .. b11 0.01194) | drift mean 0.42930 | gates σ b0 [0.0571, 0.0568, 0.0573]
+      VAL r1 0.8830 cos 0.6444 self_cos +0.0052 erank 43.7 cv 0.1208 | frame r1 0.9240
+    [backup] pushed (step 412) -> runs/alephlm0-small-a1_anchored-s1/
+    e2     450/488 loss 1.5828 nce 1.5819 mse 0.00089 acc 0.905 lr 1.51e-05 L64 22m
+    aleph: amp mean 0.00874 (b0 0.00663 .. b11 0.01195) | drift mean 0.43492 | gates σ b0 [0.0572, 0.0569, 0.0574]
+── FINAL ─────────────────────────────────────────────────────────────────────
+    mimicry R@1 (student->consensus, NOT capability): 0.8955
+    cos to target : 0.6496
+    self_cos      : +0.0035   <- isotropy; teachers .81-.98
+    effective rank: 45.3/768
+    CV            : 0.1159
+    frame-fit R@1 : 0.9315 (should be ~mimicry: reference-member alignment pins the frame)
+    CAPABILITY is decided by STS-B / SICK vs the five teachers, not here.
+No files have been modified since last commit. Skipping to prevent empty commit.
+    [backup] pushed (final) -> runs/alephlm0-small-a1_anchored-s1/