fffoivos commited on
Commit
d7e46bb
·
verified ·
1 Parent(s): cd07458

Record 3.5B early stability check

Browse files
subprojects/03_apertus_extension_and_embedding_adaptation/TAKEOVER_LOG_20260521.md CHANGED
@@ -942,3 +942,22 @@ Current next gate:
942
  - all three completed iteration `477` without OOM, skipped iterations, or
943
  NaNs;
944
  - early per-GPU throughput was roughly `7.5k-7.7k` tokens/s.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
942
  - all three completed iteration `477` without OOM, skipped iterations, or
943
  NaNs;
944
  - early per-GPU throughput was roughly `7.5k-7.7k` tokens/s.
945
+
946
+ ### 2026-05-24T14:58Z 3.5B early-stability check
947
+
948
+ - Foreground babysitting reached the intended early resumed-optimizer window:
949
+ - Vanilla reached iteration `487`;
950
+ - ReTok reached iteration `487`;
951
+ - TD layer11 reached iteration `486`.
952
+ - Health:
953
+ - all three arms still running;
954
+ - `loss scale = 1.0`;
955
+ - `number of skipped iterations = 0`;
956
+ - `number of nan iterations = 0`;
957
+ - per-GPU throughput stable at roughly `7.9k-8.1k` tokens/s after warmup.
958
+ - Observed isolated grad-norm spikes:
959
+ - Vanilla: `17.784` at iter `481`, then back near `1`;
960
+ - ReTok: `7.051` at iter `482`, then back near `1-2`;
961
+ - TD: no comparable early spike; latest observed grad norm `2.450`.
962
+ - No intervention taken. Continue babysitting until the first segment reaches
963
+ iter `585`, writes checkpoints, and the eval sidecars begin to drain.