Record 3.5B early stability check
Browse files
subprojects/03_apertus_extension_and_embedding_adaptation/TAKEOVER_LOG_20260521.md
CHANGED
|
@@ -942,3 +942,22 @@ Current next gate:
|
|
| 942 |
- all three completed iteration `477` without OOM, skipped iterations, or
|
| 943 |
NaNs;
|
| 944 |
- early per-GPU throughput was roughly `7.5k-7.7k` tokens/s.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 942 |
- all three completed iteration `477` without OOM, skipped iterations, or
|
| 943 |
NaNs;
|
| 944 |
- early per-GPU throughput was roughly `7.5k-7.7k` tokens/s.
|
| 945 |
+
|
| 946 |
+
### 2026-05-24T14:58Z 3.5B early-stability check
|
| 947 |
+
|
| 948 |
+
- Foreground babysitting reached the intended early resumed-optimizer window:
|
| 949 |
+
- Vanilla reached iteration `487`;
|
| 950 |
+
- ReTok reached iteration `487`;
|
| 951 |
+
- TD layer11 reached iteration `486`.
|
| 952 |
+
- Health:
|
| 953 |
+
- all three arms still running;
|
| 954 |
+
- `loss scale = 1.0`;
|
| 955 |
+
- `number of skipped iterations = 0`;
|
| 956 |
+
- `number of nan iterations = 0`;
|
| 957 |
+
- per-GPU throughput stable at roughly `7.9k-8.1k` tokens/s after warmup.
|
| 958 |
+
- Observed isolated grad-norm spikes:
|
| 959 |
+
- Vanilla: `17.784` at iter `481`, then back near `1`;
|
| 960 |
+
- ReTok: `7.051` at iter `482`, then back near `1-2`;
|
| 961 |
+
- TD: no comparable early spike; latest observed grad norm `2.450`.
|
| 962 |
+
- No intervention taken. Continue babysitting until the first segment reaches
|
| 963 |
+
iter `585`, writes checkpoints, and the eval sidecars begin to drain.
|