trainer: the post-stop assertions are rank-0 facts (E-044: hub_cb.steps is empty on rank 1 by construction, so the guard killed every correctly-checkpointed segment)

train/train_ounce100m.py

@@ -603,7 +603,13 @@ def main():
     # the target, but performing the save at that step is the library's decision. If it ever ordered the
     # stop ahead of the save, the session would exit 0 having pushed nothing, and every relaunch would
     # redo the same interval while `latest` stayed put -- which looks like a stuck run, not a broken one.
-    if hub_cb is not None and args.stop_after_steps:
+    if hub_cb is not None and args.stop_after_steps and rank == 0:
+        # Rank 0 only, because every fact below is a rank-0 fact: `on_save` returns early on the other
+        # ranks, so `hub_cb.steps` is empty on rank 1 by construction and this guard -- written to catch
+        # a segment that ended without a pushed checkpoint -- killed *every* correctly checkpointed
+        # segment on rank 1 with `pushed: []`. That SystemExit, not a divergent collective, is what
+        # probe v1/v4/v5/v6/v7 saw as "rank 1 exitcode 1 with error_file <N/A>": rank 1 died on its own
+        # assertion while rank 0 sat in the next all_reduce until torchrun SIGTERMed it (E-044).
         if args.stop_after_steps <= start_step:
             raise SystemExit(f"--stop-after-steps {args.stop_after_steps} is not beyond the resume point "
                              f"step {start_step}: this session would train nothing")
@@ -632,12 +638,12 @@ def main():
 
     # A segment that stopped short of the horizon does not run the validation pass, on every rank.
     #
-    # Probe 1 v1 and v4 showed the same split twice: the leg stopped early by `--stop-after-steps` died inside
-    # `evaluate()` on rank 1 with no Python traceback at all, while the leg that reached its horizon ran the
-    # same call and exported cleanly. `should_training_stop` is raised from `on_step_end`, so the two ranks
-    # can leave the inner loop one batch apart, and a collective entered in that state kills a rank without
-    # raising -- which is exactly the signature. The report's PPL comes from the horizon leg, and the
-    # mid-run legs keep their training loss, which is what §5 asks to watch on every wake.
+    # Written for what probes v1/v4 saw ("rank 1 exitcode 1, error_file <N/A>, right after a forced stop"),
+    # which was attributed to `evaluate()` being entered from a diverged collective state. E-044 later found
+    # the actual cause -- the rank-0-only push assertion firing on rank 1 -- so this skip is not what makes
+    # the forced-stop leg work. It stays, on its own merits: the held-out PPL of a 762-step segment is not
+    # worth a forward pass over 2 M tokens inside a billed session, and §5's per-wake monitoring is served by
+    # the training loss. The report's PPL comes from the horizon leg, which is where it is measured.
     forced_stop = bool(hub_cb and hub_cb.stopped_at and step < steps_planned)
     print(f"[rank {rank}] past the stop guard: forced_stop={forced_stop} "
           f"stopped_at={getattr(hub_cb, 'stopped_at', None)}", flush=True)