ounce100m-code / kernels /p4_stop_probe.py

Commit History

p4_stop_probe: E-044 repin (rank-0 stop assertions; memory gate moved onto the training-only leg)
14e982b
verified

Cion-lab commited on

p4_stop_probe: repin to b6210a29 (per-rank post-train marks + readable probe failures)
98922ef
verified

Cion-lab commited on

p4_stop_probe: E-042 repin + rehearsal of the launcher's real argv and the prune guard
a4158df
verified

Cion-lab commited on

p4_stop_probe: pins at a813e5c7 + bounded-readback rehearsal stage / tee-prefix fix
bc34e7d
verified

Cion-lab commited on

pin 7570b7b2: forced-stop legs skip the collective validation pass; probe expectations split leg1 (skip by design) from leg2 (must run it)
dd31402
verified

Cion-lab commited on

p4_stop_probe: repair the soak branch's timeout tuple (an earlier edit dropped it, which would have been a NameError only in soak mode) and right-size the leg ceilings under the notebook timeout
e7fecad
verified

Cion-lab commited on

review round 3 (probe/model-card/eval): rank-visible eval failure + cross-rank peak memory + complete run_summary; card derived-and-asserted from config; harness install --no-deps and task ids confirmed; torchrun redirects 3 + tee 3
9119e1e
verified

Cion-lab commited on

p4_stop_probe: dated scratch ckpt repo per attempt instead of clearing one (v3 died on HfApi.delete_folder's argument order in 20 s); define PROBE before the argv that uses it
e7dcd1f
verified

Cion-lab commited on

p4_stop_probe: reset the scratch ckpt repo before leg 1 (v2 was correctly refused: v1 left final@20, and stop-after-10 is not beyond it); diag glob fixed; --grad-ckpt/--no-grad-ckpt ambiguity removed
d0b40c9
verified

Cion-lab commited on

p4_stop_probe: redirect stderr only (bitmask 2) so the loss curve keeps streaming; ASCII the comment
76e690e
verified

Cion-lab commited on

p4_stop_probe: --redirects 3 --log-dir and dump each rank's own log on a failed leg, because torchrun reported error_file <N/A> for a rank-1-only crash and left the exception unsaid
622b4e1
verified

Cion-lab commited on

pin 1dd45f80 (E-037 launcher fix)
37e90cb
verified

Cion-lab commited on

E-037: torchrun's first positional IS the script -- passing sys.executable made it compile the python binary and die with 'source code cannot contain null bytes' in 8.6s. Affects the Phase 4 launcher identically at session 1. Probe also split into smoke (20 steps) and soak (180) modes
1dd45f8
verified

Cion-lab commited on

p4_stop_probe: becomes the no-checkpointing soak at the real 262,144-token step shape (180 steps, 3 pushes, a forced cold resume, the 2M-token validation pass, peak memory asserted) as well as the stop-after-steps probe -- the measurement that answers D-017 with evidence
c8523ef
verified

Cion-lab commited on

kernels: pin 91fb319f (argparse help fix)
808cf83
verified

Cion-lab commited on

kernels: pin 063fa266 (perm-identity guard)
46bb643
verified

Cion-lab commited on

p4_stop_probe: exercise --stop-after-steps, the push/verify/pointer/prune order and a cold Hub resume on 2xT4 before session 1 depends on them
1878839
verified

Cion-lab commited on