Record medium pause at completed 27B milestone for small corruption experiment
Browse files
iclr-debug-ncp-medium-135b-v1/PAUSED_FOR_SMALL_RAMP.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"stage": "paused",
|
| 3 |
+
"reason": "User requested fresh small 50-to-30 corruption run on this node",
|
| 4 |
+
"step": 25750,
|
| 5 |
+
"checkpoint": "/home/ubuntu/mstok-results/iclr-debug-ncp-medium-135b-v1/generator/milestone-iter-25750.pt",
|
| 6 |
+
"updated_utc": "2026-09-21T23:50:49.359366+00:00",
|
| 7 |
+
"controller_pid": null,
|
| 8 |
+
"child_pid": null,
|
| 9 |
+
"resume_command": "bash /home/ubuntu/mstok-results/launch-ncp-medium-135b.sh --resume"
|
| 10 |
+
}
|
iclr-debug-ncp-medium-135b-v1/RUNBOOK.md
CHANGED
|
@@ -1,5 +1,10 @@
|
|
| 1 |
# Medium two-stage NCP on full OWT
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
Production output: `/home/ubuntu/mstok-results/iclr-debug-ncp-medium-135b-v1`.
|
| 4 |
Source checkout: `/home/ubuntu/mstok-ncp-medium-135b`, pinned to
|
| 5 |
`9ec67f19c85e37b60629f945f76b28dbf0e62e65`. Keep this checkout unchanged while running.
|
|
|
|
| 1 |
# Medium two-stage NCP on full OWT
|
| 2 |
|
| 3 |
+
**Paused at user request on September 21, 2026, 23:50 UTC at step 25,750
|
| 4 |
+
(27B input positions)** to give this node to the fresh small 50%→30% corruption
|
| 5 |
+
experiment. All in-flight training updates reached the saved milestone. The
|
| 6 |
+
pause record is `PAUSED_FOR_SMALL_RAMP.json` in the production directory.
|
| 7 |
+
|
| 8 |
Production output: `/home/ubuntu/mstok-results/iclr-debug-ncp-medium-135b-v1`.
|
| 9 |
Source checkout: `/home/ubuntu/mstok-ncp-medium-135b`, pinned to
|
| 10 |
`9ec67f19c85e37b60629f945f76b28dbf0e62e65`. Keep this checkout unchanged while running.
|