Allocates ~30 GB on a chosen GPU and idles, so the slot stays ours while
v2 training is queued. Uses the existing survivecity-train image so no
extra torch install is needed on the host.
./scripts/gpu_hold.sh 3 # hold GPU 3
docker rm -f gpu-hold-3 # release
After a Colab/Kaggle session dies mid-training, the user needs a fast way
to confirm what made it to the HF Hub before the crash and to pull the
latest checkpoint locally.
scripts/check_hub_checkpoints.py:
--hub-model-id noanya/zombiee # default action: list
--hub-model-id noanya/zombiee --info # show step / loss / lr
# from trainer_state.json
--hub-model-id noanya/zombiee --download DIR # snapshot the latest
--checkpoint N # operate on a specific step
Honors HUGGINGFACE_TOKEN / HF_TOKEN for private repos. Prints a ready-to-run
`python -m training.train --resume-from-checkpoint <path>` command after a
successful download so resume on the DGX is one paste away.
notebooks/README.md:
- "My Colab/Kaggle session died — did I lose anything?" section pointing
at the recovery script.
- Document scripts/dgx_autorun.sh's tunables (MIN_FREE_GB, MAX_JOBS,
POLL_INTERVAL,…
The LNM-IIT DGX is shared and unpredictable — a GPU can be 100% full for
hours and then free up for 10 minutes. This script runs as a long-lived
watcher on the DGX login node:
- Polls nvidia-smi every POLL_INTERVAL (default 60 s).
- When any GPU has >= MIN_FREE_GB (default 10) free, launches the
SurviveCity training docker container pinned to that GPU with
--gpus '"device=N"'.
- Runs as detached containers named survivecity-train-gpuN, so the
watcher keeps scanning and can start additional runs on other GPUs
that clear the threshold, up to MAX_JOBS (default 1).
- Every container is wired to --resume-from-checkpoint <HUB_MODEL_ID>
and --push-to-hub to the same HF Hub repo, so runs that die mid-step
(OOM, preempted, host reboot) restart from the last checkpoint with
no data loss.
- Per-GPU host mount at OUTPUT_ROOT/gpuN so multiple concurrent
containers don't clobber each other's local copies; HF Hub is the
shared source of truth.
- Ctrl-C trap tears down all container…
