| # Runbook |
|
|
| Read `HANDOVER.md` first for context and current results. This file is commands only. |
|
|
| ## Local setup |
|
|
| ```bash |
| uv venv --python 3.12 .venv |
| uv pip install --python .venv/bin/python -e . torch torchvision timm opencv-python-headless "numpy<2" pandas |
| ``` |
|
|
| Common environment prefix for local runs: |
|
|
| ```bash |
| export PMDM_DATA=Task1/PackagingMaterialDifferenceMiningDataset |
| export PMDM_WORK=/tmp/pmdm_work |
| export PMDM_CKPT=/tmp/pmdm_ckpt |
| ``` |
|
|
| | Script | Purpose | Runtime | |
| |---|---|---| |
| | `scripts/local_smoke.py` | CPU pre-flight: preprocess, dataset, model forward/backward, decode, scorer, generator | ~2 min | |
| | `scripts/analyze_dataset.py` | Reproduces every dataset claim in HANDOVER §2 | ~2 min | |
| | `scripts/error_analysis.py <oof.npz>` | Error breakdown by cause and box size | seconds | |
| | `scripts/train_local.py --preprocess-only` | Stage 0 over all 300 pairs, no Modal | ~10 min | |
| | `scripts/train_local.py --fold 0 --epochs 40` | Same training loop as Modal, on a local GPU | ~45 min on A100 | |
|
|
| Archived logs and artifacts from every run so far are in `runs/` — see `runs/README.md`. |
|
|
| ## Modal |
|
|
| Data upload (once per account, ~2.5 GB): |
|
|
| ```bash |
| modal volume create pmdm-data |
| modal volume create pmdm-ckpt |
| modal volume put pmdm-data Task1/PackagingMaterialDifferenceMiningDataset/train /raw/train |
| modal volume put pmdm-data Task1/PackagingMaterialDifferenceMiningDataset/test /raw/test |
| ``` |
|
|
| | Step | Command | Hardware | Measured time | |
| |---|---|---|---| |
| | Stage 0 preprocessing | `modal run modal_app.py::prep` | CPU fan-out ×300 | few minutes | |
| | Shape/loss check | `modal run modal_app.py::smoke` | L4 | ~2 min | |
| | Synthetic data | `modal run modal_app.py::synth --shards 16 --per-shard 400` | CPU fan-out | ~20 min (not yet run) | |
| | Train one fold | `modal run --detach modal_app.py::train --fold 0 --epochs 40` | A100-40GB | ~45 min | |
| | Train with synthetic | `modal run --detach modal_app.py::train --fold 0 --epochs 40 --n-synth 4000` | A100-40GB | ~60 min | |
| | All folds | `modal run --detach modal_app.py::train_all --epochs 40` | 5 × A100-40GB | ~45 min wall | |
| | Out-of-fold candidates | `modal run modal_app.py::oof --fold 0` | L4 | ~3 min | |
| | Verifier A/B (honest) | `modal run modal_app.py::verify_ab --fold 0` | L4 | ~5 min | |
| | Verifier (all folds) | `modal run modal_app.py::verify` | L4 | ~15 min | |
| | Submission | `modal run modal_app.py::predict --folds 0,1,2,3,4 --threshold 0.31` | L4 | ~10 min | |
|
|
| Fetch results: |
|
|
| ```bash |
| modal volume get pmdm-ckpt /submission.csv ./submission.csv |
| modal volume get pmdm-ckpt /stage1_fold0_convnext_tiny/history.json ./history.json |
| modal volume get pmdm-ckpt /oof/fold0_convnext_tiny.npz /tmp/oof.npz |
| ``` |
|
|
| The trained weights are already mirrored into `checkpoints/` in this repo — see |
| `checkpoints/README.md`. Re-mirror after a new run, or push a local checkpoint back up: |
|
|
| ```bash |
| modal volume get pmdm-ckpt /stage1_fold0_convnext_tiny checkpoints/stage1_fold0_convnext_tiny |
| modal volume put pmdm-ckpt checkpoints/stage1_fold0_convnext_tiny /stage1_fold0_convnext_tiny |
| ``` |
|
|
| Use `modal volume get` for backups, never `modal volume cp` — it refuses directories while still |
| exiting 0, which is how one checkpoint was lost (`HANDOVER.md` §6). |
|
|
| ## Launch discipline |
|
|
| - **Always `--detach` for training.** A local network blip otherwise kills the run. |
| - **Confirm training actually started** — look for `[fold N] epoch 0 step 0/160` in the logs. An app |
| that reports "Created function ..." and then stops has trained nothing. |
| - Stream logs of a detached run: `modal app logs <app-id>`. |
| - `train_fold` commits the volume after every epoch and resumes automatically from `last.pt`, so an |
| interruption costs at most one epoch. To force a fresh run, delete the checkpoint directory first — |
| and back it up with `modal volume get`, not `modal volume cp` (which refuses directories while |
| still exiting 0). |
|
|
| ## Hugging Face mirror |
|
|
| The full working state lives in the private repo <https://huggingface.co/siddhant20/task1>: code, |
| docs, `runs/`, and all checkpoints. Give a teammate a collaborator invite on `siddhant20` and this |
| one link replaces both the repo tarball and Modal access. |
|
|
| ```bash |
| hf download siddhant20/task1 --local-dir task1 # everything |
| hf download siddhant20/task1 checkpoints/stage1_fold0_convnext_tiny/best.pt --local-dir . |
| ``` |
|
|
| The dataset is uploaded separately because 600 page-scale PNGs at 2.5 GB take hours on a home |
| uplink. Run it when the machine can stay awake; it is resumable, so a killed run continues rather |
| than restarting: |
|
|
| ```bash |
| .venv/bin/python scripts/push_dataset_to_hf.py |
| ``` |
|
|
| Re-push code and weights after a new run: |
|
|
| ```bash |
| python - <<'PY' |
| from huggingface_hub import HfApi |
| HfApi().upload_large_folder( |
| repo_id="siddhant20/task1", repo_type="model", folder_path=".", |
| ignore_patterns=[".venv/**", "__pycache__/**", "**/__pycache__/**", |
| ".DS_Store", "**/.DS_Store", "*.zip", ".git/**", "Task1/**"]) |
| PY |
| ``` |
|
|
| ## Order of operations |
|
|
| 1. `prep`, then `smoke`. The smoke test must show `hm_shape` at output stride 2 and finite losses. |
| 2. `train --fold 0` and read the out-of-fold F1 in `history.json`. Current reference: **0.905**. |
| 3. `synth`, then retrain fold 0 with `--n-synth 4000`. Keep the change only if F1 beats 0.905. |
| 4. `oof` per trained fold, then `verify_ab` to measure the verifier honestly before trusting it. |
| 5. `train_all`, then `predict`. |
|
|
| `--threshold` for `predict` comes from the `threshold` field in `/ckpt/oof/fold*.json`, not a guess. |
|
|
