ctopd / docs /RUNTIME_B200.md
codemaivanngu's picture
fix(runtime): automatically select and ensure compatible node runtime
e1515b2 unverified
|
Raw History Blame Contribute Delete
7.53 kB
# Develop against the existing B200 runtime
Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime
- Artifact revision: `6058422a8319cbe03958bf5d52750293e4e0da65`.
- Downloader revision: `93a118a96df88fe7dd0852b50d5b62616b701aa8`.
- Archive: `ctopd-b200-cu129-v4/runtime.tar.gz`, 16,515,065,867 bytes.
- SHA-256: `084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3`.
Download/install directly on the target host using the pinned HF instructions. Required targets are Ubuntu 22.04/glibc 2.35 and Ubuntu 24.04/glibc 2.39, Linux x86_64, with g++ for JIT and a B200 driver compatible with CUDA 12.9. Ubuntu 22 uses the pinned 30,628,766-byte ABI overlay automatically; Ubuntu 24 retains the original archive. See [compatibility evidence](COMPATIBILITY.md). Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
## Restore/check on every new node
`ensure_runtime.sh` is self-contained and can be copied from this checkout into `/workspace/storage-shared/nlp/tungks/ctopd_all/ensure_runtime.sh`. The archive stays on shared storage; the default installed prefix `/opt/ctopd-b200` belongs to the current node/pod. Shared files do not imply that another node has the runtime installed.
The **same command works on Ubuntu 22 and 24**. No OS flag or manual prefix is
needed. It detects glibc and reuses a managed installation; on Ubuntu 22, a known
original unpatched v4 in `/opt/ctopd-b200` is preserved while a compatible sibling
`/opt/ctopd-b200-u22` is restored/reused automatically. A clean host still uses
`/opt/ctopd-b200` with the overlay applied when needed. Explicit `--prefix` or
`CTOPD_RUNTIME_PREFIX` remain overrides and are never silently changed.
```bash
ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all
cp ensure_runtime.sh "$ROOT/ensure_runtime.sh" # run once from this checkout
bash "$ROOT/ensure_runtime.sh" --gpu 2 # substitute your allocated GPU; omit if scheduler exposes one
```
Use the helper from the current committed source (or its pinned GitHub commit).
The historical HF helper at revision `4f12864097bb6f1092d27bf41d9e8a6357dcf2af`
predates automatic prefix selection. Replacing that small script is sufficient;
the runtime archive and compatibility overlay pins have not changed.
Run the last line on each new node, inside a pod exposing the one B200 allocated for this check. With no GPU allocated yet, use `--install-only`; it reports `RUNTIME_INSTALLED_UNCHECKED`, not GPU readiness. With an existing runtime, `--check-only` checks without extracting. `--help` lists options.
### Choose the allocated GPU on a shared node
The helper does not select physical GPU 0 by default. It retains the caller's `CUDA_VISIBLE_DEVICES`; CUDA's device 0 in the probe is the first device in that filtered view. Every helper version supports:
```bash
CUDA_VISIBLE_DEVICES=2 bash ensure_runtime.sh --check-only
```
The current source helper also supports an explicit physical GPU index or UUID:
```bash
bash ensure_runtime.sh --gpu 2 --check-only
bash ensure_runtime.sh --gpu GPU-your-allocated-uuid --check-only
```
`--gpu` explicitly replaces the inherited visibility mask for the check and any Python command passed after `--`. Without it, the inherited mask is preserved. It does not change the parent shell, allocate a GPU, or scan for an idle device. Choose the index/UUID assigned to you; leave the flag unset when the scheduler already exposes the correct single GPU. Keep the same visibility setting in a separately launched training command. The readiness receipt records the visibility mask.
The helper automatically finds either `$ROOT/ctopd-b200-cu129-v4/runtime.tar.gz` (the manually downloaded shard layout) or `$ROOT/runtime.tar.gz`. It verifies the pinned archive size/SHA-256 before extraction, checks Linux/glibc/g++, extracts safely to a new staging directory under the target filesystem, then renames to the final prefix. On glibc 2.35 through 2.38 it downloads the small verified overlay once into `$ROOT/compat/glibc235-v2/`; an existing valid overlay is reused. It uses `CTOPD_PROXY`, existing proxy variables, or the company proxy by default; set `CTOPD_PROXY=direct` for direct access. Allow at least 45 GiB free on that node-local filesystem. It never builds dependencies or launches training. Failed extraction keeps staging evidence. If `/opt` is not writable, choose an explicit node-local location:
```bash
export CTOPD_RUNTIME_PREFIX="/var/tmp/ctopd-b200-$UID"
bash "$ROOT/ensure_runtime.sh"
```
Concurrent setup calls use a lock for the same prefix. Reuse checks its archive-identity receipt, wrapper/Python/nvcc hashes and every substituted overlay file; it does not reread the 16.5 GB archive. Default readiness additionally verifies pinned package versions (including the router), NVRTC headers, imports and a one-element CUDA memory roundtrip on the allocated B200. This is a basic runtime check, not full kernel qualification or a training result. Existing incompatible/unmanaged prefixes are preserved. Automatic selection can use the known compatible sibling; unknown or damaged selected installs still fail without replacement. The helper does not modify an active installation in place and has no recursive-delete fallback.
The script can also execute a Python command after readiness:
```bash
bash "$ROOT/ensure_runtime.sh" --gpu 2 -- -c 'import torch; print(torch.__version__)'
```
It forwards the chosen runtime as `CTOPD_RUNTIME_PREFIX` and the shared data root
as `CTOPD_ROOT` to the child process. It cannot change the parent shell; using the
forwarded command avoids manually exporting a prefix. From the source checkout:
```bash
bash ensure_runtime.sh --gpu 2 -- -m tools.h1b.exact_strprob --help
CUDA_VISIBLE_DEVICES=2 python3 tools/launch_single_b200.py --algorithm h1b-c --run-id h1b-c-s42
```
The standard launcher now selects and ensures the runtime itself before training;
no separate setup command or `--runtime` is required. It checks model/data/output
paths and committed source first. Its `--dry-run` remains read-only and never
extracts or probes a GPU. The separate H1b canary launcher uses the same policy.
This is a fresh implementation for CTOPD's CUDA 12.9 bundle. The historical GOLD/SimCT helper used different CUDA 13 prefixes and a destructive repair fallback; those paths and that fallback are not copied into CTOPD. Live installation on the company node has not been executed from this chat. Offline archive-fixture tests and Bash syntax checks are separate from the real node readiness result.
## Use source without rebuilding dependencies
Run source from this checkout through the same automatic entry point:
```bash
cd /absolute/path/to/ctopd
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
bash ensure_runtime.sh -- -m kdflow.cli.train_kd_on_policy --help
```
The CLI imports CUDA dependencies and should be checked inside the allocated GPU environment. It is not a CPU-only test. Code/configuration edits do not require rebuilding dependencies; requalify runtime changes that affect CUDA, Torch, native extensions or ABI compatibility.
Before training, update the example scripts' source/model/data/output paths and actual GPU allocation. Their current hard-coded source prefix is `/workspace/Cross-Tokenizer-OPD`; change it to this checkout so they load your edits. The original scripts assume one node with eight GPUs. No training job is launched by these instructions.