|
Download docs/RUNTIME_B200.md from codemaivanngu/ctopd: direct link, hf CLI and curl.
- Browser
- Download file 7.53 kB
-
https://huggingface.co/codemaivanngu/ctopd/resolve/main/docs/RUNTIME_B200.md
- Command line
-
hf download hf://codemaivanngu/ctopd/docs/RUNTIME_B200.md
-
curl -L -o RUNTIME_B200.md https://huggingface.co/codemaivanngu/ctopd/resolve/main/docs/RUNTIME_B200.md
7.53 kB
| # Develop against the existing B200 runtime | |
| Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime | |
| - Artifact revision: `6058422a8319cbe03958bf5d52750293e4e0da65`. | |
| - Downloader revision: `93a118a96df88fe7dd0852b50d5b62616b701aa8`. | |
| - Archive: `ctopd-b200-cu129-v4/runtime.tar.gz`, 16,515,065,867 bytes. | |
| - SHA-256: `084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3`. | |
| Download/install directly on the target host using the pinned HF instructions. Required targets are Ubuntu 22.04/glibc 2.35 and Ubuntu 24.04/glibc 2.39, Linux x86_64, with g++ for JIT and a B200 driver compatible with CUDA 12.9. Ubuntu 22 uses the pinned 30,628,766-byte ABI overlay automatically; Ubuntu 24 retains the original archive. See [compatibility evidence](COMPATIBILITY.md). Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate. | |
| ## Restore/check on every new node | |
| `ensure_runtime.sh` is self-contained and can be copied from this checkout into `/workspace/storage-shared/nlp/tungks/ctopd_all/ensure_runtime.sh`. The archive stays on shared storage; the default installed prefix `/opt/ctopd-b200` belongs to the current node/pod. Shared files do not imply that another node has the runtime installed. | |
| The **same command works on Ubuntu 22 and 24**. No OS flag or manual prefix is | |
| needed. It detects glibc and reuses a managed installation; on Ubuntu 22, a known | |
| original unpatched v4 in `/opt/ctopd-b200` is preserved while a compatible sibling | |
| `/opt/ctopd-b200-u22` is restored/reused automatically. A clean host still uses | |
| `/opt/ctopd-b200` with the overlay applied when needed. Explicit `--prefix` or | |
| `CTOPD_RUNTIME_PREFIX` remain overrides and are never silently changed. | |
| ```bash | |
| ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all | |
| cp ensure_runtime.sh "$ROOT/ensure_runtime.sh" # run once from this checkout | |
| bash "$ROOT/ensure_runtime.sh" --gpu 2 # substitute your allocated GPU; omit if scheduler exposes one | |
| ``` | |
| Use the helper from the current committed source (or its pinned GitHub commit). | |
| The historical HF helper at revision `4f12864097bb6f1092d27bf41d9e8a6357dcf2af` | |
| predates automatic prefix selection. Replacing that small script is sufficient; | |
| the runtime archive and compatibility overlay pins have not changed. | |
| Run the last line on each new node, inside a pod exposing the one B200 allocated for this check. With no GPU allocated yet, use `--install-only`; it reports `RUNTIME_INSTALLED_UNCHECKED`, not GPU readiness. With an existing runtime, `--check-only` checks without extracting. `--help` lists options. | |
| ### Choose the allocated GPU on a shared node | |
| The helper does not select physical GPU 0 by default. It retains the caller's `CUDA_VISIBLE_DEVICES`; CUDA's device 0 in the probe is the first device in that filtered view. Every helper version supports: | |
| ```bash | |
| CUDA_VISIBLE_DEVICES=2 bash ensure_runtime.sh --check-only | |
| ``` | |
| The current source helper also supports an explicit physical GPU index or UUID: | |
| ```bash | |
| bash ensure_runtime.sh --gpu 2 --check-only | |
| bash ensure_runtime.sh --gpu GPU-your-allocated-uuid --check-only | |
| ``` | |
| `--gpu` explicitly replaces the inherited visibility mask for the check and any Python command passed after `--`. Without it, the inherited mask is preserved. It does not change the parent shell, allocate a GPU, or scan for an idle device. Choose the index/UUID assigned to you; leave the flag unset when the scheduler already exposes the correct single GPU. Keep the same visibility setting in a separately launched training command. The readiness receipt records the visibility mask. | |
| The helper automatically finds either `$ROOT/ctopd-b200-cu129-v4/runtime.tar.gz` (the manually downloaded shard layout) or `$ROOT/runtime.tar.gz`. It verifies the pinned archive size/SHA-256 before extraction, checks Linux/glibc/g++, extracts safely to a new staging directory under the target filesystem, then renames to the final prefix. On glibc 2.35 through 2.38 it downloads the small verified overlay once into `$ROOT/compat/glibc235-v2/`; an existing valid overlay is reused. It uses `CTOPD_PROXY`, existing proxy variables, or the company proxy by default; set `CTOPD_PROXY=direct` for direct access. Allow at least 45 GiB free on that node-local filesystem. It never builds dependencies or launches training. Failed extraction keeps staging evidence. If `/opt` is not writable, choose an explicit node-local location: | |
| ```bash | |
| export CTOPD_RUNTIME_PREFIX="/var/tmp/ctopd-b200-$UID" | |
| bash "$ROOT/ensure_runtime.sh" | |
| ``` | |
| Concurrent setup calls use a lock for the same prefix. Reuse checks its archive-identity receipt, wrapper/Python/nvcc hashes and every substituted overlay file; it does not reread the 16.5 GB archive. Default readiness additionally verifies pinned package versions (including the router), NVRTC headers, imports and a one-element CUDA memory roundtrip on the allocated B200. This is a basic runtime check, not full kernel qualification or a training result. Existing incompatible/unmanaged prefixes are preserved. Automatic selection can use the known compatible sibling; unknown or damaged selected installs still fail without replacement. The helper does not modify an active installation in place and has no recursive-delete fallback. | |
| The script can also execute a Python command after readiness: | |
| ```bash | |
| bash "$ROOT/ensure_runtime.sh" --gpu 2 -- -c 'import torch; print(torch.__version__)' | |
| ``` | |
| It forwards the chosen runtime as `CTOPD_RUNTIME_PREFIX` and the shared data root | |
| as `CTOPD_ROOT` to the child process. It cannot change the parent shell; using the | |
| forwarded command avoids manually exporting a prefix. From the source checkout: | |
| ```bash | |
| bash ensure_runtime.sh --gpu 2 -- -m tools.h1b.exact_strprob --help | |
| CUDA_VISIBLE_DEVICES=2 python3 tools/launch_single_b200.py --algorithm h1b-c --run-id h1b-c-s42 | |
| ``` | |
| The standard launcher now selects and ensures the runtime itself before training; | |
| no separate setup command or `--runtime` is required. It checks model/data/output | |
| paths and committed source first. Its `--dry-run` remains read-only and never | |
| extracts or probes a GPU. The separate H1b canary launcher uses the same policy. | |
| This is a fresh implementation for CTOPD's CUDA 12.9 bundle. The historical GOLD/SimCT helper used different CUDA 13 prefixes and a destructive repair fallback; those paths and that fallback are not copied into CTOPD. Live installation on the company node has not been executed from this chat. Offline archive-fixture tests and Bash syntax checks are separate from the real node readiness result. | |
| ## Use source without rebuilding dependencies | |
| Run source from this checkout through the same automatic entry point: | |
| ```bash | |
| cd /absolute/path/to/ctopd | |
| export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}" | |
| bash ensure_runtime.sh -- -m kdflow.cli.train_kd_on_policy --help | |
| ``` | |
| The CLI imports CUDA dependencies and should be checked inside the allocated GPU environment. It is not a CPU-only test. Code/configuration edits do not require rebuilding dependencies; requalify runtime changes that affect CUDA, Torch, native extensions or ABI compatibility. | |
| Before training, update the example scripts' source/model/data/output paths and actual GPU allocation. Their current hard-coded source prefix is `/workspace/Cross-Tokenizer-OPD`; change it to this checkout so they load your edits. The original scripts assume one node with eight GPUs. No training job is launched by these instructions. | |