feat(runtime): add idempotent node-local ensure helper
Browse files- README.md +2 -0
- docs/RUNTIME_B200.md +32 -1
- docs/SINGLE_B200.md +2 -2
- docs/UPSTREAM.md +1 -1
- ensure_runtime.SHA256SUMS +1 -0
- ensure_runtime.sh +187 -0
- tests/test_ensure_runtime.py +123 -0
- tests/test_single_b200.py +1 -1
- tools/launch_single_b200.py +3 -2
README.md
CHANGED
|
@@ -8,6 +8,8 @@ Start editing directly in `kdflow/algorithms/`, `kdflow/token_alignment.py`, `kd
|
|
| 8 |
|
| 9 |
Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
|
| 10 |
|
|
|
|
|
|
|
| 11 |
For a full-workload, one-run-per-B200 adaptation, use [the single-B200 launcher](docs/SINGLE_B200.md). It keeps 512-response rollout/optimizer batches, 8,192 generated tokens, 100 training iterations and 32-sample MATH500 evaluation. There is no reduced-workload mode. Single-B200 full training and paper metric agreement remain unvalidated.
|
| 12 |
|
| 13 |
## Original project documentation
|
|
|
|
| 8 |
|
| 9 |
Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
|
| 10 |
|
| 11 |
+
On a new company node, run the standalone `ensure_runtime.sh`: it restores the pinned shared archive into node-local `/opt/ctopd-b200` if absent, otherwise checks and reuses the existing installation. It preserves unknown/broken installations, uses no `pip`, and starts no training. See the instructions for shared-path placement, GPU visibility, and `--install-only` before allocation.
|
| 12 |
+
|
| 13 |
For a full-workload, one-run-per-B200 adaptation, use [the single-B200 launcher](docs/SINGLE_B200.md). It keeps 512-response rollout/optimizer batches, 8,192 generated tokens, 100 training iterations and 32-sample MATH500 evaluation. There is no reduced-workload mode. Single-B200 full training and paper metric agreement remain unvalidated.
|
| 14 |
|
| 15 |
## Original project documentation
|
docs/RUNTIME_B200.md
CHANGED
|
@@ -9,11 +9,42 @@ Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-op
|
|
| 9 |
|
| 10 |
Download/install directly on the target host using the pinned HF instructions. The archive requires Linux x86_64, glibc >= 2.39, g++ for JIT, and a B200 driver compatible with CUDA 12.9. Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
Once installed, point `RUNTIME` at the actual extracted prefix and run source from this checkout:
|
| 13 |
|
| 14 |
```bash
|
| 15 |
cd /absolute/path/to/ctopd
|
| 16 |
-
RUNTIME=/
|
| 17 |
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
|
| 18 |
bash "$RUNTIME/python-b200.sh" -m kdflow.cli.train_kd_on_policy --help
|
| 19 |
```
|
|
|
|
| 9 |
|
| 10 |
Download/install directly on the target host using the pinned HF instructions. The archive requires Linux x86_64, glibc >= 2.39, g++ for JIT, and a B200 driver compatible with CUDA 12.9. Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
|
| 11 |
|
| 12 |
+
## Restore/check on every new node
|
| 13 |
+
|
| 14 |
+
`ensure_runtime.sh` is self-contained and can be copied from this checkout into `/workspace/storage-shared/nlp/tungks/ctopd_all/ensure_runtime.sh`. The archive stays on shared storage; the default installed prefix `/opt/ctopd-b200` belongs to the current node/pod. Shared files do not imply that another node has the runtime installed.
|
| 15 |
+
|
| 16 |
+
```bash
|
| 17 |
+
ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all
|
| 18 |
+
cp ensure_runtime.sh "$ROOT/ensure_runtime.sh" # run once from this checkout
|
| 19 |
+
bash "$ROOT/ensure_runtime.sh"
|
| 20 |
+
```
|
| 21 |
+
|
| 22 |
+
Run the last line on each new node, inside a pod exposing the one B200 allocated for this check. With no GPU allocated yet, use `--install-only`; it reports `RUNTIME_INSTALLED_UNCHECKED`, not GPU readiness. With an existing runtime, `--check-only` checks without extracting. `--help` lists options.
|
| 23 |
+
|
| 24 |
+
The helper automatically finds either `$ROOT/ctopd-b200-cu129-v4/runtime.tar.gz` (the manually downloaded shard layout) or `$ROOT/runtime.tar.gz`. It verifies the pinned archive size/SHA-256 before extraction, checks Linux/glibc/g++, extracts safely to a new staging directory under the target filesystem, then renames to the final prefix. Allow at least 45 GiB free on that node-local filesystem. It never downloads dependencies, builds a runtime, launches training, or replaces an existing runtime that it cannot identify. Failed extraction keeps staging evidence. If `/opt` is not writable, choose an explicit node-local location:
|
| 25 |
+
|
| 26 |
+
```bash
|
| 27 |
+
export CTOPD_RUNTIME_PREFIX="/var/tmp/ctopd-b200-$UID"
|
| 28 |
+
bash "$ROOT/ensure_runtime.sh"
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
Concurrent setup calls use a lock for the same prefix. Reuse checks its archive-identity receipt and hashes of the wrapper, portable Python and nvcc; it does not rehash every installed dependency or reread the 16.5 GB archive. Default readiness additionally verifies the pinned Python/Torch/Transformers/SGLang/FlashAttention/FlashInfer versions, NVRTC headers, imports and a one-element CUDA memory roundtrip on the allocated B200. This is a basic runtime check, not full kernel qualification or a training result. Existing incompatible/unmanaged prefixes are preserved for manual inspection; there is no recursive-delete fallback.
|
| 32 |
+
|
| 33 |
+
The script can also execute a Python command after readiness:
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
bash "$ROOT/ensure_runtime.sh" -- -c 'import torch; print(torch.__version__)'
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
This is a fresh implementation for CTOPD's CUDA 12.9 bundle. The historical GOLD/SimCT helper used different CUDA 13 prefixes and a destructive repair fallback; those paths and that fallback are not copied into CTOPD. Live installation on the company node has not been executed from this chat. Offline archive-fixture tests and Bash syntax checks are separate from the real node readiness result.
|
| 40 |
+
|
| 41 |
+
## Use source without rebuilding dependencies
|
| 42 |
+
|
| 43 |
Once installed, point `RUNTIME` at the actual extracted prefix and run source from this checkout:
|
| 44 |
|
| 45 |
```bash
|
| 46 |
cd /absolute/path/to/ctopd
|
| 47 |
+
RUNTIME="${CTOPD_RUNTIME_PREFIX:-/opt/ctopd-b200}"
|
| 48 |
export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
|
| 49 |
bash "$RUNTIME/python-b200.sh" -m kdflow.cli.train_kd_on_policy --help
|
| 50 |
```
|
docs/SINGLE_B200.md
CHANGED
|
@@ -42,14 +42,14 @@ CPU RAM and `/dev/shm` also matter: the teacher forwards a whole rollout batch a
|
|
| 42 |
With `ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all`, prepare:
|
| 43 |
|
| 44 |
```text
|
| 45 |
-
|
| 46 |
$ROOT/models/Qwen2.5-7B-Instruct/ # complete official HF snapshot + tokenizer
|
| 47 |
$ROOT/models/Llama-3.2-3B-Instruct/ # complete official HF snapshot + tokenizer
|
| 48 |
$ROOT/data/Train/math_code_20k/train.parquet
|
| 49 |
$ROOT/data/math500/test.jsonl
|
| 50 |
```
|
| 51 |
|
| 52 |
-
Use `--runtime /actual/installed/prefix`
|
| 53 |
|
| 54 |
## Print the complete plan, then run it unchanged
|
| 55 |
|
|
|
|
| 42 |
With `ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all`, prepare:
|
| 43 |
|
| 44 |
```text
|
| 45 |
+
/opt/ctopd-b200/python-b200.sh # node-local; restore using ensure_runtime.sh
|
| 46 |
$ROOT/models/Qwen2.5-7B-Instruct/ # complete official HF snapshot + tokenizer
|
| 47 |
$ROOT/models/Llama-3.2-3B-Instruct/ # complete official HF snapshot + tokenizer
|
| 48 |
$ROOT/data/Train/math_code_20k/train.parquet
|
| 49 |
$ROOT/data/math500/test.jsonl
|
| 50 |
```
|
| 51 |
|
| 52 |
+
The launcher defaults to `CTOPD_RUNTIME_PREFIX` when set, otherwise `/opt/ctopd-b200`. Use `--runtime /actual/installed/prefix` for an existing alternative installation, including the earlier `$ROOT/runtime` layout. Run the [runtime restore/check helper](RUNTIME_B200.md) on every new node. The runtime tar.gz alone is not an installed prefix. Models/data must be downloaded directly on the runtime host; the Llama HF token must belong to an account with approved model access.
|
| 53 |
|
| 54 |
## Print the complete plan, then run it unchanged
|
| 55 |
|
docs/UPSTREAM.md
CHANGED
|
@@ -8,7 +8,7 @@
|
|
| 8 |
- The source service did not expose an upstream Git commit. The local import commit starts a new history and does not claim to preserve upstream Git history.
|
| 9 |
- `docs/upstream-source-manifest.json` records original byte sizes and hashes. It describes the initial import; later development commits may intentionally change those files.
|
| 10 |
|
| 11 |
-
The initial commit preserves the imported implementation. Local additions provide documentation, line-ending attributes, a single-B200 launcher
|
| 12 |
|
| 13 |
## License metadata
|
| 14 |
|
|
|
|
| 8 |
- The source service did not expose an upstream Git commit. The local import commit starts a new history and does not claim to preserve upstream Git history.
|
| 9 |
- `docs/upstream-source-manifest.json` records original byte sizes and hashes. It describes the initial import; later development commits may intentionally change those files.
|
| 10 |
|
| 11 |
+
The initial commit preserves the imported implementation. Local additions provide documentation, line-ending attributes, a single-B200 launcher, a standalone node-local runtime setup helper and offline tests. Imported `kdflow/` code and the two original training scripts remain unchanged. Datasets and figures were excluded from the source snapshot, and no runtime archives, model weights, credentials, account metadata or experiment outputs are included.
|
| 12 |
|
| 13 |
## License metadata
|
| 14 |
|
ensure_runtime.SHA256SUMS
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
9a388b3ca183e0e771a49bdc4b61d9c6880d043da5490122c8fdf4098c3b9895 ensure_runtime.sh
|
ensure_runtime.sh
ADDED
|
@@ -0,0 +1,187 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# Standalone node-local restore/check entry point. Safe to copy into ctopd_all/.
|
| 3 |
+
set -euo pipefail
|
| 4 |
+
if [[ "$(uname -s)" != Linux || "$(uname -m)" != x86_64 ]]; then
|
| 5 |
+
echo 'CTOPD requires Linux x86_64; do not install this runtime on Windows.' >&2
|
| 6 |
+
exit 2
|
| 7 |
+
fi
|
| 8 |
+
SYSTEM_PY=""
|
| 9 |
+
for candidate in python3.12 python3; do
|
| 10 |
+
if command -v "$candidate" >/dev/null 2>&1; then SYSTEM_PY="$(command -v "$candidate")"; break; fi
|
| 11 |
+
done
|
| 12 |
+
[[ -n "$SYSTEM_PY" ]] || { echo 'Need a system Python 3 interpreter.' >&2; exit 2; }
|
| 13 |
+
exec "$SYSTEM_PY" - "$@" <<'CTOPD_ENSURE_PY'
|
| 14 |
+
import argparse
|
| 15 |
+
import copy
|
| 16 |
+
import datetime
|
| 17 |
+
import fcntl
|
| 18 |
+
import hashlib
|
| 19 |
+
import json
|
| 20 |
+
import os
|
| 21 |
+
import pathlib
|
| 22 |
+
import platform
|
| 23 |
+
import shutil
|
| 24 |
+
import subprocess
|
| 25 |
+
import sys
|
| 26 |
+
import tarfile
|
| 27 |
+
import tempfile
|
| 28 |
+
|
| 29 |
+
EXPECTED_SHA = '084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3'
|
| 30 |
+
EXPECTED_BYTES = 16515065867
|
| 31 |
+
MIN_FREE_BYTES = 45 * 1024**3
|
| 32 |
+
RECEIPT = '.ctopd-runtime.json'
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def digest(path):
|
| 36 |
+
result = hashlib.sha256()
|
| 37 |
+
with path.open('rb') as stream:
|
| 38 |
+
for block in iter(lambda: stream.read(8 * 1024 * 1024), b''):
|
| 39 |
+
result.update(block)
|
| 40 |
+
return result.hexdigest()
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def check_host():
|
| 44 |
+
libc, version = platform.libc_ver()
|
| 45 |
+
if libc != 'glibc' or tuple(map(int, version.split('.')[:2])) < (2, 39):
|
| 46 |
+
raise RuntimeError('Need glibc >= 2.39 (Ubuntu 24.04 baseline); change pod image, not system libc')
|
| 47 |
+
if not shutil.which('g++'):
|
| 48 |
+
raise RuntimeError('Host g++ is required for runtime JIT; use a prepared node image')
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def fingerprints(prefix):
|
| 52 |
+
interpreters = list(prefix.glob('python/cpython-3.12.12-*/bin/python3.12'))
|
| 53 |
+
if len(interpreters) != 1:
|
| 54 |
+
raise RuntimeError('Portable Python is missing or ambiguous in ' + str(prefix))
|
| 55 |
+
paths = [prefix / 'python-b200.sh', interpreters[0], prefix / 'cuda/bin/nvcc']
|
| 56 |
+
for path in paths:
|
| 57 |
+
if not path.is_file():
|
| 58 |
+
raise RuntimeError('Missing runtime component: ' + str(path))
|
| 59 |
+
return {str(path.relative_to(prefix)): digest(path) for path in paths}
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def restore(archive, prefix):
|
| 63 |
+
if not hasattr(tarfile, 'data_filter'):
|
| 64 |
+
raise RuntimeError('System Python needs safe tar extraction filters (Python 3.12 or patched 3.10/3.11)')
|
| 65 |
+
if archive.stat().st_size != EXPECTED_BYTES or digest(archive) != EXPECTED_SHA:
|
| 66 |
+
raise RuntimeError('Archive size/SHA-256 mismatch; reassemble numbered parts correctly before installing')
|
| 67 |
+
print('ARCHIVE_SHA256_OK=' + EXPECTED_SHA, flush=True)
|
| 68 |
+
if shutil.disk_usage(prefix.parent).free < MIN_FREE_BYTES:
|
| 69 |
+
raise RuntimeError('Need at least 45 GiB free on the target filesystem for extraction')
|
| 70 |
+
stage = pathlib.Path(tempfile.mkdtemp(prefix='.' + prefix.name + '.install-', dir=prefix.parent))
|
| 71 |
+
print('EXTRACTING_TO=' + str(stage), flush=True)
|
| 72 |
+
try:
|
| 73 |
+
with tarfile.open(archive, 'r:gz') as bundle:
|
| 74 |
+
for original in bundle:
|
| 75 |
+
path = pathlib.PurePosixPath(original.name)
|
| 76 |
+
if path.is_absolute() or '..' in path.parts or path.parts[:2] != ('opt', 'ctopd-b200'):
|
| 77 |
+
raise ValueError('Unexpected archive member: ' + str(path))
|
| 78 |
+
if len(path.parts) == 2:
|
| 79 |
+
continue
|
| 80 |
+
member = copy.copy(original)
|
| 81 |
+
member.name = str(pathlib.PurePosixPath(*path.parts[2:]))
|
| 82 |
+
if member.islnk():
|
| 83 |
+
link = pathlib.PurePosixPath(member.linkname)
|
| 84 |
+
if link.is_absolute() or '..' in link.parts or link.parts[:2] != ('opt', 'ctopd-b200'):
|
| 85 |
+
raise ValueError('Unsafe hardlink: ' + member.linkname)
|
| 86 |
+
member.linkname = str(pathlib.PurePosixPath(*link.parts[2:]))
|
| 87 |
+
bundle.extract(member, stage, filter='data')
|
| 88 |
+
receipt = {'archive_sha256': EXPECTED_SHA, 'archive_bytes': EXPECTED_BYTES,
|
| 89 |
+
'installed_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
| 90 |
+
'critical_files_sha256': fingerprints(stage)}
|
| 91 |
+
(stage / RECEIPT).write_text(json.dumps(receipt, indent=2) + '\n')
|
| 92 |
+
if prefix.exists() or prefix.is_symlink():
|
| 93 |
+
raise FileExistsError(prefix)
|
| 94 |
+
stage.rename(prefix)
|
| 95 |
+
except Exception:
|
| 96 |
+
print('INSTALL_FAILED: staging retained for inspection at ' + str(stage), file=sys.stderr)
|
| 97 |
+
raise
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
PROBE = '''import importlib, importlib.metadata as md, json, os, pathlib, sys
|
| 101 |
+
import torch
|
| 102 |
+
assert torch.cuda.device_count() == 1, 'Expose exactly the B200 allocated to this check'
|
| 103 |
+
assert 'B200' in torch.cuda.get_device_name(0), 'Expected a B200'
|
| 104 |
+
expected = {'torch': '2.11.0+cu129', 'transformers': '5.6.2', 'sglang': '0.5.12.post1',
|
| 105 |
+
'flash-attn': '2.8.3', 'flashinfer-python': '0.6.11.post1'}
|
| 106 |
+
versions = {name: md.version(name) for name in expected}
|
| 107 |
+
assert versions == expected, ('Runtime package mismatch', versions)
|
| 108 |
+
assert sys.version_info[:3] == (3, 12, 12), sys.version
|
| 109 |
+
cuda = pathlib.Path(os.environ['CUDA_HOME'])
|
| 110 |
+
assert (cuda / 'include/nvrtc.h').is_file(), 'Missing NVRTC headers'
|
| 111 |
+
for name in ['transformers', 'sglang.srt.entrypoints.engine', 'flash_attn', 'flashinfer']:
|
| 112 |
+
importlib.import_module(name)
|
| 113 |
+
assert torch.version.cuda == '12.9', torch.version.cuda
|
| 114 |
+
value = torch.tensor([1.0], device='cuda')
|
| 115 |
+
assert value.cpu().item() == 1.0
|
| 116 |
+
print(json.dumps({'versions': versions, 'python': sys.version.split()[0],
|
| 117 |
+
'cuda': torch.version.cuda, 'gpu': torch.cuda.get_device_name(0),
|
| 118 |
+
'check': 'imports + CUDA memory roundtrip; no training or full kernel suite'}))
|
| 119 |
+
'''
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def main(argv=None):
|
| 123 |
+
parser = argparse.ArgumentParser(description='Restore/check the pinned CTOPD runtime on this node; no pip, build or training')
|
| 124 |
+
parser.add_argument('--root', type=pathlib.Path,
|
| 125 |
+
default=pathlib.Path(os.environ.get('CTOPD_ROOT', '/workspace/storage-shared/nlp/tungks/ctopd_all')))
|
| 126 |
+
parser.add_argument('--prefix', type=pathlib.Path,
|
| 127 |
+
default=pathlib.Path(os.environ.get('CTOPD_RUNTIME_PREFIX', '/opt/ctopd-b200')))
|
| 128 |
+
parser.add_argument('--archive', type=pathlib.Path)
|
| 129 |
+
mode = parser.add_mutually_exclusive_group()
|
| 130 |
+
mode.add_argument('--install-only', action='store_true', help='Restore without a GPU check; reports INSTALLED_UNCHECKED')
|
| 131 |
+
mode.add_argument('--check-only', action='store_true', help='Check an existing managed runtime; never extract')
|
| 132 |
+
parser.add_argument('python_args', nargs=argparse.REMAINDER, help='After --, execute these Python arguments through the runtime wrapper')
|
| 133 |
+
args = parser.parse_args(argv)
|
| 134 |
+
forwarded = args.python_args[1:] if args.python_args[:1] == ['--'] else args.python_args
|
| 135 |
+
if args.install_only and forwarded:
|
| 136 |
+
raise ValueError('--install-only cannot also execute a runtime command')
|
| 137 |
+
raw = args.prefix.expanduser()
|
| 138 |
+
if not raw.is_absolute() or raw.is_symlink():
|
| 139 |
+
raise ValueError('Prefix must be an absolute non-symlink directory')
|
| 140 |
+
prefix = raw.resolve()
|
| 141 |
+
if len(prefix.parts) < 3 or prefix == args.root.expanduser().resolve():
|
| 142 |
+
raise ValueError('Refusing an unsafe runtime prefix: ' + str(prefix))
|
| 143 |
+
check_host()
|
| 144 |
+
if args.check_only and not prefix.exists():
|
| 145 |
+
raise FileNotFoundError('Runtime is absent; run without --check-only to restore it')
|
| 146 |
+
prefix.parent.mkdir(parents=True, exist_ok=True)
|
| 147 |
+
lock_path = prefix.parent / ('.' + prefix.name + '.ensure.lock')
|
| 148 |
+
if lock_path.is_symlink():
|
| 149 |
+
raise ValueError('Refusing symlink lock file')
|
| 150 |
+
with lock_path.open('a') as lock:
|
| 151 |
+
print('WAITING_FOR_RUNTIME_LOCK=' + str(lock_path), flush=True)
|
| 152 |
+
fcntl.flock(lock, fcntl.LOCK_EX)
|
| 153 |
+
if prefix.exists():
|
| 154 |
+
if not prefix.is_dir() or not (prefix / RECEIPT).is_file():
|
| 155 |
+
raise RuntimeError('Existing unmanaged prefix preserved; choose another --prefix or inspect it manually: ' + str(prefix))
|
| 156 |
+
receipt = json.loads((prefix / RECEIPT).read_text())
|
| 157 |
+
if receipt.get('archive_sha256') != EXPECTED_SHA or receipt.get('critical_files_sha256') != fingerprints(prefix):
|
| 158 |
+
raise RuntimeError('Existing runtime identity/integrity check failed; preserved without replacement')
|
| 159 |
+
print('REUSING_RUNTIME=' + str(prefix), flush=True)
|
| 160 |
+
else:
|
| 161 |
+
if args.check_only:
|
| 162 |
+
raise FileNotFoundError('Runtime disappeared while waiting for its lock; no extraction performed')
|
| 163 |
+
root = args.root.expanduser().resolve()
|
| 164 |
+
candidates = [root / 'ctopd-b200-cu129-v4/runtime.tar.gz', root / 'runtime.tar.gz']
|
| 165 |
+
archive = args.archive.expanduser().resolve() if args.archive else next((p for p in candidates if p.is_file()), None)
|
| 166 |
+
if archive is None or not archive.is_file():
|
| 167 |
+
raise FileNotFoundError('No runtime.tar.gz in ' + str(root) + '; download once to shared storage or pass --archive')
|
| 168 |
+
restore(archive, prefix)
|
| 169 |
+
if args.install_only:
|
| 170 |
+
print('RUNTIME_INSTALLED_UNCHECKED=' + str(prefix), flush=True)
|
| 171 |
+
else:
|
| 172 |
+
env = os.environ.copy()
|
| 173 |
+
env.update(PYTHONPATH='', HF_HUB_OFFLINE='1', TRANSFORMERS_OFFLINE='1', PYTHONNOUSERSITE='1')
|
| 174 |
+
output = subprocess.check_output(['bash', str(prefix / 'python-b200.sh'), '-c', PROBE], env=env, text=True)
|
| 175 |
+
result = json.loads(output.strip().splitlines()[-1])
|
| 176 |
+
result['checked_utc'] = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
| 177 |
+
(prefix / '.ctopd-last-check.json').write_text(json.dumps(result, indent=2) + '\n')
|
| 178 |
+
print(output, end='', flush=True)
|
| 179 |
+
print('RUNTIME_READY=' + str(prefix), flush=True)
|
| 180 |
+
print('RUNTIME_PYTHON_WRAPPER=' + str(prefix / 'python-b200.sh'), flush=True)
|
| 181 |
+
if forwarded:
|
| 182 |
+
os.execvpe('bash', ['bash', str(prefix / 'python-b200.sh'), *forwarded], os.environ.copy())
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
if __name__ == '__main__':
|
| 186 |
+
main()
|
| 187 |
+
CTOPD_ENSURE_PY
|
tests/test_ensure_runtime.py
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tiny archive fixtures; never download/install the production runtime locally."""
|
| 2 |
+
import ast
|
| 3 |
+
import contextlib
|
| 4 |
+
import hashlib
|
| 5 |
+
import io
|
| 6 |
+
import json
|
| 7 |
+
import pathlib
|
| 8 |
+
import tempfile
|
| 9 |
+
import unittest
|
| 10 |
+
|
| 11 |
+
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
| 12 |
+
SCRIPT = (ROOT / 'ensure_runtime.sh').read_text()
|
| 13 |
+
INLINE = SCRIPT.split("<<'CTOPD_ENSURE_PY'\n", 1)[1].rsplit('\nCTOPD_ENSURE_PY', 1)[0]
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
class EnsureRuntimeTests(unittest.TestCase):
|
| 17 |
+
def setUp(self):
|
| 18 |
+
self.temp = tempfile.TemporaryDirectory()
|
| 19 |
+
self.root = pathlib.Path(self.temp.name)
|
| 20 |
+
self.archive = self.root / 'runtime.tar.gz'
|
| 21 |
+
self.prefix = self.root / 'node-local/runtime'
|
| 22 |
+
self.code = {'__name__': 'ensure_fixture'}
|
| 23 |
+
exec(compile(INLINE, 'ensure_runtime.sh:inline', 'exec'), self.code)
|
| 24 |
+
# Fixture installation only: no real GPU/ML runtime or 45 GiB allocation.
|
| 25 |
+
self.code['check_host'] = lambda: None
|
| 26 |
+
self.code['MIN_FREE_BYTES'] = 0
|
| 27 |
+
|
| 28 |
+
def tearDown(self):
|
| 29 |
+
self.temp.cleanup()
|
| 30 |
+
|
| 31 |
+
def archive_with(self, extra=None):
|
| 32 |
+
import tarfile
|
| 33 |
+
files = {'opt/ctopd-b200/python-b200.sh': b'#!/bin/bash\n',
|
| 34 |
+
'opt/ctopd-b200/python/cpython-3.12.12-fixture/bin/python3.12': b'fixture-python',
|
| 35 |
+
'opt/ctopd-b200/cuda/bin/nvcc': b'fixture-nvcc'}
|
| 36 |
+
if extra:
|
| 37 |
+
files.update(extra)
|
| 38 |
+
with tarfile.open(self.archive, 'w:gz') as bundle:
|
| 39 |
+
for name, content in files.items():
|
| 40 |
+
member = tarfile.TarInfo(name)
|
| 41 |
+
member.size = len(content)
|
| 42 |
+
bundle.addfile(member, io.BytesIO(content))
|
| 43 |
+
self.code['EXPECTED_SHA'] = hashlib.sha256(self.archive.read_bytes()).hexdigest()
|
| 44 |
+
self.code['EXPECTED_BYTES'] = self.archive.stat().st_size
|
| 45 |
+
|
| 46 |
+
def run_ensure(self, *options):
|
| 47 |
+
with contextlib.redirect_stdout(io.StringIO()) as output:
|
| 48 |
+
self.code['main'](['--root', str(self.root), '--prefix', str(self.prefix), *options])
|
| 49 |
+
return output.getvalue()
|
| 50 |
+
|
| 51 |
+
def test_inline_and_probe_syntax(self):
|
| 52 |
+
ast.parse(INLINE)
|
| 53 |
+
ast.parse(self.code['PROBE'])
|
| 54 |
+
|
| 55 |
+
def test_install_and_reuse_without_archive(self):
|
| 56 |
+
self.archive_with()
|
| 57 |
+
output = self.run_ensure('--install-only')
|
| 58 |
+
self.assertIn('RUNTIME_INSTALLED_UNCHECKED=', output)
|
| 59 |
+
self.assertNotIn('RUNTIME_READY=', output)
|
| 60 |
+
receipt = self.prefix / self.code['RECEIPT']
|
| 61 |
+
before = receipt.read_bytes()
|
| 62 |
+
self.archive.unlink()
|
| 63 |
+
output = self.run_ensure('--install-only')
|
| 64 |
+
self.assertIn('REUSING_RUNTIME=', output)
|
| 65 |
+
self.assertEqual(receipt.read_bytes(), before)
|
| 66 |
+
|
| 67 |
+
def test_wrong_checksum_never_installs(self):
|
| 68 |
+
self.archive_with()
|
| 69 |
+
self.code['EXPECTED_SHA'] = '0' * 64
|
| 70 |
+
with self.assertRaisesRegex(RuntimeError, 'SHA-256 mismatch'):
|
| 71 |
+
self.run_ensure('--install-only')
|
| 72 |
+
self.assertFalse(self.prefix.exists())
|
| 73 |
+
self.assertFalse(list(self.prefix.parent.glob('.runtime.install-*')))
|
| 74 |
+
|
| 75 |
+
def test_unmanaged_existing_prefix_preserved(self):
|
| 76 |
+
self.prefix.mkdir(parents=True)
|
| 77 |
+
content = self.prefix / 'user-data'
|
| 78 |
+
content.write_text('keep')
|
| 79 |
+
with self.assertRaisesRegex(RuntimeError, 'unmanaged prefix preserved'):
|
| 80 |
+
self.run_ensure('--install-only')
|
| 81 |
+
self.assertEqual(content.read_text(), 'keep')
|
| 82 |
+
|
| 83 |
+
def test_changed_runtime_preserved_and_rejected(self):
|
| 84 |
+
self.archive_with()
|
| 85 |
+
self.run_ensure('--install-only')
|
| 86 |
+
wrapper = self.prefix / 'python-b200.sh'
|
| 87 |
+
wrapper.write_text('user change')
|
| 88 |
+
with self.assertRaisesRegex(RuntimeError, 'identity/integrity check failed'):
|
| 89 |
+
self.run_ensure('--install-only')
|
| 90 |
+
self.assertEqual(wrapper.read_text(), 'user change')
|
| 91 |
+
|
| 92 |
+
def test_traversal_rejected_and_evidence_retained(self):
|
| 93 |
+
self.archive_with({'opt/ctopd-b200/../escape': b'bad'})
|
| 94 |
+
with contextlib.redirect_stderr(io.StringIO()):
|
| 95 |
+
with self.assertRaisesRegex(ValueError, 'Unexpected archive member'):
|
| 96 |
+
self.run_ensure('--install-only')
|
| 97 |
+
self.assertFalse(self.prefix.exists())
|
| 98 |
+
self.assertFalse((self.prefix.parent / 'escape').exists())
|
| 99 |
+
self.assertEqual(len(list(self.prefix.parent.glob('.runtime.install-*'))), 1)
|
| 100 |
+
|
| 101 |
+
def test_check_only_does_not_install_or_create_parent(self):
|
| 102 |
+
with self.assertRaises(FileNotFoundError):
|
| 103 |
+
self.run_ensure('--check-only')
|
| 104 |
+
self.assertFalse(self.prefix.parent.exists())
|
| 105 |
+
|
| 106 |
+
def test_escaping_symlink_rejected(self):
|
| 107 |
+
import tarfile
|
| 108 |
+
self.archive_with()
|
| 109 |
+
with tarfile.open(self.archive, 'w:gz') as bundle:
|
| 110 |
+
member = tarfile.TarInfo('opt/ctopd-b200/escape')
|
| 111 |
+
member.type = tarfile.SYMTYPE
|
| 112 |
+
member.linkname = '/tmp/outside-ctopd'
|
| 113 |
+
bundle.addfile(member)
|
| 114 |
+
self.code['EXPECTED_SHA'] = hashlib.sha256(self.archive.read_bytes()).hexdigest()
|
| 115 |
+
self.code['EXPECTED_BYTES'] = self.archive.stat().st_size
|
| 116 |
+
with contextlib.redirect_stderr(io.StringIO()):
|
| 117 |
+
with self.assertRaises(tarfile.FilterError):
|
| 118 |
+
self.run_ensure('--install-only')
|
| 119 |
+
self.assertFalse(self.prefix.exists())
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
if __name__ == '__main__':
|
| 123 |
+
unittest.main()
|
tests/test_single_b200.py
CHANGED
|
@@ -67,7 +67,7 @@ class LauncherTests(unittest.TestCase):
|
|
| 67 |
with self.assertRaises(ValueError):
|
| 68 |
self.plan('--run-id', '../other-run')
|
| 69 |
with tempfile.TemporaryDirectory() as folder:
|
| 70 |
-
plan = self.plan('--root', folder)
|
| 71 |
for key in ['student_name_or_path', 'teacher_name_or_path']:
|
| 72 |
model = pathlib.Path(plan['flags'][key])
|
| 73 |
model.mkdir(parents=True)
|
|
|
|
| 67 |
with self.assertRaises(ValueError):
|
| 68 |
self.plan('--run-id', '../other-run')
|
| 69 |
with tempfile.TemporaryDirectory() as folder:
|
| 70 |
+
plan = self.plan('--root', folder, '--runtime', str(pathlib.Path(folder) / 'runtime'))
|
| 71 |
for key in ['student_name_or_path', 'teacher_name_or_path']:
|
| 72 |
model = pathlib.Path(plan['flags'][key])
|
| 73 |
model.mkdir(parents=True)
|
tools/launch_single_b200.py
CHANGED
|
@@ -23,7 +23,8 @@ def parser():
|
|
| 23 |
p = argparse.ArgumentParser(description=__doc__)
|
| 24 |
p.add_argument('--root', type=pathlib.Path,
|
| 25 |
default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
|
| 26 |
-
p.add_argument('--runtime', type=pathlib.Path,
|
|
|
|
| 27 |
p.add_argument('--algorithm', choices=['strict-top16', 'strict-full'], default='strict-top16')
|
| 28 |
p.add_argument('--seed', type=int, default=42, help='Original launcher default, not a verified paper seed')
|
| 29 |
p.add_argument('--micro-batch', type=int, choices=[1, 2, 4], default=4,
|
|
@@ -43,7 +44,7 @@ def build_plan(args):
|
|
| 43 |
if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,95}', run_id):
|
| 44 |
raise ValueError('run-id must be a single directory name starting with a letter or digit')
|
| 45 |
root = args.root.expanduser().resolve()
|
| 46 |
-
runtime = (args.runtime or
|
| 47 |
output = root / 'runs' / run_id
|
| 48 |
flags = {
|
| 49 |
'num_nodes': 1, 'num_gpus_per_node': 1, 'backend': 'fsdp2',
|
|
|
|
| 23 |
p = argparse.ArgumentParser(description=__doc__)
|
| 24 |
p.add_argument('--root', type=pathlib.Path,
|
| 25 |
default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
|
| 26 |
+
p.add_argument('--runtime', type=pathlib.Path,
|
| 27 |
+
help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
|
| 28 |
p.add_argument('--algorithm', choices=['strict-top16', 'strict-full'], default='strict-top16')
|
| 29 |
p.add_argument('--seed', type=int, default=42, help='Original launcher default, not a verified paper seed')
|
| 30 |
p.add_argument('--micro-batch', type=int, choices=[1, 2, 4], default=4,
|
|
|
|
| 44 |
if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,95}', run_id):
|
| 45 |
raise ValueError('run-id must be a single directory name starting with a letter or digit')
|
| 46 |
root = args.root.expanduser().resolve()
|
| 47 |
+
runtime = (args.runtime or pathlib.Path(os.environ.get('CTOPD_RUNTIME_PREFIX', '/opt/ctopd-b200'))).expanduser().resolve()
|
| 48 |
output = root / 'runs' / run_id
|
| 49 |
flags = {
|
| 50 |
'num_nodes': 1, 'num_gpus_per_node': 1, 'backend': 'fsdp2',
|