fix(runtime): support Ubuntu 22.04 with a qualified ABI overlay
Browse files- README.md +1 -1
- docs/COMPATIBILITY.md +29 -0
- docs/RUNTIME_B200.md +5 -5
- ensure_runtime.SHA256SUMS +1 -1
- ensure_runtime.sh +92 -7
- tests/test_ensure_runtime.py +98 -1
- tools/launch_single_b200.py +7 -0
README.md
CHANGED
|
@@ -8,7 +8,7 @@ Start editing directly in `kdflow/algorithms/`, `kdflow/token_alignment.py`, `kd
|
|
| 8 |
|
| 9 |
Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
|
| 10 |
|
| 11 |
-
On a new company node, run the standalone `ensure_runtime.sh`: it restores the pinned shared archive into node-local `/opt/ctopd-b200` if absent, otherwise checks and reuses the existing installation. It preserves unknown/broken installations, uses no `pip`, and starts no training. See the instructions for shared-path placement, GPU visibility, and `--install-only` before allocation.
|
| 12 |
|
| 13 |
For a full-workload, one-run-per-B200 adaptation, use [the single-B200 launcher](docs/SINGLE_B200.md). It keeps 512-response rollout/optimizer batches, 8,192 generated tokens, 100 training iterations and 32-sample MATH500 evaluation. There is no reduced-workload mode. Single-B200 full training and paper metric agreement remain unvalidated.
|
| 14 |
|
|
|
|
| 8 |
|
| 9 |
Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
|
| 10 |
|
| 11 |
+
On a new company node, run the standalone `ensure_runtime.sh`: it restores the pinned shared archive into node-local `/opt/ctopd-b200` if absent, otherwise checks and reuses the existing installation. Ubuntu 22.04/glibc 2.35 automatically uses a verified 30.6 MB compatibility overlay; Ubuntu 24.04/glibc 2.39 retains the original runtime. It preserves unknown/broken installations, uses no `pip`, and starts no training. See [compatibility requirements and evidence](docs/COMPATIBILITY.md) and the runtime instructions for shared-path placement, GPU visibility, and `--install-only` before allocation.
|
| 12 |
|
| 13 |
For a full-workload, one-run-per-B200 adaptation, use [the single-B200 launcher](docs/SINGLE_B200.md). It keeps 512-response rollout/optimizer batches, 8,192 generated tokens, 100 training iterations and 32-sample MATH500 evaluation. There is no reduced-workload mode. Single-B200 full training and paper metric agreement remain unvalidated.
|
| 14 |
|
docs/COMPATIBILITY.md
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Required node compatibility targets
|
| 2 |
+
|
| 3 |
+
Both baselines are requirements for the CTOPD runtime, recorded at the user's request on 2026-10-19:
|
| 4 |
+
|
| 5 |
+
| Required target | Evidence and current boundary |
|
| 6 |
+
| --- | --- |
|
| 7 |
+
| Ubuntu 22.04.5 LTS, glibc 2.35, Linux x86_64, B200 | User supplied OS/libc evidence from `anhhbt-quant-0-0`. The fixed runtime passed all six infrastructure probes on a real Modal B200 with Ubuntu 22.04/glibc 2.35, plus a CPU router protocol check. The actual company node still needs the delivered helper check. |
|
| 8 |
+
| Ubuntu 24.04, glibc 2.39, Linux x86_64, B200 | Original v4 passed six B200 infrastructure probes. The original artifact/path is retained and passed a fresh CPU Torch/Transformers/router-import regression on glibc 2.39. No new full-training result is implied. |
|
| 9 |
+
|
| 10 |
+
## Root cause of the Ubuntu 22.04 failure
|
| 11 |
+
|
| 12 |
+
The v4 package copied host libraries from the Ubuntu 24 author image. A read-only ELF audit found 99 bundled host libraries with GLIBC requirements above 2.35, plus `sglang_router/sglang_router_rs.abi3.so` requiring GLIBC 2.39. All other audited runtime ELF objects had requirements no higher than 2.35. Changing a shell guard cannot change these ELF requirements.
|
| 13 |
+
|
| 14 |
+
The overlay retains Python/Torch/CUDA versions and the research implementation. It substitutes a glibc-2.35 C++/gcc/zlib set and the Python/native router package from one pinned `sglang-router==0.3.2` manylinux wheel. The original image's Python router API differs despite the same version label: replacing only the binary fails on the unsupported `json_log` constructor keyword. The qualified overlay includes the matched Python files. CTOPD's actual round-robin startup, worker registration and `/generate` forwarding protocol passed; the research code is unchanged. Host libc is never bundled or replaced.
|
| 15 |
+
|
| 16 |
+
## Immutable overlay and evidence
|
| 17 |
+
|
| 18 |
+
- HF revision: `3b1d90bc044874dc20c65308ec72841fc3db22e9`.
|
| 19 |
+
- File: `compat/glibc235-v2/overlay.tar.gz`, 30,628,766 bytes.
|
| 20 |
+
- SHA-256: `21cc38ecdfc2f61714922127b67f3a0be89bf70bb2178b8fda8d3659e14f76c7`.
|
| 21 |
+
- [Full qualification receipt](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime/blob/3b1d90bc044874dc20c65308ec72841fc3db22e9/compat/glibc235-v2/qualification.json) and adjacent overlay manifest contain image IDs, dependency hashes and logs.
|
| 22 |
+
- The Ubuntu 22 GPU suite passed in 80.143386344 seconds: BF16, imports, packed FlashAttention forward/backward, FlashInfer, FSDP2/top-16 reverse-KL, patched SGLang hidden states/logprobs/sleep-wake. It used synthetic fixtures, not paper training. After the pass markers, Python emitted a multiprocessing resource-tracker cleanup `ChildProcessError`; the process still exited zero. Full company training remains unverified.
|
| 23 |
+
- The helper restored the complete original archive plus overlay on clean Ubuntu 22 in 399.609342267 seconds, reused that install in 0.244060964 seconds, and passed actual Torch CPU and router protocol checks. The complete CPU path took 412.366908248 seconds. [Restore receipt](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime/blob/3f6ca2db515415208742f0a1846a761ab361728d/compat/glibc235-v2/restore-qualification.json) tests installation independently of the separately qualified B200 image.
|
| 24 |
+
|
| 25 |
+
The helper selects the overlay for glibc 2.35 through 2.38 and keeps the original v4 path for glibc >= 2.39. Only 2.35 and 2.39 have live baseline evidence. It caches the overlay once on shared storage, checks its pinned bytes/hash, quarantines original host libraries inside the new installation, and records all substituted file hashes. The base archive is unchanged and does not need another download.
|
| 26 |
+
|
| 27 |
+
Preserve the original Ubuntu 24 path. The helper must require the verified overlay on older supported glibc hosts, rather than treating the unpatched v4 package as compatible after lowering a version check. Existing incompatible installs must be preserved and require a fresh explicit prefix, so active runtime files are not changed in place.
|
| 28 |
+
|
| 29 |
+
The exact company node is a reported reproduction target, not a lease or proof of available GPU capacity. Live free-GPU claims from earlier transcripts are not allocation evidence.
|
docs/RUNTIME_B200.md
CHANGED
|
@@ -7,7 +7,7 @@ Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-op
|
|
| 7 |
- Archive: `ctopd-b200-cu129-v4/runtime.tar.gz`, 16,515,065,867 bytes.
|
| 8 |
- SHA-256: `084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3`.
|
| 9 |
|
| 10 |
-
Download/install directly on the target host using the pinned HF instructions.
|
| 11 |
|
| 12 |
## Restore/check on every new node
|
| 13 |
|
|
@@ -25,25 +25,25 @@ Alternatively, download the same standalone helper through the public HF mirror
|
|
| 25 |
set -euo pipefail
|
| 26 |
cd /workspace/storage-shared/nlp/tungks/ctopd_all
|
| 27 |
PROXY="${HTTPS_PROXY:-${HTTP_PROXY:-http://10.30.154.118:80}}"
|
| 28 |
-
BASE=https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime/resolve/
|
| 29 |
curl -fL --retry 3 --proxy "$PROXY" --noproxy "" "$BASE/ensure_runtime.sh" -o ensure_runtime.sh
|
| 30 |
curl -fL --retry 3 --proxy "$PROXY" --noproxy "" "$BASE/ensure_runtime.SHA256SUMS" -o ensure_runtime.SHA256SUMS
|
| 31 |
sha256sum -c ensure_runtime.SHA256SUMS
|
| 32 |
bash ensure_runtime.sh
|
| 33 |
```
|
| 34 |
|
| 35 |
-
Published helper SHA-256: `
|
| 36 |
|
| 37 |
Run the last line on each new node, inside a pod exposing the one B200 allocated for this check. With no GPU allocated yet, use `--install-only`; it reports `RUNTIME_INSTALLED_UNCHECKED`, not GPU readiness. With an existing runtime, `--check-only` checks without extracting. `--help` lists options.
|
| 38 |
|
| 39 |
-
The helper automatically finds either `$ROOT/ctopd-b200-cu129-v4/runtime.tar.gz` (the manually downloaded shard layout) or `$ROOT/runtime.tar.gz`. It verifies the pinned archive size/SHA-256 before extraction, checks Linux/glibc/g++, extracts safely to a new staging directory under the target filesystem, then renames to the final prefix. Allow at least 45 GiB free on that node-local filesystem. It never
|
| 40 |
|
| 41 |
```bash
|
| 42 |
export CTOPD_RUNTIME_PREFIX="/var/tmp/ctopd-b200-$UID"
|
| 43 |
bash "$ROOT/ensure_runtime.sh"
|
| 44 |
```
|
| 45 |
|
| 46 |
-
Concurrent setup calls use a lock for the same prefix. Reuse checks its archive-identity receipt
|
| 47 |
|
| 48 |
The script can also execute a Python command after readiness:
|
| 49 |
|
|
|
|
| 7 |
- Archive: `ctopd-b200-cu129-v4/runtime.tar.gz`, 16,515,065,867 bytes.
|
| 8 |
- SHA-256: `084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3`.
|
| 9 |
|
| 10 |
+
Download/install directly on the target host using the pinned HF instructions. Required targets are Ubuntu 22.04/glibc 2.35 and Ubuntu 24.04/glibc 2.39, Linux x86_64, with g++ for JIT and a B200 driver compatible with CUDA 12.9. Ubuntu 22 uses the pinned 30,628,766-byte ABI overlay automatically; Ubuntu 24 retains the original archive. See [compatibility evidence](COMPATIBILITY.md). Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
|
| 11 |
|
| 12 |
## Restore/check on every new node
|
| 13 |
|
|
|
|
| 25 |
set -euo pipefail
|
| 26 |
cd /workspace/storage-shared/nlp/tungks/ctopd_all
|
| 27 |
PROXY="${HTTPS_PROXY:-${HTTP_PROXY:-http://10.30.154.118:80}}"
|
| 28 |
+
BASE=https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime/resolve/3f6ca2db515415208742f0a1846a761ab361728d
|
| 29 |
curl -fL --retry 3 --proxy "$PROXY" --noproxy "" "$BASE/ensure_runtime.sh" -o ensure_runtime.sh
|
| 30 |
curl -fL --retry 3 --proxy "$PROXY" --noproxy "" "$BASE/ensure_runtime.SHA256SUMS" -o ensure_runtime.SHA256SUMS
|
| 31 |
sha256sum -c ensure_runtime.SHA256SUMS
|
| 32 |
bash ensure_runtime.sh
|
| 33 |
```
|
| 34 |
|
| 35 |
+
Published helper SHA-256: `a09e1c27bd7ee6f2919b49b48d03ff2a3ebb81444aa5f578dc828624f0568bf8`. Anonymous HF download was checked byte-for-byte against the source; this verifies delivery, not execution on the company node.
|
| 36 |
|
| 37 |
Run the last line on each new node, inside a pod exposing the one B200 allocated for this check. With no GPU allocated yet, use `--install-only`; it reports `RUNTIME_INSTALLED_UNCHECKED`, not GPU readiness. With an existing runtime, `--check-only` checks without extracting. `--help` lists options.
|
| 38 |
|
| 39 |
+
The helper automatically finds either `$ROOT/ctopd-b200-cu129-v4/runtime.tar.gz` (the manually downloaded shard layout) or `$ROOT/runtime.tar.gz`. It verifies the pinned archive size/SHA-256 before extraction, checks Linux/glibc/g++, extracts safely to a new staging directory under the target filesystem, then renames to the final prefix. On glibc 2.35 through 2.38 it downloads the small verified overlay once into `$ROOT/compat/glibc235-v2/`; an existing valid overlay is reused. It uses `CTOPD_PROXY`, existing proxy variables, or the company proxy by default; set `CTOPD_PROXY=direct` for direct access. Allow at least 45 GiB free on that node-local filesystem. It never builds dependencies or launches training. Failed extraction keeps staging evidence. If `/opt` is not writable, choose an explicit node-local location:
|
| 40 |
|
| 41 |
```bash
|
| 42 |
export CTOPD_RUNTIME_PREFIX="/var/tmp/ctopd-b200-$UID"
|
| 43 |
bash "$ROOT/ensure_runtime.sh"
|
| 44 |
```
|
| 45 |
|
| 46 |
+
Concurrent setup calls use a lock for the same prefix. Reuse checks its archive-identity receipt, wrapper/Python/nvcc hashes and every substituted overlay file; it does not reread the 16.5 GB archive. Default readiness additionally verifies pinned package versions (including the router), NVRTC headers, imports and a one-element CUDA memory roundtrip on the allocated B200. This is a basic runtime check, not full kernel qualification or a training result. Existing incompatible/unmanaged prefixes are preserved. An already-installed original v4 on glibc 2.35 requires a fresh explicit node-local `--prefix`; the helper does not modify an active installation in place. There is no recursive-delete fallback.
|
| 47 |
|
| 48 |
The script can also execute a Python command after readiness:
|
| 49 |
|
ensure_runtime.SHA256SUMS
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
|
|
|
|
| 1 |
+
a09e1c27bd7ee6f2919b49b48d03ff2a3ebb81444aa5f578dc828624f0568bf8 ensure_runtime.sh
|
ensure_runtime.sh
CHANGED
|
@@ -30,6 +30,10 @@ EXPECTED_SHA = '084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3
|
|
| 30 |
EXPECTED_BYTES = 16515065867
|
| 31 |
MIN_FREE_BYTES = 45 * 1024**3
|
| 32 |
RECEIPT = '.ctopd-runtime.json'
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
|
| 35 |
def digest(path):
|
|
@@ -42,10 +46,78 @@ def digest(path):
|
|
| 42 |
|
| 43 |
def check_host():
|
| 44 |
libc, version = platform.libc_ver()
|
| 45 |
-
|
| 46 |
-
|
|
|
|
| 47 |
if not shutil.which('g++'):
|
| 48 |
raise RuntimeError('Host g++ is required for runtime JIT; use a prepared node image')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
|
| 51 |
def fingerprints(prefix):
|
|
@@ -59,7 +131,7 @@ def fingerprints(prefix):
|
|
| 59 |
return {str(path.relative_to(prefix)): digest(path) for path in paths}
|
| 60 |
|
| 61 |
|
| 62 |
-
def restore(archive, prefix):
|
| 63 |
if not hasattr(tarfile, 'data_filter'):
|
| 64 |
raise RuntimeError('System Python needs safe tar extraction filters (Python 3.12 or patched 3.10/3.11)')
|
| 65 |
if archive.stat().st_size != EXPECTED_BYTES or digest(archive) != EXPECTED_SHA:
|
|
@@ -85,9 +157,12 @@ def restore(archive, prefix):
|
|
| 85 |
raise ValueError('Unsafe hardlink: ' + member.linkname)
|
| 86 |
member.linkname = str(pathlib.PurePosixPath(*link.parts[2:]))
|
| 87 |
bundle.extract(member, stage, filter='data')
|
|
|
|
| 88 |
receipt = {'archive_sha256': EXPECTED_SHA, 'archive_bytes': EXPECTED_BYTES,
|
| 89 |
'installed_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
| 90 |
'critical_files_sha256': fingerprints(stage)}
|
|
|
|
|
|
|
| 91 |
(stage / RECEIPT).write_text(json.dumps(receipt, indent=2) + '\n')
|
| 92 |
if prefix.exists() or prefix.is_symlink():
|
| 93 |
raise FileExistsError(prefix)
|
|
@@ -102,13 +177,13 @@ import torch
|
|
| 102 |
assert torch.cuda.device_count() == 1, 'Expose exactly the B200 allocated to this check'
|
| 103 |
assert 'B200' in torch.cuda.get_device_name(0), 'Expected a B200'
|
| 104 |
expected = {'torch': '2.11.0+cu129', 'transformers': '5.6.2', 'sglang': '0.5.12.post1',
|
| 105 |
-
'flash-attn': '2.8.3', 'flashinfer-python': '0.6.11.post1'}
|
| 106 |
versions = {name: md.version(name) for name in expected}
|
| 107 |
assert versions == expected, ('Runtime package mismatch', versions)
|
| 108 |
assert sys.version_info[:3] == (3, 12, 12), sys.version
|
| 109 |
cuda = pathlib.Path(os.environ['CUDA_HOME'])
|
| 110 |
assert (cuda / 'include/nvrtc.h').is_file(), 'Missing NVRTC headers'
|
| 111 |
-
for name in ['transformers', 'sglang.srt.entrypoints.engine', 'flash_attn', 'flashinfer']:
|
| 112 |
importlib.import_module(name)
|
| 113 |
assert torch.version.cuda == '12.9', torch.version.cuda
|
| 114 |
value = torch.tensor([1.0], device='cuda')
|
|
@@ -140,7 +215,8 @@ def main(argv=None):
|
|
| 140 |
prefix = raw.resolve()
|
| 141 |
if len(prefix.parts) < 3 or prefix == args.root.expanduser().resolve():
|
| 142 |
raise ValueError('Refusing an unsafe runtime prefix: ' + str(prefix))
|
| 143 |
-
check_host()
|
|
|
|
| 144 |
if args.check_only and not prefix.exists():
|
| 145 |
raise FileNotFoundError('Runtime is absent; run without --check-only to restore it')
|
| 146 |
prefix.parent.mkdir(parents=True, exist_ok=True)
|
|
@@ -156,6 +232,14 @@ def main(argv=None):
|
|
| 156 |
receipt = json.loads((prefix / RECEIPT).read_text())
|
| 157 |
if receipt.get('archive_sha256') != EXPECTED_SHA or receipt.get('critical_files_sha256') != fingerprints(prefix):
|
| 158 |
raise RuntimeError('Existing runtime identity/integrity check failed; preserved without replacement')
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
print('REUSING_RUNTIME=' + str(prefix), flush=True)
|
| 160 |
else:
|
| 161 |
if args.check_only:
|
|
@@ -165,7 +249,8 @@ def main(argv=None):
|
|
| 165 |
archive = args.archive.expanduser().resolve() if args.archive else next((p for p in candidates if p.is_file()), None)
|
| 166 |
if archive is None or not archive.is_file():
|
| 167 |
raise FileNotFoundError('No runtime.tar.gz in ' + str(root) + '; download once to shared storage or pass --archive')
|
| 168 |
-
|
|
|
|
| 169 |
if args.install_only:
|
| 170 |
print('RUNTIME_INSTALLED_UNCHECKED=' + str(prefix), flush=True)
|
| 171 |
else:
|
|
|
|
| 30 |
EXPECTED_BYTES = 16515065867
|
| 31 |
MIN_FREE_BYTES = 45 * 1024**3
|
| 32 |
RECEIPT = '.ctopd-runtime.json'
|
| 33 |
+
# Filled from the independently qualified compatibility artifact before publication.
|
| 34 |
+
COMPAT_SHA = '21cc38ecdfc2f61714922127b67f3a0be89bf70bb2178b8fda8d3659e14f76c7'
|
| 35 |
+
COMPAT_BYTES = 30628766
|
| 36 |
+
COMPAT_REVISION = '3b1d90bc044874dc20c65308ec72841fc3db22e9'
|
| 37 |
|
| 38 |
|
| 39 |
def digest(path):
|
|
|
|
| 46 |
|
| 47 |
def check_host():
|
| 48 |
libc, version = platform.libc_ver()
|
| 49 |
+
parsed = tuple(map(int, version.split('.')[:2])) if libc == 'glibc' else (0, 0)
|
| 50 |
+
if parsed < (2, 35):
|
| 51 |
+
raise RuntimeError('Supported baselines: Ubuntu 22.04/glibc 2.35 and Ubuntu 24.04/glibc 2.39; do not replace system libc')
|
| 52 |
if not shutil.which('g++'):
|
| 53 |
raise RuntimeError('Host g++ is required for runtime JIT; use a prepared node image')
|
| 54 |
+
return parsed
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def compatibility_overlay(root):
|
| 58 |
+
if not COMPAT_SHA or not COMPAT_BYTES or not COMPAT_REVISION:
|
| 59 |
+
raise RuntimeError('glibc 2.35 requires the qualified ABI overlay; this helper has no published overlay pin yet')
|
| 60 |
+
path = root / 'compat/glibc235-v2/overlay.tar.gz'
|
| 61 |
+
if path.is_file() and path.stat().st_size == COMPAT_BYTES and digest(path) == COMPAT_SHA:
|
| 62 |
+
return path
|
| 63 |
+
if path.exists():
|
| 64 |
+
raise RuntimeError('Existing ABI overlay differs from the pinned checksum; preserved: ' + str(path))
|
| 65 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 66 |
+
fd, name = tempfile.mkstemp(prefix='.overlay-download-', dir=path.parent)
|
| 67 |
+
os.close(fd)
|
| 68 |
+
temporary = pathlib.Path(name)
|
| 69 |
+
url = ('https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime/resolve/'
|
| 70 |
+
+ COMPAT_REVISION + '/compat/glibc235-v2/overlay.tar.gz')
|
| 71 |
+
proxy = (os.environ.get('CTOPD_PROXY') or os.environ.get('HTTPS_PROXY') or os.environ.get('https_proxy')
|
| 72 |
+
or os.environ.get('HTTP_PROXY') or 'http://10.30.154.118:80')
|
| 73 |
+
command = ['curl', '-fL', '--retry', '3', url, '-o', str(temporary)]
|
| 74 |
+
if proxy.lower() != 'direct':
|
| 75 |
+
command += ['--proxy', proxy, '--noproxy', '']
|
| 76 |
+
else:
|
| 77 |
+
command += ['--noproxy', '*']
|
| 78 |
+
try:
|
| 79 |
+
print('DOWNLOADING_ABI_OVERLAY_BYTES=' + str(COMPAT_BYTES), flush=True)
|
| 80 |
+
subprocess.run(command, check=True)
|
| 81 |
+
if temporary.stat().st_size != COMPAT_BYTES or digest(temporary) != COMPAT_SHA:
|
| 82 |
+
raise RuntimeError('Downloaded ABI overlay size/SHA-256 mismatch')
|
| 83 |
+
temporary.replace(path)
|
| 84 |
+
finally:
|
| 85 |
+
temporary.unlink(missing_ok=True)
|
| 86 |
+
return path
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def apply_overlay(stage, archive):
|
| 90 |
+
if archive.stat().st_size != COMPAT_BYTES or digest(archive) != COMPAT_SHA:
|
| 91 |
+
raise RuntimeError('ABI overlay checksum mismatch')
|
| 92 |
+
disabled = stage / 'host-libs-v4-disabled'
|
| 93 |
+
if disabled.exists():
|
| 94 |
+
raise FileExistsError(disabled)
|
| 95 |
+
(stage / 'host-libs').rename(disabled)
|
| 96 |
+
with tarfile.open(archive, 'r:gz') as bundle:
|
| 97 |
+
for original in bundle:
|
| 98 |
+
path = pathlib.PurePosixPath(original.name)
|
| 99 |
+
if path.is_absolute() or '..' in path.parts or path.parts[:2] != ('opt', 'ctopd-b200'):
|
| 100 |
+
raise ValueError('Unexpected ABI overlay member: ' + str(path))
|
| 101 |
+
if len(path.parts) == 2:
|
| 102 |
+
continue
|
| 103 |
+
relative = pathlib.PurePosixPath(*path.parts[2:])
|
| 104 |
+
allowed = (relative.parts[0] == 'host-libs' or str(relative) == 'glibc235-overlay-manifest.json'
|
| 105 |
+
or (relative.parts[:2] == ('site-packages', 'sglang_router')
|
| 106 |
+
and (str(relative).endswith('.py') or str(relative) == 'site-packages/sglang_router/sglang_router_rs.abi3.so'))
|
| 107 |
+
or relative.parts[:2] == ('site-packages', 'sglang_router.libs'))
|
| 108 |
+
if not allowed or not (original.isfile() or original.isdir()):
|
| 109 |
+
raise ValueError('Disallowed ABI overlay member: ' + str(path))
|
| 110 |
+
member = copy.copy(original)
|
| 111 |
+
member.name = str(relative)
|
| 112 |
+
bundle.extract(member, stage, filter='data')
|
| 113 |
+
manifest = json.loads((stage / 'glibc235-overlay-manifest.json').read_text())
|
| 114 |
+
if manifest['base_archive_sha256'] != EXPECTED_SHA or manifest['minimum_glibc'] != '2.35':
|
| 115 |
+
raise RuntimeError('ABI overlay targets a different base runtime')
|
| 116 |
+
for row in manifest['files']:
|
| 117 |
+
relative = pathlib.PurePosixPath(row['path'])
|
| 118 |
+
if relative.is_absolute() or '..' in relative.parts or digest(stage / relative) != row['sha256']:
|
| 119 |
+
raise RuntimeError('ABI overlay payload integrity mismatch')
|
| 120 |
+
return {'sha256': COMPAT_SHA, 'files': manifest['files'], 'minimum_glibc': '2.35'}
|
| 121 |
|
| 122 |
|
| 123 |
def fingerprints(prefix):
|
|
|
|
| 131 |
return {str(path.relative_to(prefix)): digest(path) for path in paths}
|
| 132 |
|
| 133 |
|
| 134 |
+
def restore(archive, prefix, overlay=None):
|
| 135 |
if not hasattr(tarfile, 'data_filter'):
|
| 136 |
raise RuntimeError('System Python needs safe tar extraction filters (Python 3.12 or patched 3.10/3.11)')
|
| 137 |
if archive.stat().st_size != EXPECTED_BYTES or digest(archive) != EXPECTED_SHA:
|
|
|
|
| 157 |
raise ValueError('Unsafe hardlink: ' + member.linkname)
|
| 158 |
member.linkname = str(pathlib.PurePosixPath(*link.parts[2:]))
|
| 159 |
bundle.extract(member, stage, filter='data')
|
| 160 |
+
compat = apply_overlay(stage, overlay) if overlay else None
|
| 161 |
receipt = {'archive_sha256': EXPECTED_SHA, 'archive_bytes': EXPECTED_BYTES,
|
| 162 |
'installed_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(),
|
| 163 |
'critical_files_sha256': fingerprints(stage)}
|
| 164 |
+
if compat:
|
| 165 |
+
receipt['compatibility_overlay'] = compat
|
| 166 |
(stage / RECEIPT).write_text(json.dumps(receipt, indent=2) + '\n')
|
| 167 |
if prefix.exists() or prefix.is_symlink():
|
| 168 |
raise FileExistsError(prefix)
|
|
|
|
| 177 |
assert torch.cuda.device_count() == 1, 'Expose exactly the B200 allocated to this check'
|
| 178 |
assert 'B200' in torch.cuda.get_device_name(0), 'Expected a B200'
|
| 179 |
expected = {'torch': '2.11.0+cu129', 'transformers': '5.6.2', 'sglang': '0.5.12.post1',
|
| 180 |
+
'flash-attn': '2.8.3', 'flashinfer-python': '0.6.11.post1', 'sglang-router': '0.3.2'}
|
| 181 |
versions = {name: md.version(name) for name in expected}
|
| 182 |
assert versions == expected, ('Runtime package mismatch', versions)
|
| 183 |
assert sys.version_info[:3] == (3, 12, 12), sys.version
|
| 184 |
cuda = pathlib.Path(os.environ['CUDA_HOME'])
|
| 185 |
assert (cuda / 'include/nvrtc.h').is_file(), 'Missing NVRTC headers'
|
| 186 |
+
for name in ['transformers', 'sglang.srt.entrypoints.engine', 'sglang_router.launch_router', 'flash_attn', 'flashinfer']:
|
| 187 |
importlib.import_module(name)
|
| 188 |
assert torch.version.cuda == '12.9', torch.version.cuda
|
| 189 |
value = torch.tensor([1.0], device='cuda')
|
|
|
|
| 215 |
prefix = raw.resolve()
|
| 216 |
if len(prefix.parts) < 3 or prefix == args.root.expanduser().resolve():
|
| 217 |
raise ValueError('Refusing an unsafe runtime prefix: ' + str(prefix))
|
| 218 |
+
host_glibc = check_host()
|
| 219 |
+
needs_compat = host_glibc < (2, 39)
|
| 220 |
if args.check_only and not prefix.exists():
|
| 221 |
raise FileNotFoundError('Runtime is absent; run without --check-only to restore it')
|
| 222 |
prefix.parent.mkdir(parents=True, exist_ok=True)
|
|
|
|
| 232 |
receipt = json.loads((prefix / RECEIPT).read_text())
|
| 233 |
if receipt.get('archive_sha256') != EXPECTED_SHA or receipt.get('critical_files_sha256') != fingerprints(prefix):
|
| 234 |
raise RuntimeError('Existing runtime identity/integrity check failed; preserved without replacement')
|
| 235 |
+
compat = receipt.get('compatibility_overlay')
|
| 236 |
+
if needs_compat and (not compat or compat.get('sha256') != COMPAT_SHA):
|
| 237 |
+
raise RuntimeError('Existing v4 runtime is not glibc-2.35 compatible; preserved. Choose a fresh --prefix for the qualified ABI overlay')
|
| 238 |
+
if compat:
|
| 239 |
+
for row in compat['files']:
|
| 240 |
+
relative = pathlib.PurePosixPath(row['path'])
|
| 241 |
+
if relative.is_absolute() or '..' in relative.parts or digest(prefix / relative) != row['sha256']:
|
| 242 |
+
raise RuntimeError('Installed ABI overlay integrity mismatch; preserved without replacement')
|
| 243 |
print('REUSING_RUNTIME=' + str(prefix), flush=True)
|
| 244 |
else:
|
| 245 |
if args.check_only:
|
|
|
|
| 249 |
archive = args.archive.expanduser().resolve() if args.archive else next((p for p in candidates if p.is_file()), None)
|
| 250 |
if archive is None or not archive.is_file():
|
| 251 |
raise FileNotFoundError('No runtime.tar.gz in ' + str(root) + '; download once to shared storage or pass --archive')
|
| 252 |
+
overlay = compatibility_overlay(root) if needs_compat else None
|
| 253 |
+
restore(archive, prefix, overlay)
|
| 254 |
if args.install_only:
|
| 255 |
print('RUNTIME_INSTALLED_UNCHECKED=' + str(prefix), flush=True)
|
| 256 |
else:
|
tests/test_ensure_runtime.py
CHANGED
|
@@ -6,6 +6,7 @@ import io
|
|
| 6 |
import json
|
| 7 |
import pathlib
|
| 8 |
import tempfile
|
|
|
|
| 9 |
import unittest
|
| 10 |
|
| 11 |
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
|
@@ -22,9 +23,29 @@ class EnsureRuntimeTests(unittest.TestCase):
|
|
| 22 |
self.code = {'__name__': 'ensure_fixture'}
|
| 23 |
exec(compile(INLINE, 'ensure_runtime.sh:inline', 'exec'), self.code)
|
| 24 |
# Fixture installation only: no real GPU/ML runtime or 45 GiB allocation.
|
| 25 |
-
self.code['check_host'] = lambda:
|
| 26 |
self.code['MIN_FREE_BYTES'] = 0
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
def tearDown(self):
|
| 29 |
self.temp.cleanup()
|
| 30 |
|
|
@@ -52,6 +73,82 @@ class EnsureRuntimeTests(unittest.TestCase):
|
|
| 52 |
ast.parse(INLINE)
|
| 53 |
ast.parse(self.code['PROBE'])
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
def test_install_and_reuse_without_archive(self):
|
| 56 |
self.archive_with()
|
| 57 |
output = self.run_ensure('--install-only')
|
|
|
|
| 6 |
import json
|
| 7 |
import pathlib
|
| 8 |
import tempfile
|
| 9 |
+
import types
|
| 10 |
import unittest
|
| 11 |
|
| 12 |
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
|
|
|
| 23 |
self.code = {'__name__': 'ensure_fixture'}
|
| 24 |
exec(compile(INLINE, 'ensure_runtime.sh:inline', 'exec'), self.code)
|
| 25 |
# Fixture installation only: no real GPU/ML runtime or 45 GiB allocation.
|
| 26 |
+
self.code['check_host'] = lambda: (2, 39)
|
| 27 |
self.code['MIN_FREE_BYTES'] = 0
|
| 28 |
|
| 29 |
+
def make_overlay(self, forbidden=False):
|
| 30 |
+
import tarfile
|
| 31 |
+
self.archive_with({'opt/ctopd-b200/host-libs/old-incompatible.so': b'old-v4-host-library'})
|
| 32 |
+
self.overlay = self.root / 'compat/glibc235-v2/overlay.tar.gz'
|
| 33 |
+
self.overlay.parent.mkdir(parents=True)
|
| 34 |
+
files = {'host-libs/libstdc++.so.6': b'compatible-library',
|
| 35 |
+
'site-packages/sglang_router/sglang_router_rs.abi3.so': b'compatible-router'}
|
| 36 |
+
manifest = {'base_archive_sha256': self.code['EXPECTED_SHA'], 'minimum_glibc': '2.35',
|
| 37 |
+
'files': [{'path': path, 'sha256': hashlib.sha256(content).hexdigest()} for path, content in files.items()]}
|
| 38 |
+
files['glibc235-overlay-manifest.json'] = json.dumps(manifest).encode()
|
| 39 |
+
if forbidden:
|
| 40 |
+
files['site-packages/torch/unrequested.py'] = b'forbidden'
|
| 41 |
+
with tarfile.open(self.overlay, 'w:gz') as bundle:
|
| 42 |
+
for name, content in files.items():
|
| 43 |
+
member = tarfile.TarInfo('opt/ctopd-b200/' + name)
|
| 44 |
+
member.size = len(content)
|
| 45 |
+
bundle.addfile(member, io.BytesIO(content))
|
| 46 |
+
self.code.update(COMPAT_SHA=hashlib.sha256(self.overlay.read_bytes()).hexdigest(),
|
| 47 |
+
COMPAT_BYTES=self.overlay.stat().st_size, COMPAT_REVISION='fixture')
|
| 48 |
+
|
| 49 |
def tearDown(self):
|
| 50 |
self.temp.cleanup()
|
| 51 |
|
|
|
|
| 73 |
ast.parse(INLINE)
|
| 74 |
ast.parse(self.code['PROBE'])
|
| 75 |
|
| 76 |
+
def test_both_required_host_baselines_and_older_rejection(self):
|
| 77 |
+
isolated = {'__name__': 'host_test'}
|
| 78 |
+
exec(compile(INLINE, 'ensure_runtime.sh:inline', 'exec'), isolated)
|
| 79 |
+
isolated['shutil'] = types.SimpleNamespace(which=lambda _name: '/usr/bin/g++')
|
| 80 |
+
for version, expected in [('2.35', (2, 35)), ('2.39', (2, 39))]:
|
| 81 |
+
isolated['platform'] = types.SimpleNamespace(libc_ver=lambda: ('glibc', version))
|
| 82 |
+
self.assertEqual(isolated['check_host'](), expected)
|
| 83 |
+
isolated['platform'] = types.SimpleNamespace(libc_ver=lambda: ('glibc', '2.34'))
|
| 84 |
+
with self.assertRaises(RuntimeError):
|
| 85 |
+
isolated['check_host']()
|
| 86 |
+
|
| 87 |
+
def test_glibc235_does_not_silently_accept_unpatched_v4(self):
|
| 88 |
+
self.archive_with()
|
| 89 |
+
self.run_ensure('--install-only')
|
| 90 |
+
before = (self.prefix / self.code['RECEIPT']).read_bytes()
|
| 91 |
+
self.code['check_host'] = lambda: (2, 35)
|
| 92 |
+
with self.assertRaisesRegex(RuntimeError, 'not glibc-2.35 compatible'):
|
| 93 |
+
self.run_ensure('--install-only')
|
| 94 |
+
self.assertEqual((self.prefix / self.code['RECEIPT']).read_bytes(), before)
|
| 95 |
+
|
| 96 |
+
def test_glibc235_requires_real_pinned_overlay(self):
|
| 97 |
+
self.archive_with()
|
| 98 |
+
self.code['check_host'] = lambda: (2, 35)
|
| 99 |
+
self.code.update(COMPAT_SHA=None, COMPAT_BYTES=None, COMPAT_REVISION=None)
|
| 100 |
+
with self.assertRaisesRegex(RuntimeError, 'no published overlay pin'):
|
| 101 |
+
self.run_ensure('--install-only')
|
| 102 |
+
self.assertFalse(self.prefix.exists())
|
| 103 |
+
|
| 104 |
+
def test_overlay_download_verifies_bytes_and_uses_selected_proxy(self):
|
| 105 |
+
import os
|
| 106 |
+
self.make_overlay()
|
| 107 |
+
payload = self.overlay.read_bytes()
|
| 108 |
+
self.overlay.unlink()
|
| 109 |
+
for proxy in ['direct', 'http://fixture-proxy:80']:
|
| 110 |
+
calls = []
|
| 111 |
+
|
| 112 |
+
def download(command, **_kwargs):
|
| 113 |
+
calls.append(command)
|
| 114 |
+
pathlib.Path(command[command.index('-o') + 1]).write_bytes(payload)
|
| 115 |
+
|
| 116 |
+
self.code['os'] = types.SimpleNamespace(environ={'CTOPD_PROXY': proxy}, close=os.close)
|
| 117 |
+
self.code['subprocess'] = types.SimpleNamespace(run=download)
|
| 118 |
+
with contextlib.redirect_stdout(io.StringIO()):
|
| 119 |
+
self.assertEqual(self.code['compatibility_overlay'](self.root), self.overlay)
|
| 120 |
+
self.assertEqual(self.overlay.read_bytes(), payload)
|
| 121 |
+
self.assertEqual(calls[0][-2:], ['--noproxy', '*' if proxy == 'direct' else ''])
|
| 122 |
+
self.assertEqual(list(self.overlay.parent.glob('.overlay-download-*')), [])
|
| 123 |
+
self.overlay.unlink()
|
| 124 |
+
|
| 125 |
+
def test_invalid_cached_overlay_is_preserved(self):
|
| 126 |
+
self.make_overlay()
|
| 127 |
+
self.overlay.write_bytes(b'invalid-existing-overlay')
|
| 128 |
+
with self.assertRaisesRegex(RuntimeError, 'preserved'):
|
| 129 |
+
self.code['compatibility_overlay'](self.root)
|
| 130 |
+
self.assertEqual(self.overlay.read_bytes(), b'invalid-existing-overlay')
|
| 131 |
+
|
| 132 |
+
def test_glibc235_installs_overlay_and_quarantines_old_host_libraries(self):
|
| 133 |
+
self.make_overlay()
|
| 134 |
+
self.code['check_host'] = lambda: (2, 35)
|
| 135 |
+
self.run_ensure('--install-only')
|
| 136 |
+
self.assertEqual((self.prefix / 'host-libs/libstdc++.so.6').read_bytes(), b'compatible-library')
|
| 137 |
+
self.assertEqual((self.prefix / 'host-libs-v4-disabled/old-incompatible.so').read_bytes(), b'old-v4-host-library')
|
| 138 |
+
receipt = json.loads((self.prefix / self.code['RECEIPT']).read_text())
|
| 139 |
+
self.assertEqual(receipt['compatibility_overlay']['sha256'], self.code['COMPAT_SHA'])
|
| 140 |
+
self.overlay.unlink()
|
| 141 |
+
self.archive.unlink()
|
| 142 |
+
self.assertIn('REUSING_RUNTIME=', self.run_ensure('--install-only'))
|
| 143 |
+
|
| 144 |
+
def test_overlay_cannot_modify_training_package(self):
|
| 145 |
+
self.make_overlay(forbidden=True)
|
| 146 |
+
self.code['check_host'] = lambda: (2, 35)
|
| 147 |
+
with contextlib.redirect_stderr(io.StringIO()):
|
| 148 |
+
with self.assertRaisesRegex(ValueError, 'Disallowed ABI overlay member'):
|
| 149 |
+
self.run_ensure('--install-only')
|
| 150 |
+
self.assertFalse(self.prefix.exists())
|
| 151 |
+
|
| 152 |
def test_install_and_reuse_without_archive(self):
|
| 153 |
self.archive_with()
|
| 154 |
output = self.run_ensure('--install-only')
|
tools/launch_single_b200.py
CHANGED
|
@@ -153,6 +153,13 @@ print(json.dumps({'gpu': name, 'torch': torch.__version__, 'cuda': torch.version
|
|
| 153 |
env=env, text=True)
|
| 154 |
plan['source_commit'] = commit
|
| 155 |
plan['preflight'] = json.loads(result.strip().splitlines()[-1])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 156 |
plan['input_sha256'] = {}
|
| 157 |
for key in ['train_dataset_path', 'eval_math_dataset_path']:
|
| 158 |
path = pathlib.Path(plan['flags'][key])
|
|
|
|
| 153 |
env=env, text=True)
|
| 154 |
plan['source_commit'] = commit
|
| 155 |
plan['preflight'] = json.loads(result.strip().splitlines()[-1])
|
| 156 |
+
runtime_receipt = pathlib.Path(plan['runtime']) / '.ctopd-runtime.json'
|
| 157 |
+
identity = json.loads(runtime_receipt.read_text()) if runtime_receipt.is_file() else {}
|
| 158 |
+
plan['installed_runtime_identity'] = {
|
| 159 |
+
'archive_sha256': identity.get('archive_sha256'),
|
| 160 |
+
'compatibility_overlay_sha256': identity.get('compatibility_overlay', {}).get('sha256'),
|
| 161 |
+
'receipt_present': runtime_receipt.is_file(),
|
| 162 |
+
}
|
| 163 |
plan['input_sha256'] = {}
|
| 164 |
for key in ['train_dataset_path', 'eval_math_dataset_path']:
|
| 165 |
path = pathlib.Path(plan['flags'][key])
|