codemaivanngu commited on
Commit
47d5468
·
unverified ·
1 Parent(s): 0ce2d91

feat(runtime): add idempotent node-local ensure helper

Browse files
README.md CHANGED
@@ -8,6 +8,8 @@ Start editing directly in `kdflow/algorithms/`, `kdflow/token_alignment.py`, `kd
8
 
9
  Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
10
 
 
 
11
  For a full-workload, one-run-per-B200 adaptation, use [the single-B200 launcher](docs/SINGLE_B200.md). It keeps 512-response rollout/optimizer batches, 8,192 generated tokens, 100 training iterations and 32-sample MATH500 evaluation. There is no reduced-workload mode. Single-B200 full training and paper metric agreement remain unvalidated.
12
 
13
  ## Original project documentation
 
8
 
9
  Use the existing [qualified B200 runtime on Hugging Face](https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-opd-b200-runtime) with a source overlay rather than reinstalling dependencies after each code edit. See [B200 development instructions](docs/RUNTIME_B200.md).
10
 
11
+ On a new company node, run the standalone `ensure_runtime.sh`: it restores the pinned shared archive into node-local `/opt/ctopd-b200` if absent, otherwise checks and reuses the existing installation. It preserves unknown/broken installations, uses no `pip`, and starts no training. See the instructions for shared-path placement, GPU visibility, and `--install-only` before allocation.
12
+
13
  For a full-workload, one-run-per-B200 adaptation, use [the single-B200 launcher](docs/SINGLE_B200.md). It keeps 512-response rollout/optimizer batches, 8,192 generated tokens, 100 training iterations and 32-sample MATH500 evaluation. There is no reduced-workload mode. Single-B200 full training and paper metric agreement remain unvalidated.
14
 
15
  ## Original project documentation
docs/RUNTIME_B200.md CHANGED
@@ -9,11 +9,42 @@ Runtime mirror: https://huggingface.co/datasets/codemaivanngu/cross-tokenizer-op
9
 
10
  Download/install directly on the target host using the pinned HF instructions. The archive requires Linux x86_64, glibc >= 2.39, g++ for JIT, and a B200 driver compatible with CUDA 12.9. Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  Once installed, point `RUNTIME` at the actual extracted prefix and run source from this checkout:
13
 
14
  ```bash
15
  cd /absolute/path/to/ctopd
16
- RUNTIME=/absolute/path/to/installed/ctopd-b200
17
  export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
18
  bash "$RUNTIME/python-b200.sh" -m kdflow.cli.train_kd_on_policy --help
19
  ```
 
9
 
10
  Download/install directly on the target host using the pinned HF instructions. The archive requires Linux x86_64, glibc >= 2.39, g++ for JIT, and a B200 driver compatible with CUDA 12.9. Allow about 35 GiB free for parallel download/assembly or 60 GiB including installation. Model weights and datasets are separate.
11
 
12
+ ## Restore/check on every new node
13
+
14
+ `ensure_runtime.sh` is self-contained and can be copied from this checkout into `/workspace/storage-shared/nlp/tungks/ctopd_all/ensure_runtime.sh`. The archive stays on shared storage; the default installed prefix `/opt/ctopd-b200` belongs to the current node/pod. Shared files do not imply that another node has the runtime installed.
15
+
16
+ ```bash
17
+ ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all
18
+ cp ensure_runtime.sh "$ROOT/ensure_runtime.sh" # run once from this checkout
19
+ bash "$ROOT/ensure_runtime.sh"
20
+ ```
21
+
22
+ Run the last line on each new node, inside a pod exposing the one B200 allocated for this check. With no GPU allocated yet, use `--install-only`; it reports `RUNTIME_INSTALLED_UNCHECKED`, not GPU readiness. With an existing runtime, `--check-only` checks without extracting. `--help` lists options.
23
+
24
+ The helper automatically finds either `$ROOT/ctopd-b200-cu129-v4/runtime.tar.gz` (the manually downloaded shard layout) or `$ROOT/runtime.tar.gz`. It verifies the pinned archive size/SHA-256 before extraction, checks Linux/glibc/g++, extracts safely to a new staging directory under the target filesystem, then renames to the final prefix. Allow at least 45 GiB free on that node-local filesystem. It never downloads dependencies, builds a runtime, launches training, or replaces an existing runtime that it cannot identify. Failed extraction keeps staging evidence. If `/opt` is not writable, choose an explicit node-local location:
25
+
26
+ ```bash
27
+ export CTOPD_RUNTIME_PREFIX="/var/tmp/ctopd-b200-$UID"
28
+ bash "$ROOT/ensure_runtime.sh"
29
+ ```
30
+
31
+ Concurrent setup calls use a lock for the same prefix. Reuse checks its archive-identity receipt and hashes of the wrapper, portable Python and nvcc; it does not rehash every installed dependency or reread the 16.5 GB archive. Default readiness additionally verifies the pinned Python/Torch/Transformers/SGLang/FlashAttention/FlashInfer versions, NVRTC headers, imports and a one-element CUDA memory roundtrip on the allocated B200. This is a basic runtime check, not full kernel qualification or a training result. Existing incompatible/unmanaged prefixes are preserved for manual inspection; there is no recursive-delete fallback.
32
+
33
+ The script can also execute a Python command after readiness:
34
+
35
+ ```bash
36
+ bash "$ROOT/ensure_runtime.sh" -- -c 'import torch; print(torch.__version__)'
37
+ ```
38
+
39
+ This is a fresh implementation for CTOPD's CUDA 12.9 bundle. The historical GOLD/SimCT helper used different CUDA 13 prefixes and a destructive repair fallback; those paths and that fallback are not copied into CTOPD. Live installation on the company node has not been executed from this chat. Offline archive-fixture tests and Bash syntax checks are separate from the real node readiness result.
40
+
41
+ ## Use source without rebuilding dependencies
42
+
43
  Once installed, point `RUNTIME` at the actual extracted prefix and run source from this checkout:
44
 
45
  ```bash
46
  cd /absolute/path/to/ctopd
47
+ RUNTIME="${CTOPD_RUNTIME_PREFIX:-/opt/ctopd-b200}"
48
  export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"
49
  bash "$RUNTIME/python-b200.sh" -m kdflow.cli.train_kd_on_policy --help
50
  ```
docs/SINGLE_B200.md CHANGED
@@ -42,14 +42,14 @@ CPU RAM and `/dev/shm` also matter: the teacher forwards a whole rollout batch a
42
  With `ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all`, prepare:
43
 
44
  ```text
45
- $ROOT/runtime/python-b200.sh
46
  $ROOT/models/Qwen2.5-7B-Instruct/ # complete official HF snapshot + tokenizer
47
  $ROOT/models/Llama-3.2-3B-Instruct/ # complete official HF snapshot + tokenizer
48
  $ROOT/data/Train/math_code_20k/train.parquet
49
  $ROOT/data/math500/test.jsonl
50
  ```
51
 
52
- Use `--runtime /actual/installed/prefix` if the runtime was installed elsewhere. The runtime tar.gz alone is not an installed prefix. Models/data must be downloaded directly on the runtime host; the Llama HF token must belong to an account with approved model access.
53
 
54
  ## Print the complete plan, then run it unchanged
55
 
 
42
  With `ROOT=/workspace/storage-shared/nlp/tungks/ctopd_all`, prepare:
43
 
44
  ```text
45
+ /opt/ctopd-b200/python-b200.sh # node-local; restore using ensure_runtime.sh
46
  $ROOT/models/Qwen2.5-7B-Instruct/ # complete official HF snapshot + tokenizer
47
  $ROOT/models/Llama-3.2-3B-Instruct/ # complete official HF snapshot + tokenizer
48
  $ROOT/data/Train/math_code_20k/train.parquet
49
  $ROOT/data/math500/test.jsonl
50
  ```
51
 
52
+ The launcher defaults to `CTOPD_RUNTIME_PREFIX` when set, otherwise `/opt/ctopd-b200`. Use `--runtime /actual/installed/prefix` for an existing alternative installation, including the earlier `$ROOT/runtime` layout. Run the [runtime restore/check helper](RUNTIME_B200.md) on every new node. The runtime tar.gz alone is not an installed prefix. Models/data must be downloaded directly on the runtime host; the Llama HF token must belong to an account with approved model access.
53
 
54
  ## Print the complete plan, then run it unchanged
55
 
docs/UPSTREAM.md CHANGED
@@ -8,7 +8,7 @@
8
  - The source service did not expose an upstream Git commit. The local import commit starts a new history and does not claim to preserve upstream Git history.
9
  - `docs/upstream-source-manifest.json` records original byte sizes and hashes. It describes the initial import; later development commits may intentionally change those files.
10
 
11
- The initial commit preserves the imported implementation. Local additions provide documentation, line-ending attributes, a single-B200 launcher and its offline tests. Imported `kdflow/` code and the two original training scripts remain unchanged. Datasets and figures were excluded from the source snapshot, and no runtime archives, model weights, credentials, account metadata or experiment outputs are included.
12
 
13
  ## License metadata
14
 
 
8
  - The source service did not expose an upstream Git commit. The local import commit starts a new history and does not claim to preserve upstream Git history.
9
  - `docs/upstream-source-manifest.json` records original byte sizes and hashes. It describes the initial import; later development commits may intentionally change those files.
10
 
11
+ The initial commit preserves the imported implementation. Local additions provide documentation, line-ending attributes, a single-B200 launcher, a standalone node-local runtime setup helper and offline tests. Imported `kdflow/` code and the two original training scripts remain unchanged. Datasets and figures were excluded from the source snapshot, and no runtime archives, model weights, credentials, account metadata or experiment outputs are included.
12
 
13
  ## License metadata
14
 
ensure_runtime.SHA256SUMS ADDED
@@ -0,0 +1 @@
 
 
1
+ 9a388b3ca183e0e771a49bdc4b61d9c6880d043da5490122c8fdf4098c3b9895 ensure_runtime.sh
ensure_runtime.sh ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # Standalone node-local restore/check entry point. Safe to copy into ctopd_all/.
3
+ set -euo pipefail
4
+ if [[ "$(uname -s)" != Linux || "$(uname -m)" != x86_64 ]]; then
5
+ echo 'CTOPD requires Linux x86_64; do not install this runtime on Windows.' >&2
6
+ exit 2
7
+ fi
8
+ SYSTEM_PY=""
9
+ for candidate in python3.12 python3; do
10
+ if command -v "$candidate" >/dev/null 2>&1; then SYSTEM_PY="$(command -v "$candidate")"; break; fi
11
+ done
12
+ [[ -n "$SYSTEM_PY" ]] || { echo 'Need a system Python 3 interpreter.' >&2; exit 2; }
13
+ exec "$SYSTEM_PY" - "$@" <<'CTOPD_ENSURE_PY'
14
+ import argparse
15
+ import copy
16
+ import datetime
17
+ import fcntl
18
+ import hashlib
19
+ import json
20
+ import os
21
+ import pathlib
22
+ import platform
23
+ import shutil
24
+ import subprocess
25
+ import sys
26
+ import tarfile
27
+ import tempfile
28
+
29
+ EXPECTED_SHA = '084c321c0b7af1cd3a36d12a595c623567d4a37525aa0eecbc3933aa7f9b15c3'
30
+ EXPECTED_BYTES = 16515065867
31
+ MIN_FREE_BYTES = 45 * 1024**3
32
+ RECEIPT = '.ctopd-runtime.json'
33
+
34
+
35
+ def digest(path):
36
+ result = hashlib.sha256()
37
+ with path.open('rb') as stream:
38
+ for block in iter(lambda: stream.read(8 * 1024 * 1024), b''):
39
+ result.update(block)
40
+ return result.hexdigest()
41
+
42
+
43
+ def check_host():
44
+ libc, version = platform.libc_ver()
45
+ if libc != 'glibc' or tuple(map(int, version.split('.')[:2])) < (2, 39):
46
+ raise RuntimeError('Need glibc >= 2.39 (Ubuntu 24.04 baseline); change pod image, not system libc')
47
+ if not shutil.which('g++'):
48
+ raise RuntimeError('Host g++ is required for runtime JIT; use a prepared node image')
49
+
50
+
51
+ def fingerprints(prefix):
52
+ interpreters = list(prefix.glob('python/cpython-3.12.12-*/bin/python3.12'))
53
+ if len(interpreters) != 1:
54
+ raise RuntimeError('Portable Python is missing or ambiguous in ' + str(prefix))
55
+ paths = [prefix / 'python-b200.sh', interpreters[0], prefix / 'cuda/bin/nvcc']
56
+ for path in paths:
57
+ if not path.is_file():
58
+ raise RuntimeError('Missing runtime component: ' + str(path))
59
+ return {str(path.relative_to(prefix)): digest(path) for path in paths}
60
+
61
+
62
+ def restore(archive, prefix):
63
+ if not hasattr(tarfile, 'data_filter'):
64
+ raise RuntimeError('System Python needs safe tar extraction filters (Python 3.12 or patched 3.10/3.11)')
65
+ if archive.stat().st_size != EXPECTED_BYTES or digest(archive) != EXPECTED_SHA:
66
+ raise RuntimeError('Archive size/SHA-256 mismatch; reassemble numbered parts correctly before installing')
67
+ print('ARCHIVE_SHA256_OK=' + EXPECTED_SHA, flush=True)
68
+ if shutil.disk_usage(prefix.parent).free < MIN_FREE_BYTES:
69
+ raise RuntimeError('Need at least 45 GiB free on the target filesystem for extraction')
70
+ stage = pathlib.Path(tempfile.mkdtemp(prefix='.' + prefix.name + '.install-', dir=prefix.parent))
71
+ print('EXTRACTING_TO=' + str(stage), flush=True)
72
+ try:
73
+ with tarfile.open(archive, 'r:gz') as bundle:
74
+ for original in bundle:
75
+ path = pathlib.PurePosixPath(original.name)
76
+ if path.is_absolute() or '..' in path.parts or path.parts[:2] != ('opt', 'ctopd-b200'):
77
+ raise ValueError('Unexpected archive member: ' + str(path))
78
+ if len(path.parts) == 2:
79
+ continue
80
+ member = copy.copy(original)
81
+ member.name = str(pathlib.PurePosixPath(*path.parts[2:]))
82
+ if member.islnk():
83
+ link = pathlib.PurePosixPath(member.linkname)
84
+ if link.is_absolute() or '..' in link.parts or link.parts[:2] != ('opt', 'ctopd-b200'):
85
+ raise ValueError('Unsafe hardlink: ' + member.linkname)
86
+ member.linkname = str(pathlib.PurePosixPath(*link.parts[2:]))
87
+ bundle.extract(member, stage, filter='data')
88
+ receipt = {'archive_sha256': EXPECTED_SHA, 'archive_bytes': EXPECTED_BYTES,
89
+ 'installed_utc': datetime.datetime.now(datetime.timezone.utc).isoformat(),
90
+ 'critical_files_sha256': fingerprints(stage)}
91
+ (stage / RECEIPT).write_text(json.dumps(receipt, indent=2) + '\n')
92
+ if prefix.exists() or prefix.is_symlink():
93
+ raise FileExistsError(prefix)
94
+ stage.rename(prefix)
95
+ except Exception:
96
+ print('INSTALL_FAILED: staging retained for inspection at ' + str(stage), file=sys.stderr)
97
+ raise
98
+
99
+
100
+ PROBE = '''import importlib, importlib.metadata as md, json, os, pathlib, sys
101
+ import torch
102
+ assert torch.cuda.device_count() == 1, 'Expose exactly the B200 allocated to this check'
103
+ assert 'B200' in torch.cuda.get_device_name(0), 'Expected a B200'
104
+ expected = {'torch': '2.11.0+cu129', 'transformers': '5.6.2', 'sglang': '0.5.12.post1',
105
+ 'flash-attn': '2.8.3', 'flashinfer-python': '0.6.11.post1'}
106
+ versions = {name: md.version(name) for name in expected}
107
+ assert versions == expected, ('Runtime package mismatch', versions)
108
+ assert sys.version_info[:3] == (3, 12, 12), sys.version
109
+ cuda = pathlib.Path(os.environ['CUDA_HOME'])
110
+ assert (cuda / 'include/nvrtc.h').is_file(), 'Missing NVRTC headers'
111
+ for name in ['transformers', 'sglang.srt.entrypoints.engine', 'flash_attn', 'flashinfer']:
112
+ importlib.import_module(name)
113
+ assert torch.version.cuda == '12.9', torch.version.cuda
114
+ value = torch.tensor([1.0], device='cuda')
115
+ assert value.cpu().item() == 1.0
116
+ print(json.dumps({'versions': versions, 'python': sys.version.split()[0],
117
+ 'cuda': torch.version.cuda, 'gpu': torch.cuda.get_device_name(0),
118
+ 'check': 'imports + CUDA memory roundtrip; no training or full kernel suite'}))
119
+ '''
120
+
121
+
122
+ def main(argv=None):
123
+ parser = argparse.ArgumentParser(description='Restore/check the pinned CTOPD runtime on this node; no pip, build or training')
124
+ parser.add_argument('--root', type=pathlib.Path,
125
+ default=pathlib.Path(os.environ.get('CTOPD_ROOT', '/workspace/storage-shared/nlp/tungks/ctopd_all')))
126
+ parser.add_argument('--prefix', type=pathlib.Path,
127
+ default=pathlib.Path(os.environ.get('CTOPD_RUNTIME_PREFIX', '/opt/ctopd-b200')))
128
+ parser.add_argument('--archive', type=pathlib.Path)
129
+ mode = parser.add_mutually_exclusive_group()
130
+ mode.add_argument('--install-only', action='store_true', help='Restore without a GPU check; reports INSTALLED_UNCHECKED')
131
+ mode.add_argument('--check-only', action='store_true', help='Check an existing managed runtime; never extract')
132
+ parser.add_argument('python_args', nargs=argparse.REMAINDER, help='After --, execute these Python arguments through the runtime wrapper')
133
+ args = parser.parse_args(argv)
134
+ forwarded = args.python_args[1:] if args.python_args[:1] == ['--'] else args.python_args
135
+ if args.install_only and forwarded:
136
+ raise ValueError('--install-only cannot also execute a runtime command')
137
+ raw = args.prefix.expanduser()
138
+ if not raw.is_absolute() or raw.is_symlink():
139
+ raise ValueError('Prefix must be an absolute non-symlink directory')
140
+ prefix = raw.resolve()
141
+ if len(prefix.parts) < 3 or prefix == args.root.expanduser().resolve():
142
+ raise ValueError('Refusing an unsafe runtime prefix: ' + str(prefix))
143
+ check_host()
144
+ if args.check_only and not prefix.exists():
145
+ raise FileNotFoundError('Runtime is absent; run without --check-only to restore it')
146
+ prefix.parent.mkdir(parents=True, exist_ok=True)
147
+ lock_path = prefix.parent / ('.' + prefix.name + '.ensure.lock')
148
+ if lock_path.is_symlink():
149
+ raise ValueError('Refusing symlink lock file')
150
+ with lock_path.open('a') as lock:
151
+ print('WAITING_FOR_RUNTIME_LOCK=' + str(lock_path), flush=True)
152
+ fcntl.flock(lock, fcntl.LOCK_EX)
153
+ if prefix.exists():
154
+ if not prefix.is_dir() or not (prefix / RECEIPT).is_file():
155
+ raise RuntimeError('Existing unmanaged prefix preserved; choose another --prefix or inspect it manually: ' + str(prefix))
156
+ receipt = json.loads((prefix / RECEIPT).read_text())
157
+ if receipt.get('archive_sha256') != EXPECTED_SHA or receipt.get('critical_files_sha256') != fingerprints(prefix):
158
+ raise RuntimeError('Existing runtime identity/integrity check failed; preserved without replacement')
159
+ print('REUSING_RUNTIME=' + str(prefix), flush=True)
160
+ else:
161
+ if args.check_only:
162
+ raise FileNotFoundError('Runtime disappeared while waiting for its lock; no extraction performed')
163
+ root = args.root.expanduser().resolve()
164
+ candidates = [root / 'ctopd-b200-cu129-v4/runtime.tar.gz', root / 'runtime.tar.gz']
165
+ archive = args.archive.expanduser().resolve() if args.archive else next((p for p in candidates if p.is_file()), None)
166
+ if archive is None or not archive.is_file():
167
+ raise FileNotFoundError('No runtime.tar.gz in ' + str(root) + '; download once to shared storage or pass --archive')
168
+ restore(archive, prefix)
169
+ if args.install_only:
170
+ print('RUNTIME_INSTALLED_UNCHECKED=' + str(prefix), flush=True)
171
+ else:
172
+ env = os.environ.copy()
173
+ env.update(PYTHONPATH='', HF_HUB_OFFLINE='1', TRANSFORMERS_OFFLINE='1', PYTHONNOUSERSITE='1')
174
+ output = subprocess.check_output(['bash', str(prefix / 'python-b200.sh'), '-c', PROBE], env=env, text=True)
175
+ result = json.loads(output.strip().splitlines()[-1])
176
+ result['checked_utc'] = datetime.datetime.now(datetime.timezone.utc).isoformat()
177
+ (prefix / '.ctopd-last-check.json').write_text(json.dumps(result, indent=2) + '\n')
178
+ print(output, end='', flush=True)
179
+ print('RUNTIME_READY=' + str(prefix), flush=True)
180
+ print('RUNTIME_PYTHON_WRAPPER=' + str(prefix / 'python-b200.sh'), flush=True)
181
+ if forwarded:
182
+ os.execvpe('bash', ['bash', str(prefix / 'python-b200.sh'), *forwarded], os.environ.copy())
183
+
184
+
185
+ if __name__ == '__main__':
186
+ main()
187
+ CTOPD_ENSURE_PY
tests/test_ensure_runtime.py ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tiny archive fixtures; never download/install the production runtime locally."""
2
+ import ast
3
+ import contextlib
4
+ import hashlib
5
+ import io
6
+ import json
7
+ import pathlib
8
+ import tempfile
9
+ import unittest
10
+
11
+ ROOT = pathlib.Path(__file__).resolve().parents[1]
12
+ SCRIPT = (ROOT / 'ensure_runtime.sh').read_text()
13
+ INLINE = SCRIPT.split("<<'CTOPD_ENSURE_PY'\n", 1)[1].rsplit('\nCTOPD_ENSURE_PY', 1)[0]
14
+
15
+
16
+ class EnsureRuntimeTests(unittest.TestCase):
17
+ def setUp(self):
18
+ self.temp = tempfile.TemporaryDirectory()
19
+ self.root = pathlib.Path(self.temp.name)
20
+ self.archive = self.root / 'runtime.tar.gz'
21
+ self.prefix = self.root / 'node-local/runtime'
22
+ self.code = {'__name__': 'ensure_fixture'}
23
+ exec(compile(INLINE, 'ensure_runtime.sh:inline', 'exec'), self.code)
24
+ # Fixture installation only: no real GPU/ML runtime or 45 GiB allocation.
25
+ self.code['check_host'] = lambda: None
26
+ self.code['MIN_FREE_BYTES'] = 0
27
+
28
+ def tearDown(self):
29
+ self.temp.cleanup()
30
+
31
+ def archive_with(self, extra=None):
32
+ import tarfile
33
+ files = {'opt/ctopd-b200/python-b200.sh': b'#!/bin/bash\n',
34
+ 'opt/ctopd-b200/python/cpython-3.12.12-fixture/bin/python3.12': b'fixture-python',
35
+ 'opt/ctopd-b200/cuda/bin/nvcc': b'fixture-nvcc'}
36
+ if extra:
37
+ files.update(extra)
38
+ with tarfile.open(self.archive, 'w:gz') as bundle:
39
+ for name, content in files.items():
40
+ member = tarfile.TarInfo(name)
41
+ member.size = len(content)
42
+ bundle.addfile(member, io.BytesIO(content))
43
+ self.code['EXPECTED_SHA'] = hashlib.sha256(self.archive.read_bytes()).hexdigest()
44
+ self.code['EXPECTED_BYTES'] = self.archive.stat().st_size
45
+
46
+ def run_ensure(self, *options):
47
+ with contextlib.redirect_stdout(io.StringIO()) as output:
48
+ self.code['main'](['--root', str(self.root), '--prefix', str(self.prefix), *options])
49
+ return output.getvalue()
50
+
51
+ def test_inline_and_probe_syntax(self):
52
+ ast.parse(INLINE)
53
+ ast.parse(self.code['PROBE'])
54
+
55
+ def test_install_and_reuse_without_archive(self):
56
+ self.archive_with()
57
+ output = self.run_ensure('--install-only')
58
+ self.assertIn('RUNTIME_INSTALLED_UNCHECKED=', output)
59
+ self.assertNotIn('RUNTIME_READY=', output)
60
+ receipt = self.prefix / self.code['RECEIPT']
61
+ before = receipt.read_bytes()
62
+ self.archive.unlink()
63
+ output = self.run_ensure('--install-only')
64
+ self.assertIn('REUSING_RUNTIME=', output)
65
+ self.assertEqual(receipt.read_bytes(), before)
66
+
67
+ def test_wrong_checksum_never_installs(self):
68
+ self.archive_with()
69
+ self.code['EXPECTED_SHA'] = '0' * 64
70
+ with self.assertRaisesRegex(RuntimeError, 'SHA-256 mismatch'):
71
+ self.run_ensure('--install-only')
72
+ self.assertFalse(self.prefix.exists())
73
+ self.assertFalse(list(self.prefix.parent.glob('.runtime.install-*')))
74
+
75
+ def test_unmanaged_existing_prefix_preserved(self):
76
+ self.prefix.mkdir(parents=True)
77
+ content = self.prefix / 'user-data'
78
+ content.write_text('keep')
79
+ with self.assertRaisesRegex(RuntimeError, 'unmanaged prefix preserved'):
80
+ self.run_ensure('--install-only')
81
+ self.assertEqual(content.read_text(), 'keep')
82
+
83
+ def test_changed_runtime_preserved_and_rejected(self):
84
+ self.archive_with()
85
+ self.run_ensure('--install-only')
86
+ wrapper = self.prefix / 'python-b200.sh'
87
+ wrapper.write_text('user change')
88
+ with self.assertRaisesRegex(RuntimeError, 'identity/integrity check failed'):
89
+ self.run_ensure('--install-only')
90
+ self.assertEqual(wrapper.read_text(), 'user change')
91
+
92
+ def test_traversal_rejected_and_evidence_retained(self):
93
+ self.archive_with({'opt/ctopd-b200/../escape': b'bad'})
94
+ with contextlib.redirect_stderr(io.StringIO()):
95
+ with self.assertRaisesRegex(ValueError, 'Unexpected archive member'):
96
+ self.run_ensure('--install-only')
97
+ self.assertFalse(self.prefix.exists())
98
+ self.assertFalse((self.prefix.parent / 'escape').exists())
99
+ self.assertEqual(len(list(self.prefix.parent.glob('.runtime.install-*'))), 1)
100
+
101
+ def test_check_only_does_not_install_or_create_parent(self):
102
+ with self.assertRaises(FileNotFoundError):
103
+ self.run_ensure('--check-only')
104
+ self.assertFalse(self.prefix.parent.exists())
105
+
106
+ def test_escaping_symlink_rejected(self):
107
+ import tarfile
108
+ self.archive_with()
109
+ with tarfile.open(self.archive, 'w:gz') as bundle:
110
+ member = tarfile.TarInfo('opt/ctopd-b200/escape')
111
+ member.type = tarfile.SYMTYPE
112
+ member.linkname = '/tmp/outside-ctopd'
113
+ bundle.addfile(member)
114
+ self.code['EXPECTED_SHA'] = hashlib.sha256(self.archive.read_bytes()).hexdigest()
115
+ self.code['EXPECTED_BYTES'] = self.archive.stat().st_size
116
+ with contextlib.redirect_stderr(io.StringIO()):
117
+ with self.assertRaises(tarfile.FilterError):
118
+ self.run_ensure('--install-only')
119
+ self.assertFalse(self.prefix.exists())
120
+
121
+
122
+ if __name__ == '__main__':
123
+ unittest.main()
tests/test_single_b200.py CHANGED
@@ -67,7 +67,7 @@ class LauncherTests(unittest.TestCase):
67
  with self.assertRaises(ValueError):
68
  self.plan('--run-id', '../other-run')
69
  with tempfile.TemporaryDirectory() as folder:
70
- plan = self.plan('--root', folder)
71
  for key in ['student_name_or_path', 'teacher_name_or_path']:
72
  model = pathlib.Path(plan['flags'][key])
73
  model.mkdir(parents=True)
 
67
  with self.assertRaises(ValueError):
68
  self.plan('--run-id', '../other-run')
69
  with tempfile.TemporaryDirectory() as folder:
70
+ plan = self.plan('--root', folder, '--runtime', str(pathlib.Path(folder) / 'runtime'))
71
  for key in ['student_name_or_path', 'teacher_name_or_path']:
72
  model = pathlib.Path(plan['flags'][key])
73
  model.mkdir(parents=True)
tools/launch_single_b200.py CHANGED
@@ -23,7 +23,8 @@ def parser():
23
  p = argparse.ArgumentParser(description=__doc__)
24
  p.add_argument('--root', type=pathlib.Path,
25
  default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
26
- p.add_argument('--runtime', type=pathlib.Path, help='Installed prefix containing python-b200.sh')
 
27
  p.add_argument('--algorithm', choices=['strict-top16', 'strict-full'], default='strict-top16')
28
  p.add_argument('--seed', type=int, default=42, help='Original launcher default, not a verified paper seed')
29
  p.add_argument('--micro-batch', type=int, choices=[1, 2, 4], default=4,
@@ -43,7 +44,7 @@ def build_plan(args):
43
  if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,95}', run_id):
44
  raise ValueError('run-id must be a single directory name starting with a letter or digit')
45
  root = args.root.expanduser().resolve()
46
- runtime = (args.runtime or root / 'runtime').expanduser().resolve()
47
  output = root / 'runs' / run_id
48
  flags = {
49
  'num_nodes': 1, 'num_gpus_per_node': 1, 'backend': 'fsdp2',
 
23
  p = argparse.ArgumentParser(description=__doc__)
24
  p.add_argument('--root', type=pathlib.Path,
25
  default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
26
+ p.add_argument('--runtime', type=pathlib.Path,
27
+ help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
28
  p.add_argument('--algorithm', choices=['strict-top16', 'strict-full'], default='strict-top16')
29
  p.add_argument('--seed', type=int, default=42, help='Original launcher default, not a verified paper seed')
30
  p.add_argument('--micro-batch', type=int, choices=[1, 2, 4], default=4,
 
44
  if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,95}', run_id):
45
  raise ValueError('run-id must be a single directory name starting with a letter or digit')
46
  root = args.root.expanduser().resolve()
47
+ runtime = (args.runtime or pathlib.Path(os.environ.get('CTOPD_RUNTIME_PREFIX', '/opt/ctopd-b200'))).expanduser().resolve()
48
  output = root / 'runs' / run_id
49
  flags = {
50
  'num_nodes': 1, 'num_gpus_per_node': 1, 'backend': 'fsdp2',