codemaivanngu commited on
Commit
f068102
·
verified ·
1 Parent(s): ef62ebc

Publish three-seed evaluation queue and oracle tokenizer fix bbfef64f

Browse files
EVAL-QUEUE-RUNBOOK.md ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Company evaluation queue (three evaluation seeds)
2
+
3
+ This queue evaluates exactly one SFT initialization and the two existing MB4,
4
+ 312-update trajectories. It does not train new models. Atomic/fixed snapshots:
5
+ 40, 80, 120, 156, 200, 240, 280, 312; SFT is evaluated once as a snapshot.
6
+ Each snapshot runs seeds 42, 43, 44 over GSM8K, MATH500, original MBPP and LCB v6:
7
+ 17 snapshots, 51 snapshot-seed combinations, 204 benchmark cells at most.
8
+
9
+ ## Contract and interpretation
10
+
11
+ Profile `company-internal-v1` is explicit opt-in. It retains author generation
12
+ settings (temperature .6, top_p .95, one sample per request, benchmark token
13
+ caps), author GSM8K/MATH/MBPP helpers, and the pinned official LCB tester with
14
+ public and private tests. Functional LCB tasks receive the function name and
15
+ starter code in a dedicated Solution-method prompt. Math uses the author's
16
+ SymPy helpers, not math-verify. The three evaluation seeds measure sampling
17
+ variance, not independent training variance. Intermediate test-checkpoint
18
+ comparisons are exploratory; do not report the best test checkpoint as an
19
+ unbiased paper reproduction. Existing author-code/paper-spec sandbox profiles
20
+ are unchanged.
21
+
22
+ The company-internal subprocess executes generated Python under the current
23
+ user, with process time/address-space limits and a clean environment. **These
24
+ limits are not filesystem/network/container isolation.** This mode implements
25
+ the user's accepted internal-eval workflow for the namespace-restricted node.
26
+ Do not describe its timeout/security contract as identical to the paper.
27
+
28
+ Canonical fallback files total approximately 1 MB and are pinned by HF commit
29
+ and SHA256. Existing cached canonical files are reused. Company JSON copies
30
+ are compared against canonical normalized contents; incompatible copies are
31
+ not silently substituted. LCB reuses the already verified 1055-problem file,
32
+ indexes it without loading all hidden tests, and verifies each row before use.
33
+ No models or training data are downloaded. No dependencies are auto-installed.
34
+ The scorer needs numpy and sympy; parquet preparation needs pyarrow, available
35
+ in the existing datasets runtime if its installation is complete.
36
+
37
+ ## Company commands
38
+
39
+ Pull `pull-eval-queue.sh` from the published, pinned HF revision and verify its
40
+ SHA256 using the accompanying chat commands. Running the pull script creates
41
+ an isolated checkout and prints EVAL_WORK. It never changes SimCT-git.
42
+
43
+ ```bash
44
+ EVAL_WORK=/workspace/storage-shared/nlp/tungks/simct-eval-queue-REPLACE
45
+ bash "$EVAL_WORK/source/experiments/runai/transfer/prepare-eval-queue.sh"
46
+ # The following command starts GPU workers; run only when ready.
47
+ bash "$EVAL_WORK/source/experiments/runai/transfer/start-eval-queue.sh"
48
+ source "$EVAL_WORK/queue.env"
49
+ /usr/bin/python3.12 "$EVAL_WORK/source/scripts/evaluation/eval_queue.py" summarize --plan "$EVAL_PLAN"
50
+ ```
51
+
52
+ Preparation qualifies ten synthetic correct/incorrect scoring fixtures, checks
53
+ the two completed training summaries, and hashes every chosen checkpoint.
54
+ Hashing roughly 17 model snapshots may take several minutes on shared storage.
55
+ It does not consume the GPU budget. Missing snapshots stop preparation rather
56
+ than changing the selection silently.
57
+
58
+ ## Budget, ordering, resume
59
+
60
+ The first worker completing CPU qualification starts one shared 20-hour wall
61
+ clock. New snapshots stop being admitted at 18 hours (or earlier if the slowest
62
+ observed snapshot would not fit). A server watchdog ends the worker-owned
63
+ server at the deadline; process cleanup can take a few seconds beyond it.
64
+ Unfinished cells remain partial. There is no guarantee all 204 cells fit.
65
+ Both workers take snapshots independently, with barriers between these tiers:
66
+
67
+ 1. SFT, atomic312, fixed312, atomic156, fixed156.
68
+ 2. Atomic/fixed80 and240.
69
+ 3. Atomic/fixed40 and200.
70
+ 4. Atomic/fixed120 and280.
71
+
72
+ Every selected snapshot finishes all three seeds before its worker moves on.
73
+ GPU memory >=1 GiB causes waiting; existing training/oracle jobs are never
74
+ killed. Per-GPU and per-job file locks prevent duplicate queue work. Resource
75
+ ownership outside this queue still requires the operator to avoid launching
76
+ another workload onto an occupied GPU.
77
+
78
+ Workers append and fsync per-question responses and scores. Resume checks
79
+ source, checkpoint, data, request, response and server identities. Running
80
+ start-eval-queue.sh again resumes the same plan and clock; it does not grant
81
+ another 20 hours. Active duplicate workers fail their GPU lock. A failed job
82
+ halts admissions: inspect worker/server logs, fix the cause, then use
83
+ `eval_queue.py retry --plan "$EVAL_PLAN"` and start the workers again. Source or
84
+ runtime changes that break the recorded contract require a new plan and must
85
+ not be merged into old results. Do not delete state.json to extend the clock.
86
+
87
+ Summary includes a benchmark mean and sample standard deviation only when all
88
+ three seeds are complete and hashes match. A checkpoint average requires all
89
+ four benchmarks; partial results are not zero-filled. Logs and result files
90
+ stay under the plan directory. Nothing is uploaded to W&B automatically.
91
+
92
+ ## Verification boundary
93
+
94
+ Offline regression tests cover selection, tier barriers, deadline admission,
95
+ resume journals, cached responses, source-row integrity, and ten actual CPU
96
+ scorer fixtures including functional LCB. SGLang GPU startup and end-to-end
97
+ throughput still need verification on the company B200 node.
98
+
99
+ ## Mechanics tokenizer fix included
100
+
101
+ The same source bundle fixes `real_oracle.py` prompt tokenization: explicitly
102
+ request `return_dict=False`, normalize mapping/Encoding outputs into integer
103
+ IDs, and create torch.long input tensors. The old failure happened before the
104
+ first group result; it is not oracle evidence. Reuse the existing prepared
105
+ six-row mechanics JSON in the fresh bundle workspace, then run the copied
106
+ 03-run-mechanics.sh on GPU1 before starting the evaluation workers. GPU1's
107
+ eval worker will wait while mechanics owns memory. GPU0 can evaluate meanwhile.
108
+ The tokenizer regression uses CPU mocks; the corrected real-model canary has
109
+ not yet run on the company node.
pull-eval-queue.sh ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+ ROOT=/workspace/storage-shared/nlp/tungks
4
+ WORK="$(mktemp -d "$ROOT/simct-eval-queue-XXXXXXXX")"
5
+ PROXY="${MP_DOWNLOAD_PROXY:-http://10.30.154.118:80}"
6
+ fetch() { curl --fail --location --retry 3 --proxy "$PROXY" --noproxy "" "$1" -o "$2"; }
7
+ if [[ -z "${HF_REVISION:-}" ]]; then
8
+ fetch 'https://huggingface.co/api/models/codemaivanngu/simct' "$WORK/hf-info.json"
9
+ HF_REVISION="$(/usr/bin/python3.12 -c 'import json,sys; print(json.load(open(sys.argv[1]))["sha"])' "$WORK/hf-info.json")"
10
+ fi
11
+ [[ "$HF_REVISION" =~ ^[0-9a-f]{40}$ ]] || { echo 'Invalid HF revision'; exit 2; }
12
+ BASE="https://huggingface.co/codemaivanngu/simct/resolve/$HF_REVISION"
13
+ fetch "$BASE/source-manifest.json" "$WORK/source-manifest.json"
14
+ fetch "$BASE/simct-b200-portable.bundle" "$WORK/simct-b200-portable.bundle"
15
+ /usr/bin/python3.12 - "$WORK" <<'PY'
16
+ import hashlib,json,sys
17
+ from pathlib import Path
18
+ root=Path(sys.argv[1]); m=json.loads((root/'source-manifest.json').read_text())
19
+ assert m['bundle_file']=='simct-b200-portable.bundle'
20
+ assert hashlib.sha256((root/m['bundle_file']).read_bytes()).hexdigest()==m['bundle_sha256'], 'bundle checksum mismatch'
21
+ assert m['source_branch']=='vdt/ops/b200-portable'
22
+ print('BUNDLE_SHA256_PASS')
23
+ PY
24
+ git clone --branch vdt/ops/b200-portable "$WORK/simct-b200-portable.bundle" "$WORK/source"
25
+ EXPECTED="$(/usr/bin/python3.12 -c 'import json,sys; print(json.load(open(sys.argv[1]))["source_commit"])' "$WORK/source-manifest.json")"
26
+ [[ "$(git -C "$WORK/source" rev-parse HEAD)" == "$EXPECTED" ]]
27
+ for NAME in 02-prepare-mechanics.sh 03-run-mechanics.sh; do
28
+ cp "$WORK/source/experiments/runai/transfer/$NAME" "$WORK/$NAME"
29
+ done
30
+ printf 'export ABC_WORK=%q\n' "$WORK" > "$WORK/workspace.env"
31
+ printf 'export EVAL_WORK=%q\n' "$WORK" > "$WORK/eval.env"
32
+ printf 'HF_REVISION=%s\nSOURCE_COMMIT=%s\nEVAL_WORK=%s\n' "$HF_REVISION" "$EXPECTED" "$WORK"
33
+ echo "NEXT: bash $WORK/source/experiments/runai/transfer/prepare-eval-queue.sh"
34
+ echo 'No GPU started; existing training checkout unchanged.'
simct-b200-portable.bundle CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:328f4f28ef6a1e52503f24d913ea6ca4259d51ece25cf03ca5c193e5b77090ba
3
- size 4145660
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:10f42cb4b625116aeb5a492eee42364a181527171d90895e19ee941fb58b11ed
3
+ size 4159222
source-manifest.json CHANGED
@@ -1,9 +1,9 @@
1
  {
2
  "source_branch": "vdt/ops/b200-portable",
3
- "source_commit": "4b9d70a58df4aa77d2bc35228207b959fdf6bee5",
4
  "bundle_file": "simct-b200-portable.bundle",
5
- "bundle_sha256": "328f4f28ef6a1e52503f24d913ea6ca4259d51ece25cf03ca5c193e5b77090ba",
6
- "bundle_bytes": 4145660,
7
  "history": "complete",
8
  "format": "git-bundle",
9
  "reason": "HF Git receive hook rejects historical PNG/JPG blobs; bundle preserves original Git object IDs",
@@ -17,5 +17,9 @@
17
  "abc_transfer": {
18
  "pull-mp-opd-abc.sh": "779141eb7aecc11e37555c3768cb35aa5bff08545732dd14eb8adeec0a1092d5",
19
  "MP-OPD-ABC-HUONG-DAN.md": "5cb904fa6b32743cca47098e9ad0a6c900098304f097f4f63d24fe673b026a2e"
 
 
 
 
20
  }
21
  }
 
1
  {
2
  "source_branch": "vdt/ops/b200-portable",
3
+ "source_commit": "bbfef64f0d65e117b97595e704316e4afe642b8e",
4
  "bundle_file": "simct-b200-portable.bundle",
5
+ "bundle_sha256": "10f42cb4b625116aeb5a492eee42364a181527171d90895e19ee941fb58b11ed",
6
+ "bundle_bytes": 4159222,
7
  "history": "complete",
8
  "format": "git-bundle",
9
  "reason": "HF Git receive hook rejects historical PNG/JPG blobs; bundle preserves original Git object IDs",
 
17
  "abc_transfer": {
18
  "pull-mp-opd-abc.sh": "779141eb7aecc11e37555c3768cb35aa5bff08545732dd14eb8adeec0a1092d5",
19
  "MP-OPD-ABC-HUONG-DAN.md": "5cb904fa6b32743cca47098e9ad0a6c900098304f097f4f63d24fe673b026a2e"
20
+ },
21
+ "eval_queue": {
22
+ "pull-eval-queue.sh": "d20d18a6b22bae31313f13b26252f7032a290e4b229af842ddfe3a0d77ad8aa8",
23
+ "EVAL-QUEUE-RUNBOOK.md": "d66a8a79cd202626dea2998a6b28069eb9c9018a01794f8938aee14bb9c39b02"
24
  }
25
  }