Publish three-seed evaluation queue and oracle tokenizer fix bbfef64f
Browse files- EVAL-QUEUE-RUNBOOK.md +109 -0
- pull-eval-queue.sh +34 -0
- simct-b200-portable.bundle +2 -2
- source-manifest.json +7 -3
EVAL-QUEUE-RUNBOOK.md
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Company evaluation queue (three evaluation seeds)
|
| 2 |
+
|
| 3 |
+
This queue evaluates exactly one SFT initialization and the two existing MB4,
|
| 4 |
+
312-update trajectories. It does not train new models. Atomic/fixed snapshots:
|
| 5 |
+
40, 80, 120, 156, 200, 240, 280, 312; SFT is evaluated once as a snapshot.
|
| 6 |
+
Each snapshot runs seeds 42, 43, 44 over GSM8K, MATH500, original MBPP and LCB v6:
|
| 7 |
+
17 snapshots, 51 snapshot-seed combinations, 204 benchmark cells at most.
|
| 8 |
+
|
| 9 |
+
## Contract and interpretation
|
| 10 |
+
|
| 11 |
+
Profile `company-internal-v1` is explicit opt-in. It retains author generation
|
| 12 |
+
settings (temperature .6, top_p .95, one sample per request, benchmark token
|
| 13 |
+
caps), author GSM8K/MATH/MBPP helpers, and the pinned official LCB tester with
|
| 14 |
+
public and private tests. Functional LCB tasks receive the function name and
|
| 15 |
+
starter code in a dedicated Solution-method prompt. Math uses the author's
|
| 16 |
+
SymPy helpers, not math-verify. The three evaluation seeds measure sampling
|
| 17 |
+
variance, not independent training variance. Intermediate test-checkpoint
|
| 18 |
+
comparisons are exploratory; do not report the best test checkpoint as an
|
| 19 |
+
unbiased paper reproduction. Existing author-code/paper-spec sandbox profiles
|
| 20 |
+
are unchanged.
|
| 21 |
+
|
| 22 |
+
The company-internal subprocess executes generated Python under the current
|
| 23 |
+
user, with process time/address-space limits and a clean environment. **These
|
| 24 |
+
limits are not filesystem/network/container isolation.** This mode implements
|
| 25 |
+
the user's accepted internal-eval workflow for the namespace-restricted node.
|
| 26 |
+
Do not describe its timeout/security contract as identical to the paper.
|
| 27 |
+
|
| 28 |
+
Canonical fallback files total approximately 1 MB and are pinned by HF commit
|
| 29 |
+
and SHA256. Existing cached canonical files are reused. Company JSON copies
|
| 30 |
+
are compared against canonical normalized contents; incompatible copies are
|
| 31 |
+
not silently substituted. LCB reuses the already verified 1055-problem file,
|
| 32 |
+
indexes it without loading all hidden tests, and verifies each row before use.
|
| 33 |
+
No models or training data are downloaded. No dependencies are auto-installed.
|
| 34 |
+
The scorer needs numpy and sympy; parquet preparation needs pyarrow, available
|
| 35 |
+
in the existing datasets runtime if its installation is complete.
|
| 36 |
+
|
| 37 |
+
## Company commands
|
| 38 |
+
|
| 39 |
+
Pull `pull-eval-queue.sh` from the published, pinned HF revision and verify its
|
| 40 |
+
SHA256 using the accompanying chat commands. Running the pull script creates
|
| 41 |
+
an isolated checkout and prints EVAL_WORK. It never changes SimCT-git.
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
EVAL_WORK=/workspace/storage-shared/nlp/tungks/simct-eval-queue-REPLACE
|
| 45 |
+
bash "$EVAL_WORK/source/experiments/runai/transfer/prepare-eval-queue.sh"
|
| 46 |
+
# The following command starts GPU workers; run only when ready.
|
| 47 |
+
bash "$EVAL_WORK/source/experiments/runai/transfer/start-eval-queue.sh"
|
| 48 |
+
source "$EVAL_WORK/queue.env"
|
| 49 |
+
/usr/bin/python3.12 "$EVAL_WORK/source/scripts/evaluation/eval_queue.py" summarize --plan "$EVAL_PLAN"
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
Preparation qualifies ten synthetic correct/incorrect scoring fixtures, checks
|
| 53 |
+
the two completed training summaries, and hashes every chosen checkpoint.
|
| 54 |
+
Hashing roughly 17 model snapshots may take several minutes on shared storage.
|
| 55 |
+
It does not consume the GPU budget. Missing snapshots stop preparation rather
|
| 56 |
+
than changing the selection silently.
|
| 57 |
+
|
| 58 |
+
## Budget, ordering, resume
|
| 59 |
+
|
| 60 |
+
The first worker completing CPU qualification starts one shared 20-hour wall
|
| 61 |
+
clock. New snapshots stop being admitted at 18 hours (or earlier if the slowest
|
| 62 |
+
observed snapshot would not fit). A server watchdog ends the worker-owned
|
| 63 |
+
server at the deadline; process cleanup can take a few seconds beyond it.
|
| 64 |
+
Unfinished cells remain partial. There is no guarantee all 204 cells fit.
|
| 65 |
+
Both workers take snapshots independently, with barriers between these tiers:
|
| 66 |
+
|
| 67 |
+
1. SFT, atomic312, fixed312, atomic156, fixed156.
|
| 68 |
+
2. Atomic/fixed80 and240.
|
| 69 |
+
3. Atomic/fixed40 and200.
|
| 70 |
+
4. Atomic/fixed120 and280.
|
| 71 |
+
|
| 72 |
+
Every selected snapshot finishes all three seeds before its worker moves on.
|
| 73 |
+
GPU memory >=1 GiB causes waiting; existing training/oracle jobs are never
|
| 74 |
+
killed. Per-GPU and per-job file locks prevent duplicate queue work. Resource
|
| 75 |
+
ownership outside this queue still requires the operator to avoid launching
|
| 76 |
+
another workload onto an occupied GPU.
|
| 77 |
+
|
| 78 |
+
Workers append and fsync per-question responses and scores. Resume checks
|
| 79 |
+
source, checkpoint, data, request, response and server identities. Running
|
| 80 |
+
start-eval-queue.sh again resumes the same plan and clock; it does not grant
|
| 81 |
+
another 20 hours. Active duplicate workers fail their GPU lock. A failed job
|
| 82 |
+
halts admissions: inspect worker/server logs, fix the cause, then use
|
| 83 |
+
`eval_queue.py retry --plan "$EVAL_PLAN"` and start the workers again. Source or
|
| 84 |
+
runtime changes that break the recorded contract require a new plan and must
|
| 85 |
+
not be merged into old results. Do not delete state.json to extend the clock.
|
| 86 |
+
|
| 87 |
+
Summary includes a benchmark mean and sample standard deviation only when all
|
| 88 |
+
three seeds are complete and hashes match. A checkpoint average requires all
|
| 89 |
+
four benchmarks; partial results are not zero-filled. Logs and result files
|
| 90 |
+
stay under the plan directory. Nothing is uploaded to W&B automatically.
|
| 91 |
+
|
| 92 |
+
## Verification boundary
|
| 93 |
+
|
| 94 |
+
Offline regression tests cover selection, tier barriers, deadline admission,
|
| 95 |
+
resume journals, cached responses, source-row integrity, and ten actual CPU
|
| 96 |
+
scorer fixtures including functional LCB. SGLang GPU startup and end-to-end
|
| 97 |
+
throughput still need verification on the company B200 node.
|
| 98 |
+
|
| 99 |
+
## Mechanics tokenizer fix included
|
| 100 |
+
|
| 101 |
+
The same source bundle fixes `real_oracle.py` prompt tokenization: explicitly
|
| 102 |
+
request `return_dict=False`, normalize mapping/Encoding outputs into integer
|
| 103 |
+
IDs, and create torch.long input tensors. The old failure happened before the
|
| 104 |
+
first group result; it is not oracle evidence. Reuse the existing prepared
|
| 105 |
+
six-row mechanics JSON in the fresh bundle workspace, then run the copied
|
| 106 |
+
03-run-mechanics.sh on GPU1 before starting the evaluation workers. GPU1's
|
| 107 |
+
eval worker will wait while mechanics owns memory. GPU0 can evaluate meanwhile.
|
| 108 |
+
The tokenizer regression uses CPU mocks; the corrected real-model canary has
|
| 109 |
+
not yet run on the company node.
|
pull-eval-queue.sh
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
set -euo pipefail
|
| 3 |
+
ROOT=/workspace/storage-shared/nlp/tungks
|
| 4 |
+
WORK="$(mktemp -d "$ROOT/simct-eval-queue-XXXXXXXX")"
|
| 5 |
+
PROXY="${MP_DOWNLOAD_PROXY:-http://10.30.154.118:80}"
|
| 6 |
+
fetch() { curl --fail --location --retry 3 --proxy "$PROXY" --noproxy "" "$1" -o "$2"; }
|
| 7 |
+
if [[ -z "${HF_REVISION:-}" ]]; then
|
| 8 |
+
fetch 'https://huggingface.co/api/models/codemaivanngu/simct' "$WORK/hf-info.json"
|
| 9 |
+
HF_REVISION="$(/usr/bin/python3.12 -c 'import json,sys; print(json.load(open(sys.argv[1]))["sha"])' "$WORK/hf-info.json")"
|
| 10 |
+
fi
|
| 11 |
+
[[ "$HF_REVISION" =~ ^[0-9a-f]{40}$ ]] || { echo 'Invalid HF revision'; exit 2; }
|
| 12 |
+
BASE="https://huggingface.co/codemaivanngu/simct/resolve/$HF_REVISION"
|
| 13 |
+
fetch "$BASE/source-manifest.json" "$WORK/source-manifest.json"
|
| 14 |
+
fetch "$BASE/simct-b200-portable.bundle" "$WORK/simct-b200-portable.bundle"
|
| 15 |
+
/usr/bin/python3.12 - "$WORK" <<'PY'
|
| 16 |
+
import hashlib,json,sys
|
| 17 |
+
from pathlib import Path
|
| 18 |
+
root=Path(sys.argv[1]); m=json.loads((root/'source-manifest.json').read_text())
|
| 19 |
+
assert m['bundle_file']=='simct-b200-portable.bundle'
|
| 20 |
+
assert hashlib.sha256((root/m['bundle_file']).read_bytes()).hexdigest()==m['bundle_sha256'], 'bundle checksum mismatch'
|
| 21 |
+
assert m['source_branch']=='vdt/ops/b200-portable'
|
| 22 |
+
print('BUNDLE_SHA256_PASS')
|
| 23 |
+
PY
|
| 24 |
+
git clone --branch vdt/ops/b200-portable "$WORK/simct-b200-portable.bundle" "$WORK/source"
|
| 25 |
+
EXPECTED="$(/usr/bin/python3.12 -c 'import json,sys; print(json.load(open(sys.argv[1]))["source_commit"])' "$WORK/source-manifest.json")"
|
| 26 |
+
[[ "$(git -C "$WORK/source" rev-parse HEAD)" == "$EXPECTED" ]]
|
| 27 |
+
for NAME in 02-prepare-mechanics.sh 03-run-mechanics.sh; do
|
| 28 |
+
cp "$WORK/source/experiments/runai/transfer/$NAME" "$WORK/$NAME"
|
| 29 |
+
done
|
| 30 |
+
printf 'export ABC_WORK=%q\n' "$WORK" > "$WORK/workspace.env"
|
| 31 |
+
printf 'export EVAL_WORK=%q\n' "$WORK" > "$WORK/eval.env"
|
| 32 |
+
printf 'HF_REVISION=%s\nSOURCE_COMMIT=%s\nEVAL_WORK=%s\n' "$HF_REVISION" "$EXPECTED" "$WORK"
|
| 33 |
+
echo "NEXT: bash $WORK/source/experiments/runai/transfer/prepare-eval-queue.sh"
|
| 34 |
+
echo 'No GPU started; existing training checkout unchanged.'
|
simct-b200-portable.bundle
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:10f42cb4b625116aeb5a492eee42364a181527171d90895e19ee941fb58b11ed
|
| 3 |
+
size 4159222
|
source-manifest.json
CHANGED
|
@@ -1,9 +1,9 @@
|
|
| 1 |
{
|
| 2 |
"source_branch": "vdt/ops/b200-portable",
|
| 3 |
-
"source_commit": "
|
| 4 |
"bundle_file": "simct-b200-portable.bundle",
|
| 5 |
-
"bundle_sha256": "
|
| 6 |
-
"bundle_bytes":
|
| 7 |
"history": "complete",
|
| 8 |
"format": "git-bundle",
|
| 9 |
"reason": "HF Git receive hook rejects historical PNG/JPG blobs; bundle preserves original Git object IDs",
|
|
@@ -17,5 +17,9 @@
|
|
| 17 |
"abc_transfer": {
|
| 18 |
"pull-mp-opd-abc.sh": "779141eb7aecc11e37555c3768cb35aa5bff08545732dd14eb8adeec0a1092d5",
|
| 19 |
"MP-OPD-ABC-HUONG-DAN.md": "5cb904fa6b32743cca47098e9ad0a6c900098304f097f4f63d24fe673b026a2e"
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
}
|
| 21 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"source_branch": "vdt/ops/b200-portable",
|
| 3 |
+
"source_commit": "bbfef64f0d65e117b97595e704316e4afe642b8e",
|
| 4 |
"bundle_file": "simct-b200-portable.bundle",
|
| 5 |
+
"bundle_sha256": "10f42cb4b625116aeb5a492eee42364a181527171d90895e19ee941fb58b11ed",
|
| 6 |
+
"bundle_bytes": 4159222,
|
| 7 |
"history": "complete",
|
| 8 |
"format": "git-bundle",
|
| 9 |
"reason": "HF Git receive hook rejects historical PNG/JPG blobs; bundle preserves original Git object IDs",
|
|
|
|
| 17 |
"abc_transfer": {
|
| 18 |
"pull-mp-opd-abc.sh": "779141eb7aecc11e37555c3768cb35aa5bff08545732dd14eb8adeec0a1092d5",
|
| 19 |
"MP-OPD-ABC-HUONG-DAN.md": "5cb904fa6b32743cca47098e9ad0a6c900098304f097f4f63d24fe673b026a2e"
|
| 20 |
+
},
|
| 21 |
+
"eval_queue": {
|
| 22 |
+
"pull-eval-queue.sh": "d20d18a6b22bae31313f13b26252f7032a290e4b229af842ddfe3a0d77ad8aa8",
|
| 23 |
+
"EVAL-QUEUE-RUNBOOK.md": "d66a8a79cd202626dea2998a6b28069eb9c9018a01794f8938aee14bb9c39b02"
|
| 24 |
}
|
| 25 |
}
|