File size: 6,172 Bytes
b29a766 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | # CPU validation record
Date: 2026-09-20 (Asia/Bangkok).
No model weights or benchmark datasets were loaded. No GPU experiment was run.
Round 2 was implemented and its disabled-launcher guard was tested; the round
itself was not executed.
From `verl-0.7.0/examples/nash_exp`, the full test command was:
```bash
/tmp/nash-diagnostic-b-venv/bin/python3 -m pytest -q tests/diagnostic_b
```
Actual result after the author settings and boxed-answer revision: **172 passed, 10 warnings in 12.73 seconds**. The ten CVXPY
warnings report an inaccurate solver status on numerical fixtures. They are
preserved; accepted solutions must separately satisfy the declared feasibility,
certificate, and objective-gap checks. No test was skipped or weakened.
The suite covers population-standard-deviation advantages, exact original-token
segmentation, output-head/autograd identity, temperature differentiation,
held-out denominator and token-sum normalization, collision-power degeneracy and
tie-breaking, matched feasible controls, a strict Nash/Linear witness, support
identities, held-out isolation, paired prompt-cluster uncertainty, schema and
prompt validation, symbolic equivalence and parse failures, one shared generation
engine, request-level resume, cache corruption/mismatch checks, resource
preflight, configuration gates, and single-file failure logging.
The DelTA revision additionally checks the Appendix-F proxy against each
sampled output row's autograd derivative, sharing hidden coordinates across
different token IDs, weighted response/segment aggregation, production proxy
Grams and cross products, original-token prediction alignment, held-out token
sums and all-draw normalization, feature-cache geometry and checksum rejection,
geometry-specific preflight memory estimates, report labels, rejection of mixed
geometries, and reuse of generated responses when switching feature backends.
The original full-head reference and production-backend tests still pass.
The boxed-answer revision checks that the shipped prompt adds the same boxed
instruction to DAPO and held-out questions, preserves earlier DAPO messages and
the original dataset rows, and excludes reference answers. Real Math-Verify CPU
tests check nested fractions, symbolic equivalence, rejection of unboxed numbers,
selection of the final box, malformed final boxes without fallback, and escaped
LaTeX braces. Missing-setting tests now build explicitly unresolved fixtures;
they still check that configuration validation runs before external imports.
These additional checks passed:
```bash
ruff check diagnostic_b tests/diagnostic_b
ruff format --check diagnostic_b tests/diagnostic_b
python3 -m mypy --follow-imports=skip diagnostic_b
python3 -m compileall -q diagnostic_b tests/diagnostic_b
bash -n scripts/run_diagnostic_b_round1_gpu.sh scripts/run_diagnostic_b_round2_gpu.sh
```
Ruff used the parent veRL configuration. Mypy reported no issues in 14 source
files under the repository's existing configuration; that configuration ignores
errors by default outside its selected veRL modules, so this is not a claim of
strict static type coverage. Shell syntax and Python compilation checks passed.
An actual standalone CPU smoke run also completed:
```bash
/tmp/nash-diagnostic-b-venv/bin/python3 -m diagnostic_b \
--cpu-smoke --output-dir /tmp/nash-diagnostic-b-delta-smoke --resume
```
It produced compressed shared caches, direction records, CSV/JSON/Markdown/LaTeX
tables, PDF/PNG figures, and a report labeled `round_cpu_smoke`. The three-panel
scatter was visually inspected and both axes name the DelTA token-gradient proxy.
These are synthetic fixture outputs, not Qwen
results. The CLI tests additionally run smoke/resume in fresh subprocesses while
blocking imports of all model/dataset loading libraries and verify unchanged
rollout hashes and modification times on resume.
The isolated test environment used Python 3.10.12, CPU PyTorch 2.14.0+cpu,
NumPy 2.2.6, SciPy 1.15.3, CVXPY 1.7.5, Clarabel 0.11.1, pandas 2.3.3,
PyArrow 25.0.1, matplotlib 3.10.9, pytest 9.1.1, Ruff 0.12.2, mypy 1.17.0,
and Math-Verify 0.8.0. This environment was created under `/tmp`; the project
does not depend on that path for server execution.
The author selected seed 42, `sungyub/dapo-math-17k-verl`, the boxed prompt,
Math-Verify boxed-answer scoring, radius coefficient 0.4, and a requested response
cap of 32768. Dataset metadata resolved the training revision to
`3cf5c112137795c1f1d5fdae7c747871b26840a4`; no full dataset was downloaded.
A real CLI dry run with the updated Round-1 YAML stopped before external loading
with only the response/context conflict: a 32768-token response cap leaves no
space for a prompt inside the fixed 32768-token context. The requested value is
preserved pending the author's choice of a smaller fixed response cap or an
explicit remaining-context policy. Its log is
`/tmp/nash-author-settings-dry-run/diagnostic_b_round1.log`.
Real vLLM/Transformers execution, benchmark schema snapshots, memory peaks,
throughput, and measured correlations still require the author's server run.
Preflight records and checks the actual server configuration before costly
generation begins.
## Server setup validation
The simplified `scripts/setup_server.sh` has two offline Bash tests in
`tests/diagnostic_b/test_server_setup.py`: **2 passed in 1.32 seconds**.
They cover the Python 3.10 environment request, ordered editable/requirements/
vLLM/FlashAttention installs, the exact requested wheel URL, paths with spaces,
activation, reuse of an existing environment, and retry after a failed wheel
download. Package installation and wget are mocked; the Bash script runs normally.
Ruff and shell syntax checks passed. No GPU packages were installed locally.
The earlier metadata-only dependency resolution covered editable veRL, root and
experiment requirements and vLLM 0.11.0 with the existing server constraints. It
selected PyTorch 2.8.0+cu128, torchvision 0.23.0+cu128, torchaudio 2.8.0+cu128,
Transformers 4.57.6 and NumPy 1.26.4 for Python 3.10. This did not test the newly
requested FlashAttention wheel's binary imports or GPU runtime compatibility.
|