ctopd / docs /MP_OPD.md
codemaivanngu's picture
fix(mp-opd): exclude unalignable responses with reason metrics
0eeed4c unverified
|
Raw History Blame Contribute Delete
8.71 kB
# Isolated MP-OPD port: Atomic, Fixed, Random, GRASS-Chunk and ALIGN
These are experimental on-policy modes, ported from Tung's SimCT/MP-OPD checkout.
They are not aliases for Strict full/top-16, and their results must not be reported
as reproduction of the CTOPD paper's Strict objective or the full SimCT recipe.
Models, data, teacher warm start, optimizer, rollout and evaluation still come from
the CTOPD launcher. The initial adapter requires one student rank, ring size 1,
`kd_ratio=1`, `kd_temperature=1`, and `kd_loss_fn=rkl`.
## Available presets
| `tools/launch_single_b200.py --algorithm` | Registered algorithm | Intervention |
| --- | --- | --- |
| `mp-atomic` | `mp_opd_atomic` | Original atom credits |
| `mp-fixed1` through `mp-fixed5` | `mp_opd_fixed` | Hard pooling over 1 through 5 atoms |
| `mp-random1-2` | `mp_opd_random` | Hard pooling, group length sampled uniformly from [1,2] |
| `mp-random2-5` | `mp_opd_random` | Hard pooling, group length sampled uniformly from [2,5] |
| `mp-grass-chunk` | `mp_opd_grass_chunk` | Risk-adaptive shrinkage within supplied chunks; no DP |
| `mp-align` | `mp_opd_align` | Donor ALIGN v1 cyclic coefficient-space PCGrad within the same chunks |
| `mp-align-new` | `mp_opd_align_new` | Local ALIGN plus full-response head-descent guard; see [ALIGN_NEW.md](ALIGN_NEW.md) |
| `mp-trust-r` | `mp_opd_trust_r` | Strict plus mismatch scaled once per response |
| `mp-trust-b` | `mp_opd_trust_b` | Strict plus mismatch scaled once per complete optimizer batch |
| `mp-trust-p` | `mp_opd_trust_p` | Project opposing mismatch component through a complete-batch strict scale |
See [TRUST_PAPER.md](TRUST_PAPER.md) for the two-pass B/P actor contract and the
separate `simct_paper` score reconstruction. TRUST uses strict/mismatch atom types,
not the `xtoken` chunk partition used by ALIGN/GRASS.
Fixed/random groups tile each response independently. The final tail may be
shorter than the requested length/minimum. Randomness uses the configured seed
and response token IDs, independent of microbatch row/rank and global Python RNG.
This intentionally improves grouping stability over the donor's row-index seed;
it is a documented integration difference, not identical donor RNG replay.
## Run and dry-run
From this checkout, print the full workload without importing ML libraries,
creating a run, or using a GPU:
```bash
python3 tools/launch_single_b200.py --algorithm mp-align \
--runtime /opt/ctopd-b200-u22 --dry-run
python3 tools/launch_single_b200.py --algorithm mp-grass-chunk \
--runtime /opt/ctopd-b200-u22 --dry-run
python3 tools/launch_single_b200.py --algorithm mp-fixed5 --dry-run
python3 tools/launch_single_b200.py --algorithm mp-random2-5 --dry-run
```
Remove `--dry-run` only in the actual allocated B200 environment, exposing the one
GPU assigned to the run through `CUDA_VISIBLE_DEVICES`. Models/data must already
exist at the launcher's paths. Each run needs a fresh run ID. These presets retain
512 rollout/optimizer responses, max generation 8192, 100 iterations, and step-0
plus every-10-step MATH500 evaluation with 32 responses/problem. Defaults for the
two Strict presets remain unchanged. Evaluation is still MATH500, not the full
paper benchmark suite.
GRASS-Chunk and ALIGN default to `--mp-chunk-source xtoken`, using the same
vendored tokenizer/string aligner as the donor. This use does not need a learned
vocabulary projection. `--mp-chunk-source run --mp-chunk-run-length 2` explicitly
chooses the fixed-run ablation instead. `--mp-geometry diag` is a separate diagonal
geometry ablation; the default is `exact_head`. ALIGN here is v1 `align`, not
`align_new` or a full-Transformer/global-descent guard. The separately ported
`mp-align-new` has a full-response **head** guard; it still does not measure the
full Transformer gradient. The donor does not name these modes v2/v3.
## Scientific and numerical contract
Each response is byte-checked and partitioned into the donor's minimal synchronized
atoms. Terminal EOS is excluded from this mode family's credits (Strict's EOS
handling is unchanged). Atom credit is the detached teacher-minus-current-student
log-score, divided by student token count. The student loss is detached effective
credit times differentiable atom NLL, divided by the existing shared optimizer-window
token normalizer. It is a sampled path-credit surrogate and may be negative; it is
not the directly enumerated shared-vocabulary KL used by Strict.
GRASS reuses the corrected donor span-wide distortion, noise estimator and exact
LM-head Gram. The noise estimator updates per response and excludes adjacency
across chunk boundaries. Weighted signed credit is checked after dtype conversion;
evaluation uses the stored noise estimate without updating it. An invalid result
fails rather than silently switching to Atomic. ALIGN does not
claim credit conservation: its intervention is deterministic direction-conflict
projection. All geometry is detached. Geometry flags restore process-global FP32
settings; disabling `MP_OPD_GRASS_EXACT_FP32` is rejected for these two modes.
The adapter applies the configured student/teacher final-logit softcap to raw
linear heads. Token tiles use checkpointed log-prob computation; changing
`chunked_loss_size` does not set research chunk boundaries. Exact-head geometry
still materializes per-response logits and an atom Gram, so long-response GPU
memory/throughput remain unqualified. No new wheels or native extension are added.
Complete responses with `replacement_character`, `normalization_mismatch` or
`empty_response` contribute zero. The adapter never sanitizes/relabels their text
or substitutes an Atomic loss. All other atomization failures remain fatal.
The shared optimizer-window token denominator is retained, including excluded
tokens; remaining responses are not reweighted per microbatch. Exclusion counts,
reason fractions and excluded token counts are logged under `distill/mp_opd_*`.
This policy changes the effective training sample distribution and can reduce
gradient magnitude; inspect exclusions before comparing results. It prevents
unsafe cross-text scoring but does not repair the source of retokenization mismatch.
See [INVALID_RESPONSES.md](INVALID_RESPONSES.md) for the policy and qualification.
Singleton alignment chunks
leave GRASS/ALIGN unchanged; inspect singleton fractions and coefficient-change
metrics before claiming an intervention occurred.
`state_dict`/`load_state_dict` expose algorithm state and reject a changed mode or
MP-OPD contract. They do not add full training resume: existing HF model exports do
not save optimizer/RNG/GRASS noise state. The launcher continues to refuse existing
run directories. Fresh-run comparisons only are supported by this integration.
## Isolation, provenance and verification
`kdflow/algorithms/mp_opd.py` is the new adapter. Donor kernels live only in
`kdflow/algorithms/_mp_opd_port/` and are imported lazily after selecting an MP mode.
The three existing algorithm files, byte alignment, RKL and tiled loss are
byte-identical to the pre-port checkpoint. Shared edits only register names,
validate the new flags, allow them through the on-policy CLI, attach teacher
softcap metadata for MP modes, and provide launcher presets.
`MP_OPD_SOURCE.json` records the donor commit, every source/ported file hash,
line-ending/import changes, and frozen Strict source hashes. The donor repository's MIT
notice and the NVIDIA aligner's Apache-2.0 notice/license are retained alongside
the vendored files. The donor checkout's unrelated staged diagnostic files were
not modified or committed here.
Run CPU verification using an existing compatible PyTorch environment; do not
install dependencies merely to run this command:
```bash
TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v
```
The tests cover donor Atomic loss/gradient parity, Fixed1 identity, fixed/random
coverage, RNG independence, per-response/microbatch normalization, numerical tile
invariance, GRASS identities/conservation, explicit-autograd head-Gram parity with
softcapping, nontrivial ALIGN projection, native/run chunk backward, source hashes,
and existing Strict loss/gradient regression. A Python environment without torch
skips tensor tests explicitly; it is not a full verification pass.
The checked CPU environment and commands are recorded in
[MP_OPD_VALIDATION.json](MP_OPD_VALIDATION.json).
CPU tests establish implementation checks only. B200 execution, FSDP runtime
integration, full-size memory/throughput, training resume and scientific efficacy
have not been qualified for the new modes. No training or GPU job is launched by
the port or its tests.