feat(kd): add TRUST R B P and separate SimCT paper scores
Browse files- README.md +5 -0
- docs/MP_OPD.md +7 -0
- docs/SIMCT.md +4 -0
- docs/SINGLE_B200.md +2 -0
- docs/TRUST_PAPER.md +123 -0
- docs/TRUST_PAPER_SOURCE.json +70 -0
- docs/TRUST_PAPER_VALIDATION.json +45 -0
- kdflow/algorithms/__init__.py +2 -0
- kdflow/algorithms/_mp_opd_port/_mp_opd_trust.py +348 -0
- kdflow/algorithms/_mp_opd_port/_mp_opd_trust_batch.py +132 -0
- kdflow/algorithms/_mp_opd_port/_mp_opd_trust_p.py +104 -0
- kdflow/algorithms/mp_opd_trust.py +159 -0
- kdflow/algorithms/simct_paper.py +22 -0
- kdflow/algorithms/trust_training.py +78 -0
- kdflow/arguments/distillation_args.py +10 -3
- kdflow/cli/train_kd_on_policy.py +3 -2
- kdflow/mp_opd_modes.py +1 -0
- kdflow/ray/train/student_actor.py +6 -0
- tests/fixtures/trust/_mp_opd_trust.py.txt +348 -0
- tests/fixtures/trust/_mp_opd_trust_batch.py.txt +132 -0
- tests/fixtures/trust/_mp_opd_trust_p.py.txt +104 -0
- tests/fixtures/trust/_trust_r_loss.py.txt +84 -0
- tests/test_single_b200.py +14 -0
- tests/test_trust_paper.py +389 -0
- tools/launch_single_b200.py +12 -7
README.md
CHANGED
|
@@ -24,6 +24,11 @@ A separate [author-code SimCT baseline](docs/SIMCT.md) uses `--algorithm simct`
|
|
| 24 |
in the same launcher. It preserves the pinned released implementation, with
|
| 25 |
original-source loss/gradient parity tests and an explicit paper/code distinction.
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
## Original project documentation
|
| 28 |
|
| 29 |
This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
|
|
|
|
| 24 |
in the same launcher. It preserves the pinned released implementation, with
|
| 25 |
original-source loss/gradient parity tests and an explicit paper/code distinction.
|
| 26 |
|
| 27 |
+
Separate [TRUST-R/B/P and SimCT paper-score modes](docs/TRUST_PAPER.md) provide
|
| 28 |
+
`mp-trust-r`, `mp-trust-b`, `mp-trust-p` and `simct_paper`. B/P calibrate the complete
|
| 29 |
+
optimizer window; `simct_paper` corrects log-probability scores while `simct` stays
|
| 30 |
+
the released author-code baseline.
|
| 31 |
+
|
| 32 |
## Original project documentation
|
| 33 |
|
| 34 |
This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
|
docs/MP_OPD.md
CHANGED
|
@@ -18,6 +18,13 @@ the CTOPD launcher. The initial adapter requires one student rank, ring size 1,
|
|
| 18 |
| `mp-grass-chunk` | `mp_opd_grass_chunk` | Risk-adaptive shrinkage within supplied chunks; no DP |
|
| 19 |
| `mp-align` | `mp_opd_align` | Donor ALIGN v1 cyclic coefficient-space PCGrad within the same chunks |
|
| 20 |
| `mp-align-new` | `mp_opd_align_new` | Local ALIGN plus full-response head-descent guard; see [ALIGN_NEW.md](ALIGN_NEW.md) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
Fixed/random groups tile each response independently. The final tail may be
|
| 23 |
shorter than the requested length/minimum. Randomness uses the configured seed
|
|
|
|
| 18 |
| `mp-grass-chunk` | `mp_opd_grass_chunk` | Risk-adaptive shrinkage within supplied chunks; no DP |
|
| 19 |
| `mp-align` | `mp_opd_align` | Donor ALIGN v1 cyclic coefficient-space PCGrad within the same chunks |
|
| 20 |
| `mp-align-new` | `mp_opd_align_new` | Local ALIGN plus full-response head-descent guard; see [ALIGN_NEW.md](ALIGN_NEW.md) |
|
| 21 |
+
| `mp-trust-r` | `mp_opd_trust_r` | Strict plus mismatch scaled once per response |
|
| 22 |
+
| `mp-trust-b` | `mp_opd_trust_b` | Strict plus mismatch scaled once per complete optimizer batch |
|
| 23 |
+
| `mp-trust-p` | `mp_opd_trust_p` | Project opposing mismatch component through a complete-batch strict scale |
|
| 24 |
+
|
| 25 |
+
See [TRUST_PAPER.md](TRUST_PAPER.md) for the two-pass B/P actor contract and the
|
| 26 |
+
separate `simct_paper` score reconstruction. TRUST uses strict/mismatch atom types,
|
| 27 |
+
not the `xtoken` chunk partition used by ALIGN/GRASS.
|
| 28 |
|
| 29 |
Fixed/random groups tile each response independently. The final tail may be
|
| 30 |
shorter than the requested length/minimum. Randomness uses the configured seed
|
docs/SIMCT.md
CHANGED
|
@@ -3,6 +3,10 @@
|
|
| 3 |
Select `simct` to run the author's released `span_ctkd` objective. This is separate
|
| 4 |
from Strict and the MP-OPD modes: `mp-atomic`/`mp-fixed1` are not SimCT.
|
| 5 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
## Source and exact scope
|
| 7 |
|
| 8 |
The source is [sunjie279/SimCT-](https://github.com/sunjie279/SimCT-/tree/cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e),
|
|
|
|
| 3 |
Select `simct` to run the author's released `span_ctkd` objective. This is separate
|
| 4 |
from Strict and the MP-OPD modes: `mp-atomic`/`mp-fixed1` are not SimCT.
|
| 5 |
|
| 6 |
+
The separate [paper-score reconstruction](TRUST_PAPER.md), `simct_paper`, corrects
|
| 7 |
+
Eq.7-8 score normalization on this same candidate/alignment interface. The author
|
| 8 |
+
baseline described below remains unchanged.
|
| 9 |
+
|
| 10 |
## Source and exact scope
|
| 11 |
|
| 12 |
The source is [sunjie279/SimCT-](https://github.com/sunjie279/SimCT-/tree/cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e),
|
docs/SINGLE_B200.md
CHANGED
|
@@ -8,6 +8,8 @@ The separate [SimCT author-code preset](SIMCT.md), `--algorithm simct`, uses the
|
|
| 8 |
|
| 9 |
The separate [ALIGN-NEW preset](ALIGN_NEW.md), `--algorithm mp-align-new`, uses the same workload with the donor's full-response head guard. It requires exact-head geometry and adds no GPU/full-training qualification.
|
| 10 |
|
|
|
|
|
|
|
| 11 |
## Preserved training and evaluation settings
|
| 12 |
|
| 13 |
| Setting | Value |
|
|
|
|
| 8 |
|
| 9 |
The separate [ALIGN-NEW preset](ALIGN_NEW.md), `--algorithm mp-align-new`, uses the same workload with the donor's full-response head guard. It requires exact-head geometry and adds no GPU/full-training qualification.
|
| 10 |
|
| 11 |
+
The [TRUST and paper-score presets](TRUST_PAPER.md) add `mp-trust-r/b/p` (separate choices) and `simct_paper` on the same workload. TRUST-B/P require complete-window two-pass calibration; `simct_paper` retains the released candidate/alignment interface while correcting score normalization.
|
| 12 |
+
|
| 13 |
## Preserved training and evaluation settings
|
| 14 |
|
| 15 |
| Setting | Value |
|
docs/TRUST_PAPER.md
ADDED
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# TRUST-R/B/P and SimCT paper-score modes
|
| 2 |
+
|
| 3 |
+
These opt-in source modes retain the existing CTOPD data/model/optimizer/evaluation
|
| 4 |
+
pipeline. They add no SFT warmup and do not reproduce either paper's full recipe.
|
| 5 |
+
Existing Strict, SimCT author-code, Atomic/Fixed/Random, GRASS and ALIGN files stay
|
| 6 |
+
unchanged. Shared actor branches activate only for the new TRUST-B/P modes.
|
| 7 |
+
|
| 8 |
+
| Launcher algorithm | Registered mode | Loss intervention |
|
| 9 |
+
| --- | --- | --- |
|
| 10 |
+
| `mp-trust-r` | `mp_opd_trust_r` | One lambda per response: strict + lambda*mismatch |
|
| 11 |
+
| `mp-trust-b` | `mp_opd_trust_b` | One lambda for the complete optimizer batch |
|
| 12 |
+
| `mp-trust-p` | `mp_opd_trust_p` | One strict-side projection scale for the complete optimizer batch; mismatch kept |
|
| 13 |
+
| `simct` | `simct` | Unchanged released author-code raw-logit span scores |
|
| 14 |
+
| `simct_paper` | `simct_paper` | Full-native-vocabulary log-probability scores before virtual-vocabulary normalization |
|
| 15 |
+
|
| 16 |
+
## TRUST provenance and scope
|
| 17 |
+
|
| 18 |
+
The numerical kernels are copied byte for byte from
|
| 19 |
+
`sontungkieu/SimCT@529ee4b691732e1040d6a45f05fd12f58def96ec`:
|
| 20 |
+
`_mp_opd_trust.py`, `_mp_opd_trust_batch.py`, `_mp_opd_trust_p.py`. Existing frozen
|
| 21 |
+
head-pair and Gram dependencies are reused. [TRUST_PAPER_SOURCE.json](TRUST_PAPER_SOURCE.json)
|
| 22 |
+
records original fixtures and unchanged existing algorithm hashes. The donor MIT
|
| 23 |
+
notice remains in the vendored package.
|
| 24 |
+
|
| 25 |
+
Strict atoms are exactly `one_to_one` from byte-synchronized atomization; mismatch
|
| 26 |
+
atoms are `multi_token`. Unknown types fail. There is no extra `xtoken` chunking
|
| 27 |
+
for TRUST. Credits, detached rates, summed atom NLL, EOS masking and common token
|
| 28 |
+
normalizer are those of the existing MP-OPD adapter.
|
| 29 |
+
Here "strict loss" is the atomic path-credit surrogate on `one_to_one` atoms;
|
| 30 |
+
it is not the enumerated shared-vocabulary KL of the separate Strict presets.
|
| 31 |
+
|
| 32 |
+
Let S and M be the summed output-head gradients of strict/mismatch atom losses.
|
| 33 |
+
TRUST-R and B use `lambda=max(0,<S,M>/||M||^2)` with donor rules for empty or
|
| 34 |
+
degenerate sides; strict loss keeps coefficient 1. R uses one response, B the
|
| 35 |
+
entire optimizer window. P uses
|
| 36 |
+
`strict_scale=1-min(0,<S,M>)/||S||^2`, keeping mismatch coefficient 1; its donor
|
| 37 |
+
rounding protects the halfspace. With no usable strict anchor, P retains Atomic
|
| 38 |
+
and flags the waived guarantee. These are head-output-edge proxies, including
|
| 39 |
+
bias/softcap, not full-Transformer or Adam descent guarantees.
|
| 40 |
+
|
| 41 |
+
### Complete-window calibration
|
| 42 |
+
|
| 43 |
+
B/P cannot silently calibrate each microbatch. A first train-mode no-grad pass
|
| 44 |
+
streams two full-head accumulators across every response, including all
|
| 45 |
+
cross-response terms. Parameters stay fixed. The second pass restores each
|
| 46 |
+
microbatch's Python/NumPy/Torch/CUDA RNG before backward, applies one detached
|
| 47 |
+
calibration to the whole optimizer window and advances logical RNG only once.
|
| 48 |
+
Only the normal final accumulated step updates parameters. Calibration state is
|
| 49 |
+
cleared afterward, including on failures; no graphs or batch logits are stashed.
|
| 50 |
+
|
| 51 |
+
One student rank/ring size 1, a complete optimizer window starting at step 0,
|
| 52 |
+
and one finite positive token normalizer are required. Partial windows, missing
|
| 53 |
+
preparation, changed normalizers and invalid geometry fail explicitly. No legacy
|
| 54 |
+
microbatch TRUST-B alias is provided. The donor's historical error-to-Atomic
|
| 55 |
+
fallback is not reproduced: invalid geometry/memory checks fail with the cause.
|
| 56 |
+
This difference affects failure handling, not the valid-input loss parity claim.
|
| 57 |
+
|
| 58 |
+
Evaluation loss uses its own evaluation-set calibration, unit sample weights,
|
| 59 |
+
no backward/optimizer update and no training-calibration reuse. The original
|
| 60 |
+
student train/eval mode is restored. MATH500 generation evaluation is unchanged.
|
| 61 |
+
|
| 62 |
+
TRUST requires unit RKL weight/temperature and exact-head geometry.
|
| 63 |
+
`mp_opd_trust_eps_g=1e-12`; `mp_opd_trust_vocab_chunk=4096`. B/P require two
|
| 64 |
+
FP32 [vocabulary, hidden] accumulators (plus bias when present), so full-size
|
| 65 |
+
memory and throughput still need GPU qualification. No runtime rebuild is needed.
|
| 66 |
+
|
| 67 |
+
## What `simct_paper` means
|
| 68 |
+
|
| 69 |
+
This mode corrects the score calculation to
|
| 70 |
+
[SimCT v1 equations 7-8](https://arxiv.org/html/2605.07711v1#S3.SS3): first compute
|
| 71 |
+
`log_softmax` over the full native vocabulary at every position; use the first
|
| 72 |
+
position's log-probabilities for shared candidates and mean selected
|
| 73 |
+
log-probabilities for each realized span; then normalize over the virtual candidate
|
| 74 |
+
set and calculate RKL. Both sides use the same score convention. Gradients flow
|
| 75 |
+
through the full-vocabulary student normalization; teacher scores are detached.
|
| 76 |
+
|
| 77 |
+
`simct` remains untouched. `simct_paper` is a paper-score reconstruction on the
|
| 78 |
+
released author's candidate/alignment interface, **not a claim that all paper
|
| 79 |
+
semantics are implemented**. It retains greedy decoded-token alignment, the
|
| 80 |
+
released realized-span candidate set, EOS/shared-vocabulary rules, optional G(h)
|
| 81 |
+
mask (default disabled), raw teacher-head convention, common normalizer and the
|
| 82 |
+
CTOPD initialization/data. It does not add arbitrary alternative-continuation
|
| 83 |
+
forward scoring at every prefix, fix all alignment limitations, restore SFT
|
| 84 |
+
warmup, or establish equivalence to Rethinking's unavailable `simct` implementation.
|
| 85 |
+
Those are separate scientific changes, not hidden inside a score correction.
|
| 86 |
+
|
| 87 |
+
Tests independently construct Eq.7-8 candidate scores and RKL, compare complete
|
| 88 |
+
adapter loss/all student gradients, and verify invariance to independent additive
|
| 89 |
+
logit shifts at each position. Matching the released raw-logit loss is expected
|
| 90 |
+
only in special cases; numerical parity is asserted against the equations here.
|
| 91 |
+
|
| 92 |
+
## Commands and verification
|
| 93 |
+
|
| 94 |
+
From the company checkout, print each full plan without launching a GPU job:
|
| 95 |
+
|
| 96 |
+
```bash
|
| 97 |
+
git pull --ff-only
|
| 98 |
+
python3 tools/launch_single_b200.py --algorithm mp-trust-r --runtime /opt/ctopd-b200-u22 --dry-run
|
| 99 |
+
python3 tools/launch_single_b200.py --algorithm mp-trust-b --runtime /opt/ctopd-b200-u22 --dry-run
|
| 100 |
+
python3 tools/launch_single_b200.py --algorithm mp-trust-p --runtime /opt/ctopd-b200-u22 --dry-run
|
| 101 |
+
python3 tools/launch_single_b200.py --algorithm simct_paper --runtime /opt/ctopd-b200-u22 --dry-run
|
| 102 |
+
```
|
| 103 |
+
|
| 104 |
+
Only remove `--dry-run` with the actual allocated GPU, ready models/data and a new
|
| 105 |
+
run ID. The real batch512/generation8192/100-step workload remains unchanged.
|
| 106 |
+
Use an existing CPU Torch environment for tests; no dependency installation:
|
| 107 |
+
|
| 108 |
+
```bash
|
| 109 |
+
TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
The TRUST oracle separately loads original donor kernels and the TRUST-R method
|
| 113 |
+
body; frozen atom/Gram/credit helpers are source-verified shared dependencies.
|
| 114 |
+
Tests cover full loss/all student gradients, complete-batch cross-response terms,
|
| 115 |
+
projection/no-anchor behavior, dropout RNG replay, a real optimizer update through
|
| 116 |
+
the actor integration helper, evaluation state isolation and unchanged older
|
| 117 |
+
modes. CPU fixtures do not qualify B200/FSDP, full-size memory, training resume,
|
| 118 |
+
accuracy or paper-score reproduction. No GPU job is launched by this port.
|
| 119 |
+
|
| 120 |
+
The [CPU receipt](TRUST_PAPER_VALIDATION.json) records 76 passing tests and six
|
| 121 |
+
TRUST donor comparisons: maximum absolute loss error `0.0`, maximum absolute
|
| 122 |
+
student-gradient error `5.960464477539063e-08`. Paper-score checks compare against
|
| 123 |
+
independent formulas, not against the author's different raw-logit objective.
|
docs/TRUST_PAPER_SOURCE.json
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"donor_repository": "https://github.com/sontungkieu/SimCT",
|
| 3 |
+
"donor_commit": "529ee4b691732e1040d6a45f05fd12f58def96ec",
|
| 4 |
+
"paper": "https://arxiv.org/html/2605.07711v1#S3.SS3",
|
| 5 |
+
"paper_mode_scope": "Eq7-8 log-probability score correction on the released alignment/candidate interface, not full paper recipe",
|
| 6 |
+
"files": [
|
| 7 |
+
{
|
| 8 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_trust.py",
|
| 9 |
+
"sha256": "df17bda4a46fdbf5ddac20b8741915c388d8d88110f7be47eca871ee57eb32a4"
|
| 10 |
+
},
|
| 11 |
+
{
|
| 12 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_trust_batch.py",
|
| 13 |
+
"sha256": "7d526663b2bf2ddf8d9def36565499a515ff43ddb5b1e66811c3d7672a9ed6f2"
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_trust_p.py",
|
| 17 |
+
"sha256": "428eade92186b2ca4198ce7b299f875fe9836ebf9d5877d71689677366ea0be4"
|
| 18 |
+
},
|
| 19 |
+
{
|
| 20 |
+
"path": "tests/fixtures/trust/_mp_opd_trust.py.txt",
|
| 21 |
+
"sha256": "df17bda4a46fdbf5ddac20b8741915c388d8d88110f7be47eca871ee57eb32a4"
|
| 22 |
+
},
|
| 23 |
+
{
|
| 24 |
+
"path": "tests/fixtures/trust/_mp_opd_trust_batch.py.txt",
|
| 25 |
+
"sha256": "7d526663b2bf2ddf8d9def36565499a515ff43ddb5b1e66811c3d7672a9ed6f2"
|
| 26 |
+
},
|
| 27 |
+
{
|
| 28 |
+
"path": "tests/fixtures/trust/_mp_opd_trust_p.py.txt",
|
| 29 |
+
"sha256": "428eade92186b2ca4198ce7b299f875fe9836ebf9d5877d71689677366ea0be4"
|
| 30 |
+
},
|
| 31 |
+
{
|
| 32 |
+
"path": "tests/fixtures/trust/_trust_r_loss.py.txt",
|
| 33 |
+
"sha256": "343f2609f69d88626738244413e4129797657d78b03dd956f435ea6a3e88230e"
|
| 34 |
+
}
|
| 35 |
+
],
|
| 36 |
+
"unchanged_existing_files": [
|
| 37 |
+
{
|
| 38 |
+
"path": "kdflow/algorithms/mp_opd.py",
|
| 39 |
+
"sha256": "b2387d7f36502dea16c28f4391af4bc97a7e77f367d35dea2867c744e91277a6"
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"path": "kdflow/algorithms/mp_opd_align_new.py",
|
| 43 |
+
"sha256": "8479b54a2560d0c3021b796c8766c8309f89b90390981d9d91ec9044a3d13eda"
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"path": "kdflow/algorithms/simct.py",
|
| 47 |
+
"sha256": "d946926dd21fce6041e2fc3b2fd8d317d9dd8e845c5931783e1e6a2bcd70ca89"
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"path": "kdflow/algorithms/simple_ctkd.py",
|
| 51 |
+
"sha256": "fdbd99c55aeac4d061527158bdfc82a93333a42bf347eceaa134d34f5ef05952"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"path": "kdflow/algorithms/byte_align_ctkd.py",
|
| 55 |
+
"sha256": "5c6647c6fb8064ffd5f7b1ed971168cd1f3392caee62e2d75916d41931828e18"
|
| 56 |
+
},
|
| 57 |
+
{
|
| 58 |
+
"path": "kdflow/algorithms/topk_byte_align_ctkd.py",
|
| 59 |
+
"sha256": "298925da4ce973b0f2917e5ba5ad2adbc4862d7c1a73515c5c86514ac0d22ddb"
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_head_pair.py",
|
| 63 |
+
"sha256": "be9163ccd7dbdd4e5a5228be7e06ce68f062a290709383c15b934658e0a83d58"
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_align.py",
|
| 67 |
+
"sha256": "c6c0286c74215a8d30888c15b2bb5820ab77f54a45afd832db99ca6eb81cf1e0"
|
| 68 |
+
}
|
| 69 |
+
]
|
| 70 |
+
}
|
docs/TRUST_PAPER_VALIDATION.json
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"date": "2026-10-19",
|
| 3 |
+
"donor_commit": "529ee4b691732e1040d6a45f05fd12f58def96ec",
|
| 4 |
+
"platform": "WSL Linux, CPU only",
|
| 5 |
+
"python": "3.12.3",
|
| 6 |
+
"torch": "2.14.0+cpu",
|
| 7 |
+
"command": "TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v",
|
| 8 |
+
"full_suite": {"passed": 76, "skipped": 0, "elapsed_seconds": 77.342},
|
| 9 |
+
"trust_parity": {
|
| 10 |
+
"cases": 6,
|
| 11 |
+
"max_loss_abs_error": 0.0,
|
| 12 |
+
"max_gradient_abs_error": 5.960464477539063e-08,
|
| 13 |
+
"dtype": "float32",
|
| 14 |
+
"atol": 1e-6,
|
| 15 |
+
"rtol": 1e-5,
|
| 16 |
+
"coverage": "TRUST-R/B/P complete loss and all student gradients, with/without final softcap",
|
| 17 |
+
"oracle": "Original donor TRUST kernels in separate fixture namespace and original TRUST-R method body; frozen source-verified Gram/credit helpers shared"
|
| 18 |
+
},
|
| 19 |
+
"paper_score": {
|
| 20 |
+
"result": "pass",
|
| 21 |
+
"scope": "Eq7-8 score reconstruction on released alignment/candidate interface",
|
| 22 |
+
"checks": ["Independent candidate scores and RKL", "Complete adapter loss and all student gradients", "Per-position additive logit-shift loss/gradient invariance"],
|
| 23 |
+
"loss_gradient_atol": 1e-6,
|
| 24 |
+
"loss_gradient_rtol": 1e-5,
|
| 25 |
+
"shift_invariance_atol": 2e-6
|
| 26 |
+
},
|
| 27 |
+
"additional_checks": [
|
| 28 |
+
"Explicit-autograd full-batch cross-response geometry and TRUST-P opposing-component projection",
|
| 29 |
+
"Actual actor fit/evaluate dispatch with real CPU SGD: one optimizer update, train-mode dropout replay, Python/NumPy/Torch RNG advance once",
|
| 30 |
+
"Microbatch regrouping preserves full-window loss and gradients",
|
| 31 |
+
"Evaluation restores model mode and leaves weights/calibration state untouched",
|
| 32 |
+
"Common-normalizer/window guards and calibration cleanup on errors",
|
| 33 |
+
"No teacher parameter or hidden-state gradient",
|
| 34 |
+
"All existing Strict/SimCT/MP/ALIGN numerical regressions and frozen source hashes",
|
| 35 |
+
"Launcher preserves common model/data/batch/steps/evaluation settings",
|
| 36 |
+
"git diff --check"
|
| 37 |
+
],
|
| 38 |
+
"not_qualified": [
|
| 39 |
+
"Target Torch 2.11 CUDA, B200/FSDP runtime and full-size memory/throughput",
|
| 40 |
+
"Full Transformer or Adam descent guarantees",
|
| 41 |
+
"Full training resume, accuracy or either paper's full recipe",
|
| 42 |
+
"Complete SimCT paper semantics beyond Eq7-8 score correction",
|
| 43 |
+
"Parity with unavailable Rethinking simct baseline revision"
|
| 44 |
+
]
|
| 45 |
+
}
|
kdflow/algorithms/__init__.py
CHANGED
|
@@ -14,6 +14,8 @@ from kdflow.algorithms.simple_ctkd import SimpleCrossTokenizerKD
|
|
| 14 |
from kdflow.algorithms.byte_align_ctkd import ByteAlignCrossTokenizerKD
|
| 15 |
from kdflow.algorithms.topk_byte_align_ctkd import TopKByteAlignCrossTokenizerKD
|
| 16 |
from kdflow.algorithms.simct import SimCT
|
|
|
|
|
|
|
| 17 |
from kdflow.algorithms.mp_opd_align_new import AlignNewMPOnPolicyKD
|
| 18 |
from kdflow.algorithms.mp_opd import (
|
| 19 |
AtomicMPOnPolicyKD, FixedMPOnPolicyKD, RandomMPOnPolicyKD,
|
|
|
|
| 14 |
from kdflow.algorithms.byte_align_ctkd import ByteAlignCrossTokenizerKD
|
| 15 |
from kdflow.algorithms.topk_byte_align_ctkd import TopKByteAlignCrossTokenizerKD
|
| 16 |
from kdflow.algorithms.simct import SimCT
|
| 17 |
+
from kdflow.algorithms.simct_paper import SimCTPaper
|
| 18 |
+
from kdflow.algorithms.mp_opd_trust import TrustRMPOnPolicyKD, TrustBMPOnPolicyKD, TrustPMPOnPolicyKD
|
| 19 |
from kdflow.algorithms.mp_opd_align_new import AlignNewMPOnPolicyKD
|
| 20 |
from kdflow.algorithms.mp_opd import (
|
| 21 |
AtomicMPOnPolicyKD, FixedMPOnPolicyKD, RandomMPOnPolicyKD,
|
kdflow/algorithms/_mp_opd_port/_mp_opd_trust.py
ADDED
|
@@ -0,0 +1,348 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TRUST v1.0 - trusted reference update scaling for cross-tokenizer OPD.
|
| 2 |
+
|
| 3 |
+
Mismatch supervision may carry signal, but its direction and scale should be
|
| 4 |
+
calibrated against the more reliable strict/matched supervision. TRUST does not
|
| 5 |
+
touch relative credits inside mismatch spans: per calibration unit it scales the
|
| 6 |
+
whole mismatch update by ``lambda = max(0, <gS, gM> / ||gM||^2)`` and keeps the
|
| 7 |
+
strict update whole.
|
| 8 |
+
|
| 9 |
+
Two modes share this file: TRUST-R (one lambda per response) and TRUST-B (one
|
| 10 |
+
lambda per optimizer batch, with microbatch calibration an explicit legacy
|
| 11 |
+
ablation). The only intended difference between them is the
|
| 12 |
+
calibration aggregation level; the strict/mismatch partition, the proxy, and
|
| 13 |
+
the guards are identical.
|
| 14 |
+
|
| 15 |
+
The strict/mismatch partition comes from the atomizer, not from a threshold:
|
| 16 |
+
an atom whose student and teacher spans are both exactly one token over the
|
| 17 |
+
same bytes (``boundary_type == "one_to_one"``) is matched supervision, and an
|
| 18 |
+
atom that straddles a tokenization boundary (``multi_token``) is mismatch. Any
|
| 19 |
+
atom with another boundary type fails closed, because silently filing it under
|
| 20 |
+
either side would change what the calibration means.
|
| 21 |
+
|
| 22 |
+
The proxy is the GRASS LM-head geometry, reused unchanged: with per-atom
|
| 23 |
+
coefficients ``q_i = r_i`` (the OPD atom loss is a plain token-sum, so the
|
| 24 |
+
outer coefficient is 1 and the response weight is a plain sum) and token
|
| 25 |
+
reduction ``a_it = 1`` (``current_nll`` is a token sum over the atom), the
|
| 26 |
+
effective coefficients are exactly the rates the loss consumes. The Gram code
|
| 27 |
+
is shared, so a TRUST number and a GRASS number read the same geometry.
|
| 28 |
+
|
| 29 |
+
Deliberate v1 deviations from the spec text, all fail-safe rather than silent:
|
| 30 |
+
|
| 31 |
+
* Median/p10/p25/p75/p90 keys are not emitted. Metrics reduce per micro-batch
|
| 32 |
+
by sum-then-mean, which cannot express a percentile; emitting a mean under a
|
| 33 |
+
``_median`` key would lie about the statistic. Means and fractions over the
|
| 34 |
+
micro-batch responses are exact as stated (production uses
|
| 35 |
+
micro_train_batch_size=4).
|
| 36 |
+
* ``mp_opd_trust_scope`` is a float code (0.0 = response, 1.0 = batch unit) because
|
| 37 |
+
the metric table only carries finite tensors.
|
| 38 |
+
``mp_opd_trust_full_batch`` distinguishes the complete optimizer batch from
|
| 39 |
+
the legacy microbatch unit.
|
| 40 |
+
* Samples or units without both sides log 0.0 for undefined quantities (never
|
| 41 |
+
NaN: the trainer fails fast on non-finite metrics), with the eligibility
|
| 42 |
+
fractions (``has_both``/``zero_*``) as the interpretive key.
|
| 43 |
+
* TRUST requires ``exact_head`` geometry. The ``diag`` ablation zeroes every
|
| 44 |
+
cross-atom term, which would force every dot product to zero and every lambda
|
| 45 |
+
to zero: a mode that can never act. Failing closed beats a silent no-op.
|
| 46 |
+
"""
|
| 47 |
+
|
| 48 |
+
from __future__ import annotations
|
| 49 |
+
|
| 50 |
+
from dataclasses import dataclass
|
| 51 |
+
from typing import Sequence
|
| 52 |
+
|
| 53 |
+
import torch
|
| 54 |
+
|
| 55 |
+
# Section 9 fixes this: a mismatch norm at or below it carries no usable
|
| 56 |
+
# direction, so the calibration that would divide by it is skipped.
|
| 57 |
+
TRUST_EPS_G = 1e-12
|
| 58 |
+
# Only used to keep norm denominators away from zero; it never decides whether
|
| 59 |
+
# a unit calibrates or what lambda it gets.
|
| 60 |
+
TRUST_EPS_NORM = 1e-12
|
| 61 |
+
# Transient multiplier for one exact-Gram build: the [tokens, vocab] window,
|
| 62 |
+
# its Jacobian, the probabilities, and the vocabulary product temporaries.
|
| 63 |
+
# A step whose estimate exceeds half the free device memory degrades to the
|
| 64 |
+
# atomic fallback (loudly counted) instead of attempting an allocation the
|
| 65 |
+
# device cannot hold. Killed a 171-step trust_b run at 11.73 GiB attempted on
|
| 66 |
+
# 6.34 GiB free; the fallback that would have saved it did not exist yet.
|
| 67 |
+
TRUST_GRAM_TRANSIENT_FACTOR = 4.0
|
| 68 |
+
TRUST_GRAM_FREE_FRACTION = 0.5
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def gram_affordable(n_tokens: int, vocab: int, device: torch.device) -> bool:
|
| 72 |
+
"""Whether one exact Gram build fits in the currently free device memory.
|
| 73 |
+
|
| 74 |
+
Estimates ``n_tokens * vocab`` fp32 entries times the transient factor and
|
| 75 |
+
requires it under half the free bytes. On a device without memory info
|
| 76 |
+
(CPU tests) everything is affordable; the production guard this provides
|
| 77 |
+
is against the allocation, not the math.
|
| 78 |
+
"""
|
| 79 |
+
if n_tokens <= 0 or vocab <= 0:
|
| 80 |
+
raise ValueError("gram affordability needs positive token and vocab counts")
|
| 81 |
+
if device.type != "cuda" or not torch.cuda.is_available():
|
| 82 |
+
return True
|
| 83 |
+
free_bytes, _ = torch.cuda.mem_get_info(device)
|
| 84 |
+
need_bytes = float(n_tokens) * float(vocab) * 4.0 * TRUST_GRAM_TRANSIENT_FACTOR
|
| 85 |
+
return need_bytes <= float(free_bytes) * TRUST_GRAM_FREE_FRACTION
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
@dataclass(frozen=True)
|
| 89 |
+
class TrustUnitResult:
|
| 90 |
+
"""One calibration unit's lambda plus everything its diagnostics derive from."""
|
| 91 |
+
|
| 92 |
+
lam: float
|
| 93 |
+
dot: float
|
| 94 |
+
gs_norm2: float
|
| 95 |
+
gm_norm2: float
|
| 96 |
+
cosine: float
|
| 97 |
+
norm_ratio: float
|
| 98 |
+
kappa: float
|
| 99 |
+
has_strict: bool
|
| 100 |
+
has_mismatch: bool
|
| 101 |
+
calibrated: bool
|
| 102 |
+
update_cosine: float
|
| 103 |
+
update_norm_ratio: float
|
| 104 |
+
n_strict: int
|
| 105 |
+
n_mismatch: int
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def _require_finite(name: str, *tensors: torch.Tensor) -> None:
|
| 109 |
+
for tensor in tensors:
|
| 110 |
+
if not bool(torch.isfinite(tensor).all()):
|
| 111 |
+
raise ValueError(f"TRUST received a non-finite {name}")
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def trust_calibrate(
|
| 115 |
+
qs: torch.Tensor,
|
| 116 |
+
qm: torch.Tensor,
|
| 117 |
+
gram_ss: torch.Tensor,
|
| 118 |
+
gram_mm: torch.Tensor,
|
| 119 |
+
gram_sm: torch.Tensor,
|
| 120 |
+
*,
|
| 121 |
+
eps_g: float = TRUST_EPS_G,
|
| 122 |
+
eps_norm: float = TRUST_EPS_NORM,
|
| 123 |
+
) -> TrustUnitResult:
|
| 124 |
+
"""Calibrate one strict/mismatch pair from its Gram blocks.
|
| 125 |
+
|
| 126 |
+
``qs``/``qm`` are the effective strict/mismatch coefficients and
|
| 127 |
+
``gram_ss``/``gram_mm``/``gram_sm`` the matching blocks of the exact
|
| 128 |
+
LM-head Gram. The geometry is FP32; quadratic reductions promote to FP64.
|
| 129 |
+
The returned lambda is a Python float, so it can never carry a graph.
|
| 130 |
+
"""
|
| 131 |
+
_require_finite("coefficients", qs, qm)
|
| 132 |
+
_require_finite("Gram blocks", gram_ss, gram_mm, gram_sm)
|
| 133 |
+
if not (float(eps_g) >= 0.0):
|
| 134 |
+
raise ValueError(f"TRUST eps_g must be non-negative, got {eps_g!r}")
|
| 135 |
+
qs = qs.detach().to(torch.float64).reshape(-1)
|
| 136 |
+
qm = qm.detach().to(torch.float64).reshape(-1)
|
| 137 |
+
gram_ss, gram_mm, gram_sm = (g.detach().double() for g in (gram_ss, gram_mm, gram_sm))
|
| 138 |
+
n_s, n_m = int(qs.numel()), int(qm.numel())
|
| 139 |
+
if (n_s, n_m) != (int(gram_ss.shape[0]), int(gram_mm.shape[0])):
|
| 140 |
+
raise ValueError(
|
| 141 |
+
f"TRUST Gram blocks {tuple(gram_ss.shape)}/{tuple(gram_mm.shape)} "
|
| 142 |
+
f"do not match coefficient counts ({n_s}, {n_m})"
|
| 143 |
+
)
|
| 144 |
+
if tuple(gram_sm.shape) != (n_s, n_m):
|
| 145 |
+
raise ValueError(
|
| 146 |
+
f"TRUST cross Gram {tuple(gram_sm.shape)} does not match ({n_s}, {n_m})"
|
| 147 |
+
)
|
| 148 |
+
|
| 149 |
+
# Empty sides have no geometry to read: the scalars below are defined as
|
| 150 |
+
# zero rather than derived from a matmul over an empty axis.
|
| 151 |
+
gs2 = float(qs @ gram_ss @ qs) if n_s else 0.0
|
| 152 |
+
gm2 = float(qm @ gram_mm @ qm) if n_m else 0.0
|
| 153 |
+
dot = float(qs @ gram_sm @ qm) if (n_s and n_m) else 0.0
|
| 154 |
+
scales = [float(q.abs() @ g.abs() @ q.abs()) if q.numel() else 0.0
|
| 155 |
+
for q, g in ((qs, gram_ss), (qm, gram_mm))]
|
| 156 |
+
for name, norm, scale in zip(("strict", "mismatch"), (gs2, gm2), scales):
|
| 157 |
+
if norm < -1e-6 * max(scale, 1e-300):
|
| 158 |
+
raise ValueError(f"TRUST {name} Gram gives a materially negative squared norm")
|
| 159 |
+
gs2, gm2 = max(gs2, 0.), max(gm2, 0.)
|
| 160 |
+
bound = (gs2 * gm2)**.5
|
| 161 |
+
if abs(dot) > bound + 1e-6 * max((scales[0]*scales[1])**.5, 1e-300):
|
| 162 |
+
raise ValueError("TRUST Gram violates Cauchy-Schwarz")
|
| 163 |
+
dot = min(max(dot, -bound), bound)
|
| 164 |
+
return trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m,
|
| 165 |
+
eps_g=eps_g, eps_norm=eps_norm)
|
| 166 |
+
|
| 167 |
+
|
| 168 |
+
def trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *,
|
| 169 |
+
eps_g=TRUST_EPS_G, eps_norm=TRUST_EPS_NORM):
|
| 170 |
+
"""The same calibration for explicit summed head updates over a full batch."""
|
| 171 |
+
if not (0 <= float(eps_g) < float("inf")):
|
| 172 |
+
raise ValueError("TRUST eps_g must be finite and nonnegative")
|
| 173 |
+
for name, value in (("gs_norm2", gs2), ("gm_norm2", gm2), ("dot", dot)):
|
| 174 |
+
if not abs(value) < float("inf"):
|
| 175 |
+
raise ValueError(f"TRUST {name} is non-finite")
|
| 176 |
+
if gs2 < 0 or gm2 < 0:
|
| 177 |
+
raise ValueError("TRUST squared norms must be nonnegative")
|
| 178 |
+
bound = (gs2 * gm2)**.5
|
| 179 |
+
if abs(dot) > bound * (1. + 1e-6) + 1e-300:
|
| 180 |
+
raise ValueError("TRUST summed updates violate Cauchy-Schwarz")
|
| 181 |
+
dot = min(max(dot, -bound), bound)
|
| 182 |
+
|
| 183 |
+
has_strict = n_s > 0
|
| 184 |
+
has_mismatch = n_m > 0
|
| 185 |
+
calibrated = bool(has_strict and has_mismatch and gm2 > float(eps_g))
|
| 186 |
+
lam = max(0.0, dot / gm2) if calibrated else 0.0
|
| 187 |
+
|
| 188 |
+
gs = gs2**0.5
|
| 189 |
+
gm = gm2**0.5
|
| 190 |
+
cosine = dot / (gs * gm + float(eps_norm))
|
| 191 |
+
norm_ratio = gm / (gs + float(eps_norm))
|
| 192 |
+
kappa = lam * gm / (gs + float(eps_norm))
|
| 193 |
+
|
| 194 |
+
# g_pre = gS + gM and g_post = gS + lam gM, expanded in Gram scalars so no
|
| 195 |
+
# vector is ever formed. When both updates vanish they are identical.
|
| 196 |
+
pre2 = gs2 + gm2 + 2.0 * dot
|
| 197 |
+
post2 = gs2 + lam * lam * gm2 + 2.0 * lam * dot
|
| 198 |
+
inner = gs2 + lam * gm2 + (1.0 + lam) * dot
|
| 199 |
+
pre2 = max(pre2, 0.0)
|
| 200 |
+
post2 = max(post2, 0.0)
|
| 201 |
+
if pre2 <= 0.0 and post2 <= 0.0:
|
| 202 |
+
update_cosine = 1.0
|
| 203 |
+
elif pre2 <= 0.0 or post2 <= 0.0:
|
| 204 |
+
update_cosine = 0.0
|
| 205 |
+
else:
|
| 206 |
+
update_cosine = inner / (pre2**0.5 * post2**0.5 + float(eps_norm))
|
| 207 |
+
update_norm_ratio = post2**0.5 / (pre2**0.5 + float(eps_norm))
|
| 208 |
+
|
| 209 |
+
return TrustUnitResult(
|
| 210 |
+
lam=lam,
|
| 211 |
+
dot=dot,
|
| 212 |
+
gs_norm2=gs2,
|
| 213 |
+
gm_norm2=gm2,
|
| 214 |
+
cosine=cosine,
|
| 215 |
+
norm_ratio=norm_ratio,
|
| 216 |
+
kappa=kappa,
|
| 217 |
+
has_strict=has_strict,
|
| 218 |
+
has_mismatch=has_mismatch,
|
| 219 |
+
calibrated=calibrated,
|
| 220 |
+
update_cosine=update_cosine,
|
| 221 |
+
update_norm_ratio=update_norm_ratio,
|
| 222 |
+
n_strict=n_s,
|
| 223 |
+
n_mismatch=n_m,
|
| 224 |
+
)
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def strict_mask(boundary_types: Sequence[str]) -> torch.Tensor:
|
| 228 |
+
"""Strict-side mask from atomizer boundary types, failing closed.
|
| 229 |
+
|
| 230 |
+
``one_to_one`` is matched supervision, ``multi_token`` is mismatch. A
|
| 231 |
+
third kind would silently change the calibration under either choice, so
|
| 232 |
+
it raises instead of being filed anywhere.
|
| 233 |
+
"""
|
| 234 |
+
mask = []
|
| 235 |
+
for kind in boundary_types:
|
| 236 |
+
if kind == "one_to_one":
|
| 237 |
+
mask.append(True)
|
| 238 |
+
elif kind == "multi_token":
|
| 239 |
+
mask.append(False)
|
| 240 |
+
else:
|
| 241 |
+
raise ValueError(f"TRUST has no side for boundary type {kind!r}")
|
| 242 |
+
return torch.tensor(mask, dtype=torch.bool)
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def trust_response_loss(
|
| 246 |
+
current_nll: torch.Tensor,
|
| 247 |
+
rate: torch.Tensor,
|
| 248 |
+
is_strict: torch.Tensor,
|
| 249 |
+
is_mismatch: torch.Tensor,
|
| 250 |
+
result: TrustUnitResult,
|
| 251 |
+
) -> torch.Tensor:
|
| 252 |
+
"""``L_S + stopgrad(lambda) L_M`` with exactly the OPD loss coefficients.
|
| 253 |
+
|
| 254 |
+
Each side is a ``(rates.detach() * nll).sum()``, the same construction as
|
| 255 |
+
``soft_partition_loss``: the strict side keeps its credits whole and the
|
| 256 |
+
mismatch side is scaled by the detached lambda. The lambda is a Python
|
| 257 |
+
float by construction, so no graph can pass through the calibration rule.
|
| 258 |
+
"""
|
| 259 |
+
if not (bool(is_strict.any()) or bool(is_mismatch.any())):
|
| 260 |
+
raise ValueError("TRUST needs at least one strict or mismatch atom")
|
| 261 |
+
if bool((is_strict & is_mismatch).any()):
|
| 262 |
+
raise ValueError("TRUST strict and mismatch sets must be disjoint")
|
| 263 |
+
zero = current_nll.new_zeros(())
|
| 264 |
+
strict = (torch.where(is_strict, rate.detach(), zero) * current_nll).sum()
|
| 265 |
+
mismatch = (torch.where(is_mismatch, rate.detach(), zero) * current_nll).sum()
|
| 266 |
+
return strict + float(result.lam) * mismatch
|
| 267 |
+
|
| 268 |
+
|
| 269 |
+
def _counts(is_strict: torch.Tensor, token_counts: torch.Tensor) -> dict[str, float]:
|
| 270 |
+
"""Raw per-unit coverage counts; the caller decides the aggregation."""
|
| 271 |
+
is_mismatch = ~is_strict
|
| 272 |
+
n_s = int(is_strict.sum())
|
| 273 |
+
n_m = int(is_mismatch.sum())
|
| 274 |
+
total = n_s + n_m
|
| 275 |
+
return {
|
| 276 |
+
"strict_span_count": float(n_s),
|
| 277 |
+
"mismatch_span_count": float(n_m),
|
| 278 |
+
# One atom is one supervision span here: there is no higher grouping
|
| 279 |
+
# for TRUST to aggregate, so the span and atom keys coincide by
|
| 280 |
+
# definition rather than by coincidence.
|
| 281 |
+
"strict_atom_count": float(n_s),
|
| 282 |
+
"mismatch_atom_count": float(n_m),
|
| 283 |
+
"strict_token_count": float(token_counts[is_strict].sum()),
|
| 284 |
+
"mismatch_token_count": float(token_counts[is_mismatch].sum()),
|
| 285 |
+
"strict_span_fraction": float(n_s / total) if total else 0.0,
|
| 286 |
+
"mismatch_span_fraction": float(n_m / total) if total else 0.0,
|
| 287 |
+
"has_strict": float(n_s > 0),
|
| 288 |
+
"has_mismatch": float(n_m > 0),
|
| 289 |
+
"has_both": float(n_s > 0 and n_m > 0),
|
| 290 |
+
"zero_strict": float(n_s == 0),
|
| 291 |
+
"zero_mismatch": float(n_m == 0),
|
| 292 |
+
}
|
| 293 |
+
|
| 294 |
+
|
| 295 |
+
def trust_response_metrics(
|
| 296 |
+
result: TrustUnitResult,
|
| 297 |
+
is_strict: torch.Tensor,
|
| 298 |
+
token_counts: torch.Tensor,
|
| 299 |
+
*,
|
| 300 |
+
scope: str,
|
| 301 |
+
) -> dict[str, float]:
|
| 302 |
+
"""Per-unit metric values; the training loop averages them over responses."""
|
| 303 |
+
if scope not in ("response", "batch"):
|
| 304 |
+
raise ValueError(f"TRUST scope must be response|batch, got {scope!r}")
|
| 305 |
+
out = _counts(is_strict, token_counts)
|
| 306 |
+
out.update(
|
| 307 |
+
{
|
| 308 |
+
"cosine": result.cosine,
|
| 309 |
+
"cosine_negative": float(result.cosine < 0.0),
|
| 310 |
+
"mismatch_to_strict_norm_ratio": result.norm_ratio,
|
| 311 |
+
"lambda": result.lam,
|
| 312 |
+
"lambda_zero": float(result.lam <= 0.0),
|
| 313 |
+
"lambda_gt1": float(result.lam > 1.0),
|
| 314 |
+
"calibrated": float(result.calibrated),
|
| 315 |
+
"calibrated_mismatch_norm_ratio": result.kappa,
|
| 316 |
+
"update_cosine_pre_post": result.update_cosine,
|
| 317 |
+
"update_norm_ratio_post_pre": result.update_norm_ratio,
|
| 318 |
+
"scope": 0.0 if scope == "response" else 1.0,
|
| 319 |
+
}
|
| 320 |
+
)
|
| 321 |
+
return out
|
| 322 |
+
|
| 323 |
+
|
| 324 |
+
def trust_batch_metrics(
|
| 325 |
+
results: TrustUnitResult,
|
| 326 |
+
counts: dict[str, float],
|
| 327 |
+
) -> dict[str, float]:
|
| 328 |
+
"""One calibration unit's values, logged without response averaging."""
|
| 329 |
+
return {
|
| 330 |
+
"batch_strict_span_count": counts["strict_span_count"],
|
| 331 |
+
"batch_mismatch_span_count": counts["mismatch_span_count"],
|
| 332 |
+
"batch_strict_atom_count": counts["strict_atom_count"],
|
| 333 |
+
"batch_mismatch_atom_count": counts["mismatch_atom_count"],
|
| 334 |
+
"batch_strict_token_count": counts["strict_token_count"],
|
| 335 |
+
"batch_mismatch_token_count": counts["mismatch_token_count"],
|
| 336 |
+
"batch_strict_span_fraction": counts["strict_span_fraction"],
|
| 337 |
+
"batch_mismatch_span_fraction": counts["mismatch_span_fraction"],
|
| 338 |
+
"cosine": results.cosine,
|
| 339 |
+
"mismatch_to_strict_norm_ratio": results.norm_ratio,
|
| 340 |
+
"lambda": results.lam,
|
| 341 |
+
"batch_has_strict": float(results.has_strict),
|
| 342 |
+
"batch_has_mismatch": float(results.has_mismatch),
|
| 343 |
+
"batch_calibrated": float(results.calibrated),
|
| 344 |
+
"calibrated_mismatch_norm_ratio": results.kappa,
|
| 345 |
+
"update_cosine_pre_post": results.update_cosine,
|
| 346 |
+
"update_norm_ratio_post_pre": results.update_norm_ratio,
|
| 347 |
+
"scope": 1.0,
|
| 348 |
+
}
|
kdflow/algorithms/_mp_opd_port/_mp_opd_trust_batch.py
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Exact full-batch TRUST head updates, streamed without retaining forward graphs."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import contextlib
|
| 5 |
+
import torch
|
| 6 |
+
|
| 7 |
+
from ._mp_opd_grass_span import _exact_fp32, _log_partition_from_selected
|
| 8 |
+
from ._mp_opd_trust import trust_calibrate_scalars
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class TrustHeadAccumulator:
|
| 12 |
+
"""Sum strict/mismatch LM-head gradients, including cross-response terms.
|
| 13 |
+
|
| 14 |
+
Storage is two [vocab, hidden] fp32 tensors, independent of batch token count.
|
| 15 |
+
Vocabulary tiles bound transients; no [batch tokens, vocab] concatenation and
|
| 16 |
+
no atom-by-atom global Gram are constructed. Loss normalization is one common
|
| 17 |
+
factor for the entire accumulation window and cancels from lambda.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, *, softcap=None, head_bias=False, vocab_chunk=4096,
|
| 21 |
+
force_fp32=False):
|
| 22 |
+
self.softcap = softcap
|
| 23 |
+
self.head_bias = head_bias
|
| 24 |
+
self.vocab_chunk = int(vocab_chunk)
|
| 25 |
+
self.force_fp32 = bool(force_fp32)
|
| 26 |
+
if self.vocab_chunk < 1:
|
| 27 |
+
raise ValueError("TRUST vocabulary tile must be positive")
|
| 28 |
+
self.gs = self.gm = self.bs = self.bm = None
|
| 29 |
+
self.responses = []
|
| 30 |
+
|
| 31 |
+
@torch.no_grad()
|
| 32 |
+
def add_response(self, logits, hidden, labels, ranges, rate, is_strict,
|
| 33 |
+
selected_log_prob):
|
| 34 |
+
covered = ranges[-1][1]
|
| 35 |
+
logits, hidden, labels = logits[:covered], hidden[:covered], labels[:covered]
|
| 36 |
+
lengths = torch.tensor([b-a for a, b in ranges], device=logits.device)
|
| 37 |
+
if sum(b-a for a, b in ranges) != covered or ranges[0][0] != 0:
|
| 38 |
+
raise ValueError("TRUST ranges must tile the covered token prefix")
|
| 39 |
+
if any(ranges[i][1] != ranges[i+1][0] for i in range(len(ranges)-1)):
|
| 40 |
+
raise ValueError("TRUST ranges must have no gaps")
|
| 41 |
+
if not all(bool(torch.isfinite(t).all()) for t in (logits, hidden, rate)):
|
| 42 |
+
raise FloatingPointError("non-finite full-batch TRUST input")
|
| 43 |
+
vocab, width = int(logits.shape[1]), int(hidden.shape[1])
|
| 44 |
+
if self.gs is None:
|
| 45 |
+
need = 2 * vocab * width * 4 + (2 * vocab * 4 if self.head_bias else 0)
|
| 46 |
+
if logits.is_cuda:
|
| 47 |
+
free, _ = torch.cuda.mem_get_info(logits.device)
|
| 48 |
+
if need > free * 0.5:
|
| 49 |
+
raise ValueError("full-batch TRUST head accumulators exceed memory guard")
|
| 50 |
+
self.gs = torch.zeros(vocab, width, dtype=torch.float32, device=logits.device)
|
| 51 |
+
self.gm = torch.zeros_like(self.gs)
|
| 52 |
+
if self.head_bias:
|
| 53 |
+
self.bs = torch.zeros(vocab, dtype=torch.float32, device=logits.device)
|
| 54 |
+
self.bm = torch.zeros_like(self.bs)
|
| 55 |
+
if self.gs.shape != (vocab, width):
|
| 56 |
+
raise ValueError("student head shape changed within TRUST batch")
|
| 57 |
+
strict = is_strict.to(device=logits.device)
|
| 58 |
+
q = rate.detach().float().repeat_interleave(lengths)
|
| 59 |
+
token_strict = strict.repeat_interleave(lengths)
|
| 60 |
+
qs = q * token_strict
|
| 61 |
+
qm = q * ~token_strict
|
| 62 |
+
hs = hidden.detach().float() * qs.unsqueeze(1)
|
| 63 |
+
hm = hidden.detach().float() * qm.unsqueeze(1)
|
| 64 |
+
logz = _log_partition_from_selected(logits, labels, selected_log_prob[:covered]).float()
|
| 65 |
+
autocast_guard = (torch.autocast(device_type=logits.device.type, enabled=False)
|
| 66 |
+
if self.force_fp32 else contextlib.nullcontext())
|
| 67 |
+
with _exact_fp32(), autocast_guard:
|
| 68 |
+
for low in range(0, vocab, self.vocab_chunk):
|
| 69 |
+
high = min(low + self.vocab_chunk, vocab)
|
| 70 |
+
z = logits[:, low:high].detach().float()
|
| 71 |
+
delta = torch.exp(z - logz.unsqueeze(1))
|
| 72 |
+
jacobian = None
|
| 73 |
+
if self.softcap is not None:
|
| 74 |
+
jacobian = (1.0 - (z / float(self.softcap)).clamp(-1, 1).square()).clamp_min(0)
|
| 75 |
+
delta *= jacobian
|
| 76 |
+
rows = ((labels >= low) & (labels < high)).nonzero().flatten()
|
| 77 |
+
cols = labels[rows] - low
|
| 78 |
+
delta[rows, cols] -= 1.0 if jacobian is None else jacobian[rows, cols]
|
| 79 |
+
self.gs[low:high].add_(delta.T @ hs)
|
| 80 |
+
self.gm[low:high].add_(delta.T @ hm)
|
| 81 |
+
if self.head_bias:
|
| 82 |
+
self.bs[low:high].add_(qs @ delta)
|
| 83 |
+
self.bm[low:high].add_(qm @ delta)
|
| 84 |
+
self.responses.append({
|
| 85 |
+
"strict_atoms": int(strict.sum()), "mismatch_atoms": int((~strict).sum()),
|
| 86 |
+
"strict_tokens": int(lengths[strict].sum()),
|
| 87 |
+
"mismatch_tokens": int(lengths[~strict].sum()),
|
| 88 |
+
})
|
| 89 |
+
|
| 90 |
+
@torch.no_grad()
|
| 91 |
+
def finalize(self, *, eps_g=1e-12, distributed=False,
|
| 92 |
+
calibrate=trust_calibrate_scalars):
|
| 93 |
+
if distributed:
|
| 94 |
+
import torch.distributed as dist
|
| 95 |
+
# Every rank uses the same pinned student head shape. Empty ranks
|
| 96 |
+
# allocate matching buffers after exchanging shapes, before collectives.
|
| 97 |
+
shapes = [None] * dist.get_world_size()
|
| 98 |
+
dist.all_gather_object(shapes, None if self.gs is None else tuple(self.gs.shape))
|
| 99 |
+
shape = next((s for s in shapes if s is not None), None)
|
| 100 |
+
if any(s is not None and s != shape for s in shapes):
|
| 101 |
+
raise ValueError("TRUST ranks have inconsistent student head shapes")
|
| 102 |
+
if shape is not None and self.gs is None:
|
| 103 |
+
dev = torch.device("cuda", torch.cuda.current_device()) if dist.get_backend() == "nccl" else torch.device("cpu")
|
| 104 |
+
self.gs = torch.zeros(shape, dtype=torch.float32, device=dev)
|
| 105 |
+
self.gm = torch.zeros_like(self.gs)
|
| 106 |
+
if self.head_bias:
|
| 107 |
+
self.bs = torch.zeros(shape[0], dtype=torch.float32, device=dev)
|
| 108 |
+
self.bm = torch.zeros_like(self.bs)
|
| 109 |
+
if shape is not None:
|
| 110 |
+
for tensor in (self.gs, self.gm, self.bs, self.bm):
|
| 111 |
+
if tensor is not None:
|
| 112 |
+
dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
|
| 113 |
+
records = [None] * dist.get_world_size()
|
| 114 |
+
dist.all_gather_object(records, self.responses)
|
| 115 |
+
self.responses = [r for group in records for r in group]
|
| 116 |
+
gs2 = gm2 = dot = 0.0
|
| 117 |
+
if self.gs is not None:
|
| 118 |
+
for low in range(0, self.gs.shape[0], self.vocab_chunk):
|
| 119 |
+
s = self.gs[low:low+self.vocab_chunk].double()
|
| 120 |
+
m = self.gm[low:low+self.vocab_chunk].double()
|
| 121 |
+
gs2 += float(s.square().sum())
|
| 122 |
+
gm2 += float(m.square().sum())
|
| 123 |
+
dot += float((s*m).sum())
|
| 124 |
+
if self.head_bias:
|
| 125 |
+
gs2 += float(self.bs.double().square().sum())
|
| 126 |
+
gm2 += float(self.bm.double().square().sum())
|
| 127 |
+
dot += float((self.bs.double()*self.bm.double()).sum())
|
| 128 |
+
n_s = sum(r["strict_atoms"] for r in self.responses)
|
| 129 |
+
n_m = sum(r["mismatch_atoms"] for r in self.responses)
|
| 130 |
+
result = calibrate(gs2, gm2, dot, n_s, n_m, eps_g=eps_g)
|
| 131 |
+
self.gs = self.gm = self.bs = self.bm = None
|
| 132 |
+
return result
|
kdflow/algorithms/_mp_opd_port/_mp_opd_trust_p.py
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TRUST-P: project mismatch onto the strict non-conflict halfspace.
|
| 2 |
+
|
| 3 |
+
This is a new batch-level algorithm, not a correction to TRUST-R/B. Projection
|
| 4 |
+
is a known operation; efficacy or research novelty is not established here.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import math
|
| 9 |
+
from dataclasses import dataclass
|
| 10 |
+
|
| 11 |
+
import torch
|
| 12 |
+
|
| 13 |
+
from ._mp_opd_head_pair import checked_geometry
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
@dataclass(frozen=True)
|
| 17 |
+
class TrustProjectedResult:
|
| 18 |
+
strict_scale: float
|
| 19 |
+
gs_norm2: float
|
| 20 |
+
gm_norm2: float
|
| 21 |
+
dot: float
|
| 22 |
+
n_strict: int
|
| 23 |
+
n_mismatch: int
|
| 24 |
+
no_anchor: bool
|
| 25 |
+
projected: bool
|
| 26 |
+
mismatch_retained_norm_ratio: float
|
| 27 |
+
strict_descent: float
|
| 28 |
+
update_norm_ratio: float
|
| 29 |
+
update_cosine: float
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def trust_p_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *, eps_g=1e-12):
|
| 33 |
+
"""One decision for the entire optimizer batch, including cross-responses.
|
| 34 |
+
|
| 35 |
+
M_safe = M - min(0,<S,M>)/||S||^2 S; g = S+M_safe. Therefore
|
| 36 |
+
L = stopgrad(a) L_S + L_M with a=1-min(0,dot)/||S||^2.
|
| 37 |
+
With no usable S anchor, keep Atomic and explicitly waive the guarantee.
|
| 38 |
+
"""
|
| 39 |
+
if not (math.isfinite(float(eps_g)) and float(eps_g) >= 0):
|
| 40 |
+
raise ValueError("TRUST-P eps_g must be finite and nonnegative")
|
| 41 |
+
if n_s < 0 or n_m < 0:
|
| 42 |
+
raise ValueError("TRUST-P atom counts must be nonnegative")
|
| 43 |
+
pair = checked_geometry(gs2, gm2, dot)
|
| 44 |
+
s, m, d = pair.first_norm2, pair.second_norm2, pair.dot
|
| 45 |
+
no_anchor = n_s == 0 or s <= float(eps_g)
|
| 46 |
+
correction = -min(0., d)/s if not no_anchor else 0.
|
| 47 |
+
a = 1 + correction
|
| 48 |
+
if not math.isfinite(a) or a > torch.finfo(torch.float32).max:
|
| 49 |
+
raise FloatingPointError("TRUST-P strict scale is not representable in fp32")
|
| 50 |
+
# Loss coefficients are fp32. Round towards the protected halfspace, rather
|
| 51 |
+
# than letting nearest rounding reintroduce a tiny opposing component.
|
| 52 |
+
rounded = torch.tensor(a, dtype=torch.float32)
|
| 53 |
+
if float(rounded) < a:
|
| 54 |
+
rounded = torch.nextafter(rounded, torch.tensor(float("inf")))
|
| 55 |
+
a = float(rounded)
|
| 56 |
+
# Orthogonal component survives unchanged; only the opposing parallel
|
| 57 |
+
# component is removed. Projection cannot increase the M norm.
|
| 58 |
+
orthogonal2 = max(m - d*d/s, 0.) if correction > 0 else 0.
|
| 59 |
+
retained2 = orthogonal2 + (a-1+d/s)**2*s if correction > 0 else m
|
| 60 |
+
# Complete the square to avoid catastrophic cancellation for opposing arms.
|
| 61 |
+
post2 = (orthogonal2 + (a + d/s)**2*s) if correction > 0 else max(s + m + 2*d, 0.)
|
| 62 |
+
pre2 = max(s + m + 2*d, 0.)
|
| 63 |
+
cross = a*s + m + (a+1)*d
|
| 64 |
+
cosine = cross/math.sqrt(pre2*post2) if pre2 > 0 and post2 > 0 else (1. if pre2 == post2 == 0 else 0.)
|
| 65 |
+
scalars = (a, retained2, post2, pre2, cross)
|
| 66 |
+
if not all(math.isfinite(v) for v in scalars):
|
| 67 |
+
raise FloatingPointError("non-finite TRUST-P calibration")
|
| 68 |
+
return TrustProjectedResult(
|
| 69 |
+
a, s, m, d, int(n_s), int(n_m), no_anchor, correction > 0,
|
| 70 |
+
math.sqrt(retained2/m) if m > 0 else 0., a*s+d,
|
| 71 |
+
math.sqrt(post2/pre2) if pre2 > 0 else 0., max(-1., min(cosine, 1.)),
|
| 72 |
+
)
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def trust_p_response_loss(current_nll, rate, is_strict, is_mismatch, result):
|
| 76 |
+
if bool((is_strict & is_mismatch).any()) or not bool((is_strict | is_mismatch).all()):
|
| 77 |
+
raise ValueError("TRUST-P sides must be disjoint and cover all atoms")
|
| 78 |
+
coefficient = torch.where(is_strict, float(result.strict_scale), 1.).to(rate)
|
| 79 |
+
return (rate.detach() * coefficient * current_nll).sum()
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def trust_p_batch_metrics(result, counts):
|
| 83 |
+
r = result
|
| 84 |
+
out = {"batch_"+key: float(value) for key, value in counts.items()}
|
| 85 |
+
out.update({
|
| 86 |
+
"strict_scale": r.strict_scale,
|
| 87 |
+
"mismatch_scale": 1.,
|
| 88 |
+
"strict_head_norm2": r.gs_norm2,
|
| 89 |
+
"mismatch_head_norm2": r.gm_norm2,
|
| 90 |
+
"strict_mismatch_dot": r.dot,
|
| 91 |
+
"strict_mismatch_cosine": r.dot/math.sqrt(r.gs_norm2*r.gm_norm2) if r.gs_norm2 > 0 and r.gm_norm2 > 0 else 0.,
|
| 92 |
+
"projected": float(r.projected),
|
| 93 |
+
"no_anchor": float(r.no_anchor),
|
| 94 |
+
"batch_has_strict": float(r.n_strict > 0),
|
| 95 |
+
"batch_has_mismatch": float(r.n_mismatch > 0),
|
| 96 |
+
"mismatch_retained_norm_ratio": r.mismatch_retained_norm_ratio,
|
| 97 |
+
"strict_descent": r.strict_descent,
|
| 98 |
+
"strict_descent_ratio": r.strict_descent/r.gs_norm2 if r.gs_norm2 > 0 else 0.,
|
| 99 |
+
"update_norm_ratio_post_pre": r.update_norm_ratio,
|
| 100 |
+
"update_cosine_pre_post": r.update_cosine,
|
| 101 |
+
"update_ratio_defined": float(r.gs_norm2+r.gm_norm2+2*r.dot > 0),
|
| 102 |
+
"scope": 1.,
|
| 103 |
+
})
|
| 104 |
+
return out
|
kdflow/algorithms/mp_opd_trust.py
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Isolated TRUST-R/B/P adapters; B/P require a complete optimizer window."""
|
| 2 |
+
import math
|
| 3 |
+
import random
|
| 4 |
+
|
| 5 |
+
import numpy as np
|
| 6 |
+
import torch
|
| 7 |
+
|
| 8 |
+
from . import register_algorithm
|
| 9 |
+
from .mp_opd import MPOnPolicyKD, fixed_partition
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
def capture_rng():
|
| 13 |
+
return (random.getstate(), np.random.get_state(), torch.get_rng_state(),
|
| 14 |
+
torch.cuda.get_rng_state_all() if torch.cuda.is_initialized() else None)
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
def restore_rng(state):
|
| 18 |
+
random.setstate(state[0])
|
| 19 |
+
np.random.set_state(state[1])
|
| 20 |
+
torch.set_rng_state(state[2])
|
| 21 |
+
if state[3] is not None:
|
| 22 |
+
torch.cuda.set_rng_state_all(state[3])
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
class TrustMPOnPolicyKD(MPOnPolicyKD):
|
| 26 |
+
mode = 'atomic'
|
| 27 |
+
trust_mode = None
|
| 28 |
+
|
| 29 |
+
def __init__(self, *args, **kwargs):
|
| 30 |
+
super().__init__(*args, **kwargs)
|
| 31 |
+
from ._mp_opd_port._mp_opd_grass_span import _env_flag
|
| 32 |
+
self.mode = self.trust_mode
|
| 33 |
+
self.requires_optimizer_batch = self.mode in {'trust_b', 'trust_p'}
|
| 34 |
+
self.eps_g = float(self.kd.mp_opd_trust_eps_g)
|
| 35 |
+
if not math.isfinite(self.eps_g) or self.eps_g < 0:
|
| 36 |
+
raise ValueError('TRUST eps_g must be finite and nonnegative')
|
| 37 |
+
if self.kd.mp_opd_grass_geometry != 'exact_head' or not _env_flag('MP_OPD_GRASS_EXACT_FP32', True):
|
| 38 |
+
raise ValueError('TRUST requires exact_head with exact FP32 geometry enabled')
|
| 39 |
+
if self.kd.mp_opd_trust_vocab_chunk < 1:
|
| 40 |
+
raise ValueError('TRUST vocabulary tile must be positive')
|
| 41 |
+
self._collector = self._result = None
|
| 42 |
+
self._normalizer = None
|
| 43 |
+
self._batch_metrics = {}
|
| 44 |
+
|
| 45 |
+
def _partition(self, atoms, student_ids, teacher_ids):
|
| 46 |
+
return fixed_partition(len(atoms), 1), None, 0
|
| 47 |
+
|
| 48 |
+
def training_step(self, batch):
|
| 49 |
+
if self.requires_optimizer_batch:
|
| 50 |
+
if self._collector is None and self._result is None:
|
| 51 |
+
raise RuntimeError('TRUST-B/P requires prepare_optimizer_batch before scoring/backward')
|
| 52 |
+
normalizer = float(batch['avg_micro_batch_token_num'])
|
| 53 |
+
if not math.isfinite(normalizer) or normalizer <= 0:
|
| 54 |
+
raise ValueError('TRUST-B/P requires a finite positive common normalizer')
|
| 55 |
+
if self._normalizer is not None and normalizer != self._normalizer:
|
| 56 |
+
raise ValueError('TRUST-B/P requires one common normalizer across the window')
|
| 57 |
+
self._normalizer = normalizer
|
| 58 |
+
return super().training_step(batch)
|
| 59 |
+
|
| 60 |
+
def _effective_rates(self, credits, atoms, partition, hidden, labels, **_kwargs):
|
| 61 |
+
from ._mp_opd_port._mp_opd_trust import (strict_mask, trust_calibrate,
|
| 62 |
+
trust_response_metrics, gram_affordable)
|
| 63 |
+
from ._mp_opd_port._mp_opd_grass_chunk import chunk_head_gram
|
| 64 |
+
strict = strict_mask([a.boundary_type for a in atoms]).to(credits.rate.device)
|
| 65 |
+
ranges = tuple((a.student_start, a.student_end) for a in atoms)
|
| 66 |
+
if self._collector is not None:
|
| 67 |
+
with torch.no_grad():
|
| 68 |
+
logits = self._head_logits(self.student.model.lm_head, hidden, self.student_cap, True)
|
| 69 |
+
self._collector.add_response(logits, hidden.detach(), labels, ranges,
|
| 70 |
+
credits.rate, strict, -credits.student_token_nll)
|
| 71 |
+
return torch.zeros_like(credits.rate), {}
|
| 72 |
+
if self.requires_optimizer_batch:
|
| 73 |
+
result, metrics = self._result, self._batch_metrics
|
| 74 |
+
else:
|
| 75 |
+
with torch.no_grad():
|
| 76 |
+
logits = self._head_logits(self.student.model.lm_head, hidden, self.student_cap, True)
|
| 77 |
+
if not gram_affordable(labels.numel(), logits.shape[-1], logits.device):
|
| 78 |
+
raise RuntimeError('TRUST-R exact Gram exceeds memory guard; no silent Atomic fallback')
|
| 79 |
+
head = chunk_head_gram(logits, hidden.detach(), labels, ranges, ((0,len(atoms)),),
|
| 80 |
+
selected_log_prob=-credits.student_token_nll, softcap=self.student_cap,
|
| 81 |
+
head_bias=getattr(self.student.model.lm_head,'bias',None) is not None)
|
| 82 |
+
q, h = credits.rate.detach().float(), head.gram.detach().float()
|
| 83 |
+
result = trust_calibrate(q[strict], q[~strict], h[strict][:,strict],
|
| 84 |
+
h[~strict][:,~strict], h[strict][:,~strict], eps_g=self.eps_g)
|
| 85 |
+
raw = trust_response_metrics(result,strict,credits.weight,scope='response')
|
| 86 |
+
metrics = {'mp_opd_trust_r_'+k: credits.rate.new_tensor(v) for k,v in raw.items()}
|
| 87 |
+
if self.mode == 'trust_p':
|
| 88 |
+
coefficients = torch.where(strict, float(result.strict_scale), 1.).to(credits.rate)
|
| 89 |
+
else:
|
| 90 |
+
coefficients = torch.where(strict, 1., float(result.lam)).to(credits.rate)
|
| 91 |
+
return credits.rate.detach()*coefficients, metrics
|
| 92 |
+
|
| 93 |
+
def prepare_optimizer_batch(self, batches, *, batch_loader=lambda x:x, check_window=True):
|
| 94 |
+
from ._mp_opd_port._mp_opd_trust_batch import TrustHeadAccumulator
|
| 95 |
+
from ._mp_opd_port._mp_opd_trust import trust_batch_metrics
|
| 96 |
+
from ._mp_opd_port._mp_opd_trust_p import trust_p_calibrate_scalars, trust_p_batch_metrics
|
| 97 |
+
if not self.requires_optimizer_batch:
|
| 98 |
+
return None
|
| 99 |
+
if self._collector is not None or self._result is not None:
|
| 100 |
+
raise RuntimeError('TRUST optimizer window is already active')
|
| 101 |
+
if not batches:
|
| 102 |
+
raise ValueError('TRUST requires a nonempty optimizer window')
|
| 103 |
+
if check_window and (len(batches) != self.strategy.accumulated_gradient or self.strategy.step != 0):
|
| 104 |
+
raise ValueError('TRUST-B/P requires exactly one complete optimizer accumulation window at step0')
|
| 105 |
+
before = capture_rng()
|
| 106 |
+
states = []
|
| 107 |
+
self._normalizer = None
|
| 108 |
+
collector = TrustHeadAccumulator(softcap=self.student_cap,
|
| 109 |
+
head_bias=getattr(self.student.model.lm_head,'bias',None) is not None,
|
| 110 |
+
vocab_chunk=self.kd.mp_opd_trust_vocab_chunk, force_fp32=self.mode == 'trust_p')
|
| 111 |
+
self._collector = collector
|
| 112 |
+
try:
|
| 113 |
+
with torch.no_grad():
|
| 114 |
+
for batch in batches:
|
| 115 |
+
states.append(capture_rng())
|
| 116 |
+
self.training_step(batch_loader(batch))
|
| 117 |
+
after = capture_rng()
|
| 118 |
+
options = {'calibrate':trust_p_calibrate_scalars} if self.mode == 'trust_p' else {}
|
| 119 |
+
self._result = collector.finalize(eps_g=self.eps_g, distributed=False, **options)
|
| 120 |
+
records = collector.responses
|
| 121 |
+
ns = sum(r['strict_atoms'] for r in records)
|
| 122 |
+
nm = sum(r['mismatch_atoms'] for r in records)
|
| 123 |
+
counts = dict(strict_span_count=ns,mismatch_span_count=nm,strict_atom_count=ns,mismatch_atom_count=nm,
|
| 124 |
+
strict_token_count=sum(r['strict_tokens'] for r in records),
|
| 125 |
+
mismatch_token_count=sum(r['mismatch_tokens'] for r in records),
|
| 126 |
+
strict_span_fraction=ns/max(ns+nm,1),mismatch_span_fraction=nm/max(ns+nm,1))
|
| 127 |
+
metric_fn = trust_p_batch_metrics if self.mode == 'trust_p' else trust_batch_metrics
|
| 128 |
+
self._batch_metrics = {f'mp_opd_{self.mode}_'+k: torch.tensor(float(v),device=self.teacher_head.weight.device,dtype=torch.float32)
|
| 129 |
+
for k,v in metric_fn(self._result,counts).items()}
|
| 130 |
+
prefix = 'mp_opd_'+self.mode+'_'
|
| 131 |
+
self._batch_metrics.update({prefix+'full_batch':torch.tensor(1.,device=self.teacher_head.weight.device),
|
| 132 |
+
prefix+'calibration_response_count':torch.tensor(float(len(records)),device=self.teacher_head.weight.device)})
|
| 133 |
+
return {'rng_states':states,'after_rng':after}
|
| 134 |
+
except BaseException:
|
| 135 |
+
self.finish_optimizer_batch()
|
| 136 |
+
raise
|
| 137 |
+
finally:
|
| 138 |
+
self._collector = None
|
| 139 |
+
restore_rng(before)
|
| 140 |
+
|
| 141 |
+
def finish_optimizer_batch(self):
|
| 142 |
+
self._collector = self._result = None
|
| 143 |
+
self._normalizer = None
|
| 144 |
+
self._batch_metrics = {}
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
@register_algorithm('mp_opd_trust_r')
|
| 148 |
+
class TrustRMPOnPolicyKD(TrustMPOnPolicyKD):
|
| 149 |
+
trust_mode = 'trust_r'
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
@register_algorithm('mp_opd_trust_b')
|
| 153 |
+
class TrustBMPOnPolicyKD(TrustMPOnPolicyKD):
|
| 154 |
+
trust_mode = 'trust_b'
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
@register_algorithm('mp_opd_trust_p')
|
| 158 |
+
class TrustPMPOnPolicyKD(TrustMPOnPolicyKD):
|
| 159 |
+
trust_mode = 'trust_p'
|
kdflow/algorithms/simct_paper.py
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Paper Eq.7-8 scores on the released SimCT candidate/alignment interface."""
|
| 2 |
+
import types
|
| 3 |
+
|
| 4 |
+
import torch
|
| 5 |
+
|
| 6 |
+
from . import register_algorithm
|
| 7 |
+
from .simct import SimCT
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def paper_virtual_vocab(upstream, segments, student_logits, teacher_logits, student_ids, teacher_ids):
|
| 11 |
+
# Normalize over the FULL native vocabulary at EACH autoregressive position.
|
| 12 |
+
# The original builder then selects shared entries and averages span scores.
|
| 13 |
+
return type(upstream)._build_virtual_vocab_logits(upstream, segments,
|
| 14 |
+
torch.log_softmax(student_logits.float(), -1),
|
| 15 |
+
torch.log_softmax(teacher_logits.float(), -1), student_ids, teacher_ids)
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
@register_algorithm('simct_paper')
|
| 19 |
+
class SimCTPaper(SimCT):
|
| 20 |
+
def __init__(self, *args, **kwargs):
|
| 21 |
+
super().__init__(*args, **kwargs)
|
| 22 |
+
self.upstream._build_virtual_vocab_logits = types.MethodType(paper_virtual_vocab, self.upstream)
|
kdflow/algorithms/trust_training.py
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Single-rank actor integration for complete-window TRUST-B/P calibration."""
|
| 2 |
+
from collections import defaultdict
|
| 3 |
+
|
| 4 |
+
import torch
|
| 5 |
+
|
| 6 |
+
from .mp_opd_trust import restore_rng
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def fit_trust_batch(actor, train_data):
|
| 10 |
+
actor.student.train()
|
| 11 |
+
if actor.args.train.use_dynamic_bsz:
|
| 12 |
+
actor.strategy.accumulated_gradient = len(train_data)
|
| 13 |
+
actor.strategy.step = 0
|
| 14 |
+
algo = actor.kd_algorithm
|
| 15 |
+
prepared = algo.prepare_optimizer_batch(train_data,batch_loader=actor._prepare_micro_batch)
|
| 16 |
+
status = defaultdict(list)
|
| 17 |
+
try:
|
| 18 |
+
for index,batch in enumerate(train_data):
|
| 19 |
+
restore_rng(prepared['rng_states'][index])
|
| 20 |
+
micro = actor._prepare_micro_batch(batch)
|
| 21 |
+
info = algo.training_step(micro)
|
| 22 |
+
for key,value in info.items():
|
| 23 |
+
status[key].append(value.item())
|
| 24 |
+
actor.strategy.backward(info['train/loss'],actor.student,actor.optim)
|
| 25 |
+
status['train/grad_norm'].append(torch.nn.utils.clip_grad_norm_(actor.student.parameters(),float('inf')).item())
|
| 26 |
+
actor.strategy.optimizer_step(actor.optim,actor.student,actor.scheduler)
|
| 27 |
+
if actor.args.kd.use_ema_teacher and actor.strategy.step == 0:
|
| 28 |
+
actor.ema_update()
|
| 29 |
+
for name in ('response_length','total_length'):
|
| 30 |
+
if name in micro:
|
| 31 |
+
status['rollout/'+name].append(micro[name].mean().item())
|
| 32 |
+
del micro
|
| 33 |
+
restore_rng(prepared['after_rng'])
|
| 34 |
+
finally:
|
| 35 |
+
algo.finish_optimizer_batch()
|
| 36 |
+
status = {k:sum(v)/len(v) for k,v in status.items()}
|
| 37 |
+
status['train/lr'] = actor.scheduler.get_last_lr()[0]
|
| 38 |
+
return {k:actor.strategy.all_reduce(v,op='mean') for k,v in status.items()}
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
@torch.no_grad()
|
| 42 |
+
def evaluate_trust_batch(actor, eval_data):
|
| 43 |
+
"""One evaluation-set calibration; no training-state reuse or optimizer step."""
|
| 44 |
+
algo = actor.kd_algorithm
|
| 45 |
+
was_training = actor.student.training
|
| 46 |
+
actor.student.eval()
|
| 47 |
+
batches = list(eval_data)
|
| 48 |
+
|
| 49 |
+
def load(batch):
|
| 50 |
+
micro = actor._prepare_micro_batch(batch)
|
| 51 |
+
weight = micro.pop('_eval_weight',1.)
|
| 52 |
+
if float(weight) != 1.:
|
| 53 |
+
raise ValueError('Single-rank TRUST evaluation requires unit sample weights')
|
| 54 |
+
micro['avg_micro_batch_token_num'] = 1.
|
| 55 |
+
return micro
|
| 56 |
+
|
| 57 |
+
sums,total = defaultdict(float),0.
|
| 58 |
+
try:
|
| 59 |
+
if not batches:
|
| 60 |
+
return {'eval/num_tokens':0.}
|
| 61 |
+
prepared = algo.prepare_optimizer_batch(batches,batch_loader=load,check_window=False)
|
| 62 |
+
for index,batch in enumerate(batches):
|
| 63 |
+
restore_rng(prepared['rng_states'][index])
|
| 64 |
+
micro = load(batch)
|
| 65 |
+
count = float(micro['stu_loss_mask'].sum())
|
| 66 |
+
metrics = algo.training_step(micro)
|
| 67 |
+
for key,value in metrics.items():
|
| 68 |
+
# Loss already sums tokens; diagnostic values are response averages.
|
| 69 |
+
sums[key] += float(value) * (1. if key in {'train/loss','train/kd_loss'} else count)
|
| 70 |
+
total += count
|
| 71 |
+
restore_rng(prepared['after_rng'])
|
| 72 |
+
finally:
|
| 73 |
+
algo.finish_optimizer_batch()
|
| 74 |
+
actor.student.train(was_training)
|
| 75 |
+
from kdflow.utils.logging_utils import normalize_eval_metrics
|
| 76 |
+
result = {k:actor.strategy.all_reduce(v,op='sum')/max(total,1.) for k,v in sums.items()}
|
| 77 |
+
result['eval/num_tokens'] = total
|
| 78 |
+
return normalize_eval_metrics(result)
|
kdflow/arguments/distillation_args.py
CHANGED
|
@@ -23,7 +23,7 @@ class DistillationArguments:
|
|
| 23 |
default="byte_align_ctkd",
|
| 24 |
metadata={
|
| 25 |
"help": "KD algorithm for each training step.",
|
| 26 |
-
"choices": ["simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", *MP_OPD_ALGORITHMS],
|
| 27 |
}
|
| 28 |
)
|
| 29 |
kd_loss_fn: str = field(
|
|
@@ -129,6 +129,8 @@ class DistillationArguments:
|
|
| 129 |
mp_opd_align_eps_h: float = field(default=1e-12)
|
| 130 |
mp_opd_align_new_min_descent_fraction: float = field(default=0.1)
|
| 131 |
mp_opd_align_new_eps_g: float = field(default=1e-12)
|
|
|
|
|
|
|
| 132 |
|
| 133 |
def __post_init__(self):
|
| 134 |
if self.multi_teacher_config is not None:
|
|
@@ -136,18 +138,23 @@ class DistillationArguments:
|
|
| 136 |
"This release supports one teacher model per run; "
|
| 137 |
"use --teacher_name_or_path instead of --multi_teacher_config."
|
| 138 |
)
|
| 139 |
-
supported_algorithms = {"simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", *MP_OPD_ALGORITHMS}
|
| 140 |
if self.kd_algorithm not in supported_algorithms:
|
| 141 |
raise ValueError(
|
| 142 |
f"kd_algorithm must be one of {sorted(supported_algorithms)}, "
|
| 143 |
f"got {self.kd_algorithm!r}."
|
| 144 |
)
|
| 145 |
-
if self.kd_algorithm
|
| 146 |
if self.kd_ratio != 1 or self.kd_temperature != 1 or self.kd_loss_fn != 'rkl':
|
| 147 |
raise ValueError('SimCT baseline requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
|
| 148 |
if not math.isfinite(self.simct_gh_mask_threshold) or self.simct_gh_mask_threshold < 0:
|
| 149 |
raise ValueError('SimCT G(h) threshold must be finite and nonnegative')
|
| 150 |
if self.kd_algorithm in MP_OPD_ALGORITHMS:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
if self.kd_algorithm == 'mp_opd_align_new':
|
| 152 |
if self.mp_opd_grass_geometry != 'exact_head':
|
| 153 |
raise ValueError('ALIGN-NEW requires exact_head geometry')
|
|
|
|
| 23 |
default="byte_align_ctkd",
|
| 24 |
metadata={
|
| 25 |
"help": "KD algorithm for each training step.",
|
| 26 |
+
"choices": ["simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", "simct_paper", *MP_OPD_ALGORITHMS],
|
| 27 |
}
|
| 28 |
)
|
| 29 |
kd_loss_fn: str = field(
|
|
|
|
| 129 |
mp_opd_align_eps_h: float = field(default=1e-12)
|
| 130 |
mp_opd_align_new_min_descent_fraction: float = field(default=0.1)
|
| 131 |
mp_opd_align_new_eps_g: float = field(default=1e-12)
|
| 132 |
+
mp_opd_trust_eps_g: float = field(default=1e-12)
|
| 133 |
+
mp_opd_trust_vocab_chunk: int = field(default=4096)
|
| 134 |
|
| 135 |
def __post_init__(self):
|
| 136 |
if self.multi_teacher_config is not None:
|
|
|
|
| 138 |
"This release supports one teacher model per run; "
|
| 139 |
"use --teacher_name_or_path instead of --multi_teacher_config."
|
| 140 |
)
|
| 141 |
+
supported_algorithms = {"simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", "simct_paper", *MP_OPD_ALGORITHMS}
|
| 142 |
if self.kd_algorithm not in supported_algorithms:
|
| 143 |
raise ValueError(
|
| 144 |
f"kd_algorithm must be one of {sorted(supported_algorithms)}, "
|
| 145 |
f"got {self.kd_algorithm!r}."
|
| 146 |
)
|
| 147 |
+
if self.kd_algorithm in {'simct','simct_paper'}:
|
| 148 |
if self.kd_ratio != 1 or self.kd_temperature != 1 or self.kd_loss_fn != 'rkl':
|
| 149 |
raise ValueError('SimCT baseline requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
|
| 150 |
if not math.isfinite(self.simct_gh_mask_threshold) or self.simct_gh_mask_threshold < 0:
|
| 151 |
raise ValueError('SimCT G(h) threshold must be finite and nonnegative')
|
| 152 |
if self.kd_algorithm in MP_OPD_ALGORITHMS:
|
| 153 |
+
if self.kd_algorithm.startswith('mp_opd_trust_'):
|
| 154 |
+
if self.mp_opd_grass_geometry != 'exact_head':
|
| 155 |
+
raise ValueError('TRUST requires exact_head geometry')
|
| 156 |
+
if not math.isfinite(self.mp_opd_trust_eps_g) or self.mp_opd_trust_eps_g < 0 or self.mp_opd_trust_vocab_chunk < 1:
|
| 157 |
+
raise ValueError('TRUST requires finite nonnegative eps_g and positive vocabulary tile')
|
| 158 |
if self.kd_algorithm == 'mp_opd_align_new':
|
| 159 |
if self.mp_opd_grass_geometry != 'exact_head':
|
| 160 |
raise ValueError('ALIGN-NEW requires exact_head geometry')
|
kdflow/cli/train_kd_on_policy.py
CHANGED
|
@@ -22,7 +22,7 @@ from kdflow.utils.utils import get_tokenizer, load_custom_eval_fn
|
|
| 22 |
|
| 23 |
|
| 24 |
def train(args):
|
| 25 |
-
if args.kd.kd_algorithm
|
| 26 |
args.train.num_nodes * args.train.num_gpus_per_node != 1 or args.model.ring_attn_size != 1
|
| 27 |
):
|
| 28 |
raise ValueError('SimCT baseline initially requires one student rank and ring_attn_size=1')
|
|
@@ -96,11 +96,12 @@ def train(args):
|
|
| 96 |
"byte_align_ctkd",
|
| 97 |
"topk_byte_align_ctkd",
|
| 98 |
"simct",
|
|
|
|
| 99 |
*MP_OPD_ALGORITHMS,
|
| 100 |
}:
|
| 101 |
raise ValueError(
|
| 102 |
"Cross-tokenizer distillation supports simple_ctkd, "
|
| 103 |
-
"byte_align_ctkd, topk_byte_align_ctkd, simct, or an opt-in mp_opd_* mode."
|
| 104 |
)
|
| 105 |
|
| 106 |
environment_rollout = bool(args.rollout.alfworld_config)
|
|
|
|
| 22 |
|
| 23 |
|
| 24 |
def train(args):
|
| 25 |
+
if args.kd.kd_algorithm in {'simct','simct_paper'} and (
|
| 26 |
args.train.num_nodes * args.train.num_gpus_per_node != 1 or args.model.ring_attn_size != 1
|
| 27 |
):
|
| 28 |
raise ValueError('SimCT baseline initially requires one student rank and ring_attn_size=1')
|
|
|
|
| 96 |
"byte_align_ctkd",
|
| 97 |
"topk_byte_align_ctkd",
|
| 98 |
"simct",
|
| 99 |
+
"simct_paper",
|
| 100 |
*MP_OPD_ALGORITHMS,
|
| 101 |
}:
|
| 102 |
raise ValueError(
|
| 103 |
"Cross-tokenizer distillation supports simple_ctkd, "
|
| 104 |
+
"byte_align_ctkd, topk_byte_align_ctkd, simct, simct_paper, or an opt-in mp_opd_* mode."
|
| 105 |
)
|
| 106 |
|
| 107 |
environment_rollout = bool(args.rollout.alfworld_config)
|
kdflow/mp_opd_modes.py
CHANGED
|
@@ -3,4 +3,5 @@
|
|
| 3 |
MP_OPD_ALGORITHMS = (
|
| 4 |
'mp_opd_atomic', 'mp_opd_fixed', 'mp_opd_random',
|
| 5 |
'mp_opd_grass_chunk', 'mp_opd_align', 'mp_opd_align_new',
|
|
|
|
| 6 |
)
|
|
|
|
| 3 |
MP_OPD_ALGORITHMS = (
|
| 4 |
'mp_opd_atomic', 'mp_opd_fixed', 'mp_opd_random',
|
| 5 |
'mp_opd_grass_chunk', 'mp_opd_align', 'mp_opd_align_new',
|
| 6 |
+
'mp_opd_trust_r', 'mp_opd_trust_b', 'mp_opd_trust_p',
|
| 7 |
)
|
kdflow/ray/train/student_actor.py
CHANGED
|
@@ -288,6 +288,9 @@ class StudentRayActor:
|
|
| 288 |
Returns:
|
| 289 |
Training status dict
|
| 290 |
"""
|
|
|
|
|
|
|
|
|
|
| 291 |
self.student.train()
|
| 292 |
status = defaultdict(list)
|
| 293 |
|
|
@@ -338,6 +341,9 @@ class StudentRayActor:
|
|
| 338 |
@torch.no_grad()
|
| 339 |
def evaluate(self, eval_data):
|
| 340 |
"""Evaluate the current student without updating model states."""
|
|
|
|
|
|
|
|
|
|
| 341 |
was_training = self.student.training
|
| 342 |
self.student.eval()
|
| 343 |
metric_sums = defaultdict(float)
|
|
|
|
| 288 |
Returns:
|
| 289 |
Training status dict
|
| 290 |
"""
|
| 291 |
+
if getattr(self.kd_algorithm,'requires_optimizer_batch',False):
|
| 292 |
+
from kdflow.algorithms.trust_training import fit_trust_batch
|
| 293 |
+
return fit_trust_batch(self,train_data)
|
| 294 |
self.student.train()
|
| 295 |
status = defaultdict(list)
|
| 296 |
|
|
|
|
| 341 |
@torch.no_grad()
|
| 342 |
def evaluate(self, eval_data):
|
| 343 |
"""Evaluate the current student without updating model states."""
|
| 344 |
+
if getattr(self.kd_algorithm,'requires_optimizer_batch',False):
|
| 345 |
+
from kdflow.algorithms.trust_training import evaluate_trust_batch
|
| 346 |
+
return evaluate_trust_batch(self,eval_data)
|
| 347 |
was_training = self.student.training
|
| 348 |
self.student.eval()
|
| 349 |
metric_sums = defaultdict(float)
|
tests/fixtures/trust/_mp_opd_trust.py.txt
ADDED
|
@@ -0,0 +1,348 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TRUST v1.0 - trusted reference update scaling for cross-tokenizer OPD.
|
| 2 |
+
|
| 3 |
+
Mismatch supervision may carry signal, but its direction and scale should be
|
| 4 |
+
calibrated against the more reliable strict/matched supervision. TRUST does not
|
| 5 |
+
touch relative credits inside mismatch spans: per calibration unit it scales the
|
| 6 |
+
whole mismatch update by ``lambda = max(0, <gS, gM> / ||gM||^2)`` and keeps the
|
| 7 |
+
strict update whole.
|
| 8 |
+
|
| 9 |
+
Two modes share this file: TRUST-R (one lambda per response) and TRUST-B (one
|
| 10 |
+
lambda per optimizer batch, with microbatch calibration an explicit legacy
|
| 11 |
+
ablation). The only intended difference between them is the
|
| 12 |
+
calibration aggregation level; the strict/mismatch partition, the proxy, and
|
| 13 |
+
the guards are identical.
|
| 14 |
+
|
| 15 |
+
The strict/mismatch partition comes from the atomizer, not from a threshold:
|
| 16 |
+
an atom whose student and teacher spans are both exactly one token over the
|
| 17 |
+
same bytes (``boundary_type == "one_to_one"``) is matched supervision, and an
|
| 18 |
+
atom that straddles a tokenization boundary (``multi_token``) is mismatch. Any
|
| 19 |
+
atom with another boundary type fails closed, because silently filing it under
|
| 20 |
+
either side would change what the calibration means.
|
| 21 |
+
|
| 22 |
+
The proxy is the GRASS LM-head geometry, reused unchanged: with per-atom
|
| 23 |
+
coefficients ``q_i = r_i`` (the OPD atom loss is a plain token-sum, so the
|
| 24 |
+
outer coefficient is 1 and the response weight is a plain sum) and token
|
| 25 |
+
reduction ``a_it = 1`` (``current_nll`` is a token sum over the atom), the
|
| 26 |
+
effective coefficients are exactly the rates the loss consumes. The Gram code
|
| 27 |
+
is shared, so a TRUST number and a GRASS number read the same geometry.
|
| 28 |
+
|
| 29 |
+
Deliberate v1 deviations from the spec text, all fail-safe rather than silent:
|
| 30 |
+
|
| 31 |
+
* Median/p10/p25/p75/p90 keys are not emitted. Metrics reduce per micro-batch
|
| 32 |
+
by sum-then-mean, which cannot express a percentile; emitting a mean under a
|
| 33 |
+
``_median`` key would lie about the statistic. Means and fractions over the
|
| 34 |
+
micro-batch responses are exact as stated (production uses
|
| 35 |
+
micro_train_batch_size=4).
|
| 36 |
+
* ``mp_opd_trust_scope`` is a float code (0.0 = response, 1.0 = batch unit) because
|
| 37 |
+
the metric table only carries finite tensors.
|
| 38 |
+
``mp_opd_trust_full_batch`` distinguishes the complete optimizer batch from
|
| 39 |
+
the legacy microbatch unit.
|
| 40 |
+
* Samples or units without both sides log 0.0 for undefined quantities (never
|
| 41 |
+
NaN: the trainer fails fast on non-finite metrics), with the eligibility
|
| 42 |
+
fractions (``has_both``/``zero_*``) as the interpretive key.
|
| 43 |
+
* TRUST requires ``exact_head`` geometry. The ``diag`` ablation zeroes every
|
| 44 |
+
cross-atom term, which would force every dot product to zero and every lambda
|
| 45 |
+
to zero: a mode that can never act. Failing closed beats a silent no-op.
|
| 46 |
+
"""
|
| 47 |
+
|
| 48 |
+
from __future__ import annotations
|
| 49 |
+
|
| 50 |
+
from dataclasses import dataclass
|
| 51 |
+
from typing import Sequence
|
| 52 |
+
|
| 53 |
+
import torch
|
| 54 |
+
|
| 55 |
+
# Section 9 fixes this: a mismatch norm at or below it carries no usable
|
| 56 |
+
# direction, so the calibration that would divide by it is skipped.
|
| 57 |
+
TRUST_EPS_G = 1e-12
|
| 58 |
+
# Only used to keep norm denominators away from zero; it never decides whether
|
| 59 |
+
# a unit calibrates or what lambda it gets.
|
| 60 |
+
TRUST_EPS_NORM = 1e-12
|
| 61 |
+
# Transient multiplier for one exact-Gram build: the [tokens, vocab] window,
|
| 62 |
+
# its Jacobian, the probabilities, and the vocabulary product temporaries.
|
| 63 |
+
# A step whose estimate exceeds half the free device memory degrades to the
|
| 64 |
+
# atomic fallback (loudly counted) instead of attempting an allocation the
|
| 65 |
+
# device cannot hold. Killed a 171-step trust_b run at 11.73 GiB attempted on
|
| 66 |
+
# 6.34 GiB free; the fallback that would have saved it did not exist yet.
|
| 67 |
+
TRUST_GRAM_TRANSIENT_FACTOR = 4.0
|
| 68 |
+
TRUST_GRAM_FREE_FRACTION = 0.5
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def gram_affordable(n_tokens: int, vocab: int, device: torch.device) -> bool:
|
| 72 |
+
"""Whether one exact Gram build fits in the currently free device memory.
|
| 73 |
+
|
| 74 |
+
Estimates ``n_tokens * vocab`` fp32 entries times the transient factor and
|
| 75 |
+
requires it under half the free bytes. On a device without memory info
|
| 76 |
+
(CPU tests) everything is affordable; the production guard this provides
|
| 77 |
+
is against the allocation, not the math.
|
| 78 |
+
"""
|
| 79 |
+
if n_tokens <= 0 or vocab <= 0:
|
| 80 |
+
raise ValueError("gram affordability needs positive token and vocab counts")
|
| 81 |
+
if device.type != "cuda" or not torch.cuda.is_available():
|
| 82 |
+
return True
|
| 83 |
+
free_bytes, _ = torch.cuda.mem_get_info(device)
|
| 84 |
+
need_bytes = float(n_tokens) * float(vocab) * 4.0 * TRUST_GRAM_TRANSIENT_FACTOR
|
| 85 |
+
return need_bytes <= float(free_bytes) * TRUST_GRAM_FREE_FRACTION
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
@dataclass(frozen=True)
|
| 89 |
+
class TrustUnitResult:
|
| 90 |
+
"""One calibration unit's lambda plus everything its diagnostics derive from."""
|
| 91 |
+
|
| 92 |
+
lam: float
|
| 93 |
+
dot: float
|
| 94 |
+
gs_norm2: float
|
| 95 |
+
gm_norm2: float
|
| 96 |
+
cosine: float
|
| 97 |
+
norm_ratio: float
|
| 98 |
+
kappa: float
|
| 99 |
+
has_strict: bool
|
| 100 |
+
has_mismatch: bool
|
| 101 |
+
calibrated: bool
|
| 102 |
+
update_cosine: float
|
| 103 |
+
update_norm_ratio: float
|
| 104 |
+
n_strict: int
|
| 105 |
+
n_mismatch: int
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def _require_finite(name: str, *tensors: torch.Tensor) -> None:
|
| 109 |
+
for tensor in tensors:
|
| 110 |
+
if not bool(torch.isfinite(tensor).all()):
|
| 111 |
+
raise ValueError(f"TRUST received a non-finite {name}")
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def trust_calibrate(
|
| 115 |
+
qs: torch.Tensor,
|
| 116 |
+
qm: torch.Tensor,
|
| 117 |
+
gram_ss: torch.Tensor,
|
| 118 |
+
gram_mm: torch.Tensor,
|
| 119 |
+
gram_sm: torch.Tensor,
|
| 120 |
+
*,
|
| 121 |
+
eps_g: float = TRUST_EPS_G,
|
| 122 |
+
eps_norm: float = TRUST_EPS_NORM,
|
| 123 |
+
) -> TrustUnitResult:
|
| 124 |
+
"""Calibrate one strict/mismatch pair from its Gram blocks.
|
| 125 |
+
|
| 126 |
+
``qs``/``qm`` are the effective strict/mismatch coefficients and
|
| 127 |
+
``gram_ss``/``gram_mm``/``gram_sm`` the matching blocks of the exact
|
| 128 |
+
LM-head Gram. The geometry is FP32; quadratic reductions promote to FP64.
|
| 129 |
+
The returned lambda is a Python float, so it can never carry a graph.
|
| 130 |
+
"""
|
| 131 |
+
_require_finite("coefficients", qs, qm)
|
| 132 |
+
_require_finite("Gram blocks", gram_ss, gram_mm, gram_sm)
|
| 133 |
+
if not (float(eps_g) >= 0.0):
|
| 134 |
+
raise ValueError(f"TRUST eps_g must be non-negative, got {eps_g!r}")
|
| 135 |
+
qs = qs.detach().to(torch.float64).reshape(-1)
|
| 136 |
+
qm = qm.detach().to(torch.float64).reshape(-1)
|
| 137 |
+
gram_ss, gram_mm, gram_sm = (g.detach().double() for g in (gram_ss, gram_mm, gram_sm))
|
| 138 |
+
n_s, n_m = int(qs.numel()), int(qm.numel())
|
| 139 |
+
if (n_s, n_m) != (int(gram_ss.shape[0]), int(gram_mm.shape[0])):
|
| 140 |
+
raise ValueError(
|
| 141 |
+
f"TRUST Gram blocks {tuple(gram_ss.shape)}/{tuple(gram_mm.shape)} "
|
| 142 |
+
f"do not match coefficient counts ({n_s}, {n_m})"
|
| 143 |
+
)
|
| 144 |
+
if tuple(gram_sm.shape) != (n_s, n_m):
|
| 145 |
+
raise ValueError(
|
| 146 |
+
f"TRUST cross Gram {tuple(gram_sm.shape)} does not match ({n_s}, {n_m})"
|
| 147 |
+
)
|
| 148 |
+
|
| 149 |
+
# Empty sides have no geometry to read: the scalars below are defined as
|
| 150 |
+
# zero rather than derived from a matmul over an empty axis.
|
| 151 |
+
gs2 = float(qs @ gram_ss @ qs) if n_s else 0.0
|
| 152 |
+
gm2 = float(qm @ gram_mm @ qm) if n_m else 0.0
|
| 153 |
+
dot = float(qs @ gram_sm @ qm) if (n_s and n_m) else 0.0
|
| 154 |
+
scales = [float(q.abs() @ g.abs() @ q.abs()) if q.numel() else 0.0
|
| 155 |
+
for q, g in ((qs, gram_ss), (qm, gram_mm))]
|
| 156 |
+
for name, norm, scale in zip(("strict", "mismatch"), (gs2, gm2), scales):
|
| 157 |
+
if norm < -1e-6 * max(scale, 1e-300):
|
| 158 |
+
raise ValueError(f"TRUST {name} Gram gives a materially negative squared norm")
|
| 159 |
+
gs2, gm2 = max(gs2, 0.), max(gm2, 0.)
|
| 160 |
+
bound = (gs2 * gm2)**.5
|
| 161 |
+
if abs(dot) > bound + 1e-6 * max((scales[0]*scales[1])**.5, 1e-300):
|
| 162 |
+
raise ValueError("TRUST Gram violates Cauchy-Schwarz")
|
| 163 |
+
dot = min(max(dot, -bound), bound)
|
| 164 |
+
return trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m,
|
| 165 |
+
eps_g=eps_g, eps_norm=eps_norm)
|
| 166 |
+
|
| 167 |
+
|
| 168 |
+
def trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *,
|
| 169 |
+
eps_g=TRUST_EPS_G, eps_norm=TRUST_EPS_NORM):
|
| 170 |
+
"""The same calibration for explicit summed head updates over a full batch."""
|
| 171 |
+
if not (0 <= float(eps_g) < float("inf")):
|
| 172 |
+
raise ValueError("TRUST eps_g must be finite and nonnegative")
|
| 173 |
+
for name, value in (("gs_norm2", gs2), ("gm_norm2", gm2), ("dot", dot)):
|
| 174 |
+
if not abs(value) < float("inf"):
|
| 175 |
+
raise ValueError(f"TRUST {name} is non-finite")
|
| 176 |
+
if gs2 < 0 or gm2 < 0:
|
| 177 |
+
raise ValueError("TRUST squared norms must be nonnegative")
|
| 178 |
+
bound = (gs2 * gm2)**.5
|
| 179 |
+
if abs(dot) > bound * (1. + 1e-6) + 1e-300:
|
| 180 |
+
raise ValueError("TRUST summed updates violate Cauchy-Schwarz")
|
| 181 |
+
dot = min(max(dot, -bound), bound)
|
| 182 |
+
|
| 183 |
+
has_strict = n_s > 0
|
| 184 |
+
has_mismatch = n_m > 0
|
| 185 |
+
calibrated = bool(has_strict and has_mismatch and gm2 > float(eps_g))
|
| 186 |
+
lam = max(0.0, dot / gm2) if calibrated else 0.0
|
| 187 |
+
|
| 188 |
+
gs = gs2**0.5
|
| 189 |
+
gm = gm2**0.5
|
| 190 |
+
cosine = dot / (gs * gm + float(eps_norm))
|
| 191 |
+
norm_ratio = gm / (gs + float(eps_norm))
|
| 192 |
+
kappa = lam * gm / (gs + float(eps_norm))
|
| 193 |
+
|
| 194 |
+
# g_pre = gS + gM and g_post = gS + lam gM, expanded in Gram scalars so no
|
| 195 |
+
# vector is ever formed. When both updates vanish they are identical.
|
| 196 |
+
pre2 = gs2 + gm2 + 2.0 * dot
|
| 197 |
+
post2 = gs2 + lam * lam * gm2 + 2.0 * lam * dot
|
| 198 |
+
inner = gs2 + lam * gm2 + (1.0 + lam) * dot
|
| 199 |
+
pre2 = max(pre2, 0.0)
|
| 200 |
+
post2 = max(post2, 0.0)
|
| 201 |
+
if pre2 <= 0.0 and post2 <= 0.0:
|
| 202 |
+
update_cosine = 1.0
|
| 203 |
+
elif pre2 <= 0.0 or post2 <= 0.0:
|
| 204 |
+
update_cosine = 0.0
|
| 205 |
+
else:
|
| 206 |
+
update_cosine = inner / (pre2**0.5 * post2**0.5 + float(eps_norm))
|
| 207 |
+
update_norm_ratio = post2**0.5 / (pre2**0.5 + float(eps_norm))
|
| 208 |
+
|
| 209 |
+
return TrustUnitResult(
|
| 210 |
+
lam=lam,
|
| 211 |
+
dot=dot,
|
| 212 |
+
gs_norm2=gs2,
|
| 213 |
+
gm_norm2=gm2,
|
| 214 |
+
cosine=cosine,
|
| 215 |
+
norm_ratio=norm_ratio,
|
| 216 |
+
kappa=kappa,
|
| 217 |
+
has_strict=has_strict,
|
| 218 |
+
has_mismatch=has_mismatch,
|
| 219 |
+
calibrated=calibrated,
|
| 220 |
+
update_cosine=update_cosine,
|
| 221 |
+
update_norm_ratio=update_norm_ratio,
|
| 222 |
+
n_strict=n_s,
|
| 223 |
+
n_mismatch=n_m,
|
| 224 |
+
)
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def strict_mask(boundary_types: Sequence[str]) -> torch.Tensor:
|
| 228 |
+
"""Strict-side mask from atomizer boundary types, failing closed.
|
| 229 |
+
|
| 230 |
+
``one_to_one`` is matched supervision, ``multi_token`` is mismatch. A
|
| 231 |
+
third kind would silently change the calibration under either choice, so
|
| 232 |
+
it raises instead of being filed anywhere.
|
| 233 |
+
"""
|
| 234 |
+
mask = []
|
| 235 |
+
for kind in boundary_types:
|
| 236 |
+
if kind == "one_to_one":
|
| 237 |
+
mask.append(True)
|
| 238 |
+
elif kind == "multi_token":
|
| 239 |
+
mask.append(False)
|
| 240 |
+
else:
|
| 241 |
+
raise ValueError(f"TRUST has no side for boundary type {kind!r}")
|
| 242 |
+
return torch.tensor(mask, dtype=torch.bool)
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
def trust_response_loss(
|
| 246 |
+
current_nll: torch.Tensor,
|
| 247 |
+
rate: torch.Tensor,
|
| 248 |
+
is_strict: torch.Tensor,
|
| 249 |
+
is_mismatch: torch.Tensor,
|
| 250 |
+
result: TrustUnitResult,
|
| 251 |
+
) -> torch.Tensor:
|
| 252 |
+
"""``L_S + stopgrad(lambda) L_M`` with exactly the OPD loss coefficients.
|
| 253 |
+
|
| 254 |
+
Each side is a ``(rates.detach() * nll).sum()``, the same construction as
|
| 255 |
+
``soft_partition_loss``: the strict side keeps its credits whole and the
|
| 256 |
+
mismatch side is scaled by the detached lambda. The lambda is a Python
|
| 257 |
+
float by construction, so no graph can pass through the calibration rule.
|
| 258 |
+
"""
|
| 259 |
+
if not (bool(is_strict.any()) or bool(is_mismatch.any())):
|
| 260 |
+
raise ValueError("TRUST needs at least one strict or mismatch atom")
|
| 261 |
+
if bool((is_strict & is_mismatch).any()):
|
| 262 |
+
raise ValueError("TRUST strict and mismatch sets must be disjoint")
|
| 263 |
+
zero = current_nll.new_zeros(())
|
| 264 |
+
strict = (torch.where(is_strict, rate.detach(), zero) * current_nll).sum()
|
| 265 |
+
mismatch = (torch.where(is_mismatch, rate.detach(), zero) * current_nll).sum()
|
| 266 |
+
return strict + float(result.lam) * mismatch
|
| 267 |
+
|
| 268 |
+
|
| 269 |
+
def _counts(is_strict: torch.Tensor, token_counts: torch.Tensor) -> dict[str, float]:
|
| 270 |
+
"""Raw per-unit coverage counts; the caller decides the aggregation."""
|
| 271 |
+
is_mismatch = ~is_strict
|
| 272 |
+
n_s = int(is_strict.sum())
|
| 273 |
+
n_m = int(is_mismatch.sum())
|
| 274 |
+
total = n_s + n_m
|
| 275 |
+
return {
|
| 276 |
+
"strict_span_count": float(n_s),
|
| 277 |
+
"mismatch_span_count": float(n_m),
|
| 278 |
+
# One atom is one supervision span here: there is no higher grouping
|
| 279 |
+
# for TRUST to aggregate, so the span and atom keys coincide by
|
| 280 |
+
# definition rather than by coincidence.
|
| 281 |
+
"strict_atom_count": float(n_s),
|
| 282 |
+
"mismatch_atom_count": float(n_m),
|
| 283 |
+
"strict_token_count": float(token_counts[is_strict].sum()),
|
| 284 |
+
"mismatch_token_count": float(token_counts[is_mismatch].sum()),
|
| 285 |
+
"strict_span_fraction": float(n_s / total) if total else 0.0,
|
| 286 |
+
"mismatch_span_fraction": float(n_m / total) if total else 0.0,
|
| 287 |
+
"has_strict": float(n_s > 0),
|
| 288 |
+
"has_mismatch": float(n_m > 0),
|
| 289 |
+
"has_both": float(n_s > 0 and n_m > 0),
|
| 290 |
+
"zero_strict": float(n_s == 0),
|
| 291 |
+
"zero_mismatch": float(n_m == 0),
|
| 292 |
+
}
|
| 293 |
+
|
| 294 |
+
|
| 295 |
+
def trust_response_metrics(
|
| 296 |
+
result: TrustUnitResult,
|
| 297 |
+
is_strict: torch.Tensor,
|
| 298 |
+
token_counts: torch.Tensor,
|
| 299 |
+
*,
|
| 300 |
+
scope: str,
|
| 301 |
+
) -> dict[str, float]:
|
| 302 |
+
"""Per-unit metric values; the training loop averages them over responses."""
|
| 303 |
+
if scope not in ("response", "batch"):
|
| 304 |
+
raise ValueError(f"TRUST scope must be response|batch, got {scope!r}")
|
| 305 |
+
out = _counts(is_strict, token_counts)
|
| 306 |
+
out.update(
|
| 307 |
+
{
|
| 308 |
+
"cosine": result.cosine,
|
| 309 |
+
"cosine_negative": float(result.cosine < 0.0),
|
| 310 |
+
"mismatch_to_strict_norm_ratio": result.norm_ratio,
|
| 311 |
+
"lambda": result.lam,
|
| 312 |
+
"lambda_zero": float(result.lam <= 0.0),
|
| 313 |
+
"lambda_gt1": float(result.lam > 1.0),
|
| 314 |
+
"calibrated": float(result.calibrated),
|
| 315 |
+
"calibrated_mismatch_norm_ratio": result.kappa,
|
| 316 |
+
"update_cosine_pre_post": result.update_cosine,
|
| 317 |
+
"update_norm_ratio_post_pre": result.update_norm_ratio,
|
| 318 |
+
"scope": 0.0 if scope == "response" else 1.0,
|
| 319 |
+
}
|
| 320 |
+
)
|
| 321 |
+
return out
|
| 322 |
+
|
| 323 |
+
|
| 324 |
+
def trust_batch_metrics(
|
| 325 |
+
results: TrustUnitResult,
|
| 326 |
+
counts: dict[str, float],
|
| 327 |
+
) -> dict[str, float]:
|
| 328 |
+
"""One calibration unit's values, logged without response averaging."""
|
| 329 |
+
return {
|
| 330 |
+
"batch_strict_span_count": counts["strict_span_count"],
|
| 331 |
+
"batch_mismatch_span_count": counts["mismatch_span_count"],
|
| 332 |
+
"batch_strict_atom_count": counts["strict_atom_count"],
|
| 333 |
+
"batch_mismatch_atom_count": counts["mismatch_atom_count"],
|
| 334 |
+
"batch_strict_token_count": counts["strict_token_count"],
|
| 335 |
+
"batch_mismatch_token_count": counts["mismatch_token_count"],
|
| 336 |
+
"batch_strict_span_fraction": counts["strict_span_fraction"],
|
| 337 |
+
"batch_mismatch_span_fraction": counts["mismatch_span_fraction"],
|
| 338 |
+
"cosine": results.cosine,
|
| 339 |
+
"mismatch_to_strict_norm_ratio": results.norm_ratio,
|
| 340 |
+
"lambda": results.lam,
|
| 341 |
+
"batch_has_strict": float(results.has_strict),
|
| 342 |
+
"batch_has_mismatch": float(results.has_mismatch),
|
| 343 |
+
"batch_calibrated": float(results.calibrated),
|
| 344 |
+
"calibrated_mismatch_norm_ratio": results.kappa,
|
| 345 |
+
"update_cosine_pre_post": results.update_cosine,
|
| 346 |
+
"update_norm_ratio_post_pre": results.update_norm_ratio,
|
| 347 |
+
"scope": 1.0,
|
| 348 |
+
}
|
tests/fixtures/trust/_mp_opd_trust_batch.py.txt
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Exact full-batch TRUST head updates, streamed without retaining forward graphs."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import contextlib
|
| 5 |
+
import torch
|
| 6 |
+
|
| 7 |
+
from ._mp_opd_grass_span import _exact_fp32, _log_partition_from_selected
|
| 8 |
+
from ._mp_opd_trust import trust_calibrate_scalars
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class TrustHeadAccumulator:
|
| 12 |
+
"""Sum strict/mismatch LM-head gradients, including cross-response terms.
|
| 13 |
+
|
| 14 |
+
Storage is two [vocab, hidden] fp32 tensors, independent of batch token count.
|
| 15 |
+
Vocabulary tiles bound transients; no [batch tokens, vocab] concatenation and
|
| 16 |
+
no atom-by-atom global Gram are constructed. Loss normalization is one common
|
| 17 |
+
factor for the entire accumulation window and cancels from lambda.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
def __init__(self, *, softcap=None, head_bias=False, vocab_chunk=4096,
|
| 21 |
+
force_fp32=False):
|
| 22 |
+
self.softcap = softcap
|
| 23 |
+
self.head_bias = head_bias
|
| 24 |
+
self.vocab_chunk = int(vocab_chunk)
|
| 25 |
+
self.force_fp32 = bool(force_fp32)
|
| 26 |
+
if self.vocab_chunk < 1:
|
| 27 |
+
raise ValueError("TRUST vocabulary tile must be positive")
|
| 28 |
+
self.gs = self.gm = self.bs = self.bm = None
|
| 29 |
+
self.responses = []
|
| 30 |
+
|
| 31 |
+
@torch.no_grad()
|
| 32 |
+
def add_response(self, logits, hidden, labels, ranges, rate, is_strict,
|
| 33 |
+
selected_log_prob):
|
| 34 |
+
covered = ranges[-1][1]
|
| 35 |
+
logits, hidden, labels = logits[:covered], hidden[:covered], labels[:covered]
|
| 36 |
+
lengths = torch.tensor([b-a for a, b in ranges], device=logits.device)
|
| 37 |
+
if sum(b-a for a, b in ranges) != covered or ranges[0][0] != 0:
|
| 38 |
+
raise ValueError("TRUST ranges must tile the covered token prefix")
|
| 39 |
+
if any(ranges[i][1] != ranges[i+1][0] for i in range(len(ranges)-1)):
|
| 40 |
+
raise ValueError("TRUST ranges must have no gaps")
|
| 41 |
+
if not all(bool(torch.isfinite(t).all()) for t in (logits, hidden, rate)):
|
| 42 |
+
raise FloatingPointError("non-finite full-batch TRUST input")
|
| 43 |
+
vocab, width = int(logits.shape[1]), int(hidden.shape[1])
|
| 44 |
+
if self.gs is None:
|
| 45 |
+
need = 2 * vocab * width * 4 + (2 * vocab * 4 if self.head_bias else 0)
|
| 46 |
+
if logits.is_cuda:
|
| 47 |
+
free, _ = torch.cuda.mem_get_info(logits.device)
|
| 48 |
+
if need > free * 0.5:
|
| 49 |
+
raise ValueError("full-batch TRUST head accumulators exceed memory guard")
|
| 50 |
+
self.gs = torch.zeros(vocab, width, dtype=torch.float32, device=logits.device)
|
| 51 |
+
self.gm = torch.zeros_like(self.gs)
|
| 52 |
+
if self.head_bias:
|
| 53 |
+
self.bs = torch.zeros(vocab, dtype=torch.float32, device=logits.device)
|
| 54 |
+
self.bm = torch.zeros_like(self.bs)
|
| 55 |
+
if self.gs.shape != (vocab, width):
|
| 56 |
+
raise ValueError("student head shape changed within TRUST batch")
|
| 57 |
+
strict = is_strict.to(device=logits.device)
|
| 58 |
+
q = rate.detach().float().repeat_interleave(lengths)
|
| 59 |
+
token_strict = strict.repeat_interleave(lengths)
|
| 60 |
+
qs = q * token_strict
|
| 61 |
+
qm = q * ~token_strict
|
| 62 |
+
hs = hidden.detach().float() * qs.unsqueeze(1)
|
| 63 |
+
hm = hidden.detach().float() * qm.unsqueeze(1)
|
| 64 |
+
logz = _log_partition_from_selected(logits, labels, selected_log_prob[:covered]).float()
|
| 65 |
+
autocast_guard = (torch.autocast(device_type=logits.device.type, enabled=False)
|
| 66 |
+
if self.force_fp32 else contextlib.nullcontext())
|
| 67 |
+
with _exact_fp32(), autocast_guard:
|
| 68 |
+
for low in range(0, vocab, self.vocab_chunk):
|
| 69 |
+
high = min(low + self.vocab_chunk, vocab)
|
| 70 |
+
z = logits[:, low:high].detach().float()
|
| 71 |
+
delta = torch.exp(z - logz.unsqueeze(1))
|
| 72 |
+
jacobian = None
|
| 73 |
+
if self.softcap is not None:
|
| 74 |
+
jacobian = (1.0 - (z / float(self.softcap)).clamp(-1, 1).square()).clamp_min(0)
|
| 75 |
+
delta *= jacobian
|
| 76 |
+
rows = ((labels >= low) & (labels < high)).nonzero().flatten()
|
| 77 |
+
cols = labels[rows] - low
|
| 78 |
+
delta[rows, cols] -= 1.0 if jacobian is None else jacobian[rows, cols]
|
| 79 |
+
self.gs[low:high].add_(delta.T @ hs)
|
| 80 |
+
self.gm[low:high].add_(delta.T @ hm)
|
| 81 |
+
if self.head_bias:
|
| 82 |
+
self.bs[low:high].add_(qs @ delta)
|
| 83 |
+
self.bm[low:high].add_(qm @ delta)
|
| 84 |
+
self.responses.append({
|
| 85 |
+
"strict_atoms": int(strict.sum()), "mismatch_atoms": int((~strict).sum()),
|
| 86 |
+
"strict_tokens": int(lengths[strict].sum()),
|
| 87 |
+
"mismatch_tokens": int(lengths[~strict].sum()),
|
| 88 |
+
})
|
| 89 |
+
|
| 90 |
+
@torch.no_grad()
|
| 91 |
+
def finalize(self, *, eps_g=1e-12, distributed=False,
|
| 92 |
+
calibrate=trust_calibrate_scalars):
|
| 93 |
+
if distributed:
|
| 94 |
+
import torch.distributed as dist
|
| 95 |
+
# Every rank uses the same pinned student head shape. Empty ranks
|
| 96 |
+
# allocate matching buffers after exchanging shapes, before collectives.
|
| 97 |
+
shapes = [None] * dist.get_world_size()
|
| 98 |
+
dist.all_gather_object(shapes, None if self.gs is None else tuple(self.gs.shape))
|
| 99 |
+
shape = next((s for s in shapes if s is not None), None)
|
| 100 |
+
if any(s is not None and s != shape for s in shapes):
|
| 101 |
+
raise ValueError("TRUST ranks have inconsistent student head shapes")
|
| 102 |
+
if shape is not None and self.gs is None:
|
| 103 |
+
dev = torch.device("cuda", torch.cuda.current_device()) if dist.get_backend() == "nccl" else torch.device("cpu")
|
| 104 |
+
self.gs = torch.zeros(shape, dtype=torch.float32, device=dev)
|
| 105 |
+
self.gm = torch.zeros_like(self.gs)
|
| 106 |
+
if self.head_bias:
|
| 107 |
+
self.bs = torch.zeros(shape[0], dtype=torch.float32, device=dev)
|
| 108 |
+
self.bm = torch.zeros_like(self.bs)
|
| 109 |
+
if shape is not None:
|
| 110 |
+
for tensor in (self.gs, self.gm, self.bs, self.bm):
|
| 111 |
+
if tensor is not None:
|
| 112 |
+
dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
|
| 113 |
+
records = [None] * dist.get_world_size()
|
| 114 |
+
dist.all_gather_object(records, self.responses)
|
| 115 |
+
self.responses = [r for group in records for r in group]
|
| 116 |
+
gs2 = gm2 = dot = 0.0
|
| 117 |
+
if self.gs is not None:
|
| 118 |
+
for low in range(0, self.gs.shape[0], self.vocab_chunk):
|
| 119 |
+
s = self.gs[low:low+self.vocab_chunk].double()
|
| 120 |
+
m = self.gm[low:low+self.vocab_chunk].double()
|
| 121 |
+
gs2 += float(s.square().sum())
|
| 122 |
+
gm2 += float(m.square().sum())
|
| 123 |
+
dot += float((s*m).sum())
|
| 124 |
+
if self.head_bias:
|
| 125 |
+
gs2 += float(self.bs.double().square().sum())
|
| 126 |
+
gm2 += float(self.bm.double().square().sum())
|
| 127 |
+
dot += float((self.bs.double()*self.bm.double()).sum())
|
| 128 |
+
n_s = sum(r["strict_atoms"] for r in self.responses)
|
| 129 |
+
n_m = sum(r["mismatch_atoms"] for r in self.responses)
|
| 130 |
+
result = calibrate(gs2, gm2, dot, n_s, n_m, eps_g=eps_g)
|
| 131 |
+
self.gs = self.gm = self.bs = self.bm = None
|
| 132 |
+
return result
|
tests/fixtures/trust/_mp_opd_trust_p.py.txt
ADDED
|
@@ -0,0 +1,104 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TRUST-P: project mismatch onto the strict non-conflict halfspace.
|
| 2 |
+
|
| 3 |
+
This is a new batch-level algorithm, not a correction to TRUST-R/B. Projection
|
| 4 |
+
is a known operation; efficacy or research novelty is not established here.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import math
|
| 9 |
+
from dataclasses import dataclass
|
| 10 |
+
|
| 11 |
+
import torch
|
| 12 |
+
|
| 13 |
+
from ._mp_opd_head_pair import checked_geometry
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
@dataclass(frozen=True)
|
| 17 |
+
class TrustProjectedResult:
|
| 18 |
+
strict_scale: float
|
| 19 |
+
gs_norm2: float
|
| 20 |
+
gm_norm2: float
|
| 21 |
+
dot: float
|
| 22 |
+
n_strict: int
|
| 23 |
+
n_mismatch: int
|
| 24 |
+
no_anchor: bool
|
| 25 |
+
projected: bool
|
| 26 |
+
mismatch_retained_norm_ratio: float
|
| 27 |
+
strict_descent: float
|
| 28 |
+
update_norm_ratio: float
|
| 29 |
+
update_cosine: float
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def trust_p_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *, eps_g=1e-12):
|
| 33 |
+
"""One decision for the entire optimizer batch, including cross-responses.
|
| 34 |
+
|
| 35 |
+
M_safe = M - min(0,<S,M>)/||S||^2 S; g = S+M_safe. Therefore
|
| 36 |
+
L = stopgrad(a) L_S + L_M with a=1-min(0,dot)/||S||^2.
|
| 37 |
+
With no usable S anchor, keep Atomic and explicitly waive the guarantee.
|
| 38 |
+
"""
|
| 39 |
+
if not (math.isfinite(float(eps_g)) and float(eps_g) >= 0):
|
| 40 |
+
raise ValueError("TRUST-P eps_g must be finite and nonnegative")
|
| 41 |
+
if n_s < 0 or n_m < 0:
|
| 42 |
+
raise ValueError("TRUST-P atom counts must be nonnegative")
|
| 43 |
+
pair = checked_geometry(gs2, gm2, dot)
|
| 44 |
+
s, m, d = pair.first_norm2, pair.second_norm2, pair.dot
|
| 45 |
+
no_anchor = n_s == 0 or s <= float(eps_g)
|
| 46 |
+
correction = -min(0., d)/s if not no_anchor else 0.
|
| 47 |
+
a = 1 + correction
|
| 48 |
+
if not math.isfinite(a) or a > torch.finfo(torch.float32).max:
|
| 49 |
+
raise FloatingPointError("TRUST-P strict scale is not representable in fp32")
|
| 50 |
+
# Loss coefficients are fp32. Round towards the protected halfspace, rather
|
| 51 |
+
# than letting nearest rounding reintroduce a tiny opposing component.
|
| 52 |
+
rounded = torch.tensor(a, dtype=torch.float32)
|
| 53 |
+
if float(rounded) < a:
|
| 54 |
+
rounded = torch.nextafter(rounded, torch.tensor(float("inf")))
|
| 55 |
+
a = float(rounded)
|
| 56 |
+
# Orthogonal component survives unchanged; only the opposing parallel
|
| 57 |
+
# component is removed. Projection cannot increase the M norm.
|
| 58 |
+
orthogonal2 = max(m - d*d/s, 0.) if correction > 0 else 0.
|
| 59 |
+
retained2 = orthogonal2 + (a-1+d/s)**2*s if correction > 0 else m
|
| 60 |
+
# Complete the square to avoid catastrophic cancellation for opposing arms.
|
| 61 |
+
post2 = (orthogonal2 + (a + d/s)**2*s) if correction > 0 else max(s + m + 2*d, 0.)
|
| 62 |
+
pre2 = max(s + m + 2*d, 0.)
|
| 63 |
+
cross = a*s + m + (a+1)*d
|
| 64 |
+
cosine = cross/math.sqrt(pre2*post2) if pre2 > 0 and post2 > 0 else (1. if pre2 == post2 == 0 else 0.)
|
| 65 |
+
scalars = (a, retained2, post2, pre2, cross)
|
| 66 |
+
if not all(math.isfinite(v) for v in scalars):
|
| 67 |
+
raise FloatingPointError("non-finite TRUST-P calibration")
|
| 68 |
+
return TrustProjectedResult(
|
| 69 |
+
a, s, m, d, int(n_s), int(n_m), no_anchor, correction > 0,
|
| 70 |
+
math.sqrt(retained2/m) if m > 0 else 0., a*s+d,
|
| 71 |
+
math.sqrt(post2/pre2) if pre2 > 0 else 0., max(-1., min(cosine, 1.)),
|
| 72 |
+
)
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def trust_p_response_loss(current_nll, rate, is_strict, is_mismatch, result):
|
| 76 |
+
if bool((is_strict & is_mismatch).any()) or not bool((is_strict | is_mismatch).all()):
|
| 77 |
+
raise ValueError("TRUST-P sides must be disjoint and cover all atoms")
|
| 78 |
+
coefficient = torch.where(is_strict, float(result.strict_scale), 1.).to(rate)
|
| 79 |
+
return (rate.detach() * coefficient * current_nll).sum()
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def trust_p_batch_metrics(result, counts):
|
| 83 |
+
r = result
|
| 84 |
+
out = {"batch_"+key: float(value) for key, value in counts.items()}
|
| 85 |
+
out.update({
|
| 86 |
+
"strict_scale": r.strict_scale,
|
| 87 |
+
"mismatch_scale": 1.,
|
| 88 |
+
"strict_head_norm2": r.gs_norm2,
|
| 89 |
+
"mismatch_head_norm2": r.gm_norm2,
|
| 90 |
+
"strict_mismatch_dot": r.dot,
|
| 91 |
+
"strict_mismatch_cosine": r.dot/math.sqrt(r.gs_norm2*r.gm_norm2) if r.gs_norm2 > 0 and r.gm_norm2 > 0 else 0.,
|
| 92 |
+
"projected": float(r.projected),
|
| 93 |
+
"no_anchor": float(r.no_anchor),
|
| 94 |
+
"batch_has_strict": float(r.n_strict > 0),
|
| 95 |
+
"batch_has_mismatch": float(r.n_mismatch > 0),
|
| 96 |
+
"mismatch_retained_norm_ratio": r.mismatch_retained_norm_ratio,
|
| 97 |
+
"strict_descent": r.strict_descent,
|
| 98 |
+
"strict_descent_ratio": r.strict_descent/r.gs_norm2 if r.gs_norm2 > 0 else 0.,
|
| 99 |
+
"update_norm_ratio_post_pre": r.update_norm_ratio,
|
| 100 |
+
"update_cosine_pre_post": r.update_cosine,
|
| 101 |
+
"update_ratio_defined": float(r.gs_norm2+r.gm_norm2+2*r.dot > 0),
|
| 102 |
+
"scope": 1.,
|
| 103 |
+
})
|
| 104 |
+
return out
|
tests/fixtures/trust/_trust_r_loss.py.txt
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
def _trust_r_loss(
|
| 2 |
+
self,
|
| 3 |
+
credits,
|
| 4 |
+
atoms,
|
| 5 |
+
*,
|
| 6 |
+
student_logits: torch.Tensor | None,
|
| 7 |
+
student_labels: torch.Tensor | None,
|
| 8 |
+
student_hidden: torch.Tensor | None,
|
| 9 |
+
):
|
| 10 |
+
"""TRUST-R: scale this response's mismatch update by strict agreement.
|
| 11 |
+
|
| 12 |
+
The calibration unit is the response, which is exactly what
|
| 13 |
+
``_partition_loss`` is called with, so no restructuring is needed: the
|
| 14 |
+
strict/mismatch split comes from the atomizer boundary types, the Gram
|
| 15 |
+
is the GRASS exact head Gram over all atoms, and the loss keeps the
|
| 16 |
+
strict side whole while the mismatch side takes the detached lambda.
|
| 17 |
+
"""
|
| 18 |
+
if student_logits is None or student_labels is None:
|
| 19 |
+
raise RuntimeError(
|
| 20 |
+
"mp_opd trust modes require the student logits and labels"
|
| 21 |
+
)
|
| 22 |
+
if student_hidden is None:
|
| 23 |
+
raise RuntimeError(
|
| 24 |
+
"mp_opd trust modes require the student LM-head hidden states"
|
| 25 |
+
)
|
| 26 |
+
if credits.student_token_nll is None:
|
| 27 |
+
raise RuntimeError(
|
| 28 |
+
"mp_opd trust modes require per-token student NLL"
|
| 29 |
+
)
|
| 30 |
+
if credits.student_token_nll.numel() != student_logits.shape[0]:
|
| 31 |
+
raise ValueError(
|
| 32 |
+
"per-token student NLL and student logits disagree on token count"
|
| 33 |
+
)
|
| 34 |
+
|
| 35 |
+
device = credits.rate.device
|
| 36 |
+
is_strict = strict_mask([atom.boundary_type for atom in atoms]).to(device)
|
| 37 |
+
n = len(atoms)
|
| 38 |
+
atom_ranges = tuple((atom.student_start, atom.student_end) for atom in atoms)
|
| 39 |
+
covered = atom_ranges[-1][1]
|
| 40 |
+
if not gram_affordable(covered, int(student_logits.shape[1]), device):
|
| 41 |
+
raise ValueError(
|
| 42 |
+
"mp_opd trust_r Gram needs ~"
|
| 43 |
+
f"{covered * int(student_logits.shape[1]) * 16 / 1e9:.1f} GiB transient; "
|
| 44 |
+
"device cannot hold it, atomic fallback"
|
| 45 |
+
)
|
| 46 |
+
head = chunk_head_gram(
|
| 47 |
+
student_logits[:covered],
|
| 48 |
+
student_hidden[:covered],
|
| 49 |
+
student_labels[:covered],
|
| 50 |
+
atom_ranges,
|
| 51 |
+
[(0, n)],
|
| 52 |
+
selected_log_prob=(-credits.student_token_nll[:covered]).detach(),
|
| 53 |
+
softcap=self.grass_softcap,
|
| 54 |
+
head_bias=self.grass_head_bias,
|
| 55 |
+
diagonal_only=False,
|
| 56 |
+
)
|
| 57 |
+
with torch.no_grad():
|
| 58 |
+
q = credits.rate.detach().to(torch.float32)
|
| 59 |
+
gram = head.gram.detach().to(torch.float32)
|
| 60 |
+
result = trust_calibrate(
|
| 61 |
+
q[is_strict],
|
| 62 |
+
q[~is_strict],
|
| 63 |
+
gram[is_strict][:, is_strict],
|
| 64 |
+
gram[~is_strict][:, ~is_strict],
|
| 65 |
+
gram[is_strict][:, ~is_strict],
|
| 66 |
+
eps_g=self.trust_eps_g,
|
| 67 |
+
)
|
| 68 |
+
loss = trust_response_loss(
|
| 69 |
+
credits.current_nll, credits.rate, is_strict, ~is_strict, result
|
| 70 |
+
)
|
| 71 |
+
token_counts = torch.tensor(
|
| 72 |
+
[atom.student_token_count for atom in atoms],
|
| 73 |
+
dtype=torch.float32,
|
| 74 |
+
device=device,
|
| 75 |
+
)
|
| 76 |
+
raw = trust_response_metrics(result, is_strict, token_counts, scope="response")
|
| 77 |
+
metrics = {
|
| 78 |
+
self._TRUST_RESPONSE_METRIC_NAMES[key]: credits.rate.new_tensor(value)
|
| 79 |
+
for key, value in raw.items()
|
| 80 |
+
}
|
| 81 |
+
metrics.update(self._trust_static_metrics(
|
| 82 |
+
credits.rate, head.symmetry_error, head.token_count
|
| 83 |
+
))
|
| 84 |
+
return loss, metrics
|
tests/test_single_b200.py
CHANGED
|
@@ -101,6 +101,20 @@ class LauncherTests(unittest.TestCase):
|
|
| 101 |
with self.assertRaisesRegex(ValueError, 'exact_head'):
|
| 102 |
self.plan('--algorithm','mp-align-new','--mp-geometry','diag')
|
| 103 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
def test_dry_run_creates_no_files(self):
|
| 105 |
with tempfile.TemporaryDirectory() as folder:
|
| 106 |
destination = pathlib.Path(folder) / 'absent'
|
|
|
|
| 101 |
with self.assertRaisesRegex(ValueError, 'exact_head'):
|
| 102 |
self.plan('--algorithm','mp-align-new','--mp-geometry','diag')
|
| 103 |
|
| 104 |
+
def test_trust_and_simct_paper_presets_keep_common_workload(self):
|
| 105 |
+
baseline=self.plan('--algorithm','strict-full')['flags']
|
| 106 |
+
for name,mode in [('simct_paper','simct_paper'),('mp-trust-r','mp_opd_trust_r'),
|
| 107 |
+
('mp-trust-b','mp_opd_trust_b'),('mp-trust-p','mp_opd_trust_p')]:
|
| 108 |
+
flags=self.plan('--algorithm',name)['flags']
|
| 109 |
+
self.assertEqual(flags['kd_algorithm'],mode)
|
| 110 |
+
for key in ('train_batch_size','rollout_batch_size','max_steps','generate_max_len',
|
| 111 |
+
'learning_rate','eval_steps','eval_math_n_samples','student_name_or_path','train_dataset_path'):
|
| 112 |
+
self.assertEqual(flags[key],baseline[key])
|
| 113 |
+
if name.startswith('mp-trust'):
|
| 114 |
+
self.assertEqual(flags['mp_opd_grass_geometry'],'exact_head')
|
| 115 |
+
with self.assertRaisesRegex(ValueError,'exact_head'):
|
| 116 |
+
self.plan('--algorithm',name,'--mp-geometry','diag')
|
| 117 |
+
|
| 118 |
def test_dry_run_creates_no_files(self):
|
| 119 |
with tempfile.TemporaryDirectory() as folder:
|
| 120 |
destination = pathlib.Path(folder) / 'absent'
|
tests/test_trust_paper.py
ADDED
|
@@ -0,0 +1,389 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TRUST donor parity, optimizer-window integration and SimCT paper-score checks."""
|
| 2 |
+
import ast
|
| 3 |
+
import copy
|
| 4 |
+
import hashlib
|
| 5 |
+
import importlib.machinery
|
| 6 |
+
import importlib.util
|
| 7 |
+
import json
|
| 8 |
+
import pathlib
|
| 9 |
+
import random
|
| 10 |
+
import sys
|
| 11 |
+
import types
|
| 12 |
+
import unittest
|
| 13 |
+
|
| 14 |
+
import test_mp_opd_port as fixtures
|
| 15 |
+
|
| 16 |
+
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
| 17 |
+
if fixtures.HAS_TORCH:
|
| 18 |
+
import numpy as np
|
| 19 |
+
import torch
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def original_trust():
|
| 23 |
+
from kdflow.algorithms._mp_opd_port import _mp_opd_grass_span as span
|
| 24 |
+
from kdflow.algorithms._mp_opd_port import _mp_opd_head_pair as pair
|
| 25 |
+
from kdflow.algorithms._mp_opd_port._mp_opd_grass_chunk import chunk_head_gram
|
| 26 |
+
package = types.ModuleType('_trust_original')
|
| 27 |
+
package.__path__ = []
|
| 28 |
+
sys.modules[package.__name__] = package
|
| 29 |
+
sys.modules['_trust_original._mp_opd_grass_span'] = span
|
| 30 |
+
sys.modules['_trust_original._mp_opd_head_pair'] = pair
|
| 31 |
+
modules = []
|
| 32 |
+
for name in ('_mp_opd_trust','_mp_opd_trust_batch','_mp_opd_trust_p'):
|
| 33 |
+
full = '_trust_original.'+name
|
| 34 |
+
loader = importlib.machinery.SourceFileLoader(full,str(ROOT/'tests/fixtures/trust'/(name+'.py.txt')))
|
| 35 |
+
spec = importlib.util.spec_from_loader(full,loader)
|
| 36 |
+
module = importlib.util.module_from_spec(spec)
|
| 37 |
+
sys.modules[full] = module
|
| 38 |
+
loader.exec_module(module)
|
| 39 |
+
modules.append(module)
|
| 40 |
+
scope = dict(torch=torch,chunk_head_gram=chunk_head_gram,
|
| 41 |
+
**{name:getattr(modules[0],name) for name in ('strict_mask','gram_affordable','trust_calibrate','trust_response_loss','trust_response_metrics')})
|
| 42 |
+
exec(compile((ROOT/'tests/fixtures/trust/_trust_r_loss.py.txt').read_text(),'donor_trust_r','exec'),scope)
|
| 43 |
+
return (*modules,scope['_trust_r_loss'])
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
@unittest.skipUnless(fixtures.HAS_TORCH,'requires existing PyTorch; no bootstrap')
|
| 47 |
+
class TrustTests(unittest.TestCase):
|
| 48 |
+
errors = []
|
| 49 |
+
|
| 50 |
+
@classmethod
|
| 51 |
+
def tearDownClass(cls):
|
| 52 |
+
if cls.errors:
|
| 53 |
+
print('TRUST_PARITY '+json.dumps({'cases':len(cls.errors),
|
| 54 |
+
'max_loss_abs_error':max(x[0] for x in cls.errors),
|
| 55 |
+
'max_gradient_abs_error':max(x[1] for x in cls.errors)}))
|
| 56 |
+
|
| 57 |
+
def fixture(self, mode='trust_b',cap=None):
|
| 58 |
+
from kdflow.algorithms import ALGO_DICT
|
| 59 |
+
helper = fixtures.MPPortTests()
|
| 60 |
+
mp,student,teacher = helper.fixture(cap=cap)
|
| 61 |
+
mp.kd.mp_opd_trust_eps_g = 1e-12
|
| 62 |
+
mp.kd.mp_opd_trust_vocab_chunk = 3
|
| 63 |
+
mp.kd.use_ema_teacher = False
|
| 64 |
+
mp.args.train.use_dynamic_bsz = False
|
| 65 |
+
mp.strategy.accumulated_gradient,mp.strategy.step = 2,0
|
| 66 |
+
algo = ALGO_DICT['mp_opd_'+mode](mp.strategy,student,teacher,fixtures.Tokenizer(),fixtures.Tokenizer())
|
| 67 |
+
return helper,algo,student,teacher
|
| 68 |
+
|
| 69 |
+
def rows(self,algo,student,teacher,batch):
|
| 70 |
+
from kdflow.algorithms._mp_opd_port._mp_opd_credit import build_atom_credits
|
| 71 |
+
sm,tm = batch['stu_loss_mask'],batch['tea_loss_mask']
|
| 72 |
+
hidden = student(batch['stu_input_ids'])['hidden_states'][-1][sm]
|
| 73 |
+
sy,ty = batch['stu_input_ids'].roll(-1,1),batch['tea_input_ids'].roll(-1,1)
|
| 74 |
+
so=to=0
|
| 75 |
+
for row in range(sm.shape[0]):
|
| 76 |
+
sid,tid = sy[row][sm[row]].tolist(),ty[row][tm[row]].tolist()
|
| 77 |
+
atomized = algo.atomizer.atomize(sid,tid,sample_id='donor')
|
| 78 |
+
self.assertTrue(atomized.valid)
|
| 79 |
+
ns,nt = atomized.covered_student_events,atomized.covered_teacher_events
|
| 80 |
+
sh = hidden[so:so+ns]
|
| 81 |
+
sl = student.model.lm_head(sh,skip=False)
|
| 82 |
+
tl = teacher(batch['teacher_hiddens'][to:to+nt]).detach()
|
| 83 |
+
if algo.student_cap is not None:
|
| 84 |
+
sl = torch.tanh(sl/algo.student_cap)*algo.student_cap
|
| 85 |
+
tl = torch.tanh(tl/algo.teacher_cap)*algo.teacher_cap
|
| 86 |
+
labels,tlabels = sy[row][sm[row]][:ns],ty[row][tm[row]][:nt]
|
| 87 |
+
credits = build_atom_credits(atomized.atoms,sl,labels,tl,tlabels)
|
| 88 |
+
yield credits,atomized.atoms,sl,labels,sh
|
| 89 |
+
so,to = so+len(sid),to+len(tid)
|
| 90 |
+
|
| 91 |
+
def reference(self,algo,student,teacher,batches):
|
| 92 |
+
trust,batch_module,p,r_loss = original_trust()
|
| 93 |
+
if algo.mode == 'trust_r':
|
| 94 |
+
class IdentityMap(dict):
|
| 95 |
+
def __missing__(self,key):
|
| 96 |
+
return key
|
| 97 |
+
ref = types.SimpleNamespace(grass_softcap=algo.student_cap,grass_head_bias=False,
|
| 98 |
+
trust_eps_g=algo.eps_g,_TRUST_RESPONSE_METRIC_NAMES=IdentityMap(),
|
| 99 |
+
_trust_static_metrics=lambda *args:{})
|
| 100 |
+
total = student.embedding.weight.sum()*0
|
| 101 |
+
for batch in batches:
|
| 102 |
+
for credits,atoms,logits,labels,hidden in self.rows(algo,student,teacher,batch):
|
| 103 |
+
loss,_ = r_loss(ref,credits,atoms,student_logits=logits,student_labels=labels,student_hidden=hidden)
|
| 104 |
+
total = total+loss/batch['avg_micro_batch_token_num']
|
| 105 |
+
return total,None
|
| 106 |
+
collector = batch_module.TrustHeadAccumulator(softcap=algo.student_cap,
|
| 107 |
+
head_bias=False,vocab_chunk=3,force_fp32=algo.mode=='trust_p')
|
| 108 |
+
with torch.no_grad():
|
| 109 |
+
for batch in batches:
|
| 110 |
+
for credits,atoms,logits,labels,hidden in self.rows(algo,student,teacher,batch):
|
| 111 |
+
mask = trust.strict_mask([a.boundary_type for a in atoms])
|
| 112 |
+
collector.add_response(logits,hidden,labels,[(a.student_start,a.student_end) for a in atoms],
|
| 113 |
+
credits.rate,mask,-credits.student_token_nll)
|
| 114 |
+
result = collector.finalize(eps_g=algo.eps_g,**({'calibrate':p.trust_p_calibrate_scalars} if algo.mode=='trust_p' else {}))
|
| 115 |
+
fn = p.trust_p_response_loss if algo.mode=='trust_p' else trust.trust_response_loss
|
| 116 |
+
total = student.embedding.weight.sum()*0
|
| 117 |
+
for batch in batches:
|
| 118 |
+
for credits,atoms,*_ in self.rows(algo,student,teacher,batch):
|
| 119 |
+
mask = trust.strict_mask([a.boundary_type for a in atoms])
|
| 120 |
+
total = total+fn(credits.current_nll,credits.rate,mask,~mask,result)/batch['avg_micro_batch_token_num']
|
| 121 |
+
return total,result
|
| 122 |
+
|
| 123 |
+
def test_full_loss_and_all_student_gradients_match_donor(self):
|
| 124 |
+
from kdflow.algorithms.mp_opd_trust import restore_rng
|
| 125 |
+
for mode in ('trust_r','trust_b','trust_p'):
|
| 126 |
+
for cap in (None,3.):
|
| 127 |
+
with self.subTest(mode=mode,cap=cap):
|
| 128 |
+
helper,algo,student,teacher = self.fixture(mode,cap)
|
| 129 |
+
batches = [helper.batch(),helper.batch()]
|
| 130 |
+
for b in batches:
|
| 131 |
+
b['avg_micro_batch_token_num'] *= 2
|
| 132 |
+
batches[1]['teacher_hiddens'] = (batches[1]['teacher_hiddens']*1.7).detach().requires_grad_()
|
| 133 |
+
rs,rt = copy.deepcopy(student),copy.deepcopy(teacher)
|
| 134 |
+
prepared = algo.prepare_optimizer_batch(batches) if algo.requires_optimizer_batch else None
|
| 135 |
+
if prepared:
|
| 136 |
+
parts=[]
|
| 137 |
+
for i,b in enumerate(batches):
|
| 138 |
+
restore_rng(prepared['rng_states'][i])
|
| 139 |
+
parts.append(algo.training_step(b)['train/loss'])
|
| 140 |
+
actual = sum(parts)
|
| 141 |
+
else:
|
| 142 |
+
actual = sum(algo.training_step(b)['train/loss'] for b in batches)
|
| 143 |
+
expected,result = self.reference(algo,rs,rt,batches)
|
| 144 |
+
ga = torch.autograd.grad(actual,tuple(student.parameters()))
|
| 145 |
+
gb = torch.autograd.grad(expected,tuple(rs.parameters()))
|
| 146 |
+
torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
|
| 147 |
+
for a,b in zip(ga,gb):
|
| 148 |
+
torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
|
| 149 |
+
if result is not None:
|
| 150 |
+
field = 'strict_scale' if mode=='trust_p' else 'lam'
|
| 151 |
+
self.assertAlmostEqual(getattr(algo._result,field),getattr(result,field),places=6)
|
| 152 |
+
self.errors.append((float((actual-expected).detach().abs()),max(float((a-b).abs().max()) for a,b in zip(ga,gb))))
|
| 153 |
+
teacher_grad = torch.autograd.grad(algo.training_step(batches[0])['train/loss'],
|
| 154 |
+
(*teacher.parameters(),batches[0]['teacher_hiddens']),allow_unused=True)
|
| 155 |
+
self.assertTrue(all(g is None for g in teacher_grad))
|
| 156 |
+
algo.finish_optimizer_batch()
|
| 157 |
+
|
| 158 |
+
def test_full_batch_cross_response_terms_and_projected_p_match_explicit_autograd(self):
|
| 159 |
+
from kdflow.algorithms._mp_opd_port._mp_opd_trust_batch import TrustHeadAccumulator
|
| 160 |
+
from kdflow.algorithms._mp_opd_port._mp_opd_trust_p import trust_p_calibrate_scalars
|
| 161 |
+
weight = torch.zeros(2,1,requires_grad=True)
|
| 162 |
+
rows = [torch.tensor([[1.],[2.]]),torch.tensor([[-.5],[-3.]])]
|
| 163 |
+
losses=[]
|
| 164 |
+
collector = TrustHeadAccumulator(force_fp32=True,vocab_chunk=1)
|
| 165 |
+
for hidden in rows:
|
| 166 |
+
logits = hidden@weight.T
|
| 167 |
+
labels = torch.ones(2,dtype=torch.long)
|
| 168 |
+
nll = -logits.log_softmax(-1)[:,1]
|
| 169 |
+
losses.append(nll)
|
| 170 |
+
collector.add_response(logits,hidden,labels,((0,1),(1,2)),torch.ones(2),
|
| 171 |
+
torch.tensor([True,False]),-nll.detach())
|
| 172 |
+
strict = sum(loss[0] for loss in losses)
|
| 173 |
+
mismatch = sum(loss[1] for loss in losses)
|
| 174 |
+
gs = torch.autograd.grad(strict,weight,retain_graph=True)[0]
|
| 175 |
+
gm = torch.autograd.grad(mismatch,weight,retain_graph=True)[0]
|
| 176 |
+
args = (float(gs.square().sum()),float(gm.square().sum()),float((gs*gm).sum()),2,2)
|
| 177 |
+
result = collector.finalize(calibrate=trust_p_calibrate_scalars)
|
| 178 |
+
original = original_trust()[2].trust_p_calibrate_scalars(*args)
|
| 179 |
+
self.assertAlmostEqual(result.strict_scale,3.,places=5)
|
| 180 |
+
self.assertEqual(result.strict_scale,original.strict_scale)
|
| 181 |
+
projected = result.strict_scale*strict+mismatch
|
| 182 |
+
gradient = torch.autograd.grad(projected,weight)[0]
|
| 183 |
+
torch.testing.assert_close(gradient,result.strict_scale*gs+gm)
|
| 184 |
+
self.assertGreaterEqual(float((gs*gradient).sum()),0.)
|
| 185 |
+
r = original_trust()[0].trust_calibrate_scalars(*args)
|
| 186 |
+
self.assertEqual(r.lam,0.)
|
| 187 |
+
# Per-response averaging would give (1/2+1/6)/2 = 1/3 instead.
|
| 188 |
+
self.assertNotAlmostEqual(r.lam,1/3)
|
| 189 |
+
|
| 190 |
+
def test_missing_window_wrong_boundary_and_normalizers_fail(self):
|
| 191 |
+
helper,algo,*_ = self.fixture()
|
| 192 |
+
with self.assertRaisesRegex(RuntimeError,'prepare_optimizer_batch'):
|
| 193 |
+
algo.training_step(helper.batch())
|
| 194 |
+
with self.assertRaisesRegex(ValueError,'complete optimizer'):
|
| 195 |
+
algo.prepare_optimizer_batch([helper.batch()])
|
| 196 |
+
batches=[helper.batch(),helper.batch()]
|
| 197 |
+
batches[1]['avg_micro_batch_token_num'] *= 2
|
| 198 |
+
with self.assertRaisesRegex(ValueError,'common normalizer'):
|
| 199 |
+
algo.prepare_optimizer_batch(batches)
|
| 200 |
+
self.assertIsNone(algo._result)
|
| 201 |
+
self.assertIsNone(algo._collector)
|
| 202 |
+
|
| 203 |
+
def test_actor_two_pass_rng_replay_and_one_optimizer_update(self):
|
| 204 |
+
from kdflow.algorithms.mp_opd_trust import capture_rng
|
| 205 |
+
# Execute the actual actor dispatch methods without importing Ray/CUDA.
|
| 206 |
+
tree=ast.parse((ROOT/'kdflow/ray/train/student_actor.py').read_text())
|
| 207 |
+
methods={}
|
| 208 |
+
for name in ('fit','evaluate'):
|
| 209 |
+
node=next(n for n in ast.walk(tree) if isinstance(n,ast.FunctionDef) and n.name==name)
|
| 210 |
+
node.decorator_list=[]
|
| 211 |
+
scope={}
|
| 212 |
+
exec(compile(ast.Module(body=[node],type_ignores=[]),'actual_actor_'+name,'exec'),scope)
|
| 213 |
+
methods[name]=scope[name]
|
| 214 |
+
helper,algo,student,teacher = self.fixture('trust_p')
|
| 215 |
+
records=[]
|
| 216 |
+
original_forward=student.forward
|
| 217 |
+
def forward(_self,ids,**kwargs):
|
| 218 |
+
random.random(); np.random.random()
|
| 219 |
+
hidden=original_forward(ids,**kwargs)['hidden_states'][-1]
|
| 220 |
+
hidden=torch.nn.functional.dropout(hidden,p=.2,training=student.training)
|
| 221 |
+
records.append(hidden.detach().clone())
|
| 222 |
+
return {'hidden_states':[hidden]}
|
| 223 |
+
student.forward=types.MethodType(forward,student)
|
| 224 |
+
optimizer=torch.optim.SGD(student.parameters(),lr=.01)
|
| 225 |
+
scheduler=types.SimpleNamespace(get_last_lr=lambda:[.01])
|
| 226 |
+
strategy=algo.strategy
|
| 227 |
+
updates=[]
|
| 228 |
+
def backward(loss,*_):
|
| 229 |
+
strategy.step=(strategy.step+1)%strategy.accumulated_gradient
|
| 230 |
+
(loss/strategy.accumulated_gradient).backward()
|
| 231 |
+
def step(*_):
|
| 232 |
+
if strategy.step==0:
|
| 233 |
+
optimizer.step();optimizer.zero_grad();updates.append(1)
|
| 234 |
+
strategy.backward=backward
|
| 235 |
+
strategy.optimizer_step=step
|
| 236 |
+
strategy.all_reduce=lambda value,**_:value
|
| 237 |
+
actor=types.SimpleNamespace(student=student,args=algo.args,strategy=strategy,kd_algorithm=algo,
|
| 238 |
+
optim=optimizer,scheduler=scheduler,_prepare_micro_batch=lambda batch:dict(batch))
|
| 239 |
+
batches=[helper.batch(),helper.batch()]
|
| 240 |
+
for b in batches:
|
| 241 |
+
b['avg_micro_batch_token_num'] *= 2
|
| 242 |
+
observed={}
|
| 243 |
+
original_prepare=algo.prepare_optimizer_batch
|
| 244 |
+
def prepare(*args,**kwargs):
|
| 245 |
+
result=original_prepare(*args,**kwargs)
|
| 246 |
+
observed['after']=result['after_rng']
|
| 247 |
+
self.assertEqual(updates,[])
|
| 248 |
+
return result
|
| 249 |
+
algo.prepare_optimizer_batch=prepare
|
| 250 |
+
status=methods['fit'](actor,batches)
|
| 251 |
+
self.assertTrue(np.isfinite(status['train/loss']))
|
| 252 |
+
self.assertEqual(len(updates),1)
|
| 253 |
+
self.assertEqual(len(records),4)
|
| 254 |
+
for a,b in zip(records[:2],records[2:]):
|
| 255 |
+
torch.testing.assert_close(a,b)
|
| 256 |
+
after=capture_rng()
|
| 257 |
+
self.assertEqual(after[0],observed['after'][0])
|
| 258 |
+
self.assertTrue(np.array_equal(after[1][1],observed['after'][1][1]))
|
| 259 |
+
self.assertTrue(torch.equal(after[2],observed['after'][2]))
|
| 260 |
+
self.assertIsNone(algo._result)
|
| 261 |
+
algo.prepare_optimizer_batch=original_prepare
|
| 262 |
+
before=[p.detach().clone() for p in student.parameters()]
|
| 263 |
+
methods['evaluate'](actor,batches)
|
| 264 |
+
self.assertTrue(student.training)
|
| 265 |
+
self.assertIsNone(algo._result)
|
| 266 |
+
for a,b in zip(before,student.parameters()):
|
| 267 |
+
torch.testing.assert_close(a,b)
|
| 268 |
+
|
| 269 |
+
def test_only_strict_and_only_mismatch_rules(self):
|
| 270 |
+
trust,_,p,_=original_trust()
|
| 271 |
+
from kdflow.algorithms._mp_opd_port._mp_opd_trust import trust_calibrate_scalars
|
| 272 |
+
from kdflow.algorithms._mp_opd_port._mp_opd_trust_p import trust_p_calibrate_scalars
|
| 273 |
+
for args in ((1.,0.,0.,2,0),(0.,1.,0.,0,2),(0.,0.,0.,0,0),(1.,4.,-2.,1,1)):
|
| 274 |
+
self.assertEqual(trust_calibrate_scalars(*args).lam,trust.trust_calibrate_scalars(*args).lam)
|
| 275 |
+
self.assertEqual(trust_p_calibrate_scalars(*args).strict_scale,p.trust_p_calibrate_scalars(*args).strict_scale)
|
| 276 |
+
|
| 277 |
+
def test_batch_microbatch_regrouping_keeps_loss_and_gradients(self):
|
| 278 |
+
for mode in ('trust_b','trust_p'):
|
| 279 |
+
helper,algo,student,_=self.fixture(mode)
|
| 280 |
+
full=helper.batch(2)
|
| 281 |
+
one=helper.batch()
|
| 282 |
+
one['avg_micro_batch_token_num']*=2
|
| 283 |
+
algo.strategy.accumulated_gradient=1
|
| 284 |
+
algo.prepare_optimizer_batch([full])
|
| 285 |
+
whole=algo.training_step(full)['train/loss']
|
| 286 |
+
gw=torch.autograd.grad(whole,tuple(student.parameters()))
|
| 287 |
+
algo.finish_optimizer_batch()
|
| 288 |
+
algo.strategy.accumulated_gradient=2
|
| 289 |
+
algo.prepare_optimizer_batch([one,one])
|
| 290 |
+
split=algo.training_step(one)['train/loss']+algo.training_step(one)['train/loss']
|
| 291 |
+
gs=torch.autograd.grad(split,tuple(student.parameters()))
|
| 292 |
+
algo.finish_optimizer_batch()
|
| 293 |
+
torch.testing.assert_close(whole,split,atol=1e-6,rtol=1e-5)
|
| 294 |
+
for a,b in zip(gw,gs):torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
|
| 295 |
+
|
| 296 |
+
def test_trust_and_paper_argument_guards(self):
|
| 297 |
+
from kdflow.arguments.distillation_args import DistillationArguments
|
| 298 |
+
for mode in ('mp_opd_trust_r','mp_opd_trust_b','mp_opd_trust_p','simct_paper'):
|
| 299 |
+
options=dict(kd_algorithm=mode,kd_ratio=1.,kd_loss_fn='rkl')
|
| 300 |
+
self.assertEqual(DistillationArguments(**options).kd_algorithm,mode)
|
| 301 |
+
with self.assertRaises(ValueError):
|
| 302 |
+
DistillationArguments(kd_algorithm=mode)
|
| 303 |
+
if mode.startswith('mp_opd_trust_'):
|
| 304 |
+
for extra in ({'mp_opd_grass_geometry':'diag'}, {'mp_opd_trust_eps_g':float('nan')},
|
| 305 |
+
{'mp_opd_trust_vocab_chunk':0}):
|
| 306 |
+
with self.assertRaises(ValueError):DistillationArguments(**options,**extra)
|
| 307 |
+
|
| 308 |
+
|
| 309 |
+
@unittest.skipUnless(fixtures.HAS_TORCH,'requires existing PyTorch; no bootstrap')
|
| 310 |
+
class SimCTPaperTests(unittest.TestCase):
|
| 311 |
+
def pair(self):
|
| 312 |
+
from kdflow.algorithms.simct_paper import SimCTPaper
|
| 313 |
+
helper=fixtures.MPPortTests()
|
| 314 |
+
mp,student,teacher=helper.fixture()
|
| 315 |
+
mp.kd.simct_gh_mask_threshold=0.
|
| 316 |
+
return helper,SimCTPaper(mp.strategy,student,teacher,fixtures.Tokenizer(),fixtures.Tokenizer()),student,teacher
|
| 317 |
+
|
| 318 |
+
@staticmethod
|
| 319 |
+
def manual(upstream,segments,sl,tl,sid,tid):
|
| 320 |
+
rows=[[],[]]
|
| 321 |
+
spans=[i for i,(ts,te,ss,se) in enumerate(segments) if te-ts>1 or se-ss>1]
|
| 322 |
+
for logits,ids,overlap,out in ((sl,sid,upstream.student_overlap_token_ids,rows[0]),
|
| 323 |
+
(tl,tid,upstream.teacher_overlap_token_ids,rows[1])):
|
| 324 |
+
logp=logits.float().log_softmax(-1)
|
| 325 |
+
student=out is rows[0]
|
| 326 |
+
for index,(ts,te,ss,se) in enumerate(segments):
|
| 327 |
+
lo,hi=(ss,se) if student else (ts,te)
|
| 328 |
+
extras=[logp[lo:hi].gather(1,torch.tensor(ids[lo:hi])[:,None]).mean()
|
| 329 |
+
if index==span else logp.new_tensor(-1e9) for span in spans]
|
| 330 |
+
out.append(torch.cat([logp[lo,overlap],torch.stack(extras)]) if spans else logp[lo,overlap])
|
| 331 |
+
return torch.stack(rows[0]),torch.stack(rows[1])
|
| 332 |
+
|
| 333 |
+
def test_virtual_scores_loss_gradients_and_logit_shift_invariance(self):
|
| 334 |
+
_,algo,*_=self.pair()
|
| 335 |
+
torch.manual_seed(7)
|
| 336 |
+
sl=torch.randn(3,8,requires_grad=True);tl=torch.randn(2,8)
|
| 337 |
+
segments=[(0,1,0,2),(1,2,2,3)]
|
| 338 |
+
sid,tid=[1,2,0],[4,0]
|
| 339 |
+
actual=algo.upstream._build_virtual_vocab_logits(segments,sl,tl,sid,tid)
|
| 340 |
+
expected=self.manual(algo.upstream,segments,sl,tl,sid,tid)
|
| 341 |
+
for a,b in zip(actual,expected):torch.testing.assert_close(a,b)
|
| 342 |
+
loss=algo.upstream.loss_fn(*actual,reduction='sum')
|
| 343 |
+
p,q=expected[0].log_softmax(-1),expected[1].log_softmax(-1)
|
| 344 |
+
ref=(p.exp()*(p-q)).sum()
|
| 345 |
+
ga=torch.autograd.grad(loss,sl,retain_graph=True)[0]
|
| 346 |
+
gb=torch.autograd.grad(ref,sl)[0]
|
| 347 |
+
torch.testing.assert_close(loss,ref)
|
| 348 |
+
torch.testing.assert_close(ga,gb)
|
| 349 |
+
shifted_sl=(sl.detach()+torch.tensor([0.,10.,-3.])[:,None]).requires_grad_()
|
| 350 |
+
shifted_tl=tl+torch.tensor([-8.,12.])[:,None]
|
| 351 |
+
shifted=algo.upstream._build_virtual_vocab_logits(segments,shifted_sl,shifted_tl,sid,tid)
|
| 352 |
+
shifted_loss=algo.upstream.loss_fn(*shifted,reduction='sum')
|
| 353 |
+
torch.testing.assert_close(loss,shifted_loss,atol=2e-6,rtol=1e-5)
|
| 354 |
+
torch.testing.assert_close(ga,torch.autograd.grad(shifted_loss,shifted_sl)[0],atol=2e-6,rtol=1e-5)
|
| 355 |
+
torch.testing.assert_close(ga.sum(-1),torch.zeros(3),atol=1e-6,rtol=1e-5)
|
| 356 |
+
|
| 357 |
+
def test_complete_adapter_loss_and_gradient_match_manual_equations(self):
|
| 358 |
+
helper,algo,student,teacher=self.pair()
|
| 359 |
+
rs,rt=copy.deepcopy(student),copy.deepcopy(teacher)
|
| 360 |
+
batch=helper.batch(2)
|
| 361 |
+
actual=algo.training_step(batch)['train/loss']
|
| 362 |
+
sm,tm=batch['stu_loss_mask'],batch['tea_loss_mask']
|
| 363 |
+
sh=rs(batch['stu_input_ids'])['hidden_states'][-1][sm]
|
| 364 |
+
th=batch['teacher_hiddens']
|
| 365 |
+
sy,ty=batch['stu_input_ids'].roll(-1,1),batch['tea_input_ids'].roll(-1,1)
|
| 366 |
+
so=to=0;expected=sh.sum()*0
|
| 367 |
+
for row in range(sm.shape[0]):
|
| 368 |
+
sid,tid=sy[row][sm[row]].tolist(),ty[row][tm[row]].tolist()
|
| 369 |
+
segments,tid,sid=algo.upstream._align_sequences_with_spans(tid,sid)
|
| 370 |
+
sl=rs.model.lm_head(sh[so:so+len(sid)],skip=False)
|
| 371 |
+
tl=rt(th[to:to+len(tid)]).detach()
|
| 372 |
+
sv,tv=self.manual(algo.upstream,segments,sl,tl,sid,tid)
|
| 373 |
+
p,q=sv.log_softmax(-1),tv.log_softmax(-1)
|
| 374 |
+
expected=expected+(p.exp()*(p-q)).sum()/batch['avg_micro_batch_token_num']
|
| 375 |
+
so,to=so+len(sid),to+len(tid)
|
| 376 |
+
torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
|
| 377 |
+
for a,b in zip(torch.autograd.grad(actual,tuple(student.parameters())),torch.autograd.grad(expected,tuple(rs.parameters()))):
|
| 378 |
+
torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
|
| 379 |
+
|
| 380 |
+
|
| 381 |
+
class TrustPaperSourceTests(unittest.TestCase):
|
| 382 |
+
def test_source_hashes_and_frozen_existing_modes(self):
|
| 383 |
+
manifest=json.loads((ROOT/'docs/TRUST_PAPER_SOURCE.json').read_text())
|
| 384 |
+
for row in manifest['files']+manifest['unchanged_existing_files']:
|
| 385 |
+
self.assertEqual(hashlib.sha256((ROOT/row['path']).read_bytes()).hexdigest(),row['sha256'])
|
| 386 |
+
|
| 387 |
+
|
| 388 |
+
if __name__=='__main__':
|
| 389 |
+
unittest.main()
|
tools/launch_single_b200.py
CHANGED
|
@@ -25,9 +25,10 @@ def parser():
|
|
| 25 |
default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
|
| 26 |
p.add_argument('--runtime', type=pathlib.Path,
|
| 27 |
help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
|
| 28 |
-
p.add_argument('--algorithm', choices=['strict-top16', 'strict-full', 'simct', 'mp-atomic',
|
| 29 |
'mp-fixed1', 'mp-fixed2', 'mp-fixed3', 'mp-fixed4', 'mp-fixed5',
|
| 30 |
-
'mp-random1-2', 'mp-random2-5', 'mp-grass-chunk', 'mp-align', 'mp-align-new'
|
|
|
|
| 31 |
p.add_argument('--mp-chunk-source', choices=['xtoken', 'run'], default='xtoken')
|
| 32 |
p.add_argument('--mp-chunk-run-length', type=int, default=2)
|
| 33 |
p.add_argument('--mp-geometry', choices=['exact_head', 'diag'], default='exact_head')
|
|
@@ -42,8 +43,8 @@ def parser():
|
|
| 42 |
|
| 43 |
|
| 44 |
def build_plan(args):
|
| 45 |
-
if args.algorithm == 'mp-align-new' and args.mp_geometry != 'exact_head':
|
| 46 |
-
raise ValueError('ALIGN-NEW requires exact_head geometry')
|
| 47 |
if args.concurrency < 1 or args.eval_timeout < 0 or args.mp_chunk_run_length < 1:
|
| 48 |
raise ValueError('Concurrency must be positive and eval timeout nonnegative')
|
| 49 |
run_id = args.run_id or (datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%dT%H%M%S%fZ')
|
|
@@ -82,8 +83,8 @@ def build_plan(args):
|
|
| 82 |
'teacher_mem_fraction_static': 0.6, 'teacher_offload_tags': 'all',
|
| 83 |
'chunked_loss_size': 2048, 'use_wandb': False,
|
| 84 |
}
|
| 85 |
-
if args.algorithm
|
| 86 |
-
flags.update(kd_algorithm=
|
| 87 |
if args.algorithm.startswith('mp-'):
|
| 88 |
name = args.algorithm[3:]
|
| 89 |
family = 'fixed' if name.startswith('fixed') else 'random' if name.startswith('random') else name.replace('-', '_')
|
|
@@ -100,6 +101,8 @@ def build_plan(args):
|
|
| 100 |
mp_opd_grass_geometry=args.mp_geometry)
|
| 101 |
if family == 'align_new':
|
| 102 |
flags.update(mp_opd_align_new_min_descent_fraction=0.1, mp_opd_align_new_eps_g=1e-12)
|
|
|
|
|
|
|
| 103 |
command = ['bash', str(runtime / 'python-b200.sh'), '-u', '-m', 'kdflow.cli.train_kd_on_policy']
|
| 104 |
for key, value in flags.items():
|
| 105 |
command.extend(['--' + key, str(value)])
|
|
@@ -107,7 +110,9 @@ def build_plan(args):
|
|
| 107 |
'source': str(SOURCE), 'flags': flags, 'command': command,
|
| 108 |
'gradient_accumulation_microsteps': 512 // args.micro_batch,
|
| 109 |
'expected_runtime_archive_sha256': ARCHIVE_SHA,
|
| 110 |
-
'qualification': ('
|
|
|
|
|
|
|
| 111 |
if args.algorithm == 'simct' else
|
| 112 |
'experimental MP-OPD source port; no B200/full-training qualification'
|
| 113 |
if args.algorithm.startswith('mp-') else
|
|
|
|
| 25 |
default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
|
| 26 |
p.add_argument('--runtime', type=pathlib.Path,
|
| 27 |
help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
|
| 28 |
+
p.add_argument('--algorithm', choices=['strict-top16', 'strict-full', 'simct', 'simct_paper', 'mp-atomic',
|
| 29 |
'mp-fixed1', 'mp-fixed2', 'mp-fixed3', 'mp-fixed4', 'mp-fixed5',
|
| 30 |
+
'mp-random1-2', 'mp-random2-5', 'mp-grass-chunk', 'mp-align', 'mp-align-new',
|
| 31 |
+
'mp-trust-r','mp-trust-b','mp-trust-p'], default='strict-top16')
|
| 32 |
p.add_argument('--mp-chunk-source', choices=['xtoken', 'run'], default='xtoken')
|
| 33 |
p.add_argument('--mp-chunk-run-length', type=int, default=2)
|
| 34 |
p.add_argument('--mp-geometry', choices=['exact_head', 'diag'], default='exact_head')
|
|
|
|
| 43 |
|
| 44 |
|
| 45 |
def build_plan(args):
|
| 46 |
+
if (args.algorithm == 'mp-align-new' or args.algorithm.startswith('mp-trust-')) and args.mp_geometry != 'exact_head':
|
| 47 |
+
raise ValueError('ALIGN-NEW/TRUST requires exact_head geometry')
|
| 48 |
if args.concurrency < 1 or args.eval_timeout < 0 or args.mp_chunk_run_length < 1:
|
| 49 |
raise ValueError('Concurrency must be positive and eval timeout nonnegative')
|
| 50 |
run_id = args.run_id or (datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%dT%H%M%S%fZ')
|
|
|
|
| 83 |
'teacher_mem_fraction_static': 0.6, 'teacher_offload_tags': 'all',
|
| 84 |
'chunked_loss_size': 2048, 'use_wandb': False,
|
| 85 |
}
|
| 86 |
+
if args.algorithm in {'simct','simct_paper'}:
|
| 87 |
+
flags.update(kd_algorithm=args.algorithm, simct_gh_mask_threshold=0.0)
|
| 88 |
if args.algorithm.startswith('mp-'):
|
| 89 |
name = args.algorithm[3:]
|
| 90 |
family = 'fixed' if name.startswith('fixed') else 'random' if name.startswith('random') else name.replace('-', '_')
|
|
|
|
| 101 |
mp_opd_grass_geometry=args.mp_geometry)
|
| 102 |
if family == 'align_new':
|
| 103 |
flags.update(mp_opd_align_new_min_descent_fraction=0.1, mp_opd_align_new_eps_g=1e-12)
|
| 104 |
+
if family.startswith('trust_'):
|
| 105 |
+
flags.update(mp_opd_grass_geometry='exact_head',mp_opd_trust_eps_g=1e-12,mp_opd_trust_vocab_chunk=4096)
|
| 106 |
command = ['bash', str(runtime / 'python-b200.sh'), '-u', '-m', 'kdflow.cli.train_kd_on_policy']
|
| 107 |
for key, value in flags.items():
|
| 108 |
command.extend(['--' + key, str(value)])
|
|
|
|
| 110 |
'source': str(SOURCE), 'flags': flags, 'command': command,
|
| 111 |
'gradient_accumulation_microsteps': 512 // args.micro_batch,
|
| 112 |
'expected_runtime_archive_sha256': ARCHIVE_SHA,
|
| 113 |
+
'qualification': ('paper-score SimCT reconstruction; no B200/full-training qualification'
|
| 114 |
+
if args.algorithm == 'simct_paper' else
|
| 115 |
+
'author-code SimCT port; no B200/full-training qualification'
|
| 116 |
if args.algorithm == 'simct' else
|
| 117 |
'experimental MP-OPD source port; no B200/full-training qualification'
|
| 118 |
if args.algorithm.startswith('mp-') else
|