codemaivanngu commited on
Commit
619cff6
·
unverified ·
1 Parent(s): 06ba1d4

feat(kd): add TRUST R B P and separate SimCT paper scores

Browse files
README.md CHANGED
@@ -24,6 +24,11 @@ A separate [author-code SimCT baseline](docs/SIMCT.md) uses `--algorithm simct`
24
  in the same launcher. It preserves the pinned released implementation, with
25
  original-source loss/gradient parity tests and an explicit paper/code distinction.
26
 
 
 
 
 
 
27
  ## Original project documentation
28
 
29
  This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
 
24
  in the same launcher. It preserves the pinned released implementation, with
25
  original-source loss/gradient parity tests and an explicit paper/code distinction.
26
 
27
+ Separate [TRUST-R/B/P and SimCT paper-score modes](docs/TRUST_PAPER.md) provide
28
+ `mp-trust-r`, `mp-trust-b`, `mp-trust-p` and `simct_paper`. B/P calibrate the complete
29
+ optimizer window; `simct_paper` corrects log-probability scores while `simct` stays
30
+ the released author-code baseline.
31
+
32
  ## Original project documentation
33
 
34
  This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
docs/MP_OPD.md CHANGED
@@ -18,6 +18,13 @@ the CTOPD launcher. The initial adapter requires one student rank, ring size 1,
18
  | `mp-grass-chunk` | `mp_opd_grass_chunk` | Risk-adaptive shrinkage within supplied chunks; no DP |
19
  | `mp-align` | `mp_opd_align` | Donor ALIGN v1 cyclic coefficient-space PCGrad within the same chunks |
20
  | `mp-align-new` | `mp_opd_align_new` | Local ALIGN plus full-response head-descent guard; see [ALIGN_NEW.md](ALIGN_NEW.md) |
 
 
 
 
 
 
 
21
 
22
  Fixed/random groups tile each response independently. The final tail may be
23
  shorter than the requested length/minimum. Randomness uses the configured seed
 
18
  | `mp-grass-chunk` | `mp_opd_grass_chunk` | Risk-adaptive shrinkage within supplied chunks; no DP |
19
  | `mp-align` | `mp_opd_align` | Donor ALIGN v1 cyclic coefficient-space PCGrad within the same chunks |
20
  | `mp-align-new` | `mp_opd_align_new` | Local ALIGN plus full-response head-descent guard; see [ALIGN_NEW.md](ALIGN_NEW.md) |
21
+ | `mp-trust-r` | `mp_opd_trust_r` | Strict plus mismatch scaled once per response |
22
+ | `mp-trust-b` | `mp_opd_trust_b` | Strict plus mismatch scaled once per complete optimizer batch |
23
+ | `mp-trust-p` | `mp_opd_trust_p` | Project opposing mismatch component through a complete-batch strict scale |
24
+
25
+ See [TRUST_PAPER.md](TRUST_PAPER.md) for the two-pass B/P actor contract and the
26
+ separate `simct_paper` score reconstruction. TRUST uses strict/mismatch atom types,
27
+ not the `xtoken` chunk partition used by ALIGN/GRASS.
28
 
29
  Fixed/random groups tile each response independently. The final tail may be
30
  shorter than the requested length/minimum. Randomness uses the configured seed
docs/SIMCT.md CHANGED
@@ -3,6 +3,10 @@
3
  Select `simct` to run the author's released `span_ctkd` objective. This is separate
4
  from Strict and the MP-OPD modes: `mp-atomic`/`mp-fixed1` are not SimCT.
5
 
 
 
 
 
6
  ## Source and exact scope
7
 
8
  The source is [sunjie279/SimCT-](https://github.com/sunjie279/SimCT-/tree/cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e),
 
3
  Select `simct` to run the author's released `span_ctkd` objective. This is separate
4
  from Strict and the MP-OPD modes: `mp-atomic`/`mp-fixed1` are not SimCT.
5
 
6
+ The separate [paper-score reconstruction](TRUST_PAPER.md), `simct_paper`, corrects
7
+ Eq.7-8 score normalization on this same candidate/alignment interface. The author
8
+ baseline described below remains unchanged.
9
+
10
  ## Source and exact scope
11
 
12
  The source is [sunjie279/SimCT-](https://github.com/sunjie279/SimCT-/tree/cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e),
docs/SINGLE_B200.md CHANGED
@@ -8,6 +8,8 @@ The separate [SimCT author-code preset](SIMCT.md), `--algorithm simct`, uses the
8
 
9
  The separate [ALIGN-NEW preset](ALIGN_NEW.md), `--algorithm mp-align-new`, uses the same workload with the donor's full-response head guard. It requires exact-head geometry and adds no GPU/full-training qualification.
10
 
 
 
11
  ## Preserved training and evaluation settings
12
 
13
  | Setting | Value |
 
8
 
9
  The separate [ALIGN-NEW preset](ALIGN_NEW.md), `--algorithm mp-align-new`, uses the same workload with the donor's full-response head guard. It requires exact-head geometry and adds no GPU/full-training qualification.
10
 
11
+ The [TRUST and paper-score presets](TRUST_PAPER.md) add `mp-trust-r/b/p` (separate choices) and `simct_paper` on the same workload. TRUST-B/P require complete-window two-pass calibration; `simct_paper` retains the released candidate/alignment interface while correcting score normalization.
12
+
13
  ## Preserved training and evaluation settings
14
 
15
  | Setting | Value |
docs/TRUST_PAPER.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # TRUST-R/B/P and SimCT paper-score modes
2
+
3
+ These opt-in source modes retain the existing CTOPD data/model/optimizer/evaluation
4
+ pipeline. They add no SFT warmup and do not reproduce either paper's full recipe.
5
+ Existing Strict, SimCT author-code, Atomic/Fixed/Random, GRASS and ALIGN files stay
6
+ unchanged. Shared actor branches activate only for the new TRUST-B/P modes.
7
+
8
+ | Launcher algorithm | Registered mode | Loss intervention |
9
+ | --- | --- | --- |
10
+ | `mp-trust-r` | `mp_opd_trust_r` | One lambda per response: strict + lambda*mismatch |
11
+ | `mp-trust-b` | `mp_opd_trust_b` | One lambda for the complete optimizer batch |
12
+ | `mp-trust-p` | `mp_opd_trust_p` | One strict-side projection scale for the complete optimizer batch; mismatch kept |
13
+ | `simct` | `simct` | Unchanged released author-code raw-logit span scores |
14
+ | `simct_paper` | `simct_paper` | Full-native-vocabulary log-probability scores before virtual-vocabulary normalization |
15
+
16
+ ## TRUST provenance and scope
17
+
18
+ The numerical kernels are copied byte for byte from
19
+ `sontungkieu/SimCT@529ee4b691732e1040d6a45f05fd12f58def96ec`:
20
+ `_mp_opd_trust.py`, `_mp_opd_trust_batch.py`, `_mp_opd_trust_p.py`. Existing frozen
21
+ head-pair and Gram dependencies are reused. [TRUST_PAPER_SOURCE.json](TRUST_PAPER_SOURCE.json)
22
+ records original fixtures and unchanged existing algorithm hashes. The donor MIT
23
+ notice remains in the vendored package.
24
+
25
+ Strict atoms are exactly `one_to_one` from byte-synchronized atomization; mismatch
26
+ atoms are `multi_token`. Unknown types fail. There is no extra `xtoken` chunking
27
+ for TRUST. Credits, detached rates, summed atom NLL, EOS masking and common token
28
+ normalizer are those of the existing MP-OPD adapter.
29
+ Here "strict loss" is the atomic path-credit surrogate on `one_to_one` atoms;
30
+ it is not the enumerated shared-vocabulary KL of the separate Strict presets.
31
+
32
+ Let S and M be the summed output-head gradients of strict/mismatch atom losses.
33
+ TRUST-R and B use `lambda=max(0,<S,M>/||M||^2)` with donor rules for empty or
34
+ degenerate sides; strict loss keeps coefficient 1. R uses one response, B the
35
+ entire optimizer window. P uses
36
+ `strict_scale=1-min(0,<S,M>)/||S||^2`, keeping mismatch coefficient 1; its donor
37
+ rounding protects the halfspace. With no usable strict anchor, P retains Atomic
38
+ and flags the waived guarantee. These are head-output-edge proxies, including
39
+ bias/softcap, not full-Transformer or Adam descent guarantees.
40
+
41
+ ### Complete-window calibration
42
+
43
+ B/P cannot silently calibrate each microbatch. A first train-mode no-grad pass
44
+ streams two full-head accumulators across every response, including all
45
+ cross-response terms. Parameters stay fixed. The second pass restores each
46
+ microbatch's Python/NumPy/Torch/CUDA RNG before backward, applies one detached
47
+ calibration to the whole optimizer window and advances logical RNG only once.
48
+ Only the normal final accumulated step updates parameters. Calibration state is
49
+ cleared afterward, including on failures; no graphs or batch logits are stashed.
50
+
51
+ One student rank/ring size 1, a complete optimizer window starting at step 0,
52
+ and one finite positive token normalizer are required. Partial windows, missing
53
+ preparation, changed normalizers and invalid geometry fail explicitly. No legacy
54
+ microbatch TRUST-B alias is provided. The donor's historical error-to-Atomic
55
+ fallback is not reproduced: invalid geometry/memory checks fail with the cause.
56
+ This difference affects failure handling, not the valid-input loss parity claim.
57
+
58
+ Evaluation loss uses its own evaluation-set calibration, unit sample weights,
59
+ no backward/optimizer update and no training-calibration reuse. The original
60
+ student train/eval mode is restored. MATH500 generation evaluation is unchanged.
61
+
62
+ TRUST requires unit RKL weight/temperature and exact-head geometry.
63
+ `mp_opd_trust_eps_g=1e-12`; `mp_opd_trust_vocab_chunk=4096`. B/P require two
64
+ FP32 [vocabulary, hidden] accumulators (plus bias when present), so full-size
65
+ memory and throughput still need GPU qualification. No runtime rebuild is needed.
66
+
67
+ ## What `simct_paper` means
68
+
69
+ This mode corrects the score calculation to
70
+ [SimCT v1 equations 7-8](https://arxiv.org/html/2605.07711v1#S3.SS3): first compute
71
+ `log_softmax` over the full native vocabulary at every position; use the first
72
+ position's log-probabilities for shared candidates and mean selected
73
+ log-probabilities for each realized span; then normalize over the virtual candidate
74
+ set and calculate RKL. Both sides use the same score convention. Gradients flow
75
+ through the full-vocabulary student normalization; teacher scores are detached.
76
+
77
+ `simct` remains untouched. `simct_paper` is a paper-score reconstruction on the
78
+ released author's candidate/alignment interface, **not a claim that all paper
79
+ semantics are implemented**. It retains greedy decoded-token alignment, the
80
+ released realized-span candidate set, EOS/shared-vocabulary rules, optional G(h)
81
+ mask (default disabled), raw teacher-head convention, common normalizer and the
82
+ CTOPD initialization/data. It does not add arbitrary alternative-continuation
83
+ forward scoring at every prefix, fix all alignment limitations, restore SFT
84
+ warmup, or establish equivalence to Rethinking's unavailable `simct` implementation.
85
+ Those are separate scientific changes, not hidden inside a score correction.
86
+
87
+ Tests independently construct Eq.7-8 candidate scores and RKL, compare complete
88
+ adapter loss/all student gradients, and verify invariance to independent additive
89
+ logit shifts at each position. Matching the released raw-logit loss is expected
90
+ only in special cases; numerical parity is asserted against the equations here.
91
+
92
+ ## Commands and verification
93
+
94
+ From the company checkout, print each full plan without launching a GPU job:
95
+
96
+ ```bash
97
+ git pull --ff-only
98
+ python3 tools/launch_single_b200.py --algorithm mp-trust-r --runtime /opt/ctopd-b200-u22 --dry-run
99
+ python3 tools/launch_single_b200.py --algorithm mp-trust-b --runtime /opt/ctopd-b200-u22 --dry-run
100
+ python3 tools/launch_single_b200.py --algorithm mp-trust-p --runtime /opt/ctopd-b200-u22 --dry-run
101
+ python3 tools/launch_single_b200.py --algorithm simct_paper --runtime /opt/ctopd-b200-u22 --dry-run
102
+ ```
103
+
104
+ Only remove `--dry-run` with the actual allocated GPU, ready models/data and a new
105
+ run ID. The real batch512/generation8192/100-step workload remains unchanged.
106
+ Use an existing CPU Torch environment for tests; no dependency installation:
107
+
108
+ ```bash
109
+ TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v
110
+ ```
111
+
112
+ The TRUST oracle separately loads original donor kernels and the TRUST-R method
113
+ body; frozen atom/Gram/credit helpers are source-verified shared dependencies.
114
+ Tests cover full loss/all student gradients, complete-batch cross-response terms,
115
+ projection/no-anchor behavior, dropout RNG replay, a real optimizer update through
116
+ the actor integration helper, evaluation state isolation and unchanged older
117
+ modes. CPU fixtures do not qualify B200/FSDP, full-size memory, training resume,
118
+ accuracy or paper-score reproduction. No GPU job is launched by this port.
119
+
120
+ The [CPU receipt](TRUST_PAPER_VALIDATION.json) records 76 passing tests and six
121
+ TRUST donor comparisons: maximum absolute loss error `0.0`, maximum absolute
122
+ student-gradient error `5.960464477539063e-08`. Paper-score checks compare against
123
+ independent formulas, not against the author's different raw-logit objective.
docs/TRUST_PAPER_SOURCE.json ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "donor_repository": "https://github.com/sontungkieu/SimCT",
3
+ "donor_commit": "529ee4b691732e1040d6a45f05fd12f58def96ec",
4
+ "paper": "https://arxiv.org/html/2605.07711v1#S3.SS3",
5
+ "paper_mode_scope": "Eq7-8 log-probability score correction on the released alignment/candidate interface, not full paper recipe",
6
+ "files": [
7
+ {
8
+ "path": "kdflow/algorithms/_mp_opd_port/_mp_opd_trust.py",
9
+ "sha256": "df17bda4a46fdbf5ddac20b8741915c388d8d88110f7be47eca871ee57eb32a4"
10
+ },
11
+ {
12
+ "path": "kdflow/algorithms/_mp_opd_port/_mp_opd_trust_batch.py",
13
+ "sha256": "7d526663b2bf2ddf8d9def36565499a515ff43ddb5b1e66811c3d7672a9ed6f2"
14
+ },
15
+ {
16
+ "path": "kdflow/algorithms/_mp_opd_port/_mp_opd_trust_p.py",
17
+ "sha256": "428eade92186b2ca4198ce7b299f875fe9836ebf9d5877d71689677366ea0be4"
18
+ },
19
+ {
20
+ "path": "tests/fixtures/trust/_mp_opd_trust.py.txt",
21
+ "sha256": "df17bda4a46fdbf5ddac20b8741915c388d8d88110f7be47eca871ee57eb32a4"
22
+ },
23
+ {
24
+ "path": "tests/fixtures/trust/_mp_opd_trust_batch.py.txt",
25
+ "sha256": "7d526663b2bf2ddf8d9def36565499a515ff43ddb5b1e66811c3d7672a9ed6f2"
26
+ },
27
+ {
28
+ "path": "tests/fixtures/trust/_mp_opd_trust_p.py.txt",
29
+ "sha256": "428eade92186b2ca4198ce7b299f875fe9836ebf9d5877d71689677366ea0be4"
30
+ },
31
+ {
32
+ "path": "tests/fixtures/trust/_trust_r_loss.py.txt",
33
+ "sha256": "343f2609f69d88626738244413e4129797657d78b03dd956f435ea6a3e88230e"
34
+ }
35
+ ],
36
+ "unchanged_existing_files": [
37
+ {
38
+ "path": "kdflow/algorithms/mp_opd.py",
39
+ "sha256": "b2387d7f36502dea16c28f4391af4bc97a7e77f367d35dea2867c744e91277a6"
40
+ },
41
+ {
42
+ "path": "kdflow/algorithms/mp_opd_align_new.py",
43
+ "sha256": "8479b54a2560d0c3021b796c8766c8309f89b90390981d9d91ec9044a3d13eda"
44
+ },
45
+ {
46
+ "path": "kdflow/algorithms/simct.py",
47
+ "sha256": "d946926dd21fce6041e2fc3b2fd8d317d9dd8e845c5931783e1e6a2bcd70ca89"
48
+ },
49
+ {
50
+ "path": "kdflow/algorithms/simple_ctkd.py",
51
+ "sha256": "fdbd99c55aeac4d061527158bdfc82a93333a42bf347eceaa134d34f5ef05952"
52
+ },
53
+ {
54
+ "path": "kdflow/algorithms/byte_align_ctkd.py",
55
+ "sha256": "5c6647c6fb8064ffd5f7b1ed971168cd1f3392caee62e2d75916d41931828e18"
56
+ },
57
+ {
58
+ "path": "kdflow/algorithms/topk_byte_align_ctkd.py",
59
+ "sha256": "298925da4ce973b0f2917e5ba5ad2adbc4862d7c1a73515c5c86514ac0d22ddb"
60
+ },
61
+ {
62
+ "path": "kdflow/algorithms/_mp_opd_port/_mp_opd_head_pair.py",
63
+ "sha256": "be9163ccd7dbdd4e5a5228be7e06ce68f062a290709383c15b934658e0a83d58"
64
+ },
65
+ {
66
+ "path": "kdflow/algorithms/_mp_opd_port/_mp_opd_align.py",
67
+ "sha256": "c6c0286c74215a8d30888c15b2bb5820ab77f54a45afd832db99ca6eb81cf1e0"
68
+ }
69
+ ]
70
+ }
docs/TRUST_PAPER_VALIDATION.json ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "date": "2026-10-19",
3
+ "donor_commit": "529ee4b691732e1040d6a45f05fd12f58def96ec",
4
+ "platform": "WSL Linux, CPU only",
5
+ "python": "3.12.3",
6
+ "torch": "2.14.0+cpu",
7
+ "command": "TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v",
8
+ "full_suite": {"passed": 76, "skipped": 0, "elapsed_seconds": 77.342},
9
+ "trust_parity": {
10
+ "cases": 6,
11
+ "max_loss_abs_error": 0.0,
12
+ "max_gradient_abs_error": 5.960464477539063e-08,
13
+ "dtype": "float32",
14
+ "atol": 1e-6,
15
+ "rtol": 1e-5,
16
+ "coverage": "TRUST-R/B/P complete loss and all student gradients, with/without final softcap",
17
+ "oracle": "Original donor TRUST kernels in separate fixture namespace and original TRUST-R method body; frozen source-verified Gram/credit helpers shared"
18
+ },
19
+ "paper_score": {
20
+ "result": "pass",
21
+ "scope": "Eq7-8 score reconstruction on released alignment/candidate interface",
22
+ "checks": ["Independent candidate scores and RKL", "Complete adapter loss and all student gradients", "Per-position additive logit-shift loss/gradient invariance"],
23
+ "loss_gradient_atol": 1e-6,
24
+ "loss_gradient_rtol": 1e-5,
25
+ "shift_invariance_atol": 2e-6
26
+ },
27
+ "additional_checks": [
28
+ "Explicit-autograd full-batch cross-response geometry and TRUST-P opposing-component projection",
29
+ "Actual actor fit/evaluate dispatch with real CPU SGD: one optimizer update, train-mode dropout replay, Python/NumPy/Torch RNG advance once",
30
+ "Microbatch regrouping preserves full-window loss and gradients",
31
+ "Evaluation restores model mode and leaves weights/calibration state untouched",
32
+ "Common-normalizer/window guards and calibration cleanup on errors",
33
+ "No teacher parameter or hidden-state gradient",
34
+ "All existing Strict/SimCT/MP/ALIGN numerical regressions and frozen source hashes",
35
+ "Launcher preserves common model/data/batch/steps/evaluation settings",
36
+ "git diff --check"
37
+ ],
38
+ "not_qualified": [
39
+ "Target Torch 2.11 CUDA, B200/FSDP runtime and full-size memory/throughput",
40
+ "Full Transformer or Adam descent guarantees",
41
+ "Full training resume, accuracy or either paper's full recipe",
42
+ "Complete SimCT paper semantics beyond Eq7-8 score correction",
43
+ "Parity with unavailable Rethinking simct baseline revision"
44
+ ]
45
+ }
kdflow/algorithms/__init__.py CHANGED
@@ -14,6 +14,8 @@ from kdflow.algorithms.simple_ctkd import SimpleCrossTokenizerKD
14
  from kdflow.algorithms.byte_align_ctkd import ByteAlignCrossTokenizerKD
15
  from kdflow.algorithms.topk_byte_align_ctkd import TopKByteAlignCrossTokenizerKD
16
  from kdflow.algorithms.simct import SimCT
 
 
17
  from kdflow.algorithms.mp_opd_align_new import AlignNewMPOnPolicyKD
18
  from kdflow.algorithms.mp_opd import (
19
  AtomicMPOnPolicyKD, FixedMPOnPolicyKD, RandomMPOnPolicyKD,
 
14
  from kdflow.algorithms.byte_align_ctkd import ByteAlignCrossTokenizerKD
15
  from kdflow.algorithms.topk_byte_align_ctkd import TopKByteAlignCrossTokenizerKD
16
  from kdflow.algorithms.simct import SimCT
17
+ from kdflow.algorithms.simct_paper import SimCTPaper
18
+ from kdflow.algorithms.mp_opd_trust import TrustRMPOnPolicyKD, TrustBMPOnPolicyKD, TrustPMPOnPolicyKD
19
  from kdflow.algorithms.mp_opd_align_new import AlignNewMPOnPolicyKD
20
  from kdflow.algorithms.mp_opd import (
21
  AtomicMPOnPolicyKD, FixedMPOnPolicyKD, RandomMPOnPolicyKD,
kdflow/algorithms/_mp_opd_port/_mp_opd_trust.py ADDED
@@ -0,0 +1,348 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TRUST v1.0 - trusted reference update scaling for cross-tokenizer OPD.
2
+
3
+ Mismatch supervision may carry signal, but its direction and scale should be
4
+ calibrated against the more reliable strict/matched supervision. TRUST does not
5
+ touch relative credits inside mismatch spans: per calibration unit it scales the
6
+ whole mismatch update by ``lambda = max(0, <gS, gM> / ||gM||^2)`` and keeps the
7
+ strict update whole.
8
+
9
+ Two modes share this file: TRUST-R (one lambda per response) and TRUST-B (one
10
+ lambda per optimizer batch, with microbatch calibration an explicit legacy
11
+ ablation). The only intended difference between them is the
12
+ calibration aggregation level; the strict/mismatch partition, the proxy, and
13
+ the guards are identical.
14
+
15
+ The strict/mismatch partition comes from the atomizer, not from a threshold:
16
+ an atom whose student and teacher spans are both exactly one token over the
17
+ same bytes (``boundary_type == "one_to_one"``) is matched supervision, and an
18
+ atom that straddles a tokenization boundary (``multi_token``) is mismatch. Any
19
+ atom with another boundary type fails closed, because silently filing it under
20
+ either side would change what the calibration means.
21
+
22
+ The proxy is the GRASS LM-head geometry, reused unchanged: with per-atom
23
+ coefficients ``q_i = r_i`` (the OPD atom loss is a plain token-sum, so the
24
+ outer coefficient is 1 and the response weight is a plain sum) and token
25
+ reduction ``a_it = 1`` (``current_nll`` is a token sum over the atom), the
26
+ effective coefficients are exactly the rates the loss consumes. The Gram code
27
+ is shared, so a TRUST number and a GRASS number read the same geometry.
28
+
29
+ Deliberate v1 deviations from the spec text, all fail-safe rather than silent:
30
+
31
+ * Median/p10/p25/p75/p90 keys are not emitted. Metrics reduce per micro-batch
32
+ by sum-then-mean, which cannot express a percentile; emitting a mean under a
33
+ ``_median`` key would lie about the statistic. Means and fractions over the
34
+ micro-batch responses are exact as stated (production uses
35
+ micro_train_batch_size=4).
36
+ * ``mp_opd_trust_scope`` is a float code (0.0 = response, 1.0 = batch unit) because
37
+ the metric table only carries finite tensors.
38
+ ``mp_opd_trust_full_batch`` distinguishes the complete optimizer batch from
39
+ the legacy microbatch unit.
40
+ * Samples or units without both sides log 0.0 for undefined quantities (never
41
+ NaN: the trainer fails fast on non-finite metrics), with the eligibility
42
+ fractions (``has_both``/``zero_*``) as the interpretive key.
43
+ * TRUST requires ``exact_head`` geometry. The ``diag`` ablation zeroes every
44
+ cross-atom term, which would force every dot product to zero and every lambda
45
+ to zero: a mode that can never act. Failing closed beats a silent no-op.
46
+ """
47
+
48
+ from __future__ import annotations
49
+
50
+ from dataclasses import dataclass
51
+ from typing import Sequence
52
+
53
+ import torch
54
+
55
+ # Section 9 fixes this: a mismatch norm at or below it carries no usable
56
+ # direction, so the calibration that would divide by it is skipped.
57
+ TRUST_EPS_G = 1e-12
58
+ # Only used to keep norm denominators away from zero; it never decides whether
59
+ # a unit calibrates or what lambda it gets.
60
+ TRUST_EPS_NORM = 1e-12
61
+ # Transient multiplier for one exact-Gram build: the [tokens, vocab] window,
62
+ # its Jacobian, the probabilities, and the vocabulary product temporaries.
63
+ # A step whose estimate exceeds half the free device memory degrades to the
64
+ # atomic fallback (loudly counted) instead of attempting an allocation the
65
+ # device cannot hold. Killed a 171-step trust_b run at 11.73 GiB attempted on
66
+ # 6.34 GiB free; the fallback that would have saved it did not exist yet.
67
+ TRUST_GRAM_TRANSIENT_FACTOR = 4.0
68
+ TRUST_GRAM_FREE_FRACTION = 0.5
69
+
70
+
71
+ def gram_affordable(n_tokens: int, vocab: int, device: torch.device) -> bool:
72
+ """Whether one exact Gram build fits in the currently free device memory.
73
+
74
+ Estimates ``n_tokens * vocab`` fp32 entries times the transient factor and
75
+ requires it under half the free bytes. On a device without memory info
76
+ (CPU tests) everything is affordable; the production guard this provides
77
+ is against the allocation, not the math.
78
+ """
79
+ if n_tokens <= 0 or vocab <= 0:
80
+ raise ValueError("gram affordability needs positive token and vocab counts")
81
+ if device.type != "cuda" or not torch.cuda.is_available():
82
+ return True
83
+ free_bytes, _ = torch.cuda.mem_get_info(device)
84
+ need_bytes = float(n_tokens) * float(vocab) * 4.0 * TRUST_GRAM_TRANSIENT_FACTOR
85
+ return need_bytes <= float(free_bytes) * TRUST_GRAM_FREE_FRACTION
86
+
87
+
88
+ @dataclass(frozen=True)
89
+ class TrustUnitResult:
90
+ """One calibration unit's lambda plus everything its diagnostics derive from."""
91
+
92
+ lam: float
93
+ dot: float
94
+ gs_norm2: float
95
+ gm_norm2: float
96
+ cosine: float
97
+ norm_ratio: float
98
+ kappa: float
99
+ has_strict: bool
100
+ has_mismatch: bool
101
+ calibrated: bool
102
+ update_cosine: float
103
+ update_norm_ratio: float
104
+ n_strict: int
105
+ n_mismatch: int
106
+
107
+
108
+ def _require_finite(name: str, *tensors: torch.Tensor) -> None:
109
+ for tensor in tensors:
110
+ if not bool(torch.isfinite(tensor).all()):
111
+ raise ValueError(f"TRUST received a non-finite {name}")
112
+
113
+
114
+ def trust_calibrate(
115
+ qs: torch.Tensor,
116
+ qm: torch.Tensor,
117
+ gram_ss: torch.Tensor,
118
+ gram_mm: torch.Tensor,
119
+ gram_sm: torch.Tensor,
120
+ *,
121
+ eps_g: float = TRUST_EPS_G,
122
+ eps_norm: float = TRUST_EPS_NORM,
123
+ ) -> TrustUnitResult:
124
+ """Calibrate one strict/mismatch pair from its Gram blocks.
125
+
126
+ ``qs``/``qm`` are the effective strict/mismatch coefficients and
127
+ ``gram_ss``/``gram_mm``/``gram_sm`` the matching blocks of the exact
128
+ LM-head Gram. The geometry is FP32; quadratic reductions promote to FP64.
129
+ The returned lambda is a Python float, so it can never carry a graph.
130
+ """
131
+ _require_finite("coefficients", qs, qm)
132
+ _require_finite("Gram blocks", gram_ss, gram_mm, gram_sm)
133
+ if not (float(eps_g) >= 0.0):
134
+ raise ValueError(f"TRUST eps_g must be non-negative, got {eps_g!r}")
135
+ qs = qs.detach().to(torch.float64).reshape(-1)
136
+ qm = qm.detach().to(torch.float64).reshape(-1)
137
+ gram_ss, gram_mm, gram_sm = (g.detach().double() for g in (gram_ss, gram_mm, gram_sm))
138
+ n_s, n_m = int(qs.numel()), int(qm.numel())
139
+ if (n_s, n_m) != (int(gram_ss.shape[0]), int(gram_mm.shape[0])):
140
+ raise ValueError(
141
+ f"TRUST Gram blocks {tuple(gram_ss.shape)}/{tuple(gram_mm.shape)} "
142
+ f"do not match coefficient counts ({n_s}, {n_m})"
143
+ )
144
+ if tuple(gram_sm.shape) != (n_s, n_m):
145
+ raise ValueError(
146
+ f"TRUST cross Gram {tuple(gram_sm.shape)} does not match ({n_s}, {n_m})"
147
+ )
148
+
149
+ # Empty sides have no geometry to read: the scalars below are defined as
150
+ # zero rather than derived from a matmul over an empty axis.
151
+ gs2 = float(qs @ gram_ss @ qs) if n_s else 0.0
152
+ gm2 = float(qm @ gram_mm @ qm) if n_m else 0.0
153
+ dot = float(qs @ gram_sm @ qm) if (n_s and n_m) else 0.0
154
+ scales = [float(q.abs() @ g.abs() @ q.abs()) if q.numel() else 0.0
155
+ for q, g in ((qs, gram_ss), (qm, gram_mm))]
156
+ for name, norm, scale in zip(("strict", "mismatch"), (gs2, gm2), scales):
157
+ if norm < -1e-6 * max(scale, 1e-300):
158
+ raise ValueError(f"TRUST {name} Gram gives a materially negative squared norm")
159
+ gs2, gm2 = max(gs2, 0.), max(gm2, 0.)
160
+ bound = (gs2 * gm2)**.5
161
+ if abs(dot) > bound + 1e-6 * max((scales[0]*scales[1])**.5, 1e-300):
162
+ raise ValueError("TRUST Gram violates Cauchy-Schwarz")
163
+ dot = min(max(dot, -bound), bound)
164
+ return trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m,
165
+ eps_g=eps_g, eps_norm=eps_norm)
166
+
167
+
168
+ def trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *,
169
+ eps_g=TRUST_EPS_G, eps_norm=TRUST_EPS_NORM):
170
+ """The same calibration for explicit summed head updates over a full batch."""
171
+ if not (0 <= float(eps_g) < float("inf")):
172
+ raise ValueError("TRUST eps_g must be finite and nonnegative")
173
+ for name, value in (("gs_norm2", gs2), ("gm_norm2", gm2), ("dot", dot)):
174
+ if not abs(value) < float("inf"):
175
+ raise ValueError(f"TRUST {name} is non-finite")
176
+ if gs2 < 0 or gm2 < 0:
177
+ raise ValueError("TRUST squared norms must be nonnegative")
178
+ bound = (gs2 * gm2)**.5
179
+ if abs(dot) > bound * (1. + 1e-6) + 1e-300:
180
+ raise ValueError("TRUST summed updates violate Cauchy-Schwarz")
181
+ dot = min(max(dot, -bound), bound)
182
+
183
+ has_strict = n_s > 0
184
+ has_mismatch = n_m > 0
185
+ calibrated = bool(has_strict and has_mismatch and gm2 > float(eps_g))
186
+ lam = max(0.0, dot / gm2) if calibrated else 0.0
187
+
188
+ gs = gs2**0.5
189
+ gm = gm2**0.5
190
+ cosine = dot / (gs * gm + float(eps_norm))
191
+ norm_ratio = gm / (gs + float(eps_norm))
192
+ kappa = lam * gm / (gs + float(eps_norm))
193
+
194
+ # g_pre = gS + gM and g_post = gS + lam gM, expanded in Gram scalars so no
195
+ # vector is ever formed. When both updates vanish they are identical.
196
+ pre2 = gs2 + gm2 + 2.0 * dot
197
+ post2 = gs2 + lam * lam * gm2 + 2.0 * lam * dot
198
+ inner = gs2 + lam * gm2 + (1.0 + lam) * dot
199
+ pre2 = max(pre2, 0.0)
200
+ post2 = max(post2, 0.0)
201
+ if pre2 <= 0.0 and post2 <= 0.0:
202
+ update_cosine = 1.0
203
+ elif pre2 <= 0.0 or post2 <= 0.0:
204
+ update_cosine = 0.0
205
+ else:
206
+ update_cosine = inner / (pre2**0.5 * post2**0.5 + float(eps_norm))
207
+ update_norm_ratio = post2**0.5 / (pre2**0.5 + float(eps_norm))
208
+
209
+ return TrustUnitResult(
210
+ lam=lam,
211
+ dot=dot,
212
+ gs_norm2=gs2,
213
+ gm_norm2=gm2,
214
+ cosine=cosine,
215
+ norm_ratio=norm_ratio,
216
+ kappa=kappa,
217
+ has_strict=has_strict,
218
+ has_mismatch=has_mismatch,
219
+ calibrated=calibrated,
220
+ update_cosine=update_cosine,
221
+ update_norm_ratio=update_norm_ratio,
222
+ n_strict=n_s,
223
+ n_mismatch=n_m,
224
+ )
225
+
226
+
227
+ def strict_mask(boundary_types: Sequence[str]) -> torch.Tensor:
228
+ """Strict-side mask from atomizer boundary types, failing closed.
229
+
230
+ ``one_to_one`` is matched supervision, ``multi_token`` is mismatch. A
231
+ third kind would silently change the calibration under either choice, so
232
+ it raises instead of being filed anywhere.
233
+ """
234
+ mask = []
235
+ for kind in boundary_types:
236
+ if kind == "one_to_one":
237
+ mask.append(True)
238
+ elif kind == "multi_token":
239
+ mask.append(False)
240
+ else:
241
+ raise ValueError(f"TRUST has no side for boundary type {kind!r}")
242
+ return torch.tensor(mask, dtype=torch.bool)
243
+
244
+
245
+ def trust_response_loss(
246
+ current_nll: torch.Tensor,
247
+ rate: torch.Tensor,
248
+ is_strict: torch.Tensor,
249
+ is_mismatch: torch.Tensor,
250
+ result: TrustUnitResult,
251
+ ) -> torch.Tensor:
252
+ """``L_S + stopgrad(lambda) L_M`` with exactly the OPD loss coefficients.
253
+
254
+ Each side is a ``(rates.detach() * nll).sum()``, the same construction as
255
+ ``soft_partition_loss``: the strict side keeps its credits whole and the
256
+ mismatch side is scaled by the detached lambda. The lambda is a Python
257
+ float by construction, so no graph can pass through the calibration rule.
258
+ """
259
+ if not (bool(is_strict.any()) or bool(is_mismatch.any())):
260
+ raise ValueError("TRUST needs at least one strict or mismatch atom")
261
+ if bool((is_strict & is_mismatch).any()):
262
+ raise ValueError("TRUST strict and mismatch sets must be disjoint")
263
+ zero = current_nll.new_zeros(())
264
+ strict = (torch.where(is_strict, rate.detach(), zero) * current_nll).sum()
265
+ mismatch = (torch.where(is_mismatch, rate.detach(), zero) * current_nll).sum()
266
+ return strict + float(result.lam) * mismatch
267
+
268
+
269
+ def _counts(is_strict: torch.Tensor, token_counts: torch.Tensor) -> dict[str, float]:
270
+ """Raw per-unit coverage counts; the caller decides the aggregation."""
271
+ is_mismatch = ~is_strict
272
+ n_s = int(is_strict.sum())
273
+ n_m = int(is_mismatch.sum())
274
+ total = n_s + n_m
275
+ return {
276
+ "strict_span_count": float(n_s),
277
+ "mismatch_span_count": float(n_m),
278
+ # One atom is one supervision span here: there is no higher grouping
279
+ # for TRUST to aggregate, so the span and atom keys coincide by
280
+ # definition rather than by coincidence.
281
+ "strict_atom_count": float(n_s),
282
+ "mismatch_atom_count": float(n_m),
283
+ "strict_token_count": float(token_counts[is_strict].sum()),
284
+ "mismatch_token_count": float(token_counts[is_mismatch].sum()),
285
+ "strict_span_fraction": float(n_s / total) if total else 0.0,
286
+ "mismatch_span_fraction": float(n_m / total) if total else 0.0,
287
+ "has_strict": float(n_s > 0),
288
+ "has_mismatch": float(n_m > 0),
289
+ "has_both": float(n_s > 0 and n_m > 0),
290
+ "zero_strict": float(n_s == 0),
291
+ "zero_mismatch": float(n_m == 0),
292
+ }
293
+
294
+
295
+ def trust_response_metrics(
296
+ result: TrustUnitResult,
297
+ is_strict: torch.Tensor,
298
+ token_counts: torch.Tensor,
299
+ *,
300
+ scope: str,
301
+ ) -> dict[str, float]:
302
+ """Per-unit metric values; the training loop averages them over responses."""
303
+ if scope not in ("response", "batch"):
304
+ raise ValueError(f"TRUST scope must be response|batch, got {scope!r}")
305
+ out = _counts(is_strict, token_counts)
306
+ out.update(
307
+ {
308
+ "cosine": result.cosine,
309
+ "cosine_negative": float(result.cosine < 0.0),
310
+ "mismatch_to_strict_norm_ratio": result.norm_ratio,
311
+ "lambda": result.lam,
312
+ "lambda_zero": float(result.lam <= 0.0),
313
+ "lambda_gt1": float(result.lam > 1.0),
314
+ "calibrated": float(result.calibrated),
315
+ "calibrated_mismatch_norm_ratio": result.kappa,
316
+ "update_cosine_pre_post": result.update_cosine,
317
+ "update_norm_ratio_post_pre": result.update_norm_ratio,
318
+ "scope": 0.0 if scope == "response" else 1.0,
319
+ }
320
+ )
321
+ return out
322
+
323
+
324
+ def trust_batch_metrics(
325
+ results: TrustUnitResult,
326
+ counts: dict[str, float],
327
+ ) -> dict[str, float]:
328
+ """One calibration unit's values, logged without response averaging."""
329
+ return {
330
+ "batch_strict_span_count": counts["strict_span_count"],
331
+ "batch_mismatch_span_count": counts["mismatch_span_count"],
332
+ "batch_strict_atom_count": counts["strict_atom_count"],
333
+ "batch_mismatch_atom_count": counts["mismatch_atom_count"],
334
+ "batch_strict_token_count": counts["strict_token_count"],
335
+ "batch_mismatch_token_count": counts["mismatch_token_count"],
336
+ "batch_strict_span_fraction": counts["strict_span_fraction"],
337
+ "batch_mismatch_span_fraction": counts["mismatch_span_fraction"],
338
+ "cosine": results.cosine,
339
+ "mismatch_to_strict_norm_ratio": results.norm_ratio,
340
+ "lambda": results.lam,
341
+ "batch_has_strict": float(results.has_strict),
342
+ "batch_has_mismatch": float(results.has_mismatch),
343
+ "batch_calibrated": float(results.calibrated),
344
+ "calibrated_mismatch_norm_ratio": results.kappa,
345
+ "update_cosine_pre_post": results.update_cosine,
346
+ "update_norm_ratio_post_pre": results.update_norm_ratio,
347
+ "scope": 1.0,
348
+ }
kdflow/algorithms/_mp_opd_port/_mp_opd_trust_batch.py ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Exact full-batch TRUST head updates, streamed without retaining forward graphs."""
2
+ from __future__ import annotations
3
+
4
+ import contextlib
5
+ import torch
6
+
7
+ from ._mp_opd_grass_span import _exact_fp32, _log_partition_from_selected
8
+ from ._mp_opd_trust import trust_calibrate_scalars
9
+
10
+
11
+ class TrustHeadAccumulator:
12
+ """Sum strict/mismatch LM-head gradients, including cross-response terms.
13
+
14
+ Storage is two [vocab, hidden] fp32 tensors, independent of batch token count.
15
+ Vocabulary tiles bound transients; no [batch tokens, vocab] concatenation and
16
+ no atom-by-atom global Gram are constructed. Loss normalization is one common
17
+ factor for the entire accumulation window and cancels from lambda.
18
+ """
19
+
20
+ def __init__(self, *, softcap=None, head_bias=False, vocab_chunk=4096,
21
+ force_fp32=False):
22
+ self.softcap = softcap
23
+ self.head_bias = head_bias
24
+ self.vocab_chunk = int(vocab_chunk)
25
+ self.force_fp32 = bool(force_fp32)
26
+ if self.vocab_chunk < 1:
27
+ raise ValueError("TRUST vocabulary tile must be positive")
28
+ self.gs = self.gm = self.bs = self.bm = None
29
+ self.responses = []
30
+
31
+ @torch.no_grad()
32
+ def add_response(self, logits, hidden, labels, ranges, rate, is_strict,
33
+ selected_log_prob):
34
+ covered = ranges[-1][1]
35
+ logits, hidden, labels = logits[:covered], hidden[:covered], labels[:covered]
36
+ lengths = torch.tensor([b-a for a, b in ranges], device=logits.device)
37
+ if sum(b-a for a, b in ranges) != covered or ranges[0][0] != 0:
38
+ raise ValueError("TRUST ranges must tile the covered token prefix")
39
+ if any(ranges[i][1] != ranges[i+1][0] for i in range(len(ranges)-1)):
40
+ raise ValueError("TRUST ranges must have no gaps")
41
+ if not all(bool(torch.isfinite(t).all()) for t in (logits, hidden, rate)):
42
+ raise FloatingPointError("non-finite full-batch TRUST input")
43
+ vocab, width = int(logits.shape[1]), int(hidden.shape[1])
44
+ if self.gs is None:
45
+ need = 2 * vocab * width * 4 + (2 * vocab * 4 if self.head_bias else 0)
46
+ if logits.is_cuda:
47
+ free, _ = torch.cuda.mem_get_info(logits.device)
48
+ if need > free * 0.5:
49
+ raise ValueError("full-batch TRUST head accumulators exceed memory guard")
50
+ self.gs = torch.zeros(vocab, width, dtype=torch.float32, device=logits.device)
51
+ self.gm = torch.zeros_like(self.gs)
52
+ if self.head_bias:
53
+ self.bs = torch.zeros(vocab, dtype=torch.float32, device=logits.device)
54
+ self.bm = torch.zeros_like(self.bs)
55
+ if self.gs.shape != (vocab, width):
56
+ raise ValueError("student head shape changed within TRUST batch")
57
+ strict = is_strict.to(device=logits.device)
58
+ q = rate.detach().float().repeat_interleave(lengths)
59
+ token_strict = strict.repeat_interleave(lengths)
60
+ qs = q * token_strict
61
+ qm = q * ~token_strict
62
+ hs = hidden.detach().float() * qs.unsqueeze(1)
63
+ hm = hidden.detach().float() * qm.unsqueeze(1)
64
+ logz = _log_partition_from_selected(logits, labels, selected_log_prob[:covered]).float()
65
+ autocast_guard = (torch.autocast(device_type=logits.device.type, enabled=False)
66
+ if self.force_fp32 else contextlib.nullcontext())
67
+ with _exact_fp32(), autocast_guard:
68
+ for low in range(0, vocab, self.vocab_chunk):
69
+ high = min(low + self.vocab_chunk, vocab)
70
+ z = logits[:, low:high].detach().float()
71
+ delta = torch.exp(z - logz.unsqueeze(1))
72
+ jacobian = None
73
+ if self.softcap is not None:
74
+ jacobian = (1.0 - (z / float(self.softcap)).clamp(-1, 1).square()).clamp_min(0)
75
+ delta *= jacobian
76
+ rows = ((labels >= low) & (labels < high)).nonzero().flatten()
77
+ cols = labels[rows] - low
78
+ delta[rows, cols] -= 1.0 if jacobian is None else jacobian[rows, cols]
79
+ self.gs[low:high].add_(delta.T @ hs)
80
+ self.gm[low:high].add_(delta.T @ hm)
81
+ if self.head_bias:
82
+ self.bs[low:high].add_(qs @ delta)
83
+ self.bm[low:high].add_(qm @ delta)
84
+ self.responses.append({
85
+ "strict_atoms": int(strict.sum()), "mismatch_atoms": int((~strict).sum()),
86
+ "strict_tokens": int(lengths[strict].sum()),
87
+ "mismatch_tokens": int(lengths[~strict].sum()),
88
+ })
89
+
90
+ @torch.no_grad()
91
+ def finalize(self, *, eps_g=1e-12, distributed=False,
92
+ calibrate=trust_calibrate_scalars):
93
+ if distributed:
94
+ import torch.distributed as dist
95
+ # Every rank uses the same pinned student head shape. Empty ranks
96
+ # allocate matching buffers after exchanging shapes, before collectives.
97
+ shapes = [None] * dist.get_world_size()
98
+ dist.all_gather_object(shapes, None if self.gs is None else tuple(self.gs.shape))
99
+ shape = next((s for s in shapes if s is not None), None)
100
+ if any(s is not None and s != shape for s in shapes):
101
+ raise ValueError("TRUST ranks have inconsistent student head shapes")
102
+ if shape is not None and self.gs is None:
103
+ dev = torch.device("cuda", torch.cuda.current_device()) if dist.get_backend() == "nccl" else torch.device("cpu")
104
+ self.gs = torch.zeros(shape, dtype=torch.float32, device=dev)
105
+ self.gm = torch.zeros_like(self.gs)
106
+ if self.head_bias:
107
+ self.bs = torch.zeros(shape[0], dtype=torch.float32, device=dev)
108
+ self.bm = torch.zeros_like(self.bs)
109
+ if shape is not None:
110
+ for tensor in (self.gs, self.gm, self.bs, self.bm):
111
+ if tensor is not None:
112
+ dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
113
+ records = [None] * dist.get_world_size()
114
+ dist.all_gather_object(records, self.responses)
115
+ self.responses = [r for group in records for r in group]
116
+ gs2 = gm2 = dot = 0.0
117
+ if self.gs is not None:
118
+ for low in range(0, self.gs.shape[0], self.vocab_chunk):
119
+ s = self.gs[low:low+self.vocab_chunk].double()
120
+ m = self.gm[low:low+self.vocab_chunk].double()
121
+ gs2 += float(s.square().sum())
122
+ gm2 += float(m.square().sum())
123
+ dot += float((s*m).sum())
124
+ if self.head_bias:
125
+ gs2 += float(self.bs.double().square().sum())
126
+ gm2 += float(self.bm.double().square().sum())
127
+ dot += float((self.bs.double()*self.bm.double()).sum())
128
+ n_s = sum(r["strict_atoms"] for r in self.responses)
129
+ n_m = sum(r["mismatch_atoms"] for r in self.responses)
130
+ result = calibrate(gs2, gm2, dot, n_s, n_m, eps_g=eps_g)
131
+ self.gs = self.gm = self.bs = self.bm = None
132
+ return result
kdflow/algorithms/_mp_opd_port/_mp_opd_trust_p.py ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TRUST-P: project mismatch onto the strict non-conflict halfspace.
2
+
3
+ This is a new batch-level algorithm, not a correction to TRUST-R/B. Projection
4
+ is a known operation; efficacy or research novelty is not established here.
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import math
9
+ from dataclasses import dataclass
10
+
11
+ import torch
12
+
13
+ from ._mp_opd_head_pair import checked_geometry
14
+
15
+
16
+ @dataclass(frozen=True)
17
+ class TrustProjectedResult:
18
+ strict_scale: float
19
+ gs_norm2: float
20
+ gm_norm2: float
21
+ dot: float
22
+ n_strict: int
23
+ n_mismatch: int
24
+ no_anchor: bool
25
+ projected: bool
26
+ mismatch_retained_norm_ratio: float
27
+ strict_descent: float
28
+ update_norm_ratio: float
29
+ update_cosine: float
30
+
31
+
32
+ def trust_p_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *, eps_g=1e-12):
33
+ """One decision for the entire optimizer batch, including cross-responses.
34
+
35
+ M_safe = M - min(0,<S,M>)/||S||^2 S; g = S+M_safe. Therefore
36
+ L = stopgrad(a) L_S + L_M with a=1-min(0,dot)/||S||^2.
37
+ With no usable S anchor, keep Atomic and explicitly waive the guarantee.
38
+ """
39
+ if not (math.isfinite(float(eps_g)) and float(eps_g) >= 0):
40
+ raise ValueError("TRUST-P eps_g must be finite and nonnegative")
41
+ if n_s < 0 or n_m < 0:
42
+ raise ValueError("TRUST-P atom counts must be nonnegative")
43
+ pair = checked_geometry(gs2, gm2, dot)
44
+ s, m, d = pair.first_norm2, pair.second_norm2, pair.dot
45
+ no_anchor = n_s == 0 or s <= float(eps_g)
46
+ correction = -min(0., d)/s if not no_anchor else 0.
47
+ a = 1 + correction
48
+ if not math.isfinite(a) or a > torch.finfo(torch.float32).max:
49
+ raise FloatingPointError("TRUST-P strict scale is not representable in fp32")
50
+ # Loss coefficients are fp32. Round towards the protected halfspace, rather
51
+ # than letting nearest rounding reintroduce a tiny opposing component.
52
+ rounded = torch.tensor(a, dtype=torch.float32)
53
+ if float(rounded) < a:
54
+ rounded = torch.nextafter(rounded, torch.tensor(float("inf")))
55
+ a = float(rounded)
56
+ # Orthogonal component survives unchanged; only the opposing parallel
57
+ # component is removed. Projection cannot increase the M norm.
58
+ orthogonal2 = max(m - d*d/s, 0.) if correction > 0 else 0.
59
+ retained2 = orthogonal2 + (a-1+d/s)**2*s if correction > 0 else m
60
+ # Complete the square to avoid catastrophic cancellation for opposing arms.
61
+ post2 = (orthogonal2 + (a + d/s)**2*s) if correction > 0 else max(s + m + 2*d, 0.)
62
+ pre2 = max(s + m + 2*d, 0.)
63
+ cross = a*s + m + (a+1)*d
64
+ cosine = cross/math.sqrt(pre2*post2) if pre2 > 0 and post2 > 0 else (1. if pre2 == post2 == 0 else 0.)
65
+ scalars = (a, retained2, post2, pre2, cross)
66
+ if not all(math.isfinite(v) for v in scalars):
67
+ raise FloatingPointError("non-finite TRUST-P calibration")
68
+ return TrustProjectedResult(
69
+ a, s, m, d, int(n_s), int(n_m), no_anchor, correction > 0,
70
+ math.sqrt(retained2/m) if m > 0 else 0., a*s+d,
71
+ math.sqrt(post2/pre2) if pre2 > 0 else 0., max(-1., min(cosine, 1.)),
72
+ )
73
+
74
+
75
+ def trust_p_response_loss(current_nll, rate, is_strict, is_mismatch, result):
76
+ if bool((is_strict & is_mismatch).any()) or not bool((is_strict | is_mismatch).all()):
77
+ raise ValueError("TRUST-P sides must be disjoint and cover all atoms")
78
+ coefficient = torch.where(is_strict, float(result.strict_scale), 1.).to(rate)
79
+ return (rate.detach() * coefficient * current_nll).sum()
80
+
81
+
82
+ def trust_p_batch_metrics(result, counts):
83
+ r = result
84
+ out = {"batch_"+key: float(value) for key, value in counts.items()}
85
+ out.update({
86
+ "strict_scale": r.strict_scale,
87
+ "mismatch_scale": 1.,
88
+ "strict_head_norm2": r.gs_norm2,
89
+ "mismatch_head_norm2": r.gm_norm2,
90
+ "strict_mismatch_dot": r.dot,
91
+ "strict_mismatch_cosine": r.dot/math.sqrt(r.gs_norm2*r.gm_norm2) if r.gs_norm2 > 0 and r.gm_norm2 > 0 else 0.,
92
+ "projected": float(r.projected),
93
+ "no_anchor": float(r.no_anchor),
94
+ "batch_has_strict": float(r.n_strict > 0),
95
+ "batch_has_mismatch": float(r.n_mismatch > 0),
96
+ "mismatch_retained_norm_ratio": r.mismatch_retained_norm_ratio,
97
+ "strict_descent": r.strict_descent,
98
+ "strict_descent_ratio": r.strict_descent/r.gs_norm2 if r.gs_norm2 > 0 else 0.,
99
+ "update_norm_ratio_post_pre": r.update_norm_ratio,
100
+ "update_cosine_pre_post": r.update_cosine,
101
+ "update_ratio_defined": float(r.gs_norm2+r.gm_norm2+2*r.dot > 0),
102
+ "scope": 1.,
103
+ })
104
+ return out
kdflow/algorithms/mp_opd_trust.py ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Isolated TRUST-R/B/P adapters; B/P require a complete optimizer window."""
2
+ import math
3
+ import random
4
+
5
+ import numpy as np
6
+ import torch
7
+
8
+ from . import register_algorithm
9
+ from .mp_opd import MPOnPolicyKD, fixed_partition
10
+
11
+
12
+ def capture_rng():
13
+ return (random.getstate(), np.random.get_state(), torch.get_rng_state(),
14
+ torch.cuda.get_rng_state_all() if torch.cuda.is_initialized() else None)
15
+
16
+
17
+ def restore_rng(state):
18
+ random.setstate(state[0])
19
+ np.random.set_state(state[1])
20
+ torch.set_rng_state(state[2])
21
+ if state[3] is not None:
22
+ torch.cuda.set_rng_state_all(state[3])
23
+
24
+
25
+ class TrustMPOnPolicyKD(MPOnPolicyKD):
26
+ mode = 'atomic'
27
+ trust_mode = None
28
+
29
+ def __init__(self, *args, **kwargs):
30
+ super().__init__(*args, **kwargs)
31
+ from ._mp_opd_port._mp_opd_grass_span import _env_flag
32
+ self.mode = self.trust_mode
33
+ self.requires_optimizer_batch = self.mode in {'trust_b', 'trust_p'}
34
+ self.eps_g = float(self.kd.mp_opd_trust_eps_g)
35
+ if not math.isfinite(self.eps_g) or self.eps_g < 0:
36
+ raise ValueError('TRUST eps_g must be finite and nonnegative')
37
+ if self.kd.mp_opd_grass_geometry != 'exact_head' or not _env_flag('MP_OPD_GRASS_EXACT_FP32', True):
38
+ raise ValueError('TRUST requires exact_head with exact FP32 geometry enabled')
39
+ if self.kd.mp_opd_trust_vocab_chunk < 1:
40
+ raise ValueError('TRUST vocabulary tile must be positive')
41
+ self._collector = self._result = None
42
+ self._normalizer = None
43
+ self._batch_metrics = {}
44
+
45
+ def _partition(self, atoms, student_ids, teacher_ids):
46
+ return fixed_partition(len(atoms), 1), None, 0
47
+
48
+ def training_step(self, batch):
49
+ if self.requires_optimizer_batch:
50
+ if self._collector is None and self._result is None:
51
+ raise RuntimeError('TRUST-B/P requires prepare_optimizer_batch before scoring/backward')
52
+ normalizer = float(batch['avg_micro_batch_token_num'])
53
+ if not math.isfinite(normalizer) or normalizer <= 0:
54
+ raise ValueError('TRUST-B/P requires a finite positive common normalizer')
55
+ if self._normalizer is not None and normalizer != self._normalizer:
56
+ raise ValueError('TRUST-B/P requires one common normalizer across the window')
57
+ self._normalizer = normalizer
58
+ return super().training_step(batch)
59
+
60
+ def _effective_rates(self, credits, atoms, partition, hidden, labels, **_kwargs):
61
+ from ._mp_opd_port._mp_opd_trust import (strict_mask, trust_calibrate,
62
+ trust_response_metrics, gram_affordable)
63
+ from ._mp_opd_port._mp_opd_grass_chunk import chunk_head_gram
64
+ strict = strict_mask([a.boundary_type for a in atoms]).to(credits.rate.device)
65
+ ranges = tuple((a.student_start, a.student_end) for a in atoms)
66
+ if self._collector is not None:
67
+ with torch.no_grad():
68
+ logits = self._head_logits(self.student.model.lm_head, hidden, self.student_cap, True)
69
+ self._collector.add_response(logits, hidden.detach(), labels, ranges,
70
+ credits.rate, strict, -credits.student_token_nll)
71
+ return torch.zeros_like(credits.rate), {}
72
+ if self.requires_optimizer_batch:
73
+ result, metrics = self._result, self._batch_metrics
74
+ else:
75
+ with torch.no_grad():
76
+ logits = self._head_logits(self.student.model.lm_head, hidden, self.student_cap, True)
77
+ if not gram_affordable(labels.numel(), logits.shape[-1], logits.device):
78
+ raise RuntimeError('TRUST-R exact Gram exceeds memory guard; no silent Atomic fallback')
79
+ head = chunk_head_gram(logits, hidden.detach(), labels, ranges, ((0,len(atoms)),),
80
+ selected_log_prob=-credits.student_token_nll, softcap=self.student_cap,
81
+ head_bias=getattr(self.student.model.lm_head,'bias',None) is not None)
82
+ q, h = credits.rate.detach().float(), head.gram.detach().float()
83
+ result = trust_calibrate(q[strict], q[~strict], h[strict][:,strict],
84
+ h[~strict][:,~strict], h[strict][:,~strict], eps_g=self.eps_g)
85
+ raw = trust_response_metrics(result,strict,credits.weight,scope='response')
86
+ metrics = {'mp_opd_trust_r_'+k: credits.rate.new_tensor(v) for k,v in raw.items()}
87
+ if self.mode == 'trust_p':
88
+ coefficients = torch.where(strict, float(result.strict_scale), 1.).to(credits.rate)
89
+ else:
90
+ coefficients = torch.where(strict, 1., float(result.lam)).to(credits.rate)
91
+ return credits.rate.detach()*coefficients, metrics
92
+
93
+ def prepare_optimizer_batch(self, batches, *, batch_loader=lambda x:x, check_window=True):
94
+ from ._mp_opd_port._mp_opd_trust_batch import TrustHeadAccumulator
95
+ from ._mp_opd_port._mp_opd_trust import trust_batch_metrics
96
+ from ._mp_opd_port._mp_opd_trust_p import trust_p_calibrate_scalars, trust_p_batch_metrics
97
+ if not self.requires_optimizer_batch:
98
+ return None
99
+ if self._collector is not None or self._result is not None:
100
+ raise RuntimeError('TRUST optimizer window is already active')
101
+ if not batches:
102
+ raise ValueError('TRUST requires a nonempty optimizer window')
103
+ if check_window and (len(batches) != self.strategy.accumulated_gradient or self.strategy.step != 0):
104
+ raise ValueError('TRUST-B/P requires exactly one complete optimizer accumulation window at step0')
105
+ before = capture_rng()
106
+ states = []
107
+ self._normalizer = None
108
+ collector = TrustHeadAccumulator(softcap=self.student_cap,
109
+ head_bias=getattr(self.student.model.lm_head,'bias',None) is not None,
110
+ vocab_chunk=self.kd.mp_opd_trust_vocab_chunk, force_fp32=self.mode == 'trust_p')
111
+ self._collector = collector
112
+ try:
113
+ with torch.no_grad():
114
+ for batch in batches:
115
+ states.append(capture_rng())
116
+ self.training_step(batch_loader(batch))
117
+ after = capture_rng()
118
+ options = {'calibrate':trust_p_calibrate_scalars} if self.mode == 'trust_p' else {}
119
+ self._result = collector.finalize(eps_g=self.eps_g, distributed=False, **options)
120
+ records = collector.responses
121
+ ns = sum(r['strict_atoms'] for r in records)
122
+ nm = sum(r['mismatch_atoms'] for r in records)
123
+ counts = dict(strict_span_count=ns,mismatch_span_count=nm,strict_atom_count=ns,mismatch_atom_count=nm,
124
+ strict_token_count=sum(r['strict_tokens'] for r in records),
125
+ mismatch_token_count=sum(r['mismatch_tokens'] for r in records),
126
+ strict_span_fraction=ns/max(ns+nm,1),mismatch_span_fraction=nm/max(ns+nm,1))
127
+ metric_fn = trust_p_batch_metrics if self.mode == 'trust_p' else trust_batch_metrics
128
+ self._batch_metrics = {f'mp_opd_{self.mode}_'+k: torch.tensor(float(v),device=self.teacher_head.weight.device,dtype=torch.float32)
129
+ for k,v in metric_fn(self._result,counts).items()}
130
+ prefix = 'mp_opd_'+self.mode+'_'
131
+ self._batch_metrics.update({prefix+'full_batch':torch.tensor(1.,device=self.teacher_head.weight.device),
132
+ prefix+'calibration_response_count':torch.tensor(float(len(records)),device=self.teacher_head.weight.device)})
133
+ return {'rng_states':states,'after_rng':after}
134
+ except BaseException:
135
+ self.finish_optimizer_batch()
136
+ raise
137
+ finally:
138
+ self._collector = None
139
+ restore_rng(before)
140
+
141
+ def finish_optimizer_batch(self):
142
+ self._collector = self._result = None
143
+ self._normalizer = None
144
+ self._batch_metrics = {}
145
+
146
+
147
+ @register_algorithm('mp_opd_trust_r')
148
+ class TrustRMPOnPolicyKD(TrustMPOnPolicyKD):
149
+ trust_mode = 'trust_r'
150
+
151
+
152
+ @register_algorithm('mp_opd_trust_b')
153
+ class TrustBMPOnPolicyKD(TrustMPOnPolicyKD):
154
+ trust_mode = 'trust_b'
155
+
156
+
157
+ @register_algorithm('mp_opd_trust_p')
158
+ class TrustPMPOnPolicyKD(TrustMPOnPolicyKD):
159
+ trust_mode = 'trust_p'
kdflow/algorithms/simct_paper.py ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Paper Eq.7-8 scores on the released SimCT candidate/alignment interface."""
2
+ import types
3
+
4
+ import torch
5
+
6
+ from . import register_algorithm
7
+ from .simct import SimCT
8
+
9
+
10
+ def paper_virtual_vocab(upstream, segments, student_logits, teacher_logits, student_ids, teacher_ids):
11
+ # Normalize over the FULL native vocabulary at EACH autoregressive position.
12
+ # The original builder then selects shared entries and averages span scores.
13
+ return type(upstream)._build_virtual_vocab_logits(upstream, segments,
14
+ torch.log_softmax(student_logits.float(), -1),
15
+ torch.log_softmax(teacher_logits.float(), -1), student_ids, teacher_ids)
16
+
17
+
18
+ @register_algorithm('simct_paper')
19
+ class SimCTPaper(SimCT):
20
+ def __init__(self, *args, **kwargs):
21
+ super().__init__(*args, **kwargs)
22
+ self.upstream._build_virtual_vocab_logits = types.MethodType(paper_virtual_vocab, self.upstream)
kdflow/algorithms/trust_training.py ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Single-rank actor integration for complete-window TRUST-B/P calibration."""
2
+ from collections import defaultdict
3
+
4
+ import torch
5
+
6
+ from .mp_opd_trust import restore_rng
7
+
8
+
9
+ def fit_trust_batch(actor, train_data):
10
+ actor.student.train()
11
+ if actor.args.train.use_dynamic_bsz:
12
+ actor.strategy.accumulated_gradient = len(train_data)
13
+ actor.strategy.step = 0
14
+ algo = actor.kd_algorithm
15
+ prepared = algo.prepare_optimizer_batch(train_data,batch_loader=actor._prepare_micro_batch)
16
+ status = defaultdict(list)
17
+ try:
18
+ for index,batch in enumerate(train_data):
19
+ restore_rng(prepared['rng_states'][index])
20
+ micro = actor._prepare_micro_batch(batch)
21
+ info = algo.training_step(micro)
22
+ for key,value in info.items():
23
+ status[key].append(value.item())
24
+ actor.strategy.backward(info['train/loss'],actor.student,actor.optim)
25
+ status['train/grad_norm'].append(torch.nn.utils.clip_grad_norm_(actor.student.parameters(),float('inf')).item())
26
+ actor.strategy.optimizer_step(actor.optim,actor.student,actor.scheduler)
27
+ if actor.args.kd.use_ema_teacher and actor.strategy.step == 0:
28
+ actor.ema_update()
29
+ for name in ('response_length','total_length'):
30
+ if name in micro:
31
+ status['rollout/'+name].append(micro[name].mean().item())
32
+ del micro
33
+ restore_rng(prepared['after_rng'])
34
+ finally:
35
+ algo.finish_optimizer_batch()
36
+ status = {k:sum(v)/len(v) for k,v in status.items()}
37
+ status['train/lr'] = actor.scheduler.get_last_lr()[0]
38
+ return {k:actor.strategy.all_reduce(v,op='mean') for k,v in status.items()}
39
+
40
+
41
+ @torch.no_grad()
42
+ def evaluate_trust_batch(actor, eval_data):
43
+ """One evaluation-set calibration; no training-state reuse or optimizer step."""
44
+ algo = actor.kd_algorithm
45
+ was_training = actor.student.training
46
+ actor.student.eval()
47
+ batches = list(eval_data)
48
+
49
+ def load(batch):
50
+ micro = actor._prepare_micro_batch(batch)
51
+ weight = micro.pop('_eval_weight',1.)
52
+ if float(weight) != 1.:
53
+ raise ValueError('Single-rank TRUST evaluation requires unit sample weights')
54
+ micro['avg_micro_batch_token_num'] = 1.
55
+ return micro
56
+
57
+ sums,total = defaultdict(float),0.
58
+ try:
59
+ if not batches:
60
+ return {'eval/num_tokens':0.}
61
+ prepared = algo.prepare_optimizer_batch(batches,batch_loader=load,check_window=False)
62
+ for index,batch in enumerate(batches):
63
+ restore_rng(prepared['rng_states'][index])
64
+ micro = load(batch)
65
+ count = float(micro['stu_loss_mask'].sum())
66
+ metrics = algo.training_step(micro)
67
+ for key,value in metrics.items():
68
+ # Loss already sums tokens; diagnostic values are response averages.
69
+ sums[key] += float(value) * (1. if key in {'train/loss','train/kd_loss'} else count)
70
+ total += count
71
+ restore_rng(prepared['after_rng'])
72
+ finally:
73
+ algo.finish_optimizer_batch()
74
+ actor.student.train(was_training)
75
+ from kdflow.utils.logging_utils import normalize_eval_metrics
76
+ result = {k:actor.strategy.all_reduce(v,op='sum')/max(total,1.) for k,v in sums.items()}
77
+ result['eval/num_tokens'] = total
78
+ return normalize_eval_metrics(result)
kdflow/arguments/distillation_args.py CHANGED
@@ -23,7 +23,7 @@ class DistillationArguments:
23
  default="byte_align_ctkd",
24
  metadata={
25
  "help": "KD algorithm for each training step.",
26
- "choices": ["simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", *MP_OPD_ALGORITHMS],
27
  }
28
  )
29
  kd_loss_fn: str = field(
@@ -129,6 +129,8 @@ class DistillationArguments:
129
  mp_opd_align_eps_h: float = field(default=1e-12)
130
  mp_opd_align_new_min_descent_fraction: float = field(default=0.1)
131
  mp_opd_align_new_eps_g: float = field(default=1e-12)
 
 
132
 
133
  def __post_init__(self):
134
  if self.multi_teacher_config is not None:
@@ -136,18 +138,23 @@ class DistillationArguments:
136
  "This release supports one teacher model per run; "
137
  "use --teacher_name_or_path instead of --multi_teacher_config."
138
  )
139
- supported_algorithms = {"simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", *MP_OPD_ALGORITHMS}
140
  if self.kd_algorithm not in supported_algorithms:
141
  raise ValueError(
142
  f"kd_algorithm must be one of {sorted(supported_algorithms)}, "
143
  f"got {self.kd_algorithm!r}."
144
  )
145
- if self.kd_algorithm == 'simct':
146
  if self.kd_ratio != 1 or self.kd_temperature != 1 or self.kd_loss_fn != 'rkl':
147
  raise ValueError('SimCT baseline requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
148
  if not math.isfinite(self.simct_gh_mask_threshold) or self.simct_gh_mask_threshold < 0:
149
  raise ValueError('SimCT G(h) threshold must be finite and nonnegative')
150
  if self.kd_algorithm in MP_OPD_ALGORITHMS:
 
 
 
 
 
151
  if self.kd_algorithm == 'mp_opd_align_new':
152
  if self.mp_opd_grass_geometry != 'exact_head':
153
  raise ValueError('ALIGN-NEW requires exact_head geometry')
 
23
  default="byte_align_ctkd",
24
  metadata={
25
  "help": "KD algorithm for each training step.",
26
+ "choices": ["simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", "simct_paper", *MP_OPD_ALGORITHMS],
27
  }
28
  )
29
  kd_loss_fn: str = field(
 
129
  mp_opd_align_eps_h: float = field(default=1e-12)
130
  mp_opd_align_new_min_descent_fraction: float = field(default=0.1)
131
  mp_opd_align_new_eps_g: float = field(default=1e-12)
132
+ mp_opd_trust_eps_g: float = field(default=1e-12)
133
+ mp_opd_trust_vocab_chunk: int = field(default=4096)
134
 
135
  def __post_init__(self):
136
  if self.multi_teacher_config is not None:
 
138
  "This release supports one teacher model per run; "
139
  "use --teacher_name_or_path instead of --multi_teacher_config."
140
  )
141
+ supported_algorithms = {"simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", "simct_paper", *MP_OPD_ALGORITHMS}
142
  if self.kd_algorithm not in supported_algorithms:
143
  raise ValueError(
144
  f"kd_algorithm must be one of {sorted(supported_algorithms)}, "
145
  f"got {self.kd_algorithm!r}."
146
  )
147
+ if self.kd_algorithm in {'simct','simct_paper'}:
148
  if self.kd_ratio != 1 or self.kd_temperature != 1 or self.kd_loss_fn != 'rkl':
149
  raise ValueError('SimCT baseline requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
150
  if not math.isfinite(self.simct_gh_mask_threshold) or self.simct_gh_mask_threshold < 0:
151
  raise ValueError('SimCT G(h) threshold must be finite and nonnegative')
152
  if self.kd_algorithm in MP_OPD_ALGORITHMS:
153
+ if self.kd_algorithm.startswith('mp_opd_trust_'):
154
+ if self.mp_opd_grass_geometry != 'exact_head':
155
+ raise ValueError('TRUST requires exact_head geometry')
156
+ if not math.isfinite(self.mp_opd_trust_eps_g) or self.mp_opd_trust_eps_g < 0 or self.mp_opd_trust_vocab_chunk < 1:
157
+ raise ValueError('TRUST requires finite nonnegative eps_g and positive vocabulary tile')
158
  if self.kd_algorithm == 'mp_opd_align_new':
159
  if self.mp_opd_grass_geometry != 'exact_head':
160
  raise ValueError('ALIGN-NEW requires exact_head geometry')
kdflow/cli/train_kd_on_policy.py CHANGED
@@ -22,7 +22,7 @@ from kdflow.utils.utils import get_tokenizer, load_custom_eval_fn
22
 
23
 
24
  def train(args):
25
- if args.kd.kd_algorithm == 'simct' and (
26
  args.train.num_nodes * args.train.num_gpus_per_node != 1 or args.model.ring_attn_size != 1
27
  ):
28
  raise ValueError('SimCT baseline initially requires one student rank and ring_attn_size=1')
@@ -96,11 +96,12 @@ def train(args):
96
  "byte_align_ctkd",
97
  "topk_byte_align_ctkd",
98
  "simct",
 
99
  *MP_OPD_ALGORITHMS,
100
  }:
101
  raise ValueError(
102
  "Cross-tokenizer distillation supports simple_ctkd, "
103
- "byte_align_ctkd, topk_byte_align_ctkd, simct, or an opt-in mp_opd_* mode."
104
  )
105
 
106
  environment_rollout = bool(args.rollout.alfworld_config)
 
22
 
23
 
24
  def train(args):
25
+ if args.kd.kd_algorithm in {'simct','simct_paper'} and (
26
  args.train.num_nodes * args.train.num_gpus_per_node != 1 or args.model.ring_attn_size != 1
27
  ):
28
  raise ValueError('SimCT baseline initially requires one student rank and ring_attn_size=1')
 
96
  "byte_align_ctkd",
97
  "topk_byte_align_ctkd",
98
  "simct",
99
+ "simct_paper",
100
  *MP_OPD_ALGORITHMS,
101
  }:
102
  raise ValueError(
103
  "Cross-tokenizer distillation supports simple_ctkd, "
104
+ "byte_align_ctkd, topk_byte_align_ctkd, simct, simct_paper, or an opt-in mp_opd_* mode."
105
  )
106
 
107
  environment_rollout = bool(args.rollout.alfworld_config)
kdflow/mp_opd_modes.py CHANGED
@@ -3,4 +3,5 @@
3
  MP_OPD_ALGORITHMS = (
4
  'mp_opd_atomic', 'mp_opd_fixed', 'mp_opd_random',
5
  'mp_opd_grass_chunk', 'mp_opd_align', 'mp_opd_align_new',
 
6
  )
 
3
  MP_OPD_ALGORITHMS = (
4
  'mp_opd_atomic', 'mp_opd_fixed', 'mp_opd_random',
5
  'mp_opd_grass_chunk', 'mp_opd_align', 'mp_opd_align_new',
6
+ 'mp_opd_trust_r', 'mp_opd_trust_b', 'mp_opd_trust_p',
7
  )
kdflow/ray/train/student_actor.py CHANGED
@@ -288,6 +288,9 @@ class StudentRayActor:
288
  Returns:
289
  Training status dict
290
  """
 
 
 
291
  self.student.train()
292
  status = defaultdict(list)
293
 
@@ -338,6 +341,9 @@ class StudentRayActor:
338
  @torch.no_grad()
339
  def evaluate(self, eval_data):
340
  """Evaluate the current student without updating model states."""
 
 
 
341
  was_training = self.student.training
342
  self.student.eval()
343
  metric_sums = defaultdict(float)
 
288
  Returns:
289
  Training status dict
290
  """
291
+ if getattr(self.kd_algorithm,'requires_optimizer_batch',False):
292
+ from kdflow.algorithms.trust_training import fit_trust_batch
293
+ return fit_trust_batch(self,train_data)
294
  self.student.train()
295
  status = defaultdict(list)
296
 
 
341
  @torch.no_grad()
342
  def evaluate(self, eval_data):
343
  """Evaluate the current student without updating model states."""
344
+ if getattr(self.kd_algorithm,'requires_optimizer_batch',False):
345
+ from kdflow.algorithms.trust_training import evaluate_trust_batch
346
+ return evaluate_trust_batch(self,eval_data)
347
  was_training = self.student.training
348
  self.student.eval()
349
  metric_sums = defaultdict(float)
tests/fixtures/trust/_mp_opd_trust.py.txt ADDED
@@ -0,0 +1,348 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TRUST v1.0 - trusted reference update scaling for cross-tokenizer OPD.
2
+
3
+ Mismatch supervision may carry signal, but its direction and scale should be
4
+ calibrated against the more reliable strict/matched supervision. TRUST does not
5
+ touch relative credits inside mismatch spans: per calibration unit it scales the
6
+ whole mismatch update by ``lambda = max(0, <gS, gM> / ||gM||^2)`` and keeps the
7
+ strict update whole.
8
+
9
+ Two modes share this file: TRUST-R (one lambda per response) and TRUST-B (one
10
+ lambda per optimizer batch, with microbatch calibration an explicit legacy
11
+ ablation). The only intended difference between them is the
12
+ calibration aggregation level; the strict/mismatch partition, the proxy, and
13
+ the guards are identical.
14
+
15
+ The strict/mismatch partition comes from the atomizer, not from a threshold:
16
+ an atom whose student and teacher spans are both exactly one token over the
17
+ same bytes (``boundary_type == "one_to_one"``) is matched supervision, and an
18
+ atom that straddles a tokenization boundary (``multi_token``) is mismatch. Any
19
+ atom with another boundary type fails closed, because silently filing it under
20
+ either side would change what the calibration means.
21
+
22
+ The proxy is the GRASS LM-head geometry, reused unchanged: with per-atom
23
+ coefficients ``q_i = r_i`` (the OPD atom loss is a plain token-sum, so the
24
+ outer coefficient is 1 and the response weight is a plain sum) and token
25
+ reduction ``a_it = 1`` (``current_nll`` is a token sum over the atom), the
26
+ effective coefficients are exactly the rates the loss consumes. The Gram code
27
+ is shared, so a TRUST number and a GRASS number read the same geometry.
28
+
29
+ Deliberate v1 deviations from the spec text, all fail-safe rather than silent:
30
+
31
+ * Median/p10/p25/p75/p90 keys are not emitted. Metrics reduce per micro-batch
32
+ by sum-then-mean, which cannot express a percentile; emitting a mean under a
33
+ ``_median`` key would lie about the statistic. Means and fractions over the
34
+ micro-batch responses are exact as stated (production uses
35
+ micro_train_batch_size=4).
36
+ * ``mp_opd_trust_scope`` is a float code (0.0 = response, 1.0 = batch unit) because
37
+ the metric table only carries finite tensors.
38
+ ``mp_opd_trust_full_batch`` distinguishes the complete optimizer batch from
39
+ the legacy microbatch unit.
40
+ * Samples or units without both sides log 0.0 for undefined quantities (never
41
+ NaN: the trainer fails fast on non-finite metrics), with the eligibility
42
+ fractions (``has_both``/``zero_*``) as the interpretive key.
43
+ * TRUST requires ``exact_head`` geometry. The ``diag`` ablation zeroes every
44
+ cross-atom term, which would force every dot product to zero and every lambda
45
+ to zero: a mode that can never act. Failing closed beats a silent no-op.
46
+ """
47
+
48
+ from __future__ import annotations
49
+
50
+ from dataclasses import dataclass
51
+ from typing import Sequence
52
+
53
+ import torch
54
+
55
+ # Section 9 fixes this: a mismatch norm at or below it carries no usable
56
+ # direction, so the calibration that would divide by it is skipped.
57
+ TRUST_EPS_G = 1e-12
58
+ # Only used to keep norm denominators away from zero; it never decides whether
59
+ # a unit calibrates or what lambda it gets.
60
+ TRUST_EPS_NORM = 1e-12
61
+ # Transient multiplier for one exact-Gram build: the [tokens, vocab] window,
62
+ # its Jacobian, the probabilities, and the vocabulary product temporaries.
63
+ # A step whose estimate exceeds half the free device memory degrades to the
64
+ # atomic fallback (loudly counted) instead of attempting an allocation the
65
+ # device cannot hold. Killed a 171-step trust_b run at 11.73 GiB attempted on
66
+ # 6.34 GiB free; the fallback that would have saved it did not exist yet.
67
+ TRUST_GRAM_TRANSIENT_FACTOR = 4.0
68
+ TRUST_GRAM_FREE_FRACTION = 0.5
69
+
70
+
71
+ def gram_affordable(n_tokens: int, vocab: int, device: torch.device) -> bool:
72
+ """Whether one exact Gram build fits in the currently free device memory.
73
+
74
+ Estimates ``n_tokens * vocab`` fp32 entries times the transient factor and
75
+ requires it under half the free bytes. On a device without memory info
76
+ (CPU tests) everything is affordable; the production guard this provides
77
+ is against the allocation, not the math.
78
+ """
79
+ if n_tokens <= 0 or vocab <= 0:
80
+ raise ValueError("gram affordability needs positive token and vocab counts")
81
+ if device.type != "cuda" or not torch.cuda.is_available():
82
+ return True
83
+ free_bytes, _ = torch.cuda.mem_get_info(device)
84
+ need_bytes = float(n_tokens) * float(vocab) * 4.0 * TRUST_GRAM_TRANSIENT_FACTOR
85
+ return need_bytes <= float(free_bytes) * TRUST_GRAM_FREE_FRACTION
86
+
87
+
88
+ @dataclass(frozen=True)
89
+ class TrustUnitResult:
90
+ """One calibration unit's lambda plus everything its diagnostics derive from."""
91
+
92
+ lam: float
93
+ dot: float
94
+ gs_norm2: float
95
+ gm_norm2: float
96
+ cosine: float
97
+ norm_ratio: float
98
+ kappa: float
99
+ has_strict: bool
100
+ has_mismatch: bool
101
+ calibrated: bool
102
+ update_cosine: float
103
+ update_norm_ratio: float
104
+ n_strict: int
105
+ n_mismatch: int
106
+
107
+
108
+ def _require_finite(name: str, *tensors: torch.Tensor) -> None:
109
+ for tensor in tensors:
110
+ if not bool(torch.isfinite(tensor).all()):
111
+ raise ValueError(f"TRUST received a non-finite {name}")
112
+
113
+
114
+ def trust_calibrate(
115
+ qs: torch.Tensor,
116
+ qm: torch.Tensor,
117
+ gram_ss: torch.Tensor,
118
+ gram_mm: torch.Tensor,
119
+ gram_sm: torch.Tensor,
120
+ *,
121
+ eps_g: float = TRUST_EPS_G,
122
+ eps_norm: float = TRUST_EPS_NORM,
123
+ ) -> TrustUnitResult:
124
+ """Calibrate one strict/mismatch pair from its Gram blocks.
125
+
126
+ ``qs``/``qm`` are the effective strict/mismatch coefficients and
127
+ ``gram_ss``/``gram_mm``/``gram_sm`` the matching blocks of the exact
128
+ LM-head Gram. The geometry is FP32; quadratic reductions promote to FP64.
129
+ The returned lambda is a Python float, so it can never carry a graph.
130
+ """
131
+ _require_finite("coefficients", qs, qm)
132
+ _require_finite("Gram blocks", gram_ss, gram_mm, gram_sm)
133
+ if not (float(eps_g) >= 0.0):
134
+ raise ValueError(f"TRUST eps_g must be non-negative, got {eps_g!r}")
135
+ qs = qs.detach().to(torch.float64).reshape(-1)
136
+ qm = qm.detach().to(torch.float64).reshape(-1)
137
+ gram_ss, gram_mm, gram_sm = (g.detach().double() for g in (gram_ss, gram_mm, gram_sm))
138
+ n_s, n_m = int(qs.numel()), int(qm.numel())
139
+ if (n_s, n_m) != (int(gram_ss.shape[0]), int(gram_mm.shape[0])):
140
+ raise ValueError(
141
+ f"TRUST Gram blocks {tuple(gram_ss.shape)}/{tuple(gram_mm.shape)} "
142
+ f"do not match coefficient counts ({n_s}, {n_m})"
143
+ )
144
+ if tuple(gram_sm.shape) != (n_s, n_m):
145
+ raise ValueError(
146
+ f"TRUST cross Gram {tuple(gram_sm.shape)} does not match ({n_s}, {n_m})"
147
+ )
148
+
149
+ # Empty sides have no geometry to read: the scalars below are defined as
150
+ # zero rather than derived from a matmul over an empty axis.
151
+ gs2 = float(qs @ gram_ss @ qs) if n_s else 0.0
152
+ gm2 = float(qm @ gram_mm @ qm) if n_m else 0.0
153
+ dot = float(qs @ gram_sm @ qm) if (n_s and n_m) else 0.0
154
+ scales = [float(q.abs() @ g.abs() @ q.abs()) if q.numel() else 0.0
155
+ for q, g in ((qs, gram_ss), (qm, gram_mm))]
156
+ for name, norm, scale in zip(("strict", "mismatch"), (gs2, gm2), scales):
157
+ if norm < -1e-6 * max(scale, 1e-300):
158
+ raise ValueError(f"TRUST {name} Gram gives a materially negative squared norm")
159
+ gs2, gm2 = max(gs2, 0.), max(gm2, 0.)
160
+ bound = (gs2 * gm2)**.5
161
+ if abs(dot) > bound + 1e-6 * max((scales[0]*scales[1])**.5, 1e-300):
162
+ raise ValueError("TRUST Gram violates Cauchy-Schwarz")
163
+ dot = min(max(dot, -bound), bound)
164
+ return trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m,
165
+ eps_g=eps_g, eps_norm=eps_norm)
166
+
167
+
168
+ def trust_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *,
169
+ eps_g=TRUST_EPS_G, eps_norm=TRUST_EPS_NORM):
170
+ """The same calibration for explicit summed head updates over a full batch."""
171
+ if not (0 <= float(eps_g) < float("inf")):
172
+ raise ValueError("TRUST eps_g must be finite and nonnegative")
173
+ for name, value in (("gs_norm2", gs2), ("gm_norm2", gm2), ("dot", dot)):
174
+ if not abs(value) < float("inf"):
175
+ raise ValueError(f"TRUST {name} is non-finite")
176
+ if gs2 < 0 or gm2 < 0:
177
+ raise ValueError("TRUST squared norms must be nonnegative")
178
+ bound = (gs2 * gm2)**.5
179
+ if abs(dot) > bound * (1. + 1e-6) + 1e-300:
180
+ raise ValueError("TRUST summed updates violate Cauchy-Schwarz")
181
+ dot = min(max(dot, -bound), bound)
182
+
183
+ has_strict = n_s > 0
184
+ has_mismatch = n_m > 0
185
+ calibrated = bool(has_strict and has_mismatch and gm2 > float(eps_g))
186
+ lam = max(0.0, dot / gm2) if calibrated else 0.0
187
+
188
+ gs = gs2**0.5
189
+ gm = gm2**0.5
190
+ cosine = dot / (gs * gm + float(eps_norm))
191
+ norm_ratio = gm / (gs + float(eps_norm))
192
+ kappa = lam * gm / (gs + float(eps_norm))
193
+
194
+ # g_pre = gS + gM and g_post = gS + lam gM, expanded in Gram scalars so no
195
+ # vector is ever formed. When both updates vanish they are identical.
196
+ pre2 = gs2 + gm2 + 2.0 * dot
197
+ post2 = gs2 + lam * lam * gm2 + 2.0 * lam * dot
198
+ inner = gs2 + lam * gm2 + (1.0 + lam) * dot
199
+ pre2 = max(pre2, 0.0)
200
+ post2 = max(post2, 0.0)
201
+ if pre2 <= 0.0 and post2 <= 0.0:
202
+ update_cosine = 1.0
203
+ elif pre2 <= 0.0 or post2 <= 0.0:
204
+ update_cosine = 0.0
205
+ else:
206
+ update_cosine = inner / (pre2**0.5 * post2**0.5 + float(eps_norm))
207
+ update_norm_ratio = post2**0.5 / (pre2**0.5 + float(eps_norm))
208
+
209
+ return TrustUnitResult(
210
+ lam=lam,
211
+ dot=dot,
212
+ gs_norm2=gs2,
213
+ gm_norm2=gm2,
214
+ cosine=cosine,
215
+ norm_ratio=norm_ratio,
216
+ kappa=kappa,
217
+ has_strict=has_strict,
218
+ has_mismatch=has_mismatch,
219
+ calibrated=calibrated,
220
+ update_cosine=update_cosine,
221
+ update_norm_ratio=update_norm_ratio,
222
+ n_strict=n_s,
223
+ n_mismatch=n_m,
224
+ )
225
+
226
+
227
+ def strict_mask(boundary_types: Sequence[str]) -> torch.Tensor:
228
+ """Strict-side mask from atomizer boundary types, failing closed.
229
+
230
+ ``one_to_one`` is matched supervision, ``multi_token`` is mismatch. A
231
+ third kind would silently change the calibration under either choice, so
232
+ it raises instead of being filed anywhere.
233
+ """
234
+ mask = []
235
+ for kind in boundary_types:
236
+ if kind == "one_to_one":
237
+ mask.append(True)
238
+ elif kind == "multi_token":
239
+ mask.append(False)
240
+ else:
241
+ raise ValueError(f"TRUST has no side for boundary type {kind!r}")
242
+ return torch.tensor(mask, dtype=torch.bool)
243
+
244
+
245
+ def trust_response_loss(
246
+ current_nll: torch.Tensor,
247
+ rate: torch.Tensor,
248
+ is_strict: torch.Tensor,
249
+ is_mismatch: torch.Tensor,
250
+ result: TrustUnitResult,
251
+ ) -> torch.Tensor:
252
+ """``L_S + stopgrad(lambda) L_M`` with exactly the OPD loss coefficients.
253
+
254
+ Each side is a ``(rates.detach() * nll).sum()``, the same construction as
255
+ ``soft_partition_loss``: the strict side keeps its credits whole and the
256
+ mismatch side is scaled by the detached lambda. The lambda is a Python
257
+ float by construction, so no graph can pass through the calibration rule.
258
+ """
259
+ if not (bool(is_strict.any()) or bool(is_mismatch.any())):
260
+ raise ValueError("TRUST needs at least one strict or mismatch atom")
261
+ if bool((is_strict & is_mismatch).any()):
262
+ raise ValueError("TRUST strict and mismatch sets must be disjoint")
263
+ zero = current_nll.new_zeros(())
264
+ strict = (torch.where(is_strict, rate.detach(), zero) * current_nll).sum()
265
+ mismatch = (torch.where(is_mismatch, rate.detach(), zero) * current_nll).sum()
266
+ return strict + float(result.lam) * mismatch
267
+
268
+
269
+ def _counts(is_strict: torch.Tensor, token_counts: torch.Tensor) -> dict[str, float]:
270
+ """Raw per-unit coverage counts; the caller decides the aggregation."""
271
+ is_mismatch = ~is_strict
272
+ n_s = int(is_strict.sum())
273
+ n_m = int(is_mismatch.sum())
274
+ total = n_s + n_m
275
+ return {
276
+ "strict_span_count": float(n_s),
277
+ "mismatch_span_count": float(n_m),
278
+ # One atom is one supervision span here: there is no higher grouping
279
+ # for TRUST to aggregate, so the span and atom keys coincide by
280
+ # definition rather than by coincidence.
281
+ "strict_atom_count": float(n_s),
282
+ "mismatch_atom_count": float(n_m),
283
+ "strict_token_count": float(token_counts[is_strict].sum()),
284
+ "mismatch_token_count": float(token_counts[is_mismatch].sum()),
285
+ "strict_span_fraction": float(n_s / total) if total else 0.0,
286
+ "mismatch_span_fraction": float(n_m / total) if total else 0.0,
287
+ "has_strict": float(n_s > 0),
288
+ "has_mismatch": float(n_m > 0),
289
+ "has_both": float(n_s > 0 and n_m > 0),
290
+ "zero_strict": float(n_s == 0),
291
+ "zero_mismatch": float(n_m == 0),
292
+ }
293
+
294
+
295
+ def trust_response_metrics(
296
+ result: TrustUnitResult,
297
+ is_strict: torch.Tensor,
298
+ token_counts: torch.Tensor,
299
+ *,
300
+ scope: str,
301
+ ) -> dict[str, float]:
302
+ """Per-unit metric values; the training loop averages them over responses."""
303
+ if scope not in ("response", "batch"):
304
+ raise ValueError(f"TRUST scope must be response|batch, got {scope!r}")
305
+ out = _counts(is_strict, token_counts)
306
+ out.update(
307
+ {
308
+ "cosine": result.cosine,
309
+ "cosine_negative": float(result.cosine < 0.0),
310
+ "mismatch_to_strict_norm_ratio": result.norm_ratio,
311
+ "lambda": result.lam,
312
+ "lambda_zero": float(result.lam <= 0.0),
313
+ "lambda_gt1": float(result.lam > 1.0),
314
+ "calibrated": float(result.calibrated),
315
+ "calibrated_mismatch_norm_ratio": result.kappa,
316
+ "update_cosine_pre_post": result.update_cosine,
317
+ "update_norm_ratio_post_pre": result.update_norm_ratio,
318
+ "scope": 0.0 if scope == "response" else 1.0,
319
+ }
320
+ )
321
+ return out
322
+
323
+
324
+ def trust_batch_metrics(
325
+ results: TrustUnitResult,
326
+ counts: dict[str, float],
327
+ ) -> dict[str, float]:
328
+ """One calibration unit's values, logged without response averaging."""
329
+ return {
330
+ "batch_strict_span_count": counts["strict_span_count"],
331
+ "batch_mismatch_span_count": counts["mismatch_span_count"],
332
+ "batch_strict_atom_count": counts["strict_atom_count"],
333
+ "batch_mismatch_atom_count": counts["mismatch_atom_count"],
334
+ "batch_strict_token_count": counts["strict_token_count"],
335
+ "batch_mismatch_token_count": counts["mismatch_token_count"],
336
+ "batch_strict_span_fraction": counts["strict_span_fraction"],
337
+ "batch_mismatch_span_fraction": counts["mismatch_span_fraction"],
338
+ "cosine": results.cosine,
339
+ "mismatch_to_strict_norm_ratio": results.norm_ratio,
340
+ "lambda": results.lam,
341
+ "batch_has_strict": float(results.has_strict),
342
+ "batch_has_mismatch": float(results.has_mismatch),
343
+ "batch_calibrated": float(results.calibrated),
344
+ "calibrated_mismatch_norm_ratio": results.kappa,
345
+ "update_cosine_pre_post": results.update_cosine,
346
+ "update_norm_ratio_post_pre": results.update_norm_ratio,
347
+ "scope": 1.0,
348
+ }
tests/fixtures/trust/_mp_opd_trust_batch.py.txt ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Exact full-batch TRUST head updates, streamed without retaining forward graphs."""
2
+ from __future__ import annotations
3
+
4
+ import contextlib
5
+ import torch
6
+
7
+ from ._mp_opd_grass_span import _exact_fp32, _log_partition_from_selected
8
+ from ._mp_opd_trust import trust_calibrate_scalars
9
+
10
+
11
+ class TrustHeadAccumulator:
12
+ """Sum strict/mismatch LM-head gradients, including cross-response terms.
13
+
14
+ Storage is two [vocab, hidden] fp32 tensors, independent of batch token count.
15
+ Vocabulary tiles bound transients; no [batch tokens, vocab] concatenation and
16
+ no atom-by-atom global Gram are constructed. Loss normalization is one common
17
+ factor for the entire accumulation window and cancels from lambda.
18
+ """
19
+
20
+ def __init__(self, *, softcap=None, head_bias=False, vocab_chunk=4096,
21
+ force_fp32=False):
22
+ self.softcap = softcap
23
+ self.head_bias = head_bias
24
+ self.vocab_chunk = int(vocab_chunk)
25
+ self.force_fp32 = bool(force_fp32)
26
+ if self.vocab_chunk < 1:
27
+ raise ValueError("TRUST vocabulary tile must be positive")
28
+ self.gs = self.gm = self.bs = self.bm = None
29
+ self.responses = []
30
+
31
+ @torch.no_grad()
32
+ def add_response(self, logits, hidden, labels, ranges, rate, is_strict,
33
+ selected_log_prob):
34
+ covered = ranges[-1][1]
35
+ logits, hidden, labels = logits[:covered], hidden[:covered], labels[:covered]
36
+ lengths = torch.tensor([b-a for a, b in ranges], device=logits.device)
37
+ if sum(b-a for a, b in ranges) != covered or ranges[0][0] != 0:
38
+ raise ValueError("TRUST ranges must tile the covered token prefix")
39
+ if any(ranges[i][1] != ranges[i+1][0] for i in range(len(ranges)-1)):
40
+ raise ValueError("TRUST ranges must have no gaps")
41
+ if not all(bool(torch.isfinite(t).all()) for t in (logits, hidden, rate)):
42
+ raise FloatingPointError("non-finite full-batch TRUST input")
43
+ vocab, width = int(logits.shape[1]), int(hidden.shape[1])
44
+ if self.gs is None:
45
+ need = 2 * vocab * width * 4 + (2 * vocab * 4 if self.head_bias else 0)
46
+ if logits.is_cuda:
47
+ free, _ = torch.cuda.mem_get_info(logits.device)
48
+ if need > free * 0.5:
49
+ raise ValueError("full-batch TRUST head accumulators exceed memory guard")
50
+ self.gs = torch.zeros(vocab, width, dtype=torch.float32, device=logits.device)
51
+ self.gm = torch.zeros_like(self.gs)
52
+ if self.head_bias:
53
+ self.bs = torch.zeros(vocab, dtype=torch.float32, device=logits.device)
54
+ self.bm = torch.zeros_like(self.bs)
55
+ if self.gs.shape != (vocab, width):
56
+ raise ValueError("student head shape changed within TRUST batch")
57
+ strict = is_strict.to(device=logits.device)
58
+ q = rate.detach().float().repeat_interleave(lengths)
59
+ token_strict = strict.repeat_interleave(lengths)
60
+ qs = q * token_strict
61
+ qm = q * ~token_strict
62
+ hs = hidden.detach().float() * qs.unsqueeze(1)
63
+ hm = hidden.detach().float() * qm.unsqueeze(1)
64
+ logz = _log_partition_from_selected(logits, labels, selected_log_prob[:covered]).float()
65
+ autocast_guard = (torch.autocast(device_type=logits.device.type, enabled=False)
66
+ if self.force_fp32 else contextlib.nullcontext())
67
+ with _exact_fp32(), autocast_guard:
68
+ for low in range(0, vocab, self.vocab_chunk):
69
+ high = min(low + self.vocab_chunk, vocab)
70
+ z = logits[:, low:high].detach().float()
71
+ delta = torch.exp(z - logz.unsqueeze(1))
72
+ jacobian = None
73
+ if self.softcap is not None:
74
+ jacobian = (1.0 - (z / float(self.softcap)).clamp(-1, 1).square()).clamp_min(0)
75
+ delta *= jacobian
76
+ rows = ((labels >= low) & (labels < high)).nonzero().flatten()
77
+ cols = labels[rows] - low
78
+ delta[rows, cols] -= 1.0 if jacobian is None else jacobian[rows, cols]
79
+ self.gs[low:high].add_(delta.T @ hs)
80
+ self.gm[low:high].add_(delta.T @ hm)
81
+ if self.head_bias:
82
+ self.bs[low:high].add_(qs @ delta)
83
+ self.bm[low:high].add_(qm @ delta)
84
+ self.responses.append({
85
+ "strict_atoms": int(strict.sum()), "mismatch_atoms": int((~strict).sum()),
86
+ "strict_tokens": int(lengths[strict].sum()),
87
+ "mismatch_tokens": int(lengths[~strict].sum()),
88
+ })
89
+
90
+ @torch.no_grad()
91
+ def finalize(self, *, eps_g=1e-12, distributed=False,
92
+ calibrate=trust_calibrate_scalars):
93
+ if distributed:
94
+ import torch.distributed as dist
95
+ # Every rank uses the same pinned student head shape. Empty ranks
96
+ # allocate matching buffers after exchanging shapes, before collectives.
97
+ shapes = [None] * dist.get_world_size()
98
+ dist.all_gather_object(shapes, None if self.gs is None else tuple(self.gs.shape))
99
+ shape = next((s for s in shapes if s is not None), None)
100
+ if any(s is not None and s != shape for s in shapes):
101
+ raise ValueError("TRUST ranks have inconsistent student head shapes")
102
+ if shape is not None and self.gs is None:
103
+ dev = torch.device("cuda", torch.cuda.current_device()) if dist.get_backend() == "nccl" else torch.device("cpu")
104
+ self.gs = torch.zeros(shape, dtype=torch.float32, device=dev)
105
+ self.gm = torch.zeros_like(self.gs)
106
+ if self.head_bias:
107
+ self.bs = torch.zeros(shape[0], dtype=torch.float32, device=dev)
108
+ self.bm = torch.zeros_like(self.bs)
109
+ if shape is not None:
110
+ for tensor in (self.gs, self.gm, self.bs, self.bm):
111
+ if tensor is not None:
112
+ dist.all_reduce(tensor, op=dist.ReduceOp.SUM)
113
+ records = [None] * dist.get_world_size()
114
+ dist.all_gather_object(records, self.responses)
115
+ self.responses = [r for group in records for r in group]
116
+ gs2 = gm2 = dot = 0.0
117
+ if self.gs is not None:
118
+ for low in range(0, self.gs.shape[0], self.vocab_chunk):
119
+ s = self.gs[low:low+self.vocab_chunk].double()
120
+ m = self.gm[low:low+self.vocab_chunk].double()
121
+ gs2 += float(s.square().sum())
122
+ gm2 += float(m.square().sum())
123
+ dot += float((s*m).sum())
124
+ if self.head_bias:
125
+ gs2 += float(self.bs.double().square().sum())
126
+ gm2 += float(self.bm.double().square().sum())
127
+ dot += float((self.bs.double()*self.bm.double()).sum())
128
+ n_s = sum(r["strict_atoms"] for r in self.responses)
129
+ n_m = sum(r["mismatch_atoms"] for r in self.responses)
130
+ result = calibrate(gs2, gm2, dot, n_s, n_m, eps_g=eps_g)
131
+ self.gs = self.gm = self.bs = self.bm = None
132
+ return result
tests/fixtures/trust/_mp_opd_trust_p.py.txt ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TRUST-P: project mismatch onto the strict non-conflict halfspace.
2
+
3
+ This is a new batch-level algorithm, not a correction to TRUST-R/B. Projection
4
+ is a known operation; efficacy or research novelty is not established here.
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import math
9
+ from dataclasses import dataclass
10
+
11
+ import torch
12
+
13
+ from ._mp_opd_head_pair import checked_geometry
14
+
15
+
16
+ @dataclass(frozen=True)
17
+ class TrustProjectedResult:
18
+ strict_scale: float
19
+ gs_norm2: float
20
+ gm_norm2: float
21
+ dot: float
22
+ n_strict: int
23
+ n_mismatch: int
24
+ no_anchor: bool
25
+ projected: bool
26
+ mismatch_retained_norm_ratio: float
27
+ strict_descent: float
28
+ update_norm_ratio: float
29
+ update_cosine: float
30
+
31
+
32
+ def trust_p_calibrate_scalars(gs2, gm2, dot, n_s, n_m, *, eps_g=1e-12):
33
+ """One decision for the entire optimizer batch, including cross-responses.
34
+
35
+ M_safe = M - min(0,<S,M>)/||S||^2 S; g = S+M_safe. Therefore
36
+ L = stopgrad(a) L_S + L_M with a=1-min(0,dot)/||S||^2.
37
+ With no usable S anchor, keep Atomic and explicitly waive the guarantee.
38
+ """
39
+ if not (math.isfinite(float(eps_g)) and float(eps_g) >= 0):
40
+ raise ValueError("TRUST-P eps_g must be finite and nonnegative")
41
+ if n_s < 0 or n_m < 0:
42
+ raise ValueError("TRUST-P atom counts must be nonnegative")
43
+ pair = checked_geometry(gs2, gm2, dot)
44
+ s, m, d = pair.first_norm2, pair.second_norm2, pair.dot
45
+ no_anchor = n_s == 0 or s <= float(eps_g)
46
+ correction = -min(0., d)/s if not no_anchor else 0.
47
+ a = 1 + correction
48
+ if not math.isfinite(a) or a > torch.finfo(torch.float32).max:
49
+ raise FloatingPointError("TRUST-P strict scale is not representable in fp32")
50
+ # Loss coefficients are fp32. Round towards the protected halfspace, rather
51
+ # than letting nearest rounding reintroduce a tiny opposing component.
52
+ rounded = torch.tensor(a, dtype=torch.float32)
53
+ if float(rounded) < a:
54
+ rounded = torch.nextafter(rounded, torch.tensor(float("inf")))
55
+ a = float(rounded)
56
+ # Orthogonal component survives unchanged; only the opposing parallel
57
+ # component is removed. Projection cannot increase the M norm.
58
+ orthogonal2 = max(m - d*d/s, 0.) if correction > 0 else 0.
59
+ retained2 = orthogonal2 + (a-1+d/s)**2*s if correction > 0 else m
60
+ # Complete the square to avoid catastrophic cancellation for opposing arms.
61
+ post2 = (orthogonal2 + (a + d/s)**2*s) if correction > 0 else max(s + m + 2*d, 0.)
62
+ pre2 = max(s + m + 2*d, 0.)
63
+ cross = a*s + m + (a+1)*d
64
+ cosine = cross/math.sqrt(pre2*post2) if pre2 > 0 and post2 > 0 else (1. if pre2 == post2 == 0 else 0.)
65
+ scalars = (a, retained2, post2, pre2, cross)
66
+ if not all(math.isfinite(v) for v in scalars):
67
+ raise FloatingPointError("non-finite TRUST-P calibration")
68
+ return TrustProjectedResult(
69
+ a, s, m, d, int(n_s), int(n_m), no_anchor, correction > 0,
70
+ math.sqrt(retained2/m) if m > 0 else 0., a*s+d,
71
+ math.sqrt(post2/pre2) if pre2 > 0 else 0., max(-1., min(cosine, 1.)),
72
+ )
73
+
74
+
75
+ def trust_p_response_loss(current_nll, rate, is_strict, is_mismatch, result):
76
+ if bool((is_strict & is_mismatch).any()) or not bool((is_strict | is_mismatch).all()):
77
+ raise ValueError("TRUST-P sides must be disjoint and cover all atoms")
78
+ coefficient = torch.where(is_strict, float(result.strict_scale), 1.).to(rate)
79
+ return (rate.detach() * coefficient * current_nll).sum()
80
+
81
+
82
+ def trust_p_batch_metrics(result, counts):
83
+ r = result
84
+ out = {"batch_"+key: float(value) for key, value in counts.items()}
85
+ out.update({
86
+ "strict_scale": r.strict_scale,
87
+ "mismatch_scale": 1.,
88
+ "strict_head_norm2": r.gs_norm2,
89
+ "mismatch_head_norm2": r.gm_norm2,
90
+ "strict_mismatch_dot": r.dot,
91
+ "strict_mismatch_cosine": r.dot/math.sqrt(r.gs_norm2*r.gm_norm2) if r.gs_norm2 > 0 and r.gm_norm2 > 0 else 0.,
92
+ "projected": float(r.projected),
93
+ "no_anchor": float(r.no_anchor),
94
+ "batch_has_strict": float(r.n_strict > 0),
95
+ "batch_has_mismatch": float(r.n_mismatch > 0),
96
+ "mismatch_retained_norm_ratio": r.mismatch_retained_norm_ratio,
97
+ "strict_descent": r.strict_descent,
98
+ "strict_descent_ratio": r.strict_descent/r.gs_norm2 if r.gs_norm2 > 0 else 0.,
99
+ "update_norm_ratio_post_pre": r.update_norm_ratio,
100
+ "update_cosine_pre_post": r.update_cosine,
101
+ "update_ratio_defined": float(r.gs_norm2+r.gm_norm2+2*r.dot > 0),
102
+ "scope": 1.,
103
+ })
104
+ return out
tests/fixtures/trust/_trust_r_loss.py.txt ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ def _trust_r_loss(
2
+ self,
3
+ credits,
4
+ atoms,
5
+ *,
6
+ student_logits: torch.Tensor | None,
7
+ student_labels: torch.Tensor | None,
8
+ student_hidden: torch.Tensor | None,
9
+ ):
10
+ """TRUST-R: scale this response's mismatch update by strict agreement.
11
+
12
+ The calibration unit is the response, which is exactly what
13
+ ``_partition_loss`` is called with, so no restructuring is needed: the
14
+ strict/mismatch split comes from the atomizer boundary types, the Gram
15
+ is the GRASS exact head Gram over all atoms, and the loss keeps the
16
+ strict side whole while the mismatch side takes the detached lambda.
17
+ """
18
+ if student_logits is None or student_labels is None:
19
+ raise RuntimeError(
20
+ "mp_opd trust modes require the student logits and labels"
21
+ )
22
+ if student_hidden is None:
23
+ raise RuntimeError(
24
+ "mp_opd trust modes require the student LM-head hidden states"
25
+ )
26
+ if credits.student_token_nll is None:
27
+ raise RuntimeError(
28
+ "mp_opd trust modes require per-token student NLL"
29
+ )
30
+ if credits.student_token_nll.numel() != student_logits.shape[0]:
31
+ raise ValueError(
32
+ "per-token student NLL and student logits disagree on token count"
33
+ )
34
+
35
+ device = credits.rate.device
36
+ is_strict = strict_mask([atom.boundary_type for atom in atoms]).to(device)
37
+ n = len(atoms)
38
+ atom_ranges = tuple((atom.student_start, atom.student_end) for atom in atoms)
39
+ covered = atom_ranges[-1][1]
40
+ if not gram_affordable(covered, int(student_logits.shape[1]), device):
41
+ raise ValueError(
42
+ "mp_opd trust_r Gram needs ~"
43
+ f"{covered * int(student_logits.shape[1]) * 16 / 1e9:.1f} GiB transient; "
44
+ "device cannot hold it, atomic fallback"
45
+ )
46
+ head = chunk_head_gram(
47
+ student_logits[:covered],
48
+ student_hidden[:covered],
49
+ student_labels[:covered],
50
+ atom_ranges,
51
+ [(0, n)],
52
+ selected_log_prob=(-credits.student_token_nll[:covered]).detach(),
53
+ softcap=self.grass_softcap,
54
+ head_bias=self.grass_head_bias,
55
+ diagonal_only=False,
56
+ )
57
+ with torch.no_grad():
58
+ q = credits.rate.detach().to(torch.float32)
59
+ gram = head.gram.detach().to(torch.float32)
60
+ result = trust_calibrate(
61
+ q[is_strict],
62
+ q[~is_strict],
63
+ gram[is_strict][:, is_strict],
64
+ gram[~is_strict][:, ~is_strict],
65
+ gram[is_strict][:, ~is_strict],
66
+ eps_g=self.trust_eps_g,
67
+ )
68
+ loss = trust_response_loss(
69
+ credits.current_nll, credits.rate, is_strict, ~is_strict, result
70
+ )
71
+ token_counts = torch.tensor(
72
+ [atom.student_token_count for atom in atoms],
73
+ dtype=torch.float32,
74
+ device=device,
75
+ )
76
+ raw = trust_response_metrics(result, is_strict, token_counts, scope="response")
77
+ metrics = {
78
+ self._TRUST_RESPONSE_METRIC_NAMES[key]: credits.rate.new_tensor(value)
79
+ for key, value in raw.items()
80
+ }
81
+ metrics.update(self._trust_static_metrics(
82
+ credits.rate, head.symmetry_error, head.token_count
83
+ ))
84
+ return loss, metrics
tests/test_single_b200.py CHANGED
@@ -101,6 +101,20 @@ class LauncherTests(unittest.TestCase):
101
  with self.assertRaisesRegex(ValueError, 'exact_head'):
102
  self.plan('--algorithm','mp-align-new','--mp-geometry','diag')
103
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
104
  def test_dry_run_creates_no_files(self):
105
  with tempfile.TemporaryDirectory() as folder:
106
  destination = pathlib.Path(folder) / 'absent'
 
101
  with self.assertRaisesRegex(ValueError, 'exact_head'):
102
  self.plan('--algorithm','mp-align-new','--mp-geometry','diag')
103
 
104
+ def test_trust_and_simct_paper_presets_keep_common_workload(self):
105
+ baseline=self.plan('--algorithm','strict-full')['flags']
106
+ for name,mode in [('simct_paper','simct_paper'),('mp-trust-r','mp_opd_trust_r'),
107
+ ('mp-trust-b','mp_opd_trust_b'),('mp-trust-p','mp_opd_trust_p')]:
108
+ flags=self.plan('--algorithm',name)['flags']
109
+ self.assertEqual(flags['kd_algorithm'],mode)
110
+ for key in ('train_batch_size','rollout_batch_size','max_steps','generate_max_len',
111
+ 'learning_rate','eval_steps','eval_math_n_samples','student_name_or_path','train_dataset_path'):
112
+ self.assertEqual(flags[key],baseline[key])
113
+ if name.startswith('mp-trust'):
114
+ self.assertEqual(flags['mp_opd_grass_geometry'],'exact_head')
115
+ with self.assertRaisesRegex(ValueError,'exact_head'):
116
+ self.plan('--algorithm',name,'--mp-geometry','diag')
117
+
118
  def test_dry_run_creates_no_files(self):
119
  with tempfile.TemporaryDirectory() as folder:
120
  destination = pathlib.Path(folder) / 'absent'
tests/test_trust_paper.py ADDED
@@ -0,0 +1,389 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TRUST donor parity, optimizer-window integration and SimCT paper-score checks."""
2
+ import ast
3
+ import copy
4
+ import hashlib
5
+ import importlib.machinery
6
+ import importlib.util
7
+ import json
8
+ import pathlib
9
+ import random
10
+ import sys
11
+ import types
12
+ import unittest
13
+
14
+ import test_mp_opd_port as fixtures
15
+
16
+ ROOT = pathlib.Path(__file__).resolve().parents[1]
17
+ if fixtures.HAS_TORCH:
18
+ import numpy as np
19
+ import torch
20
+
21
+
22
+ def original_trust():
23
+ from kdflow.algorithms._mp_opd_port import _mp_opd_grass_span as span
24
+ from kdflow.algorithms._mp_opd_port import _mp_opd_head_pair as pair
25
+ from kdflow.algorithms._mp_opd_port._mp_opd_grass_chunk import chunk_head_gram
26
+ package = types.ModuleType('_trust_original')
27
+ package.__path__ = []
28
+ sys.modules[package.__name__] = package
29
+ sys.modules['_trust_original._mp_opd_grass_span'] = span
30
+ sys.modules['_trust_original._mp_opd_head_pair'] = pair
31
+ modules = []
32
+ for name in ('_mp_opd_trust','_mp_opd_trust_batch','_mp_opd_trust_p'):
33
+ full = '_trust_original.'+name
34
+ loader = importlib.machinery.SourceFileLoader(full,str(ROOT/'tests/fixtures/trust'/(name+'.py.txt')))
35
+ spec = importlib.util.spec_from_loader(full,loader)
36
+ module = importlib.util.module_from_spec(spec)
37
+ sys.modules[full] = module
38
+ loader.exec_module(module)
39
+ modules.append(module)
40
+ scope = dict(torch=torch,chunk_head_gram=chunk_head_gram,
41
+ **{name:getattr(modules[0],name) for name in ('strict_mask','gram_affordable','trust_calibrate','trust_response_loss','trust_response_metrics')})
42
+ exec(compile((ROOT/'tests/fixtures/trust/_trust_r_loss.py.txt').read_text(),'donor_trust_r','exec'),scope)
43
+ return (*modules,scope['_trust_r_loss'])
44
+
45
+
46
+ @unittest.skipUnless(fixtures.HAS_TORCH,'requires existing PyTorch; no bootstrap')
47
+ class TrustTests(unittest.TestCase):
48
+ errors = []
49
+
50
+ @classmethod
51
+ def tearDownClass(cls):
52
+ if cls.errors:
53
+ print('TRUST_PARITY '+json.dumps({'cases':len(cls.errors),
54
+ 'max_loss_abs_error':max(x[0] for x in cls.errors),
55
+ 'max_gradient_abs_error':max(x[1] for x in cls.errors)}))
56
+
57
+ def fixture(self, mode='trust_b',cap=None):
58
+ from kdflow.algorithms import ALGO_DICT
59
+ helper = fixtures.MPPortTests()
60
+ mp,student,teacher = helper.fixture(cap=cap)
61
+ mp.kd.mp_opd_trust_eps_g = 1e-12
62
+ mp.kd.mp_opd_trust_vocab_chunk = 3
63
+ mp.kd.use_ema_teacher = False
64
+ mp.args.train.use_dynamic_bsz = False
65
+ mp.strategy.accumulated_gradient,mp.strategy.step = 2,0
66
+ algo = ALGO_DICT['mp_opd_'+mode](mp.strategy,student,teacher,fixtures.Tokenizer(),fixtures.Tokenizer())
67
+ return helper,algo,student,teacher
68
+
69
+ def rows(self,algo,student,teacher,batch):
70
+ from kdflow.algorithms._mp_opd_port._mp_opd_credit import build_atom_credits
71
+ sm,tm = batch['stu_loss_mask'],batch['tea_loss_mask']
72
+ hidden = student(batch['stu_input_ids'])['hidden_states'][-1][sm]
73
+ sy,ty = batch['stu_input_ids'].roll(-1,1),batch['tea_input_ids'].roll(-1,1)
74
+ so=to=0
75
+ for row in range(sm.shape[0]):
76
+ sid,tid = sy[row][sm[row]].tolist(),ty[row][tm[row]].tolist()
77
+ atomized = algo.atomizer.atomize(sid,tid,sample_id='donor')
78
+ self.assertTrue(atomized.valid)
79
+ ns,nt = atomized.covered_student_events,atomized.covered_teacher_events
80
+ sh = hidden[so:so+ns]
81
+ sl = student.model.lm_head(sh,skip=False)
82
+ tl = teacher(batch['teacher_hiddens'][to:to+nt]).detach()
83
+ if algo.student_cap is not None:
84
+ sl = torch.tanh(sl/algo.student_cap)*algo.student_cap
85
+ tl = torch.tanh(tl/algo.teacher_cap)*algo.teacher_cap
86
+ labels,tlabels = sy[row][sm[row]][:ns],ty[row][tm[row]][:nt]
87
+ credits = build_atom_credits(atomized.atoms,sl,labels,tl,tlabels)
88
+ yield credits,atomized.atoms,sl,labels,sh
89
+ so,to = so+len(sid),to+len(tid)
90
+
91
+ def reference(self,algo,student,teacher,batches):
92
+ trust,batch_module,p,r_loss = original_trust()
93
+ if algo.mode == 'trust_r':
94
+ class IdentityMap(dict):
95
+ def __missing__(self,key):
96
+ return key
97
+ ref = types.SimpleNamespace(grass_softcap=algo.student_cap,grass_head_bias=False,
98
+ trust_eps_g=algo.eps_g,_TRUST_RESPONSE_METRIC_NAMES=IdentityMap(),
99
+ _trust_static_metrics=lambda *args:{})
100
+ total = student.embedding.weight.sum()*0
101
+ for batch in batches:
102
+ for credits,atoms,logits,labels,hidden in self.rows(algo,student,teacher,batch):
103
+ loss,_ = r_loss(ref,credits,atoms,student_logits=logits,student_labels=labels,student_hidden=hidden)
104
+ total = total+loss/batch['avg_micro_batch_token_num']
105
+ return total,None
106
+ collector = batch_module.TrustHeadAccumulator(softcap=algo.student_cap,
107
+ head_bias=False,vocab_chunk=3,force_fp32=algo.mode=='trust_p')
108
+ with torch.no_grad():
109
+ for batch in batches:
110
+ for credits,atoms,logits,labels,hidden in self.rows(algo,student,teacher,batch):
111
+ mask = trust.strict_mask([a.boundary_type for a in atoms])
112
+ collector.add_response(logits,hidden,labels,[(a.student_start,a.student_end) for a in atoms],
113
+ credits.rate,mask,-credits.student_token_nll)
114
+ result = collector.finalize(eps_g=algo.eps_g,**({'calibrate':p.trust_p_calibrate_scalars} if algo.mode=='trust_p' else {}))
115
+ fn = p.trust_p_response_loss if algo.mode=='trust_p' else trust.trust_response_loss
116
+ total = student.embedding.weight.sum()*0
117
+ for batch in batches:
118
+ for credits,atoms,*_ in self.rows(algo,student,teacher,batch):
119
+ mask = trust.strict_mask([a.boundary_type for a in atoms])
120
+ total = total+fn(credits.current_nll,credits.rate,mask,~mask,result)/batch['avg_micro_batch_token_num']
121
+ return total,result
122
+
123
+ def test_full_loss_and_all_student_gradients_match_donor(self):
124
+ from kdflow.algorithms.mp_opd_trust import restore_rng
125
+ for mode in ('trust_r','trust_b','trust_p'):
126
+ for cap in (None,3.):
127
+ with self.subTest(mode=mode,cap=cap):
128
+ helper,algo,student,teacher = self.fixture(mode,cap)
129
+ batches = [helper.batch(),helper.batch()]
130
+ for b in batches:
131
+ b['avg_micro_batch_token_num'] *= 2
132
+ batches[1]['teacher_hiddens'] = (batches[1]['teacher_hiddens']*1.7).detach().requires_grad_()
133
+ rs,rt = copy.deepcopy(student),copy.deepcopy(teacher)
134
+ prepared = algo.prepare_optimizer_batch(batches) if algo.requires_optimizer_batch else None
135
+ if prepared:
136
+ parts=[]
137
+ for i,b in enumerate(batches):
138
+ restore_rng(prepared['rng_states'][i])
139
+ parts.append(algo.training_step(b)['train/loss'])
140
+ actual = sum(parts)
141
+ else:
142
+ actual = sum(algo.training_step(b)['train/loss'] for b in batches)
143
+ expected,result = self.reference(algo,rs,rt,batches)
144
+ ga = torch.autograd.grad(actual,tuple(student.parameters()))
145
+ gb = torch.autograd.grad(expected,tuple(rs.parameters()))
146
+ torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
147
+ for a,b in zip(ga,gb):
148
+ torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
149
+ if result is not None:
150
+ field = 'strict_scale' if mode=='trust_p' else 'lam'
151
+ self.assertAlmostEqual(getattr(algo._result,field),getattr(result,field),places=6)
152
+ self.errors.append((float((actual-expected).detach().abs()),max(float((a-b).abs().max()) for a,b in zip(ga,gb))))
153
+ teacher_grad = torch.autograd.grad(algo.training_step(batches[0])['train/loss'],
154
+ (*teacher.parameters(),batches[0]['teacher_hiddens']),allow_unused=True)
155
+ self.assertTrue(all(g is None for g in teacher_grad))
156
+ algo.finish_optimizer_batch()
157
+
158
+ def test_full_batch_cross_response_terms_and_projected_p_match_explicit_autograd(self):
159
+ from kdflow.algorithms._mp_opd_port._mp_opd_trust_batch import TrustHeadAccumulator
160
+ from kdflow.algorithms._mp_opd_port._mp_opd_trust_p import trust_p_calibrate_scalars
161
+ weight = torch.zeros(2,1,requires_grad=True)
162
+ rows = [torch.tensor([[1.],[2.]]),torch.tensor([[-.5],[-3.]])]
163
+ losses=[]
164
+ collector = TrustHeadAccumulator(force_fp32=True,vocab_chunk=1)
165
+ for hidden in rows:
166
+ logits = hidden@weight.T
167
+ labels = torch.ones(2,dtype=torch.long)
168
+ nll = -logits.log_softmax(-1)[:,1]
169
+ losses.append(nll)
170
+ collector.add_response(logits,hidden,labels,((0,1),(1,2)),torch.ones(2),
171
+ torch.tensor([True,False]),-nll.detach())
172
+ strict = sum(loss[0] for loss in losses)
173
+ mismatch = sum(loss[1] for loss in losses)
174
+ gs = torch.autograd.grad(strict,weight,retain_graph=True)[0]
175
+ gm = torch.autograd.grad(mismatch,weight,retain_graph=True)[0]
176
+ args = (float(gs.square().sum()),float(gm.square().sum()),float((gs*gm).sum()),2,2)
177
+ result = collector.finalize(calibrate=trust_p_calibrate_scalars)
178
+ original = original_trust()[2].trust_p_calibrate_scalars(*args)
179
+ self.assertAlmostEqual(result.strict_scale,3.,places=5)
180
+ self.assertEqual(result.strict_scale,original.strict_scale)
181
+ projected = result.strict_scale*strict+mismatch
182
+ gradient = torch.autograd.grad(projected,weight)[0]
183
+ torch.testing.assert_close(gradient,result.strict_scale*gs+gm)
184
+ self.assertGreaterEqual(float((gs*gradient).sum()),0.)
185
+ r = original_trust()[0].trust_calibrate_scalars(*args)
186
+ self.assertEqual(r.lam,0.)
187
+ # Per-response averaging would give (1/2+1/6)/2 = 1/3 instead.
188
+ self.assertNotAlmostEqual(r.lam,1/3)
189
+
190
+ def test_missing_window_wrong_boundary_and_normalizers_fail(self):
191
+ helper,algo,*_ = self.fixture()
192
+ with self.assertRaisesRegex(RuntimeError,'prepare_optimizer_batch'):
193
+ algo.training_step(helper.batch())
194
+ with self.assertRaisesRegex(ValueError,'complete optimizer'):
195
+ algo.prepare_optimizer_batch([helper.batch()])
196
+ batches=[helper.batch(),helper.batch()]
197
+ batches[1]['avg_micro_batch_token_num'] *= 2
198
+ with self.assertRaisesRegex(ValueError,'common normalizer'):
199
+ algo.prepare_optimizer_batch(batches)
200
+ self.assertIsNone(algo._result)
201
+ self.assertIsNone(algo._collector)
202
+
203
+ def test_actor_two_pass_rng_replay_and_one_optimizer_update(self):
204
+ from kdflow.algorithms.mp_opd_trust import capture_rng
205
+ # Execute the actual actor dispatch methods without importing Ray/CUDA.
206
+ tree=ast.parse((ROOT/'kdflow/ray/train/student_actor.py').read_text())
207
+ methods={}
208
+ for name in ('fit','evaluate'):
209
+ node=next(n for n in ast.walk(tree) if isinstance(n,ast.FunctionDef) and n.name==name)
210
+ node.decorator_list=[]
211
+ scope={}
212
+ exec(compile(ast.Module(body=[node],type_ignores=[]),'actual_actor_'+name,'exec'),scope)
213
+ methods[name]=scope[name]
214
+ helper,algo,student,teacher = self.fixture('trust_p')
215
+ records=[]
216
+ original_forward=student.forward
217
+ def forward(_self,ids,**kwargs):
218
+ random.random(); np.random.random()
219
+ hidden=original_forward(ids,**kwargs)['hidden_states'][-1]
220
+ hidden=torch.nn.functional.dropout(hidden,p=.2,training=student.training)
221
+ records.append(hidden.detach().clone())
222
+ return {'hidden_states':[hidden]}
223
+ student.forward=types.MethodType(forward,student)
224
+ optimizer=torch.optim.SGD(student.parameters(),lr=.01)
225
+ scheduler=types.SimpleNamespace(get_last_lr=lambda:[.01])
226
+ strategy=algo.strategy
227
+ updates=[]
228
+ def backward(loss,*_):
229
+ strategy.step=(strategy.step+1)%strategy.accumulated_gradient
230
+ (loss/strategy.accumulated_gradient).backward()
231
+ def step(*_):
232
+ if strategy.step==0:
233
+ optimizer.step();optimizer.zero_grad();updates.append(1)
234
+ strategy.backward=backward
235
+ strategy.optimizer_step=step
236
+ strategy.all_reduce=lambda value,**_:value
237
+ actor=types.SimpleNamespace(student=student,args=algo.args,strategy=strategy,kd_algorithm=algo,
238
+ optim=optimizer,scheduler=scheduler,_prepare_micro_batch=lambda batch:dict(batch))
239
+ batches=[helper.batch(),helper.batch()]
240
+ for b in batches:
241
+ b['avg_micro_batch_token_num'] *= 2
242
+ observed={}
243
+ original_prepare=algo.prepare_optimizer_batch
244
+ def prepare(*args,**kwargs):
245
+ result=original_prepare(*args,**kwargs)
246
+ observed['after']=result['after_rng']
247
+ self.assertEqual(updates,[])
248
+ return result
249
+ algo.prepare_optimizer_batch=prepare
250
+ status=methods['fit'](actor,batches)
251
+ self.assertTrue(np.isfinite(status['train/loss']))
252
+ self.assertEqual(len(updates),1)
253
+ self.assertEqual(len(records),4)
254
+ for a,b in zip(records[:2],records[2:]):
255
+ torch.testing.assert_close(a,b)
256
+ after=capture_rng()
257
+ self.assertEqual(after[0],observed['after'][0])
258
+ self.assertTrue(np.array_equal(after[1][1],observed['after'][1][1]))
259
+ self.assertTrue(torch.equal(after[2],observed['after'][2]))
260
+ self.assertIsNone(algo._result)
261
+ algo.prepare_optimizer_batch=original_prepare
262
+ before=[p.detach().clone() for p in student.parameters()]
263
+ methods['evaluate'](actor,batches)
264
+ self.assertTrue(student.training)
265
+ self.assertIsNone(algo._result)
266
+ for a,b in zip(before,student.parameters()):
267
+ torch.testing.assert_close(a,b)
268
+
269
+ def test_only_strict_and_only_mismatch_rules(self):
270
+ trust,_,p,_=original_trust()
271
+ from kdflow.algorithms._mp_opd_port._mp_opd_trust import trust_calibrate_scalars
272
+ from kdflow.algorithms._mp_opd_port._mp_opd_trust_p import trust_p_calibrate_scalars
273
+ for args in ((1.,0.,0.,2,0),(0.,1.,0.,0,2),(0.,0.,0.,0,0),(1.,4.,-2.,1,1)):
274
+ self.assertEqual(trust_calibrate_scalars(*args).lam,trust.trust_calibrate_scalars(*args).lam)
275
+ self.assertEqual(trust_p_calibrate_scalars(*args).strict_scale,p.trust_p_calibrate_scalars(*args).strict_scale)
276
+
277
+ def test_batch_microbatch_regrouping_keeps_loss_and_gradients(self):
278
+ for mode in ('trust_b','trust_p'):
279
+ helper,algo,student,_=self.fixture(mode)
280
+ full=helper.batch(2)
281
+ one=helper.batch()
282
+ one['avg_micro_batch_token_num']*=2
283
+ algo.strategy.accumulated_gradient=1
284
+ algo.prepare_optimizer_batch([full])
285
+ whole=algo.training_step(full)['train/loss']
286
+ gw=torch.autograd.grad(whole,tuple(student.parameters()))
287
+ algo.finish_optimizer_batch()
288
+ algo.strategy.accumulated_gradient=2
289
+ algo.prepare_optimizer_batch([one,one])
290
+ split=algo.training_step(one)['train/loss']+algo.training_step(one)['train/loss']
291
+ gs=torch.autograd.grad(split,tuple(student.parameters()))
292
+ algo.finish_optimizer_batch()
293
+ torch.testing.assert_close(whole,split,atol=1e-6,rtol=1e-5)
294
+ for a,b in zip(gw,gs):torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
295
+
296
+ def test_trust_and_paper_argument_guards(self):
297
+ from kdflow.arguments.distillation_args import DistillationArguments
298
+ for mode in ('mp_opd_trust_r','mp_opd_trust_b','mp_opd_trust_p','simct_paper'):
299
+ options=dict(kd_algorithm=mode,kd_ratio=1.,kd_loss_fn='rkl')
300
+ self.assertEqual(DistillationArguments(**options).kd_algorithm,mode)
301
+ with self.assertRaises(ValueError):
302
+ DistillationArguments(kd_algorithm=mode)
303
+ if mode.startswith('mp_opd_trust_'):
304
+ for extra in ({'mp_opd_grass_geometry':'diag'}, {'mp_opd_trust_eps_g':float('nan')},
305
+ {'mp_opd_trust_vocab_chunk':0}):
306
+ with self.assertRaises(ValueError):DistillationArguments(**options,**extra)
307
+
308
+
309
+ @unittest.skipUnless(fixtures.HAS_TORCH,'requires existing PyTorch; no bootstrap')
310
+ class SimCTPaperTests(unittest.TestCase):
311
+ def pair(self):
312
+ from kdflow.algorithms.simct_paper import SimCTPaper
313
+ helper=fixtures.MPPortTests()
314
+ mp,student,teacher=helper.fixture()
315
+ mp.kd.simct_gh_mask_threshold=0.
316
+ return helper,SimCTPaper(mp.strategy,student,teacher,fixtures.Tokenizer(),fixtures.Tokenizer()),student,teacher
317
+
318
+ @staticmethod
319
+ def manual(upstream,segments,sl,tl,sid,tid):
320
+ rows=[[],[]]
321
+ spans=[i for i,(ts,te,ss,se) in enumerate(segments) if te-ts>1 or se-ss>1]
322
+ for logits,ids,overlap,out in ((sl,sid,upstream.student_overlap_token_ids,rows[0]),
323
+ (tl,tid,upstream.teacher_overlap_token_ids,rows[1])):
324
+ logp=logits.float().log_softmax(-1)
325
+ student=out is rows[0]
326
+ for index,(ts,te,ss,se) in enumerate(segments):
327
+ lo,hi=(ss,se) if student else (ts,te)
328
+ extras=[logp[lo:hi].gather(1,torch.tensor(ids[lo:hi])[:,None]).mean()
329
+ if index==span else logp.new_tensor(-1e9) for span in spans]
330
+ out.append(torch.cat([logp[lo,overlap],torch.stack(extras)]) if spans else logp[lo,overlap])
331
+ return torch.stack(rows[0]),torch.stack(rows[1])
332
+
333
+ def test_virtual_scores_loss_gradients_and_logit_shift_invariance(self):
334
+ _,algo,*_=self.pair()
335
+ torch.manual_seed(7)
336
+ sl=torch.randn(3,8,requires_grad=True);tl=torch.randn(2,8)
337
+ segments=[(0,1,0,2),(1,2,2,3)]
338
+ sid,tid=[1,2,0],[4,0]
339
+ actual=algo.upstream._build_virtual_vocab_logits(segments,sl,tl,sid,tid)
340
+ expected=self.manual(algo.upstream,segments,sl,tl,sid,tid)
341
+ for a,b in zip(actual,expected):torch.testing.assert_close(a,b)
342
+ loss=algo.upstream.loss_fn(*actual,reduction='sum')
343
+ p,q=expected[0].log_softmax(-1),expected[1].log_softmax(-1)
344
+ ref=(p.exp()*(p-q)).sum()
345
+ ga=torch.autograd.grad(loss,sl,retain_graph=True)[0]
346
+ gb=torch.autograd.grad(ref,sl)[0]
347
+ torch.testing.assert_close(loss,ref)
348
+ torch.testing.assert_close(ga,gb)
349
+ shifted_sl=(sl.detach()+torch.tensor([0.,10.,-3.])[:,None]).requires_grad_()
350
+ shifted_tl=tl+torch.tensor([-8.,12.])[:,None]
351
+ shifted=algo.upstream._build_virtual_vocab_logits(segments,shifted_sl,shifted_tl,sid,tid)
352
+ shifted_loss=algo.upstream.loss_fn(*shifted,reduction='sum')
353
+ torch.testing.assert_close(loss,shifted_loss,atol=2e-6,rtol=1e-5)
354
+ torch.testing.assert_close(ga,torch.autograd.grad(shifted_loss,shifted_sl)[0],atol=2e-6,rtol=1e-5)
355
+ torch.testing.assert_close(ga.sum(-1),torch.zeros(3),atol=1e-6,rtol=1e-5)
356
+
357
+ def test_complete_adapter_loss_and_gradient_match_manual_equations(self):
358
+ helper,algo,student,teacher=self.pair()
359
+ rs,rt=copy.deepcopy(student),copy.deepcopy(teacher)
360
+ batch=helper.batch(2)
361
+ actual=algo.training_step(batch)['train/loss']
362
+ sm,tm=batch['stu_loss_mask'],batch['tea_loss_mask']
363
+ sh=rs(batch['stu_input_ids'])['hidden_states'][-1][sm]
364
+ th=batch['teacher_hiddens']
365
+ sy,ty=batch['stu_input_ids'].roll(-1,1),batch['tea_input_ids'].roll(-1,1)
366
+ so=to=0;expected=sh.sum()*0
367
+ for row in range(sm.shape[0]):
368
+ sid,tid=sy[row][sm[row]].tolist(),ty[row][tm[row]].tolist()
369
+ segments,tid,sid=algo.upstream._align_sequences_with_spans(tid,sid)
370
+ sl=rs.model.lm_head(sh[so:so+len(sid)],skip=False)
371
+ tl=rt(th[to:to+len(tid)]).detach()
372
+ sv,tv=self.manual(algo.upstream,segments,sl,tl,sid,tid)
373
+ p,q=sv.log_softmax(-1),tv.log_softmax(-1)
374
+ expected=expected+(p.exp()*(p-q)).sum()/batch['avg_micro_batch_token_num']
375
+ so,to=so+len(sid),to+len(tid)
376
+ torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
377
+ for a,b in zip(torch.autograd.grad(actual,tuple(student.parameters())),torch.autograd.grad(expected,tuple(rs.parameters()))):
378
+ torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
379
+
380
+
381
+ class TrustPaperSourceTests(unittest.TestCase):
382
+ def test_source_hashes_and_frozen_existing_modes(self):
383
+ manifest=json.loads((ROOT/'docs/TRUST_PAPER_SOURCE.json').read_text())
384
+ for row in manifest['files']+manifest['unchanged_existing_files']:
385
+ self.assertEqual(hashlib.sha256((ROOT/row['path']).read_bytes()).hexdigest(),row['sha256'])
386
+
387
+
388
+ if __name__=='__main__':
389
+ unittest.main()
tools/launch_single_b200.py CHANGED
@@ -25,9 +25,10 @@ def parser():
25
  default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
26
  p.add_argument('--runtime', type=pathlib.Path,
27
  help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
28
- p.add_argument('--algorithm', choices=['strict-top16', 'strict-full', 'simct', 'mp-atomic',
29
  'mp-fixed1', 'mp-fixed2', 'mp-fixed3', 'mp-fixed4', 'mp-fixed5',
30
- 'mp-random1-2', 'mp-random2-5', 'mp-grass-chunk', 'mp-align', 'mp-align-new'], default='strict-top16')
 
31
  p.add_argument('--mp-chunk-source', choices=['xtoken', 'run'], default='xtoken')
32
  p.add_argument('--mp-chunk-run-length', type=int, default=2)
33
  p.add_argument('--mp-geometry', choices=['exact_head', 'diag'], default='exact_head')
@@ -42,8 +43,8 @@ def parser():
42
 
43
 
44
  def build_plan(args):
45
- if args.algorithm == 'mp-align-new' and args.mp_geometry != 'exact_head':
46
- raise ValueError('ALIGN-NEW requires exact_head geometry')
47
  if args.concurrency < 1 or args.eval_timeout < 0 or args.mp_chunk_run_length < 1:
48
  raise ValueError('Concurrency must be positive and eval timeout nonnegative')
49
  run_id = args.run_id or (datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%dT%H%M%S%fZ')
@@ -82,8 +83,8 @@ def build_plan(args):
82
  'teacher_mem_fraction_static': 0.6, 'teacher_offload_tags': 'all',
83
  'chunked_loss_size': 2048, 'use_wandb': False,
84
  }
85
- if args.algorithm == 'simct':
86
- flags.update(kd_algorithm='simct', simct_gh_mask_threshold=0.0)
87
  if args.algorithm.startswith('mp-'):
88
  name = args.algorithm[3:]
89
  family = 'fixed' if name.startswith('fixed') else 'random' if name.startswith('random') else name.replace('-', '_')
@@ -100,6 +101,8 @@ def build_plan(args):
100
  mp_opd_grass_geometry=args.mp_geometry)
101
  if family == 'align_new':
102
  flags.update(mp_opd_align_new_min_descent_fraction=0.1, mp_opd_align_new_eps_g=1e-12)
 
 
103
  command = ['bash', str(runtime / 'python-b200.sh'), '-u', '-m', 'kdflow.cli.train_kd_on_policy']
104
  for key, value in flags.items():
105
  command.extend(['--' + key, str(value)])
@@ -107,7 +110,9 @@ def build_plan(args):
107
  'source': str(SOURCE), 'flags': flags, 'command': command,
108
  'gradient_accumulation_microsteps': 512 // args.micro_batch,
109
  'expected_runtime_archive_sha256': ARCHIVE_SHA,
110
- 'qualification': ('author-code SimCT port; no B200/full-training qualification'
 
 
111
  if args.algorithm == 'simct' else
112
  'experimental MP-OPD source port; no B200/full-training qualification'
113
  if args.algorithm.startswith('mp-') else
 
25
  default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
26
  p.add_argument('--runtime', type=pathlib.Path,
27
  help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
28
+ p.add_argument('--algorithm', choices=['strict-top16', 'strict-full', 'simct', 'simct_paper', 'mp-atomic',
29
  'mp-fixed1', 'mp-fixed2', 'mp-fixed3', 'mp-fixed4', 'mp-fixed5',
30
+ 'mp-random1-2', 'mp-random2-5', 'mp-grass-chunk', 'mp-align', 'mp-align-new',
31
+ 'mp-trust-r','mp-trust-b','mp-trust-p'], default='strict-top16')
32
  p.add_argument('--mp-chunk-source', choices=['xtoken', 'run'], default='xtoken')
33
  p.add_argument('--mp-chunk-run-length', type=int, default=2)
34
  p.add_argument('--mp-geometry', choices=['exact_head', 'diag'], default='exact_head')
 
43
 
44
 
45
  def build_plan(args):
46
+ if (args.algorithm == 'mp-align-new' or args.algorithm.startswith('mp-trust-')) and args.mp_geometry != 'exact_head':
47
+ raise ValueError('ALIGN-NEW/TRUST requires exact_head geometry')
48
  if args.concurrency < 1 or args.eval_timeout < 0 or args.mp_chunk_run_length < 1:
49
  raise ValueError('Concurrency must be positive and eval timeout nonnegative')
50
  run_id = args.run_id or (datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%dT%H%M%S%fZ')
 
83
  'teacher_mem_fraction_static': 0.6, 'teacher_offload_tags': 'all',
84
  'chunked_loss_size': 2048, 'use_wandb': False,
85
  }
86
+ if args.algorithm in {'simct','simct_paper'}:
87
+ flags.update(kd_algorithm=args.algorithm, simct_gh_mask_threshold=0.0)
88
  if args.algorithm.startswith('mp-'):
89
  name = args.algorithm[3:]
90
  family = 'fixed' if name.startswith('fixed') else 'random' if name.startswith('random') else name.replace('-', '_')
 
101
  mp_opd_grass_geometry=args.mp_geometry)
102
  if family == 'align_new':
103
  flags.update(mp_opd_align_new_min_descent_fraction=0.1, mp_opd_align_new_eps_g=1e-12)
104
+ if family.startswith('trust_'):
105
+ flags.update(mp_opd_grass_geometry='exact_head',mp_opd_trust_eps_g=1e-12,mp_opd_trust_vocab_chunk=4096)
106
  command = ['bash', str(runtime / 'python-b200.sh'), '-u', '-m', 'kdflow.cli.train_kd_on_policy']
107
  for key, value in flags.items():
108
  command.extend(['--' + key, str(value)])
 
110
  'source': str(SOURCE), 'flags': flags, 'command': command,
111
  'gradient_accumulation_microsteps': 512 // args.micro_batch,
112
  'expected_runtime_archive_sha256': ARCHIVE_SHA,
113
+ 'qualification': ('paper-score SimCT reconstruction; no B200/full-training qualification'
114
+ if args.algorithm == 'simct_paper' else
115
+ 'author-code SimCT port; no B200/full-training qualification'
116
  if args.algorithm == 'simct' else
117
  'experimental MP-OPD source port; no B200/full-training qualification'
118
  if args.algorithm.startswith('mp-') else