fix(mp-opd): exclude unalignable responses with reason metrics
Browse files- README.md +3 -0
- docs/INVALID_RESPONSES.md +55 -0
- docs/MP_OPD.md +11 -2
- docs/MP_OPD_INVALID_RESPONSES.json +10 -0
- docs/MP_OPD_INVALID_RESPONSES_VALIDATION.json +29 -0
- kdflow/algorithms/mp_opd.py +25 -1
- tests/mp_opd_source_checks.py +12 -0
- tests/test_align_new.py +2 -1
- tests/test_invalid_response.py +174 -0
- tests/test_mp_opd_port.py +3 -3
- tests/test_simct.py +2 -1
- tests/test_trust_paper.py +2 -1
README.md
CHANGED
|
@@ -16,6 +16,9 @@ An opt-in [MP-OPD port](docs/MP_OPD.md) adds Atomic, Fixed1–5, Random1–2/2
|
|
| 16 |
GRASS-Chunk and ALIGN as separate modes. The existing Strict algorithm sources
|
| 17 |
remain unchanged. The new modes use sampled atom credits rather than Strict KL;
|
| 18 |
they have CPU verification, with B200/full training qualification still pending.
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
The separate [ALIGN-NEW port](docs/ALIGN_NEW.md), `--algorithm mp-align-new`, adds
|
| 21 |
the donor's full-response head-descent guard while keeping `mp-align` v1 unchanged.
|
|
|
|
| 16 |
GRASS-Chunk and ALIGN as separate modes. The existing Strict algorithm sources
|
| 17 |
remain unchanged. The new modes use sampled atom credits rather than Strict KL;
|
| 18 |
they have CPU verification, with B200/full training qualification still pending.
|
| 19 |
+
The shared [invalid-response policy](docs/INVALID_RESPONSES.md) excludes complete
|
| 20 |
+
responses whose final decode contains U+FFFD or whose student/teacher bytes differ,
|
| 21 |
+
with exclusion metrics and the original shared token denominator.
|
| 22 |
|
| 23 |
The separate [ALIGN-NEW port](docs/ALIGN_NEW.md), `--algorithm mp-align-new`, adds
|
| 24 |
the donor's full-response head-descent guard while keeping `mp-align` v1 unchanged.
|
docs/INVALID_RESPONSES.md
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Invalid decoded responses in MP-OPD
|
| 2 |
+
|
| 3 |
+
The 2026-10-19 company logs showed `replacement_character` in Fixed1 and
|
| 4 |
+
`normalization_mismatch` in GRASS-Chunk and ALIGN. Their root exceptions occurred
|
| 5 |
+
in the shared MP adapter; the displayed SIGTERM messages accompanied Ray cleanup.
|
| 6 |
+
Those logs do not contain the decoded strings needed to identify the underlying
|
| 7 |
+
generation/retokenization discrepancy. This patch mitigates the crash by excluding
|
| 8 |
+
responses that cannot be scored safely on synchronized byte spans.
|
| 9 |
+
|
| 10 |
+
## Policy
|
| 11 |
+
|
| 12 |
+
All MP modes, including ALIGN-NEW and TRUST-R/B/P, exclude the complete response
|
| 13 |
+
for `replacement_character`, `normalization_mismatch` or `empty_response`.
|
| 14 |
+
Student and teacher hidden-state offsets still advance by their full response
|
| 15 |
+
lengths. No text is rewritten and no alternative Atomic loss is substituted.
|
| 16 |
+
Other failures, including unsupported added tokens, remain fatal. The vendored
|
| 17 |
+
atomizer and all geometry/loss kernels are unchanged.
|
| 18 |
+
|
| 19 |
+
The existing shared optimizer-window denominator is preserved, including excluded
|
| 20 |
+
tokens. This avoids changing the weighting of valid responses according to how
|
| 21 |
+
they happen to be grouped into microbatches. Exclusion changes the effective
|
| 22 |
+
sample distribution and reduces gradient magnitude relative to processing every
|
| 23 |
+
token; it is not mathematical equivalence to training on the original batch.
|
| 24 |
+
An entirely excluded microbatch returns connected zero loss. That alone does not
|
| 25 |
+
guarantee unchanged optimizer state or weights across an entire optimizer update.
|
| 26 |
+
|
| 27 |
+
Metrics use the prefix `distill/mp_opd_`:
|
| 28 |
+
|
| 29 |
+
- `attempted_responses`, `valid_responses`, `skipped_responses`;
|
| 30 |
+
- `skipped_replacement_responses`, `skipped_normalization_responses`,
|
| 31 |
+
`skipped_empty_responses`;
|
| 32 |
+
- `skipped_replacement_fraction`, `skipped_normalization_fraction`;
|
| 33 |
+
- `skipped_student_loss_tokens`, `skipped_teacher_loss_tokens`,
|
| 34 |
+
`skipped_student_token_fraction`.
|
| 35 |
+
|
| 36 |
+
These are microbatch counts/fractions. Existing training logging aggregates
|
| 37 |
+
microbatches; a displayed count may therefore be an average rather than a run
|
| 38 |
+
total. Record exclusions when interpreting comparisons, especially if their
|
| 39 |
+
frequency differs by method. The policy does not affect `simct`, `simct_paper`,
|
| 40 |
+
or Strict.
|
| 41 |
+
|
| 42 |
+
## Verification and provenance
|
| 43 |
+
|
| 44 |
+
`tests/test_invalid_response.py` checks whole-response exclusion, hidden-state
|
| 45 |
+
offsets, the original denominator, valid-response loss/all student gradients,
|
| 46 |
+
all-excluded backward, transient UTF-8 prefix recovery, failure guards, and
|
| 47 |
+
TRUST-B/P calibration/replay. Historical source manifests are retained;
|
| 48 |
+
`MP_OPD_INVALID_RESPONSES.json` explicitly records the patched adapter hash.
|
| 49 |
+
CPU checks do not qualify B200/FSDP execution or full training. The company logs
|
| 50 |
+
were inputs to diagnosis, not successful execution evidence for this patch.
|
| 51 |
+
|
| 52 |
+
Only source changes are needed; the runtime archive does not need rebuilding.
|
| 53 |
+
Use a fresh source checkout and a new run ID for failed runs. Existing Ray actors
|
| 54 |
+
carry a packaged source snapshot, so updating files cannot repair an already
|
| 55 |
+
running actor. Preserve active jobs' checkouts and prior run outputs.
|
docs/MP_OPD.md
CHANGED
|
@@ -88,8 +88,17 @@ linear heads. Token tiles use checkpointed log-prob computation; changing
|
|
| 88 |
still materializes per-response logits and an atom Gram, so long-response GPU
|
| 89 |
memory/throughput remain unqualified. No new wheels or native extension are added.
|
| 90 |
|
| 91 |
-
|
| 92 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
leave GRASS/ALIGN unchanged; inspect singleton fractions and coefficient-change
|
| 94 |
metrics before claiming an intervention occurred.
|
| 95 |
|
|
|
|
| 88 |
still materializes per-response logits and an atom Gram, so long-response GPU
|
| 89 |
memory/throughput remain unqualified. No new wheels or native extension are added.
|
| 90 |
|
| 91 |
+
Complete responses with `replacement_character`, `normalization_mismatch` or
|
| 92 |
+
`empty_response` contribute zero. The adapter never sanitizes/relabels their text
|
| 93 |
+
or substitutes an Atomic loss. All other atomization failures remain fatal.
|
| 94 |
+
The shared optimizer-window token denominator is retained, including excluded
|
| 95 |
+
tokens; remaining responses are not reweighted per microbatch. Exclusion counts,
|
| 96 |
+
reason fractions and excluded token counts are logged under `distill/mp_opd_*`.
|
| 97 |
+
This policy changes the effective training sample distribution and can reduce
|
| 98 |
+
gradient magnitude; inspect exclusions before comparing results. It prevents
|
| 99 |
+
unsafe cross-text scoring but does not repair the source of retokenization mismatch.
|
| 100 |
+
See [INVALID_RESPONSES.md](INVALID_RESPONSES.md) for the policy and qualification.
|
| 101 |
+
Singleton alignment chunks
|
| 102 |
leave GRASS/ALIGN unchanged; inspect singleton fractions and coefficient-change
|
| 103 |
metrics before claiming an intervention occurred.
|
| 104 |
|
docs/MP_OPD_INVALID_RESPONSES.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"date": "2026-10-19",
|
| 3 |
+
"baseline_commit": "619cff667971e65d8cc9e32cf385eba9978ca479",
|
| 4 |
+
"path": "kdflow/algorithms/mp_opd.py",
|
| 5 |
+
"baseline_sha256": "b2387d7f36502dea16c28f4391af4bc97a7e77f367d35dea2867c744e91277a6",
|
| 6 |
+
"updated_sha256": "f687e45395ed6eb887e27503b7f39895f927e5dfa075e8991dd200cfb78271f1",
|
| 7 |
+
"policy": "Exclude complete responses with replacement_character, normalization_mismatch or empty_response; preserve common token denominator; count exclusions by reason; all other atomization failures remain fatal",
|
| 8 |
+
"scope": "MP-OPD family including Fixed/Random/GRASS/ALIGN/ALIGN-NEW/TRUST-R/B/P; author SimCT and simct_paper unchanged",
|
| 9 |
+
"qualification": "83 CPU tests passed with no skips in 80.283 seconds; no company-node or training qualification"
|
| 10 |
+
}
|
docs/MP_OPD_INVALID_RESPONSES_VALIDATION.json
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"date": "2026-10-19",
|
| 3 |
+
"platform": "WSL Linux, CPU only",
|
| 4 |
+
"python": "3.12.3",
|
| 5 |
+
"torch": "2.14.0+cpu",
|
| 6 |
+
"command": "TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v",
|
| 7 |
+
"full_suite": {"passed": 83, "skipped": 0, "elapsed_seconds": 80.283},
|
| 8 |
+
"regressions": [
|
| 9 |
+
"Mixed replacement response exclusion with correct teacher offsets and original denominator",
|
| 10 |
+
"Normalization mismatch exclusion and valid-response loss/all student gradient equality across MP modes and ALIGN-NEW",
|
| 11 |
+
"TRUST-R/B/P replacement and normalization exclusion, B/P full-window calibration, valid-response loss/all student gradient equality",
|
| 12 |
+
"All-replacement microbatch connected zero loss and backward",
|
| 13 |
+
"Transient replacement prefix recovery remains accepted",
|
| 14 |
+
"Unsupported added tokens remain fatal; no Atomic fallback",
|
| 15 |
+
"Historical source manifests retained with explicit adapter patch hash"
|
| 16 |
+
],
|
| 17 |
+
"unchanged_donor_parity": {
|
| 18 |
+
"trust_max_loss_abs_error": 0.0,
|
| 19 |
+
"trust_max_gradient_abs_error": 5.960464477539063e-08,
|
| 20 |
+
"simct_max_loss_abs_error": 0.0,
|
| 21 |
+
"simct_max_student_gradient_abs_error": 3.725290298461914e-09
|
| 22 |
+
},
|
| 23 |
+
"not_qualified": [
|
| 24 |
+
"Root cause of observed company generation/retokenization mismatch",
|
| 25 |
+
"Frequency of excluded responses or effect on scientific comparisons",
|
| 26 |
+
"B200/FSDP execution of this patch or full training",
|
| 27 |
+
"Full training resume or unchanged Adam update when an entire window is excluded"
|
| 28 |
+
]
|
| 29 |
+
}
|
kdflow/algorithms/mp_opd.py
CHANGED
|
@@ -203,6 +203,8 @@ class MPOnPolicyKD:
|
|
| 203 |
student_labels = micro_batch['stu_input_ids'].roll(-1, dims=1)
|
| 204 |
teacher_labels = micro_batch['tea_input_ids'].roll(-1, dims=1)
|
| 205 |
total, metrics, responses, atom_count = hidden.sum() * 0.0, {}, 0, 0
|
|
|
|
|
|
|
| 206 |
student_offset = teacher_offset = 0
|
| 207 |
for row in range(student_mask.shape[0]):
|
| 208 |
sy, ty = student_labels[row][student_mask[row]], teacher_labels[row][teacher_mask[row]]
|
|
@@ -213,7 +215,14 @@ class MPOnPolicyKD:
|
|
| 213 |
sid, tid = sy.detach().cpu().tolist(), ty.detach().cpu().tolist()
|
| 214 |
atomized = self.atomizer.atomize(sid, tid, sample_id=str(response_seed(0, sid, tid)))
|
| 215 |
if not atomized.valid:
|
| 216 |
-
if atomized.failure_reason
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 217 |
continue
|
| 218 |
raise ValueError('MP-OPD atomization failed: ' + atomized.failure_reason)
|
| 219 |
atoms = atomized.atoms
|
|
@@ -248,6 +257,21 @@ class MPOnPolicyKD:
|
|
| 248 |
info = {'train/loss': loss, 'train/kd_loss': loss,
|
| 249 |
'distill/mp_opd_valid_responses': loss.detach().new_tensor(float(responses)),
|
| 250 |
'distill/mp_opd_atoms': loss.detach().new_tensor(float(atom_count))}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 251 |
info.update({key: value / max(responses, 1) for key, value in metrics.items()})
|
| 252 |
return info
|
| 253 |
|
|
|
|
| 203 |
student_labels = micro_batch['stu_input_ids'].roll(-1, dims=1)
|
| 204 |
teacher_labels = micro_batch['tea_input_ids'].roll(-1, dims=1)
|
| 205 |
total, metrics, responses, atom_count = hidden.sum() * 0.0, {}, 0, 0
|
| 206 |
+
skipped_replacement = skipped_normalization = skipped_empty = 0
|
| 207 |
+
skipped_student_tokens = skipped_teacher_tokens = 0
|
| 208 |
student_offset = teacher_offset = 0
|
| 209 |
for row in range(student_mask.shape[0]):
|
| 210 |
sy, ty = student_labels[row][student_mask[row]], teacher_labels[row][teacher_mask[row]]
|
|
|
|
| 215 |
sid, tid = sy.detach().cpu().tolist(), ty.detach().cpu().tolist()
|
| 216 |
atomized = self.atomizer.atomize(sid, tid, sample_id=str(response_seed(0, sid, tid)))
|
| 217 |
if not atomized.valid:
|
| 218 |
+
if atomized.failure_reason in {'empty_response', 'replacement_character', 'normalization_mismatch'}:
|
| 219 |
+
# Exclude the whole unalignable response, retaining the shared
|
| 220 |
+
# optimizer-window denominator. Never sanitize or relabel tokens.
|
| 221 |
+
skipped_replacement += int(atomized.failure_reason == 'replacement_character')
|
| 222 |
+
skipped_normalization += int(atomized.failure_reason == 'normalization_mismatch')
|
| 223 |
+
skipped_empty += int(atomized.failure_reason == 'empty_response')
|
| 224 |
+
skipped_student_tokens += sy.numel()
|
| 225 |
+
skipped_teacher_tokens += ty.numel()
|
| 226 |
continue
|
| 227 |
raise ValueError('MP-OPD atomization failed: ' + atomized.failure_reason)
|
| 228 |
atoms = atomized.atoms
|
|
|
|
| 257 |
info = {'train/loss': loss, 'train/kd_loss': loss,
|
| 258 |
'distill/mp_opd_valid_responses': loss.detach().new_tensor(float(responses)),
|
| 259 |
'distill/mp_opd_atoms': loss.detach().new_tensor(float(atom_count))}
|
| 260 |
+
attempted = student_mask.shape[0]
|
| 261 |
+
exclusions = {
|
| 262 |
+
'attempted_responses': attempted,
|
| 263 |
+
'skipped_replacement_responses': skipped_replacement,
|
| 264 |
+
'skipped_normalization_responses': skipped_normalization,
|
| 265 |
+
'skipped_empty_responses': skipped_empty,
|
| 266 |
+
'skipped_responses': skipped_replacement + skipped_normalization + skipped_empty,
|
| 267 |
+
'skipped_replacement_fraction': skipped_replacement / max(attempted, 1),
|
| 268 |
+
'skipped_normalization_fraction': skipped_normalization / max(attempted, 1),
|
| 269 |
+
'skipped_student_loss_tokens': skipped_student_tokens,
|
| 270 |
+
'skipped_teacher_loss_tokens': skipped_teacher_tokens,
|
| 271 |
+
'skipped_student_token_fraction': skipped_student_tokens / max(int(student_mask.sum()), 1),
|
| 272 |
+
}
|
| 273 |
+
info.update({'distill/mp_opd_' + key: loss.detach().new_tensor(float(value))
|
| 274 |
+
for key, value in exclusions.items()})
|
| 275 |
info.update({key: value / max(responses, 1) for key, value in metrics.items()})
|
| 276 |
return info
|
| 277 |
|
tests/mp_opd_source_checks.py
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Keep historical freeze manifests while auditing the explicitly patched adapter."""
|
| 2 |
+
import hashlib
|
| 3 |
+
import json
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
def assert_source_hash(case, root, row):
|
| 7 |
+
expected = row['sha256']
|
| 8 |
+
if row['path'] == 'kdflow/algorithms/mp_opd.py':
|
| 9 |
+
patch = json.loads((root / 'docs/MP_OPD_INVALID_RESPONSES.json').read_text())
|
| 10 |
+
case.assertEqual(expected, patch['baseline_sha256'])
|
| 11 |
+
expected = patch['updated_sha256']
|
| 12 |
+
case.assertEqual(hashlib.sha256((root / row['path']).read_bytes()).hexdigest(), expected)
|
tests/test_align_new.py
CHANGED
|
@@ -9,6 +9,7 @@ import pathlib
|
|
| 9 |
import sys
|
| 10 |
import types
|
| 11 |
import unittest
|
|
|
|
| 12 |
|
| 13 |
import test_mp_opd_port as fixtures
|
| 14 |
|
|
@@ -238,7 +239,7 @@ class AlignNewSourceTests(unittest.TestCase):
|
|
| 238 |
def test_source_hashes_and_unchanged_existing_algorithms(self):
|
| 239 |
manifest = json.loads((ROOT / 'docs/ALIGN_NEW_SOURCE.json').read_text())
|
| 240 |
for row in manifest['files'] + manifest['unchanged_existing_files']:
|
| 241 |
-
|
| 242 |
|
| 243 |
|
| 244 |
if __name__ == '__main__':
|
|
|
|
| 9 |
import sys
|
| 10 |
import types
|
| 11 |
import unittest
|
| 12 |
+
from mp_opd_source_checks import assert_source_hash
|
| 13 |
|
| 14 |
import test_mp_opd_port as fixtures
|
| 15 |
|
|
|
|
| 239 |
def test_source_hashes_and_unchanged_existing_algorithms(self):
|
| 240 |
manifest = json.loads((ROOT / 'docs/ALIGN_NEW_SOURCE.json').read_text())
|
| 241 |
for row in manifest['files'] + manifest['unchanged_existing_files']:
|
| 242 |
+
assert_source_hash(self, ROOT, row)
|
| 243 |
|
| 244 |
|
| 245 |
if __name__ == '__main__':
|
tests/test_invalid_response.py
ADDED
|
@@ -0,0 +1,174 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Regression for company Fixed1/GRASS/ALIGN invalid-response crashes."""
|
| 2 |
+
import hashlib
|
| 3 |
+
import copy
|
| 4 |
+
import json
|
| 5 |
+
import pathlib
|
| 6 |
+
import unittest
|
| 7 |
+
|
| 8 |
+
import test_mp_opd_port as fixtures
|
| 9 |
+
|
| 10 |
+
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
| 11 |
+
if fixtures.HAS_TORCH:
|
| 12 |
+
import torch
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
class ReplacementTokenizer(fixtures.Tokenizer):
|
| 16 |
+
pieces = {**fixtures.Tokenizer.pieces, 5: '\ufffd'}
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
class TransientPrefixTokenizer(fixtures.Tokenizer):
|
| 20 |
+
def decode(self, ids, **kwargs):
|
| 21 |
+
if list(ids) == [1]:
|
| 22 |
+
return '\ufffd' # Incomplete byte prefix which recovers in full decode.
|
| 23 |
+
return super().decode(ids, **kwargs)
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
@unittest.skipUnless(fixtures.HAS_TORCH, 'requires existing PyTorch; no bootstrap')
|
| 27 |
+
class InvalidResponseTests(unittest.TestCase):
|
| 28 |
+
def fixture(self, mode='mp_opd_fixed'):
|
| 29 |
+
helper = fixtures.MPPortTests()
|
| 30 |
+
algo, student, teacher = helper.fixture(mode)
|
| 31 |
+
algo.kd.mp_opd_fixed_span_length = 1
|
| 32 |
+
algo.atomizer.student_tokenizer = ReplacementTokenizer()
|
| 33 |
+
return helper, algo, student, teacher
|
| 34 |
+
|
| 35 |
+
def test_fixed1_mixed_batch_loss_gradients_offsets_and_denominator(self):
|
| 36 |
+
helper, algo, student, _ = self.fixture()
|
| 37 |
+
batch = helper.batch(3)
|
| 38 |
+
batch['stu_input_ids'][0,1] = 5
|
| 39 |
+
# Give each response distinct teacher states to catch offset errors.
|
| 40 |
+
width = int(batch['tea_loss_mask'][0].sum())
|
| 41 |
+
batch['teacher_hiddens'] = (batch['teacher_hiddens'].detach()
|
| 42 |
+
* torch.tensor([1.,2.,3.]).repeat_interleave(width)[:,None]).requires_grad_()
|
| 43 |
+
actual = algo.training_step(batch)
|
| 44 |
+
clean = helper.batch(2)
|
| 45 |
+
clean['avg_micro_batch_token_num'] = batch['avg_micro_batch_token_num']
|
| 46 |
+
clean['teacher_hiddens'] = batch['teacher_hiddens'][width:]
|
| 47 |
+
expected = algo.training_step(clean)['train/loss']
|
| 48 |
+
torch.testing.assert_close(actual['train/loss'], expected)
|
| 49 |
+
actual_grads=torch.autograd.grad(actual['train/loss'],tuple(student.parameters()))
|
| 50 |
+
expected_grads=torch.autograd.grad(expected,tuple(student.parameters()))
|
| 51 |
+
for a,b in zip(actual_grads,expected_grads):
|
| 52 |
+
torch.testing.assert_close(a,b)
|
| 53 |
+
self.assertEqual(float(actual['distill/mp_opd_valid_responses']),2.)
|
| 54 |
+
self.assertEqual(float(actual['distill/mp_opd_attempted_responses']),3.)
|
| 55 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_replacement_responses']),1.)
|
| 56 |
+
self.assertAlmostEqual(float(actual['distill/mp_opd_skipped_replacement_fraction']),1/3,places=6)
|
| 57 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_student_loss_tokens']),10.)
|
| 58 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_teacher_loss_tokens']),7.)
|
| 59 |
+
self.assertAlmostEqual(float(actual['distill/mp_opd_skipped_student_token_fraction']),1/3,places=6)
|
| 60 |
+
# A removed response has no surrogate gradient, including its unique token.
|
| 61 |
+
self.assertFalse(bool(actual_grads[0][5].any()))
|
| 62 |
+
|
| 63 |
+
def test_all_replacement_responses_produce_finite_zero_and_backward(self):
|
| 64 |
+
helper,algo,student,_ = self.fixture()
|
| 65 |
+
batch=helper.batch(2)
|
| 66 |
+
batch['stu_input_ids'][:,1]=5
|
| 67 |
+
result=algo.training_step(batch)
|
| 68 |
+
self.assertEqual(float(result['train/loss'].detach()),0.)
|
| 69 |
+
self.assertEqual(float(result['distill/mp_opd_valid_responses']),0.)
|
| 70 |
+
self.assertEqual(float(result['distill/mp_opd_skipped_replacement_fraction']),1.)
|
| 71 |
+
result['train/loss'].backward()
|
| 72 |
+
self.assertTrue(all(p.grad is None or not bool(p.grad.any()) for p in student.parameters()))
|
| 73 |
+
|
| 74 |
+
def test_existing_mp_modes_skip_but_other_alignment_errors_stay_fatal(self):
|
| 75 |
+
for mode in fixtures.registry:
|
| 76 |
+
helper,algo,student,_=self.fixture(mode)
|
| 77 |
+
batch=helper.batch(2)
|
| 78 |
+
batch['stu_input_ids'][0,1]=5
|
| 79 |
+
result=algo.training_step(batch)
|
| 80 |
+
self.assertEqual(float(result['distill/mp_opd_skipped_replacement_responses']),1.)
|
| 81 |
+
self.assertTrue(all(bool(torch.isfinite(value)) for value in result.values()))
|
| 82 |
+
result['train/loss'].backward()
|
| 83 |
+
self.assertTrue(any(p.grad is not None for p in student.parameters()))
|
| 84 |
+
helper,algo,*_=self.fixture()
|
| 85 |
+
bad=helper.batch()
|
| 86 |
+
bad['tea_input_ids'][0,1]=7 # Unsupported added token remains fatal.
|
| 87 |
+
with self.assertRaisesRegex(ValueError,'unsupported_added_token'):
|
| 88 |
+
algo.training_step(bad)
|
| 89 |
+
|
| 90 |
+
def test_normalization_mismatch_loss_and_gradients_for_all_mp_modes(self):
|
| 91 |
+
import test_align_new
|
| 92 |
+
for mode in (*fixtures.registry, 'align_new'):
|
| 93 |
+
with self.subTest(mode=mode):
|
| 94 |
+
if mode == 'align_new':
|
| 95 |
+
_,algo,_,_,_,_=test_align_new.AlignNewTests().setup_pair()
|
| 96 |
+
helper=fixtures.MPPortTests()
|
| 97 |
+
student=algo.student
|
| 98 |
+
else:
|
| 99 |
+
helper,algo,student,_=self.fixture(mode)
|
| 100 |
+
batch=helper.batch(2)
|
| 101 |
+
batch['tea_input_ids'][0,1]=6
|
| 102 |
+
# Distinct hidden rows ensure a rejected first row cannot shift
|
| 103 |
+
# the teacher states used by the next valid response.
|
| 104 |
+
width=int(batch['tea_loss_mask'][0].sum())
|
| 105 |
+
batch['teacher_hiddens']=batch['teacher_hiddens'].detach().clone()
|
| 106 |
+
batch['teacher_hiddens'][:width]*=3
|
| 107 |
+
state=copy.deepcopy(algo.state_dict())
|
| 108 |
+
actual=algo.training_step(batch)
|
| 109 |
+
clean=helper.batch()
|
| 110 |
+
clean['avg_micro_batch_token_num']=batch['avg_micro_batch_token_num']
|
| 111 |
+
clean['teacher_hiddens']=batch['teacher_hiddens'][width:]
|
| 112 |
+
algo.load_state_dict(state)
|
| 113 |
+
expected=algo.training_step(clean)['train/loss']
|
| 114 |
+
torch.testing.assert_close(actual['train/loss'],expected)
|
| 115 |
+
for a,b in zip(torch.autograd.grad(actual['train/loss'],tuple(student.parameters())),
|
| 116 |
+
torch.autograd.grad(expected,tuple(student.parameters()))):
|
| 117 |
+
torch.testing.assert_close(a,b)
|
| 118 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_normalization_responses']),1.)
|
| 119 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_replacement_responses']),0.)
|
| 120 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_normalization_fraction']),.5)
|
| 121 |
+
self.assertEqual(float(actual['distill/mp_opd_skipped_responses']),1.)
|
| 122 |
+
|
| 123 |
+
def test_transient_replacement_prefix_is_not_excluded(self):
|
| 124 |
+
helper,algo,*_=self.fixture()
|
| 125 |
+
algo.atomizer.student_tokenizer=TransientPrefixTokenizer()
|
| 126 |
+
result=algo.training_step(helper.batch())
|
| 127 |
+
self.assertEqual(float(result['distill/mp_opd_valid_responses']),1.)
|
| 128 |
+
self.assertEqual(float(result['distill/mp_opd_skipped_responses']),0.)
|
| 129 |
+
|
| 130 |
+
def test_align_new_and_trust_full_window_use_same_exclusion(self):
|
| 131 |
+
import test_align_new
|
| 132 |
+
import test_trust_paper
|
| 133 |
+
_,align,_,_,_,_=test_align_new.AlignNewTests().setup_pair()
|
| 134 |
+
align.atomizer.student_tokenizer=ReplacementTokenizer()
|
| 135 |
+
batch=fixtures.MPPortTests().batch(2)
|
| 136 |
+
batch['stu_input_ids'][0,1]=5
|
| 137 |
+
result=align.training_step(batch)
|
| 138 |
+
self.assertEqual(float(result['distill/mp_opd_skipped_replacement_responses']),1.)
|
| 139 |
+
for mode,reason in ((m,r) for m in ('trust_r','trust_b','trust_p')
|
| 140 |
+
for r in ('replacement','normalization')):
|
| 141 |
+
helper,algo,student,_=test_trust_paper.TrustTests().fixture(mode)
|
| 142 |
+
algo.atomizer.student_tokenizer=ReplacementTokenizer()
|
| 143 |
+
bad,good=helper.batch(),helper.batch()
|
| 144 |
+
if reason == 'replacement':
|
| 145 |
+
bad['stu_input_ids'][0,1]=5
|
| 146 |
+
else:
|
| 147 |
+
bad['tea_input_ids'][0,1]=6
|
| 148 |
+
for b in (bad,good): b['avg_micro_batch_token_num']*=2
|
| 149 |
+
if algo.requires_optimizer_batch:
|
| 150 |
+
algo.prepare_optimizer_batch([bad,good])
|
| 151 |
+
self.assertEqual(float(algo._batch_metrics['mp_opd_'+mode+'_calibration_response_count']),1.)
|
| 152 |
+
result=algo.training_step(bad)
|
| 153 |
+
self.assertEqual(float(result['distill/mp_opd_skipped_'+reason+'_responses']),1.)
|
| 154 |
+
loss=result['train/loss']+algo.training_step(good)['train/loss']
|
| 155 |
+
full_grad=torch.autograd.grad(loss,tuple(student.parameters()))
|
| 156 |
+
algo.finish_optimizer_batch()
|
| 157 |
+
if algo.requires_optimizer_batch:
|
| 158 |
+
algo.strategy.accumulated_gradient=1
|
| 159 |
+
algo.prepare_optimizer_batch([good])
|
| 160 |
+
expected=algo.training_step(good)['train/loss']
|
| 161 |
+
torch.testing.assert_close(loss,expected,atol=1e-6,rtol=1e-5)
|
| 162 |
+
for a,b in zip(full_grad,torch.autograd.grad(expected,tuple(student.parameters()))):
|
| 163 |
+
torch.testing.assert_close(a,b,atol=1e-6,rtol=1e-5)
|
| 164 |
+
algo.finish_optimizer_batch()
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
class InvalidResponseSourceTests(unittest.TestCase):
|
| 168 |
+
def test_adapter_patch_hash(self):
|
| 169 |
+
policy=json.loads((ROOT/'docs/MP_OPD_INVALID_RESPONSES.json').read_text())
|
| 170 |
+
self.assertEqual(hashlib.sha256((ROOT/policy['path']).read_bytes()).hexdigest(),policy['updated_sha256'])
|
| 171 |
+
|
| 172 |
+
|
| 173 |
+
if __name__=='__main__':
|
| 174 |
+
unittest.main()
|
tests/test_mp_opd_port.py
CHANGED
|
@@ -239,11 +239,11 @@ assert not any(k.startswith('probe._mp_opd_port') for k in sys.modules)
|
|
| 239 |
torch.testing.assert_close((effective * weight).sum(), (rate * weight).sum().double())
|
| 240 |
self.assertLess(float(residual), 1e-12)
|
| 241 |
|
| 242 |
-
def
|
| 243 |
algo, _, _ = self.fixture('mp_opd_align')
|
| 244 |
batch = self.batch()
|
| 245 |
-
batch['tea_input_ids'][0, 1] =
|
| 246 |
-
with self.assertRaisesRegex(ValueError, '
|
| 247 |
algo.training_step(batch)
|
| 248 |
|
| 249 |
def test_noise_state_roundtrip_and_mode_guard(self):
|
|
|
|
| 239 |
torch.testing.assert_close((effective * weight).sum(), (rate * weight).sum().double())
|
| 240 |
self.assertLess(float(residual), 1e-12)
|
| 241 |
|
| 242 |
+
def test_unsupported_added_token_fails_instead_of_falling_back_to_atomic(self):
|
| 243 |
algo, _, _ = self.fixture('mp_opd_align')
|
| 244 |
batch = self.batch()
|
| 245 |
+
batch['tea_input_ids'][0, 1] = 7
|
| 246 |
+
with self.assertRaisesRegex(ValueError, 'unsupported_added_token'):
|
| 247 |
algo.training_step(batch)
|
| 248 |
|
| 249 |
def test_noise_state_roundtrip_and_mode_guard(self):
|
tests/test_simct.py
CHANGED
|
@@ -7,6 +7,7 @@ import logging
|
|
| 7 |
import pathlib
|
| 8 |
import types
|
| 9 |
import unittest
|
|
|
|
| 10 |
|
| 11 |
import test_mp_opd_port as fixtures
|
| 12 |
|
|
@@ -186,7 +187,7 @@ class SimCTSourceTests(unittest.TestCase):
|
|
| 186 |
def test_pinned_sources_and_frozen_existing_modes(self):
|
| 187 |
manifest = json.loads((ROOT / 'docs/SIMCT_SOURCE.json').read_text())
|
| 188 |
for record in manifest['files'] + manifest['unchanged_existing_files']:
|
| 189 |
-
|
| 190 |
raw = (ROOT / 'tests/fixtures/simct_upstream/span_ctkd.py.txt').read_bytes()
|
| 191 |
port = (ROOT / 'kdflow/algorithms/_simct_upstream/span_ctkd.py').read_bytes()
|
| 192 |
self.assertEqual(raw.replace(b'from kdflow.algorithms import register_algorithm',
|
|
|
|
| 7 |
import pathlib
|
| 8 |
import types
|
| 9 |
import unittest
|
| 10 |
+
from mp_opd_source_checks import assert_source_hash
|
| 11 |
|
| 12 |
import test_mp_opd_port as fixtures
|
| 13 |
|
|
|
|
| 187 |
def test_pinned_sources_and_frozen_existing_modes(self):
|
| 188 |
manifest = json.loads((ROOT / 'docs/SIMCT_SOURCE.json').read_text())
|
| 189 |
for record in manifest['files'] + manifest['unchanged_existing_files']:
|
| 190 |
+
assert_source_hash(self, ROOT, record)
|
| 191 |
raw = (ROOT / 'tests/fixtures/simct_upstream/span_ctkd.py.txt').read_bytes()
|
| 192 |
port = (ROOT / 'kdflow/algorithms/_simct_upstream/span_ctkd.py').read_bytes()
|
| 193 |
self.assertEqual(raw.replace(b'from kdflow.algorithms import register_algorithm',
|
tests/test_trust_paper.py
CHANGED
|
@@ -10,6 +10,7 @@ import random
|
|
| 10 |
import sys
|
| 11 |
import types
|
| 12 |
import unittest
|
|
|
|
| 13 |
|
| 14 |
import test_mp_opd_port as fixtures
|
| 15 |
|
|
@@ -382,7 +383,7 @@ class TrustPaperSourceTests(unittest.TestCase):
|
|
| 382 |
def test_source_hashes_and_frozen_existing_modes(self):
|
| 383 |
manifest=json.loads((ROOT/'docs/TRUST_PAPER_SOURCE.json').read_text())
|
| 384 |
for row in manifest['files']+manifest['unchanged_existing_files']:
|
| 385 |
-
|
| 386 |
|
| 387 |
|
| 388 |
if __name__=='__main__':
|
|
|
|
| 10 |
import sys
|
| 11 |
import types
|
| 12 |
import unittest
|
| 13 |
+
from mp_opd_source_checks import assert_source_hash
|
| 14 |
|
| 15 |
import test_mp_opd_port as fixtures
|
| 16 |
|
|
|
|
| 383 |
def test_source_hashes_and_frozen_existing_modes(self):
|
| 384 |
manifest=json.loads((ROOT/'docs/TRUST_PAPER_SOURCE.json').read_text())
|
| 385 |
for row in manifest['files']+manifest['unchanged_existing_files']:
|
| 386 |
+
assert_source_hash(self, ROOT, row)
|
| 387 |
|
| 388 |
|
| 389 |
if __name__=='__main__':
|