feat(simct): add isolated author baseline with loss gradient parity
Browse files- README.md +4 -0
- docs/SIMCT.md +100 -0
- docs/SIMCT_SOURCE.json +63 -0
- docs/SIMCT_VALIDATION.json +39 -0
- docs/SINGLE_B200.md +2 -0
- kdflow/algorithms/__init__.py +1 -0
- kdflow/algorithms/_simct_upstream/LICENSE.MIT.txt +21 -0
- kdflow/algorithms/_simct_upstream/__init__.py +5 -0
- kdflow/algorithms/_simct_upstream/span_ctkd.py +508 -0
- kdflow/algorithms/simct.py +89 -0
- kdflow/arguments/distillation_args.py +10 -2
- kdflow/cli/train_kd_on_policy.py +6 -1
- tests/fixtures/simct_upstream/reverse_kl_div.py.txt +27 -0
- tests/fixtures/simct_upstream/span_ctkd.py.txt +508 -0
- tests/test_simct.py +197 -0
- tests/test_single_b200.py +8 -0
- tools/launch_single_b200.py +6 -2
README.md
CHANGED
|
@@ -17,6 +17,10 @@ GRASS-Chunk and ALIGN as separate modes. The existing Strict algorithm sources
|
|
| 17 |
remain unchanged. The new modes use sampled atom credits rather than Strict KL;
|
| 18 |
they have CPU verification, with B200/full training qualification still pending.
|
| 19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
## Original project documentation
|
| 21 |
|
| 22 |
This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
|
|
|
|
| 17 |
remain unchanged. The new modes use sampled atom credits rather than Strict KL;
|
| 18 |
they have CPU verification, with B200/full training qualification still pending.
|
| 19 |
|
| 20 |
+
A separate [author-code SimCT baseline](docs/SIMCT.md) uses `--algorithm simct`
|
| 21 |
+
in the same launcher. It preserves the pinned released implementation, with
|
| 22 |
+
original-source loss/gradient parity tests and an explicit paper/code distinction.
|
| 23 |
+
|
| 24 |
## Original project documentation
|
| 25 |
|
| 26 |
This repository provides the implementation and example training scripts for `byte_align_ctkd` and `topk_byte_align_ctkd`.
|
docs/SIMCT.md
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Author-code SimCT in the common CTOPD pipeline
|
| 2 |
+
|
| 3 |
+
Select `simct` to run the author's released `span_ctkd` objective. This is separate
|
| 4 |
+
from Strict and the MP-OPD modes: `mp-atomic`/`mp-fixed1` are not SimCT.
|
| 5 |
+
|
| 6 |
+
## Source and exact scope
|
| 7 |
+
|
| 8 |
+
The source is [sunjie279/SimCT-](https://github.com/sunjie279/SimCT-/tree/cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e),
|
| 9 |
+
commit `cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e`. The author's released
|
| 10 |
+
`scripts/ctopd/qwen25_gemma2_span_mix10k_lr5e-7.sh` selects `span_ctkd` with unit
|
| 11 |
+
RKL. Locally the public mode name is `simct`; the donor registration is isolated
|
| 12 |
+
and does not register a second legacy alias. The vendor class differs only by
|
| 13 |
+
relocating the registration import. Original source fixtures, SHA-256 hashes and
|
| 14 |
+
the MIT notice are retained; see [SIMCT_SOURCE.json](SIMCT_SOURCE.json).
|
| 15 |
+
|
| 16 |
+
The adapter changes the execution interface from the author's full student
|
| 17 |
+
logits to CTOPD hidden states plus the student LM head, preserves student final
|
| 18 |
+
softcapping, and translates metrics into CTOPD's `train/*` / `distill/*` names.
|
| 19 |
+
The author's teacher uses the raw linear head, which is preserved here. It does
|
| 20 |
+
not inherit the MP-OPD adapter. Per-response alignment, shared-vocabulary mapping,
|
| 21 |
+
span construction, EOS inclusion, unmatched suffix handling, virtual vocabulary
|
| 22 |
+
and optional teacher-mass masking all use the pinned author's functions.
|
| 23 |
+
|
| 24 |
+
The adapter processes one response at a time and checkpoints the student head.
|
| 25 |
+
Empty alignment gives a connected zero loss. Nonfinite loss, invalid hidden row
|
| 26 |
+
counts and invalid token normalizers fail explicitly. Debug token/logit dumps
|
| 27 |
+
are suppressed. Unit RKL, no CE mixing, on-policy KD, one student rank and ring
|
| 28 |
+
size 1 are the supported contract. Other configurations fail before launch.
|
| 29 |
+
|
| 30 |
+
## Important distinction: released code versus paper equations
|
| 31 |
+
|
| 32 |
+
The [paper's equations 7-8](https://arxiv.org/html/2605.07711v1#S3.SS3) describe
|
| 33 |
+
length-normalized continuation **log-probability** scores. The released class
|
| 34 |
+
`_build_virtual_vocab_logits` instead averages selected **raw logits** for each
|
| 35 |
+
span and appends that value to the raw shared-vocabulary logits. Its comments
|
| 36 |
+
claim a geometric-mean probability interpretation, but position-dependent
|
| 37 |
+
full-vocabulary normalizers are absent from that computation. These two formulas
|
| 38 |
+
are generally different, including under additive logit shifts at later span
|
| 39 |
+
positions. This port preserves the released code; it does not silently replace
|
| 40 |
+
it with a reconstructed paper objective.
|
| 41 |
+
|
| 42 |
+
The released alignment is greedy over independently decoded token strings. It
|
| 43 |
+
is not the MP-OPD byte-checked atomizer, and may leave an unmatched suffix
|
| 44 |
+
unsupervised. These semantics are retained for source parity. This port is not
|
| 45 |
+
proof of paper/code equivalence, nor of parity with a different KDFlow revision
|
| 46 |
+
whose algorithm is also named `simct`.
|
| 47 |
+
|
| 48 |
+
## Launch
|
| 49 |
+
|
| 50 |
+
Inside this checkout on an allocated company node:
|
| 51 |
+
|
| 52 |
+
```bash
|
| 53 |
+
git pull --ff-only
|
| 54 |
+
python3 tools/launch_single_b200.py --algorithm simct \
|
| 55 |
+
--runtime /opt/ctopd-b200-u22 --dry-run
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
The standard library dry-run uses no GPU and creates no run. Remove `--dry-run`
|
| 59 |
+
only when ready for the real allocated single-B200 workload, with the assigned
|
| 60 |
+
GPU exposed through `CUDA_VISIBLE_DEVICES` and all models/data present. Use a new
|
| 61 |
+
run ID. No runtime rebuild or new dependency is needed for this source port.
|
| 62 |
+
|
| 63 |
+
The preset changes only the algorithm and adds `simct_gh_mask_threshold=0`
|
| 64 |
+
(disabled, as in the released author launchers). The adapter maps this isolated
|
| 65 |
+
flag to the author's optional `span_gh_mask_threshold` semantics. Direct KDFlow
|
| 66 |
+
CLI users may set `--simct_gh_mask_threshold` to a positive finite number;
|
| 67 |
+
zero disables it. Current launch presets do not enable it.
|
| 68 |
+
|
| 69 |
+
The common CTOPD recipe retains Qwen-to-Llama, the existing student initialization,
|
| 70 |
+
20k prompt source, optimizer batch 512, 100 steps, generation limit 8192, optimizer
|
| 71 |
+
and evaluation settings. It does not add SimCT's teacher-generated 10k SFT warmup
|
| 72 |
+
or its original model pairs. Compare all methods from the same checkpoint under
|
| 73 |
+
this common recipe; label the result a common-pipeline comparison, not a
|
| 74 |
+
reproduction of SimCT's published table. The launcher currently evaluates MATH500
|
| 75 |
+
only, not the paper's entire benchmark suite.
|
| 76 |
+
|
| 77 |
+
## Verification
|
| 78 |
+
|
| 79 |
+
Using an existing compatible CPU PyTorch environment:
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
The oracle runs the original saved author class and original RKL numerical bodies,
|
| 86 |
+
without importing the adapter or CTOPD's RKL. Only import/decorator wiring is
|
| 87 |
+
removed to execute those original bodies in the test process. Compare mixed
|
| 88 |
+
spans, strict-only responses, optional G(h) masks, softcapping, loss, every student
|
| 89 |
+
gradient, and alignment/mask diagnostics. An independent hand construction checks
|
| 90 |
+
virtual logits and RKL gradients. Additional checks cover teacher stop-gradient,
|
| 91 |
+
response boundaries, shared microbatch normalization, empty alignment, argument
|
| 92 |
+
validation, launcher settings and unchanged existing numerical source hashes.
|
| 93 |
+
|
| 94 |
+
The [CPU validation receipt](SIMCT_VALIDATION.json) records the 56-test pass and
|
| 95 |
+
six original-source comparisons: maximum absolute loss error `0.0`, maximum
|
| 96 |
+
absolute student-gradient error `3.725290298461914e-09` in FP32 fixtures.
|
| 97 |
+
|
| 98 |
+
CPU checks do not qualify target Torch/CUDA, B200/FSDP integration, full-response
|
| 99 |
+
memory/throughput or downstream accuracy. Per-response virtual-vocabulary logits
|
| 100 |
+
can still be large. No GPU training is launched by the port or tests.
|
docs/SIMCT_SOURCE.json
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"source_repository": "https://github.com/sunjie279/SimCT-",
|
| 3 |
+
"source_commit": "cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e",
|
| 4 |
+
"author_launcher_algorithm": "span_ctkd",
|
| 5 |
+
"local_algorithm": "simct",
|
| 6 |
+
"files": [
|
| 7 |
+
{
|
| 8 |
+
"sha256": "a83bc75c4f297c7f13e9eac809647c8c9dc228b9a3fdf90eb413d30bfc94795a",
|
| 9 |
+
"path": "tests/fixtures/simct_upstream/span_ctkd.py.txt"
|
| 10 |
+
},
|
| 11 |
+
{
|
| 12 |
+
"sha256": "abca69fdbe4f7d90a7ca990300189147230cce8f20d589c934d88f0b63c4edf6",
|
| 13 |
+
"path": "tests/fixtures/simct_upstream/reverse_kl_div.py.txt"
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
"sha256": "64bd78c0a991dcaa8e438c0d96dd61d3aacabf136c5194816c0e9e349f407e49",
|
| 17 |
+
"path": "kdflow/algorithms/_simct_upstream/span_ctkd.py"
|
| 18 |
+
},
|
| 19 |
+
{
|
| 20 |
+
"sha256": "c9f18bd09f6b760b2ceb9826447d56fd72b272ee318fb2eb48be20a68210b3be",
|
| 21 |
+
"path": "kdflow/algorithms/_simct_upstream/LICENSE.MIT.txt"
|
| 22 |
+
}
|
| 23 |
+
],
|
| 24 |
+
"unchanged_existing_files": [
|
| 25 |
+
{
|
| 26 |
+
"sha256": "fdbd99c55aeac4d061527158bdfc82a93333a42bf347eceaa134d34f5ef05952",
|
| 27 |
+
"path": "kdflow/algorithms/simple_ctkd.py"
|
| 28 |
+
},
|
| 29 |
+
{
|
| 30 |
+
"sha256": "5c6647c6fb8064ffd5f7b1ed971168cd1f3392caee62e2d75916d41931828e18",
|
| 31 |
+
"path": "kdflow/algorithms/byte_align_ctkd.py"
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"sha256": "298925da4ce973b0f2917e5ba5ad2adbc4862d7c1a73515c5c86514ac0d22ddb",
|
| 35 |
+
"path": "kdflow/algorithms/topk_byte_align_ctkd.py"
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"sha256": "b2387d7f36502dea16c28f4391af4bc97a7e77f367d35dea2867c744e91277a6",
|
| 39 |
+
"path": "kdflow/algorithms/mp_opd.py"
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"sha256": "abca69fdbe4f7d90a7ca990300189147230cce8f20d589c934d88f0b63c4edf6",
|
| 43 |
+
"path": "kdflow/loss/reverse_kl_div.py"
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"sha256": "151c81614f97b908465a952f595abe1f4dc5c0d17cea86431a7068c381125cc5",
|
| 47 |
+
"path": "kdflow/loss/chunked_loss.py"
|
| 48 |
+
},
|
| 49 |
+
{
|
| 50 |
+
"sha256": "c77cd962cb0430cd97ace5312077e847398a03da3ce1473062fd2c5c9808498a",
|
| 51 |
+
"path": "kdflow/token_alignment.py"
|
| 52 |
+
},
|
| 53 |
+
{
|
| 54 |
+
"sha256": "97a8495d68da45c05aa87ba8d87fe6db6dd520cb15de4352cb2ea0e927ddb076",
|
| 55 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_grass_chunk.py"
|
| 56 |
+
},
|
| 57 |
+
{
|
| 58 |
+
"sha256": "c6c0286c74215a8d30888c15b2bb5820ab77f54a45afd832db99ca6eb81cf1e0",
|
| 59 |
+
"path": "kdflow/algorithms/_mp_opd_port/_mp_opd_align.py"
|
| 60 |
+
}
|
| 61 |
+
],
|
| 62 |
+
"changes": "Only register_algorithm import relocated in vendored class. Adapter uses hidden states and raw heads, per-response execution, CTOPD metric names, unit-RKL/no-CE contract, no token debug logging. See SIMCT.md for paper versus code distinction."
|
| 63 |
+
}
|
docs/SIMCT_VALIDATION.json
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"date": "2026-10-19",
|
| 3 |
+
"source_commit": "cf0f33a0e6c967d4b74ea32b2dba12be01b73b9e",
|
| 4 |
+
"platform": "WSL Linux, CPU only",
|
| 5 |
+
"python": "3.12.3",
|
| 6 |
+
"torch": "2.14.0+cpu",
|
| 7 |
+
"command": "TORCH_COMPILE_DISABLE=1 OMP_NUM_THREADS=1 python -m unittest discover -s tests -v",
|
| 8 |
+
"full_suite": {
|
| 9 |
+
"passed": 56,
|
| 10 |
+
"skipped": 0,
|
| 11 |
+
"elapsed_seconds": 91.714
|
| 12 |
+
},
|
| 13 |
+
"original_source_parity": {
|
| 14 |
+
"cases": 6,
|
| 15 |
+
"max_loss_abs_error": 0.0,
|
| 16 |
+
"max_student_gradient_abs_error": 3.725290298461914e-09,
|
| 17 |
+
"atol": 1e-6,
|
| 18 |
+
"rtol": 1e-5,
|
| 19 |
+
"dtype": "float32",
|
| 20 |
+
"coverage": "Mixed spans with two responses, strict-only, student softcap, G(h) disabled and three positive thresholds",
|
| 21 |
+
"oracle": "Pinned original class and original RKL numerical bodies; only imports/decorators removed for isolated execution"
|
| 22 |
+
},
|
| 23 |
+
"additional_checks": [
|
| 24 |
+
"Independent hand-constructed virtual logits, RKL and gradients",
|
| 25 |
+
"No teacher parameter or hidden-state gradient",
|
| 26 |
+
"Response boundaries and shared microbatch normalization",
|
| 27 |
+
"Empty alignment, invalid hidden rows, arguments and topology guards",
|
| 28 |
+
"Existing Strict/MP source hashes and numerical regression tests",
|
| 29 |
+
"SimCT dry-run retains batch512, steps100, generation8192 and common optimizer/evaluation settings",
|
| 30 |
+
"Staged diff whitespace check excluding the verbatim original RKL fixture, which retains three upstream trailing-whitespace lines",
|
| 31 |
+
"Source hashes rechecked after renaming the MIT notice to enforce LF through existing .gitattributes"
|
| 32 |
+
],
|
| 33 |
+
"not_qualified": [
|
| 34 |
+
"Paper-equation equivalence: released code averages raw logits rather than log-probabilities",
|
| 35 |
+
"Parity with other KDFlow implementations named simct",
|
| 36 |
+
"Target Torch 2.11 CUDA, B200/FSDP execution and full-size memory/throughput",
|
| 37 |
+
"Full training, published scores and the original SFT warmup recipe"
|
| 38 |
+
]
|
| 39 |
+
}
|
docs/SINGLE_B200.md
CHANGED
|
@@ -4,6 +4,8 @@
|
|
| 4 |
|
| 5 |
Separate experimental MP-OPD presets (Atomic, Fixed1-5, Random1-2/2-5, GRASS-Chunk and ALIGN) are documented in [MP_OPD.md](MP_OPD.md). They retain the workload below but change the objective; they are not Strict reproduction modes. Their CPU checks do not qualify B200/FSDP execution or full-training memory requirements.
|
| 6 |
|
|
|
|
|
|
|
| 7 |
## Preserved training and evaluation settings
|
| 8 |
|
| 9 |
| Setting | Value |
|
|
|
|
| 4 |
|
| 5 |
Separate experimental MP-OPD presets (Atomic, Fixed1-5, Random1-2/2-5, GRASS-Chunk and ALIGN) are documented in [MP_OPD.md](MP_OPD.md). They retain the workload below but change the objective; they are not Strict reproduction modes. Their CPU checks do not qualify B200/FSDP execution or full-training memory requirements.
|
| 6 |
|
| 7 |
+
The separate [SimCT author-code preset](SIMCT.md), `--algorithm simct`, uses the same common workload. It adds no SFT warmup; it is not the original SimCT end-to-end recipe.
|
| 8 |
+
|
| 9 |
## Preserved training and evaluation settings
|
| 10 |
|
| 11 |
| Setting | Value |
|
kdflow/algorithms/__init__.py
CHANGED
|
@@ -13,6 +13,7 @@ def register_algorithm(name):
|
|
| 13 |
from kdflow.algorithms.simple_ctkd import SimpleCrossTokenizerKD
|
| 14 |
from kdflow.algorithms.byte_align_ctkd import ByteAlignCrossTokenizerKD
|
| 15 |
from kdflow.algorithms.topk_byte_align_ctkd import TopKByteAlignCrossTokenizerKD
|
|
|
|
| 16 |
from kdflow.algorithms.mp_opd import (
|
| 17 |
AtomicMPOnPolicyKD, FixedMPOnPolicyKD, RandomMPOnPolicyKD,
|
| 18 |
GrassChunkMPOnPolicyKD, AlignMPOnPolicyKD,
|
|
|
|
| 13 |
from kdflow.algorithms.simple_ctkd import SimpleCrossTokenizerKD
|
| 14 |
from kdflow.algorithms.byte_align_ctkd import ByteAlignCrossTokenizerKD
|
| 15 |
from kdflow.algorithms.topk_byte_align_ctkd import TopKByteAlignCrossTokenizerKD
|
| 16 |
+
from kdflow.algorithms.simct import SimCT
|
| 17 |
from kdflow.algorithms.mp_opd import (
|
| 18 |
AtomicMPOnPolicyKD, FixedMPOnPolicyKD, RandomMPOnPolicyKD,
|
| 19 |
GrassChunkMPOnPolicyKD, AlignMPOnPolicyKD,
|
kdflow/algorithms/_simct_upstream/LICENSE.MIT.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Songming Zhang
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
kdflow/algorithms/_simct_upstream/__init__.py
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pinned author's SimCT code; never register its legacy name in KDFlow."""
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
def register_algorithm(_name):
|
| 5 |
+
return lambda cls: cls
|
kdflow/algorithms/_simct_upstream/span_ctkd.py
ADDED
|
@@ -0,0 +1,508 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
import torch.nn.functional as F
|
| 3 |
+
|
| 4 |
+
from kdflow.loss import build_loss_fn
|
| 5 |
+
from . import register_algorithm
|
| 6 |
+
from kdflow.loss.cross_entropy import compute_cross_entropy
|
| 7 |
+
from kdflow.utils.logging_utils import init_logger
|
| 8 |
+
|
| 9 |
+
logger = init_logger(__name__)
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
@register_algorithm("span_ctkd")
|
| 13 |
+
class SpanCrossTokenizerKD:
|
| 14 |
+
"""Span-based Cross-Tokenizer Knowledge Distillation.
|
| 15 |
+
|
| 16 |
+
Improvement over SimpleCrossTokenizerKD: instead of discarding positions where
|
| 17 |
+
student and teacher tokens do not align 1:1, we group misaligned tokens into
|
| 18 |
+
"spans" and compute their logits as the arithmetic mean of the constituent
|
| 19 |
+
token logits (which corresponds to geometric mean after softmax).
|
| 20 |
+
|
| 21 |
+
These spans are appended to the shared (overlap) vocabulary to form a
|
| 22 |
+
"virtual common vocabulary". RKL alignment is then performed on this
|
| 23 |
+
extended vocabulary for every aligned position (both 1:1 tokens and spans).
|
| 24 |
+
|
| 25 |
+
For span segments, the overlap logits are taken from the **first** token
|
| 26 |
+
position only (not averaged across positions), because the span is treated
|
| 27 |
+
as a single virtual token occupying one position in the sequence.
|
| 28 |
+
"""
|
| 29 |
+
|
| 30 |
+
def __init__(
|
| 31 |
+
self,
|
| 32 |
+
strategy,
|
| 33 |
+
student_model,
|
| 34 |
+
teacher_lm_head,
|
| 35 |
+
student_tokenizer,
|
| 36 |
+
teacher_tokenizer,
|
| 37 |
+
**kwargs,
|
| 38 |
+
):
|
| 39 |
+
self.strategy = strategy
|
| 40 |
+
self.args = strategy.args
|
| 41 |
+
self.student = student_model
|
| 42 |
+
self.teacher_lm_head = teacher_lm_head
|
| 43 |
+
self.student_tokenizer = student_tokenizer
|
| 44 |
+
self.teacher_tokenizer = teacher_tokenizer
|
| 45 |
+
self.student_overlap_token_ids, self.teacher_overlap_token_ids = self._find_overlap_tokens()
|
| 46 |
+
self.loss_fn = build_loss_fn(self.args.kd.kd_loss_fn, self.args)
|
| 47 |
+
self._debug_count = 0 # Counter for debug logging
|
| 48 |
+
self._span_debug_count = 0 # Counter for span logits debug logging
|
| 49 |
+
|
| 50 |
+
# ------------------------------------------------------------------
|
| 51 |
+
# Overlap vocabulary (copied from simple_ctkd)
|
| 52 |
+
# ------------------------------------------------------------------
|
| 53 |
+
def _find_overlap_tokens(self):
|
| 54 |
+
"""Find the overlap tokens between student and teacher tokenizer."""
|
| 55 |
+
student_vocab = {k.replace("Ġ", "▁"): v for k, v in self.student_tokenizer.get_vocab().items()}
|
| 56 |
+
teacher_vocab = {k.replace("Ġ", "▁"): v for k, v in self.teacher_tokenizer.get_vocab().items()}
|
| 57 |
+
overlap_tokens = set(student_vocab.keys()) & set(teacher_vocab.keys())
|
| 58 |
+
student_ids = [student_vocab[token] for token in overlap_tokens]
|
| 59 |
+
teacher_ids = [teacher_vocab[token] for token in overlap_tokens]
|
| 60 |
+
stu_eos, tea_eos = self.student_tokenizer.eos_token_id, self.teacher_tokenizer.eos_token_id
|
| 61 |
+
if stu_eos not in student_ids or tea_eos not in teacher_ids:
|
| 62 |
+
student_ids.append(stu_eos)
|
| 63 |
+
teacher_ids.append(tea_eos)
|
| 64 |
+
device = self.teacher_lm_head.weight.device
|
| 65 |
+
logger.info(f"[SpanCTKD] Num of overlap_tokens between student & teacher: {len(student_ids)}")
|
| 66 |
+
return (
|
| 67 |
+
torch.tensor(student_ids, dtype=torch.long, device=device),
|
| 68 |
+
torch.tensor(teacher_ids, dtype=torch.long, device=device),
|
| 69 |
+
)
|
| 70 |
+
|
| 71 |
+
# ------------------------------------------------------------------
|
| 72 |
+
# Sequence alignment with span identification
|
| 73 |
+
# (adapted from ALM._compute_chunk_alignment)
|
| 74 |
+
# ------------------------------------------------------------------
|
| 75 |
+
def _align_sequences_with_spans(self, tea_label_ids, stu_label_ids):
|
| 76 |
+
"""Align teacher and student label tokens and identify spans.
|
| 77 |
+
|
| 78 |
+
Operates on **label ids** (i.e. input_ids shifted left by one position,
|
| 79 |
+
already filtered by loss_mask) so that alignment is consistent with the
|
| 80 |
+
next-token-prediction semantics of the logits.
|
| 81 |
+
|
| 82 |
+
Uses ``tokenizer.decode([tid])`` per token (O(n)) to correctly handle
|
| 83 |
+
all tokenizer conventions including newlines.
|
| 84 |
+
|
| 85 |
+
Args:
|
| 86 |
+
tea_label_ids: 1-D list of teacher label token ids (loss_mask positions).
|
| 87 |
+
stu_label_ids: 1-D list of student label token ids (loss_mask positions).
|
| 88 |
+
|
| 89 |
+
Returns:
|
| 90 |
+
segments: list of (tea_start, tea_end, stu_start, stu_end) where
|
| 91 |
+
indices are positions within the label id sequences. Each tuple
|
| 92 |
+
represents an aligned segment: teacher tokens ``[tea_start, tea_end)``
|
| 93 |
+
and student tokens ``[stu_start, stu_end)`` encode the same text.
|
| 94 |
+
When both ranges have length 1 it is a normal 1:1 alignment;
|
| 95 |
+
otherwise it is a span.
|
| 96 |
+
tea_label_ids_list: the teacher label ids as a Python list.
|
| 97 |
+
stu_label_ids_list: the student label ids as a Python list.
|
| 98 |
+
"""
|
| 99 |
+
if len(tea_label_ids) == 0 or len(stu_label_ids) == 0:
|
| 100 |
+
return [], [], []
|
| 101 |
+
|
| 102 |
+
tea_ids_list = tea_label_ids if isinstance(tea_label_ids, list) else tea_label_ids.cpu().tolist()
|
| 103 |
+
stu_ids_list = stu_label_ids if isinstance(stu_label_ids, list) else stu_label_ids.cpu().tolist()
|
| 104 |
+
|
| 105 |
+
# Convert token ids to text strings for alignment.
|
| 106 |
+
# Use tokenizer.decode([tid]) per token to correctly handle all
|
| 107 |
+
# tokenizer conventions (e.g. Qwen uses Ċ for \n internally, which
|
| 108 |
+
# would NOT be normalized by simple strip of ▁/Ġ prefixes).
|
| 109 |
+
tea_token_texts = [self.teacher_tokenizer.decode([tid]) for tid in tea_ids_list]
|
| 110 |
+
stu_token_texts = [self.student_tokenizer.decode([tid]) for tid in stu_ids_list]
|
| 111 |
+
|
| 112 |
+
# Greedy cumulative-text alignment (same algorithm as simple_ctkd)
|
| 113 |
+
tea_eos = self.teacher_tokenizer.eos_token
|
| 114 |
+
stu_eos = self.student_tokenizer.eos_token
|
| 115 |
+
|
| 116 |
+
i, j = 0, 0
|
| 117 |
+
boundaries = []
|
| 118 |
+
history_tea = ""
|
| 119 |
+
history_stu = ""
|
| 120 |
+
|
| 121 |
+
while i < len(tea_token_texts) and j < len(stu_token_texts):
|
| 122 |
+
is_eos_match = (tea_token_texts[i] == tea_eos and stu_token_texts[j] == stu_eos)
|
| 123 |
+
if history_tea == history_stu and (
|
| 124 |
+
tea_token_texts[i] == stu_token_texts[j] or is_eos_match
|
| 125 |
+
):
|
| 126 |
+
boundaries.append((i, j))
|
| 127 |
+
history_tea += tea_token_texts[i]
|
| 128 |
+
history_stu += stu_token_texts[j]
|
| 129 |
+
i += 1
|
| 130 |
+
j += 1
|
| 131 |
+
elif len(history_tea) > len(history_stu):
|
| 132 |
+
history_stu += stu_token_texts[j]
|
| 133 |
+
j += 1
|
| 134 |
+
elif len(history_tea) < len(history_stu):
|
| 135 |
+
history_tea += tea_token_texts[i]
|
| 136 |
+
i += 1
|
| 137 |
+
else:
|
| 138 |
+
history_tea += tea_token_texts[i]
|
| 139 |
+
history_stu += stu_token_texts[j]
|
| 140 |
+
i += 1
|
| 141 |
+
j += 1
|
| 142 |
+
|
| 143 |
+
if len(boundaries) == 0:
|
| 144 |
+
if self._debug_count < 3:
|
| 145 |
+
self._debug_count += 1
|
| 146 |
+
logger.warning(
|
| 147 |
+
f"[SpanCTKD DEBUG] No boundaries found! "
|
| 148 |
+
f"tea_token_texts({len(tea_token_texts)}): {tea_token_texts[:10]}... "
|
| 149 |
+
f"stu_token_texts({len(stu_token_texts)}): {stu_token_texts[:10]}..."
|
| 150 |
+
)
|
| 151 |
+
return [], tea_ids_list, stu_ids_list
|
| 152 |
+
|
| 153 |
+
if self._debug_count < 3:
|
| 154 |
+
self._debug_count += 1
|
| 155 |
+
logger.info(
|
| 156 |
+
f"[SpanCTKD DEBUG] Found {len(boundaries)} boundaries! "
|
| 157 |
+
f"tea_texts({len(tea_token_texts)}): {tea_token_texts[:5]}... "
|
| 158 |
+
f"stu_texts({len(stu_token_texts)}): {stu_token_texts[:5]}..."
|
| 159 |
+
)
|
| 160 |
+
|
| 161 |
+
# Convert boundaries to aligned segments (local indices within response)
|
| 162 |
+
segments = []
|
| 163 |
+
for idx in range(len(boundaries)):
|
| 164 |
+
if idx == 0:
|
| 165 |
+
local_tea_start, local_stu_start = 0, 0
|
| 166 |
+
else:
|
| 167 |
+
local_tea_start = boundaries[idx - 1][0] + 1
|
| 168 |
+
local_stu_start = boundaries[idx - 1][1] + 1
|
| 169 |
+
local_tea_end = boundaries[idx][0] + 1
|
| 170 |
+
local_stu_end = boundaries[idx][1] + 1
|
| 171 |
+
segments.append((local_tea_start, local_tea_end, local_stu_start, local_stu_end))
|
| 172 |
+
|
| 173 |
+
return segments, tea_ids_list, stu_ids_list
|
| 174 |
+
|
| 175 |
+
# ------------------------------------------------------------------
|
| 176 |
+
# Build virtual common vocabulary logits for a single sample
|
| 177 |
+
# ------------------------------------------------------------------
|
| 178 |
+
def _build_virtual_vocab_logits(
|
| 179 |
+
self,
|
| 180 |
+
segments,
|
| 181 |
+
stu_logits_aligned,
|
| 182 |
+
tea_logits_aligned,
|
| 183 |
+
stu_label_ids_list,
|
| 184 |
+
tea_label_ids_list,
|
| 185 |
+
):
|
| 186 |
+
"""Build student and teacher logit matrices on the virtual common vocabulary.
|
| 187 |
+
|
| 188 |
+
For each aligned segment we construct a logit vector of size
|
| 189 |
+
``num_overlap + num_total_spans`` where:
|
| 190 |
+
- the first ``num_overlap`` dims come from the base overlap vocabulary,
|
| 191 |
+
- the remaining ``num_total_spans`` dims correspond to span logits.
|
| 192 |
+
|
| 193 |
+
For a 1:1 aligned position the overlap-vocab logits are taken directly
|
| 194 |
+
from the single token; for a span the overlap-vocab logits are taken
|
| 195 |
+
from the **first** token position only (since the span is treated as a
|
| 196 |
+
single virtual token — other overlap tokens compete with it at the
|
| 197 |
+
first position). The span-logit dimensions are filled with ``-1e9``
|
| 198 |
+
(≈ 0 after softmax) except for the span that the current segment
|
| 199 |
+
belongs to, which gets the mean of each constituent token's logit at
|
| 200 |
+
its own token id (i.e. the model's average confidence in generating
|
| 201 |
+
this span).
|
| 202 |
+
|
| 203 |
+
Args:
|
| 204 |
+
segments: list of (tea_start, tea_end, stu_start, stu_end) —
|
| 205 |
+
indices into the label id sequences.
|
| 206 |
+
stu_logits_aligned: student logits at loss_mask positions
|
| 207 |
+
[num_stu_loss_tokens, vocab_s].
|
| 208 |
+
tea_logits_aligned: teacher logits at loss_mask positions
|
| 209 |
+
[num_tea_loss_tokens, vocab_t].
|
| 210 |
+
stu_label_ids_list: Python list of student label token ids.
|
| 211 |
+
tea_label_ids_list: Python list of teacher label token ids.
|
| 212 |
+
|
| 213 |
+
Returns:
|
| 214 |
+
stu_virtual_logits: [num_segments, num_overlap + num_spans]
|
| 215 |
+
tea_virtual_logits: [num_segments, num_overlap + num_spans]
|
| 216 |
+
"""
|
| 217 |
+
num_overlap = self.student_overlap_token_ids.shape[0]
|
| 218 |
+
device = stu_logits_aligned.device
|
| 219 |
+
|
| 220 |
+
# Identify which segments are spans (multi-token on either side)
|
| 221 |
+
span_indices = [] # indices into `segments` that are spans
|
| 222 |
+
for seg_idx, (ts, te, ss, se) in enumerate(segments):
|
| 223 |
+
if (te - ts) > 1 or (se - ss) > 1:
|
| 224 |
+
span_indices.append(seg_idx)
|
| 225 |
+
num_spans = len(span_indices)
|
| 226 |
+
# Map from segment index to its position in the span dimensions
|
| 227 |
+
seg_to_span_dim = {}
|
| 228 |
+
for dim_idx, seg_idx in enumerate(span_indices):
|
| 229 |
+
seg_to_span_dim[seg_idx] = dim_idx
|
| 230 |
+
|
| 231 |
+
virtual_dim = num_overlap + num_spans
|
| 232 |
+
|
| 233 |
+
stu_rows = []
|
| 234 |
+
tea_rows = []
|
| 235 |
+
|
| 236 |
+
for seg_idx, (ts, te, ss, se) in enumerate(segments):
|
| 237 |
+
# --- Student side ---
|
| 238 |
+
stu_seg_logits = stu_logits_aligned[ss:se] # [num_stu_tokens, vocab_s]
|
| 239 |
+
# For overlap logits, always use the FIRST token position only.
|
| 240 |
+
# The span is a single virtual token; other overlap tokens compete
|
| 241 |
+
# with it at the first position (i.e. "what comes after the prefix").
|
| 242 |
+
stu_first_logits = stu_seg_logits[0] # [vocab_s]
|
| 243 |
+
stu_overlap = stu_first_logits[self.student_overlap_token_ids] # [num_overlap]
|
| 244 |
+
|
| 245 |
+
# --- Teacher side ---
|
| 246 |
+
tea_seg_logits = tea_logits_aligned[ts:te] # [num_tea_tokens, vocab_t]
|
| 247 |
+
tea_first_logits = tea_seg_logits[0] # [vocab_t]
|
| 248 |
+
tea_overlap = tea_first_logits[self.teacher_overlap_token_ids] # [num_overlap]
|
| 249 |
+
|
| 250 |
+
if num_spans > 0:
|
| 251 |
+
# Span dimensions: default to -1e9 (negligible after softmax)
|
| 252 |
+
stu_span_dims = torch.full((num_spans,), -1e9, device=device, dtype=stu_overlap.dtype)
|
| 253 |
+
tea_span_dims = torch.full((num_spans,), -1e9, device=device, dtype=tea_overlap.dtype)
|
| 254 |
+
|
| 255 |
+
if seg_idx in seg_to_span_dim:
|
| 256 |
+
dim_pos = seg_to_span_dim[seg_idx]
|
| 257 |
+
# Span logit = mean of each constituent token's logit at
|
| 258 |
+
# its own token id. This represents the model's average
|
| 259 |
+
# confidence in generating this specific span of text.
|
| 260 |
+
# After softmax this corresponds to the geometric mean of
|
| 261 |
+
# per-token probabilities (avoiding length-dependent decay).
|
| 262 |
+
stu_span_token_ids = stu_label_ids_list[ss:se]
|
| 263 |
+
stu_self_logits = torch.stack([
|
| 264 |
+
stu_seg_logits[k, tid]
|
| 265 |
+
for k, tid in enumerate(stu_span_token_ids)
|
| 266 |
+
])
|
| 267 |
+
stu_span_dims[dim_pos] = stu_self_logits.mean()
|
| 268 |
+
|
| 269 |
+
tea_span_token_ids = tea_label_ids_list[ts:te]
|
| 270 |
+
tea_self_logits = torch.stack([
|
| 271 |
+
tea_seg_logits[k, tid]
|
| 272 |
+
for k, tid in enumerate(tea_span_token_ids)
|
| 273 |
+
])
|
| 274 |
+
tea_span_dims[dim_pos] = tea_self_logits.mean()
|
| 275 |
+
|
| 276 |
+
# Debug: print one example of span logits vs overlap logits
|
| 277 |
+
if self._span_debug_count < 1:
|
| 278 |
+
self._span_debug_count += 1
|
| 279 |
+
logger.info(
|
| 280 |
+
f"\n[SpanCTKD SPAN DEBUG] === Example span logits (seg_idx={seg_idx}) ===\n"
|
| 281 |
+
f" Student span token ids: {stu_span_token_ids}\n"
|
| 282 |
+
f" Student self_logits (per token): {stu_self_logits.detach().cpu().tolist()}\n"
|
| 283 |
+
f" Student span_dim value (mean): {stu_span_dims[dim_pos].item():.4f}\n"
|
| 284 |
+
f" Teacher span token ids: {tea_span_token_ids}\n"
|
| 285 |
+
f" Teacher self_logits (per token): {tea_self_logits.detach().cpu().tolist()}\n"
|
| 286 |
+
f" Teacher span_dim value (mean): {tea_span_dims[dim_pos].item():.4f}\n"
|
| 287 |
+
f" ---\n"
|
| 288 |
+
f" Student overlap logits (first pos) stats: mean={stu_overlap.mean().item():.4f}, "
|
| 289 |
+
f"max={stu_overlap.max().item():.4f}, min={stu_overlap.min().item():.4f}, "
|
| 290 |
+
f"std={stu_overlap.std().item():.4f}\n"
|
| 291 |
+
f" Teacher overlap logits (first pos) stats: mean={tea_overlap.mean().item():.4f}, "
|
| 292 |
+
f"max={tea_overlap.max().item():.4f}, min={tea_overlap.min().item():.4f}, "
|
| 293 |
+
f"std={tea_overlap.std().item():.4f}\n"
|
| 294 |
+
f" num_overlap={num_overlap}, num_spans={num_spans}, virtual_dim={virtual_dim}\n"
|
| 295 |
+
f" All stu_span_dims: {stu_span_dims.detach().cpu().tolist()}\n"
|
| 296 |
+
f" All tea_span_dims: {tea_span_dims.detach().cpu().tolist()}"
|
| 297 |
+
)
|
| 298 |
+
|
| 299 |
+
stu_row = torch.cat([stu_overlap, stu_span_dims]) # [virtual_dim]
|
| 300 |
+
tea_row = torch.cat([tea_overlap, tea_span_dims]) # [virtual_dim]
|
| 301 |
+
else:
|
| 302 |
+
stu_row = stu_overlap
|
| 303 |
+
tea_row = tea_overlap
|
| 304 |
+
|
| 305 |
+
stu_rows.append(stu_row)
|
| 306 |
+
tea_rows.append(tea_row)
|
| 307 |
+
|
| 308 |
+
stu_virtual_logits = torch.stack(stu_rows) # [num_segments, virtual_dim]
|
| 309 |
+
tea_virtual_logits = torch.stack(tea_rows) # [num_segments, virtual_dim]
|
| 310 |
+
return stu_virtual_logits, tea_virtual_logits
|
| 311 |
+
|
| 312 |
+
# ------------------------------------------------------------------
|
| 313 |
+
# Re-normalization multiplier G(h) safeguard
|
| 314 |
+
# ------------------------------------------------------------------
|
| 315 |
+
@torch.no_grad()
|
| 316 |
+
def _compute_segment_teacher_Z(self, segments, tea_logits_aligned, tea_label_ids_list):
|
| 317 |
+
"""Teacher mass ``Z_T(h)`` captured by the SimCT candidate space, per segment.
|
| 318 |
+
|
| 319 |
+
``Z_T(h)`` is computed on the **full-vocabulary** teacher softmax so that
|
| 320 |
+
it measures the original teacher probability mass that the candidate
|
| 321 |
+
space (shared tokens + the span realized at this position) represents:
|
| 322 |
+
|
| 323 |
+
- shared-token mass at the first teacher position, plus
|
| 324 |
+
- for a span, the geometric-mean probability of the span's teacher
|
| 325 |
+
tokens (matching the span logit used in the virtual distribution).
|
| 326 |
+
|
| 327 |
+
The re-normalization multiplier is then ``G(h) = 1 / Z_T(h)``. A large
|
| 328 |
+
``G(h)`` means the candidate space holds little teacher mass, so
|
| 329 |
+
re-normalizing over it would amplify a low-mass tail.
|
| 330 |
+
|
| 331 |
+
Returns:
|
| 332 |
+
Z: [num_segments] float tensor in (0, 1].
|
| 333 |
+
"""
|
| 334 |
+
device = tea_logits_aligned.device
|
| 335 |
+
Z = torch.ones(len(segments), device=device, dtype=torch.float32)
|
| 336 |
+
for seg_idx, (ts, te, ss, se) in enumerate(segments):
|
| 337 |
+
tea_seg_logits = tea_logits_aligned[ts:te] # [k_t, vocab_t]
|
| 338 |
+
p_first = torch.softmax(tea_seg_logits[0].float(), dim=-1) # full-vocab probs
|
| 339 |
+
m = p_first[self.teacher_overlap_token_ids].sum() # shared-token mass
|
| 340 |
+
if (te - ts) > 1 or (se - ss) > 1:
|
| 341 |
+
span_ids = tea_label_ids_list[ts:te]
|
| 342 |
+
logps = torch.stack([
|
| 343 |
+
torch.log_softmax(tea_seg_logits[k].float(), dim=-1)[tid]
|
| 344 |
+
for k, tid in enumerate(span_ids)
|
| 345 |
+
])
|
| 346 |
+
span_prob = torch.exp(logps.mean()) # geometric mean
|
| 347 |
+
Z[seg_idx] = (m + span_prob).clamp(max=1.0)
|
| 348 |
+
else:
|
| 349 |
+
Z[seg_idx] = m.clamp(max=1.0)
|
| 350 |
+
return Z
|
| 351 |
+
|
| 352 |
+
# ------------------------------------------------------------------
|
| 353 |
+
# Training step
|
| 354 |
+
# ------------------------------------------------------------------
|
| 355 |
+
def training_step(self, micro_batch):
|
| 356 |
+
"""Perform one training step with span-based cross-tokenizer KD loss.
|
| 357 |
+
|
| 358 |
+
Flow:
|
| 359 |
+
1. Student forward pass → full-sequence logits.
|
| 360 |
+
2. Teacher logits from hidden states (loss_mask positions only).
|
| 361 |
+
3. Per-sample: align sequences, identify spans, build virtual vocab
|
| 362 |
+
logits, compute RKL loss.
|
| 363 |
+
"""
|
| 364 |
+
student_input_ids = micro_batch["stu_input_ids"]
|
| 365 |
+
student_attn_mask = micro_batch["stu_attn_mask"]
|
| 366 |
+
student_loss_mask = micro_batch["stu_loss_mask"].bool()
|
| 367 |
+
teacher_input_ids = micro_batch["tea_input_ids"]
|
| 368 |
+
teacher_attn_mask = micro_batch["tea_attn_mask"]
|
| 369 |
+
teacher_loss_mask = micro_batch["tea_loss_mask"].bool()
|
| 370 |
+
teacher_hiddens = micro_batch.get("teacher_hiddens", None)
|
| 371 |
+
avg_token_num = micro_batch["avg_micro_batch_token_num"]
|
| 372 |
+
|
| 373 |
+
assert teacher_hiddens is not None, "micro_batch must contain `teacher_hiddens` for KD"
|
| 374 |
+
|
| 375 |
+
mm_kwargs = {k[3:]: v for k, v in micro_batch.items() if k.startswith("mm_")}
|
| 376 |
+
|
| 377 |
+
# Student forward
|
| 378 |
+
output = self.student(
|
| 379 |
+
student_input_ids,
|
| 380 |
+
attention_mask=student_attn_mask,
|
| 381 |
+
allgather_logits=True,
|
| 382 |
+
ring_attn_group=self.strategy.ring_attn_group,
|
| 383 |
+
**mm_kwargs,
|
| 384 |
+
)
|
| 385 |
+
student_logits = output["logits"]
|
| 386 |
+
|
| 387 |
+
# Extract label ids (next-token) at loss_mask positions — consistent
|
| 388 |
+
# with simple_ctkd which aligns on labels, not inputs.
|
| 389 |
+
student_label_ids = student_input_ids.roll(shifts=-1, dims=1)
|
| 390 |
+
teacher_label_ids = teacher_input_ids.roll(shifts=-1, dims=1)
|
| 391 |
+
|
| 392 |
+
# Flatten student logits at loss_mask positions (same as simple_ctkd)
|
| 393 |
+
student_logits_flat = student_logits[student_loss_mask] # [total_stu_loss_tokens, vocab_s]
|
| 394 |
+
|
| 395 |
+
# Free full student logits early — only loss_mask positions are needed
|
| 396 |
+
del student_logits, output
|
| 397 |
+
|
| 398 |
+
# NOTE: `teacher_hiddens` from teacher_actor is ALREADY 2D and filtered
|
| 399 |
+
# by loss_mask — shape [total_tea_loss_tokens, hidden_size]. See
|
| 400 |
+
# teacher_actor.py where hidden states are gathered only at loss_mask
|
| 401 |
+
# positions and np.concatenate-d across the micro-batch. Do NOT re-index
|
| 402 |
+
# with teacher_loss_mask (that would be shape [B, T] and mismatch).
|
| 403 |
+
teacher_hiddens = teacher_hiddens.to(self.teacher_lm_head.weight)
|
| 404 |
+
teacher_logits_flat = self.teacher_lm_head(teacher_hiddens) # [total_tea_loss_tokens, vocab_t]
|
| 405 |
+
del teacher_hiddens # free after lm_head computation
|
| 406 |
+
|
| 407 |
+
# Process each sample in the batch
|
| 408 |
+
batch_size = student_input_ids.shape[0]
|
| 409 |
+
total_loss = torch.tensor(0.0, device=student_logits_flat.device, requires_grad=True)
|
| 410 |
+
total_aligned_tokens = 0
|
| 411 |
+
total_response_tokens = 0
|
| 412 |
+
total_kept_segments = 0
|
| 413 |
+
total_masked_segments = 0
|
| 414 |
+
|
| 415 |
+
# Offset trackers for flattened logits (both are 2D across batch)
|
| 416 |
+
tea_logits_offset = 0
|
| 417 |
+
stu_logits_offset = 0
|
| 418 |
+
|
| 419 |
+
for b in range(batch_size):
|
| 420 |
+
stu_mask = student_loss_mask[b]
|
| 421 |
+
tea_mask = teacher_loss_mask[b]
|
| 422 |
+
|
| 423 |
+
stu_num_loss_tokens = stu_mask.sum().item()
|
| 424 |
+
tea_num_loss_tokens = tea_mask.sum().item()
|
| 425 |
+
|
| 426 |
+
# Student logits for this sample (already flattened by loss_mask)
|
| 427 |
+
stu_logits_b = student_logits_flat[stu_logits_offset:stu_logits_offset + stu_num_loss_tokens]
|
| 428 |
+
stu_logits_offset += stu_num_loss_tokens
|
| 429 |
+
|
| 430 |
+
# Teacher logits for this sample (already flattened by loss_mask)
|
| 431 |
+
tea_logits_b = teacher_logits_flat[tea_logits_offset:tea_logits_offset + tea_num_loss_tokens]
|
| 432 |
+
tea_logits_offset += tea_num_loss_tokens
|
| 433 |
+
|
| 434 |
+
# Extract label ids at loss_mask positions
|
| 435 |
+
stu_label_ids_b = student_label_ids[b][stu_mask].cpu().tolist()
|
| 436 |
+
tea_label_ids_b = teacher_label_ids[b][tea_mask].cpu().tolist()
|
| 437 |
+
|
| 438 |
+
# Align on label ids (next-token) and identify spans
|
| 439 |
+
segments, tea_ids_list, stu_ids_list = self._align_sequences_with_spans(
|
| 440 |
+
tea_label_ids_b, stu_label_ids_b
|
| 441 |
+
)
|
| 442 |
+
|
| 443 |
+
if len(segments) == 0:
|
| 444 |
+
total_response_tokens += max(stu_num_loss_tokens, tea_num_loss_tokens)
|
| 445 |
+
continue
|
| 446 |
+
|
| 447 |
+
# Build virtual vocab logits and compute loss.
|
| 448 |
+
# stu_logits_b and tea_logits_b are already [num_loss_tokens, vocab]
|
| 449 |
+
# and segments index directly into them.
|
| 450 |
+
stu_virtual, tea_virtual = self._build_virtual_vocab_logits(
|
| 451 |
+
segments, stu_logits_b, tea_logits_b,
|
| 452 |
+
stu_ids_list, tea_ids_list,
|
| 453 |
+
)
|
| 454 |
+
|
| 455 |
+
assert stu_virtual.shape == tea_virtual.shape, \
|
| 456 |
+
f"Virtual logit shape mismatch: student {stu_virtual.shape} vs teacher {tea_virtual.shape}"
|
| 457 |
+
|
| 458 |
+
# Compute RKL loss on virtual common vocabulary
|
| 459 |
+
sample_loss = self.loss_fn(
|
| 460 |
+
stu_virtual,
|
| 461 |
+
tea_virtual.detach(),
|
| 462 |
+
reduction="none",
|
| 463 |
+
)
|
| 464 |
+
|
| 465 |
+
# Re-normalization safeguard: drop positions whose multiplier
|
| 466 |
+
# G(h) = 1 / Z_T(h) exceeds the threshold (teacher captured mass
|
| 467 |
+
# Z_T(h) < 1/threshold). These low-mass positions would otherwise
|
| 468 |
+
# let re-normalization amplify a weak/noisy teacher tail, so they
|
| 469 |
+
# contribute no distillation gradient.
|
| 470 |
+
gh_threshold = getattr(self.args.kd, "span_gh_mask_threshold", 0.0)
|
| 471 |
+
if gh_threshold and gh_threshold > 0:
|
| 472 |
+
Z = self._compute_segment_teacher_Z(segments, tea_logits_b, tea_ids_list)
|
| 473 |
+
keep = (Z >= (1.0 / gh_threshold)).to(sample_loss.dtype) # G(h) <= threshold
|
| 474 |
+
total_kept_segments += int(keep.sum().item())
|
| 475 |
+
total_masked_segments += int((keep.numel() - keep.sum()).item())
|
| 476 |
+
sample_loss = sample_loss * keep
|
| 477 |
+
|
| 478 |
+
total_loss = total_loss + sample_loss.sum()
|
| 479 |
+
|
| 480 |
+
# Count aligned tokens (sum of all segment lengths on student side)
|
| 481 |
+
for ts, te, ss, se in segments:
|
| 482 |
+
total_aligned_tokens += max(te - ts, se - ss)
|
| 483 |
+
total_response_tokens += max(stu_num_loss_tokens, tea_num_loss_tokens)
|
| 484 |
+
|
| 485 |
+
# Normalize
|
| 486 |
+
kd_loss = total_loss / avg_token_num
|
| 487 |
+
|
| 488 |
+
align_ratio = torch.tensor(
|
| 489 |
+
total_aligned_tokens / max(total_response_tokens, 1),
|
| 490 |
+
device=student_logits_flat.device,
|
| 491 |
+
)
|
| 492 |
+
|
| 493 |
+
loss_info = {"loss": kd_loss, "kd_loss": kd_loss, "align_ratio": align_ratio}
|
| 494 |
+
|
| 495 |
+
gh_masked_total = total_kept_segments + total_masked_segments
|
| 496 |
+
loss_info["gh_mask_ratio"] = torch.tensor(
|
| 497 |
+
total_masked_segments / max(gh_masked_total, 1),
|
| 498 |
+
device=student_logits_flat.device,
|
| 499 |
+
)
|
| 500 |
+
|
| 501 |
+
if self.args.kd.kd_ratio < 1:
|
| 502 |
+
ce_label_ids = student_label_ids[student_loss_mask]
|
| 503 |
+
ce_loss = compute_cross_entropy(student_logits_flat, ce_label_ids, reduction="sum") / avg_token_num
|
| 504 |
+
loss = (1 - self.args.kd.kd_ratio) * ce_loss + self.args.kd.kd_ratio * kd_loss
|
| 505 |
+
loss_info["loss"] = loss
|
| 506 |
+
loss_info["ce_loss"] = ce_loss
|
| 507 |
+
|
| 508 |
+
return loss_info
|
kdflow/algorithms/simct.py
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Author-code SimCT baseline, integrated with CTOPD's hidden-state interface."""
|
| 2 |
+
import torch
|
| 3 |
+
from torch.utils.checkpoint import checkpoint
|
| 4 |
+
|
| 5 |
+
from . import register_algorithm
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
@register_algorithm('simct')
|
| 9 |
+
class SimCT:
|
| 10 |
+
def __init__(self, strategy, student_model, teacher_lm_head,
|
| 11 |
+
student_tokenizer, teacher_tokenizer, **kwargs):
|
| 12 |
+
# Lazy: selecting Strict/MP never imports the author's implementation.
|
| 13 |
+
from ._simct_upstream.span_ctkd import SpanCrossTokenizerKD
|
| 14 |
+
self.args = strategy.args
|
| 15 |
+
if self.args.scenario != 'on_policy_kd':
|
| 16 |
+
raise ValueError('SimCT baseline supports on-policy KD only')
|
| 17 |
+
if (self.args.kd.kd_ratio != 1 or self.args.kd.kd_temperature != 1
|
| 18 |
+
or self.args.kd.kd_loss_fn != 'rkl'):
|
| 19 |
+
raise ValueError('SimCT baseline requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
|
| 20 |
+
if (self.args.train.num_nodes * self.args.train.num_gpus_per_node != 1
|
| 21 |
+
or self.args.model.ring_attn_size != 1):
|
| 22 |
+
raise ValueError('SimCT baseline initially requires one student rank and ring_attn_size=1')
|
| 23 |
+
self.student, self.teacher_head = student_model, teacher_lm_head
|
| 24 |
+
self.upstream = SpanCrossTokenizerKD(strategy, student_model, teacher_lm_head,
|
| 25 |
+
student_tokenizer, teacher_tokenizer, **kwargs)
|
| 26 |
+
# Preserve calculations but suppress upstream logs of response token IDs/logits.
|
| 27 |
+
self.upstream._debug_count, self.upstream._span_debug_count = 3, 1
|
| 28 |
+
config = getattr(student_model.model.config, 'text_config', None) or student_model.model.config
|
| 29 |
+
self.softcap = getattr(config, 'final_logit_softcapping', None)
|
| 30 |
+
if self.softcap is not None and self.softcap <= 0:
|
| 31 |
+
raise ValueError('final_logit_softcapping must be positive')
|
| 32 |
+
|
| 33 |
+
def _student_logits(self, hidden):
|
| 34 |
+
# Match the author's actual HF student output, including final softcap.
|
| 35 |
+
logits = self.student.model.lm_head(hidden, skip=False)
|
| 36 |
+
return logits if self.softcap is None else torch.tanh(logits / self.softcap) * self.softcap
|
| 37 |
+
|
| 38 |
+
def training_step(self, micro_batch):
|
| 39 |
+
sm, tm = micro_batch['stu_loss_mask'].bool(), micro_batch['tea_loss_mask'].bool()
|
| 40 |
+
if sm.shape[0] != tm.shape[0]:
|
| 41 |
+
raise ValueError('SimCT response counts differ')
|
| 42 |
+
output = self.student(micro_batch['stu_input_ids'], attention_mask=micro_batch['stu_attn_mask'],
|
| 43 |
+
allgather_logits=True, ring_attn_group=None,
|
| 44 |
+
**(micro_batch.get('stu_multi_modal_inputs') or {}))
|
| 45 |
+
hidden = output['hidden_states'][-1][sm]
|
| 46 |
+
th = micro_batch['teacher_hiddens']
|
| 47 |
+
if isinstance(th, list):
|
| 48 |
+
th = torch.cat(th, 0)
|
| 49 |
+
if th.shape[0] != int(tm.sum()):
|
| 50 |
+
raise ValueError('SimCT teacher hidden rows do not match loss mask')
|
| 51 |
+
th = th.to(self.teacher_head.weight)
|
| 52 |
+
sy = micro_batch['stu_input_ids'].roll(-1, 1)
|
| 53 |
+
ty = micro_batch['tea_input_ids'].roll(-1, 1)
|
| 54 |
+
total = hidden.sum() * 0.0
|
| 55 |
+
aligned = response_tokens = dropped = kept = 0
|
| 56 |
+
so = to = 0
|
| 57 |
+
for row in range(sm.shape[0]):
|
| 58 |
+
sid, tid = sy[row][sm[row]].tolist(), ty[row][tm[row]].tolist()
|
| 59 |
+
sh, teacher_hidden = hidden[so:so + len(sid)], th[to:to + len(tid)]
|
| 60 |
+
so, to = so + len(sid), to + len(tid)
|
| 61 |
+
response_tokens += max(len(sid), len(tid))
|
| 62 |
+
segments, tid, sid = self.upstream._align_sequences_with_spans(tid, sid)
|
| 63 |
+
if not segments:
|
| 64 |
+
continue
|
| 65 |
+
# One response at a time; preserve the author's virtual-vocabulary math.
|
| 66 |
+
# Target memory/throughput is not qualified by the CPU parity tests.
|
| 67 |
+
sl = checkpoint(self._student_logits, sh, use_reentrant=False) if torch.is_grad_enabled() else self._student_logits(sh)
|
| 68 |
+
with torch.no_grad():
|
| 69 |
+
tl = self.teacher_head(teacher_hidden) # author's teacher head has no softcap
|
| 70 |
+
sv, tv = self.upstream._build_virtual_vocab_logits(segments, sl, tl, sid, tid)
|
| 71 |
+
losses = self.upstream.loss_fn(sv, tv.detach(), reduction='none')
|
| 72 |
+
threshold = self.args.kd.simct_gh_mask_threshold
|
| 73 |
+
if threshold > 0:
|
| 74 |
+
mass = self.upstream._compute_segment_teacher_Z(segments, tl, tid)
|
| 75 |
+
keep = mass >= 1 / threshold
|
| 76 |
+
kept += int(keep.sum())
|
| 77 |
+
dropped += keep.numel() - int(keep.sum())
|
| 78 |
+
losses = losses * keep.to(losses.dtype)
|
| 79 |
+
total = total + losses.sum()
|
| 80 |
+
aligned += sum(max(te - ts, se - ss) for ts, te, ss, se in segments)
|
| 81 |
+
denominator = micro_batch['avg_micro_batch_token_num']
|
| 82 |
+
if float(denominator) <= 0:
|
| 83 |
+
raise ValueError('SimCT requires positive shared token normalization')
|
| 84 |
+
loss = total / denominator
|
| 85 |
+
if not bool(torch.isfinite(loss)):
|
| 86 |
+
raise FloatingPointError('SimCT produced non-finite loss')
|
| 87 |
+
return {'train/loss': loss, 'train/kd_loss': loss,
|
| 88 |
+
'distill/alignment_ratio': loss.detach().new_tensor(aligned / max(response_tokens, 1)),
|
| 89 |
+
'distill/simct_gh_mask_ratio': loss.detach().new_tensor(dropped / max(dropped + kept, 1))}
|
kdflow/arguments/distillation_args.py
CHANGED
|
@@ -23,7 +23,7 @@ class DistillationArguments:
|
|
| 23 |
default="byte_align_ctkd",
|
| 24 |
metadata={
|
| 25 |
"help": "KD algorithm for each training step.",
|
| 26 |
-
"choices": ["simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", *MP_OPD_ALGORITHMS],
|
| 27 |
}
|
| 28 |
)
|
| 29 |
kd_loss_fn: str = field(
|
|
@@ -111,6 +111,9 @@ class DistillationArguments:
|
|
| 111 |
metadata={"help": "Top-k Ranking for Hierarchical Ranking Loss."}
|
| 112 |
)
|
| 113 |
|
|
|
|
|
|
|
|
|
|
| 114 |
# Opt-in MP-OPD namespace. None of these fields changes a Strict mode.
|
| 115 |
mp_opd_fixed_span_length: int = field(default=2)
|
| 116 |
mp_opd_random_min_length: int = field(default=1)
|
|
@@ -131,12 +134,17 @@ class DistillationArguments:
|
|
| 131 |
"This release supports one teacher model per run; "
|
| 132 |
"use --teacher_name_or_path instead of --multi_teacher_config."
|
| 133 |
)
|
| 134 |
-
supported_algorithms = {"simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", *MP_OPD_ALGORITHMS}
|
| 135 |
if self.kd_algorithm not in supported_algorithms:
|
| 136 |
raise ValueError(
|
| 137 |
f"kd_algorithm must be one of {sorted(supported_algorithms)}, "
|
| 138 |
f"got {self.kd_algorithm!r}."
|
| 139 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
if self.kd_algorithm in MP_OPD_ALGORITHMS:
|
| 141 |
if self.kd_ratio != 1.0 or self.kd_temperature != 1.0 or self.kd_loss_fn != 'rkl':
|
| 142 |
raise ValueError('MP-OPD requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
|
|
|
|
| 23 |
default="byte_align_ctkd",
|
| 24 |
metadata={
|
| 25 |
"help": "KD algorithm for each training step.",
|
| 26 |
+
"choices": ["simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", *MP_OPD_ALGORITHMS],
|
| 27 |
}
|
| 28 |
)
|
| 29 |
kd_loss_fn: str = field(
|
|
|
|
| 111 |
metadata={"help": "Top-k Ranking for Hierarchical Ranking Loss."}
|
| 112 |
)
|
| 113 |
|
| 114 |
+
simct_gh_mask_threshold: float = field(default=0.0,
|
| 115 |
+
metadata={'help': 'Author SimCT optional G(h) mask; 0 disables it as in released launchers.'})
|
| 116 |
+
|
| 117 |
# Opt-in MP-OPD namespace. None of these fields changes a Strict mode.
|
| 118 |
mp_opd_fixed_span_length: int = field(default=2)
|
| 119 |
mp_opd_random_min_length: int = field(default=1)
|
|
|
|
| 134 |
"This release supports one teacher model per run; "
|
| 135 |
"use --teacher_name_or_path instead of --multi_teacher_config."
|
| 136 |
)
|
| 137 |
+
supported_algorithms = {"simple_ctkd", "byte_align_ctkd", "topk_byte_align_ctkd", "simct", *MP_OPD_ALGORITHMS}
|
| 138 |
if self.kd_algorithm not in supported_algorithms:
|
| 139 |
raise ValueError(
|
| 140 |
f"kd_algorithm must be one of {sorted(supported_algorithms)}, "
|
| 141 |
f"got {self.kd_algorithm!r}."
|
| 142 |
)
|
| 143 |
+
if self.kd_algorithm == 'simct':
|
| 144 |
+
if self.kd_ratio != 1 or self.kd_temperature != 1 or self.kd_loss_fn != 'rkl':
|
| 145 |
+
raise ValueError('SimCT baseline requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
|
| 146 |
+
if not math.isfinite(self.simct_gh_mask_threshold) or self.simct_gh_mask_threshold < 0:
|
| 147 |
+
raise ValueError('SimCT G(h) threshold must be finite and nonnegative')
|
| 148 |
if self.kd_algorithm in MP_OPD_ALGORITHMS:
|
| 149 |
if self.kd_ratio != 1.0 or self.kd_temperature != 1.0 or self.kd_loss_fn != 'rkl':
|
| 150 |
raise ValueError('MP-OPD requires kd_ratio=1, kd_temperature=1, kd_loss_fn=rkl')
|
kdflow/cli/train_kd_on_policy.py
CHANGED
|
@@ -22,6 +22,10 @@ from kdflow.utils.utils import get_tokenizer, load_custom_eval_fn
|
|
| 22 |
|
| 23 |
|
| 24 |
def train(args):
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
if args.kd.kd_algorithm in MP_OPD_ALGORITHMS and (
|
| 26 |
args.train.num_nodes * args.train.num_gpus_per_node != 1
|
| 27 |
or args.model.ring_attn_size != 1
|
|
@@ -91,11 +95,12 @@ def train(args):
|
|
| 91 |
"simple_ctkd",
|
| 92 |
"byte_align_ctkd",
|
| 93 |
"topk_byte_align_ctkd",
|
|
|
|
| 94 |
*MP_OPD_ALGORITHMS,
|
| 95 |
}:
|
| 96 |
raise ValueError(
|
| 97 |
"Cross-tokenizer distillation supports simple_ctkd, "
|
| 98 |
-
"byte_align_ctkd, topk_byte_align_ctkd, or an opt-in mp_opd_* mode."
|
| 99 |
)
|
| 100 |
|
| 101 |
environment_rollout = bool(args.rollout.alfworld_config)
|
|
|
|
| 22 |
|
| 23 |
|
| 24 |
def train(args):
|
| 25 |
+
if args.kd.kd_algorithm == 'simct' and (
|
| 26 |
+
args.train.num_nodes * args.train.num_gpus_per_node != 1 or args.model.ring_attn_size != 1
|
| 27 |
+
):
|
| 28 |
+
raise ValueError('SimCT baseline initially requires one student rank and ring_attn_size=1')
|
| 29 |
if args.kd.kd_algorithm in MP_OPD_ALGORITHMS and (
|
| 30 |
args.train.num_nodes * args.train.num_gpus_per_node != 1
|
| 31 |
or args.model.ring_attn_size != 1
|
|
|
|
| 95 |
"simple_ctkd",
|
| 96 |
"byte_align_ctkd",
|
| 97 |
"topk_byte_align_ctkd",
|
| 98 |
+
"simct",
|
| 99 |
*MP_OPD_ALGORITHMS,
|
| 100 |
}:
|
| 101 |
raise ValueError(
|
| 102 |
"Cross-tokenizer distillation supports simple_ctkd, "
|
| 103 |
+
"byte_align_ctkd, topk_byte_align_ctkd, simct, or an opt-in mp_opd_* mode."
|
| 104 |
)
|
| 105 |
|
| 106 |
environment_rollout = bool(args.rollout.alfworld_config)
|
tests/fixtures/simct_upstream/reverse_kl_div.py.txt
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
|
| 3 |
+
from kdflow.loss import register_loss
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
@register_loss("rkl")
|
| 7 |
+
@torch.compile()
|
| 8 |
+
def compute_reverse_kl_div(
|
| 9 |
+
student_logits,
|
| 10 |
+
teacher_logits,
|
| 11 |
+
temperature=1.0,
|
| 12 |
+
reduction="none",
|
| 13 |
+
**kwargs
|
| 14 |
+
):
|
| 15 |
+
student_logits = student_logits / temperature
|
| 16 |
+
teacher_logits = teacher_logits / temperature
|
| 17 |
+
student_log_probs = torch.log_softmax(student_logits, -1, dtype=torch.float32)
|
| 18 |
+
teacher_log_probs = torch.log_softmax(teacher_logits, -1, dtype=torch.float32)
|
| 19 |
+
student_probs = student_log_probs.exp()
|
| 20 |
+
rkl_div = (student_probs * (student_log_probs - teacher_log_probs)).sum(-1)
|
| 21 |
+
|
| 22 |
+
if reduction == "mean":
|
| 23 |
+
rkl_div = rkl_div.mean()
|
| 24 |
+
elif reduction == "sum":
|
| 25 |
+
rkl_div = rkl_div.sum()
|
| 26 |
+
|
| 27 |
+
return rkl_div
|
tests/fixtures/simct_upstream/span_ctkd.py.txt
ADDED
|
@@ -0,0 +1,508 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
import torch.nn.functional as F
|
| 3 |
+
|
| 4 |
+
from kdflow.loss import build_loss_fn
|
| 5 |
+
from kdflow.algorithms import register_algorithm
|
| 6 |
+
from kdflow.loss.cross_entropy import compute_cross_entropy
|
| 7 |
+
from kdflow.utils.logging_utils import init_logger
|
| 8 |
+
|
| 9 |
+
logger = init_logger(__name__)
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
@register_algorithm("span_ctkd")
|
| 13 |
+
class SpanCrossTokenizerKD:
|
| 14 |
+
"""Span-based Cross-Tokenizer Knowledge Distillation.
|
| 15 |
+
|
| 16 |
+
Improvement over SimpleCrossTokenizerKD: instead of discarding positions where
|
| 17 |
+
student and teacher tokens do not align 1:1, we group misaligned tokens into
|
| 18 |
+
"spans" and compute their logits as the arithmetic mean of the constituent
|
| 19 |
+
token logits (which corresponds to geometric mean after softmax).
|
| 20 |
+
|
| 21 |
+
These spans are appended to the shared (overlap) vocabulary to form a
|
| 22 |
+
"virtual common vocabulary". RKL alignment is then performed on this
|
| 23 |
+
extended vocabulary for every aligned position (both 1:1 tokens and spans).
|
| 24 |
+
|
| 25 |
+
For span segments, the overlap logits are taken from the **first** token
|
| 26 |
+
position only (not averaged across positions), because the span is treated
|
| 27 |
+
as a single virtual token occupying one position in the sequence.
|
| 28 |
+
"""
|
| 29 |
+
|
| 30 |
+
def __init__(
|
| 31 |
+
self,
|
| 32 |
+
strategy,
|
| 33 |
+
student_model,
|
| 34 |
+
teacher_lm_head,
|
| 35 |
+
student_tokenizer,
|
| 36 |
+
teacher_tokenizer,
|
| 37 |
+
**kwargs,
|
| 38 |
+
):
|
| 39 |
+
self.strategy = strategy
|
| 40 |
+
self.args = strategy.args
|
| 41 |
+
self.student = student_model
|
| 42 |
+
self.teacher_lm_head = teacher_lm_head
|
| 43 |
+
self.student_tokenizer = student_tokenizer
|
| 44 |
+
self.teacher_tokenizer = teacher_tokenizer
|
| 45 |
+
self.student_overlap_token_ids, self.teacher_overlap_token_ids = self._find_overlap_tokens()
|
| 46 |
+
self.loss_fn = build_loss_fn(self.args.kd.kd_loss_fn, self.args)
|
| 47 |
+
self._debug_count = 0 # Counter for debug logging
|
| 48 |
+
self._span_debug_count = 0 # Counter for span logits debug logging
|
| 49 |
+
|
| 50 |
+
# ------------------------------------------------------------------
|
| 51 |
+
# Overlap vocabulary (copied from simple_ctkd)
|
| 52 |
+
# ------------------------------------------------------------------
|
| 53 |
+
def _find_overlap_tokens(self):
|
| 54 |
+
"""Find the overlap tokens between student and teacher tokenizer."""
|
| 55 |
+
student_vocab = {k.replace("Ġ", "▁"): v for k, v in self.student_tokenizer.get_vocab().items()}
|
| 56 |
+
teacher_vocab = {k.replace("Ġ", "▁"): v for k, v in self.teacher_tokenizer.get_vocab().items()}
|
| 57 |
+
overlap_tokens = set(student_vocab.keys()) & set(teacher_vocab.keys())
|
| 58 |
+
student_ids = [student_vocab[token] for token in overlap_tokens]
|
| 59 |
+
teacher_ids = [teacher_vocab[token] for token in overlap_tokens]
|
| 60 |
+
stu_eos, tea_eos = self.student_tokenizer.eos_token_id, self.teacher_tokenizer.eos_token_id
|
| 61 |
+
if stu_eos not in student_ids or tea_eos not in teacher_ids:
|
| 62 |
+
student_ids.append(stu_eos)
|
| 63 |
+
teacher_ids.append(tea_eos)
|
| 64 |
+
device = self.teacher_lm_head.weight.device
|
| 65 |
+
logger.info(f"[SpanCTKD] Num of overlap_tokens between student & teacher: {len(student_ids)}")
|
| 66 |
+
return (
|
| 67 |
+
torch.tensor(student_ids, dtype=torch.long, device=device),
|
| 68 |
+
torch.tensor(teacher_ids, dtype=torch.long, device=device),
|
| 69 |
+
)
|
| 70 |
+
|
| 71 |
+
# ------------------------------------------------------------------
|
| 72 |
+
# Sequence alignment with span identification
|
| 73 |
+
# (adapted from ALM._compute_chunk_alignment)
|
| 74 |
+
# ------------------------------------------------------------------
|
| 75 |
+
def _align_sequences_with_spans(self, tea_label_ids, stu_label_ids):
|
| 76 |
+
"""Align teacher and student label tokens and identify spans.
|
| 77 |
+
|
| 78 |
+
Operates on **label ids** (i.e. input_ids shifted left by one position,
|
| 79 |
+
already filtered by loss_mask) so that alignment is consistent with the
|
| 80 |
+
next-token-prediction semantics of the logits.
|
| 81 |
+
|
| 82 |
+
Uses ``tokenizer.decode([tid])`` per token (O(n)) to correctly handle
|
| 83 |
+
all tokenizer conventions including newlines.
|
| 84 |
+
|
| 85 |
+
Args:
|
| 86 |
+
tea_label_ids: 1-D list of teacher label token ids (loss_mask positions).
|
| 87 |
+
stu_label_ids: 1-D list of student label token ids (loss_mask positions).
|
| 88 |
+
|
| 89 |
+
Returns:
|
| 90 |
+
segments: list of (tea_start, tea_end, stu_start, stu_end) where
|
| 91 |
+
indices are positions within the label id sequences. Each tuple
|
| 92 |
+
represents an aligned segment: teacher tokens ``[tea_start, tea_end)``
|
| 93 |
+
and student tokens ``[stu_start, stu_end)`` encode the same text.
|
| 94 |
+
When both ranges have length 1 it is a normal 1:1 alignment;
|
| 95 |
+
otherwise it is a span.
|
| 96 |
+
tea_label_ids_list: the teacher label ids as a Python list.
|
| 97 |
+
stu_label_ids_list: the student label ids as a Python list.
|
| 98 |
+
"""
|
| 99 |
+
if len(tea_label_ids) == 0 or len(stu_label_ids) == 0:
|
| 100 |
+
return [], [], []
|
| 101 |
+
|
| 102 |
+
tea_ids_list = tea_label_ids if isinstance(tea_label_ids, list) else tea_label_ids.cpu().tolist()
|
| 103 |
+
stu_ids_list = stu_label_ids if isinstance(stu_label_ids, list) else stu_label_ids.cpu().tolist()
|
| 104 |
+
|
| 105 |
+
# Convert token ids to text strings for alignment.
|
| 106 |
+
# Use tokenizer.decode([tid]) per token to correctly handle all
|
| 107 |
+
# tokenizer conventions (e.g. Qwen uses Ċ for \n internally, which
|
| 108 |
+
# would NOT be normalized by simple strip of ▁/Ġ prefixes).
|
| 109 |
+
tea_token_texts = [self.teacher_tokenizer.decode([tid]) for tid in tea_ids_list]
|
| 110 |
+
stu_token_texts = [self.student_tokenizer.decode([tid]) for tid in stu_ids_list]
|
| 111 |
+
|
| 112 |
+
# Greedy cumulative-text alignment (same algorithm as simple_ctkd)
|
| 113 |
+
tea_eos = self.teacher_tokenizer.eos_token
|
| 114 |
+
stu_eos = self.student_tokenizer.eos_token
|
| 115 |
+
|
| 116 |
+
i, j = 0, 0
|
| 117 |
+
boundaries = []
|
| 118 |
+
history_tea = ""
|
| 119 |
+
history_stu = ""
|
| 120 |
+
|
| 121 |
+
while i < len(tea_token_texts) and j < len(stu_token_texts):
|
| 122 |
+
is_eos_match = (tea_token_texts[i] == tea_eos and stu_token_texts[j] == stu_eos)
|
| 123 |
+
if history_tea == history_stu and (
|
| 124 |
+
tea_token_texts[i] == stu_token_texts[j] or is_eos_match
|
| 125 |
+
):
|
| 126 |
+
boundaries.append((i, j))
|
| 127 |
+
history_tea += tea_token_texts[i]
|
| 128 |
+
history_stu += stu_token_texts[j]
|
| 129 |
+
i += 1
|
| 130 |
+
j += 1
|
| 131 |
+
elif len(history_tea) > len(history_stu):
|
| 132 |
+
history_stu += stu_token_texts[j]
|
| 133 |
+
j += 1
|
| 134 |
+
elif len(history_tea) < len(history_stu):
|
| 135 |
+
history_tea += tea_token_texts[i]
|
| 136 |
+
i += 1
|
| 137 |
+
else:
|
| 138 |
+
history_tea += tea_token_texts[i]
|
| 139 |
+
history_stu += stu_token_texts[j]
|
| 140 |
+
i += 1
|
| 141 |
+
j += 1
|
| 142 |
+
|
| 143 |
+
if len(boundaries) == 0:
|
| 144 |
+
if self._debug_count < 3:
|
| 145 |
+
self._debug_count += 1
|
| 146 |
+
logger.warning(
|
| 147 |
+
f"[SpanCTKD DEBUG] No boundaries found! "
|
| 148 |
+
f"tea_token_texts({len(tea_token_texts)}): {tea_token_texts[:10]}... "
|
| 149 |
+
f"stu_token_texts({len(stu_token_texts)}): {stu_token_texts[:10]}..."
|
| 150 |
+
)
|
| 151 |
+
return [], tea_ids_list, stu_ids_list
|
| 152 |
+
|
| 153 |
+
if self._debug_count < 3:
|
| 154 |
+
self._debug_count += 1
|
| 155 |
+
logger.info(
|
| 156 |
+
f"[SpanCTKD DEBUG] Found {len(boundaries)} boundaries! "
|
| 157 |
+
f"tea_texts({len(tea_token_texts)}): {tea_token_texts[:5]}... "
|
| 158 |
+
f"stu_texts({len(stu_token_texts)}): {stu_token_texts[:5]}..."
|
| 159 |
+
)
|
| 160 |
+
|
| 161 |
+
# Convert boundaries to aligned segments (local indices within response)
|
| 162 |
+
segments = []
|
| 163 |
+
for idx in range(len(boundaries)):
|
| 164 |
+
if idx == 0:
|
| 165 |
+
local_tea_start, local_stu_start = 0, 0
|
| 166 |
+
else:
|
| 167 |
+
local_tea_start = boundaries[idx - 1][0] + 1
|
| 168 |
+
local_stu_start = boundaries[idx - 1][1] + 1
|
| 169 |
+
local_tea_end = boundaries[idx][0] + 1
|
| 170 |
+
local_stu_end = boundaries[idx][1] + 1
|
| 171 |
+
segments.append((local_tea_start, local_tea_end, local_stu_start, local_stu_end))
|
| 172 |
+
|
| 173 |
+
return segments, tea_ids_list, stu_ids_list
|
| 174 |
+
|
| 175 |
+
# ------------------------------------------------------------------
|
| 176 |
+
# Build virtual common vocabulary logits for a single sample
|
| 177 |
+
# ------------------------------------------------------------------
|
| 178 |
+
def _build_virtual_vocab_logits(
|
| 179 |
+
self,
|
| 180 |
+
segments,
|
| 181 |
+
stu_logits_aligned,
|
| 182 |
+
tea_logits_aligned,
|
| 183 |
+
stu_label_ids_list,
|
| 184 |
+
tea_label_ids_list,
|
| 185 |
+
):
|
| 186 |
+
"""Build student and teacher logit matrices on the virtual common vocabulary.
|
| 187 |
+
|
| 188 |
+
For each aligned segment we construct a logit vector of size
|
| 189 |
+
``num_overlap + num_total_spans`` where:
|
| 190 |
+
- the first ``num_overlap`` dims come from the base overlap vocabulary,
|
| 191 |
+
- the remaining ``num_total_spans`` dims correspond to span logits.
|
| 192 |
+
|
| 193 |
+
For a 1:1 aligned position the overlap-vocab logits are taken directly
|
| 194 |
+
from the single token; for a span the overlap-vocab logits are taken
|
| 195 |
+
from the **first** token position only (since the span is treated as a
|
| 196 |
+
single virtual token — other overlap tokens compete with it at the
|
| 197 |
+
first position). The span-logit dimensions are filled with ``-1e9``
|
| 198 |
+
(≈ 0 after softmax) except for the span that the current segment
|
| 199 |
+
belongs to, which gets the mean of each constituent token's logit at
|
| 200 |
+
its own token id (i.e. the model's average confidence in generating
|
| 201 |
+
this span).
|
| 202 |
+
|
| 203 |
+
Args:
|
| 204 |
+
segments: list of (tea_start, tea_end, stu_start, stu_end) —
|
| 205 |
+
indices into the label id sequences.
|
| 206 |
+
stu_logits_aligned: student logits at loss_mask positions
|
| 207 |
+
[num_stu_loss_tokens, vocab_s].
|
| 208 |
+
tea_logits_aligned: teacher logits at loss_mask positions
|
| 209 |
+
[num_tea_loss_tokens, vocab_t].
|
| 210 |
+
stu_label_ids_list: Python list of student label token ids.
|
| 211 |
+
tea_label_ids_list: Python list of teacher label token ids.
|
| 212 |
+
|
| 213 |
+
Returns:
|
| 214 |
+
stu_virtual_logits: [num_segments, num_overlap + num_spans]
|
| 215 |
+
tea_virtual_logits: [num_segments, num_overlap + num_spans]
|
| 216 |
+
"""
|
| 217 |
+
num_overlap = self.student_overlap_token_ids.shape[0]
|
| 218 |
+
device = stu_logits_aligned.device
|
| 219 |
+
|
| 220 |
+
# Identify which segments are spans (multi-token on either side)
|
| 221 |
+
span_indices = [] # indices into `segments` that are spans
|
| 222 |
+
for seg_idx, (ts, te, ss, se) in enumerate(segments):
|
| 223 |
+
if (te - ts) > 1 or (se - ss) > 1:
|
| 224 |
+
span_indices.append(seg_idx)
|
| 225 |
+
num_spans = len(span_indices)
|
| 226 |
+
# Map from segment index to its position in the span dimensions
|
| 227 |
+
seg_to_span_dim = {}
|
| 228 |
+
for dim_idx, seg_idx in enumerate(span_indices):
|
| 229 |
+
seg_to_span_dim[seg_idx] = dim_idx
|
| 230 |
+
|
| 231 |
+
virtual_dim = num_overlap + num_spans
|
| 232 |
+
|
| 233 |
+
stu_rows = []
|
| 234 |
+
tea_rows = []
|
| 235 |
+
|
| 236 |
+
for seg_idx, (ts, te, ss, se) in enumerate(segments):
|
| 237 |
+
# --- Student side ---
|
| 238 |
+
stu_seg_logits = stu_logits_aligned[ss:se] # [num_stu_tokens, vocab_s]
|
| 239 |
+
# For overlap logits, always use the FIRST token position only.
|
| 240 |
+
# The span is a single virtual token; other overlap tokens compete
|
| 241 |
+
# with it at the first position (i.e. "what comes after the prefix").
|
| 242 |
+
stu_first_logits = stu_seg_logits[0] # [vocab_s]
|
| 243 |
+
stu_overlap = stu_first_logits[self.student_overlap_token_ids] # [num_overlap]
|
| 244 |
+
|
| 245 |
+
# --- Teacher side ---
|
| 246 |
+
tea_seg_logits = tea_logits_aligned[ts:te] # [num_tea_tokens, vocab_t]
|
| 247 |
+
tea_first_logits = tea_seg_logits[0] # [vocab_t]
|
| 248 |
+
tea_overlap = tea_first_logits[self.teacher_overlap_token_ids] # [num_overlap]
|
| 249 |
+
|
| 250 |
+
if num_spans > 0:
|
| 251 |
+
# Span dimensions: default to -1e9 (negligible after softmax)
|
| 252 |
+
stu_span_dims = torch.full((num_spans,), -1e9, device=device, dtype=stu_overlap.dtype)
|
| 253 |
+
tea_span_dims = torch.full((num_spans,), -1e9, device=device, dtype=tea_overlap.dtype)
|
| 254 |
+
|
| 255 |
+
if seg_idx in seg_to_span_dim:
|
| 256 |
+
dim_pos = seg_to_span_dim[seg_idx]
|
| 257 |
+
# Span logit = mean of each constituent token's logit at
|
| 258 |
+
# its own token id. This represents the model's average
|
| 259 |
+
# confidence in generating this specific span of text.
|
| 260 |
+
# After softmax this corresponds to the geometric mean of
|
| 261 |
+
# per-token probabilities (avoiding length-dependent decay).
|
| 262 |
+
stu_span_token_ids = stu_label_ids_list[ss:se]
|
| 263 |
+
stu_self_logits = torch.stack([
|
| 264 |
+
stu_seg_logits[k, tid]
|
| 265 |
+
for k, tid in enumerate(stu_span_token_ids)
|
| 266 |
+
])
|
| 267 |
+
stu_span_dims[dim_pos] = stu_self_logits.mean()
|
| 268 |
+
|
| 269 |
+
tea_span_token_ids = tea_label_ids_list[ts:te]
|
| 270 |
+
tea_self_logits = torch.stack([
|
| 271 |
+
tea_seg_logits[k, tid]
|
| 272 |
+
for k, tid in enumerate(tea_span_token_ids)
|
| 273 |
+
])
|
| 274 |
+
tea_span_dims[dim_pos] = tea_self_logits.mean()
|
| 275 |
+
|
| 276 |
+
# Debug: print one example of span logits vs overlap logits
|
| 277 |
+
if self._span_debug_count < 1:
|
| 278 |
+
self._span_debug_count += 1
|
| 279 |
+
logger.info(
|
| 280 |
+
f"\n[SpanCTKD SPAN DEBUG] === Example span logits (seg_idx={seg_idx}) ===\n"
|
| 281 |
+
f" Student span token ids: {stu_span_token_ids}\n"
|
| 282 |
+
f" Student self_logits (per token): {stu_self_logits.detach().cpu().tolist()}\n"
|
| 283 |
+
f" Student span_dim value (mean): {stu_span_dims[dim_pos].item():.4f}\n"
|
| 284 |
+
f" Teacher span token ids: {tea_span_token_ids}\n"
|
| 285 |
+
f" Teacher self_logits (per token): {tea_self_logits.detach().cpu().tolist()}\n"
|
| 286 |
+
f" Teacher span_dim value (mean): {tea_span_dims[dim_pos].item():.4f}\n"
|
| 287 |
+
f" ---\n"
|
| 288 |
+
f" Student overlap logits (first pos) stats: mean={stu_overlap.mean().item():.4f}, "
|
| 289 |
+
f"max={stu_overlap.max().item():.4f}, min={stu_overlap.min().item():.4f}, "
|
| 290 |
+
f"std={stu_overlap.std().item():.4f}\n"
|
| 291 |
+
f" Teacher overlap logits (first pos) stats: mean={tea_overlap.mean().item():.4f}, "
|
| 292 |
+
f"max={tea_overlap.max().item():.4f}, min={tea_overlap.min().item():.4f}, "
|
| 293 |
+
f"std={tea_overlap.std().item():.4f}\n"
|
| 294 |
+
f" num_overlap={num_overlap}, num_spans={num_spans}, virtual_dim={virtual_dim}\n"
|
| 295 |
+
f" All stu_span_dims: {stu_span_dims.detach().cpu().tolist()}\n"
|
| 296 |
+
f" All tea_span_dims: {tea_span_dims.detach().cpu().tolist()}"
|
| 297 |
+
)
|
| 298 |
+
|
| 299 |
+
stu_row = torch.cat([stu_overlap, stu_span_dims]) # [virtual_dim]
|
| 300 |
+
tea_row = torch.cat([tea_overlap, tea_span_dims]) # [virtual_dim]
|
| 301 |
+
else:
|
| 302 |
+
stu_row = stu_overlap
|
| 303 |
+
tea_row = tea_overlap
|
| 304 |
+
|
| 305 |
+
stu_rows.append(stu_row)
|
| 306 |
+
tea_rows.append(tea_row)
|
| 307 |
+
|
| 308 |
+
stu_virtual_logits = torch.stack(stu_rows) # [num_segments, virtual_dim]
|
| 309 |
+
tea_virtual_logits = torch.stack(tea_rows) # [num_segments, virtual_dim]
|
| 310 |
+
return stu_virtual_logits, tea_virtual_logits
|
| 311 |
+
|
| 312 |
+
# ------------------------------------------------------------------
|
| 313 |
+
# Re-normalization multiplier G(h) safeguard
|
| 314 |
+
# ------------------------------------------------------------------
|
| 315 |
+
@torch.no_grad()
|
| 316 |
+
def _compute_segment_teacher_Z(self, segments, tea_logits_aligned, tea_label_ids_list):
|
| 317 |
+
"""Teacher mass ``Z_T(h)`` captured by the SimCT candidate space, per segment.
|
| 318 |
+
|
| 319 |
+
``Z_T(h)`` is computed on the **full-vocabulary** teacher softmax so that
|
| 320 |
+
it measures the original teacher probability mass that the candidate
|
| 321 |
+
space (shared tokens + the span realized at this position) represents:
|
| 322 |
+
|
| 323 |
+
- shared-token mass at the first teacher position, plus
|
| 324 |
+
- for a span, the geometric-mean probability of the span's teacher
|
| 325 |
+
tokens (matching the span logit used in the virtual distribution).
|
| 326 |
+
|
| 327 |
+
The re-normalization multiplier is then ``G(h) = 1 / Z_T(h)``. A large
|
| 328 |
+
``G(h)`` means the candidate space holds little teacher mass, so
|
| 329 |
+
re-normalizing over it would amplify a low-mass tail.
|
| 330 |
+
|
| 331 |
+
Returns:
|
| 332 |
+
Z: [num_segments] float tensor in (0, 1].
|
| 333 |
+
"""
|
| 334 |
+
device = tea_logits_aligned.device
|
| 335 |
+
Z = torch.ones(len(segments), device=device, dtype=torch.float32)
|
| 336 |
+
for seg_idx, (ts, te, ss, se) in enumerate(segments):
|
| 337 |
+
tea_seg_logits = tea_logits_aligned[ts:te] # [k_t, vocab_t]
|
| 338 |
+
p_first = torch.softmax(tea_seg_logits[0].float(), dim=-1) # full-vocab probs
|
| 339 |
+
m = p_first[self.teacher_overlap_token_ids].sum() # shared-token mass
|
| 340 |
+
if (te - ts) > 1 or (se - ss) > 1:
|
| 341 |
+
span_ids = tea_label_ids_list[ts:te]
|
| 342 |
+
logps = torch.stack([
|
| 343 |
+
torch.log_softmax(tea_seg_logits[k].float(), dim=-1)[tid]
|
| 344 |
+
for k, tid in enumerate(span_ids)
|
| 345 |
+
])
|
| 346 |
+
span_prob = torch.exp(logps.mean()) # geometric mean
|
| 347 |
+
Z[seg_idx] = (m + span_prob).clamp(max=1.0)
|
| 348 |
+
else:
|
| 349 |
+
Z[seg_idx] = m.clamp(max=1.0)
|
| 350 |
+
return Z
|
| 351 |
+
|
| 352 |
+
# ------------------------------------------------------------------
|
| 353 |
+
# Training step
|
| 354 |
+
# ------------------------------------------------------------------
|
| 355 |
+
def training_step(self, micro_batch):
|
| 356 |
+
"""Perform one training step with span-based cross-tokenizer KD loss.
|
| 357 |
+
|
| 358 |
+
Flow:
|
| 359 |
+
1. Student forward pass → full-sequence logits.
|
| 360 |
+
2. Teacher logits from hidden states (loss_mask positions only).
|
| 361 |
+
3. Per-sample: align sequences, identify spans, build virtual vocab
|
| 362 |
+
logits, compute RKL loss.
|
| 363 |
+
"""
|
| 364 |
+
student_input_ids = micro_batch["stu_input_ids"]
|
| 365 |
+
student_attn_mask = micro_batch["stu_attn_mask"]
|
| 366 |
+
student_loss_mask = micro_batch["stu_loss_mask"].bool()
|
| 367 |
+
teacher_input_ids = micro_batch["tea_input_ids"]
|
| 368 |
+
teacher_attn_mask = micro_batch["tea_attn_mask"]
|
| 369 |
+
teacher_loss_mask = micro_batch["tea_loss_mask"].bool()
|
| 370 |
+
teacher_hiddens = micro_batch.get("teacher_hiddens", None)
|
| 371 |
+
avg_token_num = micro_batch["avg_micro_batch_token_num"]
|
| 372 |
+
|
| 373 |
+
assert teacher_hiddens is not None, "micro_batch must contain `teacher_hiddens` for KD"
|
| 374 |
+
|
| 375 |
+
mm_kwargs = {k[3:]: v for k, v in micro_batch.items() if k.startswith("mm_")}
|
| 376 |
+
|
| 377 |
+
# Student forward
|
| 378 |
+
output = self.student(
|
| 379 |
+
student_input_ids,
|
| 380 |
+
attention_mask=student_attn_mask,
|
| 381 |
+
allgather_logits=True,
|
| 382 |
+
ring_attn_group=self.strategy.ring_attn_group,
|
| 383 |
+
**mm_kwargs,
|
| 384 |
+
)
|
| 385 |
+
student_logits = output["logits"]
|
| 386 |
+
|
| 387 |
+
# Extract label ids (next-token) at loss_mask positions — consistent
|
| 388 |
+
# with simple_ctkd which aligns on labels, not inputs.
|
| 389 |
+
student_label_ids = student_input_ids.roll(shifts=-1, dims=1)
|
| 390 |
+
teacher_label_ids = teacher_input_ids.roll(shifts=-1, dims=1)
|
| 391 |
+
|
| 392 |
+
# Flatten student logits at loss_mask positions (same as simple_ctkd)
|
| 393 |
+
student_logits_flat = student_logits[student_loss_mask] # [total_stu_loss_tokens, vocab_s]
|
| 394 |
+
|
| 395 |
+
# Free full student logits early — only loss_mask positions are needed
|
| 396 |
+
del student_logits, output
|
| 397 |
+
|
| 398 |
+
# NOTE: `teacher_hiddens` from teacher_actor is ALREADY 2D and filtered
|
| 399 |
+
# by loss_mask — shape [total_tea_loss_tokens, hidden_size]. See
|
| 400 |
+
# teacher_actor.py where hidden states are gathered only at loss_mask
|
| 401 |
+
# positions and np.concatenate-d across the micro-batch. Do NOT re-index
|
| 402 |
+
# with teacher_loss_mask (that would be shape [B, T] and mismatch).
|
| 403 |
+
teacher_hiddens = teacher_hiddens.to(self.teacher_lm_head.weight)
|
| 404 |
+
teacher_logits_flat = self.teacher_lm_head(teacher_hiddens) # [total_tea_loss_tokens, vocab_t]
|
| 405 |
+
del teacher_hiddens # free after lm_head computation
|
| 406 |
+
|
| 407 |
+
# Process each sample in the batch
|
| 408 |
+
batch_size = student_input_ids.shape[0]
|
| 409 |
+
total_loss = torch.tensor(0.0, device=student_logits_flat.device, requires_grad=True)
|
| 410 |
+
total_aligned_tokens = 0
|
| 411 |
+
total_response_tokens = 0
|
| 412 |
+
total_kept_segments = 0
|
| 413 |
+
total_masked_segments = 0
|
| 414 |
+
|
| 415 |
+
# Offset trackers for flattened logits (both are 2D across batch)
|
| 416 |
+
tea_logits_offset = 0
|
| 417 |
+
stu_logits_offset = 0
|
| 418 |
+
|
| 419 |
+
for b in range(batch_size):
|
| 420 |
+
stu_mask = student_loss_mask[b]
|
| 421 |
+
tea_mask = teacher_loss_mask[b]
|
| 422 |
+
|
| 423 |
+
stu_num_loss_tokens = stu_mask.sum().item()
|
| 424 |
+
tea_num_loss_tokens = tea_mask.sum().item()
|
| 425 |
+
|
| 426 |
+
# Student logits for this sample (already flattened by loss_mask)
|
| 427 |
+
stu_logits_b = student_logits_flat[stu_logits_offset:stu_logits_offset + stu_num_loss_tokens]
|
| 428 |
+
stu_logits_offset += stu_num_loss_tokens
|
| 429 |
+
|
| 430 |
+
# Teacher logits for this sample (already flattened by loss_mask)
|
| 431 |
+
tea_logits_b = teacher_logits_flat[tea_logits_offset:tea_logits_offset + tea_num_loss_tokens]
|
| 432 |
+
tea_logits_offset += tea_num_loss_tokens
|
| 433 |
+
|
| 434 |
+
# Extract label ids at loss_mask positions
|
| 435 |
+
stu_label_ids_b = student_label_ids[b][stu_mask].cpu().tolist()
|
| 436 |
+
tea_label_ids_b = teacher_label_ids[b][tea_mask].cpu().tolist()
|
| 437 |
+
|
| 438 |
+
# Align on label ids (next-token) and identify spans
|
| 439 |
+
segments, tea_ids_list, stu_ids_list = self._align_sequences_with_spans(
|
| 440 |
+
tea_label_ids_b, stu_label_ids_b
|
| 441 |
+
)
|
| 442 |
+
|
| 443 |
+
if len(segments) == 0:
|
| 444 |
+
total_response_tokens += max(stu_num_loss_tokens, tea_num_loss_tokens)
|
| 445 |
+
continue
|
| 446 |
+
|
| 447 |
+
# Build virtual vocab logits and compute loss.
|
| 448 |
+
# stu_logits_b and tea_logits_b are already [num_loss_tokens, vocab]
|
| 449 |
+
# and segments index directly into them.
|
| 450 |
+
stu_virtual, tea_virtual = self._build_virtual_vocab_logits(
|
| 451 |
+
segments, stu_logits_b, tea_logits_b,
|
| 452 |
+
stu_ids_list, tea_ids_list,
|
| 453 |
+
)
|
| 454 |
+
|
| 455 |
+
assert stu_virtual.shape == tea_virtual.shape, \
|
| 456 |
+
f"Virtual logit shape mismatch: student {stu_virtual.shape} vs teacher {tea_virtual.shape}"
|
| 457 |
+
|
| 458 |
+
# Compute RKL loss on virtual common vocabulary
|
| 459 |
+
sample_loss = self.loss_fn(
|
| 460 |
+
stu_virtual,
|
| 461 |
+
tea_virtual.detach(),
|
| 462 |
+
reduction="none",
|
| 463 |
+
)
|
| 464 |
+
|
| 465 |
+
# Re-normalization safeguard: drop positions whose multiplier
|
| 466 |
+
# G(h) = 1 / Z_T(h) exceeds the threshold (teacher captured mass
|
| 467 |
+
# Z_T(h) < 1/threshold). These low-mass positions would otherwise
|
| 468 |
+
# let re-normalization amplify a weak/noisy teacher tail, so they
|
| 469 |
+
# contribute no distillation gradient.
|
| 470 |
+
gh_threshold = getattr(self.args.kd, "span_gh_mask_threshold", 0.0)
|
| 471 |
+
if gh_threshold and gh_threshold > 0:
|
| 472 |
+
Z = self._compute_segment_teacher_Z(segments, tea_logits_b, tea_ids_list)
|
| 473 |
+
keep = (Z >= (1.0 / gh_threshold)).to(sample_loss.dtype) # G(h) <= threshold
|
| 474 |
+
total_kept_segments += int(keep.sum().item())
|
| 475 |
+
total_masked_segments += int((keep.numel() - keep.sum()).item())
|
| 476 |
+
sample_loss = sample_loss * keep
|
| 477 |
+
|
| 478 |
+
total_loss = total_loss + sample_loss.sum()
|
| 479 |
+
|
| 480 |
+
# Count aligned tokens (sum of all segment lengths on student side)
|
| 481 |
+
for ts, te, ss, se in segments:
|
| 482 |
+
total_aligned_tokens += max(te - ts, se - ss)
|
| 483 |
+
total_response_tokens += max(stu_num_loss_tokens, tea_num_loss_tokens)
|
| 484 |
+
|
| 485 |
+
# Normalize
|
| 486 |
+
kd_loss = total_loss / avg_token_num
|
| 487 |
+
|
| 488 |
+
align_ratio = torch.tensor(
|
| 489 |
+
total_aligned_tokens / max(total_response_tokens, 1),
|
| 490 |
+
device=student_logits_flat.device,
|
| 491 |
+
)
|
| 492 |
+
|
| 493 |
+
loss_info = {"loss": kd_loss, "kd_loss": kd_loss, "align_ratio": align_ratio}
|
| 494 |
+
|
| 495 |
+
gh_masked_total = total_kept_segments + total_masked_segments
|
| 496 |
+
loss_info["gh_mask_ratio"] = torch.tensor(
|
| 497 |
+
total_masked_segments / max(gh_masked_total, 1),
|
| 498 |
+
device=student_logits_flat.device,
|
| 499 |
+
)
|
| 500 |
+
|
| 501 |
+
if self.args.kd.kd_ratio < 1:
|
| 502 |
+
ce_label_ids = student_label_ids[student_loss_mask]
|
| 503 |
+
ce_loss = compute_cross_entropy(student_logits_flat, ce_label_ids, reduction="sum") / avg_token_num
|
| 504 |
+
loss = (1 - self.args.kd.kd_ratio) * ce_loss + self.args.kd.kd_ratio * kd_loss
|
| 505 |
+
loss_info["loss"] = loss
|
| 506 |
+
loss_info["ce_loss"] = ce_loss
|
| 507 |
+
|
| 508 |
+
return loss_info
|
tests/test_simct.py
ADDED
|
@@ -0,0 +1,197 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Author-source oracle parity. Requires existing torch; never installs packages."""
|
| 2 |
+
import ast
|
| 3 |
+
import copy
|
| 4 |
+
import hashlib
|
| 5 |
+
import json
|
| 6 |
+
import logging
|
| 7 |
+
import pathlib
|
| 8 |
+
import types
|
| 9 |
+
import unittest
|
| 10 |
+
|
| 11 |
+
import test_mp_opd_port as fixtures
|
| 12 |
+
|
| 13 |
+
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
| 14 |
+
if fixtures.HAS_TORCH:
|
| 15 |
+
import torch
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def original_oracle_class():
|
| 19 |
+
"""Execute original numerical bodies, stripping only imports/decorators.
|
| 20 |
+
|
| 21 |
+
The oracle does not import the port, adapter, or CTOPD RKL implementation.
|
| 22 |
+
"""
|
| 23 |
+
source = ROOT / 'tests/fixtures/simct_upstream'
|
| 24 |
+
rkl_tree = ast.parse((source / 'reverse_kl_div.py.txt').read_text())
|
| 25 |
+
rkl = next(n for n in rkl_tree.body if isinstance(n, ast.FunctionDef))
|
| 26 |
+
rkl.decorator_list = []
|
| 27 |
+
scope = {'torch': torch}
|
| 28 |
+
exec(compile(ast.Module(body=[rkl], type_ignores=[]), 'original_rkl', 'exec'), scope)
|
| 29 |
+
cls_tree = ast.parse((source / 'span_ctkd.py.txt').read_text())
|
| 30 |
+
cls = next(n for n in cls_tree.body if isinstance(n, ast.ClassDef))
|
| 31 |
+
cls.decorator_list = []
|
| 32 |
+
scope.update(build_loss_fn=lambda *_: scope['compute_reverse_kl_div'],
|
| 33 |
+
logger=logging.getLogger('simct-oracle'))
|
| 34 |
+
exec(compile(ast.Module(body=[cls], type_ignores=[]), 'original_simct', 'exec'), scope)
|
| 35 |
+
return scope['SpanCrossTokenizerKD']
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
@unittest.skipUnless(fixtures.HAS_TORCH, 'requires existing PyTorch; no bootstrap')
|
| 39 |
+
class SimCTTests(unittest.TestCase):
|
| 40 |
+
parity_errors = []
|
| 41 |
+
|
| 42 |
+
@classmethod
|
| 43 |
+
def tearDownClass(cls):
|
| 44 |
+
if cls.parity_errors:
|
| 45 |
+
print('SIMCT_PARITY ' + json.dumps({
|
| 46 |
+
'cases': len(cls.parity_errors),
|
| 47 |
+
'max_loss_abs_error': max(x[0] for x in cls.parity_errors),
|
| 48 |
+
'max_student_gradient_abs_error': max(x[1] for x in cls.parity_errors)}))
|
| 49 |
+
|
| 50 |
+
def pair(self, *, threshold=0.0, cap=None):
|
| 51 |
+
from kdflow.algorithms.simct import SimCT
|
| 52 |
+
helper = fixtures.MPPortTests()
|
| 53 |
+
mp, student, teacher = helper.fixture(cap=cap)
|
| 54 |
+
args = mp.args
|
| 55 |
+
args.kd.kd_algorithm = 'simct'
|
| 56 |
+
args.kd.simct_gh_mask_threshold = args.kd.span_gh_mask_threshold = threshold
|
| 57 |
+
actual = SimCT(mp.strategy, student, teacher, fixtures.Tokenizer(), fixtures.Tokenizer())
|
| 58 |
+
ref_student, ref_teacher = copy.deepcopy(student), copy.deepcopy(teacher)
|
| 59 |
+
|
| 60 |
+
class OriginalHFOutput:
|
| 61 |
+
def __call__(self, ids, **kwargs):
|
| 62 |
+
hidden = ref_student(ids, **kwargs)['hidden_states'][-1]
|
| 63 |
+
logits = ref_student.model.lm_head(hidden, skip=False)
|
| 64 |
+
if cap is not None:
|
| 65 |
+
logits = torch.tanh(logits / cap) * cap
|
| 66 |
+
return {'logits': logits}
|
| 67 |
+
|
| 68 |
+
reference = original_oracle_class()(mp.strategy, OriginalHFOutput(), ref_teacher,
|
| 69 |
+
fixtures.Tokenizer(), fixtures.Tokenizer())
|
| 70 |
+
reference._debug_count, reference._span_debug_count = 3, 1
|
| 71 |
+
# Set iteration ordering is identical within this process. Assert explicitly.
|
| 72 |
+
self.assertTrue(torch.equal(actual.upstream.student_overlap_token_ids,
|
| 73 |
+
reference.student_overlap_token_ids))
|
| 74 |
+
return helper, actual, reference, student, ref_student, teacher, ref_teacher
|
| 75 |
+
|
| 76 |
+
def assert_pair(self, *, threshold=0.0, cap=None, copies=1, strict=False):
|
| 77 |
+
helper, actual, reference, student, rs, teacher, rt = self.pair(threshold=threshold, cap=cap)
|
| 78 |
+
batch = helper.batch(copies)
|
| 79 |
+
if strict:
|
| 80 |
+
batch['tea_input_ids'] = batch['stu_input_ids'].clone()
|
| 81 |
+
batch['tea_loss_mask'] = batch['stu_loss_mask'].clone()
|
| 82 |
+
batch['tea_attn_mask'] = batch['stu_attn_mask'].clone()
|
| 83 |
+
batch['teacher_hiddens'] = torch.randn(int(batch['tea_loss_mask'].sum()), 4, requires_grad=True)
|
| 84 |
+
a, b = actual.training_step(batch), reference.training_step(batch)
|
| 85 |
+
torch.testing.assert_close(a['train/loss'], b['loss'], atol=1e-6, rtol=1e-5)
|
| 86 |
+
torch.testing.assert_close(a['distill/alignment_ratio'], b['align_ratio'])
|
| 87 |
+
torch.testing.assert_close(a['distill/simct_gh_mask_ratio'], b['gh_mask_ratio'])
|
| 88 |
+
ga = torch.autograd.grad(a['train/loss'], tuple(student.parameters()))
|
| 89 |
+
gb = torch.autograd.grad(b['loss'], tuple(rs.parameters()))
|
| 90 |
+
for x, y in zip(ga, gb):
|
| 91 |
+
torch.testing.assert_close(x, y, atol=1e-6, rtol=1e-5)
|
| 92 |
+
self.parity_errors.append((float((a['train/loss'] - b['loss']).detach().abs()),
|
| 93 |
+
max(float((x - y).abs().max()) for x, y in zip(ga, gb))))
|
| 94 |
+
self.assertTrue(all(p.grad is None for p in teacher.parameters()))
|
| 95 |
+
self.assertTrue(all(p.grad is None for p in rt.parameters()))
|
| 96 |
+
# Assert teacher/hidden stop-gradient with autograd too, not only .grad buffers.
|
| 97 |
+
grads = torch.autograd.grad(actual.training_step(batch)['train/loss'],
|
| 98 |
+
(*teacher.parameters(), batch['teacher_hiddens']), allow_unused=True)
|
| 99 |
+
self.assertTrue(all(g is None for g in grads))
|
| 100 |
+
|
| 101 |
+
def test_mixed_spans_loss_and_all_student_gradients_match_original(self):
|
| 102 |
+
self.assert_pair(copies=2)
|
| 103 |
+
|
| 104 |
+
def test_strict_only_loss_and_gradients_match_original(self):
|
| 105 |
+
self.assert_pair(strict=True)
|
| 106 |
+
|
| 107 |
+
def test_softcap_matches_original_hf_student_output(self):
|
| 108 |
+
self.assert_pair(cap=2.5)
|
| 109 |
+
|
| 110 |
+
def test_optional_teacher_mass_mask_matches_original(self):
|
| 111 |
+
for threshold in (0.5, 2.0, 100.0):
|
| 112 |
+
with self.subTest(threshold=threshold):
|
| 113 |
+
self.assert_pair(threshold=threshold)
|
| 114 |
+
|
| 115 |
+
def test_virtual_logits_and_derivatives_match_independent_hand_calculation(self):
|
| 116 |
+
_, algo, _, _, _, _, _ = self.pair()
|
| 117 |
+
# First segment mismatches: teacher 'ab' vs student 'a','b'; next is EOS.
|
| 118 |
+
segments = [(0, 1, 0, 2), (1, 2, 2, 3)]
|
| 119 |
+
sl = torch.randn(3, 8, requires_grad=True)
|
| 120 |
+
tl = torch.randn(2, 8)
|
| 121 |
+
sv, tv = algo.upstream._build_virtual_vocab_logits(segments, sl, tl, [1, 2, 0], [4, 0])
|
| 122 |
+
ids, tids = algo.upstream.student_overlap_token_ids, algo.upstream.teacher_overlap_token_ids
|
| 123 |
+
manual_s = torch.stack([torch.cat([sl[0, ids], ((sl[0, 1] + sl[1, 2]) / 2)[None]]),
|
| 124 |
+
torch.cat([sl[2, ids], sl.new_tensor([-1e9])])])
|
| 125 |
+
manual_t = torch.stack([torch.cat([tl[0, tids], tl[0, 4][None]]),
|
| 126 |
+
torch.cat([tl[1, tids], tl.new_tensor([-1e9])])])
|
| 127 |
+
torch.testing.assert_close(sv, manual_s)
|
| 128 |
+
torch.testing.assert_close(tv, manual_t)
|
| 129 |
+
p, q = manual_s.log_softmax(-1), manual_t.log_softmax(-1)
|
| 130 |
+
expected = (p.exp() * (p - q)).sum()
|
| 131 |
+
result = algo.upstream.loss_fn(sv, tv, reduction='sum')
|
| 132 |
+
torch.testing.assert_close(result, expected)
|
| 133 |
+
torch.testing.assert_close(torch.autograd.grad(result, sl)[0],
|
| 134 |
+
torch.autograd.grad(expected, sl)[0])
|
| 135 |
+
|
| 136 |
+
def test_response_boundaries_and_microbatch_normalizer(self):
|
| 137 |
+
helper, algo, _, student, _, _, _ = self.pair()
|
| 138 |
+
full = helper.batch(copies=2)
|
| 139 |
+
one = helper.batch()
|
| 140 |
+
one['avg_micro_batch_token_num'] = full['avg_micro_batch_token_num']
|
| 141 |
+
whole = algo.training_step(full)['train/loss']
|
| 142 |
+
split = sum(algo.training_step(one)['train/loss'] for _ in range(2))
|
| 143 |
+
torch.testing.assert_close(whole, split)
|
| 144 |
+
for a, b in zip(torch.autograd.grad(whole, tuple(student.parameters())),
|
| 145 |
+
torch.autograd.grad(split, tuple(student.parameters()))):
|
| 146 |
+
torch.testing.assert_close(a, b)
|
| 147 |
+
|
| 148 |
+
def test_no_alignment_returns_connected_zero_and_bad_shapes_fail(self):
|
| 149 |
+
helper, algo, _, student, _, _, _ = self.pair()
|
| 150 |
+
batch = helper.batch()
|
| 151 |
+
batch['tea_input_ids'][:] = 5
|
| 152 |
+
loss = algo.training_step(batch)['train/loss']
|
| 153 |
+
self.assertEqual(float(loss.detach()), 0.0)
|
| 154 |
+
gradients = torch.autograd.grad(loss, tuple(student.parameters()), allow_unused=True)
|
| 155 |
+
self.assertTrue(all(g is None or not bool(g.any()) for g in gradients))
|
| 156 |
+
batch['teacher_hiddens'] = batch['teacher_hiddens'][:-1]
|
| 157 |
+
with self.assertRaisesRegex(ValueError, 'hidden rows'):
|
| 158 |
+
algo.training_step(batch)
|
| 159 |
+
|
| 160 |
+
def test_mode_registration_arguments_and_no_legacy_alias(self):
|
| 161 |
+
from kdflow.algorithms import ALGO_DICT
|
| 162 |
+
from kdflow.arguments.distillation_args import DistillationArguments
|
| 163 |
+
self.assertIn('simct', ALGO_DICT)
|
| 164 |
+
self.assertNotIn('span_ctkd', ALGO_DICT)
|
| 165 |
+
options = dict(kd_algorithm='simct', kd_ratio=1, kd_loss_fn='rkl')
|
| 166 |
+
self.assertEqual(DistillationArguments(**options).simct_gh_mask_threshold, 0)
|
| 167 |
+
for value in (-1, float('nan'), float('inf')):
|
| 168 |
+
with self.assertRaises(ValueError):
|
| 169 |
+
DistillationArguments(**options, simct_gh_mask_threshold=value)
|
| 170 |
+
with self.assertRaises(ValueError):
|
| 171 |
+
DistillationArguments(kd_algorithm='simct')
|
| 172 |
+
|
| 173 |
+
def test_unsupported_topology_and_non_on_policy_fail(self):
|
| 174 |
+
from kdflow.algorithms.simct import SimCT
|
| 175 |
+
_, algo, _, student, _, teacher, _ = self.pair()
|
| 176 |
+
algo.args.train.num_gpus_per_node = 2
|
| 177 |
+
with self.assertRaisesRegex(ValueError, 'one student rank'):
|
| 178 |
+
SimCT(algo.upstream.strategy, student, teacher, fixtures.Tokenizer(), fixtures.Tokenizer())
|
| 179 |
+
algo.args.train.num_gpus_per_node = 1
|
| 180 |
+
algo.args.scenario = 'off_policy_kd'
|
| 181 |
+
with self.assertRaisesRegex(ValueError, 'on-policy'):
|
| 182 |
+
SimCT(algo.upstream.strategy, student, teacher, fixtures.Tokenizer(), fixtures.Tokenizer())
|
| 183 |
+
|
| 184 |
+
|
| 185 |
+
class SimCTSourceTests(unittest.TestCase):
|
| 186 |
+
def test_pinned_sources_and_frozen_existing_modes(self):
|
| 187 |
+
manifest = json.loads((ROOT / 'docs/SIMCT_SOURCE.json').read_text())
|
| 188 |
+
for record in manifest['files'] + manifest['unchanged_existing_files']:
|
| 189 |
+
self.assertEqual(hashlib.sha256((ROOT / record['path']).read_bytes()).hexdigest(), record['sha256'])
|
| 190 |
+
raw = (ROOT / 'tests/fixtures/simct_upstream/span_ctkd.py.txt').read_bytes()
|
| 191 |
+
port = (ROOT / 'kdflow/algorithms/_simct_upstream/span_ctkd.py').read_bytes()
|
| 192 |
+
self.assertEqual(raw.replace(b'from kdflow.algorithms import register_algorithm',
|
| 193 |
+
b'from . import register_algorithm'), port)
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
if __name__ == '__main__':
|
| 197 |
+
unittest.main()
|
tests/test_single_b200.py
CHANGED
|
@@ -55,6 +55,14 @@ class LauncherTests(unittest.TestCase):
|
|
| 55 |
self.assertEqual(plan['flags']['train_batch_size'], 512)
|
| 56 |
self.assertEqual(plan['flags']['kd_algorithm'], 'byte_align_ctkd')
|
| 57 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
def test_mp_mode_presets_preserve_full_workload(self):
|
| 59 |
baseline = self.plan()['flags']
|
| 60 |
variants = ['mp-atomic', *['mp-fixed' + str(n) for n in range(1, 6)],
|
|
|
|
| 55 |
self.assertEqual(plan['flags']['train_batch_size'], 512)
|
| 56 |
self.assertEqual(plan['flags']['kd_algorithm'], 'byte_align_ctkd')
|
| 57 |
|
| 58 |
+
def test_simct_preset_changes_only_algorithm_and_adds_disabled_guard(self):
|
| 59 |
+
baseline = self.plan('--algorithm', 'strict-full')['flags']
|
| 60 |
+
flags = self.plan('--algorithm', 'simct')['flags']
|
| 61 |
+
self.assertEqual(flags['kd_algorithm'], 'simct')
|
| 62 |
+
self.assertEqual(flags['simct_gh_mask_threshold'], 0)
|
| 63 |
+
self.assertEqual({k: v for k, v in flags.items() if k not in {'kd_algorithm', 'simct_gh_mask_threshold'}},
|
| 64 |
+
{k: v for k, v in baseline.items() if k != 'kd_algorithm'})
|
| 65 |
+
|
| 66 |
def test_mp_mode_presets_preserve_full_workload(self):
|
| 67 |
baseline = self.plan()['flags']
|
| 68 |
variants = ['mp-atomic', *['mp-fixed' + str(n) for n in range(1, 6)],
|
tools/launch_single_b200.py
CHANGED
|
@@ -25,7 +25,7 @@ def parser():
|
|
| 25 |
default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
|
| 26 |
p.add_argument('--runtime', type=pathlib.Path,
|
| 27 |
help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
|
| 28 |
-
p.add_argument('--algorithm', choices=['strict-top16', 'strict-full', 'mp-atomic',
|
| 29 |
'mp-fixed1', 'mp-fixed2', 'mp-fixed3', 'mp-fixed4', 'mp-fixed5',
|
| 30 |
'mp-random1-2', 'mp-random2-5', 'mp-grass-chunk', 'mp-align'], default='strict-top16')
|
| 31 |
p.add_argument('--mp-chunk-source', choices=['xtoken', 'run'], default='xtoken')
|
|
@@ -80,6 +80,8 @@ def build_plan(args):
|
|
| 80 |
'teacher_mem_fraction_static': 0.6, 'teacher_offload_tags': 'all',
|
| 81 |
'chunked_loss_size': 2048, 'use_wandb': False,
|
| 82 |
}
|
|
|
|
|
|
|
| 83 |
if args.algorithm.startswith('mp-'):
|
| 84 |
name = args.algorithm[3:]
|
| 85 |
family = 'fixed' if name.startswith('fixed') else 'random' if name.startswith('random') else name.replace('-', '_')
|
|
@@ -101,7 +103,9 @@ def build_plan(args):
|
|
| 101 |
'source': str(SOURCE), 'flags': flags, 'command': command,
|
| 102 |
'gradient_accumulation_microsteps': 512 // args.micro_batch,
|
| 103 |
'expected_runtime_archive_sha256': ARCHIVE_SHA,
|
| 104 |
-
'qualification': ('
|
|
|
|
|
|
|
| 105 |
if args.algorithm.startswith('mp-') else
|
| 106 |
'unvalidated single-B200 full workload; no metric-match claim'),
|
| 107 |
'evaluation_scope': 'MATH500 accuracy over 32 responses/problem; not the full paper benchmark suite'}
|
|
|
|
| 25 |
default=pathlib.Path('/workspace/storage-shared/nlp/tungks/ctopd_all'))
|
| 26 |
p.add_argument('--runtime', type=pathlib.Path,
|
| 27 |
help='Installed prefix; default CTOPD_RUNTIME_PREFIX or /opt/ctopd-b200')
|
| 28 |
+
p.add_argument('--algorithm', choices=['strict-top16', 'strict-full', 'simct', 'mp-atomic',
|
| 29 |
'mp-fixed1', 'mp-fixed2', 'mp-fixed3', 'mp-fixed4', 'mp-fixed5',
|
| 30 |
'mp-random1-2', 'mp-random2-5', 'mp-grass-chunk', 'mp-align'], default='strict-top16')
|
| 31 |
p.add_argument('--mp-chunk-source', choices=['xtoken', 'run'], default='xtoken')
|
|
|
|
| 80 |
'teacher_mem_fraction_static': 0.6, 'teacher_offload_tags': 'all',
|
| 81 |
'chunked_loss_size': 2048, 'use_wandb': False,
|
| 82 |
}
|
| 83 |
+
if args.algorithm == 'simct':
|
| 84 |
+
flags.update(kd_algorithm='simct', simct_gh_mask_threshold=0.0)
|
| 85 |
if args.algorithm.startswith('mp-'):
|
| 86 |
name = args.algorithm[3:]
|
| 87 |
family = 'fixed' if name.startswith('fixed') else 'random' if name.startswith('random') else name.replace('-', '_')
|
|
|
|
| 103 |
'source': str(SOURCE), 'flags': flags, 'command': command,
|
| 104 |
'gradient_accumulation_microsteps': 512 // args.micro_batch,
|
| 105 |
'expected_runtime_archive_sha256': ARCHIVE_SHA,
|
| 106 |
+
'qualification': ('author-code SimCT port; no B200/full-training qualification'
|
| 107 |
+
if args.algorithm == 'simct' else
|
| 108 |
+
'experimental MP-OPD source port; no B200/full-training qualification'
|
| 109 |
if args.algorithm.startswith('mp-') else
|
| 110 |
'unvalidated single-B200 full workload; no metric-match claim'),
|
| 111 |
'evaluation_scope': 'MATH500 accuracy over 32 responses/problem; not the full paper benchmark suite'}
|