|
Download docs/INVALID_RESPONSES.md from codemaivanngu/ctopd: direct link, hf CLI and curl.
- Browser
- Download file 3.1 kB
-
https://huggingface.co/codemaivanngu/ctopd/resolve/main/docs/INVALID_RESPONSES.md
- Command line
-
hf download hf://codemaivanngu/ctopd/docs/INVALID_RESPONSES.md
-
curl -L -o INVALID_RESPONSES.md https://huggingface.co/codemaivanngu/ctopd/resolve/main/docs/INVALID_RESPONSES.md
3.1 kB
| # Invalid decoded responses in MP-OPD | |
| The 2026-10-19 company logs showed `replacement_character` in Fixed1 and | |
| `normalization_mismatch` in GRASS-Chunk and ALIGN. Their root exceptions occurred | |
| in the shared MP adapter; the displayed SIGTERM messages accompanied Ray cleanup. | |
| Those logs do not contain the decoded strings needed to identify the underlying | |
| generation/retokenization discrepancy. This patch mitigates the crash by excluding | |
| responses that cannot be scored safely on synchronized byte spans. | |
| ## Policy | |
| All MP modes, including ALIGN-NEW and TRUST-R/B/P, exclude the complete response | |
| for `replacement_character`, `normalization_mismatch` or `empty_response`. | |
| Student and teacher hidden-state offsets still advance by their full response | |
| lengths. No text is rewritten and no alternative Atomic loss is substituted. | |
| Other failures, including unsupported added tokens, remain fatal. The vendored | |
| atomizer and all geometry/loss kernels are unchanged. | |
| The existing shared optimizer-window denominator is preserved, including excluded | |
| tokens. This avoids changing the weighting of valid responses according to how | |
| they happen to be grouped into microbatches. Exclusion changes the effective | |
| sample distribution and reduces gradient magnitude relative to processing every | |
| token; it is not mathematical equivalence to training on the original batch. | |
| An entirely excluded microbatch returns connected zero loss. That alone does not | |
| guarantee unchanged optimizer state or weights across an entire optimizer update. | |
| Metrics use the prefix `distill/mp_opd_`: | |
| - `attempted_responses`, `valid_responses`, `skipped_responses`; | |
| - `skipped_replacement_responses`, `skipped_normalization_responses`, | |
| `skipped_empty_responses`; | |
| - `skipped_replacement_fraction`, `skipped_normalization_fraction`; | |
| - `skipped_student_loss_tokens`, `skipped_teacher_loss_tokens`, | |
| `skipped_student_token_fraction`. | |
| These are microbatch counts/fractions. Existing training logging aggregates | |
| microbatches; a displayed count may therefore be an average rather than a run | |
| total. Record exclusions when interpreting comparisons, especially if their | |
| frequency differs by method. The policy does not affect `simct`, `simct_paper`, | |
| or Strict. | |
| ## Verification and provenance | |
| `tests/test_invalid_response.py` checks whole-response exclusion, hidden-state | |
| offsets, the original denominator, valid-response loss/all student gradients, | |
| all-excluded backward, transient UTF-8 prefix recovery, failure guards, and | |
| TRUST-B/P calibration/replay. Historical source manifests are retained; | |
| `MP_OPD_INVALID_RESPONSES.json` explicitly records the patched adapter hash. | |
| CPU checks do not qualify B200/FSDP execution or full training. The company logs | |
| were inputs to diagnosis, not successful execution evidence for this patch. | |
| Only source changes are needed; the runtime archive does not need rebuilding. | |
| Use a fresh source checkout and a new run ID for failed runs. Existing Ray actors | |
| carry a packaged source snapshot, so updating files cannot repair an already | |
| running actor. Preserve active jobs' checkouts and prior run outputs. | |