Publish experimental commitment-classifier pilot

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,17 @@
+MIT License
+
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+associated documentation files (the "Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the
+following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial
+portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
+LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO
+EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -0,0 +1,152 @@
+---
+license: mit
+language:
+- en
+library_name: transformers
+pipeline_tag: text-classification
+base_model: vllm-sr/Vela-1.0-Encoder-307M
+base_model_relation: finetune
+datasets:
+- TwilightTechie/commitment-statements-pilot
+tags:
+- commitment-detection
+- decision-detection
+- experimental
+- vela
+widget:
+- text: We have decided to update the document today.
+- text: I like mountains.
+---
+
+# Vela commitment classifier — experimental pilot
+
+A short English statement classifier that detects an **expressed decision, firm
+intention, or commitment**. This is a personal fine-tuning experiment derived from
+Vela, not an official vLLM Semantic Router release or an endorsed product.
+
+| Label | Meaning |
+| --- | --- |
+| `DECISION_EXPRESSED` | An affirmed chosen action, firm intention, commitment, explicit rejection or chosen inaction. Explicitly reported past choices also count. |
+| `NO_DECISION_EXPRESSED` | Preferences, questions, requests, possibilities, predictions, example quotations, or completed-action reports without an expressed choice. |
+
+It does not determine whether a promise was fulfilled, infer an unspoken mental
+state, extract tasks, or decide whether a statement deserves a decision-log entry.
+Routine firm process commitments count as positive under this broad contract.
+
+## Quick start
+
+```bash
+pip install torch transformers==4.57.6
+```
+
+```python
+from transformers import pipeline
+
+classifier = pipeline(
+    "text-classification",
+    model="TwilightTechie/vela-commitment-classifier-307m",
+    tokenizer="TwilightTechie/vela-commitment-classifier-307m",
+    device=-1,  # CPU
+)
+print(classifier("We have decided to update the document today."))
+print(classifier("I like mountains."))
+```
+
+The first example was classified as `DECISION_EXPRESSED`; the second as
+`NO_DECISION_EXPRESSED`. A GPU or hosted endpoint is not required for inference.
+Scores are model softmax outputs, **not calibrated certainty**. Run only trusted
+standard library code; this model requires no `trust_remote_code=True`.
+
+Keep inputs to **256 tokens including special tokens**. You can enforce this
+before classification:
+
+```python
+text = "We agreed to postpone the rollout until the pilot finishes."
+if len(classifier.tokenizer(text)["input_ids"]) > 256:
+    raise ValueError("This pilot's input budget is 256 tokens")
+print(classifier(text))
+```
+
+The foundation has a larger architectural context capacity; this fine-tuned
+pilot has not established quality on long documents or conversations.
+
+## Training and provenance
+
+- Foundation: [`vllm-sr/Vela-1.0-Encoder-307M`](https://huggingface.co/vllm-sr/Vela-1.0-Encoder-307M).
+- Exact foundation revision: `720ab37904ce15054068d429381bfae7549f00f6`.
+- Architecture: ModernBERT sequence classifier with a fresh two-label task head.
+- Method: complete checkpoint, updating the final two encoder layers and task
+  head; other encoder parameters stayed frozen and passed preservation checks.
+- Trainable parameters: 10,622,210.
+- Training data: 118 short, synthetic English examples. Development data: 14.
+  Assistant-authored examples were accepted by the user for this pilot; this
+  does not constitute independent expert annotation. Original row review and
+  provenance fields remain unchanged in the dataset.
+- 100 optimizer steps, seed 42, batch size 2, gradient accumulation 2.
+- Encoder learning rate 0.00002; head learning rate 0.0001; maximum length 256.
+- BF16 forward autocast, FP32 parameters; FP32 development evaluation.
+- Hardware: one NVIDIA L4 on Modal; peak allocated training memory 1.3697 GiB.
+- Selection: development macro F1 at steps 0, 25, 50, 75, 100. Step 75 selected.
+- Remote environment: Python 3.12, Torch 2.8.0+cu128, Transformers 4.57.6,
+  PEFT 0.18.1, Accelerate 1.10.1, scikit-learn 1.7.2.
+
+The training recipe and environment are included under `reproducibility/`.
+The training code comes from the public
+[vLLM Semantic Router repository](https://github.com/vllm-project/semantic-router).
+`source-files.json` captures hashes of the exact trainer modules used; the
+experiment was made in a local checkout, not claimed as an upstream release.
+No private source document or credentials are included.
+
+## Development results — not independent testing
+
+All rows below use the same **14-example development set**:
+
+| Classifier | Correct | Macro F1 |
+| --- | --- | --- |
+| Fixed keyword rule | 12/14 | 0.8542 |
+| TF-IDF + logistic regression | 13/14 | 0.9282 |
+| Decision 2.0 Kai, fixed yes/no question | 11/14 | 0.7754 |
+| This selected Vela checkpoint | 14/14 | 1.0000 |
+
+The neural checkpoint was selected using this development set. Additional
+training examples were authored after inspecting earlier development errors.
+Therefore, **14/14 does not imply general accuracy or enterprise readiness**.
+FP32 GPU and local CPU reload produced the same class predictions; maximum
+probability difference was about 0.00000185. This is artifact verification,
+not independent quality evaluation.
+
+A 16-example same-author synthetic test set remains unpublished and unevaluated.
+A fresh, realistic, independently annotated evaluation set is still needed.
+
+## Intended use and limitations
+
+Use this for learning, local experimentation, and a candidate classifier in
+human-reviewed workflows. It has not been qualified for automatic authoritative
+logging, production routing, or consequential decisions.
+
+- English-only evaluated scope despite the multilingual foundation.
+- Very small authored dataset; wording and topic bias are likely.
+- Standalone short statements only; multi-turn reference resolution is not tested.
+- Conditional, canceled, sarcastic and context-dependent statements need a
+  separately reviewed contract; predictions may be unreliable there.
+- A past decision may be superseded; this classifier does not identify current
+  policy, author authority, or supersession links.
+- High scores can be overconfident. Calibration and realistic false-positive
+  rates have not been measured.
+- Transformers 4.57.6 emitted a known class of Mistral regex warnings when loading
+  this non-Mistral tokenizer locally. No tokenizer patch was applied. Basic
+  reload checks passed; a broader tokenizer audit has not been performed.
+
+## License and attribution
+
+The pinned Vela model card declares MIT; it identifies
+[`jhu-clsp/mmBERT-base`](https://huggingface.co/jhu-clsp/mmBERT-base) as its upstream
+foundation, whose model card also declares MIT. The fine-tuned model is shared
+under MIT with those upstream declarations retained. See `LICENSE` and
+`UPSTREAM_LICENSES.md` for the recorded absence of separate upstream notice files.
+No upstream copyright owner or year has been invented.
+
+The reproducibility code and synthetic dataset are distributed separately under
+Apache-2.0, following the Semantic Router repository's code/data context. See
+`reproducibility/LICENSE` and the dataset card. This model license does not
+relicense upstream training corpora or unrelated dependencies.

UPSTREAM_LICENSES.md

@@ -0,0 +1,27 @@
+# Upstream license declarations and provenance
+
+This distribution contains a fine-tuned derivative of
+[`vllm-sr/Vela-1.0-Encoder-307M`](https://huggingface.co/vllm-sr/Vela-1.0-Encoder-307M),
+using revision `720ab37904ce15054068d429381bfae7549f00f6`.
+That revision's model card declares `license: mit`. Its upstream model is
+[`jhu-clsp/mmBERT-base`](https://huggingface.co/jhu-clsp/mmBERT-base), whose queried
+model card also declares MIT.
+
+The queried pinned Vela files and upstream mmBERT files provide no separate
+LICENSE or NOTICE file. This omission is recorded rather than manufacturing
+an upstream copyright notice or assigning ownership of their weights to the
+fine-tuning author. The model license file contains the standard MIT permission
+and warranty terms, sourced from the
+[SPDX MIT license text](https://github.com/spdx/license-list-data/blob/main/text/MIT.txt).
+Any authoritative upstream copyright notices subsequently supplied should be
+preserved with this distribution. These statements reflect published license
+declarations rather than a comprehensive audit of upstream training corpora.
+
+The new classifier head and final two encoder layers were updated in a personal
+training experiment. This derivative is not an official upstream release and
+does not imply endorsement by vLLM Semantic Router or JHU.
+
+The accompanying reproduction source and synthetic dataset retain Apache-2.0
+licensing from the Semantic Router experiment's source/data context. Their
+license copies are separate from the MIT model license. No private reference
+document, original test examples or credentials are distributed.

config.json

@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "ModernBertForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "id2label": {
+    "0": "NO_DECISION_EXPRESSED",
+    "1": "DECISION_EXPRESSED"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "label2id": {
+    "DECISION_EXPRESSED": 1,
+    "NO_DECISION_EXPRESSED": 0
+  },
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 160000,
+  "mask_token_id": 4,
+  "max_position_embeddings": 32768,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "problem_type": "single_label_classification",
+  "repad_logits_with_grad": false,
+  "rope_scaling": {
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 4.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "yarn",
+    "truncate": true
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "transformers_version": "4.57.6",
+  "vocab_size": 256000
+}

label-contract.json

@@ -0,0 +1,11 @@
+{
+  "problem_type": "single_label_classification",
+  "label2id": {
+    "NO_DECISION_EXPRESSED": 0,
+    "DECISION_EXPRESSED": 1
+  },
+  "id2label": {
+    "0": "NO_DECISION_EXPRESSED",
+    "1": "DECISION_EXPRESSED"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcadfb68f0ea218ca6b015a9517af234049e3953cb4376d2c2ffc3803917dcb4
+size 1230141424

reproducibility/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

reproducibility/README.md

@@ -0,0 +1,37 @@
+# Reproduce the pilot
+
+Use a Linux GPU environment matching `environment.txt`. This is the exact
+training source; dependency versions and data hashes are in `training.json`.
+Hardware and numerical differences can change the selected checkpoint.
+
+Download the foundation at its pinned revision and this release's synthetic
+train/validation data. Run from this directory; choose a new output directory.
+
+```python
+from huggingface_hub import snapshot_download
+
+snapshot_download(
+    "vllm-sr/Vela-1.0-Encoder-307M",
+    revision="720ab37904ce15054068d429381bfae7549f00f6",
+    local_dir="base",
+)
+snapshot_download(
+    "TwilightTechie/commitment-statements-pilot", repo_type="dataset", local_dir="dataset"
+)
+```
+
+```bash
+python -m sequence_repair.train \
+  --method full --fresh-head --trainable-last-layers 2 \
+  --base base --base-id vllm-sr/Vela-1.0-Encoder-307M \
+  --base-revision 720ab37904ce15054068d429381bfae7549f00f6 \
+  --contract ../label-contract.json \
+  --train dataset/data/train.jsonl --dev dataset/data/validation.jsonl \
+  --output rerun --steps 100 --batch-size 2 --accumulate 2 \
+  --max-length 256 --learning-rate 0.00002 --head-learning-rate 0.0001 \
+  --eval-every 25 --evaluation-dtype float32 --selection macro-f1 --seed 42
+```
+
+The original run selected step 75 using development macro F1. No reserved test
+examples are distributed or used by this command. Before running, verify the
+training and validation file hashes against `training.json`.

reproducibility/environment.txt

@@ -0,0 +1,64 @@
+accelerate==1.10.1
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.7
+aiosignal==1.3.2
+attrs==25.3.0
+cbor2==5.7.0
+certifi==2026.7.22
+charset-normalizer==3.5.2
+cloudpickle==3.1.2
+filelock==4.0.12
+frozenlist==1.6.0
+fsspec==2026.9.0
+grpclib==0.4.8
+h2==4.2.0
+hf-xet==1.7.0
+hpack==4.1.0
+huggingface_hub==0.36.2
+hyperframe==6.1.0
+idna==3.20
+Jinja2==3.1.6
+joblib==1.6.0
+MarkupSafe==3.0.4
+mpmath==1.3.0
+multidict==6.4.4
+networkx==3.7
+numpy==2.5.3
+nvidia-cublas-cu12==12.8.4.1
+nvidia-cuda-cupti-cu12==12.8.90
+nvidia-cuda-nvrtc-cu12==12.8.93
+nvidia-cuda-runtime-cu12==12.8.90
+nvidia-cudnn-cu12==9.10.2.21
+nvidia-cufft-cu12==11.3.3.83
+nvidia-cufile-cu12==1.13.1.3
+nvidia-curand-cu12==10.3.9.90
+nvidia-cusolver-cu12==11.7.3.90
+nvidia-cusparse-cu12==12.5.8.93
+nvidia-cusparselt-cu12==0.7.1
+nvidia-nccl-cu12==2.27.3
+nvidia-nvjitlink-cu12==12.8.93
+nvidia-nvtx-cu12==12.8.90
+packaging==26.3
+peft==0.18.1
+propcache==0.3.1
+protobuf==6.31.1
+psutil==7.2.2
+PyYAML==6.0.3
+regex==2026.9.29
+requests==2.34.2
+safetensors==0.8.0
+scikit-learn==1.7.2
+scipy==1.18.1
+setuptools==84.0.0
+sympy==1.14.0
+threadpoolctl==3.7.0
+tokenizers==0.22.2
+torch==2.8.0
+tqdm==4.70.1
+transformers==4.57.6
+triton==3.4.0
+typing_extensions==4.16.0
+urllib3==2.8.0
+uv==0.7.19
+wheel==0.45.1
+yarl==1.20.0

reproducibility/sequence_repair/__init__.py

@@ -0,0 +1 @@
+"""Reproducible continuation of fixed-label sequence classifiers."""

reproducibility/sequence_repair/complete_adapter.py

@@ -0,0 +1,94 @@
+"""Make a legacy sequence adapter self-contained with its original task head."""
+
+# ruff: noqa: PLC0415
+import argparse
+import hashlib
+import json
+import shutil
+from pathlib import Path
+
+from .model import load_model
+from .task_head import verify_saved_task_head
+
+
+def main():
+    import torch
+    from safetensors.torch import load_file, save_file
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--adapter", type=Path, required=True)
+    parser.add_argument("--contract", required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    if args.output.exists():
+        raise ValueError("Refusing to overwrite a completed adapter")
+    config = json.loads((args.adapter / "adapter_config.json").read_text())
+    if "head" in (config.get("modules_to_save") or []):
+        raise ValueError("Adapter already owns its task head")
+    torch.set_num_threads(8)
+    model, tokenizer, _labels, _ids = load_model(args.base, args.contract, args.adapter)
+    model.eval()
+    tokens = tokenizer(
+        ["A task head parity probe.", "核对分类头参数是否保持不变。"],
+        padding=True,
+        return_tensors="pt",
+    )
+    with torch.inference_mode():
+        before = model(**tokens).logits
+    head = model.get_base_model().head.state_dict()
+    weights = load_file(args.adapter / "adapter_model.safetensors", device="cpu")
+    for name, tensor in head.items():
+        key = "base_model.model.head." + name
+        if key in weights:
+            raise ValueError("Unexpected existing head tensor in legacy adapter")
+        weights[key] = tensor.detach().cpu().contiguous()
+    args.output.mkdir(parents=True)
+    for path in args.adapter.iterdir():
+        if path.is_file() and path.name not in {
+            "adapter_config.json",
+            "adapter_model.safetensors",
+            "task-head.json",
+        }:
+            shutil.copy2(path, args.output / path.name)
+    config["modules_to_save"] = [*(config.get("modules_to_save") or []), "head"]
+    (args.output / "adapter_config.json").write_text(
+        json.dumps(config, indent=2) + "\n"
+    )
+    save_file(
+        weights, args.output / "adapter_model.safetensors", metadata={"format": "pt"}
+    )
+    restored, _tokenizer, _labels, _ids = load_model(
+        args.base, args.contract, args.output
+    )
+    restored.eval()
+    with torch.inference_mode():
+        after = restored(**tokens).logits
+    if not bool(torch.isfinite(before).all() and torch.isfinite(after).all()):
+        raise ValueError("Task-head completion produced non-finite logits")
+    torch.testing.assert_close(before, after, rtol=1e-6, atol=1e-6)
+    scope = verify_saved_task_head(restored, args.output / "adapter_model.safetensors")
+    (args.output / "task-head.json").write_text(json.dumps(scope, indent=2) + "\n")
+    receipt = {
+        "operation": "Preserve the existing sequence task head in a legacy adapter; no training or encoder change",
+        "source_adapter_sha256": hashlib.sha256(
+            (args.adapter / "adapter_model.safetensors").read_bytes()
+        ).hexdigest(),
+        "completed_adapter_sha256": hashlib.sha256(
+            (args.output / "adapter_model.safetensors").read_bytes()
+        ).hexdigest(),
+        "base_model": config["base_model_name_or_path"],
+        "base_revision": config.get("revision"),
+        "head_tensors": list(head),
+        "task_head": scope,
+        "maximum_absolute_logit_difference": float((before - after).abs().max()),
+        "quality_evaluation": False,
+    }
+    (args.output / "task-head-completion.json").write_text(
+        json.dumps(receipt, indent=2) + "\n"
+    )
+    print(json.dumps(receipt))
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/context.py

@@ -0,0 +1,85 @@
+"""Construct measured context probes without truncating the task-bearing payload."""
+
+
+def insert_payload(
+    payload,
+    background,
+    position,
+    budget,
+    count_tokens,
+    *,
+    fill_to_budget=True,
+    neutral_suffix=" note"
+):
+    if position not in {"head", "middle", "tail"}:
+        raise ValueError("Unknown payload position")
+    if not payload or not background.strip() or budget <= 0:
+        raise ValueError("Payload, background and token budget must be nonempty")
+    if count_tokens(payload) > budget:
+        raise ValueError("Payload itself exceeds the token budget")
+    if count_tokens(background) <= count_tokens(""):
+        raise ValueError("Background must contribute actual tokens")
+
+    def build(repetitions):
+        left = {"head": 0, "middle": repetitions // 2, "tail": repetitions}[position]
+        prefix = "\n".join([background] * left)
+        suffix = "\n".join([background] * (repetitions - left))
+        text = (
+            (prefix + "\n" if prefix else "")
+            + payload
+            + ("\n" + suffix if suffix else "")
+        )
+        start = len(prefix) + 1 if prefix else 0
+        return text, start
+
+    measured = {}
+
+    def measure(repetitions):
+        if repetitions not in measured:
+            measured[repetitions] = count_tokens(build(repetitions)[0])
+        return measured[repetitions]
+
+    # Repeated complete paragraphs usually have a constant token increment.
+    # Use that only to find a close starting point; exact measurements still
+    # bracket and determine the result, including boundary-token differences.
+    increment = max(1, measure(2) - measure(1))
+    estimate = min(budget, max(0, 1 + (budget - measure(1)) // increment))
+    if measure(estimate) <= budget:
+        low, high, distance = estimate, estimate, 1
+        while high < budget:
+            high = min(budget, estimate + distance)
+            if measure(high) > budget:
+                break
+            low, distance = high, distance * 2
+    else:
+        low, high, distance = estimate, estimate, 1
+        while low > 0:
+            low = max(0, estimate - distance)
+            if measure(low) <= budget:
+                break
+            high, distance = low, distance * 2
+    while low < high:
+        middle = (low + high + 1) // 2
+        if measure(middle) <= budget:
+            low = middle
+        else:
+            high = middle - 1
+    text, start = build(low)
+    actual = measure(low)
+    if fill_to_budget:
+        text += neutral_suffix * (budget - actual)
+        actual = count_tokens(text)
+        if actual != budget:
+            raise ValueError(
+                "Neutral suffix does not fill this tokenizer's exact budget"
+            )
+    if actual > budget or text[start : start + len(payload)] != payload:
+        raise ValueError(
+            "Context construction corrupted its payload or exceeded budget"
+        )
+    return {
+        "text": text,
+        "payload_start": start,
+        "payload_end": start + len(payload),
+        "actual_tokens": actual,
+    }

reproducibility/sequence_repair/corpus.py

@@ -0,0 +1,55 @@
+"""Frozen task corpus receipts and source-group partitioning."""
+
+import hashlib
+import json
+from pathlib import Path
+
+from .data import assert_disjoint, normalized_text
+
+
+def group_split(group, seed=20260913):
+    bucket = int(hashlib.sha256(f"{seed}:{group}".encode()).hexdigest()[:8], 16) % 100
+    return "train" if bucket < 70 else "dev" if bucket < 85 else "test"  # noqa: PLR2004
+
+
+def write_corpus(output, splits, contract, provenance):
+    output = Path(output)
+    if output.exists() and any(output.iterdir()):
+        raise ValueError("Refusing to overwrite a frozen corpus")
+    unique = {}
+    for split, rows in splits.items():
+        seen, retained, retained_keys = {}, [], set()
+        for row in rows:
+            key = normalized_text(row["text"])
+            previous = seen.setdefault(key, row["label"])
+            if previous != row["label"]:
+                raise ValueError("Contradictory labels for identical normalized text")
+            if key in unique and unique[key] != split:
+                raise ValueError("Duplicate text crosses corpus splits")
+            unique[key] = split
+            if key not in retained_keys:
+                retained.append(row)
+                retained_keys.add(key)
+        splits[split] = retained
+    for first, second in [("train", "dev"), ("train", "test"), ("dev", "test")]:
+        assert_disjoint(splits[first], splits[second])
+    output.mkdir(parents=True, exist_ok=True)
+    files = {}
+    for split, rows in splits.items():
+        content = "".join(
+            json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in rows
+        )
+        (output / f"{split}.jsonl").write_text(content)
+        files[split] = {
+            "rows": len(rows),
+            "sha256": hashlib.sha256(content.encode()).hexdigest(),
+        }
+    (output / "contract.json").write_text(json.dumps(contract, indent=2) + "\n")
+    manifest = {
+        "files": files,
+        "provenance": provenance,
+        "test_used_for_selection": False,
+        "split_policy": "Group-disjoint before sampling; exact NFKC/case/whitespace overlap rejected",
+    }
+    (output / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n")
+    return manifest

reproducibility/sequence_repair/data.py

@@ -0,0 +1,107 @@
+"""Dependency-light sequence data contracts and classification metrics."""
+
+import hashlib
+import json
+import unicodedata
+from collections import Counter
+from pathlib import Path
+
+
+def normalized_text(text):
+    return " ".join(unicodedata.normalize("NFKC", text).casefold().split())
+
+
+def load_contract(path):
+    config = json.loads(Path(path).read_text())
+    id_to_label = {int(index): label for index, label in config["id2label"].items()}
+    if set(id_to_label) != set(range(len(id_to_label))):
+        raise ValueError("Label IDs must be contiguous from zero")
+    if config["label2id"] != {label: index for index, label in id_to_label.items()}:
+        raise ValueError("id2label and label2id must be a bijection")
+    return config["label2id"], id_to_label
+
+
+def read_records(paths, label_to_id):
+    records, ids, texts = [], set(), {}
+    for path in paths:
+        for line in Path(path).read_text().split("\n"):
+            if not line.strip():
+                continue
+            row = json.loads(line)
+            if not isinstance(row.get("text"), str) or not row["text"].strip():
+                raise ValueError("A classification row needs nonempty text")
+            if not row.get("id") or row["id"] in ids:
+                raise ValueError("Missing or duplicate row ID")
+            if not row.get("group_id"):
+                raise ValueError("Each row needs a source/translation/template group")
+            if row.get("label") not in label_to_id:
+                raise ValueError("Unknown classification label")
+            normalized = normalized_text(row["text"])
+            previous = texts.setdefault(normalized, row["label"])
+            if previous != row["label"]:
+                raise ValueError("Identical text has contradictory labels")
+            ids.add(row["id"])
+            records.append(row)
+    if not records:
+        raise ValueError(
+            "Empty datasets cannot produce training or evaluation evidence"
+        )
+    return records
+
+
+def assert_disjoint(train, dev):
+    for key, transform in [("group_id", str), ("text", normalized_text), ("id", str)]:
+        overlap = {transform(row[key]) for row in train} & {
+            transform(row[key]) for row in dev
+        }
+        if overlap:
+            raise ValueError(f"Training/development overlap in {key}: {len(overlap)}")
+
+
+def file_receipts(paths):
+    return [
+        {
+            "file": Path(path).name,
+            "sha256": hashlib.sha256(Path(path).read_bytes()).hexdigest(),
+        }
+        for path in paths
+    ]
+
+
+def classification_metrics(gold, predicted, id_to_label):
+    if not gold or len(gold) != len(predicted):
+        raise ValueError("Empty or mismatched prediction evidence")
+    count = len(id_to_label)
+    confusion = [[0] * count for _ in range(count)]
+    for index, expected in enumerate(gold):
+        actual = predicted[index]
+        if expected not in id_to_label or actual not in id_to_label:
+            raise ValueError("Prediction outside the frozen label contract")
+        confusion[expected][actual] += 1
+    per_label = {}
+    for index, label in id_to_label.items():
+        tp = confusion[index][index]
+        support = sum(confusion[index])
+        selected = sum(row[index] for row in confusion)
+        per_label[label] = {
+            "support": support,
+            "tp": tp,
+            "fp": selected - tp,
+            "fn": support - tp,
+            "precision": tp / selected if selected else 0.0,
+            "recall": tp / support if support else 0.0,
+            "f1": 2 * tp / (selected + support) if selected + support else 0.0,
+        }
+    present = [row for row in per_label.values() if row["support"]]
+    return {
+        "rows": len(gold),
+        "accuracy": sum(confusion[index][index] for index in range(count)) / len(gold),
+        "macro_f1": sum(row["f1"] for row in per_label.values()) / count,
+        "macro_f1_present_labels": sum(row["f1"] for row in present) / len(present),
+        "per_label": per_label,
+        "confusion": confusion,
+    }
+
+
+def label_counts(records):
+    return dict(Counter(row["label"] for row in records))

reproducibility/sequence_repair/evaluate.py

@@ -0,0 +1,147 @@
+"""Explicit streaming evaluation; never invoked on test data during training."""
+
+# ruff: noqa: PLC0415
+
+import argparse
+import json
+import time
+from pathlib import Path
+
+from .data import classification_metrics, file_receipts, read_records
+from .model import load_model
+
+LONG_BATCH_THRESHOLD = 2048
+
+
+def evaluate_records(
+    model,
+    tokenizer,
+    records,
+    label_to_id,
+    id_to_label,
+    max_length,
+    batch_size=16,
+    dtype="bfloat16",
+):
+    import torch
+
+    device = next(model.parameters()).device
+    if model.config.problem_type != "single_label_classification":
+        raise ValueError("Softmax evaluation requires a single-label classifier")
+    was_training = model.training
+    model.eval()
+    encoded = [
+        tokenizer(row["text"], truncation=False, padding=False) for row in records
+    ]
+    lengths = [len(row["input_ids"]) for row in encoded]
+    if max(lengths) > max_length:
+        raise ValueError(
+            "Evaluation input exceeds the explicit budget; truncation is forbidden"
+        )
+    order = sorted(range(len(records)), key=lengths.__getitem__)
+    results, timings, cursor = [None] * len(records), [], 0
+    while cursor < len(order):
+        size = 1 if lengths[order[cursor]] > LONG_BATCH_THRESHOLD else batch_size
+        indices = order[cursor : cursor + size]
+        if any(lengths[index] > LONG_BATCH_THRESHOLD for index in indices):
+            indices = indices[:1]
+        cursor += len(indices)
+        batch = tokenizer.pad(
+            [encoded[index] for index in indices], padding=True, return_tensors="pt"
+        )
+        batch = {key: value.to(device) for key, value in batch.items()}
+        if device.type == "cuda":
+            torch.cuda.synchronize()
+        start = time.perf_counter()
+        with torch.inference_mode(), torch.autocast(
+            device_type=device.type,
+            dtype=getattr(torch, dtype),
+            enabled=device.type == "cuda" and dtype != "float32",
+        ):
+            logits = model(**batch).logits.float()
+        if not bool(torch.isfinite(logits).all()):
+            raise ValueError("Non-finite evaluation logits")
+        probabilities = logits.softmax(-1).cpu().tolist()
+        if device.type == "cuda":
+            torch.cuda.synchronize()
+        timings.append((time.perf_counter() - start) * 1000)
+        for position, index in enumerate(indices):
+            scores = probabilities[position]
+            results[index] = {
+                "id": records[index]["id"],
+                "prediction": max(range(len(scores)), key=scores.__getitem__),
+                "probabilities": scores,
+                "tokens": lengths[index],
+            }
+    gold = [label_to_id[row["label"]] for row in records]
+    predicted = [row["prediction"] for row in results]
+    metrics = classification_metrics(gold, predicted, id_to_label)
+    metrics["breakdowns"] = {}
+    for field in ["source", "language", "length_bucket", "position"]:
+        metrics["breakdowns"][field] = {}
+        for group in sorted({str(row.get(field, "unspecified")) for row in records}):
+            indices = [
+                i
+                for i, row in enumerate(records)
+                if str(row.get(field, "unspecified")) == group
+            ]
+            metrics["breakdowns"][field][group] = classification_metrics(
+                [gold[i] for i in indices], [predicted[i] for i in indices], id_to_label
+            )
+    metrics["actual_tokens"] = {"min": min(lengths), "max": max(lengths)}
+    metrics["forward_batches_ms"] = timings
+    metrics["precision"] = {
+        "parameters": "float32",
+        "forward_autocast": (
+            dtype if device.type == "cuda" and dtype != "float32" else None
+        ),
+        "device_type": device.type,
+    }
+    if was_training:
+        model.train()
+    return metrics, results
+
+
+def main():
+    import torch
+
+    torch.set_num_threads(8)
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--adapter")
+    parser.add_argument("--contract", required=True)
+    parser.add_argument("--data", nargs="+", required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--max-length", type=int, default=32768)
+    parser.add_argument("--device", default="cuda")
+    parser.add_argument("--dtype", choices=["float32", "bfloat16"], default="bfloat16")
+    args = parser.parse_args()
+    if args.output.exists():
+        raise ValueError("Refusing to overwrite evaluation evidence")
+    model, tokenizer, labels, ids = load_model(args.base, args.contract, args.adapter)
+    if args.max_length > model.config.max_position_embeddings:
+        raise ValueError("Evaluation budget exceeds checkpoint position capacity")
+    model.to(args.device)
+    rows = read_records(args.data, labels)
+    metrics, predictions = evaluate_records(
+        model, tokenizer, rows, labels, ids, args.max_length, dtype=args.dtype
+    )
+    metrics["data_files"] = file_receipts(args.data)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(metrics, indent=2) + "\n")
+    args.output.with_suffix(".predictions.jsonl").write_text(
+        "".join(json.dumps(row) + "\n" for row in predictions)
+    )
+    print(
+        json.dumps(
+            {
+                key: metrics[key]
+                for key in ["rows", "accuracy", "macro_f1", "actual_tokens"]
+            }
+        ),
+        flush=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/export.py

@@ -0,0 +1,172 @@
+"""Freeze a selected adapter or full model as a standard HF sequence checkpoint."""
+
+# ruff: noqa: PLC0415
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+from .data import load_contract
+from .model import configuration_overrides, load_model, load_task_config
+from .optimization import validate_method, verify_full_checkpoint
+from .runtime_mapping import RUNTIME_TASKS, write_runtime_mappings
+from .task_head import (
+    assert_task_head_preserved,
+    task_head_scope,
+    verify_saved_task_head,
+)
+
+
+def validate_provenance(args):
+    validate_method(args.method, args.adapter, False)
+    run = json.loads(args.run_manifest.read_text())
+    if run.get("test_used") is not False:
+        raise ValueError("Candidate provenance must explicitly exclude test selection")
+    if run.get("base_revision") != args.base_revision:
+        raise ValueError("Export base revision differs from the training receipt")
+    if run.get("base_model", args.base_id) != args.base_id:
+        raise ValueError("Export base model differs from the training receipt")
+    if run.get("method", "lora") != args.method:
+        raise ValueError("Export method differs from the training receipt")
+    if args.method == "full":
+        origin = json.loads((Path(args.base) / "training-origin.json").read_text())
+        expected = {
+            "method": "full",
+            "base_model": args.base_id,
+            "base_revision": args.base_revision,
+        }
+        if origin != expected:
+            raise ValueError("Full checkpoint origin differs from the training receipt")
+        config = json.loads((Path(args.base) / "config.json").read_text())
+        labels, ids = load_contract(args.contract)
+        contract = configuration_overrides(args.contract)
+        contract.update(label2id=labels, id2label={str(k): v for k, v in ids.items()})
+        if any(config.get(key) != value for key, value in contract.items()):
+            raise ValueError("Export contract differs from the trained full checkpoint")
+
+
+def main():
+    import torch
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--base-id", required=True)
+    parser.add_argument("--base-revision", required=True)
+    parser.add_argument("--method", choices=["lora", "full"], default="lora")
+    parser.add_argument("--adapter", type=Path)
+    parser.add_argument("--contract", required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--run-manifest", type=Path, required=True)
+    parser.add_argument("--runtime-task", choices=RUNTIME_TASKS, required=True)
+    args = parser.parse_args()
+    if args.output.exists() and any(args.output.iterdir()):
+        raise ValueError("Refusing to overwrite a frozen candidate")
+    validate_provenance(args)
+    torch.set_num_threads(8)
+    model, tokenizer, labels, _ = load_model(args.base, args.contract, args.adapter)
+    model.eval()
+    if args.method == "full":
+        verify_full_checkpoint(model, args.base)
+        head = task_head_scope(model)
+    else:
+        head = verify_saved_task_head(model, args.adapter / "adapter_model.safetensors")
+    tokens = tokenizer(
+        ["Numerical checkpoint equivalence probe.", "用于核对模型数值的输入。"],
+        padding=True,
+        return_tensors="pt",
+    )
+    with torch.inference_mode():
+        before = model(**tokens).logits
+        exported = (
+            model.merge_and_unload(safe_merge=True) if args.method == "lora" else model
+        )
+        after = exported(**tokens).logits
+    assert_task_head_preserved(head, exported)
+    torch.testing.assert_close(before, after, atol=2e-4, rtol=2e-4)
+    if type(exported).__name__ != "ModernBertForSequenceClassification":
+        raise ValueError(
+            "Export must preserve the standard supported HF sequence architecture"
+        )
+    if exported.config.label2id != labels:
+        raise ValueError("Export changed label IDs")
+    exported.config._name_or_path = args.base_id
+    args.output.mkdir(parents=True, exist_ok=True)
+    exported.save_pretrained(args.output, safe_serialization=True)
+    verify_full_checkpoint(exported, args.output)
+    tokenizer.save_pretrained(args.output)
+    restored = (
+        type(exported)
+        .from_pretrained(
+            args.output,
+            config=load_task_config(args.output),
+            torch_dtype=torch.float32,
+            attn_implementation="sdpa",
+        )
+        .eval()
+    )
+    verify_full_checkpoint(restored, args.output)
+    assert_task_head_preserved(head, restored)
+    with torch.inference_mode():
+        reloaded_logits = restored(**tokens).logits
+    torch.testing.assert_close(after, reloaded_logits, atol=0, rtol=0)
+    mapping_files = write_runtime_mappings(args.output, labels, args.runtime_task)
+    receipt = {
+        "method": args.method,
+        "base_model": args.base_id,
+        "base_revision": args.base_revision,
+        "run_manifest_sha256": hashlib.sha256(
+            args.run_manifest.read_bytes()
+        ).hexdigest(),
+        "parameter_count": sum(
+            parameter.numel() for parameter in exported.parameters()
+        ),
+        "encoder_parameter_count": sum(
+            parameter.numel() for parameter in exported.model.parameters()
+        ),
+        "label2id": labels,
+        "runtime_task": args.runtime_task,
+        "runtime_mapping_files": mapping_files,
+        "task": exported.config.problem_type,
+        "classifier_pooling": exported.config.classifier_pooling,
+        "max_position_embeddings": exported.config.max_position_embeddings,
+        "position_capacity_is_quality_evidence": False,
+        "weight_dtype": "float32",
+        "task_head": head,
+        "task_head_preserved_after_reload": True,
+        "all_tensors_preserved_after_reload": True,
+        "test_used_for_selection": False,
+        "files": {
+            path.name: hashlib.sha256(path.read_bytes()).hexdigest()
+            for path in sorted(args.output.iterdir())
+            if path.is_file()
+        },
+    }
+    if args.method == "lora":
+        receipt.update(
+            adapter_sha256=hashlib.sha256(
+                (args.adapter / "adapter_model.safetensors").read_bytes()
+            ).hexdigest(),
+            task_head_preserved_after_merge_and_reload=True,
+            merge_probe_max_absolute_difference=float((before - after).abs().max()),
+        )
+    (args.output / "candidate-lock.json").write_text(
+        json.dumps(receipt, indent=2) + "\n"
+    )
+    print(
+        json.dumps(
+            {
+                key: receipt[key]
+                for key in (
+                    "method",
+                    "parameter_count",
+                    "all_tensors_preserved_after_reload",
+                )
+            }
+        ),
+        flush=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/fetch_sources.py

@@ -0,0 +1,48 @@
+"""Fetch only the pinned public dataset files used by the repair recipes."""
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+
+def main():
+    from huggingface_hub import hf_hub_download  # noqa: PLC0415
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--manifest",
+        type=Path,
+        default=Path(__file__).parent / "data/source-files.json",
+    )
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    receipts = []
+    for item in json.loads(args.manifest.read_text()):
+        directory = args.output / item["repo"].replace("/", "--")
+        path = Path(
+            hf_hub_download(
+                repo_id=item["repo"],
+                repo_type="dataset",
+                filename=item["file"],
+                revision=item["revision"],
+                local_dir=directory,
+            )
+        )
+        digest = hashlib.sha256(path.read_bytes()).hexdigest()
+        if digest != item["sha256"]:
+            raise ValueError(
+                f"Pinned dataset file hash differs: {item['repo']}/{item['file']}"
+            )
+        receipts.append({**item, "verified_sha256": digest})
+        print(
+            json.dumps({"repo": item["repo"], "file": item["file"], "verified": True}),
+            flush=True,
+        )
+    (args.output / "verified-source-files.json").write_text(
+        json.dumps(receipts, indent=2) + "\n"
+    )
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/hydrate_annotations.py

@@ -0,0 +1,95 @@
+"""Reconstruct reviewed requests from pinned source data and text-free label sidecars."""
+
+# ruff: noqa: PLC0415
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+
+def sha256_text(text):
+    return hashlib.sha256(text.encode()).hexdigest()
+
+
+def hydrate_records(sidecar, sources):
+    repository = sidecar["source"]["repository"]
+    result = []
+    if repository == "CohereLabs/aya_dataset":
+        lookup = {}
+        for row in sources:
+            if not row.get("user_id"):
+                continue
+            key = (
+                sha256_text(row["inputs"]),
+                row["language_code"],
+                "aya-author-" + sha256_text(row["user_id"]),
+            )
+            lookup[key] = row["inputs"]
+        for row in sidecar["records"]:
+            key = (row["source_text_sha256"], row["language"], row["group_id"])
+            if key not in lookup:
+                raise ValueError(
+                    "Reviewed Aya request or author differs from the pinned source"
+                )
+            result.append({**row, "text": lookup[key]})
+    elif repository == "databricks/databricks-dolly-15k":
+        for row in sidecar["records"]:
+            source = sources[row["source_row"]]
+            instruction, context = (
+                source["instruction"].strip(),
+                source["context"].strip(),
+            )
+            text = instruction + ("\n\n" + context if context else "")
+            if sha256_text(text) != row["source_text_sha256"]:
+                raise ValueError(
+                    "Reviewed Dolly request differs from the pinned source"
+                )
+            result.append(
+                {**row, "text": text, "instruction": instruction, "context": context}
+            )
+    else:
+        raise ValueError("Unknown reviewed source")
+    return result
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--sidecar", type=Path, required=True)
+    parser.add_argument("--source", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    if args.output.exists():
+        raise ValueError("Refusing to overwrite reviewed annotations")
+    sidecar = json.loads(args.sidecar.read_text())
+    if (
+        hashlib.sha256(args.source.read_bytes()).hexdigest()
+        != sidecar["source"]["file_sha256"]
+    ):
+        raise ValueError("Source file hash differs from the reviewed revision")
+    if sidecar["source"]["repository"] == "CohereLabs/aya_dataset":
+        from pyarrow import parquet
+
+        sources = parquet.read_table(
+            args.source, columns=["inputs", "language_code", "user_id"]
+        ).to_pylist()
+    else:
+        sources = [
+            json.loads(line)
+            for line in args.source.read_text().split("\n")
+            if line.strip()
+        ]
+    rows = hydrate_records(sidecar, sources)
+    content = "".join(
+        json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n" for row in rows
+    )
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(content)
+    print(
+        json.dumps(
+            {"rows": len(rows), "sha256": hashlib.sha256(content.encode()).hexdigest()}
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/initialize.py

@@ -0,0 +1,76 @@
+"""Initialize a fresh task adapter and trainable head from a fixed encoder base."""
+
+# ruff: noqa: PLC0415
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+from .model import save_adapter
+from .optimization import load_trainable_model
+
+
+def main():
+    import torch
+    from peft import LoraConfig, TaskType, get_peft_model
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--base-id", required=True)
+    parser.add_argument("--base-revision", required=True)
+    parser.add_argument("--contract", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--seed", type=int, default=20260913)
+    parser.add_argument("--rank", type=int, default=32)
+    parser.add_argument("--alpha", type=int, default=64)
+    parser.add_argument("--dropout", type=float, default=0.05)
+    args = parser.parse_args()
+    if args.output.exists():
+        raise ValueError("Refusing to overwrite an initialized adapter")
+    if min(args.rank, args.alpha) <= 0 or not 0 <= args.dropout < 1:
+        raise ValueError("Invalid LoRA dimensions or dropout")
+    torch.set_num_threads(8)
+    torch.manual_seed(args.seed)
+    model, tokenizer, _labels, _ids = load_trainable_model(
+        args.base, args.contract, None, "full", fresh_head=True
+    )
+    model = get_peft_model(
+        model,
+        LoraConfig(
+            task_type=TaskType.SEQ_CLS,
+            r=args.rank,
+            lora_alpha=args.alpha,
+            lora_dropout=args.dropout,
+            target_modules=["attn.Wqkv", "attn.Wo", "mlp.Wi", "mlp.Wo"],
+            modules_to_save=["head", "classifier"],
+            bias="none",
+            revision=args.base_revision,
+        ),
+    )
+    save_adapter(model, tokenizer, args.output, args.base_id, args.base_revision)
+    (args.output / "contract.json").write_bytes(args.contract.read_bytes())
+    receipt = {
+        "base_model": args.base_id,
+        "base_revision": args.base_revision,
+        "seed": args.seed,
+        "rank": args.rank,
+        "alpha": args.alpha,
+        "dropout": args.dropout,
+        "classifier_pooling": model.config.classifier_pooling,
+        "adapter_sha256": hashlib.sha256(
+            (args.output / "adapter_model.safetensors").read_bytes()
+        ).hexdigest(),
+        "trainable_parameters": sum(
+            parameter.numel()
+            for parameter in model.parameters()
+            if parameter.requires_grad
+        ),
+    }
+    (args.output / "initialization.json").write_text(
+        json.dumps(receipt, indent=2) + "\n"
+    )
+    print(json.dumps(receipt))
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/model.py

@@ -0,0 +1,94 @@
+"""HF standard sequence checkpoints with an unchanged label contract."""
+
+# Heavy imports stay optional for data-contract tests.
+# ruff: noqa: PLC0415
+
+import json
+from pathlib import Path
+
+from .data import load_contract
+from .task_head import verify_saved_task_head
+
+
+def configuration_overrides(contract):
+    """Validate optional standard HF task settings without guessing a pooling mode."""
+    content = json.loads(Path(contract).read_text())
+    problem_type = content.get("problem_type", "single_label_classification")
+    if problem_type not in {
+        "single_label_classification",
+        "multi_label_classification",
+    }:
+        raise ValueError("Unsupported classification problem type")
+    overrides = {"problem_type": problem_type}
+    if "classifier_pooling" in content:
+        pooling = content["classifier_pooling"]
+        if pooling not in {"mean", "cls"}:
+            raise ValueError("Sequence pooling must be explicitly mean or cls")
+        overrides["classifier_pooling"] = pooling
+    return overrides
+
+
+def load_task_config(base, **overrides):
+    """Keep optional execution settings on the configuration that owns them."""
+    from transformers import AutoConfig
+
+    config = AutoConfig.from_pretrained(base, **overrides)
+    if hasattr(config, "reference_compile"):
+        config.reference_compile = False
+    return config
+
+
+def load_model(base, contract, adapter=None, trainable=False):
+    import torch
+    from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+    label_to_id, id_to_label = load_contract(contract)
+    overrides = configuration_overrides(contract)
+    tokenizer = AutoTokenizer.from_pretrained(base)
+    config = load_task_config(
+        base,
+        num_labels=len(label_to_id),
+        label2id=label_to_id,
+        id2label=id_to_label,
+        **overrides,
+    )
+    model, loading = AutoModelForSequenceClassification.from_pretrained(
+        base,
+        config=config,
+        torch_dtype=torch.float32,
+        attn_implementation="sdpa",
+        output_loading_info=True,
+    )
+    missing_encoder = [
+        name for name in loading["missing_keys"] if name.startswith("model.")
+    ]
+    if missing_encoder:
+        raise ValueError(f"Checkpoint is missing encoder tensors: {missing_encoder}")
+    if adapter:
+        from peft import PeftModel
+
+        model = PeftModel.from_pretrained(model, adapter, is_trainable=trainable)
+    elif trainable:
+        raise ValueError("Continuation requires an explicit initial adapter")
+    if model.config.label2id != label_to_id:
+        raise ValueError("The loaded classifier changed the label contract")
+    return model, tokenizer, label_to_id, id_to_label
+
+
+def save_adapter(model, tokenizer, destination, base_id, base_revision):
+    import json
+
+    destination.mkdir(parents=True, exist_ok=True)
+    model.peft_config["default"].base_model_name_or_path = base_id
+    model.peft_config["default"].revision = base_revision
+    model.save_pretrained(destination)
+    scope = verify_saved_task_head(model, destination / "adapter_model.safetensors")
+    (destination / "task-head.json").write_text(json.dumps(scope, indent=2) + "\n")
+    tokenizer.save_pretrained(destination)
+    (destination / "label_mapping.json").write_text(
+        json.dumps(
+            {"label2id": model.config.label2id, "id2label": model.config.id2label},
+            indent=2,
+        )
+        + "\n"
+    )

reproducibility/sequence_repair/optimization.py

@@ -0,0 +1,220 @@
+"""Explicit adapter or full-encoder optimization and checkpoint ownership."""
+
+# ruff: noqa: PLC0415
+
+import hashlib
+import json
+import math
+from pathlib import Path
+
+from .model import load_model, save_adapter
+from .task_head import task_head_scope, tensor_digest
+
+
+def configure_trainable_tail(model, last_layers):
+    """Restrict a complete ModernBERT checkpoint to its tail and entire task head."""
+    if (
+        type(model).__name__ != "ModernBertForSequenceClassification"
+        or hasattr(model, "peft_config")
+        or hasattr(model, "get_base_model")
+    ):
+        raise ValueError("Tail training requires a complete ModernBERT classifier")
+    layers = model.model.layers
+    if type(last_layers) is not int or not 1 <= last_layers <= len(layers):
+        raise ValueError("Trainable tail must contain one through all encoder layers")
+    start = len(layers) - last_layers
+    modules = [*layers[start:], model.head, model.classifier]
+    selected = {
+        id(parameter) for module in modules for parameter in module.parameters()
+    }
+    named = dict(model.named_parameters())
+    if not selected or not selected <= {id(parameter) for parameter in named.values()}:
+        raise ValueError("Trainable modules do not belong to the classifier")
+    if any(
+        id(parameter) in selected for parameter in model.model.final_norm.parameters()
+    ):
+        raise ValueError("Encoder final normalization must remain frozen")
+    model.requires_grad_(False)
+    for module in modules:
+        module.requires_grad_(True)
+    tensors = {
+        name: {"shape": list(parameter.shape), "numel": parameter.numel()}
+        for name, parameter in named.items()
+        if parameter.requires_grad
+    }
+    return {
+        "architecture": type(model).__name__,
+        "layer_indices": list(range(start, len(layers))),
+        "trainable_tensors": tensors,
+        "trainable_parameters": sum(item["numel"] for item in tensors.values()),
+        "encoder_final_norm_trainable": False,
+    }
+
+
+def frozen_parameter_receipt(model):
+    """Hash frozen parameters once so an optional restricted update is auditable."""
+    return {
+        name: {
+            "shape": list(parameter.shape),
+            "dtype": str(parameter.dtype),
+            "sha256": tensor_digest(parameter),
+        }
+        for name, parameter in model.named_parameters()
+        if not parameter.requires_grad
+    }
+
+
+def assert_frozen_parameters_preserved(expected, model):
+    if frozen_parameter_receipt(model) != expected:
+        raise ValueError("Frozen parameter scope or tensor values changed")
+
+
+def validate_method(method, adapter, fresh_head):
+    if method not in {"lora", "full"}:
+        raise ValueError("Training method must be lora or full")
+    if method == "lora" and (adapter is None or fresh_head):
+        raise ValueError("LoRA continuation requires its initialized adapter")
+    if method == "full" and adapter is not None:
+        raise ValueError(
+            "Full training starts from a complete checkpoint, not an adapter"
+        )
+
+
+def load_trainable_model(base, contract, adapter, method, fresh_head=False):
+    validate_method(method, adapter, fresh_head)
+    model, tokenizer, labels, ids = load_model(
+        base, contract, adapter, trainable=method == "lora"
+    )
+    if method == "full":
+        if type(model).__name__ != "ModernBertForSequenceClassification":
+            raise ValueError("Full sequence training requires a ModernBERT classifier")
+        model.requires_grad_(True)
+        if fresh_head:
+            from torch import nn
+
+            head_parameter = next(model.head.parameters())
+            model.head = type(model.head)(model.config).to(
+                device=head_parameter.device, dtype=head_parameter.dtype
+            )
+            classifier = model.classifier
+            model.classifier = nn.Linear(
+                classifier.in_features,
+                classifier.out_features,
+                bias=classifier.bias is not None,
+                device=classifier.weight.device,
+                dtype=classifier.weight.dtype,
+            )
+            # Fresh modules let HF apply its native initialization without
+            # mutating the initialization guards on loaded checkpoint tensors.
+            model.head.apply(model._init_weights)
+            # ModernBERT initializes the final classifier on its owning task
+            # module, not when visiting the bare Linear child.
+            model._init_weights(model)
+    return model, tokenizer, labels, ids
+
+
+def optimizer_groups(model, learning_rate, head_learning_rate=None):
+    rates = [learning_rate]
+    if head_learning_rate is not None:
+        rates.append(head_learning_rate)
+    if any(not math.isfinite(rate) or rate <= 0 for rate in rates):
+        raise ValueError("Learning rates must be finite and positive")
+    base = model.get_base_model() if hasattr(model, "get_base_model") else model
+    head_ids = {
+        id(parameter)
+        for module in (base.head, base.classifier)
+        for parameter in module.parameters()
+        if parameter.requires_grad
+    }
+    encoder, head = [], []
+    for parameter in model.parameters():
+        if parameter.requires_grad:
+            (head if id(parameter) in head_ids else encoder).append(parameter)
+    groups = []
+    for name, parameters, rate in (
+        ("encoder", encoder, learning_rate),
+        ("task_head", head, head_learning_rate or learning_rate),
+    ):
+        if parameters:
+            groups.append(
+                {"name": name, "params": parameters, "lr": rate, "initial_lr": rate}
+            )
+    if not groups:
+        raise ValueError("No trainable parameters")
+    return groups
+
+
+def initial_artifact_receipt(base, adapter):
+    """Hash the actual local initialization, including every weight shard."""
+    root = Path(base)
+    if not root.is_dir():
+        raise ValueError("Training requires a downloaded local base checkpoint")
+    names = {"config.json", "tokenizer.json", "tokenizer_config.json"}
+    for path in root.iterdir():
+        if (
+            path.suffix == ".safetensors"
+            or path.name
+            in {
+                "model.safetensors.index.json",
+                "pytorch_model.bin",
+                "pytorch_model.bin.index.json",
+            }
+            or path.name.startswith("pytorch_model-")
+        ):
+            names.add(path.name)
+    files = {}
+    for name in sorted(names):
+        path = root / name
+        if path.is_file():
+            with path.open("rb") as stream:
+                files[name] = hashlib.file_digest(stream, "sha256").hexdigest()
+    result = {"base_files": files}
+    if adapter is not None:
+        for filename in ("adapter_model.safetensors", "adapter_config.json"):
+            path = Path(adapter) / filename
+            if filename == "adapter_config.json" and not path.is_file():
+                result[filename] = None
+                continue
+            with path.open("rb") as stream:
+                result[filename] = hashlib.file_digest(stream, "sha256").hexdigest()
+    return result
+
+
+def verify_full_checkpoint(model, checkpoint):
+    """Require every tensor of a complete sequence checkpoint to be preserved."""
+    if type(model).__name__ != "ModernBertForSequenceClassification":
+        raise ValueError("Full checkpoints must own the actual encoder weights")
+    from safetensors.torch import load_file
+
+    saved = load_file(str(Path(checkpoint) / "model.safetensors"), device="cpu")
+    state = model.state_dict()
+    if saved.keys() != state.keys():
+        raise ValueError("Full checkpoint omitted or introduced model tensors")
+    import torch
+
+    for name, value in state.items():
+        if saved[name].dtype != value.dtype or not torch.equal(
+            saved[name], value.detach().cpu()
+        ):
+            raise ValueError(f"Full checkpoint changed tensor: {name}")
+
+
+def save_training_checkpoint(model, tokenizer, destination, method, base_id, revision):
+    if method == "lora":
+        save_adapter(model, tokenizer, destination, base_id, revision)
+        return
+    if method != "full" or hasattr(model, "peft_config"):
+        raise ValueError("Full checkpoints must own the actual encoder weights")
+    destination.mkdir(parents=True, exist_ok=True)
+    expected = task_head_scope(model)
+    model.save_pretrained(destination, safe_serialization=True)
+    verify_full_checkpoint(model, destination)
+    tokenizer.save_pretrained(destination)
+    (destination / "task-head.json").write_text(json.dumps(expected, indent=2) + "\n")
+    (destination / "training-origin.json").write_text(
+        json.dumps(
+            {"method": method, "base_model": base_id, "base_revision": revision},
+            indent=2,
+        )
+        + "\n"
+    )

reproducibility/sequence_repair/prepare_context.py

@@ -0,0 +1,158 @@
+"""Frozen context curriculum and held-out stress variants from task source groups."""
+
+# Chinese punctuation is intentional in language-specific task markers.
+# ruff: noqa: PLC0415, RUF001
+import argparse
+import hashlib
+import json
+from collections import defaultdict
+from pathlib import Path
+
+from .context import insert_payload
+from .corpus import write_corpus
+from .data import read_records
+
+LANGUAGE_ALIASES = {
+    "ara": "ar",
+    "eng": "en",
+    "zho": "zh",
+    "spa": "es",
+    "fra": "fr",
+    "deu": "de",
+    "jpn": "ja",
+}
+
+TASK_MARKERS = {
+    "ar": "المهمة المطلوبة:\n",
+    "de": "Angefragte Aufgabe:\n",
+    "en": "Requested task:\n",
+    "es": "Tarea solicitada:\n",
+    "fr": "Tâche demandée :\n",
+    "ja": "依頼されたタスク：\n",
+    "zh": "需要完成的任务：\n",
+}
+
+
+def task_payload(row):
+    """Mark a task in its declared language without rewriting its source text."""
+    language = LANGUAGE_ALIASES.get(row["language"], row["language"])
+    if language not in TASK_MARKERS:
+        raise ValueError(f"No context task marker for language: {language}")
+    return TASK_MARKERS[language] + row["text"]
+
+
+def choose_payloads(rows, languages, per_label):
+    pools = defaultdict(list)
+    for row in rows:
+        language = LANGUAGE_ALIASES.get(row.get("language"), row.get("language"))
+        if language in languages:
+            pools[(language, row["label"])].append(row)
+    result = []
+    for (language, _label), pool in sorted(pools.items()):
+        seen = set()
+        for row in sorted(
+            pool, key=lambda item: hashlib.sha256(item["id"].encode()).hexdigest()
+        ):
+            if row["group_id"] in seen:
+                continue
+            seen.add(row["group_id"])
+            result.append({**row, "language": language})
+            if len(seen) >= per_label:
+                break
+    return result
+
+
+def main():
+    from transformers import AutoTokenizer
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--corpus", type=Path, required=True)
+    parser.add_argument("--tokenizer", required=True)
+    parser.add_argument("--backgrounds", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--train-per-label", type=int, default=3)
+    parser.add_argument("--eval-per-label", type=int, default=1)
+    parser.add_argument(
+        "--budgets", type=int, nargs="+", default=[4096, 8192, 16384, 32768]
+    )
+    parser.add_argument(
+        "--languages", nargs="+", choices=sorted(TASK_MARKERS), default=["en", "zh"]
+    )
+    parser.add_argument("--payload-sources", nargs="+")
+    args = parser.parse_args()
+    if min(args.train_per_label, args.eval_per_label, *args.budgets) <= 0:
+        raise ValueError("Context budgets and sample counts must be positive")
+    backgrounds = json.loads(args.backgrounds.read_text())
+    if len({text for group in backgrounds.values() for text in group.values()}) != sum(
+        len(group) for group in backgrounds.values()
+    ):
+        raise ValueError("Background text must differ across languages and partitions")
+    tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
+    contract = json.loads((args.corpus / "contract.json").read_text())
+
+    def count(text):
+        return len(tokenizer(text, truncation=False, padding=False)["input_ids"])
+
+    splits = {}
+    for split in ["train", "dev", "test"]:
+        rows = read_records([args.corpus / f"{split}.jsonl"], contract["label2id"])
+        if args.payload_sources:
+            rows = [row for row in rows if row.get("source") in args.payload_sources]
+        chosen = choose_payloads(
+            rows,
+            args.languages,
+            args.train_per_label if split == "train" else args.eval_per_label,
+        )
+        if not chosen:
+            raise ValueError(f"No eligible {split} source groups")
+        output = []
+        for row in chosen:
+            language = row["language"]
+            payload = task_payload(row)
+            for budget in args.budgets:
+                for position in ["head", "middle", "tail"]:
+                    built = insert_payload(
+                        payload, backgrounds[split][language], position, budget, count
+                    )
+                    output.append(
+                        {
+                            **row,
+                            "id": f"context:{row['id']}:{budget}:{position}",
+                            "text": built["text"],
+                            "source": "authored-context-on-frozen-source",
+                            "parent_source": row["source"],
+                            "parent_id": row["id"],
+                            "length_bucket": str(budget),
+                            "position": position,
+                            "actual_tokens": built["actual_tokens"],
+                            "payload_start": built["payload_start"],
+                            "payload_end": built["payload_end"],
+                        }
+                    )
+        splits[split] = output
+        print(
+            json.dumps(
+                {"split": split, "payloads": len(chosen), "variants": len(output)}
+            ),
+            flush=True,
+        )
+    provenance = {
+        "parent_manifest_sha256": hashlib.sha256(
+            (args.corpus / "manifest.json").read_bytes()
+        ).hexdigest(),
+        "backgrounds_sha256": hashlib.sha256(args.backgrounds.read_bytes()).hexdigest(),
+        "tokenizer_sha256": hashlib.sha256(
+            (Path(args.tokenizer) / "tokenizer.json").read_bytes()
+        ).hexdigest(),
+        "train_payloads_per_label_language": args.train_per_label,
+        "eval_payloads_per_label_language": args.eval_per_label,
+        "budgets": args.budgets,
+        "languages": args.languages,
+        "scope": "Authored repeated multi-paragraph context around frozen source requests, not natural long-document accuracy. Label is the explicitly marked requested task. Paired length/position variants retain the parent source group and partition.",
+        "test_used": False,
+    }
+    print(json.dumps(write_corpus(args.output, splits, contract, provenance), indent=2))
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/prepare_retention.py

@@ -0,0 +1,87 @@
+"""Cache complete FP32 teacher outputs for explicitly marked replay TRAIN rows."""
+
+# ruff: noqa: PLC0415
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+
+from .data import read_records
+from .model import load_model
+from .optimization import initial_artifact_receipt, verify_full_checkpoint
+from .retention import row_identity, tokenizer_file_receipt, write_retention_cache
+
+
+def main():
+    import torch
+
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--contract", required=True)
+    parser.add_argument("--train", nargs="+", required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--max-length", type=int, default=32768)
+    parser.add_argument("--device", default="cuda")
+    args = parser.parse_args()
+    if args.output.exists():
+        raise ValueError("Refusing to overwrite retention targets")
+    if args.max_length <= 0:
+        raise ValueError("The teacher input budget must be positive")
+    initialization = initial_artifact_receipt(args.base, None)
+    model, tokenizer, labels, _ = load_model(args.base, args.contract)
+    if model.config.problem_type != "single_label_classification":
+        raise ValueError("Retention targets require single-label classification")
+    verify_full_checkpoint(model, args.base)
+    if args.max_length > model.config.max_position_embeddings:
+        raise ValueError("Teacher input budget exceeds checkpoint capacity")
+    rows = read_records(args.train, labels)
+    if any(row.get("split", "train") != "train" for row in rows):
+        raise ValueError("Teacher targets must not use development or test rows")
+    if any(not isinstance(row.get("retention_replay", False), bool) for row in rows):
+        raise ValueError("retention_replay must be a boolean TRAIN annotation")
+    replay = [row for row in rows if row.get("retention_replay", False)]
+    if not replay:
+        raise ValueError("Mark the old TRAIN rows with retention_replay: true")
+    encoded = [
+        tokenizer(row["text"], truncation=False, padding=False) for row in replay
+    ]
+    if any(len(value["input_ids"]) > args.max_length for value in encoded):
+        raise ValueError(
+            "A replay row exceeds the teacher budget; truncation is forbidden"
+        )
+    torch.set_num_threads(8)
+    torch.set_float32_matmul_precision("highest")
+    torch.backends.cuda.matmul.allow_tf32 = False
+    torch.backends.cudnn.allow_tf32 = False
+    model = model.to(args.device).eval()
+    records = []
+    with torch.inference_mode():
+        for row, value in zip(replay, encoded, strict=True):
+            batch = {
+                name: torch.tensor([ids], device=args.device)
+                for name, ids in value.items()
+            }
+            logits = model(**batch).logits
+            if logits.dtype != torch.float32 or not bool(torch.isfinite(logits).all()):
+                raise ValueError("Teacher outputs must be finite native FP32 logits")
+            records.append(
+                {**row_identity(row, value), "logits": logits[0].cpu().tolist()}
+            )
+    manifest = write_retention_cache(
+        args.output,
+        initialization=initialization,
+        contract_sha256=hashlib.sha256(Path(args.contract).read_bytes()).hexdigest(),
+        tokenizer_files=tokenizer_file_receipt(args.base),
+        label_to_id=labels,
+        records=records,
+    )
+    print(
+        json.dumps(
+            {"manifest": str(manifest), "rows": len(records), "dtype": "float32"}
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/retention.py

@@ -0,0 +1,410 @@
+"""Optional, identity-bound TRAIN replay targets and categorical retention loss."""
+
+# Torch stays optional for cache identity and data checks.
+# ruff: noqa: PLC0415
+
+import hashlib
+import json
+import math
+import re
+import struct
+from collections.abc import Mapping
+from pathlib import Path
+
+_LOGIT_RANK = 2
+_MINIMUM_LABELS = 2
+
+
+def masked_teacher_kl(
+    logits, teacher_logits, eligible, *, examples_per_step, temperature=2.0
+):
+    """Return detached T² KL(teacher || student), summed over a logical batch."""
+    import torch
+    from torch.nn import functional
+
+    if type(examples_per_step) is not int or examples_per_step <= 0:
+        raise ValueError("Logical example count must be a positive integer")
+    if (
+        isinstance(temperature, bool)
+        or not isinstance(temperature, (int, float))
+        or not math.isfinite(temperature)
+        or temperature <= 0
+    ):
+        raise ValueError("Retention temperature must be finite and positive")
+    if (
+        logits.ndim != _LOGIT_RANK
+        or logits.shape[1] < _MINIMUM_LABELS
+        or logits.shape[0] > examples_per_step
+        or teacher_logits.shape != logits.shape
+        or eligible.shape != logits.shape[:1]
+        or eligible.dtype != torch.bool
+        or not logits.is_floating_point()
+        or not teacher_logits.is_floating_point()
+        or teacher_logits.device != logits.device
+        or eligible.device != logits.device
+        or not torch.isfinite(logits).all()
+    ):
+        raise ValueError(
+            "Retention logits and mask must have compatible finite geometry"
+        )
+    if not eligible.any():
+        # Connected zero without summing possibly huge finite masked logits.
+        return logits.float()[:0].sum()
+    student = logits[eligible].float() / temperature
+    teacher = teacher_logits.detach()[eligible].float() / temperature
+    if not torch.isfinite(teacher).all():
+        raise ValueError("Eligible teacher logits must be finite")
+    student_logp = functional.log_softmax(student, dim=-1)
+    teacher_logp = functional.log_softmax(teacher, dim=-1)
+    loss = functional.kl_div(
+        student_logp, teacher_logp, log_target=True, reduction="sum"
+    ) * (temperature**2 / examples_per_step)
+    if not torch.isfinite(loss):
+        raise ValueError("Retention loss must be finite")
+    return loss
+
+
+def _sha(path):
+    with Path(path).open("rb") as stream:
+        return hashlib.file_digest(stream, "sha256").hexdigest()
+
+
+def _digest_map(value):
+    if not isinstance(value, dict) or not value:
+        raise ValueError("Artifact receipts must contain file digests")
+    for name, digest in value.items():
+        if (
+            not isinstance(name, str)
+            or not name
+            or Path(name).is_absolute()
+            or ".." in Path(name).parts
+            or not isinstance(digest, str)
+            or re.fullmatch(r"[0-9a-f]{64}", digest) is None
+        ):
+            raise ValueError(
+                "Artifact receipt contains an invalid relative name or SHA"
+            )
+
+
+def tokenizer_file_receipt(base):
+    """Bind the complete local fast-tokenizer files used by ModernBERT."""
+    base = Path(base)
+    names = {
+        "tokenizer.json",
+        "tokenizer_config.json",
+        "special_tokens_map.json",
+        "added_tokens.json",
+        "vocab.json",
+        "vocab.txt",
+        "merges.txt",
+        "tokenizer.model",
+        "spiece.model",
+        "sentencepiece.bpe.model",
+        "chat_template.jinja",
+    }
+    names.update(
+        str(path.relative_to(base))
+        for path in (base / "chat_templates").glob("*.jinja")
+    )
+    result = {
+        name: _sha(base / name) for name in sorted(names) if (base / name).is_file()
+    }
+    if not {"tokenizer.json", "tokenizer_config.json"} <= result.keys():
+        raise ValueError("Retention requires the complete local fast tokenizer")
+    return result
+
+
+def row_identity(row, encoded):
+    """Bind raw text and every unpadded model input, without storing text in targets."""
+    if any(
+        not isinstance(row.get(key), str) or not row[key]
+        for key in ("id", "group_id", "text", "label")
+    ):
+        raise ValueError("Retention rows require string ID, group, text and label")
+    if (
+        not isinstance(encoded, Mapping)
+        or not {"input_ids", "attention_mask"} <= encoded.keys()
+    ):
+        raise ValueError("Retention requires complete unpadded tokenizer inputs")
+    encoded = dict(encoded)
+    if not encoded.keys() <= {
+        "input_ids",
+        "attention_mask",
+        "token_type_ids",
+        "position_ids",
+    }:
+        raise ValueError("Unexpected tokenizer input in retention identity")
+    if not isinstance(encoded["input_ids"], list):
+        raise ValueError("Retention input IDs must be an unpadded integer list")
+    size = len(encoded["input_ids"])
+    if (
+        not size
+        or any(
+            not isinstance(values, list)
+            or len(values) != size
+            or any(type(value) is not int or value < 0 for value in values)
+            for values in encoded.values()
+        )
+        or any(value != 1 for value in encoded["attention_mask"])
+    ):
+        raise ValueError("Retention inputs must be complete unpadded integer sequences")
+    token_bytes = json.dumps(encoded, sort_keys=True, separators=(",", ":")).encode()
+    return {
+        "id": row["id"],
+        "group_id": row["group_id"],
+        "label": row["label"],
+        "text_sha256": hashlib.sha256(row["text"].encode()).hexdigest(),
+        "tokens_sha256": hashlib.sha256(token_bytes).hexdigest(),
+    }
+
+
+def _labels(value):
+    if (
+        not isinstance(value, dict)
+        or len(value) < _MINIMUM_LABELS
+        or any(
+            not isinstance(name, str) or not name or type(index) is not int
+            for name, index in value.items()
+        )
+        or sorted(value.values()) != list(range(len(value)))
+    ):
+        raise ValueError("Retention labels must be a contiguous bijection")
+
+
+def _record(record, labels):
+    expected = {"id", "group_id", "label", "text_sha256", "tokens_sha256", "logits"}
+    if not isinstance(record, dict) or set(record) != expected:
+        raise ValueError("Retention target fields differ from the schema")
+    if any(
+        not isinstance(record[key], str) or not record[key]
+        for key in ("id", "group_id", "label")
+    ):
+        raise ValueError("Invalid retention target identity")
+    _digest_map({key: record[key] for key in ("text_sha256", "tokens_sha256")})
+    values = record["logits"]
+    if (
+        record["label"] not in labels
+        or not isinstance(values, list)
+        or len(values) != len(labels)
+    ):
+        raise ValueError("Retention logits differ from the label geometry")
+    for value in values:
+        if (
+            isinstance(value, bool)
+            or not isinstance(value, (int, float))
+            or not math.isfinite(value)
+        ):
+            raise ValueError("Cached teacher logits must be finite float32 values")
+        try:
+            exact = struct.unpack("f", struct.pack("f", value))[0]
+        except (OverflowError, struct.error) as error:
+            raise ValueError("Cached teacher logit exceeds float32") from error
+        if exact != value:
+            raise ValueError("Cached teacher logit was not serialized from float32")
+
+
+def _metadata(initialization, contract_sha256, tokenizer_files, label_to_id):
+    if not isinstance(initialization, dict) or set(initialization) != {"base_files"}:
+        raise ValueError("Retention requires an exact complete-checkpoint receipt")
+    _digest_map(initialization["base_files"])
+    _digest_map({"contract": contract_sha256})
+    _digest_map(tokenizer_files)
+    _labels(label_to_id)
+    for name, digest in tokenizer_files.items():
+        if (
+            name in initialization["base_files"]
+            and initialization["base_files"][name] != digest
+        ):
+            raise ValueError("Tokenizer and initialization receipts disagree")
+
+
+def write_retention_cache(
+    destination,
+    *,
+    initialization,
+    contract_sha256,
+    tokenizer_files,
+    label_to_id,
+    records,
+):
+    """Write a new portable cache; the loader separately verifies admitted TRAIN scope."""
+    _metadata(initialization, contract_sha256, tokenizer_files, label_to_id)
+    records = list(records)
+    for record in records:
+        _record(record, label_to_id)
+    records.sort(key=lambda record: record["id"])
+    ids = [record["id"] for record in records]
+    if not ids or len(ids) != len(set(ids)):
+        raise ValueError("Retention targets must be nonempty with unique IDs")
+    destination = Path(destination)
+    destination.mkdir(parents=True, exist_ok=False)
+    targets = destination / "targets.jsonl"
+    with targets.open("w") as stream:
+        for record in records:
+            stream.write(json.dumps(record, sort_keys=True, allow_nan=False) + "\n")
+    manifest = {
+        "version": 1,
+        "split": "train",
+        "dtype": "float32",
+        "initialization": initialization,
+        "contract_sha256": contract_sha256,
+        "tokenizer_files": tokenizer_files,
+        "label_to_id": label_to_id,
+        "replay_ids": ids,
+        "rows": len(records),
+        "records_file": targets.name,
+        "records_sha256": _sha(targets),
+    }
+    path = destination / "manifest.json"
+    path.write_text(json.dumps(manifest, indent=2, allow_nan=False) + "\n")
+    return path
+
+
+class RetentionCache:
+    """Targets for explicitly designated old TRAIN rows; unknown rows are rejected."""
+
+    @classmethod
+    def load(
+        cls,
+        path,
+        *,
+        initialization,
+        contract_sha256,
+        tokenizer_files,
+        train_rows,
+        tokenized_rows,
+        label_to_id,
+        replay_ids,
+    ):
+        _metadata(initialization, contract_sha256, tokenizer_files, label_to_id)
+        path = Path(path)
+        raw = path.read_bytes()
+        manifest = json.loads(raw)
+        expected = {
+            "version",
+            "split",
+            "dtype",
+            "initialization",
+            "contract_sha256",
+            "tokenizer_files",
+            "label_to_id",
+            "replay_ids",
+            "rows",
+            "records_file",
+            "records_sha256",
+        }
+        if not isinstance(manifest, dict) or set(manifest) != expected:
+            raise ValueError("Retention manifest fields differ from the schema")
+        if (
+            type(manifest["version"]) is not int
+            or manifest["version"] != 1
+            or manifest["split"] != "train"
+            or manifest["dtype"] != "float32"
+        ):
+            raise ValueError("Retention requires version1 float32 TRAIN targets")
+        for key, value in (
+            ("initialization", initialization),
+            ("contract_sha256", contract_sha256),
+            ("tokenizer_files", tokenizer_files),
+            ("label_to_id", label_to_id),
+        ):
+            if manifest[key] != value:
+                raise ValueError(f"Retention cache differs from the actual {key}")
+        identities, marked = {}, set()
+        for row in train_rows:
+            if row.get("split", "train") != "train":
+                raise ValueError(
+                    "Retention targets cannot include development or test rows"
+                )
+            if type(row.get("retention_replay", False)) is not bool:
+                raise ValueError("TRAIN retention_replay designation must be boolean")
+            if row["id"] in identities or row["id"] not in tokenized_rows:
+                raise ValueError("Missing tokens or duplicate TRAIN retention ID")
+            identities[row["id"]] = row_identity(row, tokenized_rows[row["id"]])
+            if row["label"] not in label_to_id:
+                raise ValueError("Unknown TRAIN retention label")
+            if row.get("retention_replay", False):
+                marked.add(row["id"])
+        replay_ids = list(replay_ids)
+        if (
+            not marked
+            or len(replay_ids) != len(set(replay_ids))
+            or set(replay_ids) != marked
+            or manifest["replay_ids"] != sorted(marked)
+        ):
+            raise ValueError("Cache must cover exactly the designated old TRAIN subset")
+        filename = manifest["records_file"]
+        if (
+            not isinstance(filename, str)
+            or Path(filename).name != filename
+            or filename in {"", ".", ".."}
+        ):
+            raise ValueError("Retention records must use a relative filename")
+        targets = path.parent / filename
+        data = targets.read_bytes()
+        if hashlib.sha256(data).hexdigest() != manifest["records_sha256"]:
+            raise ValueError("Retention target bytes changed")
+        records = [json.loads(line) for line in data.splitlines() if line.strip()]
+        if (
+            type(manifest["rows"]) is not int
+            or len(records) != manifest["rows"]
+            or len(records) != len(marked)
+        ):
+            raise ValueError("Retention target coverage is incomplete")
+        cached = {}
+        for record in records:
+            _record(record, label_to_id)
+            identity = {key: value for key, value in record.items() if key != "logits"}
+            if (
+                record["id"] not in marked
+                or record["id"] in cached
+                or identity != identities[record["id"]]
+            ):
+                raise ValueError(
+                    "Retention target does not match its old TRAIN identity"
+                )
+            values = record["logits"]
+            maximum = max(values)
+            eligible = (
+                values.count(maximum) == 1
+                and values.index(maximum) == label_to_id[record["label"]]
+            )
+            cached[record["id"]] = (values, eligible)
+        result = cls()
+        result._identities, result._cached, result._width = (
+            identities,
+            cached,
+            len(label_to_id),
+        )
+        result.receipt = {
+            "manifest_sha256": hashlib.sha256(raw).hexdigest(),
+            "records_sha256": manifest["records_sha256"],
+            "replay_rows": len(cached),
+            "eligible_rows": sum(eligible for _, eligible in cached.values()),
+        }
+        return result
+
+    def targets(self, rows, device):
+        import torch
+
+        values, masks = [], []
+        for row in rows:
+            identity = self._identities.get(row["id"])
+            if (
+                identity is None
+                or any(row[key] != identity[key] for key in ("id", "group_id", "label"))
+                or hashlib.sha256(row["text"].encode()).hexdigest()
+                != identity["text_sha256"]
+            ):
+                raise ValueError(
+                    "Retention batch is not part of the validated TRAIN corpus"
+                )
+            logits, eligible = self._cached.get(row["id"], ([0.0] * self._width, False))
+            values.append(logits)
+            masks.append(eligible)
+        return (
+            torch.tensor(values, dtype=torch.float32, device=device).reshape(
+                len(rows), self._width
+            ),
+            torch.tensor(masks, dtype=torch.bool, device=device),
+        )

reproducibility/sequence_repair/runtime_mapping.py

@@ -0,0 +1,58 @@
+"""Serialize classifier label sidecars consumed by the router's Go loaders."""
+
+import json
+from pathlib import Path
+
+RUNTIME_TASKS = (
+    "domain",
+    "fact-check",
+    "modality",
+    "feedback",
+    "prompt-guard",
+    "safety",
+    "hazard",
+)
+
+
+def write_runtime_mappings(destination, labels, task):
+    """Write both HF and router layouts from one contiguous label bijection."""
+    if task not in RUNTIME_TASKS:
+        raise ValueError("Unknown runtime classifier task")
+    if not labels or set(labels.values()) != set(range(len(labels))):
+        raise ValueError("Runtime label IDs must be a contiguous bijection")
+    if task == "modality" and labels != {"AR": 0, "DIFFUSION": 1, "BOTH": 2}:
+        raise ValueError("Modality must preserve the native AR/DIFFUSION/BOTH IDs")
+    if task == "fact-check" and set(labels) != {
+        "NO_FACT_CHECK_NEEDED",
+        "FACT_CHECK_NEEDED",
+    }:
+        raise ValueError("FactCheck requires the router's two semantic labels")
+    reverse = {str(index): label for label, index in labels.items()}
+    generic = {
+        "label2id": labels,
+        "id2label": reverse,
+        "label_to_idx": labels,
+        "idx_to_label": reverse,
+    }
+    files = {"label_mapping.json": generic}
+    if task == "domain":
+        files["category_mapping.json"] = {
+            "category_to_idx": labels,
+            "idx_to_category": reverse,
+        }
+    sidecars = {
+        "fact-check": "fact_check_mapping.json",
+        "modality": "modality_mapping.json",
+        "prompt-guard": "jailbreak_type_mapping.json",
+    }
+    if task in sidecars:
+        files[sidecars[task]] = {
+            "label_to_idx": labels,
+            "idx_to_label": reverse,
+        }
+    for name, data in files.items():
+        path = Path(destination) / name
+        if path.exists():
+            raise ValueError("Refusing to overwrite an existing runtime mapping")
+        path.write_text(json.dumps(data, indent=2) + "\n")
+    return sorted(files)

reproducibility/sequence_repair/selection.py

@@ -0,0 +1,158 @@
+"""Development-only binary operating points under an explicit false-positive budget."""
+
+import hashlib
+import json
+import math
+from collections import Counter, defaultdict
+from fractions import Fraction
+
+BINARY_FP_SELECTION = "binary-fp-budget-recall"
+BINARY_LABEL_COUNT = 2
+
+
+def validate_binary_selection(label_to_id, positive_label, budget):
+    """Require an unambiguous binary ontology and a finite, explicit FP budget."""
+    if (
+        len(label_to_id) != BINARY_LABEL_COUNT
+        or set(label_to_id.values()) != {0, 1}
+        or any(type(index) is not int for index in label_to_id.values())
+    ):
+        raise ValueError("FP-budget selection requires exactly two label IDs: 0 and 1")
+    if positive_label not in label_to_id:
+        raise ValueError("FP-budget selection requires an exact positive label")
+    if (
+        isinstance(budget, bool)
+        or not isinstance(budget, (int, float))
+        or not math.isfinite(budget)
+        or not 0 <= budget <= 1
+    ):
+        raise ValueError("Selection false-positive budget must be finite in [0, 1]")
+
+
+def validate_selection_options(selection, label_to_id, positive_label, budget):
+    if selection == BINARY_FP_SELECTION:
+        validate_binary_selection(label_to_id, positive_label, budget)
+    elif positive_label is not None or budget is not None:
+        raise ValueError("Binary selection options require binary-fp-budget-recall")
+
+
+def binary_selection_support(records, label_to_id, positive_label):
+    counts = Counter(row["label"] for row in records)
+    if set(counts) != set(label_to_id):
+        raise ValueError("FP-budget development data must observe both binary labels")
+    if any(not row.get("id") for row in records) or len(
+        {row["id"] for row in records}
+    ) != len(records):
+        raise ValueError("Development IDs must be nonempty and unique")
+    return {
+        "positive": counts[positive_label],
+        "negative": len(records) - counts[positive_label],
+    }
+
+
+def binary_checkpoint_key(operating_point):
+    """Tie policy also applies across checkpoints; exact ties keep the earlier step."""
+    if not operating_point["feasible"]:
+        return None
+    return (
+        operating_point["recall"],
+        -operating_point["confusion"]["fp"],
+        operating_point["threshold"],
+    )
+
+
+def fit_binary_operating_point(
+    records, predictions, label_to_id, positive_label, budget
+):
+    """Sweep complete tied-score groups with p(positive) >= threshold in [0, 1]."""
+    validate_binary_selection(label_to_id, positive_label, budget)
+    support = binary_selection_support(records, label_to_id, positive_label)
+    by_id = {row["id"]: row for row in predictions}
+    if len(by_id) != len(predictions) or set(by_id) != {row["id"] for row in records}:
+        raise ValueError("Development prediction IDs must match each row exactly once")
+    positive_id = label_to_id[positive_label]
+    grouped = defaultdict(lambda: [0, 0])
+    evidence = []
+    for row in records:
+        scores = by_id[row["id"]]["probabilities"]
+        if (
+            len(scores) != BINARY_LABEL_COUNT
+            or any(
+                isinstance(score, bool)
+                or not isinstance(score, (int, float))
+                or not math.isfinite(score)
+                or not 0 <= score <= 1
+                for score in scores
+            )
+            or not math.isclose(sum(scores), 1.0, rel_tol=0, abs_tol=1e-5)
+        ):
+            raise ValueError("Expected finite normalized binary probabilities")
+        score = scores[positive_id]
+        is_positive = row["label"] == positive_label
+        grouped[score][0 if is_positive else 1] += 1
+        evidence.append(
+            {"id": row["id"], "label": row["label"], "probabilities": scores}
+        )
+    # Decimal budget arithmetic avoids floor(0.3 * 10) rounding surprises without
+    # an epsilon that could admit an extra false positive.
+    maximum_fp = math.floor(Fraction(str(budget)) * support["negative"])
+    receipt = {
+        "metric": BINARY_FP_SELECTION,
+        "positive_label": positive_label,
+        "positive_label_id": positive_id,
+        "negative_label": next(
+            label for label in label_to_id if label != positive_label
+        ),
+        "false_positive_budget": budget,
+        "maximum_false_positives": maximum_fp,
+        "support": support,
+        "comparison": "positive_probability >= threshold",
+        "threshold_range": [0, 1],
+        "tie_break": ["maximum_recall", "fewer_false_positives", "higher_threshold"],
+        "evidence_sha256": hashlib.sha256(
+            json.dumps(
+                sorted(evidence, key=lambda row: row["id"]),
+                sort_keys=True,
+                separators=(",", ":"),
+                allow_nan=False,
+            ).encode()
+        ).hexdigest(),
+        "test_used": False,
+        "feasible": False,
+        "threshold": None,
+        "recall": None,
+        "precision": None,
+        "false_positive_rate": None,
+        "confusion": None,
+    }
+    best_key, tp, fp = None, 0, 0
+    for threshold in sorted({0.0, 1.0, *grouped}, reverse=True):
+        positive_count, negative_count = grouped[threshold]
+        tp += positive_count
+        fp += negative_count
+        if fp > maximum_fp:
+            continue
+        key = (tp, -fp, threshold)
+        if best_key is not None and key <= best_key:
+            continue
+        best_key = key
+        receipt.update(
+            {
+                "feasible": True,
+                "threshold": threshold,
+                "recall": tp / support["positive"],
+                "precision": tp / (tp + fp) if tp + fp else 0.0,
+                "false_positive_rate": fp / support["negative"],
+                "confusion": {
+                    "tp": tp,
+                    "fp": fp,
+                    "fn": support["positive"] - tp,
+                    "tn": support["negative"] - fp,
+                },
+            }
+        )
+    if not receipt["feasible"]:
+        receipt["infeasible_reason"] = (
+            "Negative examples at probability 1 exceed the budget even at threshold 1"
+        )
+    return receipt

reproducibility/sequence_repair/task_head.py

@@ -0,0 +1,102 @@
+"""Inspect the effective task head and verify its adapter/merged storage."""
+
+# ruff: noqa: PLC0415
+
+import hashlib
+
+
+def _modules(model):
+    base = model.get_base_model() if hasattr(model, "get_base_model") else model
+    if type(base).__name__ not in {
+        "ModernBertForSequenceClassification",
+        "ModernBertForTokenClassification",
+    }:
+        raise ValueError(
+            "Task-head auditing requires a supported ModernBERT task model"
+        )
+    for name in ("head", "classifier"):
+        module = getattr(base, name)
+        owned = hasattr(module, "modules_to_save")
+        if owned:
+            active = module.active_adapter
+            if isinstance(active, list):
+                if len(active) != 1:
+                    raise ValueError("Task-head auditing requires one active adapter")
+                active = active[0]
+            module = module.modules_to_save[active]
+        yield name, module, owned
+
+
+def tensor_digest(tensor):
+    import torch
+
+    value = tensor.detach().cpu().contiguous()
+    if not bool(torch.isfinite(value).all()):
+        raise ValueError("Task head contains non-finite tensors")
+    return hashlib.sha256(value.view(torch.uint8).numpy().tobytes()).hexdigest()
+
+
+def task_head_scope(model):
+    """Describe actual effective tensors, including deliberately inherited heads."""
+    tensors = {}
+    for name, module, owned in _modules(model):
+        trainable = dict(module.named_parameters())
+        for key, tensor in module.state_dict().items():
+            parameter = trainable.get(key)
+            tensors[f"{name}.{key}"] = {
+                "shape": list(tensor.shape),
+                "dtype": str(tensor.dtype),
+                "numel": tensor.numel(),
+                "trainable": parameter is not None and parameter.requires_grad,
+                "adapter_owned": owned,
+                "sha256": tensor_digest(tensor),
+            }
+    base = model.get_base_model() if hasattr(model, "get_base_model") else model
+    return {
+        "architecture": type(base).__name__,
+        "modules_to_save": (
+            model.peft_config["default"].modules_to_save
+            if hasattr(model, "get_base_model")
+            else None
+        ),
+        "trainable_task_head_parameters": sum(
+            item["numel"] for item in tensors.values() if item["trainable"]
+        ),
+        "tensors": tensors,
+    }
+
+
+def verify_saved_task_head(model, adapter_path):
+    """Reject an adapter that omits or changes an effective trainable task tensor."""
+    from safetensors.torch import load_file
+
+    saved = load_file(str(adapter_path), device="cpu")
+    scope = task_head_scope(model)
+    for name, item in scope["tensors"].items():
+        key = f"base_model.model.{name}"
+        value = saved.get(key)
+        item["saved_key"] = key if value is not None else None
+        if value is None:
+            if item["trainable"] or item["adapter_owned"]:
+                raise ValueError(f"Adapter omitted effective task tensor: {name}")
+            continue
+        if (
+            list(value.shape) != item["shape"]
+            or str(value.dtype) != item["dtype"]
+            or tensor_digest(value) != item["sha256"]
+        ):
+            raise ValueError(f"Adapter changed effective task tensor: {name}")
+    return scope
+
+
+def assert_task_head_preserved(expected, model):
+    """Compare tensor identity across PEFT merge and serialized model reload."""
+    observed = task_head_scope(model)
+    identity = ("shape", "dtype", "sha256")
+    expected_tensors = expected["tensors"]
+    if expected_tensors.keys() != observed["tensors"].keys():
+        raise ValueError("Task-head tensor names changed")
+    for name, item in expected_tensors.items():
+        if any(item[key] != observed["tensors"][name][key] for key in identity):
+            raise ValueError(f"Task-head tensor changed: {name}")
+    return observed

reproducibility/sequence_repair/train.py

@@ -0,0 +1,791 @@
+"""Explicit single-GPU classifier training with optional development selection."""
+
+# ruff: noqa: PLC0415
+
+import argparse
+import hashlib
+import json
+import math
+import random
+import time
+from collections import Counter, defaultdict
+from pathlib import Path
+
+from .data import assert_disjoint, file_receipts, label_counts, read_records
+from .evaluate import evaluate_records
+from .optimization import (
+    assert_frozen_parameters_preserved,
+    configure_trainable_tail,
+    frozen_parameter_receipt,
+    initial_artifact_receipt,
+    load_trainable_model,
+    optimizer_groups,
+    save_training_checkpoint,
+    validate_method,
+    verify_full_checkpoint,
+)
+from .retention import RetentionCache, masked_teacher_kl, tokenizer_file_receipt
+from .selection import (
+    BINARY_FP_SELECTION,
+    binary_checkpoint_key,
+    binary_selection_support,
+    fit_binary_operating_point,
+    validate_selection_options,
+)
+from .task_head import task_head_scope
+from .training_order import load_training_order
+
+
+def microbatch_loss(logits, labels, accumulation_steps):
+    from torch.nn import functional
+
+    if accumulation_steps <= 0:
+        raise ValueError("Gradient accumulation must be positive")
+    # Each microbatch has the same number of examples. HF Trainer loss-kwargs
+    # normalization is intentionally not involved in this explicit loop.
+    return (
+        functional.cross_entropy(logits.float(), labels, reduction="mean")
+        / accumulation_steps
+    )
+
+
+def global_batch_loss(logits, labels, examples_per_step):
+    """Normalize variable microbatches by the same sampled optimizer-step size."""
+    from torch.nn import functional
+
+    if examples_per_step <= 0:
+        raise ValueError("Optimizer-step example count must be positive")
+    return (
+        functional.cross_entropy(logits.float(), labels, reduction="sum")
+        / examples_per_step
+    )
+
+
+def token_budget_microbatches(indices, lengths, token_budget):
+    """Stable length ordering without dropping or duplicating sampled examples."""
+    if token_budget <= 0:
+        raise ValueError("Microbatch token budget must be positive")
+    batches, current = [], []
+    for index in sorted(indices, key=lengths.__getitem__):
+        length = lengths[index]
+        if length > token_budget:
+            raise ValueError("A sampled example exceeds the microbatch token budget")
+        if current and length * (len(current) + 1) > token_budget:
+            batches.append(current)
+            current = []
+        current.append(index)
+    if current:
+        batches.append(current)
+    return batches
+
+
+def selection_score(metrics, selection):
+    if selection == "macro-f1":
+        return metrics["macro_f1"]
+    if selection in {
+        "length-macro-f1",
+        "source-macro-f1",
+        "source-present-macro-f1",
+    }:
+        field = "length_bucket" if selection == "length-macro-f1" else "source"
+        metric = (
+            "macro_f1_present_labels"
+            if selection == "source-present-macro-f1"
+            else "macro_f1"
+        )
+        groups = list(metrics["breakdowns"][field].values())
+        if not groups:
+            raise ValueError("Grouped selection requires measured groups")
+        return sum(group[metric] for group in groups) / len(groups)
+    raise ValueError("Unknown development selection metric")
+
+
+def length_sampling_pools(rows):
+    buckets = defaultdict(lambda: defaultdict(list))
+    for index, row in enumerate(rows):
+        buckets[str(row.get("length_bucket", "unspecified"))][row["label"]].append(
+            index
+        )
+    return [list(labels.values()) for labels in buckets.values()]
+
+
+def sample_length_balanced(pools, rng, balance_labels):
+    bucket = rng.choice(pools)
+    pool = (
+        rng.choice(bucket)
+        if balance_labels
+        else [index for label in bucket for index in label]
+    )
+    return rng.choice(pool)
+
+
+def source_sampling_pools(rows, balance_lengths):
+    sources = defaultdict(lambda: defaultdict(lambda: defaultdict(list)))
+    for index, row in enumerate(rows):
+        if not row.get("source"):
+            raise ValueError("Source-balanced sampling requires explicit sources")
+        length = (
+            str(row.get("length_bucket", "unspecified")) if balance_lengths else "all"
+        )
+        sources[row["source"]][length][row["label"]].append(index)
+    return [
+        [list(labels.values()) for labels in lengths.values()]
+        for lengths in sources.values()
+    ]
+
+
+def source_sampling_weights(rows, specification):
+    """Resolve explicit probabilities in the same order as source pools."""
+    if specification is None:
+        return None
+    weights = json.loads(specification)
+    sources = list(dict.fromkeys(row.get("source") for row in rows))
+    if (
+        not sources
+        or not all(sources)
+        or not isinstance(weights, dict)
+        or set(weights) != set(sources)
+    ):
+        raise ValueError("Source weights must name every eligible source exactly")
+    values = [weights[source] for source in sources]
+    if any(
+        isinstance(value, bool)
+        or not isinstance(value, (int, float))
+        or not math.isfinite(value)
+        or value <= 0
+        for value in values
+    ):
+        raise ValueError("Source weights must be finite positive numbers")
+    total = sum(values)
+    if not math.isfinite(total):
+        raise ValueError("Total source weight must be finite")
+    return [value / total for value in values]
+
+
+def sample_source_balanced(pools, rng, balance_labels, source_weights=None):
+    source = (
+        rng.choice(pools)
+        if source_weights is None
+        else rng.choices(pools, weights=source_weights, k=1)[0]
+    )
+    return sample_length_balanced(source, rng, balance_labels)
+
+
+def sampling_exposure(rows, drawn):
+    """Report sampling counts separately from class/length selection metrics."""
+    sources = defaultdict(
+        lambda: {"rows": 0, "draws": 0, "unique_rows": 0, "label_draws": Counter()}
+    )
+    for index, row in enumerate(rows):
+        item = sources[row.get("source", "unspecified")]
+        count = drawn[index]
+        item["rows"] += 1
+        item["draws"] += count
+        item["unique_rows"] += int(count > 0)
+        item["label_draws"][row["label"]] += count
+    return dict(sources)
+
+
+def main():
+    import torch
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--base", required=True)
+    parser.add_argument("--base-id", required=True)
+    parser.add_argument("--base-revision", required=True)
+    parser.add_argument("--adapter", type=Path)
+    parser.add_argument("--method", choices=["lora", "full"], default="lora")
+    parser.add_argument("--fresh-head", action="store_true")
+    parser.add_argument("--contract", required=True)
+    parser.add_argument("--train", nargs="+", required=True)
+    parser.add_argument("--dev", nargs="+")
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--steps", type=int, default=400)
+    parser.add_argument("--batch-size", type=int, default=8)
+    parser.add_argument("--accumulate", type=int, default=2)
+    parser.add_argument("--max-length", type=int, default=2048)
+    parser.add_argument("--learning-rate", type=float, default=2e-5)
+    parser.add_argument("--head-learning-rate", type=float)
+    parser.add_argument("--trainable-last-layers", type=int)
+    parser.add_argument("--retention-targets", type=Path)
+    parser.add_argument("--retention-weight", type=float)
+    parser.add_argument("--retention-temperature", type=float)
+    parser.add_argument("--eval-every", type=int, default=100)
+    parser.add_argument(
+        "--evaluation-dtype", choices=["bfloat16", "float32"], default="bfloat16"
+    )
+    parser.add_argument(
+        "--selection",
+        choices=[
+            "macro-f1",
+            "length-macro-f1",
+            "source-macro-f1",
+            "source-present-macro-f1",
+            BINARY_FP_SELECTION,
+        ],
+        default="macro-f1",
+    )
+    parser.add_argument("--positive-label")
+    parser.add_argument("--selection-false-positive-budget", type=float)
+    parser.add_argument("--balanced-sampling", action="store_true")
+    parser.add_argument("--length-balanced-sampling", action="store_true")
+    parser.add_argument("--source-balanced-sampling", action="store_true")
+    parser.add_argument(
+        "--source-weights", help="JSON object of explicit source weights"
+    )
+    parser.add_argument("--microbatch-token-budget", type=int)
+    parser.add_argument("--training-order", type=Path)
+    mode = parser.add_mutually_exclusive_group()
+    mode.add_argument("--probe-only", action="store_true")
+    mode.add_argument(
+        "--train-only",
+        action="store_true",
+        help="Skip development data and selection; save only the final checkpoint",
+    )
+    parser.add_argument("--seed", type=int, default=20260913)
+    args = parser.parse_args()
+    if args.train_only:
+        if args.dev:
+            parser.error("--train-only must not receive development data")
+        if (
+            args.selection != "macro-f1"
+            or args.positive_label is not None
+            or args.selection_false_positive_budget is not None
+        ):
+            parser.error("--train-only does not select a development operating point")
+    elif not args.dev:
+        parser.error("--dev is required unless --train-only is set")
+    validate_method(args.method, args.adapter, args.fresh_head)
+    if args.trainable_last_layers is not None and args.method != "full":
+        raise ValueError("A trainable encoder tail requires full checkpoint training")
+    if args.retention_targets is None:
+        if args.retention_weight is not None or args.retention_temperature is not None:
+            raise ValueError("Retention settings require explicit TRAIN targets")
+    else:
+        if args.method != "full" or args.fresh_head:
+            raise ValueError("Retention requires the unchanged complete initial head")
+        args.retention_weight = (
+            0.5 if args.retention_weight is None else args.retention_weight
+        )
+        args.retention_temperature = (
+            2.0 if args.retention_temperature is None else args.retention_temperature
+        )
+        if any(
+            not math.isfinite(value) or value <= 0
+            for value in (args.retention_weight, args.retention_temperature)
+        ):
+            raise ValueError(
+                "Retention weight and temperature must be finite and positive"
+            )
+    if args.training_order and (
+        args.balanced_sampling
+        or args.length_balanced_sampling
+        or args.source_balanced_sampling
+        or args.source_weights is not None
+    ):
+        raise ValueError("An explicit training order cannot also configure sampling")
+    if (
+        min(
+            args.steps,
+            args.batch_size,
+            args.accumulate,
+            args.max_length,
+            args.eval_every,
+        )
+        <= 0
+        or args.learning_rate <= 0
+        or (
+            args.microbatch_token_budget is not None
+            and args.microbatch_token_budget <= 0
+        )
+    ):
+        raise ValueError("Training budgets and learning rate must be positive")
+    if args.output.exists() and any(args.output.iterdir()):
+        raise ValueError("Refusing to overwrite a training run")
+    torch.set_num_threads(8)
+    torch.manual_seed(args.seed)
+    rng = random.Random(args.seed)
+    initial_artifacts = initial_artifact_receipt(args.base, args.adapter)
+    model, tokenizer, label_to_id, id_to_label = load_trainable_model(
+        args.base, args.contract, args.adapter, args.method, args.fresh_head
+    )
+    if model.config.problem_type != "single_label_classification":
+        raise ValueError("This training loop requires single-label targets")
+    if args.retention_targets is not None:
+        verify_full_checkpoint(model, args.base)
+    trainable_scope = (
+        configure_trainable_tail(model, args.trainable_last_layers)
+        if args.trainable_last_layers is not None
+        else None
+    )
+    frozen_parameters = (
+        frozen_parameter_receipt(model) if trainable_scope is not None else None
+    )
+    if not args.train_only:
+        validate_selection_options(
+            args.selection,
+            label_to_id,
+            args.positive_label,
+            args.selection_false_positive_budget,
+        )
+    if args.max_length > model.config.max_position_embeddings:
+        raise ValueError("Training budget exceeds checkpoint position capacity")
+    rows, dev = (
+        read_records(args.train, label_to_id),
+        [] if args.train_only else read_records(args.dev, label_to_id),
+    )
+    assert_disjoint(rows, dev)
+    if args.selection == BINARY_FP_SELECTION:
+        binary_selection_support(dev, label_to_id, args.positive_label)
+    train, rejected, pools = [], [], defaultdict(list)
+    for row in rows:
+        encoded = tokenizer(row["text"], truncation=False, padding=False)
+        if len(encoded["input_ids"]) > args.max_length:
+            rejected.append(
+                {
+                    "id": row["id"],
+                    "reason": "explicit context budget",
+                    "tokens": len(encoded["input_ids"]),
+                }
+            )
+            continue
+        index = len(train)
+        train.append((row, encoded))
+        pools[row["label"]].append(index)
+    if not train:
+        raise ValueError("No training rows fit the explicit budget")
+    retention = None
+    if args.retention_targets is not None:
+        if any(
+            not isinstance(row.get("retention_replay", False), bool) for row, _ in train
+        ):
+            raise ValueError("retention_replay must be a boolean TRAIN annotation")
+        retention = RetentionCache.load(
+            args.retention_targets,
+            initialization=initial_artifacts,
+            contract_sha256=hashlib.sha256(
+                Path(args.contract).read_bytes()
+            ).hexdigest(),
+            tokenizer_files=tokenizer_file_receipt(args.base),
+            train_rows=[row for row, _ in train],
+            tokenized_rows={row["id"]: encoded for row, encoded in train},
+            label_to_id=label_to_id,
+            replay_ids=[
+                row["id"] for row, _ in train if row.get("retention_replay", False)
+            ],
+        )
+    train_lengths = [len(encoded["input_ids"]) for _, encoded in train]
+    if (
+        args.microbatch_token_budget is not None
+        and max(train_lengths) > args.microbatch_token_budget
+    ):
+        raise ValueError(
+            "Microbatch token budget cannot fit the longest eligible example"
+        )
+    training_order = (
+        load_training_order(
+            args.training_order,
+            [row for row, _ in train],
+            args.train,
+            steps=args.steps,
+            global_batch=args.batch_size * args.accumulate,
+        )
+        if args.training_order
+        else None
+    )
+    pool_values = list(pools.values())
+    length_pools = length_sampling_pools([row for row, _encoded in train])
+    balance_sources = args.source_balanced_sampling or args.source_weights is not None
+    source_weights = source_sampling_weights(
+        [row for row, _encoded in train], args.source_weights
+    )
+    source_pools = (
+        source_sampling_pools(
+            [row for row, _encoded in train], args.length_balanced_sampling
+        )
+        if balance_sources
+        else None
+    )
+    model.gradient_checkpointing_enable(
+        gradient_checkpointing_kwargs={"use_reentrant": False}
+    )
+    model.to("cuda")
+    model.train()
+    parameters = [
+        parameter for parameter in model.parameters() if parameter.requires_grad
+    ]
+    groups = optimizer_groups(model, args.learning_rate, args.head_learning_rate)
+    optimizer = torch.optim.AdamW(groups, weight_decay=0.01)
+    args.output.mkdir(parents=True, exist_ok=True)
+    metadata = {
+        "base_model": args.base_id,
+        "base_revision": args.base_revision,
+        "method": args.method,
+        "fresh_head": args.fresh_head,
+        "trainable_scope": trainable_scope,
+        "frozen_parameters": frozen_parameters,
+        "retention": (
+            {
+                "targets": retention.receipt,
+                "weight": args.retention_weight,
+                "temperature": args.retention_temperature,
+                "normalization": "all examples in the optimizer step",
+            }
+            if retention is not None
+            else None
+        ),
+        "initial_artifacts": initial_artifacts,
+        "initial_adapter_sha256": initial_artifacts.get("adapter_model.safetensors"),
+        "initial_adapter_config_sha256": initial_artifacts.get("adapter_config.json"),
+        "contract_sha256": hashlib.sha256(Path(args.contract).read_bytes()).hexdigest(),
+        "classifier_pooling": model.config.classifier_pooling,
+        "task_head": task_head_scope(model),
+        "problem_type": model.config.problem_type,
+        "train_files": file_receipts(args.train),
+        "dev_files": file_receipts(args.dev or []),
+        "train_only": args.train_only,
+        "checkpoint_policy": (
+            "final-only" if args.train_only else "development-and-final"
+        ),
+        "label2id": label_to_id,
+        "train_rows": len(train),
+        "dev_rows": len(dev),
+        "train_label_counts": label_counts([row for row, _ in train]),
+        "rejected": rejected,
+        "seed": args.seed,
+        "steps": args.steps,
+        "batch_size": args.batch_size,
+        "accumulate": args.accumulate,
+        "examples_per_optimizer_step": args.batch_size * args.accumulate,
+        "microbatch_token_budget": args.microbatch_token_budget,
+        "dropout_trajectory_equivalence": args.microbatch_token_budget is None,
+        "max_length": args.max_length,
+        "learning_rate": args.learning_rate,
+        "optimizer_groups": [
+            {
+                "name": group["name"],
+                "learning_rate": group["initial_lr"],
+                "parameters": sum(parameter.numel() for parameter in group["params"]),
+            }
+            for group in groups
+        ],
+        "balanced_sampling": args.balanced_sampling,
+        "training_order": training_order.receipt if training_order else None,
+        "length_balanced_sampling": args.length_balanced_sampling,
+        "source_balanced_sampling": balance_sources,
+        "source_sampling_probabilities": (
+            dict(
+                zip(
+                    dict.fromkeys(row["source"] for row, _ in train),
+                    source_weights,
+                    strict=True,
+                )
+            )
+            if source_weights is not None
+            else None
+        ),
+        "implementation_sha256": hashlib.sha256(
+            Path(__file__).read_bytes()
+        ).hexdigest(),
+        "selection": None if args.train_only else args.selection,
+        "evaluation_dtype": None if args.train_only else args.evaluation_dtype,
+        "test_used": False,
+        "precision": "FP32 parameters and explicit FP32 mean CE / BF16 autocast",
+        "attention": "sdpa",
+        "checkpointing": "non-reentrant",
+        "parameters": sum(parameter.numel() for parameter in model.parameters()),
+        "trainable_parameters": sum(parameter.numel() for parameter in parameters),
+    }
+    if args.selection == BINARY_FP_SELECTION:
+        metadata["binary_selection"] = {
+            "positive_label": args.positive_label,
+            "false_positive_budget": args.selection_false_positive_budget,
+            "support": binary_selection_support(dev, label_to_id, args.positive_label),
+            "implementation_sha256": hashlib.sha256(
+                Path(__file__).with_name("selection.py").read_bytes()
+            ).hexdigest(),
+        }
+    (args.output / "run.json").write_text(json.dumps(metadata, indent=2) + "\n")
+    print(
+        json.dumps(
+            {
+                "event": "ready",
+                "train_rows": len(train),
+                "rejected_rows": len(rejected),
+                "trainable_parameters": metadata["trainable_parameters"],
+            }
+        ),
+        flush=True,
+    )
+    best = -1.0
+    best_key = None
+    drawn = Counter()
+
+    def evaluate_checkpoint(step):
+        nonlocal best, best_key
+        metrics, predictions = evaluate_records(
+            model,
+            tokenizer,
+            dev,
+            label_to_id,
+            id_to_label,
+            args.max_length,
+            dtype=args.evaluation_dtype,
+        )
+        operating_point = None
+        if args.selection == BINARY_FP_SELECTION:
+            operating_point = fit_binary_operating_point(
+                dev,
+                predictions,
+                label_to_id,
+                args.positive_label,
+                args.selection_false_positive_budget,
+            )
+            metrics["binary_fp_budget"] = operating_point
+            key = binary_checkpoint_key(operating_point)
+            score = operating_point["recall"] if key is not None else -1.0
+            (args.output / f"dev-predictions-step-{step}.jsonl").write_text(
+                "".join(json.dumps(row) + "\n" for row in predictions)
+            )
+        else:
+            score = selection_score(metrics, args.selection)
+            key = (score,)
+        (args.output / f"sampling-step-{step}.json").write_text(
+            json.dumps(sampling_exposure([row for row, _ in train], drawn), indent=2)
+            + "\n"
+        )
+        (args.output / f"dev-step-{step}.json").write_text(
+            json.dumps(metrics, indent=2) + "\n"
+        )
+        print(
+            json.dumps(
+                {
+                    "event": "dev",
+                    "step": step,
+                    "score": score,
+                    "accuracy": metrics["accuracy"],
+                    "macro_f1": metrics["macro_f1"],
+                }
+            ),
+            flush=True,
+        )
+        if key is not None and (best_key is None or key > best_key):
+            best = score
+            best_key = key
+            if frozen_parameters is not None:
+                assert_frozen_parameters_preserved(frozen_parameters, model)
+            save_training_checkpoint(
+                model,
+                tokenizer,
+                args.output
+                / ("best-adapter" if args.method == "lora" else "best-model"),
+                args.method,
+                args.base_id,
+                args.base_revision,
+            )
+            (args.output / "selection.json").write_text(
+                json.dumps(
+                    {
+                        "step": step,
+                        "score": score,
+                        "metric": args.selection,
+                        "test_used": False,
+                        **(
+                            {"operating_point": operating_point}
+                            if operating_point is not None
+                            else {}
+                        ),
+                    },
+                    indent=2,
+                )
+                + "\n"
+            )
+
+    def draw_microbatch_indices():
+        return [
+            (
+                sample_source_balanced(
+                    source_pools, rng, args.balanced_sampling, source_weights
+                )
+                if balance_sources
+                else (
+                    sample_length_balanced(length_pools, rng, args.balanced_sampling)
+                    if args.length_balanced_sampling
+                    else (
+                        rng.choice(rng.choice(pool_values))
+                        if args.balanced_sampling
+                        else rng.randrange(len(train))
+                    )
+                )
+            )
+            for _ in range(args.batch_size)
+        ]
+
+    if not args.probe_only and not args.train_only:
+        evaluate_checkpoint(0)
+    for step in range(1, args.steps + 1):
+        optimizer.zero_grad(set_to_none=True)
+        torch.cuda.reset_peak_memory_stats()
+        torch.cuda.synchronize()
+        start, step_loss, lengths = time.perf_counter(), 0.0, []
+        step_ce_loss, step_retention_loss, retention_examples = 0.0, 0.0, 0
+        if training_order:
+            planned = training_order.indices_for_step(step)
+            microbatches = (
+                token_budget_microbatches(
+                    planned, train_lengths, args.microbatch_token_budget
+                )
+                if args.microbatch_token_budget is not None
+                else [
+                    planned[start : start + args.batch_size]
+                    for start in range(0, len(planned), args.batch_size)
+                ]
+            )
+        elif args.microbatch_token_budget is None:
+            microbatches = (draw_microbatch_indices() for _ in range(args.accumulate))
+        else:
+            sampled = [
+                index
+                for _ in range(args.accumulate)
+                for index in draw_microbatch_indices()
+            ]
+            microbatches = token_budget_microbatches(
+                sampled, train_lengths, args.microbatch_token_budget
+            )
+        microbatch_count, max_padded_tokens = 0, 0
+        for indices in microbatches:
+            microbatch_count += 1
+            max_padded_tokens = max(
+                max_padded_tokens, max(train_lengths[i] for i in indices) * len(indices)
+            )
+            drawn.update(indices)
+            selected = [train[index] for index in indices]
+            lengths.extend(len(encoded["input_ids"]) for _, encoded in selected)
+            batch = tokenizer.pad(
+                [encoded for _, encoded in selected], padding=True, return_tensors="pt"
+            )
+            batch = {key: value.to("cuda") for key, value in batch.items()}
+            gold = torch.tensor(
+                [label_to_id[row["label"]] for row, _ in selected], device="cuda"
+            )
+            with torch.autocast("cuda", dtype=torch.bfloat16):
+                logits = model(**batch).logits
+                loss = (
+                    microbatch_loss(logits, gold, args.accumulate)
+                    if args.microbatch_token_budget is None
+                    else global_batch_loss(
+                        logits, gold, args.batch_size * args.accumulate
+                    )
+                )
+            step_ce_loss += float(loss.detach())
+            if retention is not None:
+                teacher_logits, eligible = retention.targets(
+                    [row for row, _ in selected], device=logits.device
+                )
+                preservation = args.retention_weight * masked_teacher_kl(
+                    logits,
+                    teacher_logits,
+                    eligible,
+                    examples_per_step=args.batch_size * args.accumulate,
+                    temperature=args.retention_temperature,
+                )
+                loss = loss + preservation
+                step_retention_loss += float(preservation.detach())
+                retention_examples += int(eligible.sum())
+            if not bool(torch.isfinite(loss)):
+                raise ValueError(f"Non-finite loss at step {step}")
+            loss.backward()
+            step_loss += float(loss.detach())
+            del logits, loss, batch
+        gradients = [
+            parameter.grad for parameter in parameters if parameter.grad is not None
+        ]
+        if not gradients or not all(
+            bool(torch.isfinite(gradient).all()) for gradient in gradients
+        ):
+            raise ValueError(f"Missing or non-finite gradients at step {step}")
+        norm = torch.nn.utils.clip_grad_norm_(parameters, 1.0)
+        warmup = max(1, round(args.steps * 0.06))
+        rate = (
+            step / warmup
+            if step <= warmup
+            else 0.5
+            * (1 + math.cos(math.pi * (step - warmup) / max(1, args.steps - warmup)))
+        )
+        for group in optimizer.param_groups:
+            group["lr"] = group["initial_lr"] * rate
+        optimizer.step()
+        if training_order:
+            trace = training_order.record_step(step, microbatches)
+            with (args.output / "actual-training-order.jsonl").open("a") as stream:
+                stream.write(json.dumps(trace) + "\n")
+        torch.cuda.synchronize()
+        log = {
+            "step": step,
+            "loss": step_loss,
+            "cross_entropy_loss": step_ce_loss,
+            "retention_loss": step_retention_loss,
+            "retention_examples": retention_examples,
+            "grad_norm": float(norm),
+            "finite": True,
+            "gradient_tensors": len(gradients),
+            "max_tokens": max(lengths),
+            "mean_tokens": sum(lengths) / len(lengths),
+            "microbatches": microbatch_count,
+            "max_padded_tokens": max_padded_tokens,
+            "step_seconds": time.perf_counter() - start,
+            "peak_allocated_gib": torch.cuda.max_memory_allocated() / 2**30,
+            "peak_reserved_gib": torch.cuda.max_memory_reserved() / 2**30,
+        }
+        with (args.output / "steps.jsonl").open("a") as stream:
+            stream.write(json.dumps(log) + "\n")
+        if step == 1 or step % 10 == 0 or args.probe_only:
+            print(json.dumps(log), flush=True)
+        if (
+            not args.probe_only
+            and not args.train_only
+            and (step % args.eval_every == 0 or step == args.steps)
+        ):
+            evaluate_checkpoint(step)
+    if training_order:
+        order_receipt = training_order.finish()
+        order_receipt["actual_trace_sha256"] = hashlib.sha256(
+            (args.output / "actual-training-order.jsonl").read_bytes()
+        ).hexdigest()
+        (args.output / "training-order-completed.json").write_text(
+            json.dumps(order_receipt, indent=2) + "\n"
+        )
+    (args.output / "sampling-final.json").write_text(
+        json.dumps(sampling_exposure([row for row, _ in train], drawn), indent=2) + "\n"
+    )
+    if frozen_parameters is not None:
+        assert_frozen_parameters_preserved(frozen_parameters, model)
+    if not args.probe_only:
+        save_training_checkpoint(
+            model,
+            tokenizer,
+            args.output / ("last-adapter" if args.method == "lora" else "last-model"),
+            args.method,
+            args.base_id,
+            args.base_revision,
+        )
+        if best_key is None and not args.train_only:
+            raise ValueError(
+                "No checkpoint met the development FP budget; last checkpoint and "
+                "infeasibility receipts were retained without selecting a best checkpoint"
+            )
+    print(
+        json.dumps(
+            {
+                "event": "complete",
+                "best_dev_score": (
+                    best if not (args.probe_only or args.train_only) else None
+                ),
+            }
+        ),
+        flush=True,
+    )
+
+
+if __name__ == "__main__":
+    main()

reproducibility/sequence_repair/training_order.py

@@ -0,0 +1,134 @@
+"""Replay a complete, content-bound training order without changing optimization."""
+
+import hashlib
+import json
+from collections import Counter
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from .data import file_receipts
+
+
+def ordered_ids_sha256(ids):
+    """Canonical identity preserves UTF-8 IDs, duplicates, and their full order."""
+    return hashlib.sha256(
+        json.dumps(ids, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
+    ).hexdigest()
+
+
+def _unique_object(pairs):
+    result = {}
+    for key, value in pairs:
+        if key in result:
+            raise ValueError("Duplicate training-order field")
+        result[key] = value
+    return result
+
+
+@dataclass
+class TrainingOrder:
+    """A step can only complete once, with every planned example accounted for."""
+
+    indices: tuple[int, ...]
+    eligible_ids: tuple[str, ...]
+    steps: int
+    global_batch: int
+    receipt: dict
+    completed_steps: int = field(default=0, init=False)
+
+    def indices_for_step(self, step):
+        if (
+            type(step) is not int
+            or step != self.completed_steps + 1
+            or step > self.steps
+        ):
+            raise ValueError("Training order requires the next unconsumed step")
+        start = (step - 1) * self.global_batch
+        return list(self.indices[start : start + self.global_batch])
+
+    def record_step(self, step, microbatches):
+        planned = self.indices_for_step(step)
+        actual = [index for batch in microbatches for index in batch]
+        if Counter(planned) != Counter(actual):
+            raise ValueError("Microbatches dropped or duplicated planned examples")
+        self.completed_steps = step
+        return {
+            "step": step,
+            "planned_ids": [self.eligible_ids[index] for index in planned],
+            "microbatch_ids": [
+                [self.eligible_ids[index] for index in batch] for batch in microbatches
+            ],
+        }
+
+    def finish(self):
+        if self.completed_steps != self.steps:
+            raise ValueError("Training ended before consuming the complete order")
+        return {
+            **self.receipt,
+            "completed_steps": self.completed_steps,
+            "completed_draws": self.completed_steps * self.global_batch,
+        }
+
+
+def load_training_order(path, rows, train_paths, *, steps, global_batch):
+    """Validate exact source bytes, eligible IDs, budgets, and planned draws."""
+    raw = Path(path).read_bytes()
+    value = json.loads(raw, object_pairs_hook=_unique_object)
+    required = {
+        "version",
+        "train_files",
+        "eligible_ids_sha256",
+        "steps",
+        "global_batch",
+        "ids",
+        "ids_sha256",
+    }
+    if not isinstance(value, dict) or set(value) != required:
+        raise ValueError("Training order has unknown or missing fields")
+    if type(value["version"]) is not int or value["version"] != 1:
+        raise ValueError("Unsupported training-order version")
+    for name, expected in (("steps", steps), ("global_batch", global_batch)):
+        if (
+            type(expected) is not int
+            or expected <= 0
+            or type(value[name]) is not int
+            or value[name] != expected
+        ):
+            raise ValueError("Training order does not match the optimizer budget")
+    if value["train_files"] != file_receipts(train_paths):
+        raise ValueError("Training order does not match TRAIN file contents")
+    eligible = [row.get("id") for row in rows]
+    if (
+        not eligible
+        or any(not isinstance(key, str) or not key for key in eligible)
+        or len(set(eligible)) != len(eligible)
+    ):
+        raise ValueError("Eligible TRAIN IDs must be nonempty unique strings")
+    if value["eligible_ids_sha256"] != ordered_ids_sha256(eligible):
+        raise ValueError("Training order does not match eligible TRAIN IDs")
+    ids = value["ids"]
+    if (
+        not isinstance(ids, list)
+        or len(ids) != steps * global_batch
+        or any(not isinstance(key, str) or not key for key in ids)
+    ):
+        raise ValueError("Training order needs exactly steps times global-batch IDs")
+    if value["ids_sha256"] != ordered_ids_sha256(ids):
+        raise ValueError("Training-order ID sequence hash mismatch")
+    positions = {key: index for index, key in enumerate(eligible)}
+    if any(key not in positions for key in ids):
+        raise ValueError("Training order references unknown or filtered TRAIN IDs")
+    receipt = {
+        "file": Path(path).name,
+        "sha256": hashlib.sha256(raw).hexdigest(),
+        **{key: value[key] for key in sorted(required - {"ids"})},
+        "draws": len(ids),
+        "unique_rows": len(set(ids)),
+    }
+    return TrainingOrder(
+        tuple(positions[key] for key in ids),
+        tuple(eligible),
+        steps,
+        global_batch,
+        receipt,
+    )

reproducibility/source-files.json

@@ -0,0 +1,22 @@
+{
+  "runtime_mapping.py": "ec0c553ef625001c09c53a27659cc5162629e303bdef6a93c6091767f49b7fd9",
+  "retention.py": "3c8bc8aa60388508bc400bd68a6de2ed6ac18b48a24d3acd66d8e591223736eb",
+  "complete_adapter.py": "bc9a9365b1740b18d9cbe59d5250d8b5d214a0e133850c4921857c8e870ea13a",
+  "hydrate_annotations.py": "ea0c44b51afddbeb7dabf084db006eed545022ed819fa40682f4be846b207d62",
+  "export.py": "2f3ec1b038eae154ad32b28dc3cdc465e57e8d8b6fa842c2b67b0a66d3938fa9",
+  "task_head.py": "9d2a19743980c35e224c752a35e1e91a7e82dee3430f83b9008a4d01270a5a42",
+  "prepare_context.py": "9f9fe1ebd52e97d3e6901dc42bbf81f7e0703b7f2293703eabd89acd900a254a",
+  "corpus.py": "ba3d4f91607db085e55ae1c76701b9b1319fea5ee7dece9851d4ee7375bca108",
+  "prepare_retention.py": "56f48cc26b4889f93b43734e66ad8aaccd0046701519278b6b446d495f605ff4",
+  "initialize.py": "ade17c61ba94231a83e124e368bda4f468574cbb77689b6f71694c6489bb94a6",
+  "evaluate.py": "41617631838a1f72e33d402470a9bebd1ab0d7b42b80ac7e8c424a487068c562",
+  "selection.py": "764150454d150ad8971956faa2198d84380c8ee28f23ea63a650b07d045fc735",
+  "data.py": "84f922c20a2f3b029f7d363cf7f96f98cd239669f7777de099c63cfa1a1ef199",
+  "model.py": "c85ca12237e8735e20289e6e630803c59a6919dd473021d42291956c4b32f434",
+  "optimization.py": "71dde43dbe288cd8fab636a9266905f7898bac66ae155ded5158e37786492082",
+  "__init__.py": "04cd2a4c037dd1cc2070d57bea979cacb76d4660313a190fdf612007cb33b88e",
+  "context.py": "8e51031478aca4a825a77d057b2a66a9d09d80e42fb66ccbec3c527f3a891303",
+  "fetch_sources.py": "1abd30d33f456357553facdbde11116134835a087009c63c433faeccf8214638",
+  "train.py": "8b91020b3fcf93ae5925e9d15597d6f37ea3ca5d5c03745a2096b76be246e038",
+  "training_order.py": "88492bbb97b93743d637474595cffbc82e79d18bd115245adf60ffadea3bef44"
+}

reproducibility/training.json

@@ -0,0 +1,26 @@
+{
+  "steps": 100,
+  "gpu": "NVIDIA L4",
+  "torch": "2.8.0+cu128",
+  "cuda": "12.8",
+  "python": "3.12.10",
+  "selection": {
+    "step": 75,
+    "score": 1.0,
+    "metric": "macro-f1",
+    "test_used": false
+  },
+  "base_model": "vllm-sr/Vela-1.0-Encoder-307M",
+  "base_revision": "720ab37904ce15054068d429381bfae7549f00f6",
+  "method": "full; fresh head; final two encoder layers",
+  "batch_size": 2,
+  "accumulation": 2,
+  "max_length": 256,
+  "seed": 42,
+  "learning_rate": 2e-05,
+  "head_learning_rate": 0.0001,
+  "dataset_files": {
+    "train.jsonl": "69022836ee506922a74b3583088a242fcbedc5fd181c4dd7bcea4e7eca60f814",
+    "dev.jsonl": "5909fe10961b0c6500a5e7d6ac9efd99b114c16f3545d45cd04036a070dbd1c3"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,55 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

task-head.json

@@ -0,0 +1,49 @@
+{
+  "architecture": "ModernBertForSequenceClassification",
+  "modules_to_save": null,
+  "trainable_task_head_parameters": 592130,
+  "tensors": {
+    "head.dense.weight": {
+      "shape": [
+        768,
+        768
+      ],
+      "dtype": "torch.float32",
+      "numel": 589824,
+      "trainable": true,
+      "adapter_owned": false,
+      "sha256": "01a8bfc3e8af91a9901f5088ec0fb1cdb1f1b8c47526a5d5e2fe5a94d7383003"
+    },
+    "head.norm.weight": {
+      "shape": [
+        768
+      ],
+      "dtype": "torch.float32",
+      "numel": 768,
+      "trainable": true,
+      "adapter_owned": false,
+      "sha256": "fd989126d90b0d85ba4e8863c791567d526c8d6bed155fc05672489c906a91b5"
+    },
+    "classifier.weight": {
+      "shape": [
+        2,
+        768
+      ],
+      "dtype": "torch.float32",
+      "numel": 1536,
+      "trainable": true,
+      "adapter_owned": false,
+      "sha256": "84ca652ca12370d26a59fc4039312c112dd5079c834a04f43cc48c568b0873ee"
+    },
+    "classifier.bias": {
+      "shape": [
+        2
+      ],
+      "dtype": "torch.float32",
+      "numel": 2,
+      "trainable": true,
+      "adapter_owned": false,
+      "sha256": "eace58362ea99f963002fb0ba0a36ab6d2ddfc1a504668108853dd5dbf1cba92"
+    }
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
+size 34363188

tokenizer_config.json

@@ -0,0 +1,2018 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<2mass>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "6": {
+      "content": "[@BOS@]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "7": {
+      "content": "<unused0>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "8": {
+      "content": "<unused1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "9": {
+      "content": "<unused2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "10": {
+      "content": "<unused3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "11": {
+      "content": "<unused4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "12": {
+      "content": "<unused5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "13": {
+      "content": "<unused6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "14": {
+      "content": "<unused7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "15": {
+      "content": "<unused8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "16": {
+      "content": "<unused9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "17": {
+      "content": "<unused10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "18": {
+      "content": "<unused11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "19": {
+      "content": "<unused12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "20": {
+      "content": "<unused13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "21": {
+      "content": "<unused14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "22": {
+      "content": "<unused15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "23": {
+      "content": "<unused16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "24": {
+      "content": "<unused17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "25": {
+      "content": "<unused18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "26": {
+      "content": "<unused19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "27": {
+      "content": "<unused20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "28": {
+      "content": "<unused21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "29": {
+      "content": "<unused22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "30": {
+      "content": "<unused23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "31": {
+      "content": "<unused24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32": {
+      "content": "<unused25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "33": {
+      "content": "<unused26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "34": {
+      "content": "<unused27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "35": {
+      "content": "<unused28>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "36": {
+      "content": "<unused29>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "37": {
+      "content": "<unused30>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "38": {
+      "content": "<unused31>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "39": {
+      "content": "<unused32>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "40": {
+      "content": "<unused33>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "41": {
+      "content": "<unused34>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "42": {
+      "content": "<unused35>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "43": {
+      "content": "<unused36>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "44": {
+      "content": "<unused37>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "45": {
+      "content": "<unused38>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "46": {
+      "content": "<unused39>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "47": {
+      "content": "<unused40>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "48": {
+      "content": "<unused41>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "49": {
+      "content": "<unused42>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50": {
+      "content": "<unused43>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "51": {
+      "content": "<unused44>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "52": {
+      "content": "<unused45>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "53": {
+      "content": "<unused46>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "54": {
+      "content": "<unused47>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "55": {
+      "content": "<unused48>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "56": {
+      "content": "<unused49>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "57": {
+      "content": "<unused50>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "58": {
+      "content": "<unused51>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "59": {
+      "content": "<unused52>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "60": {
+      "content": "<unused53>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "61": {
+      "content": "<unused54>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "62": {
+      "content": "<unused55>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "63": {
+      "content": "<unused56>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "64": {
+      "content": "<unused57>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65": {
+      "content": "<unused58>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "66": {
+      "content": "<unused59>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "67": {
+      "content": "<unused60>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "68": {
+      "content": "<unused61>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "69": {
+      "content": "<unused62>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "70": {
+      "content": "<unused63>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "71": {
+      "content": "<unused64>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "72": {
+      "content": "<unused65>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "73": {
+      "content": "<unused66>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "74": {
+      "content": "<unused67>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "75": {
+      "content": "<unused68>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "76": {
+      "content": "<unused69>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "77": {
+      "content": "<unused70>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "78": {
+      "content": "<unused71>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79": {
+      "content": "<unused72>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "80": {
+      "content": "<unused73>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "81": {
+      "content": "<unused74>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "82": {
+      "content": "<unused75>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "83": {
+      "content": "<unused76>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "84": {
+      "content": "<unused77>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "85": {
+      "content": "<unused78>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "86": {
+      "content": "<unused79>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "87": {
+      "content": "<unused80>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "88": {
+      "content": "<unused81>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "89": {
+      "content": "<unused82>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "90": {
+      "content": "<unused83>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "91": {
+      "content": "<unused84>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "92": {
+      "content": "<unused85>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "93": {
+      "content": "<unused86>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "94": {
+      "content": "<unused87>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "95": {
+      "content": "<unused88>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "96": {
+      "content": "<unused89>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "97": {
+      "content": "<unused90>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "98": {
+      "content": "<unused91>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "99": {
+      "content": "<unused92>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100": {
+      "content": "<unused93>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "101": {
+      "content": "<unused94>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102": {
+      "content": "<unused95>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "103": {
+      "content": "<unused96>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "104": {
+      "content": "<unused97>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "105": {
+      "content": "<unused98>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "106": {
+      "content": "<start_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<end_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "109": {
+      "content": "\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "110": {
+      "content": "\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "111": {
+      "content": "\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "112": {
+      "content": "\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "113": {
+      "content": "\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "114": {
+      "content": "\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "115": {
+      "content": "\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "116": {
+      "content": "\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "117": {
+      "content": "\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "118": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "119": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "120": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "121": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "122": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "123": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "124": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "125": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "126": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "127": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "129": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "130": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "131": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "132": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "133": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "134": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "135": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "136": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "137": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "138": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "139": {
+      "content": "▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "140": {
+      "content": "▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "141": {
+      "content": "▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "142": {
+      "content": "▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "143": {
+      "content": "▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "144": {
+      "content": "▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "145": {
+      "content": "▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "146": {
+      "content": "▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "147": {
+      "content": "▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "148": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "149": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "150": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "152": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "153": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "154": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "155": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "156": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "157": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "158": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "159": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "160": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "161": {
+      "content": "▁▁▁���▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "162": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "163": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "164": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "165": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "166": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "167": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "168": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "169": {
+      "content": "<table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "170": {
+      "content": "<caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "171": {
+      "content": "<thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "172": {
+      "content": "<tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "173": {
+      "content": "<tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "174": {
+      "content": "<tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "175": {
+      "content": "<th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "176": {
+      "content": "<td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "177": {
+      "content": "</table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "178": {
+      "content": "</caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "179": {
+      "content": "</thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "180": {
+      "content": "</tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "181": {
+      "content": "</tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "182": {
+      "content": "</tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "183": {
+      "content": "</th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "184": {
+      "content": "</td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "185": {
+      "content": "<h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "186": {
+      "content": "<h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "187": {
+      "content": "<h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "188": {
+      "content": "<h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "189": {
+      "content": "<h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "190": {
+      "content": "<h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "191": {
+      "content": "<blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "192": {
+      "content": "</h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "193": {
+      "content": "</h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "194": {
+      "content": "</h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "195": {
+      "content": "</h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "196": {
+      "content": "</h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "197": {
+      "content": "</h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "198": {
+      "content": "</blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "199": {
+      "content": "<strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200": {
+      "content": "<em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "201": {
+      "content": "<b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "202": {
+      "content": "<i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "203": {
+      "content": "<u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "204": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "205": {
+      "content": "<sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "206": {
+      "content": "<sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "207": {
+      "content": "<code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "208": {
+      "content": "</strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "209": {
+      "content": "</em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "210": {
+      "content": "</b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "211": {
+      "content": "</i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "212": {
+      "content": "</u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "213": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "214": {
+      "content": "</sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "215": {
+      "content": "</sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "216": {
+      "content": "</code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255968": {
+      "content": "[toxicity=0]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255969": {
+      "content": "\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255970": {
+      "content": "\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255971": {
+      "content": "\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255972": {
+      "content": "\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255973": {
+      "content": "\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255974": {
+      "content": "\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255975": {
+      "content": "\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255976": {
+      "content": "\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255977": {
+      "content": "\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255978": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255979": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255980": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255981": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255982": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255983": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255984": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255985": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255986": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255987": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255988": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255989": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255990": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255991": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255992": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255993": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255994": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255995": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255996": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255997": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255998": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255999": {
+      "content": "<unused99>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 32768,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

training-origin.json

@@ -0,0 +1,5 @@
+{
+  "method": "full",
+  "base_model": "vllm-sr/Vela-1.0-Encoder-307M",
+  "base_revision": "720ab37904ce15054068d429381bfae7549f00f6"
+}