Publish Laya Mini CPU research artifacts, A/B results and limitations

Original, control A and experimental B; CPU exports, optional FP32 reference checkpoints, text-free evaluation records and bilingual documentation. B remains experimental: composite criterion failed; FINAL not evaluated.

BENCHMARKS.md

@@ -0,0 +1,100 @@
+# Benchmark methodology and complete decision
+
+These measurements describe this experiment, not standard full-dataset leaderboard submissions. There is deliberately no `model-index` claiming otherwise.
+
+## Controlled A/B, recorded DEV predictions
+
+There are 156 new text/premise families per domain, two held-out description styles and all cyclic option rotations: four for AG News, three for SNLI. Keys are opaque. Total: 312 texts, 2,184 requests. Repeated variants of a text are not independent observations.
+
+Original historical partitions, DEV and reserved FINAL are separated by source/text/premise checks. This does not establish absence of pretraining exposure in BERT or Laya, or of all semantic paraphrases. Description pools are disjoint strings but remain lexically similar.
+
+Evaluation used the same local CPU, four threads, batches of eight sorted by length, PyTorch FP32 and ONNX Runtime INT8 for both A and B. Confidence intervals use paired percentile bootstrap over families, 10,000 replicates, seed 20260926. They are conditional on these two trained models, not confidence intervals over retraining runs.
+
+| Domain / precision | A | B | B−A, percentage points | Paired 95% CI |
+| --- | ---: | ---: | ---: | ---: |
+| AG News FP32 | 57.05% | 89.02% | +31.97 | [26.92, 37.02] |
+| AG News INT8 | 54.89% | 88.46% | +33.57 | [28.76, 38.38] |
+| SNLI FP32 | 53.42% | 71.05% | +17.63 | [13.78, 21.37] |
+| SNLI INT8 | 50.11% | 65.60% | +15.49 | [11.97, 19.02] |
+
+### Preregistered gates, not reinterpreted after the result
+
+1. At least +10 accuracy points per domain and precision, with lower paired CI >0: all four pass.
+2. B order instability at most half A: AG FP32/INT8 and SNLI FP32 pass; **SNLI INT8 fails**.
+3. Fixed-validation lower global accuracy-difference CI ≥−2 points and no task point estimate below −4: both precisions pass.
+4. Upper CI for change in FP32/INT8 disagreement ≤+2 points, on each DEV domain and fixed validation overall: all pass.
+5. CPU payload 80–90 decimal MB: met.
+
+**Composite result: FAIL. No checkpoint promotion and no FINAL evaluation.** No threshold, checkpoint or quantization setting was changed after inspecting DEV.
+
+Order instability is the fraction of text/style pairs whose canonical class changes across rotations, averaged over styles. On SNLI INT8, A has 181 unstable cells out of 312, B has 105. Half of A allows at most 90 whole unstable cells, not 105. Accuracy is checked separately because a consistently wrong model can be stable.
+
+### SNLI by description style, INT8
+
+| Style | A accuracy | B accuracy | Gain | B instability |
+| --- | ---: | ---: | ---: | ---: |
+| Short | 67.09% | 73.08% | +5.98 points | 21.79% |
+| Long | 33.12% | 58.12% | +25.00 points | 45.51% |
+
+The aggregate +10-point accuracy gate was per domain/precision, not per style. It would be incorrect to claim that every formulation improved by ten points.
+
+### Quantization is not harmless
+
+| Domain | A FP32/INT8 disagreement | B disagreement | Change, points | Paired 95% CI |
+| --- | ---: | ---: | ---: | ---: |
+| AG News | 18.59% | 2.08% | −16.51 | [−19.23, −13.86] |
+| SNLI | 22.76% | 11.00% | −11.75 | [−15.17, −8.33] |
+
+On SNLI's 936 requests, A changes 213 predictions: 95 correct→wrong, 64 wrong→correct, 54 wrong→different-wrong. B changes 103: 73 correct→wrong, 22 wrong→correct, 8 wrong→different-wrong. B loses 51 net correct answers, **5.45 points**, versus A's 31 / 3.31 points.
+
+Thus B has fewer changes but a larger accuracy loss. B's instability also rises from 16.03% in FP32 to 33.65% in INT8. Do not use the disagreement reduction as a blanket quantization-robustness claim.
+
+## Historical fixed validation used in A/B
+
+2,000 requests / 1,980 families, previously consulted during development. Same batch/CPU conditions as DEV.
+
+| Precision | A | B | Difference, points | Paired 95% CI |
+| --- | ---: | ---: | ---: | ---: |
+| FP32 | 76.15% | 77.20% | +1.05 | [+0.20, +1.90] |
+| INT8 | 76.05% | 75.65% | −0.40 | [−1.31, +0.55] |
+
+The worst task point regression is −1.00 point. The overall increase in FP32/INT8 changes is +0.15 point, CI [−0.90, +1.20]. Non-regression guards pass on this development set; this is not a blind confirmation or a demonstrated inter-seed improvement.
+
+## Earlier original-model evaluations — NOT A/B scores
+
+The original model was selected by historical validation. Earlier test results had already been seen before increasing training data, so the end-to-end development process was not entirely blind. COPA was excluded from training and checkpoint selection.
+
+| Original INT8, historical test | Student | Laya teacher |
+| --- | ---: | ---: |
+| AG News, 600 requests | 91.00% | 91.17% |
+| BoolQ, 600 | 69.00% | 83.33% |
+| Emotion, 600 | 91.17% | 59.00% |
+| SNLI, 600 | 78.17% | 76.33% |
+| SST5, 600 | 45.00% | 33.33% |
+| Balanced total, 3,000 | 74.87% | 68.63% |
+| COPA out of domain, 400 | 49.25% | 69.75% |
+
+These teacher annotations belong to the historical pipeline, not a fresh remeasurement using the A/B FP32 teacher cache. Supervised labels supplement distillation, so in-domain teacher-score comparisons do not establish general student superiority. COPA random-choice reference is 50%. An old descriptive per-answer-vocabulary majority oracle of 100% was not a usable predictor and is omitted from the publication summaries.
+
+Original historical FP32/INT8 overall test accuracy: 75.00% / 74.87%. Batch size 16, four CPU threads, unlike the newer A/B batch-eight evaluation. Neither test-3000 nor COPA was rerun for A or B.
+
+## CPU benchmark
+
+Linux x86-64, Intel Core i7-1260P, ONNX Runtime 1.24.4, two threads, batch one, four options, three warm-up calls per shape and 30 timed repetitions. Synthetic tokens, model already loaded, tokenization/network/loading excluded. Peak RSS is over all four sequence lengths.
+
+| Variant | 128-token median | p95 | Peak RSS |
+| --- | ---: | ---: | ---: |
+| Original, current session | 21.36 ms | 33.24 ms | 195.74 MiB |
+| Experimental B | 21.55 ms | 28.83 ms | 196.00 MiB |
+
+Control A's median is 21.43 ms; its full measurement record is in `benchmarks/cpu/control-a.json`. These observations do not establish a speed difference. Raw individual timings/IQR were not retained. Background load, clock and power states were not controlled sufficiently to explain the original's older 10.79 ms figure; that separate-session number must not be used to claim a 2× slowdown or speedup.
+
+```bash
+python benchmark_cpu.py models/experimental-b --threads 2 --repeats 30 --output my-cpu.json
+```
+
+The benchmark uses Unix `resource`; Windows is not supported by this measurement script as written. The inference runtime itself does not depend on `resource`, but Windows/ARM64 compatibility remains untested. Dynamic quantization kernels and predictions can vary by CPU and batch composition.
+
+## What the evidence does not establish
+
+Reliable decisions on new tasks, arbitrary paraphrases, broad out-of-distribution robustness, calibrated confidence, performance on every CPU, bitwise training reproducibility, variance across seeds, or game-playing ability. None of those claims follows from small model files, software test success or a successful ONNX export.

LICENSE_NOTES.md

@@ -0,0 +1,37 @@
+# Source-specific rights and unresolved redistribution questions
+
+**There is no single permissive license asserted for this mixed-source research release.** The model-card `license: other` and descriptive license name point to this notice; they do not create a new license, clear upstream rights or authorize every commercial use.
+
+The owner requested a public research release with these reservations explicit. Public accessibility does not resolve copyright, database rights, contract terms or whether particular dataset terms apply to derived weights. This is a source inventory and limitation notice, not legal advice or a legal determination of derivative-work status.
+
+## Teacher and backbone
+
+| Component | Pinned revision | Declared model-card license |
+| --- | --- | --- |
+| [Laya / convaiinnovations](https://huggingface.co/convaiinnovations/laya) | `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851` | Apache-2.0 |
+| [Google compact BERT](https://huggingface.co/google/bert_uncased_L-6_H-512_A-8) | `dd53ec6ca9d05e0a91b309c4e137f31988888071` | Apache-2.0 |
+
+The Laya Python package used was 0.3.20; its [source repository](https://github.com/NandhaKishorM/laya) should also be reviewed for applicable notices and terms. The Apache-2.0 declarations are not a separate warranty about the teacher's pretraining data, outputs or every downstream distribution scenario. No broader permission for this student is inferred solely from those tags. A copy of the [Apache License 2.0 text](licenses/Apache-2.0.txt) is included for components declaring it; it is not a blanket relicensing of this repository or the mixed data.
+
+## Training and evaluation datasets
+
+| Source | Pinned revision | Evidence and limitation |
+| --- | --- | --- |
+| [AG News / fancyzhx](https://huggingface.co/datasets/fancyzhx/ag_news) | `eb185aade064a813bc0b7f42de02595523103ca4` | Pinned card says `unknown` and describes research / non-commercial activity; no blanket commercial or redistribution clearance asserted. |
+| [SST5 / SetFit](https://huggingface.co/datasets/SetFit/sst5) | `e51bdcd8cd3a30da231967c1a249ba59361279a3` | No license declaration in the pinned card; review-text rights not resolved here. |
+| [BoolQ / Google](https://huggingface.co/datasets/google/boolq) | `35b264d03638db9f4ce671b711558bf7ff0f80d5` | Card declares [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/); preserve attribution and applicable obligations. |
+| [SNLI / Stanford NLP](https://huggingface.co/datasets/stanfordnlp/snli) | `cdb5c3d5eed6ead6e5a341c8e56e669bb666725b` | Card declares [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/); preserve attribution and applicable obligations. |
+| [Emotion / DAIR.AI](https://huggingface.co/datasets/dair-ai/emotion) | `cab853a1dbdf4c42c2b3ef2173804746df8825fe` | Card says the dataset should be used for educational and research purposes only. |
+| [COPA through SuperGLUE](https://huggingface.co/datasets/aps/super_glue) | `3de24cf8022e94f4ee4b9d55a6f539891524d646` | Historical evaluation only, not fitting; SuperGLUE refers to original task licenses, with COPA-specific rights unresolved in this project. |
+
+Exact pinned-card links, original homepages, source partitions and provenance are recorded in `research/corpus-manifest.json`. No source corpus or teacher-cache contents are redistributed. Historical reports that contained example-specific answer vocabularies have been reduced to aggregate summaries before publication.
+
+## Practical restrictions
+
+- Treat this as experimental research material, not a cleared commercial product or a universally licensed foundation model.
+- Verify source-specific rights and obtain any necessary permission before redistribution or commercial deployment. A public download is not itself that permission.
+- Retain source attribution and this notice when sharing permitted derivatives. Applicable upstream licenses/terms are not superseded by this notice.
+- No license is invented for third-party corpus text or any right the maintainer does not hold.
+- Research source files have no newly added blanket license grant in this snapshot. Review applicable upstream code licenses as well as data/model conditions before reuse.
+
+The scientific claims are similarly limited: all variants have `quality_validated: false`, probabilities are uncalibrated, and the latest composite acceptance criterion failed. Publication must not be represented as certification of safety, generality, legal clearance or endorsement by upstream authors.

PUBLICATION_CHECKS.json

@@ -0,0 +1,55 @@
+{
+  "research_snapshot": "2026-09-26",
+  "package_scope": "Original, control A and experimental B; CPU exports and optional FP32 reference checkpoints.",
+  "software_unit_tests_passed": 57,
+  "hugging_face_model_card_validation": "passed",
+  "recomputed_statistics": {
+    "dev_rows": 2184,
+    "fixed_validation_rows": 2000,
+    "robustness_report_exactly_reproduced": true,
+    "fixed_validation_report_exactly_reproduced": true,
+    "new_inference_for_recomputation": false
+  },
+  "runtime": {
+    "runtime_checks": {
+      "original": {
+        "payload_bytes": 87157026,
+        "CPUExecutionProvider": true,
+        "known_example_answers": {
+          "example-canonical-request.json": "sports",
+          "example-news-request.json": "science"
+        }
+      },
+      "control-a": {
+        "payload_bytes": 87155068,
+        "CPUExecutionProvider": true,
+        "known_example_answers": {
+          "example-canonical-request.json": "sports",
+          "example-news-request.json": "science"
+        }
+      },
+      "experimental-b": {
+        "payload_bytes": 87155068,
+        "CPUExecutionProvider": true,
+        "known_example_answers": {
+          "example-canonical-request.json": "sports",
+          "example-news-request.json": "sports"
+        }
+      }
+    },
+    "torch_imported": false,
+    "note": "Known interface checks, not independent quality evaluations."
+  },
+  "quality_validated": false,
+  "dev_composite_passed": false,
+  "final_evaluated": false,
+  "privacy": {
+    "raw_dataset_text_in_recorded_decisions": false,
+    "source_family_ids_replaced_preserving_sort_order": true,
+    "credentials_included": false,
+    "private_conversation_transcripts_included": false,
+    "account_balances_included": false,
+    "final_input_files_included": false
+  },
+  "limitations": "Passing software checks and model-card validation are not evidence of general decision reliability or legal clearance."
+}

README.md

@@ -0,0 +1,154 @@
+---
+language:
+- en
+license: other
+license_name: research-use-with-unresolved-upstream-rights
+license_link: https://huggingface.co/VynoDePal/laya-mini-cpu-research/blob/main/LICENSE_NOTES.md
+library_name: onnx
+pipeline_tag: text-classification
+base_model: google/bert_uncased_L-6_H-512_A-8
+base_model_relation: finetune
+datasets:
+- fancyzhx/ag_news
+- google/boolq
+- dair-ai/emotion
+- stanfordnlp/snli
+- SetFit/sst5
+tags:
+- bert
+- onnx
+- int8
+- cpu
+- knowledge-distillation
+- multiple-choice
+- experimental
+- research
+inference: false
+---
+
+# Laya Mini CPU — research checkpoints and controlled A/B results
+
+**English decision models, approximately 87.16 MB each. Research artifacts, not a validated general-purpose decision agent.**
+
+[Français](README_FR.md) · [Benchmarks](BENCHMARKS.md) · [Research notes](RESEARCH.md) · [Reproducibility](REPRODUCIBILITY.md) · [Rights and restrictions](LICENSE_NOTES.md)
+
+Maintained by [VynoDePal](https://huggingface.co/VynoDePal). Research snapshot: 26 September 2026. This is an independent student-model experiment using [Laya](https://huggingface.co/convaiinnovations/laya) as teacher, not an official Laya release or an affiliation claim.
+
+## Read this before choosing a checkpoint
+
+**Experimental B improves accuracy on the tested close reformulations, but fails the preregistered composite acceptance criterion. It is NOT promoted as a replacement for the original.** None of these variants is validated for reliable general decisions. The reserved FINAL set was not evaluated.
+
+The main experiment covers **312 texts / 2,184 correlated requests, two domains, two description styles and one training seed**. It is not an official full-dataset AG News or SNLI leaderboard evaluation.
+
+| DEV domain / precision | Control A accuracy | Experimental B accuracy | A order instability | B order instability | Accuracy + stability gate |
+| --- | ---: | ---: | ---: | ---: | --- |
+| AG News FP32 | 57.05% | 89.02% | 61.54% | 3.53% | Pass |
+| AG News INT8 | 54.89% | 88.46% | 65.06% | 6.41% | Pass |
+| SNLI FP32 | 53.42% | 71.05% | 48.40% | 16.03% | Pass |
+| **SNLI INT8** | **50.11%** | **65.60%** | **58.01%** | **33.65%** | **FAIL: B must be ≤29.01% unstable** |
+
+- The SNLI INT8 **short-style gain is only 5.98 percentage points**, not the 15.49-point average across styles.
+- B loses **5.45 points on SNLI from FP32 to INT8**, despite fewer FP32/INT8 prediction changes than A. Fewer changes do not establish better quantization robustness.
+- Confidence intervals resample text/premise families, not individual rotations. They do **not** measure variation across training seeds.
+- `experiment-status.json` records `quality_validated: false`, `dev_passed: false`, the failed gate and `final_evaluated: false`.
+
+## Model variants
+
+| Directory | Role | CPU model payload | FP32 checkpoint |
+| --- | --- | ---: | ---: |
+| `models/original/` | Historical reference, before the A/B continuation | 87,157,026 bytes | 152,909,884 bytes |
+| `models/control-a/` | Two more epochs using fixed descriptions | 87,155,068 bytes | 152,909,884 bytes |
+| `models/experimental-b/` | Same continuation, with controlled description augmentation | 87,155,068 bytes | 152,909,884 bytes |
+
+Payload means **ONNX INT8 + tokenizer + manifest**; 1 MB = 1,000,000 bytes. Each variant satisfies the 80–90 MB model-file target. The whole repository is larger because it contains multiple variants, optional FP32 reference checkpoints, code and evidence. Python, dependencies and working RAM are extra.
+
+There is deliberately **no default model at the repository root**. Choose a variant explicitly. Original is a historical reference, not a claim that it is generally reliable. Checkpoints under `checkpoints/` are for inspection, comparison and possible re-export; downloading them is unnecessary for CPU INT8 inference.
+
+## Download only the CPU variant you need
+
+Install the Hugging Face CLI if needed (`python -m pip install huggingface_hub`), then:
+
+```bash
+hf download VynoDePal/laya-mini-cpu-research \
+  --include 'models/experimental-b/*' runtime.py schema.py requirements-runtime.txt 'examples/*' \
+  --local-dir laya-mini
+cd laya-mini
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install -r requirements-runtime.txt
+python runtime.py models/experimental-b examples/example-canonical-request.json --threads 2
+```
+
+Use `models/original` or `models/control-a` to select another variant. The custom runtime uses only `CPUExecutionProvider`; it does not require PyTorch, a GPU, paid inference or network access after installation. It is **not** a Transformers `pipeline()` checkpoint, an Ollama/GGUF model or a hosted chat endpoint.
+
+```python
+import json
+from pathlib import Path
+from runtime import DecisionSession
+
+session = DecisionSession("models/experimental-b", threads=2)
+request = json.loads(Path("examples/example-canonical-request.json").read_text())
+result = session.predict(request["state"], request["questions"])
+print(result["answers"])
+```
+
+Keep the session loaded between requests. Inputs, instructions and option descriptions should be in English. The two included tennis examples are previously known interface checks, **not held-out evidence**. B answers `sports` on both in the recorded smoke test; the original fails the long-description version. Do not infer general reliability from either anecdote.
+
+### Interface and limits
+
+- `choice`: chooses a key from a dictionary of option descriptions.
+- `score`: returns the expectation over an ordered list of levels.
+- `noul`: returns a soft boolean score.
+- Software supports 2–20 options; training covered only 2–6.
+- Maximum total context: 512 WordPiece tokens. Instructions/options have a 192-token budget; oversized question definitions are rejected. Long states may be truncated; check `truncated_state`.
+- Probabilities are **not calibrated**. A high `max_probability` is not a factual correctness guarantee.
+- No free-form generation, vision, screen reading or automatic action execution. A game-playing system would still need an adapter and game-specific evaluation.
+
+## CPU performance
+
+Intel Core i7-1260P, Linux x86-64, ONNX Runtime 1.24.4, two threads, one question / four options, warm model, 30 repetitions. Synthetic inputs; excludes tokenization and loading.
+
+| B input tokens | Median latency |
+| --- | ---: |
+| 64 | 11.08 ms |
+| 128 | 21.55 ms |
+| 256 | 44.83 ms |
+| 512 | 111.79 ms |
+
+B's 128-token p95 is **28.83 ms**, and peak process RSS is **196 MiB** across the benchmark. Original / A / B 128-token medians measured in the same session are **21.36 / 21.43 / 21.55 ms**. No meaningful speed ranking is established. The original's older 10.79 ms result came from a different session and is not a matched comparison; the difference is unexplained.
+
+Raw per-repetition timings were not retained, so an IQR is unavailable. Windows, ARM64 and old/low-end CPUs were not validated. **“Runs easily on every CPU” is not established.** More settings and p95 values are in [BENCHMARKS.md](BENCHMARKS.md).
+
+## Architecture and training
+
+- **38,223,873 parameters**: a pretrained six-layer, 512-wide BERT backbone, an additional decision layer, question-type embeddings and an option-marker scoring head.
+- Backbone: `google/bert_uncased_L-6_H-512_A-8`, revision `dd53ec6ca9d05e0a91b309c4e137f31988888071`.
+- Teacher: `convaiinnovations/laya`, revision `55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851`, package `laya==0.3.20`.
+- Loss: teacher KL at temperature 1 + 2 × cross-entropy against actual labels. This is **distillation plus supervised learning**, not direct quantization of the original Laya weights.
+- A/B both continue the same original checkpoint for exactly two epochs / 5,048 steps, seed 42, batch 32, identical schedules, shared padding and last-checkpoint selection. There is no DEV-based checkpoint selection.
+- B replaces exactly 12,000 AG News and 12,000 SNLI descriptions, 50% of each training pool. All other fields/tasks remain unchanged. Both arms share **teacher targets on fixed prompts**, not teacher re-annotations of the variants.
+- Student: CUDA BF16 with FP32 matmul mode `high` (TF32 allowed; kernels not traced). Shared teacher cache: FP32 CUDA without autocast/TF32. Bitwise training reproducibility is not established.
+- Export: ONNX opset 18; dynamic INT8 linear-weight quantization, per channel, `reduce_range=True`; embeddings remain FP32. The reduced range uses seven effective weight bits in INT8 storage. No separate calibration corpus was used.
+
+The 80,758 training rows comprise 24,000 AG News, 24,000 SNLI, 15,966 Emotion, 8,533 SST5 and 8,259 BoolQ examples. The original model's development history and different historical evaluations are documented separately; **old test/COPA scores must not be attributed to A or B**.
+
+## Evidence and reproducibility boundary
+
+- [A/B report](benchmarks/ab/dev-report.json): all acceptance criteria, per-style diagnostics and confidence intervals.
+- [Frozen protocol](benchmarks/ab/protocol.json): hyperparameters and cryptographic commitments to inputs, including unevaluated FINAL files.
+- [Text-free recorded decisions](benchmarks/ab/decisions-no-text.jsonl): 2,184 DEV + 2,000 fixed-validation rows, allowing aggregate statistics to be recomputed without rerunning inference.
+- [CPU reports](benchmarks/cpu): measurements for all three variants in the same session.
+- [Historical results](benchmarks/history): clearly scoped to the original or the pre-A/B audit.
+- [Research sources](research/src), checkpoint metadata and `SHA256SUMS`: code/provenance and integrity evidence.
+
+No raw training corpus, teacher-cache contents, FINAL requests/descriptions, credentials, account balances or private conversation transcripts are included. The final-set preparation script is omitted. Published hashes/code are not a cryptographic guarantee that nobody could reconstruct held-out public examples; **the claim is that FINAL was not evaluated in this experiment**.
+
+Recomputing the recorded statistics is reproducible here. **Training from scratch and rerunning the original DEV inputs are not fully self-contained**, because their data/cache artifacts are not redistributed. FP32 checkpoints do not remove that limitation. See [REPRODUCIBILITY.md](REPRODUCIBILITY.md).
+
+## Rights, safety and status
+
+This public research snapshot is released at the owner's request **with unresolved upstream rights explicitly disclosed**. The `other` license tag is a pointer to source-specific restrictions, not a newly granted unified license. The Apache-2.0 model cards of the teacher/backbone do not by themselves clear the mixed training-data rights. Emotion is research/education only, AG News has unknown/non-commercial wording, and the pinned SST5 card declares no license. See [LICENSE_NOTES.md](LICENSE_NOTES.md) before redistribution or commercial use.
+
+Do not use these experimental models for consequential decisions or irreversible actions without independent validation and safeguards. The size target is met; reliable general decision intelligence is **not**.
+
+The engineering/review process used Codex and Claude Code assistance. Claude reviewed supplied excerpts and measurements, did not execute these tests, and did not provide independent human peer review. The reported total Vast.ai credit decrease was approximately **US$0.36**, including approximately **US$0.125 for A/B**; these are observations, not final invoices or a reproducible cost quote. Training instances were removed. No new paid job is required to use these CPU models.

README_FR.md

@@ -0,0 +1,61 @@
+# Laya Mini CPU — guide français
+
+[Carte complète en anglais](README.md) · [Benchmarks](BENCHMARKS.md) · [Recherche](RESEARCH.md) · [Reproductibilité](REPRODUCIBILITY.md) · [Droits](LICENSE_NOTES.md)
+
+Ce dépôt de recherche contient **trois variantes réellement entraînées**, d'environ **87,16 Mo chacune** pour l'inférence CPU. Il ne s'agit ni d'une simple quantification des poids originaux de Laya, ni d'un chatbot, ni d'un agent de jeu prêt à l'emploi.
+
+## Quel modèle choisir ?
+
+- `models/original/` : référence historique avant la dernière expérience, pas une garantie de fiabilité générale.
+- `models/control-a/` : contrôle ayant reçu deux epochs supplémentaires avec descriptions fixes.
+- `models/experimental-b/` : même continuation avec remplacement contrôlé de descriptions AG News/SNLI.
+
+**B progresse sur les formulations proches étudiées, mais ne passe pas tous les critères préenregistrés.** Il reste expérimental et n'est pas promu comme remplacement de l'original. Aucun modèle n'est sélectionné par défaut à la racine du dépôt.
+
+| DEV en INT8 | A | B |
+| --- | ---: | ---: |
+| AG News, catégories d'actualité | 54,89 % | 88,46 % |
+| SNLI, inférence entre phrases | 50,11 % | 65,60 % |
+
+Ces résultats concernent **312 textes, 2 184 requêtes corrélées, deux domaines et une seule graine**. Ils ne constituent pas les scores officiels sur les jeux de test complets.
+
+La garde échouée est la stabilité SNLI après quantification : **33,65 %** des couples texte/style changent de réponse selon la rotation des options, contre un maximum admissible de **29,01 %**. Sur le style SNLI court, le gain n'est que de 5,98 points. B perd 5,45 points SNLI entre FP32 et INT8. Le jeu FINAL réservé n'a pas été évalué ; aucun réglage supplémentaire n'a été fait sur ce DEV consommé.
+
+## Installation minimale
+
+Installer le CLI Hugging Face si nécessaire (`python -m pip install huggingface_hub`), puis télécharger uniquement la variante souhaitée :
+
+```bash
+hf download VynoDePal/laya-mini-cpu-research \
+  --include 'models/experimental-b/*' runtime.py schema.py requirements-runtime.txt 'examples/*' \
+  --local-dir laya-mini
+cd laya-mini
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install -r requirements-runtime.txt
+python runtime.py models/experimental-b examples/example-canonical-request.json --threads 2
+```
+
+L'inférence est locale, sur CPU, sans PyTorch ni accès réseau après installation. Les entrées doivent être en **anglais**. Le modèle choisit parmi des options ou produit un score ; il ne génère pas de texte libre. Les probabilités ne sont pas calibrées.
+
+Le logiciel accepte 2 à 20 options, mais l'entraînement n'en couvre que 2 à 6. Le contexte est limité à 512 tokens WordPiece. Vérifier `truncated_state` pour les observations longues. Les exemples de tennis sont déjà connus : leur réussite éventuelle n'est pas un benchmark indépendant.
+
+## Taille et performances
+
+- Modèle CPU B : **87 155 068 octets**, tokenizer et manifeste inclus, soit 87,155 Mo décimaux / 83,118 Mio.
+- Sur Intel Core i7-1260P, Linux x86-64, deux threads : **21,55 ms médians à 128 tokens**, modèle chaud, hors tokenisation et chargement ; p95 28,83 ms.
+- Pic RSS du benchmark : **196 Mio**. La RAM et les dépendances dépassent donc la taille des fichiers.
+- A et l'original remesurés dans la même session ont des latences proches. L'ancien chiffre de 10,79 ms provient d'une autre session, non comparable directement.
+- Windows, ARM64 et les CPU anciens ne sont pas testés. « Fonctionner aisément sur n'importe quel CPU » n'est pas garanti.
+
+Les gros checkpoints FP32 de `checkpoints/` sont facultatifs et servent à l'inspection/comparaison/réexport. Le dépôt complet contient plusieurs modèles : sa taille totale dépasse volontairement 90 Mo, mais chaque variante CPU respecte la cible.
+
+## Recherche et limites de la publication
+
+Le dépôt conserve les gains **et** le résultat négatif, les statistiques par style, le protocole, les hashes, le code et les mesures CPU. Les résultats historiques du test à 3 000 exemples et de COPA restent attribués uniquement à l'original : ils n'ont pas été réévalués pour A/B.
+
+Les décisions enregistrées sont publiées sans les textes des corpus, avec identifiants de familles anonymisés conservant l'ordre utile au bootstrap. Elles permettent de recalculer les tableaux et intervalles. En revanche, le réentraînement complet et la nouvelle inférence DEV ne sont pas autonomes sans les données/cache non redistribués.
+
+Le professeur Laya et le backbone BERT déclarent Apache-2.0, mais les données ont des droits hétérogènes, parfois incomplets ou limités à la recherche. **Une publication publique n'équivaut pas à une autorisation commerciale ni à une clarification juridique de ces droits.** Lire les [réserves de licence](LICENSE_NOTES.md). Aucun corpus brut, contenu FINAL, secret ou solde de compte n'est téléversé.
+
+Le coût Vast.ai cumulé constaté est d'environ 0,36 USD, dont 0,125 USD pour A/B ; ce n'est pas une facture définitive. Les instances ont été détruites. L'objectif de taille est atteint ; la fiabilité générale recherchée ne l'est pas encore.

REPRODUCIBILITY.md

@@ -0,0 +1,76 @@
+# Reproducibility and artifact boundaries
+
+## What is included
+
+- Three ONNX CPU variants, with their original tokenizer and manifest.
+- Three FP32 reference checkpoints and training metadata. These are optional for ordinary inference and are not 80–90 MB files.
+- The custom runtime and CPU benchmark code.
+- The frozen A/B protocol, checkpoint provenance, historical corpus manifest, aggregate reports and hashes.
+- Recorded A/B DEV/fixed-validation decisions with dataset text and private identifiers removed.
+- Selected research source snapshots. Training scripts are historical reference material, not a complete one-command reproduction bundle.
+
+## What is not included
+
+Raw corpus text, teacher-cache contents, full DEV input prompts, reserved FINAL text/descriptions, the final-set preparation script, GPU orchestration, credentials, account balances, instance IDs and private conversations. Only FINAL file hashes are published as commitments. Generic algorithms and public source datasets do not provide cryptographic secrecy of the holdout; the verifiable experimental status is **FINAL not evaluated**.
+
+This release can reproduce **statistics from recorded decisions** and run new CPU inference. It cannot independently recreate the whole training experiment or rerun its exact DEV prompts without the omitted data/cache. Public dataset revisions and the historical corpus manifest help describe the source material but do not establish reuse rights.
+
+## Verify downloaded files
+
+For a complete snapshot:
+
+```bash
+sha256sum -c SHA256SUMS
+```
+
+If downloading only one model, filter the checksum list to those files instead of expecting the whole-repository check to pass. Model payloads include only `model.int8.onnx`, `tokenizer.json` and `manifest.json` in one variant directory. Source weights remain unchanged from the measured experiment.
+
+Checkpoints use the same custom architecture in `research/src/model.py`. They are not standalone `AutoModel.from_pretrained()` directories. A checkpoint must be paired with its architecture, tokenizer and question serialization. The matching tokenizer is under the corresponding `models/` directory.
+
+## Recompute recorded A/B statistics, without inference
+
+The local reference used **NumPy 2.5.3**. Install `requirements-stats.txt` in a suitable Python 3.12 environment. No PyTorch or GPU is needed for this operation.
+
+Run this from the repository root:
+
+```bash
+python - <<'PY'
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path("research/src").resolve()))
+from ab_evaluate import fixed_summary, robustness_summary
+
+rows = [json.loads(line) for line in Path("benchmarks/ab/decisions-no-text.jsonl").open()]
+dev = [row for row in rows if row["style"] != "fixed"]
+fixed = [row for row in rows if row["style"] == "fixed"]
+expected = json.loads(Path("benchmarks/ab/dev-report.json").read_text())
+robustness = robustness_summary(dev)
+validation = fixed_summary(fixed)
+assert robustness == expected["robustness"]
+assert validation == expected["fixed_validation"]
+assert not (robustness["passed"] and validation["passed"])
+print("Recorded statistics reproduced; the composite criterion fails as reported.")
+PY
+```
+
+The 4,184 records comprise 2,184 DEV and 2,000 historical fixed-validation requests, each with the four A/B × FP32/INT8 predictions/probability vectors. Family identifiers are replaced with fixed-width ordinal IDs preserving the original sorted-family order. Thus both dependence groups and bootstrap index ordering are retained without exposing original identifiers.
+
+For DEV, `canonical_label` and probability indices are canonical domain classes. For historical fixed validation, they follow each request's original option order; they are suitable for per-request correctness and flip comparisons, **not** a pooled semantic-class confusion matrix without the omitted option definitions. Do not treat every row as independent or interpret these anonymized predictions as a redistributed text dataset.
+
+The expected original prediction-file SHA256 is `608003d576107260205f758b012ba9a42020c28d1d4d35b682930903dd3b4e7b`. The text-free derivative necessarily has a different hash, `a982420e45e2e16ece356619196cf49bfe36fe8adac2b208e4c9763b3d2b3f99`. The original predictions themselves are not published.
+
+## Inference and export reproducibility
+
+Use `requirements-runtime.txt` and the README commands for ONNX CPU inference. Dynamic quantization can produce different results on other CPUs or different batches. The published predictions are the reference observations, not a promise of bitwise equality on every environment.
+
+Remote export used PyTorch 2.8.0+cu126, Transformers 4.57.6, ONNX 1.20.1 and ONNX Runtime 1.24.4. Local A/B quality evaluation used ONNX Runtime 1.24.4 and NumPy 2.5.3. Dynamic per-channel INT8 used `reduce_range=True` and no separate calibration dataset. FP32 embeddings were retained.
+
+Re-exporting the same checkpoint can depend on library/platform versions; byte-identical ONNX output is not guaranteed. Synthetic-shape FP32 ONNX parity checks had maximum logit errors of about 2.68×10⁻⁶ for A and 3.76×10⁻⁶ for B; this verifies export mechanics, not task accuracy.
+
+## Research-source limitations
+
+`research/src/` contains copies of the actual model, loss, batching, cache-validation, training and evaluation logic. Some historical entry points refer to the original workspace layout or require omitted corpus/configuration files. They are supplied for inspection, not advertised as runnable full training recipes from this snapshot.
+
+No training, teacher annotation or FINAL inference should be triggered to reproduce the published statistical table. A new experiment requires its own explicit protocol, data access/rights review and compute budget. The old development sets cannot be rebranded as fresh tests.

RESEARCH.md

@@ -0,0 +1,53 @@
+# Research notes: from a size target to an invariance experiment
+
+## Objective and scope
+
+The requested target was an English decision model with an **80–90 decimal MB inference payload**, usable on CPU. Reliable general decisions were the aspiration, not an achieved result. Directly shrinking the teacher by ordinary INT8 quantization was not the chosen solution: a smaller pretrained BERT student was trained with teacher distributions and ground-truth labels.
+
+All reported numbers are project measurements unless explicitly described as claims in a cited source. This document is an engineering research note, not a peer-reviewed paper.
+
+## Development chronology
+
+1. **Architecture feasibility.** A 38.2M-parameter student was exported to about 87 MB using INT8 linear matrices and FP32 embeddings. Prototype size checks were not quality evidence.
+2. **Historical training.** A first 24,000-example run was followed by an 80,758-example run. The second was selected on validation; its epoch-6 checkpoint is `original`. The historical test improved, but COPA remained 49.25%, close to the 50% random reference. This did not meet the generality objective.
+3. **Wording diagnostic.** A known tennis sentence exposed sensitivity to longer option descriptions and option ordering. Correct marker positions, absence of truncation and FP32 export parity were checked; the error also existed before INT8. One sentence and its rotations were not treated as an independent benchmark.
+4. **Expanded development audit.** 60 AG News + 60 SNLI texts, four styles, two key families and cyclic rotations produced 3,360 requests per student precision. A bounded teacher audit completed 24 of these texts / 672 requests, not the entire 120-text pool. Teacher results were stronger in some conditions but also showed wording sensitivity. Aggregate evidence is in `benchmarks/history/pre-ab-robustness-audit.json`.
+5. **Preregistered A/B continuation.** New DEV/FINAL families and disjoint description pools were reserved. A and B shared start weights, optimizer/loss, teacher targets, batches/padding and selection rule. B alone changed exactly half the AG/SNLI descriptions. The protocol and files were hashed before training.
+6. **Result.** Localized reformulation accuracy improved substantially. One preregistered stability guard still failed in SNLI INT8. The experiment ended with B experimental, FINAL unevaluated, original preserved and no post-result tuning.
+
+## Why the contrast matters
+
+A also receives the two extra epochs. Therefore B versus A measures the chosen description-augmentation contrast more cleanly than B versus the earlier original checkpoint would. The controls do not eliminate single-seed stochastic training uncertainty.
+
+Both arms learn from the same Laya distributions on **fixed inputs**. This promotes invariance to semantically matched reformulations; it is not imitation of Laya's outputs on each reformulated prompt. Ground-truth cross-entropy remains present, so distillation is only part of the supervision.
+
+The cache contains 80,758 annotations. On 64 pilot comparisons, decoded probability argmax agreed and maximum normalized-probability discrepancy was approximately 9.884×10⁻⁵. Native `choice` tie-breaking after rounding was not separately verified. No teacher truncation occurred; its longest encoded input was 383 tokens. A warning about `choice` with 11+ options lies outside this experiment's 2–6-option data.
+
+Student training used BF16 and allowed TF32 for eligible FP32 matmuls; teacher annotations used strict FP32 CUDA. A seed is not a guarantee of deterministic kernels or bitwise reproducibility.
+
+## Main finding and unresolved issue
+
+Augmentation produces positive measured changes on close, held-out formulations within these two domains. It does **not** establish arbitrary-prompt robustness or unseen-task intelligence. The remaining problem is particularly visible in the interaction between SNLI descriptions, option rotations and INT8.
+
+Fewer FP32/INT8 prediction differences can coexist with a larger net accuracy drop. For B, 73 SNLI decisions change from correct to wrong and only 22 recover; A has 95 losses and 64 recoveries. The directional balance, not only the total disagreement rate, matters.
+
+The next research direction would investigate this quantization-related loss under a **new, explicit protocol**. DEV is now consumed; it cannot honestly become a fresh test for variants chosen using its errors. The reserved FINAL must not be turned into a tuning set. No such next training run is included here.
+
+## Literature and primary documentation consulted
+
+- [GLiClass: Generalist Lightweight Model for Sequence Classification Tasks](https://arxiv.org/html/2508.07662v1), Stepanov et al., 2025: text/label scoring and deliberate training variation motivated possible alternatives. No GLiClass weights were trained or benchmarked in this project.
+- [GLiClass small v1.0 card/config](https://huggingface.co/knowledgator/gliclass-small-v1.0): “small” did not imply an 80–90 MB artifact. The consulted configuration's 128,003×768 word embedding table alone has 98,306,304 parameters. This is a size calculation, not an observed quantized-model benchmark.
+- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://huggingface.co/papers/2110.08207), Sanh et al., 2021: the consulted abstract motivates prompt diversity and task-held-out evaluation. It does not predict performance for this much smaller encoder. Only metadata/abstract were reviewed for this source.
+- [ONNX Runtime quantization](https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html): quantization can reduce accuracy; dynamic quantization is commonly used for transformers; per-channel and reduced-range configurations have hardware-dependent tradeoffs.
+- [PyTorch 2.8 reproducibility notes](https://raw.githubusercontent.com/pytorch/pytorch/v2.8.0/docs/source/notes/randomness.rst): reproducibility across versions, platforms and CPU/GPU is not guaranteed, even with matching seeds.
+- [Laya](https://huggingface.co/convaiinnovations/laya) and its [source repository](https://github.com/NandhaKishorM/laya): teacher interface, architecture and reported capabilities. The student is not assumed to inherit all teacher capabilities or calibration.
+
+More precise consulted sections and timestamps are in `research/references.json` and `research/runtime-references-checked.json`. Reading a source is not reproducing its experiments.
+
+## Review and research integrity
+
+The owner requested recurring discussion with Claude Code. Actual exchanges covered semantic equivalence, cache/label ordering, experiment controls, quantization and interpretation. Suggestions were checked against local code and saved predictions; the final review supported non-promotion. One timed-out request produced no review and is not counted as approval.
+
+Claude's conversational connection received excerpts and measurements, not direct workspace/shell access. It did not execute the tests. Two AI reviews are not independent human semantic validation or peer review. Private conversation IDs and transcripts are deliberately not published.
+
+The trained exports are real, not random probe weights or padded files. The local pipeline had 57 passing unit tests, but software correctness checks are not certification of decision quality. Checkpoint sizes, hashes, mechanical export checks, research conclusions and quality gates are kept separate.

SHA256SUMS

@@ -0,0 +1,58 @@
+fe712bdfec1d7dffc809139de81571f5a3a87847b8907e9a866a92dfca8e3369  BENCHMARKS.md
+cc992b2c269c6e24b027c57c2c8ce02121748c9f2e17da8bade9721e4f2016ee  LICENSE_NOTES.md
+b0cb8a5965b88db04ea8e6ae141e0d880a081a3391ccbf1bf0f933c889b5f3a5  PUBLICATION_CHECKS.json
+e2ed8ff3e09f941b917497b145fe45fcb230fb76eeee8b895198b58329d4dbfe  README.md
+079f860db190333db8dba1954dfad88c60fbda1a05728f38003d94ab9fbe9609  README_FR.md
+79852a1ae240ff20c00984c789349fb1fe3180063defee448f1de8cf33b62cf3  REPRODUCIBILITY.md
+4425e580d8f75e439c72668c1839f22dad215b495154f7f01ba958efe8c1730b  RESEARCH.md
+d6eb989eb736559dcac886a47c7be217a11c88f48a62728baff88954f78fec05  benchmark_cpu.py
+a982420e45e2e16ece356619196cf49bfe36fe8adac2b208e4c9763b3d2b3f99  benchmarks/ab/decisions-no-text.jsonl
+50880c10e5167034f151765662c7b28c188ea99d1c9cbcc6e2db9c63a7599a77  benchmarks/ab/dev-report.json
+143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366  benchmarks/ab/protocol.json
+9749e9eaa665a549d9c570d0c48fdfdd83be7c7e3ccdfc2d33ef8048d90f6ad4  benchmarks/cpu/control-a.json
+a5ea41cefb34c958d506f9e1103ef65718c53a7c80dbe744f531ba0c88ca525d  benchmarks/cpu/experimental-b.json
+ed2e2cbae4e233909e00186bafa2abe2543e8de84ffed677f8823e2f7b67a8d0  benchmarks/cpu/original-current.json
+61c50584cc5948781d922598db8e43fbfdd2adf0c05d9d0f3ce1aad600dd0e40  benchmarks/history/original-copa-400-summary.json
+dc9eddd55ecf3cf960600252dd6c13efa8fe00c6c0020aa3fa65b92acd6b3ed9  benchmarks/history/original-cpu-earlier-session.json
+3a492329c63c6ee335855a6dfc11a2bee1ebfae282c85d681d090de4a48dc5ea  benchmarks/history/original-test-3000-summary.json
+79b3a029073007e6cd4d02f0de5883294b86f0b157644c9d0f6f885005fe36f9  benchmarks/history/pre-ab-robustness-audit.json
+1da45906bfc7f513867875c5c5902a4909593f949d623b1eda4c25051cd2c53e  checkpoints/control-a/model.safetensors
+e1f24303468c4e9f3222a2cfd42651accf99ed5b4943f3d14f130486d4ea4491  checkpoints/control-a/training.json
+7dfc87cdeeb3b6885d54fce02beea64f55d7d93679ed1cc9850c533c3888e36c  checkpoints/experimental-b/model.safetensors
+d2638c9dd2f3f8dae44670c0f1c39acf707f7fe76038ed4dc705df3fb883c7c8  checkpoints/experimental-b/training.json
+5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008  checkpoints/original/model.safetensors
+3890e0ff94e4c0e0be926b9cba7d4fb7f2fa3a18e0b5957d47d11fc1f410de49  checkpoints/original/training.json
+3fb2dc9676522962dd1884ac2bef8ff56e1813bc7eed4317d96c805533e7b438  examples/example-canonical-request.json
+70333d48b9503ea7e3a8bce234d2ce3673a005168f4ae6ecbbfe88147335beb2  examples/example-news-request.json
+a3156a3526dec6985b7d7ccf9a69e25c8ae6680fc183aca03c328d234e46fc01  experiment-status.json
+cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  licenses/Apache-2.0.txt
+d89ad68d5052361601017b76706847e1a3a8a8f1f91cc3f0c7d701bf77f8e725  models/control-a/manifest.json
+30c633d3bd6742714f3198c5e6d3176c05b312e967c6da2523ad6f6ff25e43b6  models/control-a/model.int8.onnx
+19a57dcc34c9491ae5049f69d49db9081ecf2556f728b31f24aafadfdc2025de  models/control-a/tokenizer.json
+d9ae2b6a6b628d8a91d07bc44a9501b2cd4ae7b1e12efbd52f5d63c69813653a  models/experimental-b/manifest.json
+d0aa20e2a2b2e2b054c4db668b98452362ce58968f24645a9f0f07543c0e1e31  models/experimental-b/model.int8.onnx
+19a57dcc34c9491ae5049f69d49db9081ecf2556f728b31f24aafadfdc2025de  models/experimental-b/tokenizer.json
+319241a7af9e5328e337e73ed26f2b283e0167fd08d2169286dacd8175b7f498  models/original/manifest.json
+18b45a1a7979c4f98e93a4521bb59d0b1690c898670b44d484346b4849ab4fda  models/original/model.int8.onnx
+19a57dcc34c9491ae5049f69d49db9081ecf2556f728b31f24aafadfdc2025de  models/original/tokenizer.json
+2b96ffca45b3009f89bb3029f093223fdf289f28659343f0b6622bff043bf8a1  requirements-runtime.txt
+5c12d85e7c08038d1dd9ffbf8119d12ce808885230a47e2bf65dcd9f114659d5  requirements-stats.txt
+329afcca5672d548697a2948468641e61f266d4f5fc5515037e6b1d50c17b3f5  research/corpus-manifest.json
+5bce0fa7fd07ef2f57d0bc3649e24830c211941c3253c35cc8b806f5ffb79c9a  research/exports/control-a.json
+f635cfbab2de7b8a0cf40f2e20f067434da5b0f6e956608b906c4d2709affec8  research/exports/experimental-b.json
+a0fc4d6bac89f02a15588323e28cecdee6f846c4ec3b37cd519fbaaab1e6fb26  research/original-selection.json
+5fa8d32d2a1833b60ebc7179d9f11f8d58f8b2067434c7181bb9d5a0cb2f540f  research/references.json
+5439d40e6fbba568aa35b12e9a22c438f5b3c8570c81da7fc58376f4537d3200  research/runtime-references-checked.json
+0e0addbe20762637ccb8cd2d29d06351e8b6c2a9e571064e360ef676c557a464  research/src/ab_evaluate.py
+10d3ee98200d36ea79c60d4d2ef95e49e44c741f2747921b5d3b03f51cf8f0f1  research/src/ab_experiment.py
+93bc78d052d2b9bc6b36a42d0e039911fcb364d080b27da32fa81e2e2340c04b  research/src/ab_train.py
+9863df1f565fc9a4ca4bf9b4fea7c705d4698d609ee37fe153b370c2801cc79b  research/src/audit_robustness.py
+8d0cee025d53511162e6ca7991727cc733c26b75f8f09b2982d2a8047d85d503  research/src/audit_teacher.py
+f75f03f0df896cb0c0b32f95a932a68616026323b821b91543696c7c4166f177  research/src/build_corpus.py
+a518cab84b9b02885a7a97f369acefd5b9279a2a56b21962156ae6f6d313197f  research/src/evaluate_model.py
+e3470d5ba85fd97a4cb7804688043ad178e171d5dd25a9137118998d08a87e3f  research/src/model.py
+9b46fc8df3977fdfea2a22176aadfaea73b545b66cecd0b5454eb510917816a5  research/src/pipeline.py
+f9106f00c93d95946100ac13df374598abf9db0d8fa76c04f1d99964e3cd17b9  research/src/requirements-build.txt
+9db14f28d71221f7743a6e3602e5d6f07790cf11f4f215da181f050eecde0c10  research/src/schema.py
+3e059a7b71af78ffd6cb3c2fb99f3ecd5f0a29d9e7c7e278c1328731aa3b44eb  runtime.py
+9db14f28d71221f7743a6e3602e5d6f07790cf11f4f215da181f050eecde0c10  schema.py

benchmark_cpu.py

@@ -0,0 +1,71 @@
+import argparse
+import gc
+import json
+import platform
+import resource
+import time
+from pathlib import Path
+
+import numpy as np
+import onnxruntime as ort
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("model_directory")
+    parser.add_argument("--threads", type=int, default=2)
+    parser.add_argument("--repeats", type=int, default=20)
+    parser.add_argument("--output", required=True)
+    arguments = parser.parse_args()
+    if arguments.threads < 1 or arguments.repeats < 5:
+        parser.error("Use positive threads and at least five repeats")
+    directory = Path(arguments.model_directory)
+    manifest = json.loads((directory / "manifest.json").read_text())
+    settings = ort.SessionOptions()
+    settings.intra_op_num_threads = arguments.threads
+    settings.inter_op_num_threads = 1
+    settings.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
+    session = ort.InferenceSession(str(directory / "model.int8.onnx"), settings, providers=["CPUExecutionProvider"])
+    generator = np.random.default_rng(42)
+    measurements = []
+    for length in [64, 128, 256, 512]:
+        inputs = {
+            "input_ids": generator.integers(105, 10000, size=(1, length), dtype=np.int64),
+            "attention_mask": np.ones((1, length), dtype=np.int64),
+            "marker_pos": np.array([[4, 8, 12, 16]], dtype=np.int64),
+            "marker_mask": np.ones((1, 4), dtype=np.bool_),
+            "qtype": np.zeros(1, dtype=np.int64),
+        }
+        for _ in range(3):
+            session.run(None, inputs)
+        timings = []
+        for _ in range(arguments.repeats):
+            started = time.perf_counter()
+            session.run(None, inputs)
+            timings.append((time.perf_counter() - started) * 1000)
+        measurements.append({
+            "tokens": length, "batch_size": 1, "options": 4,
+            "p50_ms": float(np.median(timings)),
+            "p95_ms": float(np.percentile(timings, 95)),
+        })
+    peak_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    report = {
+        "architecture": manifest["architecture"],
+        "trained": manifest["trained"],
+        "machine": platform.machine(),
+        "platform": platform.platform(),
+        "onnxruntime": ort.__version__,
+        "threads": arguments.threads,
+        "repeats": arguments.repeats,
+        "peak_process_RSS_MiB": peak_rss / (1024 * 1024 if platform.system() == "Darwin" else 1024),
+        "measurements": measurements,
+        "limitations": "Synthetic inputs; warm model; excludes tokenization, network and loading. Not a quality evaluation or a guarantee for another CPU.",
+    }
+    Path(arguments.output).write_text(json.dumps(report, indent=2) + "\n")
+    print(json.dumps(report, indent=2))
+    del session
+    gc.collect()
+
+
+if __name__ == "__main__":
+    main()

benchmarks/ab/dev-report.json

@@ -0,0 +1,625 @@
+{
+  "protocol_sha256": "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366",
+  "split": "dev",
+  "texts_per_domain": 156,
+  "variant_requests": 2184,
+  "fixed_requests": 2000,
+  "passed": false,
+  "robustness": {
+    "domains": {
+      "ag_news": {
+        "fp32": {
+          "A_accuracy": 0.5705128205128205,
+          "B_accuracy": 0.8902243589743589,
+          "gain": {
+            "delta": 0.31971153846153844,
+            "ci95": [
+              0.2692307692307692,
+              0.3701923076923077
+            ],
+            "clusters": 156
+          },
+          "A_order_instability": 0.6153846153846154,
+          "B_order_instability": 0.035256410256410256,
+          "passed": true
+        },
+        "int8": {
+          "A_accuracy": 0.5488782051282052,
+          "B_accuracy": 0.8846153846153846,
+          "gain": {
+            "delta": 0.33573717948717946,
+            "ci95": [
+              0.28764022435897435,
+              0.38383413461538435
+            ],
+            "clusters": 156
+          },
+          "A_order_instability": 0.6506410256410257,
+          "B_order_instability": 0.0641025641025641,
+          "passed": true
+        },
+        "quantization": {
+          "A_flip_rate": 0.1858974358974359,
+          "B_flip_rate": 0.020833333333333332,
+          "change": {
+            "delta": -0.16506410256410256,
+            "ci95": [
+              -0.19230769230769232,
+              -0.13862179487179488
+            ],
+            "clusters": 156
+          },
+          "passed": true
+        },
+        "styles": {
+          "0": {
+            "A_fp32": {
+              "accuracy": 0.7435897435897436,
+              "order_instability": 0.391025641025641,
+              "confusion": [
+                [
+                  114,
+                  2,
+                  23,
+                  17
+                ],
+                [
+                  0,
+                  100,
+                  33,
+                  23
+                ],
+                [
+                  4,
+                  5,
+                  121,
+                  26
+                ],
+                [
+                  3,
+                  1,
+                  23,
+                  129
+                ]
+              ]
+            },
+            "A_int8": {
+              "accuracy": 0.717948717948718,
+              "order_instability": 0.4358974358974359,
+              "confusion": [
+                [
+                  110,
+                  3,
+                  29,
+                  14
+                ],
+                [
+                  0,
+                  105,
+                  28,
+                  23
+                ],
+                [
+                  4,
+                  6,
+                  122,
+                  24
+                ],
+                [
+                  2,
+                  3,
+                  40,
+                  111
+                ]
+              ]
+            },
+            "B_fp32": {
+              "accuracy": 0.8974358974358975,
+              "order_instability": 0.019230769230769232,
+              "confusion": [
+                [
+                  141,
+                  0,
+                  12,
+                  3
+                ],
+                [
+                  0,
+                  156,
+                  0,
+                  0
+                ],
+                [
+                  2,
+                  10,
+                  127,
+                  17
+                ],
+                [
+                  4,
+                  0,
+                  16,
+                  136
+                ]
+              ]
+            },
+            "B_int8": {
+              "accuracy": 0.8878205128205128,
+              "order_instability": 0.04487179487179487,
+              "confusion": [
+                [
+                  140,
+                  0,
+                  13,
+                  3
+                ],
+                [
+                  0,
+                  156,
+                  0,
+                  0
+                ],
+                [
+                  3,
+                  11,
+                  123,
+                  19
+                ],
+                [
+                  5,
+                  0,
+                  16,
+                  135
+                ]
+              ]
+            }
+          },
+          "1": {
+            "A_fp32": {
+              "accuracy": 0.3974358974358974,
+              "order_instability": 0.8397435897435898,
+              "confusion": [
+                [
+                  35,
+                  19,
+                  51,
+                  51
+                ],
+                [
+                  13,
+                  38,
+                  49,
+                  56
+                ],
+                [
+                  13,
+                  10,
+                  77,
+                  56
+                ],
+                [
+                  5,
+                  9,
+                  44,
+                  98
+                ]
+              ]
+            },
+            "A_int8": {
+              "accuracy": 0.3798076923076923,
+              "order_instability": 0.8653846153846154,
+              "confusion": [
+                [
+                  37,
+                  16,
+                  34,
+                  69
+                ],
+                [
+                  16,
+                  35,
+                  37,
+                  68
+                ],
+                [
+                  12,
+                  8,
+                  65,
+                  71
+                ],
+                [
+                  9,
+                  11,
+                  36,
+                  100
+                ]
+              ]
+            },
+            "B_fp32": {
+              "accuracy": 0.8830128205128205,
+              "order_instability": 0.05128205128205128,
+              "confusion": [
+                [
+                  138,
+                  0,
+                  14,
+                  4
+                ],
+                [
+                  1,
+                  155,
+                  0,
+                  0
+                ],
+                [
+                  5,
+                  8,
+                  121,
+                  22
+                ],
+                [
+                  4,
+                  0,
+                  15,
+                  137
+                ]
+              ]
+            },
+            "B_int8": {
+              "accuracy": 0.8814102564102564,
+              "order_instability": 0.08333333333333333,
+              "confusion": [
+                [
+                  134,
+                  0,
+                  16,
+                  6
+                ],
+                [
+                  0,
+                  156,
+                  0,
+                  0
+                ],
+                [
+                  5,
+                  7,
+                  122,
+                  22
+                ],
+                [
+                  2,
+                  0,
+                  16,
+                  138
+                ]
+              ]
+            }
+          }
+        }
+      },
+      "snli": {
+        "fp32": {
+          "A_accuracy": 0.5341880341880342,
+          "B_accuracy": 0.7104700854700855,
+          "gain": {
+            "delta": 0.1762820512820513,
+            "ci95": [
+              0.13782051282051283,
+              0.21367521367521367
+            ],
+            "clusters": 156
+          },
+          "A_order_instability": 0.483974358974359,
+          "B_order_instability": 0.16025641025641027,
+          "passed": true
+        },
+        "int8": {
+          "A_accuracy": 0.5010683760683761,
+          "B_accuracy": 0.655982905982906,
+          "gain": {
+            "delta": 0.15491452991452992,
+            "ci95": [
+              0.11965811965811966,
+              0.19017094017094016
+            ],
+            "clusters": 156
+          },
+          "A_order_instability": 0.5801282051282052,
+          "B_order_instability": 0.33653846153846156,
+          "passed": false
+        },
+        "quantization": {
+          "A_flip_rate": 0.22756410256410256,
+          "B_flip_rate": 0.11004273504273504,
+          "change": {
+            "delta": -0.11752136752136752,
+            "ci95": [
+              -0.1517094017094017,
+              -0.08333333333333333
+            ],
+            "clusters": 156
+          },
+          "passed": true
+        },
+        "styles": {
+          "0": {
+            "A_fp32": {
+              "accuracy": 0.7200854700854701,
+              "order_instability": 0.1346153846153846,
+              "confusion": [
+                [
+                  133,
+                  6,
+                  17
+                ],
+                [
+                  24,
+                  63,
+                  69
+                ],
+                [
+                  4,
+                  11,
+                  141
+                ]
+              ]
+            },
+            "A_int8": {
+              "accuracy": 0.6709401709401709,
+              "order_instability": 0.23076923076923078,
+              "confusion": [
+                [
+                  133,
+                  4,
+                  19
+                ],
+                [
+                  39,
+                  40,
+                  77
+                ],
+                [
+                  3,
+                  12,
+                  141
+                ]
+              ]
+            },
+            "B_fp32": {
+              "accuracy": 0.7692307692307693,
+              "order_instability": 0.0641025641025641,
+              "confusion": [
+                [
+                  135,
+                  11,
+                  10
+                ],
+                [
+                  20,
+                  86,
+                  50
+                ],
+                [
+                  4,
+                  13,
+                  139
+                ]
+              ]
+            },
+            "B_int8": {
+              "accuracy": 0.7307692307692307,
+              "order_instability": 0.21794871794871795,
+              "confusion": [
+                [
+                  138,
+                  7,
+                  11
+                ],
+                [
+                  31,
+                  70,
+                  55
+                ],
+                [
+                  6,
+                  16,
+                  134
+                ]
+              ]
+            }
+          },
+          "1": {
+            "A_fp32": {
+              "accuracy": 0.3482905982905983,
+              "order_instability": 0.8333333333333334,
+              "confusion": [
+                [
+                  36,
+                  34,
+                  86
+                ],
+                [
+                  29,
+                  43,
+                  84
+                ],
+                [
+                  28,
+                  44,
+                  84
+                ]
+              ]
+            },
+            "A_int8": {
+              "accuracy": 0.3311965811965812,
+              "order_instability": 0.9294871794871795,
+              "confusion": [
+                [
+                  45,
+                  34,
+                  77
+                ],
+                [
+                  44,
+                  51,
+                  61
+                ],
+                [
+                  50,
+                  47,
+                  59
+                ]
+              ]
+            },
+            "B_fp32": {
+              "accuracy": 0.6517094017094017,
+              "order_instability": 0.2564102564102564,
+              "confusion": [
+                [
+                  100,
+                  29,
+                  27
+                ],
+                [
+                  23,
+                  66,
+                  67
+                ],
+                [
+                  3,
+                  14,
+                  139
+                ]
+              ]
+            },
+            "B_int8": {
+              "accuracy": 0.5811965811965812,
+              "order_instability": 0.4551282051282051,
+              "confusion": [
+                [
+                  97,
+                  30,
+                  29
+                ],
+                [
+                  42,
+                  43,
+                  71
+                ],
+                [
+                  6,
+                  18,
+                  132
+                ]
+              ]
+            }
+          }
+        }
+      }
+    },
+    "passed": false
+  },
+  "fixed_validation": {
+    "passed": true,
+    "fp32": {
+      "A_accuracy": 0.7615,
+      "B_accuracy": 0.772,
+      "gain": {
+        "delta": 0.0105,
+        "ci95": [
+          0.002006999422753962,
+          0.01899050474762619
+        ],
+        "clusters": 1980
+      },
+      "tasks": {
+        "ag_news": {
+          "A_accuracy": 0.94,
+          "B_accuracy": 0.935,
+          "delta": -0.005
+        },
+        "boolq": {
+          "A_accuracy": 0.6625,
+          "B_accuracy": 0.6875,
+          "delta": 0.025
+        },
+        "emotion": {
+          "A_accuracy": 0.9175,
+          "B_accuracy": 0.92,
+          "delta": 0.0025
+        },
+        "snli": {
+          "A_accuracy": 0.805,
+          "B_accuracy": 0.825,
+          "delta": 0.02
+        },
+        "sst5": {
+          "A_accuracy": 0.4825,
+          "B_accuracy": 0.4925,
+          "delta": 0.01
+        }
+      },
+      "passed": true
+    },
+    "int8": {
+      "A_accuracy": 0.7605,
+      "B_accuracy": 0.7565,
+      "gain": {
+        "delta": -0.004,
+        "ci95": [
+          -0.013058928494809557,
+          0.0054917623564653024
+        ],
+        "clusters": 1980
+      },
+      "tasks": {
+        "ag_news": {
+          "A_accuracy": 0.9325,
+          "B_accuracy": 0.93,
+          "delta": -0.0025
+        },
+        "boolq": {
+          "A_accuracy": 0.665,
+          "B_accuracy": 0.655,
+          "delta": -0.01
+        },
+        "emotion": {
+          "A_accuracy": 0.9075,
+          "B_accuracy": 0.9125,
+          "delta": 0.005
+        },
+        "snli": {
+          "A_accuracy": 0.8025,
+          "B_accuracy": 0.795,
+          "delta": -0.0075
+        },
+        "sst5": {
+          "A_accuracy": 0.495,
+          "B_accuracy": 0.49,
+          "delta": -0.005
+        }
+      },
+      "passed": true
+    },
+    "quantization": {
+      "change": {
+        "delta": 0.0015,
+        "ci95": [
+          -0.008995614692653673,
+          0.011970074812967581
+        ],
+        "clusters": 1980
+      },
+      "passed": true
+    }
+  },
+  "cpu_evaluation_seconds": {
+    "A": 225.80804631299543,
+    "B": 250.6982191290008
+  },
+  "prediction_sha256": "608003d576107260205f758b012ba9a42020c28d1d4d35b682930903dd3b4e7b",
+  "quality_validated_for_general_decisions": false,
+  "instability_definition": "Fraction of text/style pairs whose canonical prediction changes across cyclic rotations; average over styles",
+  "uncertainty": "Paired family-cluster percentile bootstrap, 10000 replicates, seed20260926; not uncertainty over training seeds"
+}

benchmarks/ab/protocol.json

@@ -0,0 +1,82 @@
+{
+  "purpose": "Single-seed controlled continuation; no claim of general decision intelligence",
+  "start_sha256": "5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008",
+  "teacher_revision": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+  "seed": 42,
+  "epochs": 2,
+  "batch_size": 32,
+  "steps_per_arm": 5048,
+  "encoder_learning_rate": 2e-05,
+  "head_learning_rate": 0.0001,
+  "weight_decay": 0.01,
+  "warmup_fraction": 0.06,
+  "label_weight": 2,
+  "temperature": 1,
+  "selection": "terminal checkpoint after exactly epoch 2, identically for A and B",
+  "augmentation": "static exact half of each AG/SNLI pool, no filtering by teacher/student correctness",
+  "variant_counts": {
+    "ag_news:3": 3041,
+    "snli:0": 2939,
+    "snli:2": 3045,
+    "ag_news:0": 2950,
+    "snli:1": 3020,
+    "ag_news:1": 2970,
+    "ag_news:2": 3039,
+    "snli:3": 2996
+  },
+  "shared_targets": "one Laya FP32 CUDA cache on fixed A inputs",
+  "batch_schedule": "fixed A lengths; shared padding maxima over both arms; seed42+epoch",
+  "student_precision": "BF16 CUDA if supported, both arms identical",
+  "evaluation": "156 new unique families per domain and split, two held-out styles, all cyclic rotations, opaque keys",
+  "bootstrap": {
+    "seed": 20260926,
+    "replicates": 10000,
+    "cluster": "text/premise",
+    "interval": "two-sided percentile 95%"
+  },
+  "criteria": {
+    "dev_gain_each_domain_each_precision_min": 0.1,
+    "dev_gain_ci_lower_strict_min": 0,
+    "order_instability_B_to_A_max": 0.5,
+    "fixed_global_delta_ci_lower_min": -0.02,
+    "fixed_each_task_point_delta_min": -0.04,
+    "flip_increase_ci_upper_max": 0.02,
+    "flip_scope": "each DEV domain and fixed-validation global, paired family bootstrap",
+    "final": "only after all DEV gates pass; same robustness gates on reserved FINAL, no tuning",
+    "model_bytes_range": [
+      80000000,
+      90000000
+    ]
+  },
+  "budget": {
+    "total_task_cap_usd": 10,
+    "both_arms_cap_usd": 3,
+    "local_deadline_hours": 2,
+    "provider_cleanup_slack_hours": 2,
+    "bandwidth_reserve_gb_each_direction": 20
+  },
+  "train_rows": 80758,
+  "maximum_sequence_lengths": {
+    "A": 362,
+    "B": 393
+  },
+  "files": {
+    "dev-texts.jsonl": "e327763b227b5118f57b04be4da1ffb19aab2986e002041ce81d43b64218c104",
+    "test.jsonl": "47cc633fd7e1ae38a52c271d2ca36ded1d9002e9a7f84a62b1dc7b3c1adf3cb9",
+    "dev-requests.jsonl": "636db7df8f9d5fe5b5a4ab32851174ed7c02846c772c2c6decde69ba26d21461",
+    "final-requests.jsonl": "bb923215006f19601979e34832dfaaa1177a719210945e728065ed8b06de7c93",
+    "train.jsonl": "1f1ac0097b120404cabf1e315887907461a434d1369b693c82e2a910e899ab84",
+    "train-B.jsonl": "00553ac4ee1181195cc3d8689e9bf21bc6904936712d4f3a3f50d6cbf91fbbe7",
+    "validation.jsonl": "3e2840cd09a2372e812dbc8f37904abf31dff4c2a74c09f5aec00b9c3bc2e1ce",
+    "ood.jsonl": "cdad1edf8d687786bfaba10c0d3dfebb8216f94dba26afcb92d4a0699d8eac15",
+    "descriptions.json": "a0c9638e4659cc494df4e94ddc99daca9571b2ca6d94f7677620059621de90a4",
+    "final-texts.jsonl": "457f0e77ab4f5296106cbc33b9d1c5925106eb91000caee5e8fc37f0f0a5091a"
+  },
+  "licenses": "Private research only; unresolved dataset-license restrictions inherited from release",
+  "limitations": [
+    "Two-agent semantic review is not independent human review",
+    "Lexically similar reformulations; not broad prompt OOD",
+    "One seed does not measure training variance",
+    "Historical validation already used; old test and COPA not used for selection"
+  ]
+}

benchmarks/cpu/control-a.json

@@ -0,0 +1,41 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "trained": true,
+  "machine": "x86_64",
+  "platform": "Linux-7.0.0-34-generic-x86_64-inconnu-with-glibc2.43",
+  "onnxruntime": "1.24.4",
+  "threads": 2,
+  "repeats": 30,
+  "peak_process_RSS_MiB": 195.97265625,
+  "measurements": [
+    {
+      "tokens": 64,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 11.29639999999199,
+      "p95_ms": 23.609198799022124
+    },
+    {
+      "tokens": 128,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 21.42894699863973,
+      "p95_ms": 21.64968539873371
+    },
+    {
+      "tokens": 256,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 43.88379150259425,
+      "p95_ms": 44.729615551477764
+    },
+    {
+      "tokens": 512,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 105.77025650127325,
+      "p95_ms": 115.75684060080675
+    }
+  ],
+  "limitations": "Synthetic inputs; warm model; excludes tokenization, network and loading. Not a quality evaluation or a guarantee for another CPU."
+}

benchmarks/cpu/experimental-b.json

@@ -0,0 +1,41 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "trained": true,
+  "machine": "x86_64",
+  "platform": "Linux-7.0.0-34-generic-x86_64-inconnu-with-glibc2.43",
+  "onnxruntime": "1.24.4",
+  "threads": 2,
+  "repeats": 30,
+  "peak_process_RSS_MiB": 196.0,
+  "measurements": [
+    {
+      "tokens": 64,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 11.079036998125957,
+      "p95_ms": 11.426868901253329
+    },
+    {
+      "tokens": 128,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 21.55238850173191,
+      "p95_ms": 28.829175600912993
+    },
+    {
+      "tokens": 256,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 44.82588450264302,
+      "p95_ms": 45.46837649868394
+    },
+    {
+      "tokens": 512,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 111.79175250072149,
+      "p95_ms": 125.3648807516583
+    }
+  ],
+  "limitations": "Synthetic inputs; warm model; excludes tokenization, network and loading. Not a quality evaluation or a guarantee for another CPU."
+}

benchmarks/cpu/original-current.json

@@ -0,0 +1,41 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "trained": true,
+  "machine": "x86_64",
+  "platform": "Linux-7.0.0-34-generic-x86_64-inconnu-with-glibc2.43",
+  "onnxruntime": "1.24.4",
+  "threads": 2,
+  "repeats": 30,
+  "peak_process_RSS_MiB": 195.7421875,
+  "measurements": [
+    {
+      "tokens": 64,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 11.094615500041982,
+      "p95_ms": 11.69740975165041
+    },
+    {
+      "tokens": 128,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 21.357893499953207,
+      "p95_ms": 33.236518100966336
+    },
+    {
+      "tokens": 256,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 44.78299600305036,
+      "p95_ms": 49.30807600176195
+    },
+    {
+      "tokens": 512,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 108.33085199919879,
+      "p95_ms": 136.28926924830006
+    }
+  ],
+  "limitations": "Synthetic inputs; warm model; excludes tokenization, network and loading. Not a quality evaluation or a guarantee for another CPU."
+}

benchmarks/history/original-copa-400-summary.json

@@ -0,0 +1,113 @@
+{
+  "purpose": "historical_development_evaluation_for_original_weights_only",
+  "source_report_sha256": "d61d4526f7db3b3a96dcc1c02bf98334fec3a889db3c5e56d5ab8c9d1044b02a",
+  "rows": 400,
+  "calibrated": false,
+  "not_evaluated_for_control_A_or_experimental_B": true,
+  "development_limitation": "Earlier test outcomes were observed during development; this is not an entirely blind end-to-end test. COPA was excluded from fitting and checkpoint selection.",
+  "publication_redactions": "Raw example-specific option vocabularies, by-task tables, workspace paths, and descriptive-oracle majority baseline omitted.",
+  "runtime": {
+    "batch_size": 16,
+    "threads": 4,
+    "quantized_load_seconds": 0.24128931399900466,
+    "floating_load_seconds": 0.7911271149641834,
+    "encoding_seconds": 0.04855810495791957,
+    "collation_seconds": 0.0023445660481229424,
+    "evaluation_wall_seconds_excluding_output_writes": 11.097406168002635
+  },
+  "quantization_comparison": {
+    "rows": 400,
+    "labelled_rows": 400,
+    "prediction_agreement": 0.8775,
+    "accuracy_drop": 0.0225,
+    "accuracy_drop_definition": "floating accuracy minus quantized accuracy; positive means quantization is worse",
+    "floating_correct_quantized_wrong": 29,
+    "quantized_correct_floating_wrong": 20,
+    "logits": {
+      "valid_option_values": 800,
+      "mean_absolute_difference": 0.2002160781857674,
+      "max_absolute_difference": 1.521109938621521
+    },
+    "probabilities": {
+      "valid_option_values": 800,
+      "mean_absolute_difference": 0.04965776814257573,
+      "max_absolute_difference": 0.2278980684992542
+    }
+  },
+  "floating": {
+    "overall": {
+      "rows": 400,
+      "labelled_rows": 400,
+      "missing_label_rows": 0,
+      "truncated_rows": 0,
+      "correct": 206,
+      "ground_truth_accuracy": 0.515,
+      "teacher_ground_truth_accuracy": 0.6975,
+      "teacher_agreement": 0.4775,
+      "teacher_to_student_kl": 0.15669801502894715,
+      "score_mae": null,
+      "score_labelled_rows": 0,
+      "noul_brier": null,
+      "noul_labelled_rows": 0,
+      "ece_10_bins": 0.0948756083782432,
+      "uniform_random_baseline_accuracy": 0.5
+    },
+    "by_dataset": {
+      "copa": {
+        "rows": 400,
+        "labelled_rows": 400,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 206,
+        "ground_truth_accuracy": 0.515,
+        "teacher_ground_truth_accuracy": 0.6975,
+        "teacher_agreement": 0.4775,
+        "teacher_to_student_kl": 0.15669801502894715,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.0948756083782432,
+        "uniform_random_baseline_accuracy": 0.5
+      }
+    }
+  },
+  "quantized": {
+    "overall": {
+      "rows": 400,
+      "labelled_rows": 400,
+      "missing_label_rows": 0,
+      "truncated_rows": 0,
+      "correct": 197,
+      "ground_truth_accuracy": 0.4925,
+      "teacher_ground_truth_accuracy": 0.6975,
+      "teacher_agreement": 0.49,
+      "teacher_to_student_kl": 0.14061569682439606,
+      "score_mae": null,
+      "score_labelled_rows": 0,
+      "noul_brier": null,
+      "noul_labelled_rows": 0,
+      "ece_10_bins": 0.08862892688434312,
+      "uniform_random_baseline_accuracy": 0.5
+    },
+    "by_dataset": {
+      "copa": {
+        "rows": 400,
+        "labelled_rows": 400,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 197,
+        "ground_truth_accuracy": 0.4925,
+        "teacher_ground_truth_accuracy": 0.6975,
+        "teacher_agreement": 0.49,
+        "teacher_to_student_kl": 0.14061569682439606,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.08862892688434312,
+        "uniform_random_baseline_accuracy": 0.5
+      }
+    }
+  }
+}

benchmarks/history/original-cpu-earlier-session.json

@@ -0,0 +1,41 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "trained": true,
+  "machine": "x86_64",
+  "platform": "Linux-7.0.0-34-generic-x86_64-inconnu-with-glibc2.43",
+  "onnxruntime": "1.24.4",
+  "threads": 2,
+  "repeats": 50,
+  "peak_process_RSS_MiB": 195.95703125,
+  "measurements": [
+    {
+      "tokens": 64,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 5.586627001321176,
+      "p95_ms": 6.451607048438743
+    },
+    {
+      "tokens": 128,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 10.78688699999475,
+      "p95_ms": 11.632900599579441
+    },
+    {
+      "tokens": 256,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 22.02214199860464,
+      "p95_ms": 30.23740555290714
+    },
+    {
+      "tokens": 512,
+      "batch_size": 1,
+      "options": 4,
+      "p50_ms": 63.14534450211795,
+      "p95_ms": 69.91638834915648
+    }
+  ],
+  "limitations": "Synthetic inputs; warm model; excludes tokenization, network and loading. Not a quality evaluation or a guarantee for another CPU."
+}

benchmarks/history/original-test-3000-summary.json

@@ -0,0 +1,249 @@
+{
+  "purpose": "historical_development_evaluation_for_original_weights_only",
+  "source_report_sha256": "057c53c602a6356c39e593879548b45e742e9eb193c4f8561f1b69eb96595692",
+  "rows": 3000,
+  "calibrated": false,
+  "not_evaluated_for_control_A_or_experimental_B": true,
+  "development_limitation": "Earlier test outcomes were observed during development; this is not an entirely blind end-to-end test. COPA was excluded from fitting and checkpoint selection.",
+  "publication_redactions": "Raw example-specific option vocabularies, by-task tables, workspace paths, and descriptive-oracle majority baseline omitted.",
+  "runtime": {
+    "batch_size": 16,
+    "threads": 4,
+    "quantized_load_seconds": 0.2457776820519939,
+    "floating_load_seconds": 0.8067200410296209,
+    "encoding_seconds": 0.7069170550093986,
+    "collation_seconds": 0.024417184351477772,
+    "evaluation_wall_seconds_excluding_output_writes": 58.057117588992696
+  },
+  "quantization_comparison": {
+    "rows": 3000,
+    "labelled_rows": 3000,
+    "prediction_agreement": 0.937,
+    "accuracy_drop": 0.0013333333333333333,
+    "accuracy_drop_definition": "floating accuracy minus quantized accuracy; positive means quantization is worse",
+    "floating_correct_quantized_wrong": 88,
+    "quantized_correct_floating_wrong": 84,
+    "logits": {
+      "valid_option_values": 12000,
+      "mean_absolute_difference": 0.4191225695454632,
+      "max_absolute_difference": 2.950909733772278
+    },
+    "probabilities": {
+      "valid_option_values": 12000,
+      "mean_absolute_difference": 0.03555099927094927,
+      "max_absolute_difference": 0.5201866639682322
+    }
+  },
+  "floating": {
+    "overall": {
+      "rows": 3000,
+      "labelled_rows": 3000,
+      "missing_label_rows": 0,
+      "truncated_rows": 0,
+      "correct": 2250,
+      "ground_truth_accuracy": 0.75,
+      "teacher_ground_truth_accuracy": 0.6863333333333334,
+      "teacher_agreement": 0.6693333333333333,
+      "teacher_to_student_kl": 0.4394546882317112,
+      "score_mae": 0.7635296451689558,
+      "score_labelled_rows": 600,
+      "noul_brier": 0.22620426238182129,
+      "noul_labelled_rows": 600,
+      "ece_10_bins": 0.03069561032879222,
+      "uniform_random_baseline_accuracy": 0.29000000000000004
+    },
+    "by_dataset": {
+      "ag_news": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 545,
+        "ground_truth_accuracy": 0.9083333333333333,
+        "teacher_ground_truth_accuracy": 0.9116666666666666,
+        "teacher_agreement": 0.9333333333333333,
+        "teacher_to_student_kl": 0.14562351508071758,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.02919148973125299,
+        "uniform_random_baseline_accuracy": 0.25
+      },
+      "boolq": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 406,
+        "ground_truth_accuracy": 0.6766666666666666,
+        "teacher_ground_truth_accuracy": 0.8333333333333334,
+        "teacher_agreement": 0.7033333333333334,
+        "teacher_to_student_kl": 0.46581474517803506,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": 0.22620426238182129,
+        "noul_labelled_rows": 600,
+        "ece_10_bins": 0.14024903248642293,
+        "uniform_random_baseline_accuracy": 0.5
+      },
+      "emotion": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 545,
+        "ground_truth_accuracy": 0.9083333333333333,
+        "teacher_ground_truth_accuracy": 0.59,
+        "teacher_agreement": 0.625,
+        "teacher_to_student_kl": 0.7589576663432547,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.07844001209439692,
+        "uniform_random_baseline_accuracy": 0.16666666666666666
+      },
+      "snli": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 470,
+        "ground_truth_accuracy": 0.7833333333333333,
+        "teacher_ground_truth_accuracy": 0.7633333333333333,
+        "teacher_agreement": 0.7033333333333334,
+        "teacher_to_student_kl": 0.302764245855786,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.038207785506182396,
+        "uniform_random_baseline_accuracy": 0.3333333333333333
+      },
+      "sst5": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 284,
+        "ground_truth_accuracy": 0.47333333333333333,
+        "teacher_ground_truth_accuracy": 0.3333333333333333,
+        "teacher_agreement": 0.38166666666666665,
+        "teacher_to_student_kl": 0.5241132687007625,
+        "score_mae": 0.7635296451689558,
+        "score_labelled_rows": 600,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.08556262397129752,
+        "uniform_random_baseline_accuracy": 0.2
+      }
+    }
+  },
+  "quantized": {
+    "overall": {
+      "rows": 3000,
+      "labelled_rows": 3000,
+      "missing_label_rows": 0,
+      "truncated_rows": 0,
+      "correct": 2246,
+      "ground_truth_accuracy": 0.7486666666666667,
+      "teacher_ground_truth_accuracy": 0.6863333333333334,
+      "teacher_agreement": 0.6763333333333333,
+      "teacher_to_student_kl": 0.3660636905206232,
+      "score_mae": 0.7933817943202061,
+      "score_labelled_rows": 600,
+      "noul_brier": 0.21363519516235982,
+      "noul_labelled_rows": 600,
+      "ece_10_bins": 0.022693194229737538,
+      "uniform_random_baseline_accuracy": 0.29000000000000004
+    },
+    "by_dataset": {
+      "ag_news": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 546,
+        "ground_truth_accuracy": 0.91,
+        "teacher_ground_truth_accuracy": 0.9116666666666666,
+        "teacher_agreement": 0.9283333333333333,
+        "teacher_to_student_kl": 0.11830738113161032,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.026812594538814332,
+        "uniform_random_baseline_accuracy": 0.25
+      },
+      "boolq": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 414,
+        "ground_truth_accuracy": 0.69,
+        "teacher_ground_truth_accuracy": 0.8333333333333334,
+        "teacher_agreement": 0.73,
+        "teacher_to_student_kl": 0.3804193201203644,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": 0.21363519516235982,
+        "noul_labelled_rows": 600,
+        "ece_10_bins": 0.08913086434644102,
+        "uniform_random_baseline_accuracy": 0.5
+      },
+      "emotion": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 547,
+        "ground_truth_accuracy": 0.9116666666666666,
+        "teacher_ground_truth_accuracy": 0.59,
+        "teacher_agreement": 0.6266666666666667,
+        "teacher_to_student_kl": 0.6541046259251772,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.12286200104658901,
+        "uniform_random_baseline_accuracy": 0.16666666666666666
+      },
+      "snli": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 469,
+        "ground_truth_accuracy": 0.7816666666666666,
+        "teacher_ground_truth_accuracy": 0.7633333333333333,
+        "teacher_agreement": 0.7066666666666667,
+        "teacher_to_student_kl": 0.20679542158256595,
+        "score_mae": null,
+        "score_labelled_rows": 0,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.10508906707488629,
+        "uniform_random_baseline_accuracy": 0.3333333333333333
+      },
+      "sst5": {
+        "rows": 600,
+        "labelled_rows": 600,
+        "missing_label_rows": 0,
+        "truncated_rows": 0,
+        "correct": 270,
+        "ground_truth_accuracy": 0.45,
+        "teacher_ground_truth_accuracy": 0.3333333333333333,
+        "teacher_agreement": 0.39,
+        "teacher_to_student_kl": 0.47069170384339826,
+        "score_mae": 0.7933817943202061,
+        "score_labelled_rows": 600,
+        "noul_brier": null,
+        "noul_labelled_rows": 0,
+        "ece_10_bins": 0.04569868374631497,
+        "uniform_random_baseline_accuracy": 0.2
+      }
+    }
+  }
+}

benchmarks/history/pre-ab-robustness-audit.json

@@ -0,0 +1,2154 @@
+{
+  "protocol_sha256": "f5f0a74e0ad1321e58d5ee59188aa962cb4fe95686f3a41a3114600fd653fcdf",
+  "texts": 120,
+  "requests": 3360,
+  "teacher_complete": false,
+  "teacher_requests_available": 672,
+  "teacher_subset_complete": true,
+  "teacher_base_texts": 24,
+  "teacher_comparison_cells_same_texts": [
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8958333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.7291666666666666,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.875,
+      "accuracy_ci95_text_bootstrap": [
+        0.7083333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8333333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.5833333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8541666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.6458333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8541666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.6458333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8333333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.5833333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.5416666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.3333333333333333,
+        0.7291666666666666
+      ],
+      "all_rotations_correct": 0.25,
+      "order_instability": 0.6666666666666666,
+      "rotation_zero_accuracy": 0.6666666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.5208333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.3125,
+        0.7291666666666666
+      ],
+      "all_rotations_correct": 0.25,
+      "order_instability": 0.6666666666666666,
+      "rotation_zero_accuracy": 0.6666666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8541666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.6666666666666666,
+        1.0
+      ],
+      "all_rotations_correct": 0.75,
+      "order_instability": 0.16666666666666666,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.4375,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.6875
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.4166666666666667,
+      "rotation_zero_accuracy": 0.5
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.4375,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.6875
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.3333333333333333,
+      "rotation_zero_accuracy": 0.5
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8958333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.7291666666666666,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.875,
+      "accuracy_ci95_text_bootstrap": [
+        0.7083333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8333333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.5833333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8541666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.6458333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.875,
+      "accuracy_ci95_text_bootstrap": [
+        0.7083333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8333333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.5833333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.5416666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.3333333333333333,
+        0.75
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.5833333333333334,
+      "rotation_zero_accuracy": 0.6666666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.5208333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.31197916666666686,
+        0.75
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.5833333333333334,
+      "rotation_zero_accuracy": 0.6666666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8958333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.7291666666666666,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.4583333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.20833333333333334,
+        0.7083333333333334
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.5
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.4583333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.20833333333333334,
+        0.7083333333333334
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.5
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 48,
+      "accuracy": 0.8958333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.7291666666666666,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.75,
+      "accuracy_ci95_text_bootstrap": [
+        0.5277777777777778,
+        0.9444444444444445
+      ],
+      "all_rotations_correct": 0.6666666666666666,
+      "order_instability": 0.16666666666666666,
+      "rotation_zero_accuracy": 0.75
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.16666666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.08333333333333333,
+        0.25000000000000006
+      ],
+      "all_rotations_correct": 0.0,
+      "order_instability": 0.6666666666666666,
+      "rotation_zero_accuracy": 0.0
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.1111111111111111,
+      "accuracy_ci95_text_bootstrap": [
+        0.027777777777777776,
+        0.19444444444444445
+      ],
+      "all_rotations_correct": 0.0,
+      "order_instability": 0.6666666666666666,
+      "rotation_zero_accuracy": 0.0
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.8055555555555555,
+      "accuracy_ci95_text_bootstrap": [
+        0.5833333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.75,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.611111111111111,
+      "accuracy_ci95_text_bootstrap": [
+        0.4444444444444444,
+        0.7777777777777777
+      ],
+      "all_rotations_correct": 0.25,
+      "order_instability": 0.75,
+      "rotation_zero_accuracy": 0.4166666666666667
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.4444444444444444,
+      "accuracy_ci95_text_bootstrap": [
+        0.3055555555555555,
+        0.5833333333333334
+      ],
+      "all_rotations_correct": 0.0,
+      "order_instability": 0.9166666666666666,
+      "rotation_zero_accuracy": 0.3333333333333333
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.611111111111111,
+      "accuracy_ci95_text_bootstrap": [
+        0.3611111111111111,
+        0.861111111111111
+      ],
+      "all_rotations_correct": 0.5,
+      "order_instability": 0.4166666666666667,
+      "rotation_zero_accuracy": 0.6666666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.3333333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.5277777777777778
+      ],
+      "all_rotations_correct": 0.16666666666666666,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.25
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.4166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.19444444444444442,
+        0.6666666666666666
+      ],
+      "all_rotations_correct": 0.25,
+      "order_instability": 0.4166666666666667,
+      "rotation_zero_accuracy": 0.3333333333333333
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.8333333333333334,
+      "accuracy_ci95_text_bootstrap": [
+        0.5833333333333334,
+        1.0
+      ],
+      "all_rotations_correct": 0.8333333333333334,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.8055555555555557,
+      "accuracy_ci95_text_bootstrap": [
+        0.611111111111111,
+        0.9722222222222222
+      ],
+      "all_rotations_correct": 0.6666666666666666,
+      "order_instability": 0.25,
+      "rotation_zero_accuracy": 0.8333333333333334
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.6944444444444443,
+      "accuracy_ci95_text_bootstrap": [
+        0.4444444444444444,
+        0.9166666666666666
+      ],
+      "all_rotations_correct": 0.6666666666666666,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.6666666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.4166666666666667,
+      "accuracy_ci95_text_bootstrap": [
+        0.19444444444444445,
+        0.6388888888888888
+      ],
+      "all_rotations_correct": 0.25,
+      "order_instability": 0.4166666666666667,
+      "rotation_zero_accuracy": 0.25
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.36111111111111116,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.5555555555555555
+      ],
+      "all_rotations_correct": 0.16666666666666666,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.16666666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.38888888888888884,
+      "accuracy_ci95_text_bootstrap": [
+        0.13888888888888887,
+        0.6388888888888888
+      ],
+      "all_rotations_correct": 0.25,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.3333333333333333
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.4166666666666667,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.6666666666666666
+      ],
+      "all_rotations_correct": 0.4166666666666667,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.4166666666666667
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.3333333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.5277777777777778
+      ],
+      "all_rotations_correct": 0.08333333333333333,
+      "order_instability": 0.8333333333333334,
+      "rotation_zero_accuracy": 0.3333333333333333
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.3333333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.13888888888888887,
+        0.5277777777777778
+      ],
+      "all_rotations_correct": 0.08333333333333333,
+      "order_instability": 0.5833333333333334,
+      "rotation_zero_accuracy": 0.5
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "requests": 36,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.75,
+        1.0
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.0,
+      "rotation_zero_accuracy": 0.9166666666666666
+    }
+  ],
+  "teacher_comparison_style_deltas_same_texts": [
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.041666666666666664,
+      "ci95": [
+        -0.125,
+        0.0
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.020833333333333332,
+      "ci95": [
+        -0.0625,
+        0.0
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0,
+      "ci95": [
+        0.0,
+        0.0
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.3541666666666667,
+      "ci95": [
+        -0.5208333333333334,
+        -0.1875
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.3541666666666667,
+      "ci95": [
+        -0.5208333333333334,
+        -0.1875
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.020833333333333332,
+      "ci95": [
+        -0.0625,
+        0.125
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.4583333333333333,
+      "ci95": [
+        -0.7083333333333334,
+        -0.20833333333333334
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.4375,
+      "ci95": [
+        -0.6875,
+        -0.1875
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0625,
+      "ci95": [
+        0.0,
+        0.1875
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.0625,
+      "ci95": [
+        -0.1875,
+        0.0
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "delta": 0.0,
+      "ci95": [
+        0.0,
+        0.0
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0,
+      "ci95": [
+        0.0,
+        0.0
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.375,
+      "ci95": [
+        -0.5833333333333334,
+        -0.16666666666666666
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.3541666666666667,
+      "ci95": [
+        -0.5416666666666666,
+        -0.16666666666666666
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0625,
+      "ci95": [
+        0.0,
+        0.1875
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.4583333333333333,
+      "ci95": [
+        -0.7083333333333334,
+        -0.20833333333333334
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.4166666666666667,
+      "ci95": [
+        -0.6458333333333334,
+        -0.1875
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0625,
+      "ci95": [
+        0.0,
+        0.1875
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.75,
+      "ci95": [
+        -0.9166666666666666,
+        -0.5000000000000001
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.8055555555555555,
+      "ci95": [
+        -0.9722222222222223,
+        -0.5555555555555557
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.05555555555555556,
+      "ci95": [
+        0.0,
+        0.16666666666666666
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.30555555555555564,
+      "ci95": [
+        -0.5277777777777778,
+        -0.05555555555555559
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.47222222222222227,
+      "ci95": [
+        -0.6944444444444445,
+        -0.22222222222222224
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "delta": -0.1388888888888889,
+      "ci95": [
+        -0.2777777777777778,
+        -0.02777777777777778
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.5833333333333334,
+      "ci95": [
+        -0.7777777777777778,
+        -0.36111111111111116
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.5000000000000001,
+      "ci95": [
+        -0.7222222222222223,
+        -0.2777777777777778
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.08333333333333333,
+      "ci95": [
+        0.0,
+        0.19513888888889147
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.3888888888888889,
+      "ci95": [
+        -0.5555555555555557,
+        -0.2222222222222222
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.3333333333333334,
+      "ci95": [
+        -0.5000000000000001,
+        -0.16666666666666666
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0,
+      "ci95": [
+        0.0,
+        0.0
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.4166666666666667,
+      "ci95": [
+        -0.6388888888888888,
+        -0.19444444444444445
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.27777777777777773,
+      "ci95": [
+        -0.5277777777777778,
+        -0.05555555555555555
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0,
+      "ci95": [
+        0.0,
+        0.0
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 12,
+      "delta": -0.47222222222222227,
+      "ci95": [
+        -0.6944444444444445,
+        -0.25000000000000006
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 12,
+      "delta": -0.36111111111111116,
+      "ci95": [
+        -0.5833333333333334,
+        -0.16666666666666666
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "teacher",
+      "texts": 12,
+      "delta": 0.0,
+      "ci95": [
+        0.0,
+        0.0
+      ]
+    }
+  ],
+  "cells": [
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.9208333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.85,
+        0.9791666666666666
+      ],
+      "all_rotations_correct": 0.9,
+      "order_instability": 0.05,
+      "rotation_zero_accuracy": 0.9333333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.9166666666666666,
+      "accuracy_ci95_text_bootstrap": [
+        0.8458333333333333,
+        0.975
+      ],
+      "all_rotations_correct": 0.8833333333333333,
+      "order_instability": 0.06666666666666667,
+      "rotation_zero_accuracy": 0.9333333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.9041666666666667,
+      "accuracy_ci95_text_bootstrap": [
+        0.8333333333333334,
+        0.9625
+      ],
+      "all_rotations_correct": 0.8666666666666667,
+      "order_instability": 0.1,
+      "rotation_zero_accuracy": 0.8666666666666667
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.9208333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.85,
+        0.975
+      ],
+      "all_rotations_correct": 0.8833333333333333,
+      "order_instability": 0.08333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5708333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.4791666666666667,
+        0.6625
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.5666666666666667,
+      "rotation_zero_accuracy": 0.6833333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5666666666666667,
+      "accuracy_ci95_text_bootstrap": [
+        0.475,
+        0.6625
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.6,
+      "rotation_zero_accuracy": 0.6833333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5375,
+      "accuracy_ci95_text_bootstrap": [
+        0.43333333333333335,
+        0.6458333333333334
+      ],
+      "all_rotations_correct": 0.4,
+      "order_instability": 0.48333333333333334,
+      "rotation_zero_accuracy": 0.6333333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5375,
+      "accuracy_ci95_text_bootstrap": [
+        0.43333333333333335,
+        0.6417708333333337
+      ],
+      "all_rotations_correct": 0.38333333333333336,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.65
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.925,
+      "accuracy_ci95_text_bootstrap": [
+        0.8583333333333333,
+        0.9833333333333333
+      ],
+      "all_rotations_correct": 0.9166666666666666,
+      "order_instability": 0.03333333333333333,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.925,
+      "accuracy_ci95_text_bootstrap": [
+        0.8583333333333333,
+        0.975
+      ],
+      "all_rotations_correct": 0.9,
+      "order_instability": 0.05,
+      "rotation_zero_accuracy": 0.9333333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.9125,
+      "accuracy_ci95_text_bootstrap": [
+        0.8416666666666667,
+        0.9709375000000003
+      ],
+      "all_rotations_correct": 0.9,
+      "order_instability": 0.05,
+      "rotation_zero_accuracy": 0.9166666666666666
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.9083333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.8333333333333334,
+        0.975
+      ],
+      "all_rotations_correct": 0.9,
+      "order_instability": 0.03333333333333333,
+      "rotation_zero_accuracy": 0.9
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5875,
+      "accuracy_ci95_text_bootstrap": [
+        0.49166666666666664,
+        0.6833333333333333
+      ],
+      "all_rotations_correct": 0.38333333333333336,
+      "order_instability": 0.5833333333333334,
+      "rotation_zero_accuracy": 0.7
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5791666666666667,
+      "accuracy_ci95_text_bootstrap": [
+        0.48333333333333334,
+        0.675
+      ],
+      "all_rotations_correct": 0.4,
+      "order_instability": 0.5666666666666667,
+      "rotation_zero_accuracy": 0.6833333333333333
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.525,
+      "accuracy_ci95_text_bootstrap": [
+        0.425,
+        0.625
+      ],
+      "all_rotations_correct": 0.3333333333333333,
+      "order_instability": 0.6,
+      "rotation_zero_accuracy": 0.6
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "requests": 240,
+      "accuracy": 0.5333333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.43333333333333335,
+        0.6333333333333333
+      ],
+      "all_rotations_correct": 0.31666666666666665,
+      "order_instability": 0.6,
+      "rotation_zero_accuracy": 0.6
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.7833333333333333,
+      "accuracy_ci95_text_bootstrap": [
+        0.6722222222222223,
+        0.8833333333333333
+      ],
+      "all_rotations_correct": 0.7666666666666667,
+      "order_instability": 0.05,
+      "rotation_zero_accuracy": 0.8
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.8,
+      "accuracy_ci95_text_bootstrap": [
+        0.6944444444444444,
+        0.8944444444444445
+      ],
+      "all_rotations_correct": 0.7833333333333333,
+      "order_instability": 0.06666666666666667,
+      "rotation_zero_accuracy": 0.8
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.14444444444444443,
+      "accuracy_ci95_text_bootstrap": [
+        0.09999999999999999,
+        0.1888888888888889
+      ],
+      "all_rotations_correct": 0.0,
+      "order_instability": 0.7666666666666667,
+      "rotation_zero_accuracy": 0.11666666666666667
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.20555555555555557,
+      "accuracy_ci95_text_bootstrap": [
+        0.15555555555555553,
+        0.2555555555555556
+      ],
+      "all_rotations_correct": 0.016666666666666666,
+      "order_instability": 0.8,
+      "rotation_zero_accuracy": 0.18333333333333332
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.5222222222222221,
+      "accuracy_ci95_text_bootstrap": [
+        0.42777777777777787,
+        0.6166666666666667
+      ],
+      "all_rotations_correct": 0.2833333333333333,
+      "order_instability": 0.6166666666666667,
+      "rotation_zero_accuracy": 0.48333333333333334
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.49444444444444435,
+      "accuracy_ci95_text_bootstrap": [
+        0.4111111111111111,
+        0.5777777777777777
+      ],
+      "all_rotations_correct": 0.16666666666666666,
+      "order_instability": 0.7666666666666667,
+      "rotation_zero_accuracy": 0.43333333333333335
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.24444444444444444,
+      "accuracy_ci95_text_bootstrap": [
+        0.16666666666666666,
+        0.3279166666666672
+      ],
+      "all_rotations_correct": 0.06666666666666667,
+      "order_instability": 0.55,
+      "rotation_zero_accuracy": 0.23333333333333334
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.3277777777777778,
+      "accuracy_ci95_text_bootstrap": [
+        0.22777777777777777,
+        0.4333333333333333
+      ],
+      "all_rotations_correct": 0.2,
+      "order_instability": 0.4166666666666667,
+      "rotation_zero_accuracy": 0.3
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.6055555555555555,
+      "accuracy_ci95_text_bootstrap": [
+        0.4888888888888889,
+        0.7166666666666666
+      ],
+      "all_rotations_correct": 0.5166666666666667,
+      "order_instability": 0.18333333333333332,
+      "rotation_zero_accuracy": 0.6
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "fixed",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.5722222222222222,
+      "accuracy_ci95_text_bootstrap": [
+        0.45,
+        0.6888888888888889
+      ],
+      "all_rotations_correct": 0.5166666666666667,
+      "order_instability": 0.16666666666666666,
+      "rotation_zero_accuracy": 0.5666666666666667
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.3222222222222222,
+      "accuracy_ci95_text_bootstrap": [
+        0.22777777777777777,
+        0.41680555555555615
+      ],
+      "all_rotations_correct": 0.11666666666666667,
+      "order_instability": 0.5166666666666667,
+      "rotation_zero_accuracy": 0.31666666666666665
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.3055555555555555,
+      "accuracy_ci95_text_bootstrap": [
+        0.21666666666666667,
+        0.39444444444444443
+      ],
+      "all_rotations_correct": 0.11666666666666667,
+      "order_instability": 0.5,
+      "rotation_zero_accuracy": 0.25
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.36111111111111105,
+      "accuracy_ci95_text_bootstrap": [
+        0.27777777777777773,
+        0.45000000000000007
+      ],
+      "all_rotations_correct": 0.13333333333333333,
+      "order_instability": 0.6,
+      "rotation_zero_accuracy": 0.38333333333333336
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.3888888888888889,
+      "accuracy_ci95_text_bootstrap": [
+        0.27777777777777773,
+        0.5
+      ],
+      "all_rotations_correct": 0.31666666666666665,
+      "order_instability": 0.23333333333333334,
+      "rotation_zero_accuracy": 0.38333333333333336
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.3111111111111111,
+      "accuracy_ci95_text_bootstrap": [
+        0.23333333333333334,
+        0.3944444444444445
+      ],
+      "all_rotations_correct": 0.05,
+      "order_instability": 0.8166666666666667,
+      "rotation_zero_accuracy": 0.3
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "requests": 180,
+      "accuracy": 0.31111111111111106,
+      "accuracy_ci95_text_bootstrap": [
+        0.21666666666666667,
+        0.4111111111111111
+      ],
+      "all_rotations_correct": 0.15,
+      "order_instability": 0.43333333333333335,
+      "rotation_zero_accuracy": 0.36666666666666664
+    }
+  ],
+  "paired_style_minus_fixed_accuracy": [
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.016666666666666666,
+      "ci95": [
+        -0.0625,
+        0.029166666666666667
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "delta": 0.004166666666666667,
+      "ci95": [
+        -0.0375,
+        0.05
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.35,
+      "ci95": [
+        -0.44583333333333336,
+        -0.25
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.35,
+      "ci95": [
+        -0.44583333333333336,
+        -0.25
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.38333333333333336,
+      "ci95": [
+        -0.49166666666666664,
+        -0.2708333333333333
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.37916666666666665,
+      "ci95": [
+        -0.4875,
+        -0.2625
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.0125,
+      "ci95": [
+        -0.0625,
+        0.03333333333333333
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.016666666666666666,
+      "ci95": [
+        -0.06666666666666667,
+        0.016666666666666666
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.3375,
+      "ci95": [
+        -0.4375,
+        -0.23333333333333334
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.3458333333333333,
+      "ci95": [
+        -0.44583333333333336,
+        -0.24583333333333332
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.4,
+      "ci95": [
+        -0.5041666666666667,
+        -0.29572916666666627
+      ]
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.39166666666666666,
+      "ci95": [
+        -0.4875,
+        -0.2875
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.6388888888888888,
+      "ci95": [
+        -0.7444444444444444,
+        -0.5277777777777779
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.5944444444444444,
+      "ci95": [
+        -0.711111111111111,
+        -0.4722222222222223
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.2611111111111112,
+      "ci95": [
+        -0.3944444444444445,
+        -0.12222222222222223
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.3055555555555555,
+      "ci95": [
+        -0.4222222222222223,
+        -0.1833333333333334
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.5388888888888889,
+      "ci95": [
+        -0.6444444444444444,
+        -0.4277777777777778
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.47222222222222227,
+      "ci95": [
+        -0.5833333333333335,
+        -0.35555555555555557
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.2833333333333334,
+      "ci95": [
+        -0.3777777777777778,
+        -0.1888888888888889
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.26666666666666666,
+      "ci95": [
+        -0.35555555555555557,
+        -0.1833333333333333
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.24444444444444446,
+      "ci95": [
+        -0.34458333333333335,
+        -0.14444444444444446
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.18333333333333332,
+      "ci95": [
+        -0.2833333333333333,
+        -0.08888888888888889
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "fp32",
+      "texts": 60,
+      "delta": -0.2944444444444444,
+      "ci95": [
+        -0.40555555555555556,
+        -0.1833333333333334
+      ]
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "model": "int8",
+      "texts": 60,
+      "delta": -0.2611111111111111,
+      "ci95": [
+        -0.3722222222222223,
+        -0.15555555555555556
+      ]
+    }
+  ],
+  "quantization": [
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "fixed",
+      "flip_rate": 0.0125,
+      "mean_total_variation": 0.024744551112909283,
+      "max_total_variation": 0.18720267309731498
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "short",
+      "flip_rate": 0.025,
+      "mean_total_variation": 0.05402983263895219,
+      "max_total_variation": 0.3253621078860362
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long",
+      "flip_rate": 0.07916666666666666,
+      "mean_total_variation": 0.08043902622275094,
+      "max_total_variation": 0.29786696594217227
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "opaque",
+      "style": "long_named",
+      "flip_rate": 0.05416666666666667,
+      "mean_total_variation": 0.07193859290985224,
+      "max_total_variation": 0.2695192629062798
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "fixed",
+      "flip_rate": 0.025,
+      "mean_total_variation": 0.0325915520093269,
+      "max_total_variation": 0.25381608388696264
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "short",
+      "flip_rate": 0.020833333333333332,
+      "mean_total_variation": 0.054007280016479024,
+      "max_total_variation": 0.24714846232905585
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long",
+      "flip_rate": 0.058333333333333334,
+      "mean_total_variation": 0.07145547211515127,
+      "max_total_variation": 0.3291973613101394
+    },
+    {
+      "dataset": "ag_news",
+      "keys": "semantic",
+      "style": "long_named",
+      "flip_rate": 0.05,
+      "mean_total_variation": 0.07524021821331758,
+      "max_total_variation": 0.20842211509776506
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "fixed",
+      "flip_rate": 0.03333333333333333,
+      "mean_total_variation": 0.08573770352915598,
+      "max_total_variation": 0.25184783845498604
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "short",
+      "flip_rate": 0.25555555555555554,
+      "mean_total_variation": 0.051259924176761874,
+      "max_total_variation": 0.19821004365653805
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long",
+      "flip_rate": 0.17777777777777778,
+      "mean_total_variation": 0.040942129139137515,
+      "max_total_variation": 0.16281755752157304
+    },
+    {
+      "dataset": "snli",
+      "keys": "opaque",
+      "style": "long_named",
+      "flip_rate": 0.28888888888888886,
+      "mean_total_variation": 0.03427239545958374,
+      "max_total_variation": 0.18796492410460625
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "fixed",
+      "flip_rate": 0.07222222222222222,
+      "mean_total_variation": 0.14792474831060787,
+      "max_total_variation": 0.31995717234851995
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "short",
+      "flip_rate": 0.15555555555555556,
+      "mean_total_variation": 0.04652275458967452,
+      "max_total_variation": 0.1480863505686345
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long",
+      "flip_rate": 0.18333333333333332,
+      "mean_total_variation": 0.029969332447007797,
+      "max_total_variation": 0.08305757636515404
+    },
+    {
+      "dataset": "snli",
+      "keys": "semantic",
+      "style": "long_named",
+      "flip_rate": 0.15555555555555556,
+      "mean_total_variation": 0.037899451248389115,
+      "max_total_variation": 0.10310639855638526
+    }
+  ],
+  "limitations": [
+    "Development validation already used for checkpoint selection",
+    "Only AG News and SNLI choice tasks",
+    "One alternative wording set per style",
+    "No score/noul or unseen-task conclusions",
+    "No weight changes, no GPU rental, no COPA evaluation"
+  ]
+}

checkpoints/control-a/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1da45906bfc7f513867875c5c5902a4909593f949d623b1eda4c25051cd2c53e
+size 152909884

checkpoints/control-a/training.json

@@ -0,0 +1,14 @@
+{
+  "trained": true,
+  "quality_validated": false,
+  "arm": "A",
+  "steps": 5048,
+  "selection": "terminal checkpoint after exactly epoch 2, identically for A and B",
+  "start_sha256": "5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008",
+  "protocol_sha256": "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366",
+  "shared_cache_sha256": "2857aaa26b592d6a2add814e9764b841ca1cbe43479c9c085d6f56d9efb1843a",
+  "shared_schedule_sha256": "d495a21cc69ddc56bcaea92c0cb4db020f9120814e5653456c166fe093677896",
+  "seed": 42,
+  "bf16": true,
+  "elapsed_seconds": 304.6381041610148
+}

checkpoints/experimental-b/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7dfc87cdeeb3b6885d54fce02beea64f55d7d93679ed1cc9850c533c3888e36c
+size 152909884

checkpoints/experimental-b/training.json

@@ -0,0 +1,14 @@
+{
+  "trained": true,
+  "quality_validated": false,
+  "arm": "B",
+  "steps": 5048,
+  "selection": "terminal checkpoint after exactly epoch 2, identically for A and B",
+  "start_sha256": "5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008",
+  "protocol_sha256": "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366",
+  "shared_cache_sha256": "2857aaa26b592d6a2add814e9764b841ca1cbe43479c9c085d6f56d9efb1843a",
+  "shared_schedule_sha256": "d495a21cc69ddc56bcaea92c0cb4db020f9120814e5653456c166fe093677896",
+  "seed": 42,
+  "bf16": true,
+  "elapsed_seconds": 288.4059911170043
+}

checkpoints/original/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008
+size 152909884

checkpoints/original/training.json

@@ -0,0 +1,93 @@
+{
+  "trained": true,
+  "smoke_test": false,
+  "quality_validated": false,
+  "base_model": "google/bert_uncased_L-6_H-512_A-8",
+  "base_revision": "dd53ec6ca9d05e0a91b309c4e137f31988888071",
+  "seed": 42,
+  "best_epoch": 6,
+  "selection": "validation ground-truth accuracy; teacher KL as tiebreaker",
+  "train_sha256": "9f1e076aedb2bb9f9bfd353933d7af863a0943dd681660ff4856b16bec5b8e2b",
+  "validation_sha256": "8cfabd613dde690e6bbf8092132f0100ecbb1041ec996bfa1de9dd6fe9420e49",
+  "hyperparameters": {
+    "epochs": 8,
+    "batch_size": 32,
+    "encoder_learning_rate": 2e-05,
+    "head_learning_rate": 0.0001,
+    "label_weight": 2.0,
+    "bf16_training": true
+  },
+  "elapsed_seconds": 1027.408421811997,
+  "training_rows": 80758,
+  "validation_rows": 2000,
+  "truncated_training_states": 0,
+  "teacher_sources": [
+    "convaiinnovations/laya"
+  ],
+  "history": [
+    {
+      "rows": 2000,
+      "teacher_KL": 0.4694688792228699,
+      "teacher_agreement": 0.573,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.62,
+      "epoch": 1
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.4274449801445007,
+      "teacher_agreement": 0.6015,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.683,
+      "epoch": 2
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.4418224000930786,
+      "teacher_agreement": 0.636,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.7365,
+      "epoch": 3
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.4513157923221588,
+      "teacher_agreement": 0.6375,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.734,
+      "epoch": 4
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.49970576214790347,
+      "teacher_agreement": 0.6315,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.75,
+      "epoch": 5
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.46601298499107363,
+      "teacher_agreement": 0.647,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.7595,
+      "epoch": 6
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.5017643827199936,
+      "teacher_agreement": 0.649,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.759,
+      "epoch": 7
+    },
+    {
+      "rows": 2000,
+      "teacher_KL": 0.5062610116004944,
+      "teacher_agreement": 0.649,
+      "labelled_rows": 2000,
+      "ground_truth_accuracy": 0.759,
+      "epoch": 8
+    }
+  ]
+}

examples/example-canonical-request.json

@@ -0,0 +1,15 @@
+{
+  "state": "The tennis player won the tournament after defeating the defending champion in the final match.",
+  "questions": {
+    "topic": {
+      "type": "choice",
+      "instructions": "Choose the topic that best describes this news item.",
+      "criteria": {
+        "world": "World",
+        "sports": "Sports",
+        "business": "Business",
+        "science": "Sci/Tech"
+      }
+    }
+  }
+}

examples/example-news-request.json

@@ -0,0 +1,15 @@
+{
+  "state": "The tennis player won the tournament after defeating the defending champion in the final match.",
+  "questions": {
+    "topic": {
+      "type": "choice",
+      "instructions": "Choose the topic that best describes this news item.",
+      "criteria": {
+        "world": "World: international events, politics and public affairs",
+        "sports": "Sports: matches, athletes and competitions",
+        "business": "Business: companies, markets and the economy",
+        "science": "Science and technology: research, computing and inventions"
+      }
+    }
+  }
+}

experiment-status.json

@@ -0,0 +1,32 @@
+{
+  "status": "experimental_not_promoted",
+  "subject_variant": "models/experimental-b",
+  "trained": true,
+  "quality_validated": false,
+  "dev_passed": false,
+  "final_evaluated": false,
+  "historical_test_3000_evaluated_for_these_weights": false,
+  "copa_evaluated_for_these_weights": false,
+  "model_payload_bytes": 87155068,
+  "protocol_sha256": "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366",
+  "failed_gate": {
+    "dataset": "snli",
+    "precision": "int8",
+    "metric": "fraction of text/style pairs with changing canonical prediction across cyclic rotations",
+    "A": 0.5801282051282052,
+    "B": 0.33653846153846156,
+    "required_B_max": 0.2900641025641026,
+    "A_unstable_cells": 181,
+    "B_unstable_cells": 105,
+    "total_cells": 312,
+    "maximum_allowed_integer_count": 90
+  },
+  "evidence": "benchmarks/ab/dev-report.json",
+  "limitations": [
+    "Positive gains on close reformulations only; general decision reliability not established.",
+    "Single seed (42); bootstrap intervals do not measure training variance.",
+    "DEV consumed; no post-result tuning or FINAL evaluation.",
+    "Fewer FP32/INT8 flips do not prove lower accuracy loss from quantization.",
+    "CPU compatibility and speed beyond Linux x86-64 i7-1260P not validated."
+  ]
+}

licenses/Apache-2.0.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

models/control-a/manifest.json

@@ -0,0 +1,32 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "parameters": 38223873,
+  "base_model": "google/bert_uncased_L-6_H-512_A-8",
+  "language": "en",
+  "max_sequence_length": 512,
+  "max_options": 20,
+  "question_types": [
+    "choice",
+    "score",
+    "noul"
+  ],
+  "quantization": "dynamic INT8 linear weights, per-channel, reduce_range=True; FP32 embeddings",
+  "opset": 18,
+  "trained": true,
+  "quality_validated": false,
+  "calibrated": false,
+  "provenance": {
+    "trained": true,
+    "quality_validated": false,
+    "arm": "A",
+    "steps": 5048,
+    "selection": "terminal checkpoint after exactly epoch 2, identically for A and B",
+    "start_sha256": "5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008",
+    "protocol_sha256": "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366",
+    "shared_cache_sha256": "2857aaa26b592d6a2add814e9764b841ca1cbe43479c9c085d6f56d9efb1843a",
+    "shared_schedule_sha256": "d495a21cc69ddc56bcaea92c0cb4db020f9120814e5653456c166fe093677896",
+    "seed": 42,
+    "bf16": true,
+    "elapsed_seconds": 304.6381041610148
+  }
+}

models/control-a/model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:30c633d3bd6742714f3198c5e6d3176c05b312e967c6da2523ad6f6ff25e43b6
+size 86443598

models/experimental-b/manifest.json

@@ -0,0 +1,32 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "parameters": 38223873,
+  "base_model": "google/bert_uncased_L-6_H-512_A-8",
+  "language": "en",
+  "max_sequence_length": 512,
+  "max_options": 20,
+  "question_types": [
+    "choice",
+    "score",
+    "noul"
+  ],
+  "quantization": "dynamic INT8 linear weights, per-channel, reduce_range=True; FP32 embeddings",
+  "opset": 18,
+  "trained": true,
+  "quality_validated": false,
+  "calibrated": false,
+  "provenance": {
+    "trained": true,
+    "quality_validated": false,
+    "arm": "B",
+    "steps": 5048,
+    "selection": "terminal checkpoint after exactly epoch 2, identically for A and B",
+    "start_sha256": "5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008",
+    "protocol_sha256": "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366",
+    "shared_cache_sha256": "2857aaa26b592d6a2add814e9764b841ca1cbe43479c9c085d6f56d9efb1843a",
+    "shared_schedule_sha256": "d495a21cc69ddc56bcaea92c0cb4db020f9120814e5653456c166fe093677896",
+    "seed": 42,
+    "bf16": true,
+    "elapsed_seconds": 288.4059911170043
+  }
+}

models/experimental-b/model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0aa20e2a2b2e2b054c4db668b98452362ce58968f24645a9f0f07543c0e1e31
+size 86443598

models/original/manifest.json

@@ -0,0 +1,111 @@
+{
+  "architecture": "BERT 6x512 + 1 decision layer + option-marker scorer",
+  "parameters": 38223873,
+  "base_model": "google/bert_uncased_L-6_H-512_A-8",
+  "language": "en",
+  "max_sequence_length": 512,
+  "max_options": 20,
+  "question_types": [
+    "choice",
+    "score",
+    "noul"
+  ],
+  "quantization": "dynamic INT8 linear weights, per-channel, reduce_range=True; FP32 embeddings",
+  "opset": 18,
+  "trained": true,
+  "quality_validated": false,
+  "calibrated": false,
+  "provenance": {
+    "trained": true,
+    "smoke_test": false,
+    "quality_validated": false,
+    "base_model": "google/bert_uncased_L-6_H-512_A-8",
+    "base_revision": "dd53ec6ca9d05e0a91b309c4e137f31988888071",
+    "seed": 42,
+    "best_epoch": 6,
+    "selection": "validation ground-truth accuracy; teacher KL as tiebreaker",
+    "train_sha256": "9f1e076aedb2bb9f9bfd353933d7af863a0943dd681660ff4856b16bec5b8e2b",
+    "validation_sha256": "8cfabd613dde690e6bbf8092132f0100ecbb1041ec996bfa1de9dd6fe9420e49",
+    "hyperparameters": {
+      "epochs": 8,
+      "batch_size": 32,
+      "encoder_learning_rate": 2e-05,
+      "head_learning_rate": 0.0001,
+      "label_weight": 2.0,
+      "bf16_training": true
+    },
+    "elapsed_seconds": 1027.408421811997,
+    "training_rows": 80758,
+    "validation_rows": 2000,
+    "truncated_training_states": 0,
+    "teacher_sources": [
+      "convaiinnovations/laya"
+    ],
+    "history": [
+      {
+        "rows": 2000,
+        "teacher_KL": 0.4694688792228699,
+        "teacher_agreement": 0.573,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.62,
+        "epoch": 1
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.4274449801445007,
+        "teacher_agreement": 0.6015,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.683,
+        "epoch": 2
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.4418224000930786,
+        "teacher_agreement": 0.636,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.7365,
+        "epoch": 3
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.4513157923221588,
+        "teacher_agreement": 0.6375,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.734,
+        "epoch": 4
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.49970576214790347,
+        "teacher_agreement": 0.6315,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.75,
+        "epoch": 5
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.46601298499107363,
+        "teacher_agreement": 0.647,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.7595,
+        "epoch": 6
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.5017643827199936,
+        "teacher_agreement": 0.649,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.759,
+        "epoch": 7
+      },
+      {
+        "rows": 2000,
+        "teacher_KL": 0.5062610116004944,
+        "teacher_agreement": 0.649,
+        "labelled_rows": 2000,
+        "ground_truth_accuracy": 0.759,
+        "epoch": 8
+      }
+    ]
+  }
+}

models/original/model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18b45a1a7979c4f98e93a4521bb59d0b1690c898670b44d484346b4849ab4fda
+size 86443598

requirements-runtime.txt

@@ -0,0 +1,3 @@
+onnxruntime==1.24.4
+numpy>=1.26,<3
+tokenizers>=0.22,<0.23

requirements-stats.txt

@@ -0,0 +1 @@
+numpy==2.5.3

research/exports/control-a.json

@@ -0,0 +1,46 @@
+{
+  "status": "trained_export_requires_quality_evaluation",
+  "parameters": 38223873,
+  "fp32_onnx_bytes": 153022850,
+  "int8_onnx_bytes": 86443598,
+  "tokenizer_bytes": 710348,
+  "payload_bytes": 87155068,
+  "payload_MB_decimal": 87.155068,
+  "payload_MiB_binary": 83.11754989624023,
+  "fits_80_to_90_MB": true,
+  "checks": [
+    {
+      "batch": 1,
+      "sequence": 24,
+      "options": 2,
+      "fp32_max_logit_error": 1.3709068298339844e-06,
+      "int8_max_probability_difference": 0.03501194715499878
+    },
+    {
+      "batch": 3,
+      "sequence": 96,
+      "options": 5,
+      "fp32_max_logit_error": 2.5033950805664062e-06,
+      "int8_max_probability_difference": 0.030787497758865356
+    },
+    {
+      "batch": 1,
+      "sequence": 256,
+      "options": 10,
+      "fp32_max_logit_error": 2.682209014892578e-06,
+      "int8_max_probability_difference": 0.014649853110313416
+    },
+    {
+      "batch": 1,
+      "sequence": 512,
+      "options": 20,
+      "fp32_max_logit_error": 1.862645149230957e-06,
+      "int8_max_probability_difference": 0.04665376618504524
+    }
+  ],
+  "limitations": [
+    "Checks validate export mechanics, not English decision quality.",
+    "Payload excludes the runtime, Python, temporary training files and RAM.",
+    "CPU portability beyond the tested machine requires separate validation."
+  ]
+}

research/exports/experimental-b.json

@@ -0,0 +1,46 @@
+{
+  "status": "trained_export_requires_quality_evaluation",
+  "parameters": 38223873,
+  "fp32_onnx_bytes": 153022850,
+  "int8_onnx_bytes": 86443598,
+  "tokenizer_bytes": 710348,
+  "payload_bytes": 87155068,
+  "payload_MB_decimal": 87.155068,
+  "payload_MiB_binary": 83.11754989624023,
+  "fits_80_to_90_MB": true,
+  "checks": [
+    {
+      "batch": 1,
+      "sequence": 24,
+      "options": 2,
+      "fp32_max_logit_error": 4.76837158203125e-07,
+      "int8_max_probability_difference": 0.006500035524368286
+    },
+    {
+      "batch": 3,
+      "sequence": 96,
+      "options": 5,
+      "fp32_max_logit_error": 3.159046173095703e-06,
+      "int8_max_probability_difference": 0.029584228992462158
+    },
+    {
+      "batch": 1,
+      "sequence": 256,
+      "options": 10,
+      "fp32_max_logit_error": 3.7550926208496094e-06,
+      "int8_max_probability_difference": 0.010657772421836853
+    },
+    {
+      "batch": 1,
+      "sequence": 512,
+      "options": 20,
+      "fp32_max_logit_error": 1.9669532775878906e-06,
+      "int8_max_probability_difference": 0.040227651596069336
+    }
+  ],
+  "limitations": [
+    "Checks validate export mechanics, not English decision quality.",
+    "Payload excludes the runtime, Python, temporary training files and RAM.",
+    "CPU portability beyond the tested machine requires separate validation."
+  ]
+}

research/original-selection.json

@@ -0,0 +1,29 @@
+{
+  "selected": "refined",
+  "criterion": "validation ground-truth accuracy; teacher KL as tiebreaker",
+  "same_holdout_input_sha256_verified": true,
+  "holdout_input_sha256": {
+    "validation": "3e2840cd09a2372e812dbc8f37904abf31dff4c2a74c09f5aec00b9c3bc2e1ce",
+    "test": "47cc633fd7e1ae38a52c271d2ca36ded1d9002e9a7f84a62b1dc7b3c1adf3cb9",
+    "ood": "cdad1edf8d687786bfaba10c0d3dfebb8216f94dba26afcb92d4a0699d8eac15"
+  },
+  "candidates": {
+    "baseline": {
+      "training_rows": 24000,
+      "best_epoch": 4,
+      "validation_accuracy": 0.627,
+      "teacher_KL": 0.43756801056861877
+    },
+    "refined": {
+      "training_rows": 80758,
+      "best_epoch": 6,
+      "validation_accuracy": 0.7595,
+      "teacher_KL": 0.46601298499107363
+    }
+  },
+  "development_limitations": [
+    "First-pass tests were inspected before increasing the training data; this is not a fully blind end-to-end benchmark.",
+    "Checkpoint and run selection use validation, never test or OOD accuracy.",
+    "All COPA data are excluded from fitting and checkpoint selection."
+  ]
+}

research/references.json

@@ -0,0 +1,87 @@
+{
+  "consulted_on": "2026-09-26",
+  "scope": "Primary-source reading for a design review, not a reproduction of published model benchmarks.",
+  "sources": [
+    {
+      "title": "GLiClass: Generalist Lightweight Model for Sequence Classification Tasks",
+      "authors": "Ihor Stepanov et al.",
+      "year": 2025,
+      "urls": [
+        "https://arxiv.org/html/2508.07662v1",
+        "https://huggingface.co/api/papers/2508.07662",
+        "https://github.com/Knowledgator/GLiClass"
+      ],
+      "verified_sections": [
+        "Abstract",
+        "Architecture: scoring text and label representations",
+        "Training data: logic/NLI and text-length/label-density streams",
+        "Training framework"
+      ],
+      "findings": [
+        "The paper describes pooled text and label representations scored using a dot product or a learned scorer.",
+        "The training description includes logic/NLI examples and deliberate variation of text length and label density.",
+        "These are possible design directions, not evidence that changing our 38M-parameter model will improve its generalization.",
+        "Dataset grouping before augmentation remains necessary in our own protocol; do not copy a random split after duplication."
+      ]
+    },
+    {
+      "title": "GLiClass small v1.0 model card and configuration",
+      "urls": [
+        "https://huggingface.co/knowledgator/gliclass-small-v1.0",
+        "https://huggingface.co/knowledgator/gliclass-small-v1.0/raw/main/config.json"
+      ],
+      "verified_facts": {
+        "card_parameter_count": 144000000,
+        "encoder_type": "deberta-v2",
+        "hidden_size": 768,
+        "layers": 6,
+        "vocabulary": 128003,
+        "word_embedding_parameters": 98306304
+      },
+      "findings": [
+        "The card reports Apache-2.0 and synthetic training data; this does not resolve the licenses of our current training datasets.",
+        "The adjective small does not imply an 80–90 MB INT8 artifact: this checkpoint's word embeddings alone require about 98.3 MB at one byte per parameter, before scales and the rest of the network.",
+        "No GLiClass weights were downloaded, trained, or benchmarked during this review."
+      ]
+    },
+    {
+      "title": "Multitask Prompted Training Enables Zero-Shot Task Generalization",
+      "authors": "Victor Sanh et al.",
+      "year": 2021,
+      "urls": [
+        "https://huggingface.co/papers/2110.08207",
+        "https://huggingface.co/api/papers/2110.08207"
+      ],
+      "verified_sections": [
+        "Title, publication metadata and abstract"
+      ],
+      "findings": [
+        "The abstract describes explicit multitask learning with diverse prompt formulations and evaluation on fully held-out tasks.",
+        "This motivates a task-held-out evaluation protocol; it does not predict a score for our compact encoder, whose architecture and scale differ.",
+        "Only the abstract was inspected for this source, not a full reproduction of its experiments."
+      ]
+    },
+    {
+      "title": "ONNX Runtime — Quantize ONNX models",
+      "urls": [
+        "https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html"
+      ],
+      "verified_sections": [
+        "Quantization debugging",
+        "Dynamic versus static quantization",
+        "Data type selection",
+        "Reduce-range and per-channel"
+      ],
+      "findings": [
+        "Quantization can harm accuracy; comparing weights and activations can help localize errors and identify tensors to keep unquantized.",
+        "The documentation recommends dynamic quantization in general for transformers.",
+        "The current model already uses per-channel quantization and reduce_range=True.",
+        "Small aggregate accuracy changes do not establish local parity, probability calibration, or portability across every CPU."
+      ]
+    }
+  ],
+  "failed_research_routes": [
+    "Both Hugging Face paper_search connector calls returned 'Tool paper_search not found'. Research continued through primary-source HTTP requests.",
+    "An initial HTML parsing attempt required an unavailable bs4 module; the successful read used Python standard-library HTML cleanup without installing dependencies."
+  ]
+}

research/runtime-references-checked.json

@@ -0,0 +1,18 @@
+[
+  {
+    "url": "https://raw.githubusercontent.com/pytorch/pytorch/v2.8.0/docs/source/notes/randomness.rst",
+    "checked_at_utc": "2026-09-26T11:33:44.146287+00:00",
+    "snippets": [
+      "_reproducibility: Reproducibility =============== Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds. However, there are some steps you can take to limit the number of sources of nondeterministic behavior for a specific platform, device, and PyTorch release. First, you can control sources of randomness",
+      "e between CPU and GPU executions, even when using identical seeds. However, there are some steps you can take to limit the number of sources of nondeterministic behavior for a specific platform, device, and PyTorch release. First, you can control sources of randomness that can cause multiple executions of your application to behave differently. Second, you can configure PyTorch to avoid using nondeterministic algorithms for some operations, so that multiple cal"
+    ]
+  },
+  {
+    "url": "https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html",
+    "checked_at_utc": "2026-09-26T11:33:45.102559+00:00",
+    "snippets": [
+      "efer to the Method selection section. When to use reduce-range and per-channel quantization? Reduce-range will quantize the weights to 7 bits. It is designed for the U8S8 format on AVX2 and AVX512 (non-VNNI) machines to mitigate saturation issues . This is not needed on machines supporting VNNI. Per-channel quantization can improve the accuracy for models whose weight ranges are large. Try it if the accuracy loss is large. On AVX2 and AVX512 machines, you wi",
+      " 0 uniquely after quantization, it will result in accuracy errors. ONNX quantization representation format There are two ways to represent quantized ONNX models: Operator-oriented (QOperator) : All the quantized operators have their own ONNX definitions, like QLinearConv, MatMulInteger and etc. Tensor-oriented (QDQ; Quantize and DeQuantize) : This format inserts DeQuantizeLinear(QuantizeLinear(tensor)) between the original operators to simulate the quant"
+    ]
+  }
+]

research/src/ab_evaluate.py

@@ -0,0 +1,211 @@
+import argparse
+import json
+import time
+from collections import defaultdict
+from pathlib import Path
+
+import numpy as np
+
+from ab_experiment import checked_feature, family_key, file_hash, write_json, write_rows
+from schema import collate, options_for, read_rows
+
+
+def clustered_change(rows, value):
+    grouped = defaultdict(list)
+    for row in rows:
+        grouped[row["family_group"]].append(float(value(row)))
+    if not grouped:
+        raise ValueError("Empty paired comparison")
+    groups = [grouped[key] for key in sorted(grouped)]
+    sums = np.array([sum(group) for group in groups])
+    counts = np.array([len(group) for group in groups])
+    generator = np.random.default_rng(20260926)
+    draws = []
+    for offset in range(0, 10000, 500):
+        indices = generator.integers(0, len(groups), size=(500, len(groups)))
+        draws.extend((sums[indices].sum(axis=1) / counts[indices].sum(axis=1)).tolist())
+    return {"delta": float(sums.sum() / counts.sum()), "ci95": np.quantile(draws, [0.025, 0.975]).tolist(), "clusters": len(groups)}
+
+
+def accuracy(rows, name):
+    return float(np.mean([row["prediction"][name] == row["canonical_label"] for row in rows]))
+
+
+def confusion(rows, name):
+    labels = max(max(row["canonical_label"], row["prediction"][name]) for row in rows) + 1
+    matrix = [[0] * labels for _ in range(labels)]
+    for row in rows:
+        matrix[row["canonical_label"]][row["prediction"][name]] += 1
+    return matrix
+
+
+def instability(rows, name):
+    grouped = defaultdict(set)
+    for row in rows:
+        grouped[(row["family_group"], row["style"])].add(row["prediction"][name])
+    return float(np.mean([len(predictions) > 1 for predictions in grouped.values()]))
+
+
+def flips(row, arm):
+    return int(row["prediction"][arm + "_fp32"] != row["prediction"][arm + "_int8"])
+
+
+def robustness_summary(rows):
+    summary = {"domains": {}, "passed": True}
+    for domain in ["ag_news", "snli"]:
+        subset = [row for row in rows if row["dataset"] == domain]
+        result = {}
+        for precision in ["fp32", "int8"]:
+            left, right = "A_" + precision, "B_" + precision
+            gain = clustered_change(subset, lambda row: int(row["prediction"][right] == row["canonical_label"]) - int(row["prediction"][left] == row["canonical_label"]))
+            order_a, order_b = instability(subset, left), instability(subset, right)
+            passed = gain["delta"] >= 0.10 and gain["ci95"][0] > 0 and order_b <= order_a / 2
+            result[precision] = {"A_accuracy": accuracy(subset, left), "B_accuracy": accuracy(subset, right),
+                                 "gain": gain, "A_order_instability": order_a, "B_order_instability": order_b, "passed": passed}
+            summary["passed"] &= passed
+        change = clustered_change(subset, lambda row: flips(row, "B") - flips(row, "A"))
+        result["quantization"] = {"A_flip_rate": float(np.mean([flips(row, "A") for row in subset])),
+                                  "B_flip_rate": float(np.mean([flips(row, "B") for row in subset])),
+                                  "change": change, "passed": change["ci95"][1] <= 0.02}
+        summary["passed"] &= result["quantization"]["passed"]
+        result["styles"] = {}
+        for style in sorted({row["style"] for row in subset}):
+            cell = [row for row in subset if row["style"] == style]
+            result["styles"][str(style)] = {
+                name: {"accuracy": accuracy(cell, name), "order_instability": instability(cell, name), "confusion": confusion(cell, name)}
+                for name in ["A_fp32", "A_int8", "B_fp32", "B_int8"]
+            }
+        summary["domains"][domain] = result
+    return summary
+
+
+def fixed_summary(rows):
+    summary = {"passed": True}
+    for precision in ["fp32", "int8"]:
+        left, right = "A_" + precision, "B_" + precision
+        metric = lambda row: int(row["prediction"][right] == row["canonical_label"]) - int(row["prediction"][left] == row["canonical_label"])
+        gain = clustered_change(rows, metric)
+        tasks = {}
+        for domain in sorted({row["dataset"] for row in rows}):
+            subset = [row for row in rows if row["dataset"] == domain]
+            tasks[domain] = {"A_accuracy": accuracy(subset, left), "B_accuracy": accuracy(subset, right), "delta": float(np.mean([metric(row) for row in subset]))}
+        passed = gain["ci95"][0] >= -0.02 and all(task["delta"] >= -0.04 for task in tasks.values())
+        summary[precision] = {"A_accuracy": accuracy(rows, left), "B_accuracy": accuracy(rows, right), "gain": gain, "tasks": tasks, "passed": passed}
+        summary["passed"] &= passed
+    change = clustered_change(rows, lambda row: flips(row, "B") - flips(row, "A"))
+    summary["quantization"] = {"change": change, "passed": change["ci95"][1] <= 0.02}
+    summary["passed"] &= summary["quantization"]["passed"]
+    return summary
+
+
+def fixed_requests(rows):
+    output = []
+    for row in rows:
+        question = row["questions"]["decision"]
+        labels, _ = options_for(question)
+        expected = row["expected"]["decision"]
+        canonical = int(expected) if question["type"] == "noul" else labels.index(str(expected))
+        output.append({
+            "request_id": row["source_group"] + "|fixed", "source_group": row["source_group"],
+            "family_group": row["dataset"] + ":" + family_key(row), "dataset": row["dataset"],
+            "state": row["state"], "question": question, "canonical_label": canonical,
+            "semantic_order": list(range(len(labels))), "style": "fixed", "rotation": 0,
+        })
+    return output
+
+
+def infer(requests, result_directory):
+    import onnxruntime as ort
+    import torch
+    from safetensors.torch import load_file
+    from tokenizers import Tokenizer
+    from model import MiniDecisionModel
+
+    torch.set_num_threads(4)
+    tokenizer = Tokenizer.from_file(str(result_directory / "A/tokenizer.json"))
+    features = [checked_feature(row, tokenizer) for row in requests]
+    ordered = sorted(range(len(requests)), key=lambda index: len(features[index]["input_ids"]))
+    records = [{**row, "prediction": {}, "probabilities": {}} for row in requests]
+    timings = {}
+    for arm in ["A", "B"]:
+        if file_hash(result_directory / arm / "tokenizer.json") != file_hash(result_directory / "A/tokenizer.json"):
+            raise ValueError("A and B tokenizers differ")
+        model = MiniDecisionModel()
+        model.load_state_dict(load_file(result_directory / arm / "model.safetensors"))
+        model.eval()
+        settings = ort.SessionOptions()
+        settings.intra_op_num_threads = 4
+        settings.inter_op_num_threads = 1
+        session = ort.InferenceSession(str(result_directory / arm / "model/model.int8.onnx"), settings, providers=["CPUExecutionProvider"])
+        started = time.perf_counter()
+        with torch.inference_mode():
+            for offset in range(0, len(ordered), 8):
+                indices = ordered[offset:offset + 8]
+                arrays = collate([features[index] for index in indices])
+                floating = model(**{name: torch.from_numpy(value) for name, value in arrays.items()}).numpy()
+                quantized = session.run(None, arrays)[0]
+                for name, logits in [(arm + "_fp32", floating), (arm + "_int8", quantized)]:
+                    if not np.isfinite(logits).all():
+                        raise ValueError("Nonfinite logits")
+                    for position, index in enumerate(indices):
+                        order = requests[index]["semantic_order"]
+                        values = torch.tensor(logits[position, :len(order)]).softmax(-1).numpy()
+                        canonical = np.empty(len(order))
+                        canonical[order] = values
+                        records[index]["probabilities"][name] = canonical.tolist()
+                        records[index]["prediction"][name] = int(canonical.argmax())
+                if offset % 800 == 0:
+                    print(json.dumps({"evaluation_arm": arm, "requests": min(offset + 8, len(ordered)), "seconds": time.perf_counter() - started}), flush=True)
+        timings[arm] = time.perf_counter() - started
+        del model, session
+    return records, timings
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--data", type=Path, required=True)
+    parser.add_argument("--results", type=Path, required=True)
+    parser.add_argument("--split", choices=["dev", "final"], default="dev")
+    arguments = parser.parse_args()
+    directory = arguments.data
+    protocol = json.loads((directory / "protocol.json").read_text())
+    from ab_train import PROTOCOL_SHA256
+    if file_hash(directory / "protocol.json") != PROTOCOL_SHA256:
+        raise ValueError("Frozen evaluation protocol changed")
+    if file_hash(directory / "validation.jsonl") != protocol["files"]["validation.jsonl"]:
+        raise ValueError("Fixed validation changed")
+    provenance = [json.loads((arguments.results / arm / "training.json").read_text()) for arm in ["A", "B"]]
+    for key in ["start_sha256", "protocol_sha256", "shared_cache_sha256", "shared_schedule_sha256", "seed", "steps", "selection", "bf16"]:
+        if provenance[0][key] != provenance[1][key]:
+            raise ValueError(f"A/B training provenance differs: {key}")
+    if provenance[0]["protocol_sha256"] != PROTOCOL_SHA256 or provenance[0]["steps"] != protocol["steps_per_arm"]:
+        raise ValueError("Training did not follow this protocol")
+    if arguments.split == "final" and not json.loads((directory / "dev-report.json").read_text())["passed"]:
+        raise ValueError("FINAL remains sealed because DEV did not pass")
+    destination = directory / f"{arguments.split}-predictions.jsonl"
+    if destination.exists() or (directory / f"{arguments.split}-report.json").exists():
+        raise FileExistsError("Evaluation already exists")
+    request_file = directory / f"{arguments.split}-requests.jsonl"
+    if file_hash(request_file) != protocol["files"][request_file.name]:
+        raise ValueError("Evaluation input changed")
+    requests = read_rows(request_file)
+    fixed = fixed_requests(read_rows(directory / "validation.jsonl")) if arguments.split == "dev" else []
+    predictions, timings = infer(requests + fixed, arguments.results)
+    write_rows(destination, predictions)
+    robust = robustness_summary(predictions[:len(requests)])
+    regression = fixed_summary(predictions[len(requests):]) if fixed else None
+    report = {
+        "protocol_sha256": file_hash(directory / "protocol.json"), "split": arguments.split,
+        "texts_per_domain": 156, "variant_requests": len(requests), "fixed_requests": len(fixed),
+        "passed": robust["passed"] and (regression is None or regression["passed"]),
+        "robustness": robust, "fixed_validation": regression, "cpu_evaluation_seconds": timings,
+        "prediction_sha256": file_hash(destination), "quality_validated_for_general_decisions": False,
+        "instability_definition": "Fraction of text/style pairs whose canonical prediction changes across cyclic rotations; average over styles",
+        "uncertainty": "Paired family-cluster percentile bootstrap, 10000 replicates, seed20260926; not uncertainty over training seeds",
+    }
+    write_json(directory / f"{arguments.split}-report.json", report)
+    print(json.dumps(report, indent=2), flush=True)
+
+
+if __name__ == "__main__":
+    main()

research/src/ab_experiment.py

@@ -0,0 +1,212 @@
+import copy
+import hashlib
+import json
+import math
+import random
+import sys
+from pathlib import Path
+
+RELEASE = Path(__file__).resolve().parents[2] / "outputs/laya-mini-cpu"
+if RELEASE.is_dir():
+    sys.path.insert(0, str(RELEASE))
+
+import build_corpus
+from schema import encode_question, normalized_probabilities, options_for
+
+SEED = 42
+TEACHER_REVISION = "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"
+START_SHA256 = "5f3cfb2292b5ffb24a7e44e0c7dbad6d80e483bce439b02d431124b8572b4008"
+DOMAINS = ("ag_news", "snli")
+
+
+def file_hash(path):
+    with Path(path).open("rb") as handle:
+        return hashlib.file_digest(handle, "sha256").hexdigest()
+
+
+def write_json(path, value):
+    with Path(path).open("x", encoding="utf-8") as handle:
+        json.dump(value, handle, ensure_ascii=False, indent=2)
+        handle.write("\n")
+
+
+def write_rows(path, rows):
+    with Path(path).open("x", encoding="utf-8") as handle:
+        for row in rows:
+            handle.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
+
+
+def validate_pools(pools):
+    if set(pools) != {"train", "dev", "final"}:
+        raise ValueError("Expected TRAIN/DEV/FINAL pools")
+    seen = set()
+    for split, domains in pools.items():
+        if set(domains) != set(DOMAINS):
+            raise ValueError("Missing domain")
+        for dataset, variants in domains.items():
+            if len(variants) < 2:
+                raise ValueError("At least two variants required per pool")
+            for descriptions in variants:
+                if len(descriptions) != len(build_corpus.CHOICES[dataset]):
+                    raise ValueError("Incorrect class count")
+                for description in descriptions:
+                    normalized = build_corpus.normalize_text(description).casefold()
+                    if normalized in seen:
+                        raise ValueError("Description pools must be disjoint")
+                    seen.add(normalized)
+
+
+def class_order(row, descriptions=None):
+    descriptions = descriptions or build_corpus.CHOICES[row["dataset"]]
+    question = row.get("question") or row["questions"]["decision"]
+    order = [list(descriptions).index(value) for value in question["criteria"].values()]
+    if sorted(order) != list(range(len(descriptions))):
+        raise ValueError("Criteria must be a class permutation")
+    return order
+
+
+def replace_descriptions(row, descriptions):
+    result = copy.deepcopy(row)
+    criteria = result["questions"]["decision"]["criteria"]
+    order = class_order(row)
+    if len(descriptions) != len(order):
+        raise ValueError("Description count mismatch")
+    result["questions"]["decision"]["criteria"] = {
+        key: descriptions[semantic] for key, semantic in zip(criteria, order)
+    }
+    return result
+
+
+def augmentation_groups(rows, seed):
+    selected = set()
+    for dataset in DOMAINS:
+        groups = [row["source_group"] for row in rows if row["dataset"] == dataset]
+        if len(groups) != len(set(groups)) or len(groups) % 2:
+            raise ValueError("Need unique even-sized domain pools for exactly 50 percent")
+        groups.sort(key=lambda group: build_corpus.digest(seed, "augment", group))
+        selected.update(groups[:len(groups) // 2])
+    return selected
+
+
+def remap_probabilities(values, source_order, target_order):
+    count = len(source_order)
+    if sorted(source_order) != list(range(count)) or sorted(target_order) != list(range(count)):
+        raise ValueError("Class orders must be permutations")
+    probabilities = normalized_probabilities(values, count)
+    canonical = dict(zip(source_order, probabilities.tolist()))
+    return [canonical[index] for index in target_order]
+
+
+def validate_cache(fixed_rows, annotated_rows):
+    expected = {row["source_group"]: row for row in fixed_rows}
+    observed = {row["source_group"]: row for row in annotated_rows}
+    if len(expected) != len(fixed_rows) or len(observed) != len(annotated_rows) or expected.keys() != observed.keys():
+        raise ValueError("Teacher cache groups differ from fixed corpus")
+    for group, fixed in expected.items():
+        cached = observed[group]
+        question = fixed["questions"]["decision"]
+        if cached["state"] != fixed["state"] or cached["question"] != question:
+            raise ValueError("Teacher cache does not match the fixed input")
+        if "label" in cached and cached["label"] != fixed["expected"]["decision"]:
+            raise ValueError("Teacher cache label differs")
+        labels, _ = options_for(question)
+        if options_for(cached["question"])[0] != labels:
+            raise ValueError("Teacher cache probability order differs from the fixed input")
+        normalized_probabilities(cached["probabilities"], len(labels))
+    return observed
+
+
+def family_key(row):
+    text = row["state"]
+    if row["dataset"] == "snli":
+        if not text.startswith("Premise: ") or "\nHypothesis: " not in text:
+            raise ValueError("Invalid SNLI state")
+        text = text[len("Premise: "):].split("\nHypothesis: ", 1)[0]
+    return build_corpus.state_key(text)
+
+
+def checked_feature(row, tokenizer):
+    question = row.get("question") or row["questions"]["decision"]
+    feature = encode_question(tokenizer, row["state"], question)
+    if feature["truncated_state"]:
+        raise ValueError("A/B inputs must not be truncated")
+    return feature
+
+
+def gold_class(row):
+    question = row["questions"]["decision"]
+    labels, _ = options_for(question)
+    return class_order(row)[labels.index(str(row["expected"]["decision"]))]
+
+
+def select_holdout(candidates, count, forbidden_families, split):
+    if not candidates:
+        raise ValueError("No candidates")
+    classes = len(build_corpus.CHOICES[candidates[0]["dataset"]])
+    if count < 1 or count % classes:
+        raise ValueError("Balanced selection requires a multiple of class count")
+    used = set(forbidden_families)
+    ordered = sorted(candidates, key=lambda row: build_corpus.digest(SEED, split, row["source_group"]))
+    selected = []
+    for label in range(classes):
+        chosen = []
+        for row in ordered:
+            family = family_key(row)
+            if gold_class(row) == label and family not in used:
+                chosen.append(row)
+                used.add(family)
+            if len(chosen) == count // classes:
+                break
+        if len(chosen) != count // classes:
+            raise ValueError("Insufficient disjoint families")
+        selected.extend(chosen)
+    return selected
+
+
+def evaluation_requests(records, pool):
+    requests = []
+    for row in records:
+        order = class_order(row)
+        criteria = row["questions"]["decision"]["criteria"]
+        key_by_class = dict(zip(order, criteria))
+        for style, descriptions in enumerate(pool[row["dataset"]]):
+            for rotation in range(len(order)):
+                rotated = order[rotation:] + order[:rotation]
+                question = {**row["questions"]["decision"], "criteria": {
+                    key_by_class[label]: descriptions[label] for label in rotated
+                }}
+                requests.append({
+                    "request_id": f"{row['source_group']}|style{style}|{rotation}",
+                    "source_group": row["source_group"], "family_group": family_key(row),
+                    "dataset": row["dataset"], "state": row["state"], "question": question,
+                    "canonical_label": gold_class(row), "label": row["expected"]["decision"],
+                    "semantic_order": rotated, "style": style, "rotation": rotation,
+                })
+    return requests
+
+
+def reserved_cost(hourly_rate, maximum_hours, down_per_gb, up_per_gb):
+    values = [hourly_rate, maximum_hours, down_per_gb, up_per_gb]
+    if not all(math.isfinite(value) and value >= 0 for value in values):
+        raise ValueError("Invalid cost values")
+    return hourly_rate * (maximum_hours + 2) + 20 * (down_per_gb + up_per_gb) + 0.2
+
+
+def budget_allowed(prior_spend, reservation):
+    return (all(math.isfinite(value) and value >= 0 for value in [prior_spend, reservation])
+            and reservation <= 3 and prior_spend + reservation <= 10)
+
+
+def batch_indices(lengths, batch_size, seed, epoch):
+    if batch_size < 1 or not lengths:
+        raise ValueError("Need nonempty data and positive batch size")
+    generator = random.Random(seed + epoch)
+    indices = list(range(len(lengths)))
+    generator.shuffle(indices)
+    batches = []
+    window = batch_size * 20
+    for offset in range(0, len(indices), window):
+        ordered = sorted(indices[offset:offset + window], key=lambda index: lengths[index])
+        batches.extend(ordered[start:start + batch_size] for start in range(0, len(ordered), batch_size))
+    generator.shuffle(batches)
+    return batches

research/src/ab_train.py

@@ -0,0 +1,334 @@
+import argparse
+import copy
+import gc
+import json
+import math
+import random
+import shutil
+import time
+from pathlib import Path
+
+import numpy as np
+import torch
+from safetensors.torch import load_file, save_file
+from tokenizers import Tokenizer
+
+from ab_experiment import (
+    DOMAINS, START_SHA256, TEACHER_REVISION, batch_indices, checked_feature,
+    class_order, file_hash, remap_probabilities, validate_cache, write_json, write_rows,
+)
+from model import MiniDecisionModel
+from pipeline import answer_probabilities, distillation_loss, dummy_inputs, export_model, prepare_rows, training_batch
+from schema import normalized_probabilities, options_for, read_rows
+
+
+PROTOCOL_SHA256 = "143e0890279a6fe339b8a0cc6f394c61e9d806e9b9b1800272418716f3a5b366"
+
+
+def prepare_pair(fixed, variants, cached, tokenizer, pools):
+    mapping = validate_cache(fixed, cached)
+    if len(fixed) != len(variants):
+        raise ValueError("A and B counts differ")
+    paired = {"A": [], "B": []}
+    for original, variant in zip(fixed, variants):
+        restored = copy.deepcopy(variant)
+        if original["dataset"] in DOMAINS:
+            if list(original["questions"]["decision"]["criteria"]) != list(variant["questions"]["decision"]["criteria"]):
+                raise ValueError("A/B changed key order")
+            domain = original["dataset"]
+            for key, semantic in zip(original["questions"]["decision"]["criteria"], class_order(original)):
+                allowed = {descriptions[semantic] for descriptions in pools["train"][domain]}
+                allowed.add(original["questions"]["decision"]["criteria"][key])
+                if variant["questions"]["decision"]["criteria"][key] not in allowed:
+                    raise ValueError("Variant description is assigned to the wrong class")
+            restored["questions"]["decision"]["criteria"] = original["questions"]["decision"]["criteria"]
+        if restored != original:
+            raise ValueError("A/B differ beyond allowed descriptions")
+        target = mapping[original["source_group"]]
+        values = target["probabilities"]
+        if original["dataset"] in DOMAINS:
+            order = class_order(original)
+            values = remap_probabilities(values, order, order)
+        for arm, row in [("A", original), ("B", variant)]:
+            feature = checked_feature(row, tokenizer)
+            annotation = {**target, "question": row["questions"]["decision"], "probabilities": values}
+            prepared = prepare_rows([annotation], tokenizer)[0]
+            paired[arm].append((feature, prepared[1], prepared[2]))
+    return paired
+
+
+def shared_batch(paired, indices, arm, device):
+    inputs, targets, labels = training_batch([paired[arm][index] for index in indices], device)
+    length = max(len(paired[version][index][0]["input_ids"]) for version in ["A", "B"] for index in indices)
+    padding = length - inputs["input_ids"].shape[1]
+    for name in ["input_ids", "attention_mask"]:
+        inputs[name] = torch.nn.functional.pad(inputs[name], (0, padding), value=0)
+    return inputs, targets, labels
+
+
+def encode_teacher(teacher, row):
+    from laya.common import build_sequence
+
+    question = row["questions"]["decision"]
+    teacher._check_question("decision", question)
+    internal = {"decision": teacher._to_internal(question)}
+    items = teacher._encode_state(row["state"], ["decision"], internal)
+    full_ids, markers = build_sequence(teacher.tok, row["state"], internal["decision"], 8192, 8192)
+    if len(items) != 1 or items[0]["ids"] != full_ids or items[0]["markers"] != markers:
+        raise ValueError(f"Teacher would truncate {row['source_group']}")
+    return row, items, internal
+
+
+@torch.inference_mode()
+def teacher_batch(teacher, entries):
+    from laya.common import collate_items
+
+    batch = collate_items([entry[1] for entry in entries], teacher.tok.pad_token_id)
+    logits, actions = teacher._forward(batch)
+    if (teacher.device.type != "cuda" or teacher.amp_enabled
+            or torch.backends.cuda.matmul.allow_tf32 or torch.backends.cudnn.allow_tf32
+            or torch.get_float32_matmul_precision() != "highest"
+            or next(teacher.model.parameters()).dtype != torch.float32):
+        raise RuntimeError("Teacher precision/device changed")
+    records = []
+    for position, (row, items, internal) in enumerate(entries):
+        answer = teacher._decode_answers(logits, actions, items, ["decision"], internal, position)["decision"]
+        question = row["questions"]["decision"]
+        labels, _ = options_for(question)
+        values = answer_probabilities(answer, question, labels)
+        records.append({
+            "source_group": row["source_group"], "dataset": row["dataset"],
+            "state": row["state"], "question": question, "label": row["expected"]["decision"],
+            "probabilities": normalized_probabilities(values, len(labels)).tolist(),
+        })
+    return records
+
+
+def remaining_seconds(deadline):
+    return deadline - time.time()
+
+
+def make_cache(directory, deadline):
+    from huggingface_hub import snapshot_download
+    import laya
+
+    if (directory / "teacher-cache.jsonl").exists():
+        raise FileExistsError("Teacher cache already exists; do not regenerate")
+    rows = read_rows(directory / "train.jsonl")
+    if len({row["source_group"] for row in rows}) != len(rows):
+        raise ValueError("Cache needs unique row-level source groups")
+    location = snapshot_download("convaiinnovations/laya", revision=TEACHER_REVISION, token=False,
+                                 allow_patterns=["model.safetensors", "rl_agent_config.json", "encoder/config.json", "tokenizer/*"])
+    teacher_hash = file_hash(Path(location) / "model.safetensors")
+    if teacher_hash != "891102d372688fc2a094dac56a384bc537b87c63f21f9f3dac0be2b7cbc8d86c":
+        raise ValueError("Pinned teacher weights differ")
+    teacher = laya.load(location, device="cuda", token=False)
+    teacher.amp_enabled = False
+    teacher.dtype = torch.float32
+    teacher.model.float().eval()
+    torch.backends.cuda.matmul.allow_tf32 = False
+    torch.backends.cudnn.allow_tf32 = False
+    torch.set_float32_matmul_precision("highest")
+    started = time.perf_counter()
+    encoded = [encode_teacher(teacher, row) for row in rows]
+    encoded.sort(key=lambda entry: len(entry[1][0]["ids"]))
+    preprocessing = time.perf_counter() - started
+    pilot = [encoded[index] for index in np.linspace(0, len(encoded) - 1, 64, dtype=int)]
+    teacher_batch(teacher, pilot[:16])
+    torch.cuda.synchronize()
+    started = time.perf_counter()
+    results = [record for offset in range(0, len(pilot), 16) for record in teacher_batch(teacher, pilot[offset:offset + 16])]
+    torch.cuda.synchronize()
+    elapsed = time.perf_counter() - started
+    checks = []
+    covered = set()
+    for entry, result in zip(pilot, results):
+        row = entry[0]
+        kind = row["questions"]["decision"]["type"]
+        question = row["questions"]["decision"]
+        answer = teacher.predict(row["state"], {"decision": question})["answers"]["decision"]
+        labels, _ = options_for(question)
+        reference = normalized_probabilities(answer_probabilities(answer, question, labels), len(labels))
+        difference = float(np.max(np.abs(reference - result["probabilities"])))
+        if difference > 0.001 or int(reference.argmax()) != int(np.argmax(result["probabilities"])):
+            raise ValueError("Teacher batch/public mismatch")
+        checks.append({"type": kind, "max_probability_delta": difference})
+        covered.add(kind)
+    estimate = elapsed / len(pilot) * len(encoded) * 2
+    if covered != {row["questions"]["decision"]["type"] for row in rows}:
+        raise ValueError("Teacher pilot misses a question type")
+    training_estimate = json.loads((directory / "pre-cache-training-pilot.json").read_text())["both_arms_seconds_conservative"]
+    write_json(directory / "teacher-pilot.json", {
+        "teacher_revision": TEACHER_REVISION, "teacher_sha256": teacher_hash,
+        "precision": "FP32 CUDA, no autocast, no TF32", "pilot_rows": 64,
+        "pilot_seconds": elapsed, "preprocessing_seconds": preprocessing,
+        "conservative_cache_seconds": estimate, "public_parity": checks,
+        "truncated_teacher_rows": 0, "maximum_teacher_tokens": max(len(entry[1][0]["ids"]) for entry in encoded),
+    })
+    if remaining_seconds(deadline) < estimate + training_estimate + 900:
+        raise RuntimeError("Measured annotation throughput leaves insufficient time for both arms and collection")
+    started = time.perf_counter()
+    output_path = directory / "teacher-cache.jsonl"
+    with output_path.open("x") as handle:
+        for offset in range(0, len(encoded), 16):
+            if remaining_seconds(deadline) < training_estimate + 900:
+                raise RuntimeError("Cache time bound reached")
+            for record in teacher_batch(teacher, encoded[offset:offset + 16]):
+                handle.write(json.dumps(record, separators=(",", ":"), ensure_ascii=False) + "\n")
+            if offset % 4000 == 0:
+                handle.flush()
+                print(json.dumps({"cache_rows": min(offset + 16, len(encoded)), "elapsed": time.perf_counter() - started}), flush=True)
+    validate_cache(rows, read_rows(output_path))
+    write_json(directory / "teacher-cache-meta.json", {
+        "rows": len(rows), "fixed_input_sha256": file_hash(directory / "train.jsonl"),
+        "cache_sha256": file_hash(output_path), "teacher_revision": TEACHER_REVISION,
+        "precision": "FP32 CUDA, no autocast, no TF32", "elapsed_seconds": time.perf_counter() - started,
+    })
+
+
+def fit(directory, checkpoint, deadline):
+    protocol = json.loads((directory / "protocol.json").read_text())
+    if file_hash(checkpoint / "model.safetensors") != START_SHA256:
+        raise ValueError("Wrong initial checkpoint")
+    if any((directory / arm).exists() for arm in ["A", "B"]):
+        raise FileExistsError("A/B output already exists; do not restart the experiment")
+    cache_meta = json.loads((directory / "teacher-cache-meta.json").read_text())
+    if file_hash(directory / "teacher-cache.jsonl") != cache_meta["cache_sha256"]:
+        raise ValueError("Teacher cache changed since verification")
+    fixed, variants = read_rows(directory / "train.jsonl"), read_rows(directory / "train-B.jsonl")
+    tokenizer = Tokenizer.from_file(str(checkpoint / "tokenizer.json"))
+    pools = json.loads((directory / "descriptions.json").read_text())
+    paired = prepare_pair(fixed, variants, read_rows(directory / "teacher-cache.jsonl"), tokenizer, pools)
+    lengths = [len(row[0]["input_ids"]) for row in paired["A"]]
+    schedules = [batch_indices(lengths, protocol["batch_size"], protocol["seed"], epoch) for epoch in range(protocol["epochs"])]
+    total_steps = sum(len(batches) for batches in schedules)
+    if total_steps != protocol["steps_per_arm"]:
+        raise ValueError("Actual step count differs from protocol")
+    write_json(directory / "batch-schedule.json", schedules)
+    schedule_hash = file_hash(directory / "batch-schedule.json")
+    use_amp = torch.cuda.is_bf16_supported()
+    torch.set_float32_matmul_precision("high")
+    for arm in ["A", "B"]:
+        random.seed(protocol["seed"])
+        np.random.seed(protocol["seed"])
+        torch.manual_seed(protocol["seed"])
+        torch.cuda.manual_seed_all(protocol["seed"])
+        model = MiniDecisionModel()
+        model.load_state_dict(load_file(checkpoint / "model.safetensors"), strict=True)
+        model.to("cuda").train()
+        optimizer = torch.optim.AdamW([
+            {"params": model.encoder.parameters(), "lr": protocol["encoder_learning_rate"]},
+            {"params": [parameter for name, parameter in model.named_parameters() if not name.startswith("encoder.")], "lr": protocol["head_learning_rate"]},
+        ], weight_decay=protocol["weight_decay"])
+        warmup = max(1, int(total_steps * protocol["warmup_fraction"]))
+        def multiplier(step):
+            return (step + 1) / warmup if step < warmup else max(0.05, (total_steps - step) / (total_steps - warmup))
+        scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, multiplier)
+        started = time.perf_counter()
+        steps = 0
+        for epoch, batches in enumerate(schedules):
+            for indices in batches:
+                if remaining_seconds(deadline) < 900:
+                    raise RuntimeError("Training deadline preserves artifact retrieval reserve")
+                inputs, targets, labels = shared_batch(paired, indices, arm, "cuda")
+                optimizer.zero_grad(set_to_none=True)
+                with torch.autocast("cuda", dtype=torch.bfloat16, enabled=use_amp):
+                    loss = distillation_loss(model(**inputs), targets, labels, protocol["label_weight"])
+                if not torch.isfinite(loss):
+                    raise RuntimeError("Nonfinite training loss")
+                loss.backward()
+                torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+                optimizer.step()
+                scheduler.step()
+                steps += 1
+                if steps == 100 and arm == "A":
+                    torch.cuda.synchronize()
+                    elapsed = time.perf_counter() - started
+                    estimated_remaining = elapsed / steps * (2 * total_steps - steps) * 1.5 + 900
+                    write_json(directory / "training-pilot.json", {"steps": steps, "seconds": elapsed, "both_arms_remaining_seconds_conservative": estimated_remaining})
+                    if estimated_remaining > remaining_seconds(deadline):
+                        raise RuntimeError("Measured training throughput exceeds the guarded lease")
+                if steps % 200 == 0:
+                    print(json.dumps({"arm": arm, "epoch": epoch + 1, "step": steps, "total_steps": total_steps, "loss": loss.item(), "seconds": time.perf_counter() - started}), flush=True)
+        output = directory / arm
+        output.mkdir()
+        save_file({name: value.detach().cpu().contiguous() for name, value in model.state_dict().items()}, output / "model.safetensors")
+        shutil.copyfile(checkpoint / "tokenizer.json", output / "tokenizer.json")
+        write_json(output / "training.json", {
+            "trained": True, "quality_validated": False, "arm": arm, "steps": steps,
+            "selection": protocol["selection"], "start_sha256": START_SHA256,
+            "protocol_sha256": file_hash(directory / "protocol.json"),
+            "shared_cache_sha256": file_hash(directory / "teacher-cache.jsonl"),
+            "shared_schedule_sha256": schedule_hash, "seed": protocol["seed"],
+            "bf16": use_amp, "elapsed_seconds": time.perf_counter() - started,
+        })
+        del model, optimizer, scheduler
+        gc.collect()
+        torch.cuda.empty_cache()
+
+
+def training_pilot(directory, checkpoint):
+    if file_hash(checkpoint / "model.safetensors") != START_SHA256:
+        raise ValueError("Wrong pilot checkpoint")
+    torch.manual_seed(42)
+    torch.set_float32_matmul_precision("high")
+    model = MiniDecisionModel()
+    model.load_state_dict(load_file(checkpoint / "model.safetensors"), strict=True)
+    model.cuda().train()
+    optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
+    inputs = {name: value.cuda() for name, value in dummy_inputs(32, 512, 6).items()}
+    targets = torch.full((32, 6), 1 / 6, device="cuda")
+    labels = torch.arange(32, device="cuda") % 6
+    started = None
+    for step in range(35):
+        if step == 5:
+            torch.cuda.synchronize()
+            started = time.perf_counter()
+        optimizer.zero_grad(set_to_none=True)
+        with torch.autocast("cuda", dtype=torch.bfloat16, enabled=torch.cuda.is_bf16_supported()):
+            loss = distillation_loss(model(**inputs), targets, labels, 2)
+        loss.backward()
+        torch.nn.utils.clip_grad_norm_(model.parameters(), 1)
+        optimizer.step()
+    torch.cuda.synchronize()
+    elapsed = time.perf_counter() - started
+    write_json(directory / "pre-cache-training-pilot.json", {
+        "steps_timed": 30, "warmup_steps": 5, "batch": 32, "sequence": 512, "options": 6,
+        "seconds": elapsed, "both_arms_seconds_conservative": elapsed / 30 * 10096 * 1.5,
+        "targets": "synthetic uniform for throughput only; no pilot weights saved or reused",
+    })
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("action", choices=["pilot", "cache", "fit", "export"])
+    parser.add_argument("--data", type=Path, required=True)
+    parser.add_argument("--checkpoint", type=Path, required=True)
+    parser.add_argument("--deadline", type=float, required=True)
+    arguments = parser.parse_args()
+    torch.set_num_threads(4)
+    protocol = json.loads((arguments.data / "protocol.json").read_text())
+    if file_hash(arguments.data / "protocol.json") != PROTOCOL_SHA256:
+        raise ValueError("Frozen protocol changed")
+    for name in ["train.jsonl", "train-B.jsonl", "descriptions.json"]:
+        if file_hash(arguments.data / name) != protocol["files"][name]:
+            raise ValueError(f"Frozen input changed: {name}")
+    if arguments.action == "pilot":
+        training_pilot(arguments.data, arguments.checkpoint)
+    elif arguments.action == "cache":
+        make_cache(arguments.data, arguments.deadline)
+    elif arguments.action == "fit":
+        fit(arguments.data, arguments.checkpoint, arguments.deadline)
+    else:
+        for arm in ["A", "B"]:
+            checkpoint = arguments.data / arm
+            model = MiniDecisionModel()
+            model.load_state_dict(load_file(checkpoint / "model.safetensors"))
+            provenance = json.loads((checkpoint / "training.json").read_text())
+            torch.manual_seed(42)
+            report = export_model(model, checkpoint / "model", provenance, checkpoint / "tokenizer.json")
+            write_json(checkpoint / "export-report.json", report)
+
+
+if __name__ == "__main__":
+    main()

research/src/audit_robustness.py

@@ -0,0 +1,432 @@
+import argparse
+import hashlib
+import json
+import shutil
+import sys
+import time
+import urllib.request
+from collections import defaultdict
+from pathlib import Path
+
+import numpy as np
+
+
+ROOT = Path(__file__).resolve().parents[2]
+RELEASE = ROOT / "outputs/laya-mini-cpu"
+LAB = Path(__file__).resolve().parent
+DIRECTORY = LAB / "robustness-audit"
+SEED = 20260926
+sys.path.insert(0, str(RELEASE))
+
+import build_corpus
+from schema import collate, encode_question, normalized_probabilities, read_rows
+
+
+SEMANTIC_KEYS = {
+    "ag_news": ["world", "sports", "business", "science"],
+    "snli": ["entailment", "neutral", "contradiction"],
+}
+SHORT = {
+    "ag_news": ["world news", "sporting events", "business news", "science or technology"],
+    "snli": ["entailed by the premise", "not determined by the premise", "inconsistent with the premise"],
+}
+LONG = {
+    "ag_news": [
+        "International events, politics, diplomacy and public affairs.",
+        "Competitive sports, athletes, teams, tournaments and match results.",
+        "Companies, financial markets, trade, industries and the economy.",
+        "Scientific research, computing, technological products and inventions.",
+    ],
+    "snli": [
+        "The hypothesis must be true if the premise is true; it is supported by the stated facts.",
+        "The premise does not establish whether the hypothesis is true or false; either remains possible.",
+        "The hypothesis cannot be true if the premise is true; the two statements are incompatible.",
+    ],
+}
+
+
+def digest_file(path):
+    with Path(path).open("rb") as handle:
+        return hashlib.file_digest(handle, "sha256").hexdigest()
+
+
+def write_json(path, value):
+    with Path(path).open("x", encoding="utf-8") as handle:
+        json.dump(value, handle, ensure_ascii=False, indent=2)
+        handle.write("\n")
+
+
+def write_rows(path, rows):
+    with Path(path).open("x", encoding="utf-8") as handle:
+        for row in rows:
+            handle.write(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n")
+
+
+def build_variants(record):
+    dataset = record["dataset"]
+    fixed = list(build_corpus.CHOICES[dataset])
+    original = record["questions"]["decision"]
+    base_order = [fixed.index(value) for value in original["criteria"].values()]
+    original_keys = {fixed.index(value): key for key, value in original["criteria"].items()}
+    named = [f"{name}: {description}" for name, description in
+             zip(build_corpus.SOURCE_BY_NAME[dataset].labels, LONG[dataset])]
+    styles = {"fixed": fixed, "short": SHORT[dataset], "long": LONG[dataset], "long_named": named}
+    variants = []
+    for keys in ["opaque", "semantic"]:
+        key_mapping = original_keys if keys == "opaque" else dict(enumerate(SEMANTIC_KEYS[dataset]))
+        for style, descriptions in styles.items():
+            for rotation in range(len(base_order)):
+                order = base_order[rotation:] + base_order[:rotation]
+                question = {**original, "criteria": {key_mapping[index]: descriptions[index] for index in order}}
+                variants.append({
+                    "request_id": f"{record['source_group']}|{keys}|{style}|{rotation}",
+                    "source_group": record["source_group"], "dataset": dataset,
+                    "state": record["state"], "question": question,
+                    "canonical_label": record["canonical_label"],
+                    "label": key_mapping[record["canonical_label"]],
+                    "semantic_order": order, "keys": keys, "style": style, "rotation": rotation,
+                })
+    return variants
+
+
+def canonical_probabilities(values, semantic_order):
+    count = len(semantic_order)
+    if sorted(semantic_order) != list(range(count)):
+        raise ValueError("Semantic order must be a permutation")
+    normalized = normalized_probabilities(values, count)
+    canonical = np.empty(count, dtype=np.float64)
+    canonical[semantic_order] = normalized
+    return canonical.tolist()
+
+
+def select_balanced(rows, dataset, count):
+    label_count = len(SEMANTIC_KEYS[dataset])
+    if count <= 0 or count % label_count:
+        raise ValueError("Count must be positive and divisible by the class count")
+    selected, families = [], set()
+    for label in range(label_count):
+        candidates = [row for row in rows if row["dataset"] == dataset and row["canonical_label"] == label]
+        candidates.sort(key=lambda row: build_corpus.digest(SEED, "audit", row["source_group"]))
+        chosen = []
+        for row in candidates:
+            if row["family_group"] not in families:
+                chosen.append(row)
+                families.add(row["family_group"])
+            if len(chosen) == count // label_count:
+                break
+        if len(chosen) != count // label_count:
+            raise ValueError("Not enough distinct families for balanced sampling")
+        selected.extend(chosen)
+    return selected
+
+
+def interval(values):
+    values = np.asarray(values, dtype=np.float64)
+    if values.ndim != 1 or not len(values):
+        raise ValueError("Bootstrap needs nonempty text-level values")
+    generator = np.random.default_rng(SEED)
+    means = values[generator.integers(0, len(values), size=(4000, len(values)))].mean(axis=1)
+    return [float(value) for value in np.quantile(means, [0.025, 0.975])]
+
+
+def grouped_accuracy(rows, model):
+    grouped = defaultdict(list)
+    for row in rows:
+        grouped[row["source_group"]].append(float(row["prediction"][model] == row["canonical_label"]))
+    return {group: float(np.mean(values)) for group, values in grouped.items()}
+
+
+def paired_interval(reference, variant):
+    if not reference or set(reference) != set(variant):
+        raise ValueError("Paired comparisons require identical nonempty source groups")
+    differences = [variant[group] - reference[group] for group in sorted(reference)]
+    return {"texts": len(differences), "delta": float(np.mean(differences)), "ci95": interval(differences)}
+
+
+def summarize_cell(rows, model):
+    groups = defaultdict(list)
+    for row in rows:
+        groups[row["source_group"]].append(row)
+    scores = grouped_accuracy(rows, model)
+    original = [row for row in rows if row["rotation"] == 0]
+    return {
+        "texts": len(groups), "requests": len(rows),
+        "accuracy": float(np.mean(list(scores.values()))),
+        "accuracy_ci95_text_bootstrap": interval(list(scores.values())),
+        "all_rotations_correct": float(np.mean([value == 1.0 for value in scores.values()])),
+        "order_instability": float(np.mean([
+            len({row["prediction"][model] for row in values}) > 1 for values in groups.values()
+        ])),
+        "rotation_zero_accuracy": float(np.mean([
+            row["prediction"][model] == row["canonical_label"] for row in original
+        ])),
+    }
+
+
+def retrieve_source(source, original_split):
+    import pyarrow.parquet as parquet
+
+    filename = source.files[original_split]
+    suffix = ".parquet" if source.format == "parquet" else ".jsonl"
+    path = DIRECTORY / "sources" / f"{source.name}-{original_split}-{source.revision}{suffix}"
+    path.parent.mkdir(parents=True, exist_ok=True)
+    if not path.exists():
+        if shutil.disk_usage(DIRECTORY).free < 256_000_000:
+            raise RuntimeError("Keep at least 256 MB disk free; source download not started")
+        with urllib.request.urlopen(source.url(filename), timeout=120) as response:
+            payload = response.read(64_000_001)
+        if len(payload) > 64_000_000:
+            raise RuntimeError("Source exceeds the audit download bound")
+        path.write_bytes(payload)
+    if source.format == "parquet":
+        return parquet.read_table(path).to_pylist()
+    return read_rows(path)
+
+
+def prepare():
+    if (DIRECTORY / "protocol.json").exists():
+        raise FileExistsError("Audit protocol already exists")
+    DIRECTORY.mkdir(parents=True, exist_ok=True)
+    manifest = json.loads((RELEASE / "reports/corpus-manifest.json").read_text())
+    validation, enriched = [], []
+    forbidden_groups = set()
+    for pool in manifest["pools"]:
+        for source_split, selection in pool["source_splits"].items():
+            if pool["output_split"] != "validation":
+                forbidden_groups.update(f"{pool['dataset']}:{source_split}:{index}" for index in selection["selected_indices"])
+    for pool in manifest["pools"]:
+        if pool["output_split"] != "validation":
+            continue
+        source = build_corpus.SOURCE_BY_NAME[pool["dataset"]]
+        for source_split, selection in pool["source_splits"].items():
+            raw_rows = retrieve_source(source, source_split)
+            for index in selection["selected_indices"]:
+                raw = raw_rows[index]
+                record = build_corpus.render_record(source, source_split, index, raw, "validation", manifest["seed"])
+                if record["source_group"] in forbidden_groups:
+                    raise ValueError("Validation source group overlaps another split")
+                validation.append(record)
+                if source.name in SEMANTIC_KEYS:
+                    family_text = raw["premise"] if source.name == "snli" else raw["text"]
+                    enriched.append({**record, "canonical_label": raw["label"],
+                                     "family_group": build_corpus.state_key(family_text)})
+            print(json.dumps({"recovered_validation": source.name, "rows": len(selection["selected_indices"])}), flush=True)
+    validation.sort(key=lambda row: (build_corpus.digest(manifest["seed"], "output-order", "validation", row["source_group"]), row["source_group"]))
+    payload = "".join(json.dumps(row, ensure_ascii=False, separators=(",", ":")) + "\n" for row in validation).encode()
+    checksum = hashlib.sha256(payload).hexdigest()
+    if checksum != manifest["files"]["validation"]["sha256"]:
+        raise ValueError(f"Reconstructed validation hash differs: {checksum}")
+    write_rows(DIRECTORY / "validation.jsonl", validation)
+    selected = [row for dataset in SEMANTIC_KEYS for row in select_balanced(enriched, dataset, 60)]
+    requests = [variant for record in selected for variant in build_variants(record)]
+    tokenizer, _ = build_corpus.load_tokenizer(RELEASE / "model/tokenizer.json")
+    lengths = []
+    for request in requests:
+        feature = encode_question(tokenizer, request["state"], request["question"])
+        if feature["truncated_state"]:
+            raise ValueError("Audit variant would truncate the student state")
+        lengths.append(len(feature["input_ids"]))
+    write_rows(DIRECTORY / "selected.jsonl", selected)
+    write_rows(DIRECTORY / "requests.jsonl", requests)
+    write_json(DIRECTORY / "protocol.json", {
+        "seed": SEED, "purpose": "Exploratory development audit, not a held-out generalization benchmark",
+        "validation_sha256": checksum, "original_validation_rows": len(validation),
+        "selection": "60 per task, label-balanced hash sampling; unique normalized premises for SNLI",
+        "base_texts": len(selected), "requests": len(requests),
+        "source_groups": [row["source_group"] for row in selected],
+        "selected_sha256": digest_file(DIRECTORY / "selected.jsonl"),
+        "requests_sha256": digest_file(DIRECTORY / "requests.jsonl"),
+        "maximum_student_tokens": max(lengths), "student_truncation": False,
+        "keys": ["opaque original per-example keys", "semantic keys"],
+        "styles": {"fixed": build_corpus.CHOICES, "short": SHORT, "long": LONG,
+                   "long_named": "Source class name followed by the identical long description"},
+        "order": "All cyclic rotations of original validation order, not all permutations",
+        "statistics": "Paired source-text bootstrap, 4000 resamples, exploratory marginal 95% intervals; no multiplicity correction",
+        "primary_contrasts": [
+            "AG News: FP32, opaque keys, long without prefix minus fixed descriptions, mean over rotations",
+            "SNLI: FP32, opaque keys, long without prefix minus fixed descriptions, mean over rotations",
+        ],
+        "secondary_analyses": "Other styles, key families, order sensitivity, INT8 and teacher comparisons are exploratory",
+        "teacher": "convaiinnovations/laya@55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851",
+        "teacher_runtime": "laya==0.3.20, CPU, verify single/batch parity and token budgets",
+        "student_int8_batch": 1,
+        "limitations": ["Development validation already used for checkpoint selection", "Only AG News and SNLI choice tasks",
+                        "One alternative wording set per style", "No score/noul or unseen-task conclusions",
+                        "No weight changes, no GPU rental, no COPA evaluation"],
+    })
+    print(json.dumps({"validation_sha256": checksum, "texts": len(selected), "requests": len(requests), "maximum_tokens": max(lengths)}), flush=True)
+
+
+def inference_record(request, probabilities):
+    metadata = {key: request[key] for key in ["request_id", "source_group", "dataset", "canonical_label", "keys", "style", "rotation"]}
+    metadata["probabilities"] = probabilities
+    metadata["prediction"] = {name: int(np.argmax(values)) for name, values in probabilities.items()}
+    return metadata
+
+
+def run_student():
+    import onnxruntime as ort
+    import torch
+    from safetensors.torch import load_file
+    from model import MiniDecisionModel
+
+    torch.set_num_threads(4)
+    requests = read_rows(DIRECTORY / "requests.jsonl")
+    tokenizer, _ = build_corpus.load_tokenizer(RELEASE / "model/tokenizer.json")
+    prepared = [(request, encode_question(tokenizer, request["state"], request["question"])) for request in requests]
+    prepared.sort(key=lambda pair: len(pair[1]["input_ids"]))
+    model = MiniDecisionModel().eval()
+    model.load_state_dict(load_file(LAB / "collected/checkpoint/model.safetensors"), strict=True)
+    settings = ort.SessionOptions()
+    settings.intra_op_num_threads = 2
+    settings.inter_op_num_threads = 1
+    quantized = ort.InferenceSession(str(RELEASE / "model/model.int8.onnx"), settings, providers=["CPUExecutionProvider"])
+    started = time.perf_counter()
+    with (DIRECTORY / "student.jsonl").open("x", encoding="utf-8") as handle, torch.inference_mode():
+        for offset in range(0, len(prepared), 8):
+            chunk = prepared[offset:offset + 8]
+            arrays = collate([feature for _, feature in chunk])
+            floating = model(**{name: torch.from_numpy(value) for name, value in arrays.items()}).softmax(-1).numpy()
+            for position, (request, feature) in enumerate(chunk):
+                logits = quantized.run(None, collate([feature]))[0][0]
+                exponentials = np.exp(logits - logits.max())
+                count = len(feature["labels"])
+                values = {
+                    "fp32": canonical_probabilities(floating[position, :count], request["semantic_order"]),
+                    "int8": canonical_probabilities(exponentials / exponentials.sum(), request["semantic_order"]),
+                }
+                handle.write(json.dumps(inference_record(request, values)) + "\n")
+            handle.flush()
+            if offset % 256 == 0:
+                print(json.dumps({"student_done": min(offset + 8, len(prepared)), "total": len(prepared),
+                                  "elapsed_seconds": time.perf_counter() - started}), flush=True)
+    write_json(DIRECTORY / "student-meta.json", {
+        "decisions": len(prepared), "elapsed_seconds": time.perf_counter() - started,
+        "fp32_threads": 4, "fp32_batch": 8, "int8_threads": 2, "int8_batch": 1,
+        "checkpoint_sha256": digest_file(LAB / "collected/checkpoint/model.safetensors"),
+        "int8_sha256": digest_file(RELEASE / "model/model.int8.onnx"),
+    })
+
+
+def teacher_coverage(requests, teacher):
+    expected = {row["request_id"]: row for row in requests}
+    observed = {row["request_id"]: row for row in teacher}
+    if len(observed) != len(teacher) or not set(observed) <= set(expected):
+        raise ValueError("Teacher predictions have duplicate or unknown request IDs")
+    for identifier, row in observed.items():
+        if any(row[field] != expected[identifier][field]
+               for field in ["source_group", "dataset", "canonical_label", "keys", "style", "rotation"]):
+            raise ValueError("Teacher prediction metadata differs from the frozen request")
+    groups = {row["source_group"] for row in teacher}
+    required = {row["request_id"] for row in requests if row["source_group"] in groups}
+    if set(observed) != required:
+        raise ValueError("Teacher coverage must contain all variants for every included text")
+    return groups
+
+
+def confusion_matrix(rows, model, label_count):
+    matrix = np.zeros((label_count, label_count), dtype=np.int64)
+    for row in rows:
+        matrix[row["canonical_label"], row["prediction"][model]] += 1
+    return matrix.tolist()
+
+
+def diagnostics():
+    students = read_rows(DIRECTORY / "student.jsonl")
+    teachers = read_rows(DIRECTORY / "teacher.jsonl")
+    matrices, margins = [], []
+    for dataset, labels in SEMANTIC_KEYS.items():
+        for keys in ["opaque", "semantic"]:
+            for style in ["fixed", "short", "long", "long_named"]:
+                subset = [row for row in students if row["dataset"] == dataset and row["keys"] == keys and row["style"] == style]
+                reference = [row for row in teachers if row["dataset"] == dataset and row["keys"] == keys and row["style"] == style]
+                for model, rows in [("fp32", subset), ("int8", subset), ("teacher", reference)]:
+                    matrices.append({"dataset": dataset, "keys": keys, "style": style, "model": model,
+                                     "labels": labels, "texts": len({row["source_group"] for row in rows}),
+                                     "matrix_true_rows_predicted_columns": confusion_matrix(rows, model, len(labels))})
+                disagreements = [row for row in subset if row["prediction"]["fp32"] != row["prediction"]["int8"]]
+                values = [float(np.sort(row["probabilities"]["fp32"])[-1] - np.sort(row["probabilities"]["fp32"])[-2])
+                          for row in disagreements]
+                margins.append({"dataset": dataset, "keys": keys, "style": style,
+                                "disagreements": len(disagreements),
+                                "fp32_margin_min_median_p90_max": [float(value) for value in np.quantile(values, [0, 0.5, 0.9, 1])] if values else None})
+    result = {"purpose": "Post-audit exploratory diagnostics; repeated rotations are not independent examples",
+              "confusion_matrices": matrices, "quantization_disagreement_margins": margins}
+    write_json(DIRECTORY / "diagnostics.json", result)
+    print(json.dumps({"snli_opaque_fp32": [row for row in matrices if row["dataset"] == "snli" and row["keys"] == "opaque" and row["model"] == "fp32"],
+                      "snli_opaque_margins": [row for row in margins if row["dataset"] == "snli" and row["keys"] == "opaque"]}, indent=2))
+
+
+def report():
+    requests = read_rows(DIRECTORY / "requests.jsonl")
+    expected_ids = {row["request_id"] for row in requests}
+    rows = read_rows(DIRECTORY / "student.jsonl")
+    if len(rows) != len(expected_ids) or {row["request_id"] for row in rows} != expected_ids:
+        raise ValueError("Student predictions do not cover exactly the frozen audit requests")
+    models = ["fp32", "int8"]
+    teacher_path = DIRECTORY / "teacher.jsonl"
+    teacher = read_rows(teacher_path) if teacher_path.exists() else []
+    teacher_groups = teacher_coverage(requests, teacher)
+    teacher_complete = len(teacher) == len(expected_ids) and {row["request_id"] for row in teacher} == expected_ids
+    if teacher:
+        by_id = {row["request_id"]: row for row in teacher}
+        for row in rows:
+            if row["request_id"] not in by_id:
+                continue
+            match = by_id[row["request_id"]]
+            row["prediction"].update(match["prediction"])
+            row["probabilities"].update(match["probabilities"])
+    if teacher_complete:
+        models.append("teacher")
+    cells, deltas, quantization = [], [], []
+    for dataset in SEMANTIC_KEYS:
+        for keys in ["opaque", "semantic"]:
+            baseline = [row for row in rows if row["dataset"] == dataset and row["keys"] == keys and row["style"] == "fixed"]
+            for style in ["fixed", "short", "long", "long_named"]:
+                subset = [row for row in rows if row["dataset"] == dataset and row["keys"] == keys and row["style"] == style]
+                for model in models:
+                    cells.append({"dataset": dataset, "keys": keys, "style": style, "model": model, **summarize_cell(subset, model)})
+                    if style != "fixed":
+                        deltas.append({"dataset": dataset, "keys": keys, "style": style, "model": model,
+                                       **paired_interval(grouped_accuracy(baseline, model), grouped_accuracy(subset, model))})
+                distances = [float(np.abs(np.asarray(row["probabilities"]["fp32"]) - row["probabilities"]["int8"]).sum() / 2) for row in subset]
+                quantization.append({"dataset": dataset, "keys": keys, "style": style,
+                                     "flip_rate": float(np.mean([row["prediction"]["fp32"] != row["prediction"]["int8"] for row in subset])),
+                                     "mean_total_variation": float(np.mean(distances)), "max_total_variation": max(distances)})
+    paired_teacher_cells, paired_teacher_deltas = [], []
+    for dataset in SEMANTIC_KEYS:
+        for keys in ["opaque", "semantic"]:
+            baseline = [row for row in rows if row["source_group"] in teacher_groups and row["dataset"] == dataset and row["keys"] == keys and row["style"] == "fixed"]
+            for style in ["fixed", "short", "long", "long_named"]:
+                subset = [row for row in rows if row["source_group"] in teacher_groups and row["dataset"] == dataset and row["keys"] == keys and row["style"] == style]
+                if not subset:
+                    continue
+                for model in ["fp32", "int8", "teacher"]:
+                    paired_teacher_cells.append({"dataset": dataset, "keys": keys, "style": style, "model": model,
+                                                 **summarize_cell(subset, model)})
+                    if style != "fixed":
+                        paired_teacher_deltas.append({"dataset": dataset, "keys": keys, "style": style, "model": model,
+                                                      **paired_interval(grouped_accuracy(baseline, model), grouped_accuracy(subset, model))})
+    teacher_meta = DIRECTORY / "teacher-meta.json"
+    teacher_subset_complete = json.loads(teacher_meta.read_text())["complete"] if teacher_meta.exists() else False
+    result = {"protocol_sha256": digest_file(DIRECTORY / "protocol.json"), "texts": len({row["source_group"] for row in rows}),
+              "requests": len(rows), "teacher_complete": teacher_complete, "teacher_requests_available": len(teacher),
+              "teacher_subset_complete": teacher_subset_complete, "teacher_base_texts": len(teacher_groups),
+              "teacher_comparison_cells_same_texts": paired_teacher_cells,
+              "teacher_comparison_style_deltas_same_texts": paired_teacher_deltas,
+              "cells": cells, "paired_style_minus_fixed_accuracy": deltas, "quantization": quantization,
+              "limitations": json.loads((DIRECTORY / "protocol.json").read_text())["limitations"]}
+    output = DIRECTORY / ("report.json" if teacher else "student-report.json")
+    write_json(output, result)
+    print(json.dumps({"output": str(output), "teacher_complete": teacher_complete, "cells": cells}, indent=2))
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("command", choices=["prepare", "student", "report", "diagnostics"])
+    arguments = parser.parse_args()
+    {"prepare": prepare, "student": run_student, "report": report, "diagnostics": diagnostics}[arguments.command]()
+
+
+if __name__ == "__main__":
+    main()