Publish Sev-4B v0.2.0 SwarmTraces security-evidence research checkpoint

DATA_PROVENANCE.md

@@ -0,0 +1,43 @@
+# Sev research release provenance
+
+This note describes `macmacmacmac/Sev-4B@v0.2.0-swarmtraces-research`. It packages the selected security-evidence checkpoint as a research artifact despite three failed retention checks. It is not field validation or a replacement for source-specific data permissions.
+
+## Current model lineage
+
+The selected run is `sev-r2-swarmtraces-4b-retention-v1/00-trial-0`. It warm-starts `sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint`, which trained from [jaredpalmer/kev-4b at `485ace8703592fcf405488b262449990824cfed1`](https://huggingface.co/jaredpalmer/kev-4b/tree/485ace8703592fcf405488b262449990824cfed1). The backbone is [Qwen/Qwen3.5-4B-Base at `1001bb4d826a52d1f399e183466143f4da7b741b`](https://huggingface.co/Qwen/Qwen3.5-4B-Base/tree/1001bb4d826a52d1f399e183466143f4da7b741b).
+
+The [v0.1.0 synthetic-policy preview](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.1.0-research) is a separate historical branch. The selected checkpoint does not continue it or inherit its published performance claims.
+
+The final continuation used one epoch, learning rate `2.5e-6`, batch 4, accumulation 2, seed 4, and rank-16 LoRA. It retained every prior curriculum record and added the reviewed SwarmTraces component:
+
+| Training input | Records |
+| --- | ---: |
+| ExCyTIn evidence questions | 899 |
+| Original GUIDE incident view | 1,799 |
+| General public classification and authored-rule replay | 1,200 |
+| Native Sysmon evidence | 364 |
+| Richer GUIDE detector view | 1,730 |
+| Complete SwarmTraces programs | 446 |
+| Total | 6,438 |
+
+The [published curriculum manifest](https://huggingface.co/datasets/macmacmacmac/openai-agent-swarmtraces/blob/main/collection/v2/experiments/swarmtraces-v1/curriculum-v1/manifest.json) has SHA256 `61e3c1898b8a03eb178355d3c8c5050e85bd217285c700df9b4506ec9e3680f8`. It pins the component manifests and partition bytes. The retained parent curriculum manifest is `5ae5dbfd48e9a2717397c0197ed8db016db87d69abc30d51813878ef734beba3`; the SwarmTraces static component is `1f304228dbbcad5b4aab5d9ce582738b8b7109214c1fdfd56c617fa5f1db583b`.
+
+The raw trial checkpoint keeps temperature 1.0. Its `head.pt` SHA256 is `b29abc817f2340aa510fcd4b2c813455ef38102508120ffc35f7917ce359c27e`; its adapter SHA256 is `b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420`. The release applies the saved calibration-only temperature `1.6245047927124707`, fitted on 2,534 questions. The release head SHA256 is `1d50c37b4db60c22ae8e6bb24f5055056701bb16e332964350d76df57dc82af0`; its learned tensors are unchanged. Raw evaluation evidence remains attributable to the original bytes.
+
+The selected run answers 65/83 manual SwarmTraces questions and 345/354 parser-derived questions correctly. Native Sysmon, ExCyTIn, original Sysmon and general correctness counts match the parent. It passes 34/37 registered checks. General raw negative log loss and older GUIDE raw negative log loss/Brier remain failures. The release preserves those outcomes; applying calibration does not retrospectively pass the experiment.
+
+## Source and dataset terms
+
+Kev source is Apache-2.0, copyright Jared Palmer, 2026. Sev additions retain that [source license](LICENSE) and [upstream attribution](NOTICE). The Kev adapter/head card and the [pinned Qwen backbone license](https://huggingface.co/Qwen/Qwen3.5-4B-Base/blob/1001bb4d826a52d1f399e183466143f4da7b741b/LICENSE) identify Apache-2.0 for those model artifacts. Preserve the applicable notices when redistributing source or model packages.
+
+Dataset grants remain separate. ExCyTIn, GUIDE, OTRF logs, public classification replay and generated rule records retain their respective source terms and provenance. The [collection catalog](https://huggingface.co/datasets/macmacmacmac/Sev-behavioral-research-v1) records these boundaries. The mixed curriculum and transfer entries publish manifests and provenance only, not all referenced payloads. Availability of a model, catalog entry or mirror does not grant rights to redistribute every underlying dataset.
+
+SwarmTraces upstream reuse terms remain unresolved. Its [dedicated mirror](https://huggingface.co/datasets/macmacmacmac/openai-agent-swarmtraces) retains that limitation. Author-redacted recovered artifacts are not guaranteed free of secrets, and their source text is not a verified record of executed actions. The conversion preserves separate observations and annotations with absent timestamps left explicit. It does not create verified origin, intent, ownership or execution-outcome labels. No CC BY or Apache grant is asserted over third-party SwarmTraces content.
+
+The Kev ancestor already had its own public-dataset and generated-rule training history. The native parent and selected continuation also train directly on the retained public replay shown above. This is not a wholly synthetic training lineage. The package's model license does not replace dataset terms or erase inherited training provenance.
+
+## Historical synthetic release
+
+The original [v0.1.0 release](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.1.0-research) trained on 2,094 behavioral windows and 240 authored Kev rule records. Its behavioral labels described simulated policy priors, with no captured humans, live LLM executions or network captures. Its weak origin-classification results remain historical evidence, not results for v0.2.0.
+
+The original behavioral records retain [CC BY 4.0](https://github.com/maceip/Sev/blob/main/research/collector/DATA_LICENSE); authored upstream rule records retain their own Apache-2.0 provenance. The [behavioral manifest](https://jevalin-agent-datasets-095713295645.s3.eu-central-1.amazonaws.com/behavioral_v1/manifest.json) remains pinned at SHA256 `83f311ca5d0feddedc16e06272d881542d48ec2b3b0cbc3e292572f391f2300d`. Those notices do not relicense the expanded collection or this release's public-source inputs.

NOTICE

@@ -1,12 +1,17 @@
-Sev research preview
+Sev security-evidence research releases
 
 Sev is derived from Kev, Copyright 2026 Jared Palmer, under Apache-2.0.
 Sev research additions are Copyright 2026 the Sev contributors.
 The Qwen3.5 base model is provided by the Qwen team under Apache-2.0.
-The released adapter is initialized from jaredpalmer/kev-4b.
+The v0.2.0 SwarmTraces adapter continues the Sev native security curriculum,
+whose parent is the pinned jaredpalmer/kev-4b checkpoint. The older v0.1.0
+synthetic actor-origin adapter remains a separate historical release.
 
 The Apache-2.0 source/model license does not relicense third-party datasets.
-Original behavioral simulator records use CC BY 4.0 as described in
-research/collector/DATA_LICENSE. Authored Kev replay rules retain Apache-2.0
-and upstream attribution. Private captures and raw third-party replay are
-not included in the new Sev dataset or checkpoint artifacts.
+Training sources include SwarmTraces, OTRF Security-Datasets, ExCyTIn,
+Microsoft GUIDE, public classification replay, and authored Kev rules.
+Each source retains its own notices and terms. No upstream SwarmTraces
+dataset license declaration was located. A model/source license is not a
+grant to redistribute third-party training artifacts.
+Original behavioral simulator records, used in the historical v0.1.0 branch,
+use CC BY 4.0 as described in research/collector/DATA_LICENSE.

README.md

@@ -7,6 +7,7 @@ base_model_relation: adapter
 library_name: peft
 datasets:
 - macmacmacmac/Sev-behavioral-research-v1
+- macmacmacmac/openai-agent-swarmtraces
 tags:
 - sev
 - research
@@ -15,79 +16,169 @@ tags:
 - decision-model
 - lora
 ---
-# Sev-4B Research Preview
-
-Sev-4B is an exploratory baseline for classifying ordered observations as authored human, fixed-script, or agent policies. **This checkpoint predicts human for almost every window and has not demonstrated useful agent detection.** Its IID accuracy is 34.03% against 33.33% chance; agent recall is 1.04%. We release the checkpoint, synthetic inputs, generator, and evidence so researchers can reproduce and improve this result.
-
-This is a LoRA adapter and pointer head for Qwen3.5-4B-Base, initialized from Kev-4B. It scores supplied choices without generating text. It requires the [Sev/Kev runtime](https://github.com/maceip/Sev); loading the LoRA adapter into a text-generation pipeline omits the decision head.
+# Sev-4B: security evidence and SwarmTraces
+
+Sev-4B scores supplied answers to questions about security logs and recovered
+programs. This research release, **v0.2.0-swarmtraces-research**, adds checked
+SwarmTraces request-code examples to a curriculum of native security logs,
+incident triage and general decision tasks. It is a Qwen3.5-4B LoRA adapter
+with a decision head. It does not generate text.
+
+On 83 manually annotated development questions from 40 programs excluded from
+training, this checkpoint answers **65 correctly**, compared with its immediate
+parent's 58. All seven gains concern which credentials belong to a request.
+The main log-reading panels retain the parent's correctness counts. Three of
+37 registered continuation checks fail; the full failures remain below.
+
+The earlier synthetic actor-origin preview remains available at
+[`v0.1.0-research`](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.1.0-research).
+That historical preview is not the parent used in the comparisons on this page.
+This release broadens the task to security evidence interpretation. It does
+not establish reliable human-versus-agent identification.
 
 ## Use
 
+The adapter requires the [Sev runtime](https://github.com/maceip/Sev). A generic
+text-generation pipeline does not load its decision head.
+
 ```bash
 git clone https://github.com/maceip/Sev.git
 cd Sev
 uv sync --extra serve
 uv run python -m kev.serve \
-  --run macmacmacmac/Sev-4B@v0.1.0-research --port 8009
+  --run macmacmacmac/Sev-4B@v0.2.0-swarmtraces-research --port 8009
 ```
 
-The [repository example](https://github.com/maceip/Sev#try-it) sends an actual development observation after removing its target and audit metadata. The package and API aliases keep their upstream `kev` names. Treat probabilities and the derived `confidence` field as research outputs, not evidence of a person's or agent's identity. The serving implementation permits longer input than training used; accuracy beyond the 384-token training state limit is untested.
+Send a TypeSafe-shaped request to `POST /v1/systemone`. This illustrative
+request asks a credential-binding question; it is not a reported benchmark row.
+
+```json
+{
+  "state": "const token = sessionStorage.getItem('api_token');\nfetch('/records', {headers: {Authorization: `Bearer ${token}`}});",
+  "questions": {
+    "credential_source": {
+      "type": "choice",
+      "instructions": "Where does the Authorization credential for the /records request come from?",
+      "criteria": {
+        "session": "The api_token value read from sessionStorage",
+        "response": "A response from an earlier request",
+        "unknown": "The visible code does not establish the source"
+      }
+    }
+  }
+}
+```
+
+The checkpoint ships with temperature **1.6245047927124707**. Set
+`KEV_TEMPERATURE=1.0` to inspect raw probabilities. Published evaluation uses
+the fp32 CUDA path with merged LoRA; use `KEV_BACKEND=torch KEV_DTYPE=fp32 KEV_MERGE=1`
+when reproducing it. Default accelerated serving can differ numerically.
 
-## Training
+## Training and source lineage
 
 | Setting | Value |
 | --- | --- |
 | Backbone | `Qwen/Qwen3.5-4B-Base@1001bb4d826a52d1f399e183466143f4da7b741b` |
-| Warm start | `jaredpalmer/kev-4b@485ace8703592fcf405488b262449990824cfed1` |
-| Run | `sev-r2-research-4b-v1/00-trial-0` |
-| Inputs | 2,094 authored behavioral windows and 240 authored Kev rule records |
+| Immediate parent | `sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint` |
+| Selected run | `sev-r2-swarmtraces-4b-retention-v1/00-trial-0` |
+| Training records / questions | 6,438 / 9,178 |
+| New SwarmTraces programs / questions | 446 / 903 |
+| Retained curriculum records | 5,992 |
 | Epochs / seed | 1 / 4 |
-| Learning rate | `1e-5` |
+| Learning rate | `2.5e-6` |
 | Batch / accumulation | 4 / 2 |
-| LoRA / head | rank 16, all targets / 256 dimensions |
-| Precision | fp32 frozen weights, bf16 autocast, gradient checkpointing |
-| Updates / forward tokens | 292 / 738,327 |
-| Training coverage | 2,334 requested and seen, zero rejected or truncated |
-| Hardware | One NVIDIA H100 80 GB |
-
-The [registered plan](https://github.com/maceip/Sev/blob/main/experiments/sev-research-4b-v1.json) records every override. Use the frozen suite and `kev.experiment` or `modal_app.py::study` to repeat it. The checkpoint includes full training configuration, metrics, log, and source hashes. The training suite manifest SHA-256 is `372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5`.
-
-The continuation excludes the earlier screen's 400 public replay rows. This does not erase the parent model's training history. The Apache-licensed Kev parent was trained on public classification datasets and authored rules. No private phone captures, URLQuery records, retired enterprise synthetic data, or executed LLM traces enter this continuation. `legacy_policy` means authored upstream Kev rules, not retired Sev traffic.
-
-## Evaluation
-
-| Full observation panel | Accuracy | Parent accuracy | Human recall | Script recall | Agent recall |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| IID development | 34.03% | 33.33% | 98.96% | 2.08% | 1.04% |
-| Held-out tasks and policy shift | 33.85% | 33.59% | 98.18% | 3.39% | 0.00% |
-
-Each panel contains 96 episodes from 32 matched scenario groups. IID has 196 full-observation windows and the shifted panel has 279. Each episode has equal weight; its eligible windows share that weight. Probabilities are not pooled into an episode ensemble. Paired bootstrap resampling keeps all origins and views in each scenario group together. The IID gain over the parent is 0.69 percentage points, with a 95% interval of 0.00 to 1.74 points over 1,000 resamples. This small development result does not establish useful discrimination.
-
-| Full observation panel | Raw NLL | Shipped-temperature NLL | Raw ECE | Shipped-temperature ECE |
-| --- | ---: | ---: | ---: | ---: |
-| IID development | 1.1142 | 1.0998 | 0.0658 | 0.0291 |
-| Held-out tasks and policy shift | 1.1021 | 1.0975 | 0.0677 | 0.0392 |
-
-Uniform three-class predictions have NLL 1.0986. Low ECE near chance is not evidence of an effective detector. The parent comparison uses the unchanged pinned Kev checkpoint, scored on identical inputs. There is no fresh general-task retention evaluation for this continuation.
-
-These development panels have informed earlier research and are not untouched confirmation. The separate native final test remains unscored. The release suite's `test.jsonl` is empty. `promotable=false` in the original trial receipt remains unchanged: this is an explicitly limited research publication, not an automatic quality promotion.
-
-## Calibration
-
-The shipped `head.pt` stores temperature 2.0, fitted only on 182 full-view calibration windows from 96 episodes and 32 scenario groups. Fitting gives equal weight to episodes, then equal weight to their windows. Five-fold diagnostics hold out complete scenario groups. Episode-weighted calibration ECE is 0.0655 raw and 0.0616 out of fold. The full fit's lower in-sample ECE is not an unseen-data guarantee.
-
-The experiment also recorded a different, window-weighted diagnostic fit of 1.07177. It is preserved in `result.json`; **2.0 is the shipped temperature**. The release calibration copy changes temperature metadata only: learned head tensors and adapter bytes are unchanged, and the raw training checkpoint remains untouched. See `calibration.json`, `calibration-sources.json`, and `SHA256SUMS` for receipts.
-
-## Data Limits And Intended Use
-
-Labels identify authored policies in a single small simulator. There are no captured humans, live LLM executions, measured browser clocks, or network captures in the training corpus. Logical timing and policy probabilities are authored, not estimates of enterprise populations. Human, script, and agent policies have overlapping possible actions.
-
-Task holdouts also change policy priors within the same controller implementation. They do not test independent controllers or real agent swarms. Ordered windows preserve complete interactions, but the short context can omit useful earlier actions and cross-window dependencies. Internal consistency tests do not establish realism.
-
-Use this release to study evidence representations, sequence context, class bias, calibration, and synthetic-data design. It does not support operational blocking, human-owner attribution, maliciousness judgments, or claims that a real-world action conclusively came from an AI agent.
-
-## Provenance And License
-
-Code and adapter/head weights are Apache-2.0, with Kev and Qwen attribution in `NOTICE` and `BASE_LICENSE`. Original generated behavioral data are CC BY 4.0; the 240 upstream authored rule records retain Apache-2.0. Dataset licenses are separate from model licenses. See the [data and inherited-training provenance](https://github.com/maceip/Sev/blob/main/docs/releases/sev-research-provenance.md).
-
-The [release evidence](https://github.com/maceip/Sev/tree/main/docs/releases/v0.1.0-research) includes source and checkpoint hashes, raw predictions, grouped comparison results, and calibration receipts. The artifact is maintained by [maceip](https://github.com/maceip), using Kev by Jared Palmer and the Qwen team's backbone.
+| LoRA / head | Rank 16, all targets / 256 dimensions |
+| Precision | fp32 frozen weights, bf16 autocast |
+| Updates / forward tokens | 805 / 1,447,855 |
+| Rejected or truncated training records | 0 |
+
+The immediate parent starts from pinned Kev-4B and adds native Sysmon captures,
+ExCyTIn evidence, GUIDE triage and detector features, and general public/rule
+replay. The final continuation retains those records and adds complete,
+context-fitting recovered programs. It does not descend from the published
+v0.1.0 synthetic actor-origin checkpoint. [Training configuration](training_config.json),
+[provenance](provenance.json) and [source lineage](DATA_PROVENANCE.md) identify
+the exact inputs and ancestors.
+
+The [dataset collection](https://huggingface.co/datasets/macmacmacmac/Sev-behavioral-research-v1)
+and [SwarmTraces repository](https://huggingface.co/datasets/macmacmacmac/openai-agent-swarmtraces)
+contain source inventories, the lossless conversion and frozen code-reading
+suites. Some mixed-curriculum entries contain metadata only. Catalog inclusion
+does not mean a source was used for training.
+
+SwarmTraces contributes real recovered artifacts. The normalization preserves
+189,579 artifacts; the model trains on the 446 programs that pass the static
+label and context checks. The source does not contain reliable event timestamps,
+and recovered code does not itself prove execution. No blanket maliciousness,
+controller identity or human-ownership labels were inferred from the source.
+
+## Development results
+
+These counts compare the selected checkpoint with its immediate internal
+parent on identical questions, not with v0.1.0. They are development results.
+
+| Panel | Parent correct | This release correct |
+| --- | ---: | ---: |
+| Manual SwarmTraces | 58/83 | 65/83 |
+| Parser-derived SwarmTraces | 324/354 | 345/354 |
+| Native Sysmon | 409/410 | 409/410 |
+| ExCyTIn | 398/398 | 398/398 |
+| Original Sysmon | 62/64 | 62/64 |
+| General | 86/105 | 86/105 |
+| Older GUIDE | 127/231 | 130/231 |
+| Richer GUIDE | 623/1,064 | 638/1,064 |
+
+The manual panel gains seven correct answers with no new errors. Its paired
+whole-program bootstrap difference is +8.43 percentage points, with a 95%
+interval of +3.46 to +14.63 on this selected panel. Exact-text, reconstruction
+and program-template groups were kept apart from training. The panel was reused
+for two learning-rate experiments, so it is not untouched confirmation. No
+locked final test was scored. No matched continuation without SwarmTraces was
+run, so the source's isolated contribution is not established.
+
+An earlier `5e-6` continuation answered 66/83 manual questions correctly, but
+reduced older GUIDE true-positive answers from 8 to 5. This lower-rate release
+answers 7 of those 37 true-positive questions correctly. Richer GUIDE
+true-positive answers increase from 73/279 to 101/279; that remains limited
+recall. Python questions stay at 14/21, and manual flow questions do not improve.
+
+## Registered failures and calibration
+
+This checkpoint passes **34 of 37** continuation checks. These three raw
+probability-quality checks fail:
+
+| Check | Measured | Maximum permitted |
+| --- | ---: | ---: |
+| General raw NLL | 0.55743 | 0.55165 |
+| Older GUIDE raw NLL | 0.85633 | 0.82567 |
+| Older GUIDE raw Brier | 0.52551 | 0.49922 |
+
+Every probability-retention check after applying the separately fitted global
+temperature passes. This does not change the failed raw checks or mark the
+experiment as passing. Publication makes the research checkpoint available with
+its tradeoffs; it is not a claim of general superiority or field validation.
+[Evaluation](evaluation.json) preserves all checks and comparisons.
+
+The shipped temperature is the registered micro-NLL fit on 2,534 calibration
+questions only. Five-fold diagnostics keep the 437 whole calibration groups
+together across task families. Calibration ECE is 0.06260 raw and 0.03870 out
+of fold. This diagnostic describes calibration data, not deployment accuracy.
+[Calibration](calibration.json) records the fit, intervals and grouping.
+
+Only temperature metadata was added to a copy of the raw trained checkpoint.
+Learned head tensors, adapter and tokenizer bytes remain unchanged.
+[Integrity evidence](checkpoint-integrity.json) and [SHA256SUMS](SHA256SUMS)
+identify the release files. Training and inference support different context
+limits; inputs longer than the admitted training examples remain unvalidated.
+
+## License and attribution
+
+Sev source and released adapter/head weights carry Apache-2.0 notices. Preserve
+[LICENSE](LICENSE), [NOTICE](NOTICE) and the Qwen [BASE_LICENSE](BASE_LICENSE).
+Sev builds on Kev by Jared Palmer and Qwen3.5 by the Qwen team.
+
+Training sources retain their own terms. We have not located an upstream
+SwarmTraces dataset license declaration; this model's license does not grant
+rights to its source artifacts. The original synthetic data notice applies
+only to the historical synthetic source, not to the expanded collection.
+See [data provenance](DATA_PROVENANCE.md).

SHA256SUMS

@@ -1,20 +1,22 @@
+34448b82c17d60fec9b65b1f093c115ddbaadc04beb1b0140b6bfed2e012a930  .gitattributes
 50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32  BASE_LICENSE
+66d89d8131fdbd57f4357f8e3f4bde232e74ca0adbfd8314edcf21f395d78c0c  DATA_PROVENANCE.md
 6b08bb37982c233aa12bcbdf19106da12f3f4fcf800773ea42e28ebeddd34fb8  LICENSE
-859135197bdac84f3e4fb27463b1abe71cd0344f08a5b4eb970f50684bcceae7  NOTICE
-d032908fe8fda54de4a3a3b29eaa3d6d2ba4bef16266373411bab6688fbedaf4  README.md
+c435cbd92703e3e3fe455a8526c7a02890d0b1a543f3a46c4ece969da605c0a5  NOTICE
+186795e17386163aa95a6e18b46af3a556722fe9c076d520e7315d86f6296ff7  README.md
 8a05dfd6c5e7f61a62e6db5d8de8093a8dbbd105aa7ee7b5e7b73f6249f45617  adapter_config.json
-7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c  adapter_model.safetensors
-be16ec2fdfb3b28b63888ede85cce428a8106785a2a64b37215550833176d875  calibration-sources.json
-ca2b2eacb602dd61b87139cd5eafe2daf4d3fdd659c8cb8924538567259808bc  calibration.json
+b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420  adapter_model.safetensors
+459aa9d26078634c229642eb4ed5e8b9b31570313286c1db9edcfcacd4ea08c7  calibration-sources.json
+40f71c5c6bbda523e75e60dc289769c353a34b4aaf4974e55144b2a1bbb51bfe  calibration.json
 a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  chat_template.jinja
-68a0de83be179f88ee0f0d324db65ae581c0cfd89177b0bf6a56364a80164686  checkpoint-integrity.json
-8327fc07f1354720311b1ba2e09b84ef7bf946857a1bcd1c6d7c832bc3a97e6b  class-metrics.json
-14c148b41d9c5070f2800b938daf6e647ea3019cafcd2046f739226177469c64  evaluation.json
-dc1f550c7914388af77e607c70fe8801558a89a3847d37ea37d3923b28521c3e  head.pt
-f540a4587f44f4a3bdda19f68a42377a56c56d464443beaaf8fe57b593beb85f  provenance.json
-ec4bfe6436093b6ba096921bb95b2643e8418c84d1e5cbbc596d78f2c744338c  result.json
+0210038fb9eee9de75bca08d5ae75dc29bbf9855121f74cf6e3784a3784c06b7  checkpoint-integrity.json
+a72cba7d6a3d851f6b2d06e746f97b1024a5c2be3d69b411e53ae034e15e6ae2  evaluation.json
+1d50c37b4db60c22ae8e6bb24f5055056701bb16e332964350d76df57dc82af0  head.pt
+5712631ff496a274abe8abecc84f8f259c8e58d6bf1f4a73638d969f2cce8227  provenance.json
+d4966edb60f8d222f2b93e7336577d3836dc1757d67a52090c07f63423026337  release.json
+9e6143f4cf6055c8644458f6920b7138a1a348cc254422e9d3173c268b97c4c6  result.json
 06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523  tokenizer.json
 8671bed7c852ce9e661be94f179a7b4ffd091c2a65aea0363e5501c20318ee45  tokenizer_config.json
-e5b54da6b803d7e6d5c0fdfc902b8095a123243a652ec996ab411da0c818ce09  train.log
-f1936818cecded5fe659330a6e0a1ad631d9b5b1eb2014982634a58f7d6f7d68  training_config.json
-20b9314de861f32641b4f6a16ae8b06ba273426333da40c2820e5f591936d7a7  training_metrics.json
+298409efc1ea833998a489624a0083ab4aefedc4afc4b847f7c0469228cf92c3  train.log
+daf81a93394c6cd9e38cc4b0e049f54910048ffebf4de34f6ded2339d85478cb  training_config.json
+71cc83469fc7e21c7d5fc1a44ad2b2db8cdb36356a227bbcd80bed76fd512776  training_metrics.json

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c
+oid sha256:b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420
 size 129924032

calibration-sources.json

@@ -1,86 +1,474 @@
 {
-  "schema": "sev-research-calibration-sources-v1",
-  "split": "calibration",
-  "view": "full",
-  "temperature": 1.9999999999999998,
-  "weighting": "equal episodes, then equal windows within each episode",
-  "partition_sha256": "eebb5ddeb322b16bbb312ddb6927b1d71226b72e4b01d323c698d1e2a3c4f38d",
-  "predictions": {
-    "path": "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/rows.json",
-    "sha256": "00909b4d6b838c534d8d403d84a578fea1ddaac5e30a2b488c2d1c138c0d05ab"
+  "rows_sha256": "5dc5633f59c9da863561e1ae34fbc1c87199053cf1397d4592b921ec5ff69790",
+  "partition": "calibration",
+  "question_counts_by_source": {
+    "agnews": 48,
+    "amazon": 16,
+    "banking77": 16,
+    "boolq": 16,
+    "compositional": 40,
+    "dbpedia14": 16,
+    "excytin_file": 80,
+    "excytin_parent_join": 80,
+    "excytin_process": 120,
+    "excytin_signin": 120,
+    "imdb": 16,
+    "legacy_policy": 40,
+    "mnli": 16,
+    "sev_guide_detector": 1071,
+    "sev_guide_triage": 169,
+    "sev_swarmtraces_static": 211,
+    "sev_sysmon_native_file": 72,
+    "sev_sysmon_native_join": 86,
+    "sev_sysmon_native_other_effect": 96,
+    "sev_sysmon_native_process": 69,
+    "sev_sysmon_native_registry": 72,
+    "sst5": 16,
+    "trec": 16,
+    "yelp": 32
   },
-  "sources": {
-    "sev_behavioral_artifact_handoff": {
-      "windows": 63,
-      "episodes": 24,
-      "groups": 8,
-      "record_ids_sha256": "ec2841fda8a390bd2d73d828e1d3efac393d6cc583f94188ef25588a85945877",
-      "raw": {
-        "acc": 0.34722222222222215,
-        "nll": 1.0806702183699979,
-        "brier": 0.6537004910585257,
-        "ece": 0.09166280712269316
-      },
-      "calibrated": {
-        "acc": 0.34722222222222215,
-        "nll": 1.0876702845061252,
-        "brier": 0.6589248476452055,
-        "ece": 0.04426971102023442
-      }
-    },
-    "sev_behavioral_authorized_export": {
-      "windows": 40,
-      "episodes": 24,
-      "groups": 8,
-      "record_ids_sha256": "255e72d055b44aa16ec7f11fe905f4c251fe28ced7f801120abef039c930115f",
-      "raw": {
-        "acc": 0.3333333333333333,
-        "nll": 1.104011417056653,
-        "brier": 0.6698152020985213,
-        "ece": 0.04556619047184074
-      },
-      "calibrated": {
-        "acc": 0.3333333333333333,
-        "nll": 1.0996799880244128,
-        "brier": 0.6672264414223563,
-        "ece": 0.04735674179920909
-      }
-    },
-    "sev_behavioral_lookup_catalog": {
-      "windows": 25,
-      "episodes": 24,
-      "groups": 8,
-      "record_ids_sha256": "8cd835d6e3fefa25803ffbc579240aa9c4ae264163027eaa186db00580e4b2e1",
-      "raw": {
-        "acc": 0.3333333333333333,
-        "nll": 1.0929036665832168,
-        "brier": 0.6633979694737079,
-        "ece": 0.10676289475303495
-      },
-      "calibrated": {
-        "acc": 0.3333333333333333,
-        "nll": 1.087115774278896,
-        "brier": 0.6586049595361453,
-        "ece": 0.052123650581165686
-      }
-    },
-    "sev_behavioral_update_export": {
-      "windows": 54,
-      "episodes": 24,
-      "groups": 8,
-      "record_ids_sha256": "f83b45d350186cfa47db970f9311408fbde0fb68bb884adc1a3644a36b18f52c",
-      "raw": {
-        "acc": 0.3229166666666667,
-        "nll": 1.11786418934943,
-        "brier": 0.6813036856150539,
-        "ece": 0.09059596392971296
-      },
-      "calibrated": {
-        "acc": 0.3229166666666667,
-        "nll": 1.1024371940848587,
-        "brier": 0.6695271712596828,
-        "ece": 0.060665733732341603
-      }
-    }
-  }
+  "fold_assignment_by_whole_group": {
+    "00499ec7fe8f44800d9870099fd51fad92a90a16e7dde23a597d8eecbde40c33": 2,
+    "00f36cc73084a7f13e7b7ac8a582b4c496c3585e496225e163fbeab602d3fb01": 0,
+    "02d857d8ccbd122614301bc25e5d2e85b75d316c01f987c8aa844c6858a2ff25": 3,
+    "041cb4d4db55df55921150dfbca521e5963aafca45d14bbcace6b33150c82e0a": 2,
+    "04529d23bfd5d7cd7636d63dbede52c47d2416058e8d2e9ef2c1879fe94cc2a6": 3,
+    "075404932f83a1013f2d227fcc82c56a3be0a264e34b3b75873e87a2b4edfc43": 4,
+    "0830d84f6e8815fae1534e4cb94985054e69540b503ccbd2d38df3f8028cf396": 0,
+    "099ce79915720358385d6042cd7d451e420521871c97c16c0bcf194d89582bcd": 4,
+    "0bfe0d35601938e7e4077e00da02992772870cc0e18b6cf0f585ab7fe9b65df4": 0,
+    "0f12436e9a901f77d6a2ca7dbebdeb16fac4207c9210510e5a8c00b20b4ab95d": 1,
+    "11a889a11ab8a80373829bd1a1ebbdfa15b9acaf6dfed494636290715b0308b2": 3,
+    "1259b6d797a0e7412716eddeef04fe8b183695b4ea3a49959164e4d0a778edfd": 2,
+    "16a91f37a46c9fdf8dc33b014a9d9137ba2e1684444931de11ea2a12b74f2c64": 0,
+    "183ec1e92d380a1e327d241312e3e0cb01a41de90973db2a3e5dd2a05d6f06ca": 4,
+    "18dccb14b28c4c5b95c45a4297178fdeb04d9eea37942c0794073bdb4d1b1a36": 2,
+    "1aa7ca0351abc7244f3d40d0fa144cafc0d2fa9a786df7ba90e3578f50f43156": 2,
+    "1c11e4dced3a9133cbd2fd03664e425e5688faf1242220b400b215d586ad71cf": 0,
+    "1dd0ae3520074c0e7e86edf26ed1d23bd0e7ea4b5a5dcb3da81336e115efecdc": 2,
+    "221ba1ef07d606ce1bd65e32582f39f463a48486bc5c9eb0f2493d4268345be1": 4,
+    "229faaa3ddd04245f4c293348ae53a36ce1c39395d2463c31aaf4129d8fc740c": 4,
+    "24b419b2c6ae56d1347c1cbe05987dc23d1c07442ea811143b89eff23facf0b4": 1,
+    "26142a9f871e1a3db6673da8eb091482e46a35b9b935d29132417c4b17a54c02": 3,
+    "267aed10626844f2f78398c97d2156baf05cf04c02b4032bb5cde18b1aba6604": 3,
+    "269b07282750471d8647645f530371e3e421cf919cf5715511e5789445249b4f": 1,
+    "29725763217576a16b0493ac12492e860a04173839488eeb2fa0df66d9636c47": 2,
+    "2c38ea943f402f6392ceef212c295130088313291924ecb973dab9afc2e70e2e": 4,
+    "2ea406b423e605f2655ab45fdc06889f6e27d7545bbbd9f1e1b4e069b25851d3": 1,
+    "2fa36c8f6b489beec15e38246d510fefb919c42c8bb1959c5a8e87f39d695c2f": 1,
+    "318cb25c9ce681abb939933bf0ae3a117e05f80dadab5eb742c568a4148aa7c4": 1,
+    "328069edaba0ba9458de0b4dd72148d9e1455b1aa249fba4df08931d05988c84": 2,
+    "3444beb48cd8d9ff13fc1afe794638f57fdda39e7adbc6acd8e5e0bd7ad100e0": 2,
+    "348d121ace5402ecb06bac82b307cc5ac4c08f879285dc18e3c1b1bf66ce5c98": 2,
+    "357cdb24dba8ff458aa904796c2dc41ad31a19cb6bfbb7f730ad906df6ad9273": 4,
+    "361098d9fa55cfaf96c3774143b09d3f353325d3345f27bb1b2bb37f6c1e318e": 2,
+    "3897c587bcc08bfef8420412ecfd49dd4c07ee6f37a8c206f9f04646d4d8b083": 3,
+    "3c6e79c5e05446362355b1a49b61b133a6f7ba02afd20ba81d1940218b78d850": 4,
+    "423dd9f13af9aad60197e7780dd4ba37e36e202c59cae28d9af166554f927e3a": 1,
+    "425c23f55483c6075edbde9d6c4f974a39e84df381f4b93c52bde961f74d89c0": 4,
+    "42d0491fda16a85d4d1f871865f77c5739a4e869187be3fd0dc17fdf6657a2cb": 0,
+    "475633eada5fb59725c031525cbe9cbb339703087b8c4cc0e4881b7519533dca": 0,
+    "477e6424d8e1dcff27c0c2c0cc02556835152d79e99533e18a3da87b0916b08d": 3,
+    "487ff1bee218aabcc9ee66052e43f14f5f17fbc11c691db2b2570724745f8e46": 4,
+    "4a212dd243e1ee3e4ef19f7f4b422d89025e97f1027ce350a3ded9958c672d37": 0,
+    "4e6771b0fb7916f5ad35ef2dc1fffbdea320dc18ff968b23332cce6b2ab45617": 4,
+    "50bbf29a0d9ed1382b495c6e0528c72e8face503cd10d0ed6e7ea9ea1caf948a": 2,
+    "519f0b956e9a7bde6ea08f25c5844412843e819778e462d43c9c03ad5edd2b94": 0,
+    "539c6e76362dae5370cb880e7848a29b0e94a7f60eb95accfedecf6d9dbf00cf": 1,
+    "53f0fe28b3e03ec00f774cad9dbbdefe2449ef33be918ad017c3d60dfc7bf0c3": 1,
+    "57b05080bb3bf2b9133ef8db4dde66ca80356ab035e3c90cdc05c9d7060c823d": 1,
+    "5d1a8fd3abb2b01fd17550a6ea97e100b7eda158ae7d3bf7c4131a72d7727ab8": 2,
+    "610f5684b7f77ae665a254d8b411f37de5d4dba4068c33902501700990c36be4": 2,
+    "61899e3d3ee2d5181e6c3fcb0118fcbe8e7f87b4ae2b3e669bff6d375156b8d3": 4,
+    "62f37080d06004a2907a356f0fd1870b1ec729b8827cd7ac12b49677160b99bc": 4,
+    "672978e9bd0102eb62a096c6001e5876711a71426290b9af0f046e18cc879c51": 0,
+    "677999c3987447d3c07408cbbcacad17f27e3b6e22f68e02669832014d8725ff": 0,
+    "69ab508a375bcae09e649ba6bad3ed48dd10d05de101e034eaecbaa3f5c37d88": 3,
+    "6f24fd1e699471f0b9ded2621f3022b68bf833224f7791c56cd9d281356efc4e": 3,
+    "71459fc71afa74685b60d529b9259be1630db836b5dea7d1fc04a1bc961a8cd7": 1,
+    "73bcfe2773823b20e92f2aae19d6b4bf01ba0b13d90092221d0becfb606ff2ea": 3,
+    "743408b3f822cbcaf60f377a5b3c083cb6ffe2b4de9c89a5b6bf8feb98cd2617": 0,
+    "748198b98b24100a96b2c82dfa35c6234cb82e29d74d195af5f19685c16c20d0": 4,
+    "7618231c73729f55e50cb2c0ec930b5ef7faa5f7c31a6ea9adab136c0501483c": 0,
+    "79666236d8e13cfd58e358a514335e17277fbd4c4055654cb28d51b51aa416fa": 3,
+    "79b6484f4739f7a9877fd03cef00c4f8851f7bf7bf02b89ce19af0ec10ca1f9c": 4,
+    "7ab73488ae24f6b2c0334f1a1ce50f3329ae4af6ada2715795627e63bf5e56c1": 3,
+    "7c9ad9098793924b293b0b518510c4653e26d3bac1b7c506f3a76ebbd499b079": 4,
+    "7d99fb134a77b53ff7a0751a097afd675cec0da2c5a1c184972e69ef21fc0370": 2,
+    "819cab28d1fcaed1274d71ccba29668121e36e799c9f708d2b7e1a1b6f7fa9ab": 4,
+    "83a2d2d6c6dc3e8120625b78d83f72b7f5dd33b0dd2756028aab83dfdd1ffd00": 2,
+    "842cc82bb9039c364344352e59fe6b798e00eec1ce9e79d3e8c21dd76ce804d9": 3,
+    "84ee77922e17cc31100828da9ce759001d86a75e8bf1c01e6122f66f5e0bb8ba": 4,
+    "880fe1f53b1c54246ff1b1e34701c3d18a706c2f4208a1a223f92fb9bfc28b76": 4,
+    "89a1bc6ec8d4f7a6225d935e69af6c1e86592e4cd81a73765f196fe1c17d1b85": 0,
+    "8bde4f7aa4d7d4eed7f79fd92fb3ed59eba5235b4e5416bce4c7bc843c3e29af": 0,
+    "8d5871380f260d1a703e5e699075aff9abccb7d0586c94cecd0415e3a0eb0949": 2,
+    "90b574274f68eb5e24980c381e1942312b72dc1319b816885bf65e18b9e54038": 3,
+    "9158dd63f7edaa8365a743916d031fed575819f8eded9ba6c317dcb62a6b23b9": 1,
+    "9250afb478ca5d830eb1a6ad1f41cbb2182d00ea86b1efbdd927ec30928eeb0a": 1,
+    "92819eb7fb9311343a907bccf0fc1dfa81a961bfeed7d5ee392948fd95631364": 0,
+    "92ba815dc7e421a4b7ea98853b3ebf7edbc9fed3e97c612fc797bf265bfcd21b": 1,
+    "95b2cf319cee7a53c40d83efbfa7680d14e78c08edae155a3f905a7937502e1b": 1,
+    "9822412b2490e553eb9ce3725aee78f96dbcb877d58a15cbd174adb0400e4dcc": 4,
+    "9a38073114b26a5d9db6f3f53004fbb1589e73015f6db4a02a6449ade9fdde1e": 1,
+    "9bddb7e0a1d0e713043380611aa7a16b8cbc039e547d5d6574696cd0e1d4039a": 3,
+    "9e790c7f3664ec167d693ad3e8578c0f0f012836a56d0c25a78257b25b147709": 0,
+    "9f569e6d1471e40e909510d7b908e6beb7ac973a35ba7f462f664d40d5a6a8d7": 2,
+    "a0303de6d5353a919f6250e14d972405a916e9848d1cc6d4bbda17e94ef2b566": 3,
+    "a13628ae64e96b57cf12ba3be7060e5c732d9b39299dd42e0ee9912b214c2dfb": 2,
+    "a2ac9b1f6bfd534c1d2b41432fd8746b32c8918ada08a9b64f5e7894bb788413": 3,
+    "a5e645d11b05458f4c11e61fa7e9ddbb142392cde856db6c6f44c9ee1c7cfe70": 0,
+    "a6c5e7f0aabbc4df811798a9f5cdce564faa8b9709113bc4c34bf6911dd09d1f": 0,
+    "a77a05c9fcb537f39a95134c0f461d90c4302233a8b80d0cdd207db37be8cc19": 2,
+    "a7e04e0ca6b8c9a6f9c16feb44beba00dcc91af0d64e452aff8cd4d84a7796f4": 0,
+    "agnews/train/101434": 1,
+    "agnews/train/101841": 1,
+    "agnews/train/103464": 1,
+    "agnews/train/1838": 2,
+    "agnews/train/2778": 2,
+    "agnews/train/30876": 2,
+    "agnews/train/44038": 3,
+    "agnews/train/50813": 0,
+    "agnews/train/55417": 0,
+    "agnews/train/62009": 1,
+    "agnews/train/75316": 3,
+    "agnews/train/85441": 0,
+    "agnews/train/8621": 4,
+    "agnews/train/92122": 0,
+    "agnews/train/95856": 1,
+    "agnews/train/96325": 3,
+    "amazon/train/101520": 3,
+    "amazon/train/10720": 2,
+    "amazon/train/113503": 4,
+    "amazon/train/13204": 2,
+    "amazon/train/141381": 1,
+    "amazon/train/189583": 3,
+    "amazon/train/193840": 2,
+    "amazon/train/27682": 2,
+    "amazon/train/36553": 0,
+    "amazon/train/37076": 0,
+    "amazon/train/42646": 0,
+    "amazon/train/60569": 0,
+    "amazon/train/62071": 4,
+    "amazon/train/69675": 1,
+    "amazon/train/73928": 0,
+    "amazon/train/95046": 0,
+    "b1d3ce4cf13a9a547d1c9c7b370279c8643baf0f5da78ad372d63c6b56929bd6": 3,
+    "b1ef55c84c759c6b7d2560d94a1fbc7d4b6f9a0e511ea687b386b55e23fd4524": 1,
+    "b3babd2bcf5fd348a9fd60b286bcd20a88da1a1556dc7604eb6aefdd23f45cc2": 2,
+    "b63d9d2757887aa51cb931c30d306b2824d78ed1c4045d8f18a55701bc2f9244": 1,
+    "b7fd68958b5cfce063a2745ecbdbfa918901b1e036454e0e2f30c502e7ca5803": 0,
+    "banking77/train/1170": 0,
+    "banking77/train/2352": 3,
+    "banking77/train/2670": 3,
+    "banking77/train/362": 1,
+    "banking77/train/4212": 4,
+    "banking77/train/4829": 0,
+    "banking77/train/4837": 4,
+    "banking77/train/5134": 1,
+    "banking77/train/6582": 4,
+    "banking77/train/7304": 0,
+    "banking77/train/7829": 3,
+    "banking77/train/784": 3,
+    "banking77/train/8110": 4,
+    "banking77/train/9261": 3,
+    "banking77/train/96": 3,
+    "banking77/train/9867": 1,
+    "be247539e57c5c62752fbaf32142d7b2c7a1083bab71ac9538a7849966beb84f": 4,
+    "be35f32c962592e2cb919c79dec4a32bbe9a6860c93b8a45364a98b23db3b71e": 0,
+    "boolq/train/2064": 2,
+    "boolq/train/2941": 1,
+    "boolq/train/3016": 4,
+    "boolq/train/3171": 2,
+    "boolq/train/3247": 4,
+    "boolq/train/4548": 2,
+    "boolq/train/465": 1,
+    "boolq/train/4886": 3,
+    "boolq/train/5591": 4,
+    "boolq/train/6031": 3,
+    "boolq/train/6825": 1,
+    "boolq/train/695": 4,
+    "boolq/train/7839": 0,
+    "boolq/train/825": 4,
+    "boolq/train/8783": 1,
+    "boolq/train/8787": 2,
+    "c02b7ff3b7f793990e6415bce6d06f7ae01871f396e08a622cd4ecbc90d9d2d4": 2,
+    "c1f754ae69f2828935a19a50c9f255db11751f7a2f560a3366e77278f26d8ae5": 3,
+    "c2c3acaaa02566c38f9ca4489b02f85d1f67213e611cd5c08b945bf80498e14d": 2,
+    "c300a83c49301a90f31e4f72d410e78a55ec01ac4012b3b5376b9547bf501d0d": 4,
+    "c63b152da1217ee621ae83e30f9289043e5fa341a06ab2c312798d5f8c646bc3": 3,
+    "c7fbf51f62094aac9fca2c756d11454306b61b4c29068f930142c67936de30c9": 1,
+    "c97a7f1be8bce0f4c5571b701897bd2a39f76fc2cdfce59ff6576e03c25bf135": 4,
+    "cb062f2c4214c034ae25cb0e516968613728a2eb64e14ea906cd93f88f36ccc9": 1,
+    "cbd7c8aa465fb037ef130cd046523f12882b584a7276c912cc8bfb3c78aa6f66": 1,
+    "cc1d416dec07790e00bda53831f9adb3e655c8e4917b85e356178db447a0724a": 3,
+    "ccf69c4c7ab07005a221d68d7a15101b726d3ce587fb61f4550aadc6ed7a1eae": 2,
+    "cd8d7512f322650b499c579f1fed1edd49306fa9dc3bea935faaced29d3d641c": 1,
+    "composition/v7-composition/rand:if(0,not(1),and(2,3))/0": 2,
+    "composition/v7-composition/rand:not(if(0,not(1),or(2,3)))/0": 4,
+    "composition/v7-composition/rand:or(0,and(1,2))/0": 4,
+    "composition/v7-composition/rand:or(0,or(1,or(2,3)))/0": 3,
+    "composition/v7-composition/rand:or(and(0,1),and(2,3))/0": 2,
+    "composition/v7-composition/rand:or(or(0,1),or(2,3))/0": 0,
+    "composition/v7-composition/rand:unless(0,and(1,2))/0": 2,
+    "composition/v7-composition/rand:unless(not(or(0,1)),not(and(2,3)))/0": 3,
+    "composition/v7-composition/rand:unless(or(0,1),and(2,3))/0": 3,
+    "composition/v7-composition/rand:unless(or(not(0),unless(1,2)),3)/0": 4,
+    "d57182545fdc6c4a565e3c8d84dcddeee87c18647ce6a27d531fa1ec9e859f75": 0,
+    "d60031e7eb0d6c99149aaaf80c4a0c3a3ddc2f5338cc8f299111c4007612d0d3": 3,
+    "d96aca9fd4e509ae47f53fdeac366e29c35c3d6e00a50431467fa5d38abede58": 1,
+    "dbpedia14/train/103965": 0,
+    "dbpedia14/train/133544": 4,
+    "dbpedia14/train/142027": 2,
+    "dbpedia14/train/142687": 1,
+    "dbpedia14/train/170048": 3,
+    "dbpedia14/train/178589": 0,
+    "dbpedia14/train/350925": 3,
+    "dbpedia14/train/367859": 2,
+    "dbpedia14/train/411444": 2,
+    "dbpedia14/train/426152": 3,
+    "dbpedia14/train/430585": 4,
+    "dbpedia14/train/445823": 2,
+    "dbpedia14/train/462388": 4,
+    "dbpedia14/train/57204": 3,
+    "dbpedia14/train/61328": 4,
+    "dbpedia14/train/97525": 0,
+    "dc2da7d54a4979185d586e0952009e0f3b10593358f750a4b41fe481a6228097": 4,
+    "dc38ea06ceffc71a0e722ae5087c81d29ca6f370a133ad273b86eebad2bdfcb7": 0,
+    "dcfbd121238b76104762f45f9decccd7b36bfe9531876f1ff99533e114d8b5c1": 3,
+    "de1b3c77c56692a4c68528e1f026c323a57cb0fe0a1609eb000ecc20b6ef519b": 4,
+    "df07c29379ecc67e05a72f3c3f212b43e98a39363af772af98f4b278ab507780": 2,
+    "e4280c0207daf1f825826429bcc3732571a9befb86364b040e0e905ed8f4978a": 3,
+    "e538bcad8e7da3f3c3e9b570a9fd22fd6fc3479ca91918ae2baaea5e742f46fd": 4,
+    "e544bdc087e91f802be16f67d009ef7bd3b92a13b8d356edbc7c5d71ebbbee86": 2,
+    "e7088e7244c807f9935643e2cf093f02b15ca69e362fdbe923f1a85d9a6c9d72": 0,
+    "ea1eb0fb9d3f315ee492f8ce4f556d0333b6feade638af0c075a5f8d966c054e": 0,
+    "eab1ab1d2744f9b826126b369397673aabaad117a27cf575baff824bf79d0581": 3,
+    "ec2a4d8d9a1f01b22dfaa1839ec35a40a4373400c2f5f8f486768682f196e8f7": 4,
+    "ec8ffb8e238e15ae189ecb89bf75bac87629972a04d32b051ec65d70b3792bfe": 3,
+    "excytin-incident_5": 1,
+    "f23cf292ccddaea5d1480bbaa348f4bbec1f338ec62225d22ea54f2f595d7682": 2,
+    "f4788c0d6da5eff15b004d43796385a5bc61f31c36527f81eef8fbeedaed4eba": 0,
+    "f6a0e34bc6e7efddc234d90e60b8783767b7250af70cf25cca230a3638e3e434": 2,
+    "f72bd080dc73fba94a90b9f4bf8a79830dc74d564a24d72c5282883f6893ce4d": 2,
+    "f8cee5052b6f7f3c998c920c6504883915b55dc57392808a253d3f82a38ade51": 4,
+    "fa01b23b5c12b48899914863c0b45ac80939e81cc1f978f4d5ebf836268ce50d": 1,
+    "fa965f9322839e52214b133a456bb1ce274e93175fd04df9b89fa87ca430ba6f": 3,
+    "fc61c13973ccddaae6b9a9d01fae876bc7d968c212575e550db651cc7564464f": 4,
+    "imdb/train/10025": 4,
+    "imdb/train/10495": 1,
+    "imdb/train/11233": 3,
+    "imdb/train/13887": 4,
+    "imdb/train/16581": 1,
+    "imdb/train/18230": 0,
+    "imdb/train/18445": 2,
+    "imdb/train/19308": 4,
+    "imdb/train/19651": 0,
+    "imdb/train/20668": 1,
+    "imdb/train/22475": 3,
+    "imdb/train/23621": 3,
+    "imdb/train/2638": 3,
+    "imdb/train/5139": 0,
+    "imdb/train/614": 1,
+    "imdb/train/8968": 0,
+    "mnli/train/127796": 1,
+    "mnli/train/145863": 4,
+    "mnli/train/153623": 1,
+    "mnli/train/154356": 0,
+    "mnli/train/176070": 4,
+    "mnli/train/177065": 3,
+    "mnli/train/194867": 4,
+    "mnli/train/238065": 4,
+    "mnli/train/26448": 0,
+    "mnli/train/270460": 3,
+    "mnli/train/271014": 0,
+    "mnli/train/282280": 2,
+    "mnli/train/3061": 2,
+    "mnli/train/74568": 2,
+    "mnli/train/79641": 3,
+    "mnli/train/98987": 3,
+    "otrf-native/1d4be09fa1a078b2b9b040e062b551960e35deab69ebdc8c27a2fb32bc010076": 3,
+    "otrf-native/67eb1f409d797dc97902ad614c82fe0c9b84a50221385178a145fe47a4fc03f0": 2,
+    "otrf-native/68888ef6ab395837e591e2907cd125821d09c5426be69aa4061fd0a0d013f4d2": 2,
+    "otrf-native/8509642c585ca4179828522e5b36bac4a628f0f4c8e81b66c59d3a548515d61d": 0,
+    "sst5/train/1203": 3,
+    "sst5/train/1272": 2,
+    "sst5/train/1307": 1,
+    "sst5/train/2007": 1,
+    "sst5/train/2813": 3,
+    "sst5/train/4590": 1,
+    "sst5/train/4672": 0,
+    "sst5/train/4673": 4,
+    "sst5/train/5770": 3,
+    "sst5/train/6597": 1,
+    "sst5/train/6821": 1,
+    "sst5/train/7169": 0,
+    "sst5/train/7338": 2,
+    "sst5/train/7401": 0,
+    "sst5/train/7503": 2,
+    "sst5/train/8048": 0,
+    "swarm-component/R0000011": 0,
+    "swarm-component/R0000077": 1,
+    "swarm-component/R0000373": 0,
+    "swarm-component/R0000471": 2,
+    "swarm-component/R0001019": 3,
+    "swarm-component/R0001129": 4,
+    "swarm-component/R0001187": 1,
+    "swarm-component/R0002932": 0,
+    "swarm-component/R0002968": 2,
+    "swarm-component/R0003511": 1,
+    "swarm-component/R0003667": 0,
+    "swarm-component/R0004727": 3,
+    "swarm-component/R0004885": 2,
+    "swarm-component/R0005634": 0,
+    "swarm-component/R0005881": 4,
+    "swarm-component/R0005904": 3,
+    "swarm-component/R0006488": 2,
+    "swarm-component/R0006694": 2,
+    "swarm-component/R0006896": 0,
+    "swarm-component/R0007144": 1,
+    "swarm-component/R0007661": 4,
+    "swarm-component/R0007892": 3,
+    "swarm-component/R0008007": 0,
+    "swarm-component/R0008389": 2,
+    "swarm-component/R0008401": 2,
+    "swarm-component/R0008559": 1,
+    "swarm-component/R0008699": 1,
+    "swarm-component/R0009389": 1,
+    "swarm-component/R0009392": 3,
+    "swarm-component/R0009402": 3,
+    "swarm-component/R0009411": 4,
+    "swarm-component/R0009601": 0,
+    "swarm-component/R0009692": 1,
+    "swarm-component/R0009823": 2,
+    "swarm-component/R0010274": 1,
+    "swarm-component/R0010400": 2,
+    "swarm-component/R0010618": 3,
+    "swarm-component/R0011117": 4,
+    "swarm-component/R0011639": 0,
+    "swarm-component/R0012190": 1,
+    "swarm-component/R0012458": 3,
+    "swarm-component/R0012849": 1,
+    "swarm-component/R0014589": 2,
+    "swarm-component/R0014654": 2,
+    "swarm-component/R0015100": 0,
+    "swarm-component/R0016127": 4,
+    "swarm-component/R0016415": 3,
+    "swarm-component/R0017170": 1,
+    "swarm-component/R0019072": 2,
+    "swarm-component/R0019298": 2,
+    "swarm-component/R0019733": 1,
+    "swarm-component/R0020060": 4,
+    "swarm-component/R0020534": 3,
+    "swarm-component/R0020939": 1,
+    "swarm-component/R0021323": 3,
+    "swarm-component/R0022509": 1,
+    "swarm-component/R0022827": 4,
+    "swarm-component/R0022873": 3,
+    "swarm-component/R0023399": 2,
+    "swarm-component/R0023567": 2,
+    "swarm-component/R0023776": 4,
+    "swarm-component/R0024532": 4,
+    "swarm-component/R0024644": 4,
+    "swarm-component/R0024898": 1,
+    "swarm-component/R0025731": 1,
+    "swarm-component/R0027596": 0,
+    "swarm-component/R0028886": 1,
+    "swarm-component/R0031561": 3,
+    "swarm-component/R0031694": 4,
+    "swarm-component/R0031927": 2,
+    "swarm-component/R0035144": 1,
+    "swarm-component/R0036036": 4,
+    "swarm-component/R0036206": 2,
+    "swarm-component/R0038167": 4,
+    "swarm-component/R0038251": 2,
+    "swarm-component/R0038452": 1,
+    "swarm-component/R0038633": 2,
+    "swarm-component/R0040302": 4,
+    "swarm-component/R0041141": 1,
+    "swarm-component/R0042114": 4,
+    "swarm-component/R0043643": 1,
+    "swarm-component/R0044813": 1,
+    "swarm-component/R0044904": 3,
+    "swarm-component/R0044954": 0,
+    "swarm-component/R0045177": 3,
+    "swarm-component/R0046203": 3,
+    "swarm-component/R0046285": 4,
+    "swarm-component/R0046434": 0,
+    "swarm-component/R0046694": 3,
+    "swarm-component/R0046813": 4,
+    "swarm-component/R0046975": 4,
+    "swarm-component/R0047014": 1,
+    "swarm-component/R0047174": 0,
+    "swarm-component/R0047180": 4,
+    "swarm-component/R0047288": 0,
+    "swarm-component/R0047753": 2,
+    "swarm-component/R0048233": 2,
+    "swarm-component/R0050222": 4,
+    "swarm-component/R0050265": 3,
+    "swarm-component/R0050299": 2,
+    "swarm-component/R0050565": 1,
+    "swarm-component/R0053242": 0,
+    "swarm-component/R0082618": 0,
+    "swarm-component/R0082689": 2,
+    "swarm-component/R0083001": 0,
+    "swarm-component/R0127498": 1,
+    "trec/train/1120": 1,
+    "trec/train/1140": 1,
+    "trec/train/1343": 0,
+    "trec/train/1612": 2,
+    "trec/train/2030": 2,
+    "trec/train/2336": 4,
+    "trec/train/2962": 4,
+    "trec/train/3021": 3,
+    "trec/train/3530": 3,
+    "trec/train/3685": 4,
+    "trec/train/4176": 4,
+    "trec/train/4363": 3,
+    "trec/train/5154": 3,
+    "trec/train/5435": 0,
+    "trec/train/614": 1,
+    "trec/train/857": 2,
+    "v3-control-age_eligibility-0001": 4,
+    "v3-control-age_eligibility-0002": 3,
+    "v3-control-age_eligibility-0003": 3,
+    "v3-control-age_eligibility-0004": 0,
+    "v3-control-age_eligibility-0007": 4,
+    "v3-control-late_fee-0004": 3,
+    "v3-control-quantity_limit-0007": 0,
+    "v3-control-return_window-0003": 4,
+    "v3-control-return_window-0006": 2,
+    "v3-control-return_window-0007": 2,
+    "v3-control-sla_response-0001": 1,
+    "v3-control-sla_response-0003": 3,
+    "v3-control-sla_response-0005": 1,
+    "v3-control-sla_response-0006": 0,
+    "v3-control-sla_response-0007": 4,
+    "v3-control-spend_threshold-0004": 1,
+    "v3-control-spend_threshold-0005": 0,
+    "v3-control-volume_discount-0000": 1,
+    "v3-control-volume_discount-0002": 1,
+    "v3-control-volume_discount-0005": 0,
+    "yelp/train/10956": 0,
+    "yelp/train/186057": 1,
+    "yelp/train/194502": 0,
+    "yelp/train/221928": 2,
+    "yelp/train/242003": 0,
+    "yelp/train/286936": 1,
+    "yelp/train/291134": 3,
+    "yelp/train/31319": 1,
+    "yelp/train/319489": 0,
+    "yelp/train/386581": 2,
+    "yelp/train/394666": 4,
+    "yelp/train/408648": 0,
+    "yelp/train/527065": 3,
+    "yelp/train/564946": 4,
+    "yelp/train/630291": 2,
+    "yelp/train/75560": 0
+  },
+  "folds": 5,
+  "seed": 20260927,
+  "groups": 437,
+  "source_overridden_only_in_cv_copies": true,
+  "state_text_included": false
 }

calibration.json

@@ -1,313 +1,161 @@
 {
-  "schema": "sev-research-calibrated-copy-v1",
-  "created_at": "2026-09-25T16:37:54.666347+00:00",
-  "checkpoint": "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint",
+  "temperature": 1.6245047927124707,
+  "split": "calibration",
+  "n": 2534,
+  "records": 1892,
   "fit": {
-    "schema": "sev-research-calibration-v1",
-    "temperature": 1.9999999999999998,
-    "split": "calibration",
-    "view": "full",
-    "question": "operator_origin",
-    "method": "kev.metrics.fit_temperature",
-    "aggregation": "macro",
-    "task": "episode parent_id",
-    "points": 121,
-    "temperature_range": [
-      0.25,
-      4.0
-    ],
-    "weighting": "equal episodes, then equal windows within each episode",
-    "windows": 182,
-    "episodes": 96,
-    "groups": 32,
-    "record_ids_sha256": "46528c7e8fa5d195456afb8352b64a1393b26852a868eb58fa7aee9d41e1d93b",
-    "excluded_views": {
-      "prefix_loss": 148,
-      "sampled_loss": 152
-    },
-    "raw": {
-      "acc": 0.3342013888888889,
-      "nll": 1.0988623728398248,
-      "brier": 0.6670543370614523,
-      "ece": 0.06554797243603253
-    },
-    "calibrated": {
-      "acc": 0.3342013888888889,
-      "nll": 1.094225810223573,
-      "brier": 0.6635708549658472,
-      "ece": 0.04216089485176933
+    "aggregation": "micro",
+    "points": 81
+  },
+  "range": [
+    0.25,
+    4.0
+  ],
+  "method": "Minimum question-weighted raw-logit NLL on the registered 81-point grid",
+  "rows_sha256": "5dc5633f59c9da863561e1ae34fbc1c87199053cf1397d4592b921ec5ff69790",
+  "suite_sha256": "61e3c1898b8a03eb178355d3c8c5050e85bd217285c700df9b4506ec9e3680f8",
+  "partition_sha256": "8504531b423590392325f9e1d5fd291280122e6ca038483268e05bfe94970f77",
+  "raw": {
+    "n": 2534,
+    "nll": 0.4793191240878706,
+    "acc": 0.7750591949486977,
+    "ece": 0.06259762046608341,
+    "brier": 0.277178481678158,
+    "mean_conf": 0.8376568154147812,
+    "confident_error_rate": 0.010655090765588003,
+    "coverage_at_0_9": 0.6298342541436464,
+    "accuracy_at_0_9": 0.9830827067669173,
+    "coverage_at_5pct_error": 0.6566692975532754,
+    "coverage_at_1pct_error": 0.5256511444356748,
+    "aurc": 0.05421101630801481,
+    "error_rate_at_0_9": 0.016917293233082706,
+    "confidence_bias": 0.06259762046608353,
+    "top_bins": {
+      "0.9": {
+        "n": 1596,
+        "errors": 27,
+        "error_rate": 0.016917293233082706
+      },
+      "0.95": {
+        "n": 1564,
+        "errors": 22,
+        "error_rate": 0.014066496163682864
+      },
+      "0.99": {
+        "n": 1498,
+        "errors": 18,
+        "error_rate": 0.012016021361815754
+      }
     },
-    "cross_validation": {
-      "raw": {
-        "n": 182,
-        "ece": 0.030378155838447805,
-        "brier": 0.6564945291407126,
-        "nll": 1.0838391530675782,
-        "confident_error_rate": 0.0,
-        "coverage_at_5pct_error": 0.01098901098901099
-      },
-      "out_of_fold": {
-        "n": 182,
-        "ece": 0.02879543978652912,
-        "brier": 0.6607993702192454,
-        "nll": 1.0900062934624115,
-        "confident_error_rate": 0.0,
-        "coverage_at_5pct_error": 0.0
-      },
-      "ece_ci95": {
-        "raw": [
-          0.013784180181410588,
-          0.07820699846672471
-        ],
-        "out_of_fold": [
-          0.015176680491519571,
-          0.06251847627618651
-        ],
-        "delta": [
-          -0.045586065150116414,
-          0.03442859983142514
-        ]
-      },
-      "separated": false,
-      "temperatures": [
-        1.5511447618337346,
-        2.578740616879158,
-        1.6624757922855755,
-        2.3510958125672174,
-        2.70069989233638
-      ],
-      "folds": 5,
-      "seed": 0,
-      "samples": 1000,
-      "groups": 32,
-      "unit": "latent scenario group_id, across all original sources",
-      "fit_weighting": "equal episodes, then equal windows within each episode",
-      "canonical_metric_weighting": "windows; canonical raw/out_of_fold metrics and ECE intervals",
-      "episode_weighted": {
-        "raw": {
-          "acc": 0.3342013888888889,
-          "nll": 1.0988623728398248,
-          "brier": 0.6670543370614523,
-          "ece": 0.06554797243603253
-        },
-        "out_of_fold": {
-          "acc": 0.3342013888888889,
-          "nll": 1.0963892736496,
-          "brier": 0.6651579277109647,
-          "ece": 0.061642028028613405
-        }
-      },
-      "fold_by_group": {
-        "0a96bbbde731f77629d6": 0,
-        "0fd4a07c8a94efc8213a": 4,
-        "127dba0a0b6d112fdd60": 0,
-        "1a7d4eff40c538e9531b": 2,
-        "25f1f108010761f574ec": 0,
-        "42172cc6c0db60ca5758": 1,
-        "459d3a949d71768ba55d": 4,
-        "4c3fe25e9bd641be2ddb": 0,
-        "4c6ad83158be617c1548": 4,
-        "53c1f89988e203c21d7f": 1,
-        "635e51a0d3a3a3da2ba7": 4,
-        "7a15bda3e41ccfee7241": 1,
-        "7f573f974db9fa0d872d": 2,
-        "86f21fea379a7a9fe79e": 4,
-        "896e70c3467400fe789c": 3,
-        "89a112d154659e4b8585": 0,
-        "8de646371957ac833b8f": 2,
-        "9267379e905f6045e976": 2,
-        "9343076eec6e0c57c601": 0,
-        "95cf7a4cbfcb6a117bfc": 1,
-        "96651d23eb4107dd8e5a": 3,
-        "99fba8dee219cdf16eef": 3,
-        "9f8e8516bf49e0ec08d5": 0,
-        "a10ba4c34ba318f3829a": 3,
-        "ac1f21ab8ebefecfe315": 3,
-        "ace673b4d894d1f1740e": 2,
-        "ae5e6811a635cc23dfbd": 1,
-        "b786a1db55e5f652c1cf": 4,
-        "b8d7d7b733ad6a06fade": 2,
-        "d983184d237cd08a0327": 1,
-        "e90cd441191e7f18968b": 3,
-        "ed1a95e2dd7115022ea3": 1
+    "selective": {
+      "0.5": {
+        "coverage": 0.5173638516179953,
+        "accuracy": 0.9908466819221968,
+        "confidence_cutoff": 0.9961011329746132
+      },
+      "0.8": {
+        "coverage": 0.8011049723756906,
+        "accuracy": 0.8605911330049261,
+        "confidence_cutoff": 0.5618087607487383
       }
     },
-    "per_source": {
-      "sev_behavioral_artifact_handoff": {
-        "windows": 63,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "ec2841fda8a390bd2d73d828e1d3efac393d6cc583f94188ef25588a85945877",
-        "raw": {
-          "acc": 0.34722222222222215,
-          "nll": 1.0806702183699979,
-          "brier": 0.6537004910585257,
-          "ece": 0.09166280712269316
-        },
-        "calibrated": {
-          "acc": 0.34722222222222215,
-          "nll": 1.0876702845061252,
-          "brier": 0.6589248476452055,
-          "ece": 0.04426971102023442
-        }
-      },
-      "sev_behavioral_authorized_export": {
-        "windows": 40,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "255e72d055b44aa16ec7f11fe905f4c251fe28ced7f801120abef039c930115f",
-        "raw": {
-          "acc": 0.3333333333333333,
-          "nll": 1.104011417056653,
-          "brier": 0.6698152020985213,
-          "ece": 0.04556619047184074
-        },
-        "calibrated": {
-          "acc": 0.3333333333333333,
-          "nll": 1.0996799880244128,
-          "brier": 0.6672264414223563,
-          "ece": 0.04735674179920909
-        }
-      },
-      "sev_behavioral_lookup_catalog": {
-        "windows": 25,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "8cd835d6e3fefa25803ffbc579240aa9c4ae264163027eaa186db00580e4b2e1",
-        "raw": {
-          "acc": 0.3333333333333333,
-          "nll": 1.0929036665832168,
-          "brier": 0.6633979694737079,
-          "ece": 0.10676289475303495
-        },
-        "calibrated": {
-          "acc": 0.3333333333333333,
-          "nll": 1.087115774278896,
-          "brier": 0.6586049595361453,
-          "ece": 0.052123650581165686
-        }
-      },
-      "sev_behavioral_update_export": {
-        "windows": 54,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "f83b45d350186cfa47db970f9311408fbde0fb68bb884adc1a3644a36b18f52c",
-        "raw": {
-          "acc": 0.3229166666666667,
-          "nll": 1.11786418934943,
-          "brier": 0.6813036856150539,
-          "ece": 0.09059596392971296
-        },
-        "calibrated": {
-          "acc": 0.3229166666666667,
-          "nll": 1.1024371940848587,
-          "brier": 0.6695271712596828,
-          "ece": 0.060665733732341603
-        }
+    "score_mae": 0.29175006232796596,
+    "ranked_probability_score": 0.05078441644312436
+  },
+  "calibrated_in_sample": {
+    "n": 2534,
+    "nll": 0.4493040826220053,
+    "acc": 0.7750591949486977,
+    "ece": 0.04011689914688196,
+    "brier": 0.2632611879225425,
+    "mean_conf": 0.7930262042717668,
+    "confident_error_rate": 0.00749802683504341,
+    "coverage_at_0_9": 0.6045777426992897,
+    "accuracy_at_0_9": 0.987597911227154,
+    "coverage_at_5pct_error": 0.6606156274664562,
+    "coverage_at_1pct_error": 0.5284135753749013,
+    "aurc": 0.05443998254725242,
+    "error_rate_at_0_9": 0.012402088772845953,
+    "confidence_bias": 0.01796700932306916,
+    "top_bins": {
+      "0.9": {
+        "n": 1532,
+        "errors": 19,
+        "error_rate": 0.012402088772845953
+      },
+      "0.95": {
+        "n": 1342,
+        "errors": 14,
+        "error_rate": 0.010432190760059613
+      },
+      "0.99": {
+        "n": 773,
+        "errors": 0,
+        "error_rate": 0.0
       }
     },
-    "development_read": false,
-    "test_read": false,
-    "scope": "Calibration evidence on authored simulated controllers; no field accuracy or release gate.",
-    "suite_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-    "partition_sha256": "eebb5ddeb322b16bbb312ddb6927b1d71226b72e4b01d323c698d1e2a3c4f38d",
-    "predictions": {
-      "path": "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/rows.json",
-      "sha256": "00909b4d6b838c534d8d403d84a578fea1ddaac5e30a2b488c2d1c138c0d05ab"
+    "selective": {
+      "0.5": {
+        "coverage": 0.5185477505919495,
+        "accuracy": 0.9908675799086758,
+        "confidence_cutoff": 0.9587920809563001
+      },
+      "0.8": {
+        "coverage": 0.8003157063930545,
+        "accuracy": 0.8594674556213018,
+        "confidence_cutoff": 0.47705693179262204
+      }
     },
-    "input_sha256": {
-      "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/manifest.json": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-      "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/calibration.jsonl": "eebb5ddeb322b16bbb312ddb6927b1d71226b72e4b01d323c698d1e2a3c4f38d",
-      "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/provenance.json": "f540a4587f44f4a3bdda19f68a42377a56c56d464443beaaf8fe57b593beb85f",
-      "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/result.json": "ec4bfe6436093b6ba096921bb95b2643e8418c84d1e5cbbc596d78f2c744338c",
-      "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/rows.json": "00909b4d6b838c534d8d403d84a578fea1ddaac5e30a2b488c2d1c138c0d05ab",
-      "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/report.json": "6bee81d641d7003c7715c0b26bcec77fe244e59e847e52e7f14f7336d88f59f9",
-      "/Users/mac/Sev-release-20260925/kev/checkpoint.py": "9f1dbc5d3d7d3050aca4ee34e0578c9ad76b7a2a7467dfc7a4ad65e3c471db59",
-      "/Users/mac/Sev-release-20260925/kev/metrics.py": "db0930d470c0ce16f16dcaf726d49a7a103973a5296cefc5259d1e68707d882b",
-      "/Users/mac/Sev-release-20260925/scripts/report_sev_behavioral_round.py": "5beee8c302040233e2950ebe2d3f8944e84d6202f08b95943d679690bcdad8d8",
-      "/Users/mac/Sev-release-20260925/scripts/calibrate_sev_research.py": "f9fa4df59fda060e856f4e2a691e8fc0aeeeec836f1ddbaa43b3db5a39a874cf"
+    "score_mae": 0.3042583923313257,
+    "ranked_probability_score": 0.04677850406395365
+  },
+  "cross_validation": {
+    "raw": {
+      "n": 2534,
+      "ece": 0.06259762046608341,
+      "brier": 0.277178481678158,
+      "nll": 0.4793191240878706,
+      "confident_error_rate": 0.010655090765588003,
+      "coverage_at_5pct_error": 0.6566692975532754
     },
-    "source_checkpoint": "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
-    "source_files": {
-      "README.md": {
-        "sha256": "e0e50894b914f1765ef42c1eade74b42c613fb0330b06339f9502cc6f2e25e79",
-        "bytes": 5164
-      },
-      "adapter_config.json": {
-        "sha256": "8a05dfd6c5e7f61a62e6db5d8de8093a8dbbd105aa7ee7b5e7b73f6249f45617",
-        "bytes": 1271
-      },
-      "adapter_model.safetensors": {
-        "sha256": "7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c",
-        "bytes": 129924032
-      },
-      "chat_template.jinja": {
-        "sha256": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
-        "bytes": 7756
-      },
-      "head.pt": {
-        "sha256": "af6bfd215bf0b7eff92612a83b21cb56754ca87f0bf9f62fdfcfbef275d9eab6",
-        "bytes": 5248895
-      },
-      "tokenizer.json": {
-        "sha256": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523",
-        "bytes": 19989325
-      },
-      "tokenizer_config.json": {
-        "sha256": "8671bed7c852ce9e661be94f179a7b4ffd091c2a65aea0363e5501c20318ee45",
-        "bytes": 1128
-      },
-      "training_config.json": {
-        "sha256": "f1936818cecded5fe659330a6e0a1ad631d9b5b1eb2014982634a58f7d6f7d68",
-        "bytes": 1889
-      },
-      "training_metrics.json": {
-        "sha256": "20b9314de861f32641b4f6a16ae8b06ba273426333da40c2820e5f591936d7a7",
-        "bytes": 321
-      }
+    "out_of_fold": {
+      "n": 2534,
+      "ece": 0.038704806329517796,
+      "brier": 0.2637612413375006,
+      "nll": 0.4500281806306658,
+      "confident_error_rate": 0.00749802683504341,
+      "coverage_at_5pct_error": 0.659431728492502
     },
-    "source_temperature": 1.0,
-    "head_tensors": {
-      "k.bias": {
-        "shape": [
-          256
-        ],
-        "dtype": "torch.float32",
-        "sha256": "b8d56826b48ca7768c910704d155b6f9368481658f4b7be5e6e87c9e3a77109a"
-      },
-      "k.weight": {
-        "shape": [
-          256,
-          2560
-        ],
-        "dtype": "torch.float32",
-        "sha256": "7caa9354998b823ae4ca2c87b849b240bf474cfb8479b43e8a34e012e1fb2d97"
-      },
-      "q.bias": {
-        "shape": [
-          256
-        ],
-        "dtype": "torch.float32",
-        "sha256": "341acaf84e9296a1a6676e25789b2ce97c8632a408ea31c3a626f48e840496e4"
-      },
-      "q.weight": {
-        "shape": [
-          256,
-          2560
-        ],
-        "dtype": "torch.float32",
-        "sha256": "b32e0568a714d926fe5faef0cffc7f0215c61f0f4874e2138303a7641a4cbfca"
-      }
+    "ece_ci95": {
+      "raw": [
+        0.038399055202893245,
+        0.11368661824213783
+      ],
+      "out_of_fold": [
+        0.024813975731529547,
+        0.07371171725362363
+      ],
+      "delta": [
+        -0.04984892118238256,
+        0.0032724258916628857
+      ]
     },
-    "calibration_source_sha256": {
-      "kev/checkpoint.py": "9f1dbc5d3d7d3050aca4ee34e0578c9ad76b7a2a7467dfc7a4ad65e3c471db59",
-      "kev/metrics.py": "db0930d470c0ce16f16dcaf726d49a7a103973a5296cefc5259d1e68707d882b",
-      "scripts/report_sev_behavioral_round.py": "5beee8c302040233e2950ebe2d3f8944e84d6202f08b95943d679690bcdad8d8",
-      "scripts/calibrate_sev_research.py": "f9fa4df59fda060e856f4e2a691e8fc0aeeeec836f1ddbaa43b3db5a39a874cf"
-    }
+    "separated": false,
+    "temperatures": [
+      1.5691681957935013,
+      1.6817928305074288,
+      1.5691681957935013,
+      1.6245047927124707,
+      1.5691681957935013
+    ],
+    "folds": 5,
+    "seed": 20260927,
+    "samples": 1000,
+    "groups": 437,
+    "unit": "source-stratified (source, group); sibling questions and variants stay together"
   },
-  "output_head_sha256": "dc1f550c7914388af77e607c70fe8801558a89a3847d37ea37d3923b28521c3e",
-  "source_unchanged": true,
-  "learned_tensors_unchanged": true,
-  "adapter_unchanged": true
+  "cv_grouping": "Exact group IDs across source families; one constant source stratum",
+  "development_used_for_fit": false,
+  "test_read": false,
+  "note": "CV is a diagnostic within calibration data, not fresh field validation. The shipped value is the full calibration fit."
 }

checkpoint-integrity.json

@@ -1,62 +1,18 @@
 {
-  "schema": "sev-calibrated-checkpoint-integrity-v1",
-  "pass": true,
-  "verified_at": "2026-09-25T16:42:29.007740+00:00",
-  "source_checkpoint": "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
-  "calibrated_checkpoint": "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint",
-  "suite_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-  "raw_head_sha256": "af6bfd215bf0b7eff92612a83b21cb56754ca87f0bf9f62fdfcfbef275d9eab6",
-  "calibrated_head_sha256": "dc1f550c7914388af77e607c70fe8801558a89a3847d37ea37d3923b28521c3e",
-  "adapter_sha256": "7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c",
-  "raw_temperature": 1.0,
-  "released_temperature": 1.9999999999999998,
-  "training": {
-    "wall_seconds": 371.20887875556946,
-    "records_seen": 2334,
-    "requested_records": 2334,
-    "truncated_records": 0,
-    "rejected_records": 0,
-    "optimizer_steps": 292,
-    "forward_tokens": 738327,
-    "peak_device_bytes": 18913074688,
-    "device": "cuda",
-    "dtype": "bf16",
-    "batch": 4,
-    "peak_rss_bytes": 31561670656
+  "raw_checkpoint": "runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint",
+  "raw_sha256": {
+    "head.pt": "b29abc817f2340aa510fcd4b2c813455ef38102508120ffc35f7917ce359c27e",
+    "adapter_model.safetensors": "b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420"
   },
-  "training_sources": {
-    "sev_behavioral_lookup_catalog": 391,
-    "sev_behavioral_update_export": 533,
-    "sev_behavioral_artifact_handoff": 739,
-    "sev_behavioral_authorized_export": 431,
-    "compositional": 160,
-    "legacy_policy": 80
-  },
-  "warm_start": {
-    "init_from": "jaredpalmer/kev-4b@485ace8703592fcf405488b262449990824cfed1",
-    "resolved": "/__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--jaredpalmer--kev-4b/snapshots/485ace8703592fcf405488b262449990824cfed1",
-    "adapter_sha256": "9797de69a42188e411b17b7b4fcb66a23374dcebc21d71a7a66f836b5d34df2b",
-    "head_sha256": "d8f796da36ff7bd7c0fb9496b452139bb7851af4fc82b07b500b682d3f721d6a",
-    "adapter_tensors": 496
-  },
-  "adapter_tensors": 496,
+  "release_head_sha256": "1d50c37b4db60c22ae8e6bb24f5055056701bb16e332964350d76df57dc82af0",
+  "release_adapter_sha256": "b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420",
   "head_tensors": {
-    "q.weight": {
-      "shape": [
-        256,
-        2560
-      ],
-      "dtype": "torch.float32",
-      "raw_bytes_sha256": "b32e0568a714d926fe5faef0cffc7f0215c61f0f4874e2138303a7641a4cbfca",
-      "byte_identical": true
-    },
-    "q.bias": {
+    "k.bias": {
       "shape": [
         256
       ],
       "dtype": "torch.float32",
-      "raw_bytes_sha256": "341acaf84e9296a1a6676e25789b2ce97c8632a408ea31c3a626f48e840496e4",
-      "byte_identical": true
+      "sha256": "5c890db00f616c6ba9ac37e78938340109101fbf635e8337f0bfe523f07e1ffa"
     },
     "k.weight": {
       "shape": [
@@ -64,399 +20,32 @@
         2560
       ],
       "dtype": "torch.float32",
-      "raw_bytes_sha256": "7caa9354998b823ae4ca2c87b849b240bf474cfb8479b43e8a34e012e1fb2d97",
-      "byte_identical": true
+      "sha256": "8aa345e5f3d84606a49a3ef5b86cce277140ffc528536b5dd346c4deae4dfa4a"
     },
-    "k.bias": {
+    "q.bias": {
       "shape": [
         256
       ],
       "dtype": "torch.float32",
-      "raw_bytes_sha256": "b8d56826b48ca7768c910704d155b6f9368481658f4b7be5e6e87c9e3a77109a",
-      "byte_identical": true
-    }
-  },
-  "calibration": {
-    "windows": 182,
-    "episodes": 96,
-    "groups": 32,
-    "temperature": 1.9999999999999998,
-    "raw": {
-      "acc": 0.3342013888888889,
-      "nll": 1.0988623728398248,
-      "brier": 0.6670543370614523,
-      "ece": 0.06554797243603253
+      "sha256": "dca14ce4ebf8115cc3a06a7ebeff39762bf2760ec4f83180c56a83d20a834119"
     },
-    "calibrated": {
-      "acc": 0.3342013888888889,
-      "nll": 1.094225810223573,
-      "brier": 0.6635708549658472,
-      "ece": 0.04216089485176933
-    },
-    "cross_validation": {
-      "raw": {
-        "n": 182,
-        "ece": 0.030378155838447805,
-        "brier": 0.6564945291407126,
-        "nll": 1.0838391530675782,
-        "confident_error_rate": 0.0,
-        "coverage_at_5pct_error": 0.01098901098901099
-      },
-      "out_of_fold": {
-        "n": 182,
-        "ece": 0.02879543978652912,
-        "brier": 0.6607993702192454,
-        "nll": 1.0900062934624115,
-        "confident_error_rate": 0.0,
-        "coverage_at_5pct_error": 0.0
-      },
-      "ece_ci95": {
-        "raw": [
-          0.013784180181410588,
-          0.07820699846672471
-        ],
-        "out_of_fold": [
-          0.015176680491519571,
-          0.06251847627618651
-        ],
-        "delta": [
-          -0.045586065150116414,
-          0.03442859983142514
-        ]
-      },
-      "separated": false,
-      "temperatures": [
-        1.5511447618337346,
-        2.578740616879158,
-        1.6624757922855755,
-        2.3510958125672174,
-        2.70069989233638
-      ],
-      "folds": 5,
-      "seed": 0,
-      "samples": 1000,
-      "groups": 32,
-      "unit": "latent scenario group_id, across all original sources",
-      "fit_weighting": "equal episodes, then equal windows within each episode",
-      "canonical_metric_weighting": "windows; canonical raw/out_of_fold metrics and ECE intervals",
-      "episode_weighted": {
-        "raw": {
-          "acc": 0.3342013888888889,
-          "nll": 1.0988623728398248,
-          "brier": 0.6670543370614523,
-          "ece": 0.06554797243603253
-        },
-        "out_of_fold": {
-          "acc": 0.3342013888888889,
-          "nll": 1.0963892736496,
-          "brier": 0.6651579277109647,
-          "ece": 0.061642028028613405
-        }
-      },
-      "fold_by_group": {
-        "0a96bbbde731f77629d6": 0,
-        "0fd4a07c8a94efc8213a": 4,
-        "127dba0a0b6d112fdd60": 0,
-        "1a7d4eff40c538e9531b": 2,
-        "25f1f108010761f574ec": 0,
-        "42172cc6c0db60ca5758": 1,
-        "459d3a949d71768ba55d": 4,
-        "4c3fe25e9bd641be2ddb": 0,
-        "4c6ad83158be617c1548": 4,
-        "53c1f89988e203c21d7f": 1,
-        "635e51a0d3a3a3da2ba7": 4,
-        "7a15bda3e41ccfee7241": 1,
-        "7f573f974db9fa0d872d": 2,
-        "86f21fea379a7a9fe79e": 4,
-        "896e70c3467400fe789c": 3,
-        "89a112d154659e4b8585": 0,
-        "8de646371957ac833b8f": 2,
-        "9267379e905f6045e976": 2,
-        "9343076eec6e0c57c601": 0,
-        "95cf7a4cbfcb6a117bfc": 1,
-        "96651d23eb4107dd8e5a": 3,
-        "99fba8dee219cdf16eef": 3,
-        "9f8e8516bf49e0ec08d5": 0,
-        "a10ba4c34ba318f3829a": 3,
-        "ac1f21ab8ebefecfe315": 3,
-        "ace673b4d894d1f1740e": 2,
-        "ae5e6811a635cc23dfbd": 1,
-        "b786a1db55e5f652c1cf": 4,
-        "b8d7d7b733ad6a06fade": 2,
-        "d983184d237cd08a0327": 1,
-        "e90cd441191e7f18968b": 3,
-        "ed1a95e2dd7115022ea3": 1
-      }
-    },
-    "per_source": {
-      "sev_behavioral_artifact_handoff": {
-        "windows": 63,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "ec2841fda8a390bd2d73d828e1d3efac393d6cc583f94188ef25588a85945877",
-        "raw": {
-          "acc": 0.34722222222222215,
-          "nll": 1.0806702183699979,
-          "brier": 0.6537004910585257,
-          "ece": 0.09166280712269316
-        },
-        "calibrated": {
-          "acc": 0.34722222222222215,
-          "nll": 1.0876702845061252,
-          "brier": 0.6589248476452055,
-          "ece": 0.04426971102023442
-        }
-      },
-      "sev_behavioral_authorized_export": {
-        "windows": 40,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "255e72d055b44aa16ec7f11fe905f4c251fe28ced7f801120abef039c930115f",
-        "raw": {
-          "acc": 0.3333333333333333,
-          "nll": 1.104011417056653,
-          "brier": 0.6698152020985213,
-          "ece": 0.04556619047184074
-        },
-        "calibrated": {
-          "acc": 0.3333333333333333,
-          "nll": 1.0996799880244128,
-          "brier": 0.6672264414223563,
-          "ece": 0.04735674179920909
-        }
-      },
-      "sev_behavioral_lookup_catalog": {
-        "windows": 25,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "8cd835d6e3fefa25803ffbc579240aa9c4ae264163027eaa186db00580e4b2e1",
-        "raw": {
-          "acc": 0.3333333333333333,
-          "nll": 1.0929036665832168,
-          "brier": 0.6633979694737079,
-          "ece": 0.10676289475303495
-        },
-        "calibrated": {
-          "acc": 0.3333333333333333,
-          "nll": 1.087115774278896,
-          "brier": 0.6586049595361453,
-          "ece": 0.052123650581165686
-        }
-      },
-      "sev_behavioral_update_export": {
-        "windows": 54,
-        "episodes": 24,
-        "groups": 8,
-        "record_ids_sha256": "f83b45d350186cfa47db970f9311408fbde0fb68bb884adc1a3644a36b18f52c",
-        "raw": {
-          "acc": 0.3229166666666667,
-          "nll": 1.11786418934943,
-          "brier": 0.6813036856150539,
-          "ece": 0.09059596392971296
-        },
-        "calibrated": {
-          "acc": 0.3229166666666667,
-          "nll": 1.1024371940848587,
-          "brier": 0.6695271712596828,
-          "ece": 0.060665733732341603
-        }
-      }
-    }
-  },
-  "recorded_runtime_source_hashes": {
-    "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-    "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
-    "kev/api.py": "7bffacfb762c626b8bc2f670f350295af5ccb0883dbe90239c7d2f8e5ef56582",
-    "kev/autoresearch.py": "f7d7fabb9c0df7065bee3fec4aa4bee7028c5d1565847aefe2d74e3d7d40ed8a",
-    "kev/benchmark.py": "8d486659a9036102386b077fcbd197fc0d85ae7752a4711666d6b3a95622b26f",
-    "kev/calibrate.py": "c1eff4744fabd349e8abca86777a7aa0cbea44904c8223d3b61fca3d43741519",
-    "kev/checkpoint.py": "9f1dbc5d3d7d3050aca4ee34e0578c9ad76b7a2a7467dfc7a4ad65e3c471db59",
-    "kev/compare.py": "3606dbf98bb305d66420158498cb837304d2edb8dab01cdaf2ec42964bef3435",
-    "kev/composition.py": "f335ed17e18e0a544893db5e22b9a059e6ce1b2e14dbb863ac7d7bbf8f3e0536",
-    "kev/contrastive.py": "cbb979aa5d40265ad0e64695f94b281d91751fa405811ddfa8212fede111edcf",
-    "kev/data.py": "a5a5e200f56b60e4bf30a195a4cc87a74afddae15999ec48c7543a4f9b59f71c",
-    "kev/device.py": "d1677fd98ec0979c7284546306e34e0d09298ca4042fc39eb2b32a74f4e975c4",
-    "kev/evaluate.py": "999264f2837dcfbf2ec93601aa4745e701698ab850a43ad89324672a6604965c",
-    "kev/experiment.py": "1a5ca47d2b3d236e53c87e8c4197704f162c6692aa008521d31ea491945693f0",
-    "kev/jev.py": "acd4cc3f1844e438cc83a8d409c15ef78a5ef64d39ea10f583d75a6c7646b243",
-    "kev/metrics.py": "db0930d470c0ce16f16dcaf726d49a7a103973a5296cefc5259d1e68707d882b",
-    "kev/mlx_model.py": "f582428796faf6bf962b772a69a6227ac3259ccbd3215c0c8b55ce28dbbd0909",
-    "kev/model.py": "eac0f56c807b85b5c4077483d922e6bb03f07890de2835efd9372cf20cd01b8e",
-    "kev/plot.py": "0874bcff2885d8155a1cceade0de8a275a7163d3c3fd6aa0c294e8ccf5ff7e02",
-    "kev/predictors.py": "b2a66dd9f0f0a8026bc94603383bacfdd622e75d1c935173997175606626a9b3",
-    "kev/publish.py": "8abc9bcf4a01365697b05cf3c5ad0013462bd92d005f5616954e40ab1100c7aa",
-    "kev/serve.py": "c210b1f4598e64b49c399f73603ed372f4314c165ceda50e1340c0466ec1c735",
-    "kev/study_v3.py": "7891150aa623e4479c7c789d4dc4662f186132b37b7bac1e1f072b9e08ee13e3",
-    "kev/suite.py": "c5e5e27f2fed871b547a021d180b0a7f4a0542a488cce4ee02ad936f433441db",
-    "kev/train.py": "a2d85632d73760fb3efe51627ff4fde6af3100ed1c34e75d290c1b00c9899aa9",
-    "kev/transfer_v9.py": "0902409742151250a28af1fd8f42258b70fcc161f3deed3ee35fe3f473f3c763",
-    "modal_app.py": "5d55aa2a454aba2ba6e4a392ed77ab2bc849e9823b8298a41ce1b8fe4217344b",
-    "pyproject.toml": "7c17fbe9efcadda3eb488b59adf1b6db5938cd33af4d3bd389ce4359b73fcc7d",
-    "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
-  },
-  "current_release_source_hashes": {
-    "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-    "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
-    "kev/api.py": "7bffacfb762c626b8bc2f670f350295af5ccb0883dbe90239c7d2f8e5ef56582",
-    "kev/autoresearch.py": "f7d7fabb9c0df7065bee3fec4aa4bee7028c5d1565847aefe2d74e3d7d40ed8a",
-    "kev/benchmark.py": "8d486659a9036102386b077fcbd197fc0d85ae7752a4711666d6b3a95622b26f",
-    "kev/calibrate.py": "c1eff4744fabd349e8abca86777a7aa0cbea44904c8223d3b61fca3d43741519",
-    "kev/checkpoint.py": "9f1dbc5d3d7d3050aca4ee34e0578c9ad76b7a2a7467dfc7a4ad65e3c471db59",
-    "kev/compare.py": "3606dbf98bb305d66420158498cb837304d2edb8dab01cdaf2ec42964bef3435",
-    "kev/composition.py": "f335ed17e18e0a544893db5e22b9a059e6ce1b2e14dbb863ac7d7bbf8f3e0536",
-    "kev/contrastive.py": "cbb979aa5d40265ad0e64695f94b281d91751fa405811ddfa8212fede111edcf",
-    "kev/data.py": "a5a5e200f56b60e4bf30a195a4cc87a74afddae15999ec48c7543a4f9b59f71c",
-    "kev/device.py": "d1677fd98ec0979c7284546306e34e0d09298ca4042fc39eb2b32a74f4e975c4",
-    "kev/evaluate.py": "999264f2837dcfbf2ec93601aa4745e701698ab850a43ad89324672a6604965c",
-    "kev/experiment.py": "1a5ca47d2b3d236e53c87e8c4197704f162c6692aa008521d31ea491945693f0",
-    "kev/jev.py": "acd4cc3f1844e438cc83a8d409c15ef78a5ef64d39ea10f583d75a6c7646b243",
-    "kev/metrics.py": "db0930d470c0ce16f16dcaf726d49a7a103973a5296cefc5259d1e68707d882b",
-    "kev/mlx_model.py": "f582428796faf6bf962b772a69a6227ac3259ccbd3215c0c8b55ce28dbbd0909",
-    "kev/model.py": "eac0f56c807b85b5c4077483d922e6bb03f07890de2835efd9372cf20cd01b8e",
-    "kev/plot.py": "0874bcff2885d8155a1cceade0de8a275a7163d3c3fd6aa0c294e8ccf5ff7e02",
-    "kev/predictors.py": "b2a66dd9f0f0a8026bc94603383bacfdd622e75d1c935173997175606626a9b3",
-    "kev/publish.py": "8abc9bcf4a01365697b05cf3c5ad0013462bd92d005f5616954e40ab1100c7aa",
-    "kev/serve.py": "c210b1f4598e64b49c399f73603ed372f4314c165ceda50e1340c0466ec1c735",
-    "kev/study_v3.py": "7891150aa623e4479c7c789d4dc4662f186132b37b7bac1e1f072b9e08ee13e3",
-    "kev/suite.py": "c5e5e27f2fed871b547a021d180b0a7f4a0542a488cce4ee02ad936f433441db",
-    "kev/train.py": "a2d85632d73760fb3efe51627ff4fde6af3100ed1c34e75d290c1b00c9899aa9",
-    "kev/transfer_v9.py": "0902409742151250a28af1fd8f42258b70fcc161f3deed3ee35fe3f473f3c763",
-    "modal_app.py": "5d55aa2a454aba2ba6e4a392ed77ab2bc849e9823b8298a41ce1b8fe4217344b",
-    "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
-    "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
-  },
-  "release_metadata_drift": {
-    "pyproject.toml": {
-      "trained_sha256": "7c17fbe9efcadda3eb488b59adf1b6db5938cd33af4d3bd389ce4359b73fcc7d",
-      "release_sha256": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
-      "changed_fields": [
-        "project.description",
-        "project.urls"
+    "q.weight": {
+      "shape": [
+        256,
+        2560
       ],
-      "all_other_toml_equal": true
+      "dtype": "torch.float32",
+      "sha256": "9d7e952f5c8e175aa031568bcf08a004ef352a6b93901b54cd5433f3aaa8aacf"
     }
   },
-  "recorded_git_commit": "d8b1413f428a9a8f4e3d6091acf3d9ed8445c864",
-  "recorded_commit_source_mismatches": [],
-  "checks": {
-    "suite_and_plan_match_registered_hashes": true,
-    "registration_names_fresh_study": true,
-    "configuration_matches_registered_recipe": true,
-    "result_binds_execution_provenance": true,
-    "raw_checkpoint_hashes_match_scored_artifact": true,
-    "raw_temperature_verified": true,
-    "trained_and_evaluated_suite_match": true,
-    "actual_trainer_arguments_match_plan": true,
-    "head_metadata_matches_training_arguments": true,
-    "base_and_architecture_match_plan": true,
-    "warm_start_matches_pinned_clean_parent": true,
-    "training_membership_is_registered_authored_subset": true,
-    "complete_training_without_rejection": true,
-    "no_locked_test_evaluation": true,
-    "recorded_mechanism_checks_pass": true,
-    "calibration_complete_evaluation": true,
-    "development_complete_evaluation": true,
-    "only_pyproject_differs_from_recorded_runtime_sources": true,
-    "original_project_file_matches_training": true,
-    "project_drift_is_description_and_urls_only": true,
-    "learned_head_tensor_names_match": true,
-    "head_tensor:q.weight": true,
-    "head_tensor:q.bias": true,
-    "head_tensor:k.weight": true,
-    "head_tensor:k.bias": true,
-    "adapter_has_expected_finite_tensors": true,
-    "adapter_is_byte_identical": true,
-    "copied_file:adapter_model.safetensors": true,
-    "copied_file:tokenizer_config.json": true,
-    "copied_file:tokenizer.json": true,
-    "copied_file:README.md": true,
-    "copied_file:training_metrics.json": true,
-    "copied_file:adapter_config.json": true,
-    "copied_file:chat_template.jinja": true,
-    "copied_file:training_config.json": true,
-    "calibration_receipt_matches_saved_metadata": true,
-    "all_calibration_input_hashes_still_match": true,
-    "per_source_receipt_matches_fit": true,
-    "temperature_independently_recomputed": true,
-    "calibration_population_exact": true,
-    "cv_is_latent_group_disjoint": true,
-    "calibration_excludes_development_and_test": true,
-    "all_inputs_stable_through_verification": true
-  },
-  "artifact_sha256": {
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/README.md": "e0e50894b914f1765ef42c1eade74b42c613fb0330b06339f9502cc6f2e25e79",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/adapter_config.json": "8a05dfd6c5e7f61a62e6db5d8de8093a8dbbd105aa7ee7b5e7b73f6249f45617",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/adapter_model.safetensors": "7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/chat_template.jinja": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/head.pt": "af6bfd215bf0b7eff92612a83b21cb56754ca87f0bf9f62fdfcfbef275d9eab6",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/tokenizer.json": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/tokenizer_config.json": "8671bed7c852ce9e661be94f179a7b4ffd091c2a65aea0363e5501c20318ee45",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/training_config.json": "f1936818cecded5fe659330a6e0a1ad631d9b5b1eb2014982634a58f7d6f7d68",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint/training_metrics.json": "20b9314de861f32641b4f6a16ae8b06ba273426333da40c2820e5f591936d7a7",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/README.md": "e0e50894b914f1765ef42c1eade74b42c613fb0330b06339f9502cc6f2e25e79",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/adapter_config.json": "8a05dfd6c5e7f61a62e6db5d8de8093a8dbbd105aa7ee7b5e7b73f6249f45617",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/adapter_model.safetensors": "7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/calibration-sources.json": "be16ec2fdfb3b28b63888ede85cce428a8106785a2a64b37215550833176d875",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/calibration.json": "ca2b2eacb602dd61b87139cd5eafe2daf4d3fdd659c8cb8924538567259808bc",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/chat_template.jinja": "a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/head.pt": "dc1f550c7914388af77e607c70fe8801558a89a3847d37ea37d3923b28521c3e",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/tokenizer.json": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/tokenizer_config.json": "8671bed7c852ce9e661be94f179a7b4ffd091c2a65aea0363e5501c20318ee45",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/training_config.json": "f1936818cecded5fe659330a6e0a1ad631d9b5b1eb2014982634a58f7d6f7d68",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/calibrated-checkpoint/training_metrics.json": "20b9314de861f32641b4f6a16ae8b06ba273426333da40c2820e5f591936d7a7",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/provenance.json": "f540a4587f44f4a3bdda19f68a42377a56c56d464443beaaf8fe57b593beb85f",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/result.json": "ec4bfe6436093b6ba096921bb95b2643e8418c84d1e5cbbc596d78f2c744338c",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/train.log": "e5b54da6b803d7e6d5c0fdfc902b8095a123243a652ec996ab411da0c818ce09",
-    "/Users/mac/Sev/runs/sev-research-release-20260925/training-registration.json": "d46082575bf06ecbba776df6476be2cef694ba4f1e1c7ecb8f1b4048c33cb160",
-    "/Users/mac/Sev-release-20260925/experiments/sev-research-4b-v1.json": "af05762d5194a556cc6a518b0fc0860e6942157f0790b48ae80fe85098c5d94f",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/manifest.json": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/train.jsonl": "0aa668b5cc927db65e6b7a0890bc47ff7061d0ec271ff239d7d120f362964a6e",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/calibration.jsonl": "eebb5ddeb322b16bbb312ddb6927b1d71226b72e4b01d323c698d1e2a3c4f38d",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/development.jsonl": "bd76cba90825cfb2699ff8a4419c99f7b395541ed1cfe51c4a69305451f659e5",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/rows.json": "00909b4d6b838c534d8d403d84a578fea1ddaac5e30a2b488c2d1c138c0d05ab",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/report.json": "6bee81d641d7003c7715c0b26bcec77fe244e59e847e52e7f14f7336d88f59f9",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/temperature.json": "497cbf805d98eeaa745023e5242e0a0dfb62ed8e588a6dbec1e46fe0ac08a3cc",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/development/report.json": "3c3966f1f1ffbc4e55a634c359d02590f2cd1f5e2ff071b98d4dca610ab8a2d6",
-    "/Users/mac/Sev/runs/sev-r2-behavioral-4b-parents-v1/00-kev-4b@485ace8703592fcf405488b262449990824cfed1/provenance.json": "38b41be0ab1c27020c14da51c62cb13a76a8b140d5d89dbfc48078abe18fe277",
-    "/Users/mac/Sev/pyproject.toml": "7c17fbe9efcadda3eb488b59adf1b6db5938cd33af4d3bd389ce4359b73fcc7d",
-    "/Users/mac/Sev-release-20260925/kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-    "/Users/mac/Sev-release-20260925/kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
-    "/Users/mac/Sev-release-20260925/kev/api.py": "7bffacfb762c626b8bc2f670f350295af5ccb0883dbe90239c7d2f8e5ef56582",
-    "/Users/mac/Sev-release-20260925/kev/autoresearch.py": "f7d7fabb9c0df7065bee3fec4aa4bee7028c5d1565847aefe2d74e3d7d40ed8a",
-    "/Users/mac/Sev-release-20260925/kev/benchmark.py": "8d486659a9036102386b077fcbd197fc0d85ae7752a4711666d6b3a95622b26f",
-    "/Users/mac/Sev-release-20260925/kev/calibrate.py": "c1eff4744fabd349e8abca86777a7aa0cbea44904c8223d3b61fca3d43741519",
-    "/Users/mac/Sev-release-20260925/kev/checkpoint.py": "9f1dbc5d3d7d3050aca4ee34e0578c9ad76b7a2a7467dfc7a4ad65e3c471db59",
-    "/Users/mac/Sev-release-20260925/kev/compare.py": "3606dbf98bb305d66420158498cb837304d2edb8dab01cdaf2ec42964bef3435",
-    "/Users/mac/Sev-release-20260925/kev/composition.py": "f335ed17e18e0a544893db5e22b9a059e6ce1b2e14dbb863ac7d7bbf8f3e0536",
-    "/Users/mac/Sev-release-20260925/kev/contrastive.py": "cbb979aa5d40265ad0e64695f94b281d91751fa405811ddfa8212fede111edcf",
-    "/Users/mac/Sev-release-20260925/kev/data.py": "a5a5e200f56b60e4bf30a195a4cc87a74afddae15999ec48c7543a4f9b59f71c",
-    "/Users/mac/Sev-release-20260925/kev/device.py": "d1677fd98ec0979c7284546306e34e0d09298ca4042fc39eb2b32a74f4e975c4",
-    "/Users/mac/Sev-release-20260925/kev/evaluate.py": "999264f2837dcfbf2ec93601aa4745e701698ab850a43ad89324672a6604965c",
-    "/Users/mac/Sev-release-20260925/kev/experiment.py": "1a5ca47d2b3d236e53c87e8c4197704f162c6692aa008521d31ea491945693f0",
-    "/Users/mac/Sev-release-20260925/kev/jev.py": "acd4cc3f1844e438cc83a8d409c15ef78a5ef64d39ea10f583d75a6c7646b243",
-    "/Users/mac/Sev-release-20260925/kev/metrics.py": "db0930d470c0ce16f16dcaf726d49a7a103973a5296cefc5259d1e68707d882b",
-    "/Users/mac/Sev-release-20260925/kev/mlx_model.py": "f582428796faf6bf962b772a69a6227ac3259ccbd3215c0c8b55ce28dbbd0909",
-    "/Users/mac/Sev-release-20260925/kev/model.py": "eac0f56c807b85b5c4077483d922e6bb03f07890de2835efd9372cf20cd01b8e",
-    "/Users/mac/Sev-release-20260925/kev/plot.py": "0874bcff2885d8155a1cceade0de8a275a7163d3c3fd6aa0c294e8ccf5ff7e02",
-    "/Users/mac/Sev-release-20260925/kev/predictors.py": "b2a66dd9f0f0a8026bc94603383bacfdd622e75d1c935173997175606626a9b3",
-    "/Users/mac/Sev-release-20260925/kev/publish.py": "8abc9bcf4a01365697b05cf3c5ad0013462bd92d005f5616954e40ab1100c7aa",
-    "/Users/mac/Sev-release-20260925/kev/serve.py": "c210b1f4598e64b49c399f73603ed372f4314c165ceda50e1340c0466ec1c735",
-    "/Users/mac/Sev-release-20260925/kev/study_v3.py": "7891150aa623e4479c7c789d4dc4662f186132b37b7bac1e1f072b9e08ee13e3",
-    "/Users/mac/Sev-release-20260925/kev/suite.py": "c5e5e27f2fed871b547a021d180b0a7f4a0542a488cce4ee02ad936f433441db",
-    "/Users/mac/Sev-release-20260925/kev/train.py": "a2d85632d73760fb3efe51627ff4fde6af3100ed1c34e75d290c1b00c9899aa9",
-    "/Users/mac/Sev-release-20260925/kev/transfer_v9.py": "0902409742151250a28af1fd8f42258b70fcc161f3deed3ee35fe3f473f3c763",
-    "/Users/mac/Sev-release-20260925/modal_app.py": "5d55aa2a454aba2ba6e4a392ed77ab2bc849e9823b8298a41ce1b8fe4217344b",
-    "/Users/mac/Sev-release-20260925/pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
-    "/Users/mac/Sev-release-20260925/uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466",
-    "/Users/mac/Sev-release-20260925/scripts/report_sev_behavioral_round.py": "5beee8c302040233e2950ebe2d3f8944e84d6202f08b95943d679690bcdad8d8",
-    "/Users/mac/Sev-release-20260925/scripts/calibrate_sev_research.py": "f9fa4df59fda060e856f4e2a691e8fc0aeeeec836f1ddbaa43b3db5a39a874cf"
-  },
-  "locked_test_read": false,
-  "limits": [
-    "Artifact consistency and calibration verification only; no fresh inference or retraining.",
-    "Forward-token total is matched across trainer/result receipts, not independently reconstructed.",
-    "Warm-start adapter identity is compared with the prior pinned-parent evaluation receipt; parent adapter was not downloaded.",
-    "Calibration data were previously used in research. Calibration is not a field-accuracy or publication-quality guarantee.",
-    "Recorded runtime source hashes are authoritative because the recorded git base does not include uncommitted release changes."
+  "head_tensor_count": 4,
+  "adapter_tensor_count": 496,
+  "all_tensors_finite": true,
+  "head_tensors_unchanged": true,
+  "adapter_and_tokenizer_bytes_unchanged": true,
+  "raw_checkpoint_unchanged": true,
+  "only_head_changes": [
+    "temperature",
+    "temperature_fit"
   ]
 }

class-metrics.json

@@ -1,956 +0,0 @@
-{
-  "schema": "sev-research-independent-evaluation-v1",
-  "created_at": "2026-09-25T16:42:32.814870+00:00",
-  "pass": true,
-  "suite_manifest_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-  "input_sha256": {
-    "/Users/mac/Sev/runs/sev-research-release-20260925/paired-report.json": "14c148b41d9c5070f2800b938daf6e647ea3019cafcd2046f739226177469c64",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/manifest.json": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/calibration.jsonl": "eebb5ddeb322b16bbb312ddb6927b1d71226b72e4b01d323c698d1e2a3c4f38d",
-    "/Users/mac/Sev-release-20260925/evals/sev/behavioral-research-v1/development.jsonl": "bd76cba90825cfb2699ff8a4419c99f7b395541ed1cfe51c4a69305451f659e5",
-    "/Users/mac/Sev/runs/sev-r2-behavioral-4b-parents-v1/00-kev-4b@485ace8703592fcf405488b262449990824cfed1/calibration/rows.json": "e307a8170f6c1c159e252c8e6c55c88f51e37a784e0a4579bd5625c6742952cf",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/calibration/rows.json": "00909b4d6b838c534d8d403d84a578fea1ddaac5e30a2b488c2d1c138c0d05ab",
-    "/Users/mac/Sev/runs/sev-r2-behavioral-4b-parents-v1/00-kev-4b@485ace8703592fcf405488b262449990824cfed1/development/rows.json": "9ca8eeeb877ea120e5f4c5d54fa72f9bf31f200bbf460e14e6ec85c223162d56",
-    "/Users/mac/Sev/runs/sev-r2-research-4b-v1/00-trial-0/development/rows.json": "994e39a231bc64037f392e606c5fbd3c455f79e68f7b34ba44517238b269b26a"
-  },
-  "test_read": false,
-  "method": {
-    "point_metrics": "Independent NumPy log-softmax NLL, multiclass Brier and ten-bin weighted ECE. Raw confidence/Brier use saved float32 probabilities to match the documented canonical metric contract; tempered probabilities come from logits. Every episode has total weight one divided over its original full-view windows. No probability ensemble.",
-    "paired_intervals": "Independent paired 1000-draw seed-0 percentile bootstrap by group_id alone. Group draw multiplicities applied to row weights. All accuracy/NLL/Brier/ECE point values and interval endpoints match paired-report.json within 1e-12.",
-    "temperature": "Independent 121-point log-grid minimization of full calibration NLL with equal episode weights. Development labels do not fit temperatures. Intervals condition on the calibration fits.",
-    "joins": "All calibration/development predictions match frozen IDs, class order, targets, groups and episodes. Inference temperature is 1.0; saved probabilities match logits within 1e-5. No input hash changed.",
-    "maximum_point_discrepancy": 1.5543122344752192e-15
-  },
-  "calibration": {
-    "full_equal_episode_temperatures": {
-      "parent": 4.0,
-      "candidate": 1.9999999999999998
-    },
-    "candidate_experiment_micro_temperature": 1.0717734625362931,
-    "parent_grid_upper_bound": 4.0,
-    "windows": 182,
-    "episodes": 96,
-    "groups": 32,
-    "calibration_metrics_are_in_sample": true
-  },
-  "panels": {
-    "calibration/full/all/all": {
-      "windows": 182,
-      "episodes": 96,
-      "groups": 32,
-      "episode_weight_sum": 96.0,
-      "raw": {
-        "parent": {
-          "acc": 0.3333333333333333,
-          "nll": 1.7039948885030407,
-          "brier": 0.9366734114222882,
-          "ece": 0.3836640270561081,
-          "nll_minus_uniform": 0.6053825998349309
-        },
-        "candidate": {
-          "acc": 0.3342013888888888,
-          "nll": 1.0988623728398248,
-          "brier": 0.6670543370614525,
-          "ece": 0.06554797243603248,
-          "nll_minus_uniform": 0.00025008417171501485
-        }
-      },
-      "calibrated": {
-        "parent": {
-          "acc": 0.3333333333333333,
-          "nll": 1.1516241769020006,
-          "brier": 0.7011284844520594,
-          "ece": 0.11261178491667191,
-          "nll_minus_uniform": 0.0530118882338908
-        },
-        "candidate": {
-          "acc": 0.3342013888888888,
-          "nll": 1.0942258102235733,
-          "brier": 0.6635708549658471,
-          "ece": 0.04216089485176933,
-          "nll_minus_uniform": -0.004386478444536435
-        }
-      }
-    },
-    "development/full/iid/all": {
-      "windows": 196,
-      "episodes": 96,
-      "groups": 32,
-      "episode_weight_sum": 96.0,
-      "raw": {
-        "parent": {
-          "acc": 0.3333333333333332,
-          "nll": 1.712452357373193,
-          "brier": 0.9303227524837444,
-          "ece": 0.40316209525167973,
-          "nll_minus_uniform": 0.6138400687050831
-        },
-        "candidate": {
-          "acc": 0.3402777777777777,
-          "nll": 1.114185953853556,
-          "brier": 0.6767587666407534,
-          "ece": 0.0658244924811593,
-          "nll_minus_uniform": 0.015573665185446117
-        },
-        "paired": {
-          "acc": {
-            "candidate_minus_parent": 0.006944444444444475,
-            "ci95": [
-              -5.551115123125783e-17,
-              0.01736111111111116
-            ],
-            "parent_ci95": [
-              0.3333333333333332,
-              0.3333333333333334
-            ],
-            "candidate_ci95": [
-              0.33333333333333326,
-              0.3506944444444444
-            ]
-          },
-          "nll": {
-            "candidate_minus_parent": -0.598266403519637,
-            "ci95": [
-              -0.6785650197211481,
-              -0.5198138791551934
-            ],
-            "parent_ci95": [
-              1.6411516919873783,
-              1.7821554572493037
-            ],
-            "candidate_ci95": [
-              1.0932970868965703,
-              1.1377853746292819
-            ]
-          },
-          "brier": {
-            "candidate_minus_parent": -0.253563985842991,
-            "ci95": [
-              -0.2882144767124644,
-              -0.2179524081333052
-            ],
-            "parent_ci95": [
-              0.9005605959813184,
-              0.9600325280394626
-            ],
-            "candidate_ci95": [
-              0.6623924005681694,
-              0.6929750831612851
-            ]
-          },
-          "ece": {
-            "candidate_minus_parent": -0.33733760277052044,
-            "ci95": [
-              -0.36700635899334366,
-              -0.30629448376287555
-            ],
-            "parent_ci95": [
-              0.3845718384896557,
-              0.4223828872836136
-            ],
-            "candidate_ci95": [
-              0.04670505012772756,
-              0.08779944049177658
-            ]
-          }
-        }
-      },
-      "calibrated": {
-        "parent": {
-          "acc": 0.3333333333333332,
-          "nll": 1.1395642135708883,
-          "brier": 0.6924498378688599,
-          "ece": 0.1225919918902803,
-          "nll_minus_uniform": 0.04095192490277855
-        },
-        "candidate": {
-          "acc": 0.3402777777777777,
-          "nll": 1.099795297211755,
-          "brier": 0.6673327111796757,
-          "ece": 0.029085900496782724,
-          "nll_minus_uniform": 0.0011830085436452809
-        },
-        "paired": {
-          "acc": {
-            "candidate_minus_parent": 0.006944444444444475,
-            "ci95": [
-              -5.551115123125783e-17,
-              0.01736111111111116
-            ],
-            "parent_ci95": [
-              0.3333333333333332,
-              0.3333333333333334
-            ],
-            "candidate_ci95": [
-              0.33333333333333326,
-              0.3506944444444444
-            ]
-          },
-          "nll": {
-            "candidate_minus_parent": -0.039768916359133266,
-            "ci95": [
-              -0.0541526606128827,
-              -0.02516884604006643
-            ],
-            "parent_ci95": [
-              1.1281309744714123,
-              1.150621803326349
-            ],
-            "candidate_ci95": [
-              1.0898406047820133,
-              1.1100987332218029
-            ]
-          },
-          "brier": {
-            "candidate_minus_parent": -0.0251171266891842,
-            "ci95": [
-              -0.03573663793465235,
-              -0.01491019414543692
-            ],
-            "parent_ci95": [
-              0.6844439570327296,
-              0.7007751716641649
-            ],
-            "candidate_ci95": [
-              0.6604923908009388,
-              0.6744589579107121
-            ]
-          },
-          "ece": {
-            "candidate_minus_parent": -0.09350609139349758,
-            "ci95": [
-              -0.11225927559402635,
-              -0.06097575023830562
-            ],
-            "parent_ci95": [
-              0.11518648560004421,
-              0.13576747842445488
-            ],
-            "candidate_ci95": [
-              0.016975309751749493,
-              0.060832712514700446
-            ]
-          }
-        }
-      }
-    },
-    "development/full/heldout_task/all": {
-      "windows": 279,
-      "episodes": 96,
-      "groups": 32,
-      "episode_weight_sum": 96.0,
-      "raw": {
-        "parent": {
-          "acc": 0.33593749999999994,
-          "nll": 1.7575686233451442,
-          "brier": 0.9328596715471842,
-          "ece": 0.38062690344193295,
-          "nll_minus_uniform": 0.6589563346770344
-        },
-        "candidate": {
-          "acc": 0.33854166666666646,
-          "nll": 1.1020826039546936,
-          "brier": 0.6685505178342491,
-          "ece": 0.06767651412584706,
-          "nll_minus_uniform": 0.003470315286583814
-        },
-        "paired": {
-          "acc": {
-            "candidate_minus_parent": 0.0026041666666665186,
-            "ci95": [
-              -0.012152777777777792,
-              0.01822916666666663
-            ],
-            "parent_ci95": [
-              0.3333333333333332,
-              0.3411458333333333
-            ],
-            "candidate_ci95": [
-              0.3246310763888887,
-              0.35329861111111116
-            ]
-          },
-          "nll": {
-            "candidate_minus_parent": -0.6554860193904506,
-            "ci95": [
-              -0.7191252619922539,
-              -0.5925014370566782
-            ],
-            "parent_ci95": [
-              1.6936048834769093,
-              1.8243095747652278
-            ],
-            "candidate_ci95": [
-              1.0909955635083946,
-              1.1148722338770993
-            ]
-          },
-          "brier": {
-            "candidate_minus_parent": -0.26430915371293506,
-            "ci95": [
-              -0.28718910807802966,
-              -0.23881191694116355
-            ],
-            "parent_ci95": [
-              0.908673057864882,
-              0.9557397760683716
-            ],
-            "candidate_ci95": [
-              0.6602464516032021,
-              0.6782263566624999
-            ]
-          },
-          "ece": {
-            "candidate_minus_parent": -0.3129503893160859,
-            "ci95": [
-              -0.34660047272419636,
-              -0.2808933130120531
-            ],
-            "parent_ci95": [
-              0.3655869447937234,
-              0.39906087104179055
-            ],
-            "candidate_ci95": [
-              0.04217660861029082,
-              0.09521924116463996
-            ]
-          }
-        }
-      },
-      "calibrated": {
-        "parent": {
-          "acc": 0.33593749999999994,
-          "nll": 1.1503900787443782,
-          "brier": 0.7000942462701811,
-          "ece": 0.11705101643165554,
-          "nll_minus_uniform": 0.05177779007626837
-        },
-        "candidate": {
-          "acc": 0.33854166666666646,
-          "nll": 1.0975078649634014,
-          "brier": 0.6657823404671227,
-          "ece": 0.03921704440404433,
-          "nll_minus_uniform": -0.0011044237047084238
-        },
-        "paired": {
-          "acc": {
-            "candidate_minus_parent": 0.0026041666666665186,
-            "ci95": [
-              -0.012152777777777792,
-              0.01822916666666663
-            ],
-            "parent_ci95": [
-              0.3333333333333332,
-              0.3411458333333333
-            ],
-            "candidate_ci95": [
-              0.3246310763888887,
-              0.35329861111111116
-            ]
-          },
-          "nll": {
-            "candidate_minus_parent": -0.052882213780976794,
-            "ci95": [
-              -0.062443593811576194,
-              -0.04190493793031237
-            ],
-            "parent_ci95": [
-              1.1410787694045739,
-              1.1593915727679056
-            ],
-            "candidate_ci95": [
-              1.091928362600967,
-              1.1035442157202628
-            ]
-          },
-          "brier": {
-            "candidate_minus_parent": -0.03431190580305843,
-            "ci95": [
-              -0.04143880164165239,
-              -0.026395950941262136
-            ],
-            "parent_ci95": [
-              0.6935373129262573,
-              0.706784911160115
-            ],
-            "candidate_ci95": [
-              0.6617785284376593,
-              0.6702643963140507
-            ]
-          },
-          "ece": {
-            "candidate_minus_parent": -0.0778339720276112,
-            "ci95": [
-              -0.11444951609054545,
-              -0.04209215226387989
-            ],
-            "parent_ci95": [
-              0.1088484189991902,
-              0.13090558733426141
-            ],
-            "candidate_ci95": [
-              0.010719860472098339,
-              0.07335949517961426
-            ]
-          }
-        }
-      }
-    },
-    "development/full/all/all": {
-      "windows": 475,
-      "episodes": 192,
-      "groups": 64,
-      "episode_weight_sum": 192.0,
-      "raw": {
-        "parent": {
-          "acc": 0.3346354166666669,
-          "nll": 1.73501049035917,
-          "brier": 0.9315912120154644,
-          "ece": 0.39189449934680637,
-          "nll_minus_uniform": 0.6363982016910603
-        },
-        "candidate": {
-          "acc": 0.3394097222222226,
-          "nll": 1.1081342789041249,
-          "brier": 0.6726546422375007,
-          "ece": 0.056655958235590886,
-          "nll_minus_uniform": 0.009521990236015077
-        },
-        "paired": {
-          "acc": {
-            "candidate_minus_parent": 0.004774305555555691,
-            "ci95": [
-              -0.003917100694444441,
-              0.013888888888888895
-            ],
-            "parent_ci95": [
-              0.3333333333333333,
-              0.3372395833333335
-            ],
-            "candidate_ci95": [
-              0.3311631944444446,
-              0.34810112847222235
-            ]
-          },
-          "nll": {
-            "candidate_minus_parent": -0.6268762114550452,
-            "ci95": [
-              -0.6790899859377325,
-              -0.574909379457072
-            ],
-            "parent_ci95": [
-              1.6811977782718917,
-              1.78661079285623
-            ],
-            "candidate_ci95": [
-              1.0961882249343542,
-              1.121428990046415
-            ]
-          },
-          "brier": {
-            "candidate_minus_parent": -0.25893656977796364,
-            "ci95": [
-              -0.28024348498046536,
-              -0.23624394979708538
-            ],
-            "parent_ci95": [
-              0.9092211056538773,
-              0.9509438619550089
-            ],
-            "candidate_ci95": [
-              0.664610827238441,
-              0.6815656878154811
-            ]
-          },
-          "ece": {
-            "candidate_minus_parent": -0.3352385411112155,
-            "ci95": [
-              -0.3512828651139513,
-              -0.31080435666201395
-            ],
-            "parent_ci95": [
-              0.37815344075238194,
-              0.40561665467498564
-            ],
-            "candidate_ci95": [
-              0.04569651446212631,
-              0.07531915688931974
-            ]
-          }
-        }
-      },
-      "calibrated": {
-        "parent": {
-          "acc": 0.3346354166666669,
-          "nll": 1.1449771461576332,
-          "brier": 0.6962720420695208,
-          "ece": 0.11982150416096789,
-          "nll_minus_uniform": 0.04636485748952346
-        },
-        "candidate": {
-          "acc": 0.3394097222222226,
-          "nll": 1.0986515810875772,
-          "brier": 0.6665575258233997,
-          "ece": 0.024607196074886876,
-          "nll_minus_uniform": 3.929241946742934e-05
-        },
-        "paired": {
-          "acc": {
-            "candidate_minus_parent": 0.004774305555555691,
-            "ci95": [
-              -0.003917100694444441,
-              0.013888888888888895
-            ],
-            "parent_ci95": [
-              0.3333333333333333,
-              0.3372395833333335
-            ],
-            "candidate_ci95": [
-              0.3311631944444446,
-              0.34810112847222235
-            ]
-          },
-          "nll": {
-            "candidate_minus_parent": -0.04632556507005603,
-            "ci95": [
-              -0.05491666451541059,
-              -0.037191163770480214
-            ],
-            "parent_ci95": [
-              1.1365041969928558,
-              1.1526221927426759
-            ],
-            "candidate_ci95": [
-              1.0931171370730461,
-              1.1043579084594946
-            ]
-          },
-          "brier": {
-            "candidate_minus_parent": -0.029714516246121092,
-            "ci95": [
-              -0.036132595842729695,
-              -0.02284046811142687
-            ],
-            "parent_ci95": [
-              0.6901428934681393,
-              0.701587223080451
-            ],
-            "candidate_ci95": [
-              0.6627210641166761,
-              0.6705910357128937
-            ]
-          },
-          "ece": {
-            "candidate_minus_parent": -0.09521430808608101,
-            "ci95": [
-              -0.10827761667473262,
-              -0.0701737586224902
-            ],
-            "parent_ci95": [
-              0.11338467129999455,
-              0.12711460228131327
-            ],
-            "candidate_ci95": [
-              0.015289219037466687,
-              0.04742637735158731
-            ]
-          }
-        }
-      }
-    }
-  },
-  "origin_confusion": {
-    "calibration/full/all/all": {
-      "parent": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            0.0,
-            31.999999999999993,
-            0.0
-          ],
-          [
-            0.0,
-            31.999999999999996,
-            0.0
-          ],
-          [
-            0.0,
-            32.0,
-            0.0
-          ]
-        ],
-        "recall": {
-          "human": 0.0,
-          "script": 1.0,
-          "agent": 0.0
-        },
-        "prediction_share": {
-          "human": 0.0,
-          "script": 0.9999999999999999,
-          "agent": 0.0
-        },
-        "origin_episode_support": {
-          "human": 31.999999999999993,
-          "script": 31.999999999999996,
-          "agent": 32.0
-        },
-        "window_prediction_counts": {
-          "human": 0,
-          "script": 182,
-          "agent": 0
-        }
-      },
-      "candidate": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            31.749999999999993,
-            0.25,
-            0.0
-          ],
-          [
-            31.66666666666666,
-            0.3333333333333333,
-            0.0
-          ],
-          [
-            31.75,
-            0.25,
-            0.0
-          ]
-        ],
-        "recall": {
-          "human": 0.9921875,
-          "script": 0.010416666666666668,
-          "agent": 0.0
-        },
-        "prediction_share": {
-          "human": 0.9913194444444443,
-          "script": 0.008680555555555554,
-          "agent": 0.0
-        },
-        "origin_episode_support": {
-          "human": 31.999999999999993,
-          "script": 31.999999999999993,
-          "agent": 32.0
-        },
-        "window_prediction_counts": {
-          "human": 179,
-          "script": 3,
-          "agent": 0
-        }
-      }
-    },
-    "development/full/iid/all": {
-      "parent": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            0.0,
-            31.99999999999999,
-            0.0
-          ],
-          [
-            0.0,
-            31.99999999999999,
-            0.0
-          ],
-          [
-            0.0,
-            31.99999999999999,
-            0.0
-          ]
-        ],
-        "recall": {
-          "human": 0.0,
-          "script": 1.0,
-          "agent": 0.0
-        },
-        "prediction_share": {
-          "human": 0.0,
-          "script": 1.0,
-          "agent": 0.0
-        },
-        "origin_episode_support": {
-          "human": 31.99999999999999,
-          "script": 31.99999999999999,
-          "agent": 31.99999999999999
-        },
-        "window_prediction_counts": {
-          "human": 0,
-          "script": 196,
-          "agent": 0
-        }
-      },
-      "candidate": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            31.666666666666657,
-            0.3333333333333333,
-            0.0
-          ],
-          [
-            31.333333333333325,
-            0.6666666666666666,
-            0.0
-          ],
-          [
-            30.166666666666657,
-            1.4999999999999998,
-            0.3333333333333333
-          ]
-        ],
-        "recall": {
-          "human": 0.9895833333333334,
-          "script": 0.020833333333333336,
-          "agent": 0.01041666666666667
-        },
-        "prediction_share": {
-          "human": 0.9704861111111113,
-          "script": 0.026041666666666678,
-          "agent": 0.0034722222222222238
-        },
-        "origin_episode_support": {
-          "human": 31.99999999999999,
-          "script": 31.999999999999993,
-          "agent": 31.99999999999999
-        },
-        "window_prediction_counts": {
-          "human": 188,
-          "script": 7,
-          "agent": 1
-        }
-      }
-    },
-    "development/full/heldout_task/all": {
-      "parent": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            0.0,
-            31.99999999999998,
-            0.0
-          ],
-          [
-            0.0,
-            31.99999999999999,
-            0.0
-          ],
-          [
-            0.0,
-            31.74999999999998,
-            0.25
-          ]
-        ],
-        "recall": {
-          "human": 0.0,
-          "script": 1.0,
-          "agent": 0.007812500000000005
-        },
-        "prediction_share": {
-          "human": 0.0,
-          "script": 0.9973958333333334,
-          "agent": 0.0026041666666666683
-        },
-        "origin_episode_support": {
-          "human": 31.99999999999998,
-          "script": 31.99999999999999,
-          "agent": 31.99999999999998
-        },
-        "window_prediction_counts": {
-          "human": 0,
-          "script": 278,
-          "agent": 1
-        }
-      },
-      "candidate": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            31.416666666666647,
-            0.5833333333333333,
-            0.0
-          ],
-          [
-            30.916666666666654,
-            1.0833333333333333,
-            0.0
-          ],
-          [
-            31.216666666666647,
-            0.7833333333333333,
-            0.0
-          ]
-        ],
-        "recall": {
-          "human": 0.9817708333333334,
-          "script": 0.03385416666666668,
-          "agent": 0.0
-        },
-        "prediction_share": {
-          "human": 0.9744791666666668,
-          "script": 0.025520833333333347,
-          "agent": 0.0
-        },
-        "origin_episode_support": {
-          "human": 31.99999999999998,
-          "script": 31.999999999999986,
-          "agent": 31.999999999999982
-        },
-        "window_prediction_counts": {
-          "human": 270,
-          "script": 9,
-          "agent": 0
-        }
-      }
-    },
-    "development/full/all/all": {
-      "parent": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            0.0,
-            64.00000000000006,
-            0.0
-          ],
-          [
-            0.0,
-            64.00000000000004,
-            0.0
-          ],
-          [
-            0.0,
-            63.75000000000006,
-            0.25
-          ]
-        ],
-        "recall": {
-          "human": 0.0,
-          "script": 1.0,
-          "agent": 0.0039062499999999965
-        },
-        "prediction_share": {
-          "human": 0.0,
-          "script": 0.9986979166666666,
-          "agent": 0.0013020833333333322
-        },
-        "origin_episode_support": {
-          "human": 64.00000000000006,
-          "script": 64.00000000000004,
-          "agent": 64.00000000000006
-        },
-        "window_prediction_counts": {
-          "human": 0,
-          "script": 474,
-          "agent": 1
-        }
-      },
-      "candidate": {
-        "labels": [
-          "human",
-          "script",
-          "agent"
-        ],
-        "rows": "true origin",
-        "columns": "predicted origin",
-        "confusion_episode_mass": [
-          [
-            63.0833333333334,
-            0.9166666666666665,
-            0.0
-          ],
-          [
-            62.25000000000004,
-            1.7499999999999998,
-            0.0
-          ],
-          [
-            61.38333333333338,
-            2.283333333333333,
-            0.3333333333333333
-          ]
-        ],
-        "recall": {
-          "human": 0.9856770833333333,
-          "script": 0.02734374999999998,
-          "agent": 0.00520833333333333
-        },
-        "prediction_share": {
-          "human": 0.9724826388888888,
-          "script": 0.025781249999999974,
-          "agent": 0.0017361111111111095
-        },
-        "origin_episode_support": {
-          "human": 64.00000000000007,
-          "script": 64.00000000000004,
-          "agent": 64.00000000000004
-        },
-        "window_prediction_counts": {
-          "human": 458,
-          "script": 16,
-          "agent": 1
-        }
-      }
-    }
-  },
-  "training": {
-    "records": 2334,
-    "training_seconds": 371.20887875556946,
-    "trial_wall_seconds": 610.9060554900001,
-    "optimizer_steps": 292,
-    "transfer_evaluated": false,
-    "mechanism_and_coverage_gates_passed": true,
-    "promotable": false
-  },
-  "interpretation": [
-    "The candidate predicts human on about 97% of equal-episode-weighted full development windows. Per-class recall is a necessary release limitation.",
-    "IID candidate agent recall is 1.0417%; shifted agent recall is zero. Temperature does not alter these argmax decisions.",
-    "Uniform three-class NLL is ln(3)=1.0986122886681098. Raw candidate NLL is worse than uniform on IID and shifted development; calibrated NLL is very close to uniform. Low ECE does not establish useful discrimination.",
-    "Each development regime contains 32 latent groups with three matched origin episodes per group. Intervals describe this authored-policy population, not real human or LLM traffic. The shift changes tasks and policy priors within one controller implementation.",
-    "No fresh transfer evaluation was run. Older mixed-replay retention results cannot be assigned to this checkpoint."
-  ]
-}

head.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dc1f550c7914388af77e607c70fe8801558a89a3847d37ea37d3923b28521c3e
-size 5256511
+oid sha256:1d50c37b4db60c22ae8e6bb24f5055056701bb16e332964350d76df57dc82af0
+size 5251327

provenance.json

@@ -2,7 +2,7 @@
   "config": {
     "epochs": 1,
     "seed": 4,
-    "lr": 1e-05,
+    "lr": 2.5e-06,
     "lora": 16,
     "accum": 2,
     "batch": 4,
@@ -23,17 +23,17 @@
     "focal_gamma": 0.0,
     "base": "Qwen/Qwen3.5-4B-Base",
     "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
-    "init_from": "jaredpalmer/kev-4b@485ace8703592fcf405488b262449990824cfed1",
+    "init_from": "/runs/sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint",
     "lora_targets": "all",
     "head_dim": 256,
     "weights_dtype": "fp32",
-    "option_isolation": 0,
-    "special_embeddings": 0,
     "dtype": "bf16",
-    "checkpointing": 1
+    "checkpointing": 1,
+    "option_isolation": 0,
+    "special_embeddings": 0
   },
-  "config_sha256": "1abcf6795472953561dc92bf54d59e9f10fd5e81d08d558f44197e56299077af",
-  "suite_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
+  "config_sha256": "958d0b20b3ffd1c572e0abf221383a450e0f8e8d3fff3cb745235a371e6a025d",
+  "suite_sha256": "61e3c1898b8a03eb178355d3c8c5050e85bd217285c700df9b4506ec9e3680f8",
   "source_hashes": {
     "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
     "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
@@ -61,21 +61,21 @@
     "kev/suite.py": "c5e5e27f2fed871b547a021d180b0a7f4a0542a488cce4ee02ad936f433441db",
     "kev/train.py": "a2d85632d73760fb3efe51627ff4fde6af3100ed1c34e75d290c1b00c9899aa9",
     "kev/transfer_v9.py": "0902409742151250a28af1fd8f42258b70fcc161f3deed3ee35fe3f473f3c763",
-    "modal_app.py": "5d55aa2a454aba2ba6e4a392ed77ab2bc849e9823b8298a41ce1b8fe4217344b",
-    "pyproject.toml": "7c17fbe9efcadda3eb488b59adf1b6db5938cd33af4d3bd389ce4359b73fcc7d",
+    "modal_app.py": "639577258f4d581daefa51dad639961280ce92de9da256316ab4124f005a53fb",
+    "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
     "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
   },
-  "git_commit": "d8b1413f428a9a8f4e3d6091acf3d9ed8445c864",
+  "git_commit": "76e180880160721082a065bf1882495e756a643a",
   "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
   "torch": "2.8.0+cu128",
   "device": "cuda",
   "gpu": "NVIDIA H100 80GB HBM3",
   "legacy_checkpoint": false,
   "measured_checkpoint": {
-    "requested": "/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
-    "resolved": "/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
-    "head_sha256": "af6bfd215bf0b7eff92612a83b21cb56754ca87f0bf9f62fdfcfbef275d9eab6",
-    "adapter_sha256": "7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c",
+    "requested": "/runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint",
+    "resolved": "/runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint",
+    "head_sha256": "b29abc817f2340aa510fcd4b2c813455ef38102508120ffc35f7917ce359c27e",
+    "adapter_sha256": "b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420",
     "inference_temperature": 1.0
   }
 }

release.json

@@ -0,0 +1,36 @@
+{
+  "version": "v0.2.0-swarmtraces-research",
+  "repo": "macmacmacmac/Sev-4B",
+  "selected_run": "sev-r2-swarmtraces-4b-retention-v1/00-trial-0",
+  "raw_checkpoint": {
+    "head.pt": "b29abc817f2340aa510fcd4b2c813455ef38102508120ffc35f7917ce359c27e",
+    "adapter_model.safetensors": "b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420"
+  },
+  "temperature": 1.6245047927124707,
+  "release_head_sha256": "1d50c37b4db60c22ae8e6bb24f5055056701bb16e332964350d76df57dc82af0",
+  "registered_gates": {
+    "passed": 34,
+    "total": 37,
+    "overall_passed": false,
+    "failed": [
+      "general_raw_nll",
+      "guide_raw_nll",
+      "guide_raw_brier"
+    ]
+  },
+  "selection": "User-authorized research publication of the lower-rate continuation, with improved program-reading accuracy and better older GUIDE true-positive retention than the earlier SwarmTraces run. Does not convert failed experimental adoption checks to passes.",
+  "published_baseline_before": "8c19e9357e1f1ebf6c3d6eb5ed2d60a61981fe22",
+  "historical_tag_preserved": [
+    {
+      "name": "v0.1.0-research",
+      "target": "baaa0d277df1077591f6c0a151be4419a0dbd1b4"
+    }
+  ],
+  "comparison_sha256": "a72cba7d6a3d851f6b2d06e746f97b1024a5c2be3d69b411e53ae034e15e6ae2",
+  "source_runtime_matches_origin_main": true,
+  "checkpoint_raw_weights_preserved": true,
+  "dataset_repositories": [
+    "macmacmacmac/Sev-behavioral-research-v1",
+    "macmacmacmac/openai-agent-swarmtraces"
+  ]
+}

result.json

@@ -1,331 +1,690 @@
 {
-  "objective": -1.1040644052465771,
+  "objective": -0.15033816061880909,
   "paired_flip": null,
   "unknowable": null,
   "clean": {
-    "n": 475,
-    "nll": 1.1040644052465771,
-    "acc": 0.3452631578947368,
-    "ece": 0.057981423538388685,
-    "brier": 0.6695330728421766,
-    "mean_conf": 0.3934063210268794,
-    "confident_error_rate": 0.0,
-    "coverage_at_0_9": 0.0,
-    "accuracy_at_0_9": null,
-    "coverage_at_5pct_error": 0.0,
-    "coverage_at_1pct_error": 0.0,
-    "aurc": 0.6226976418791688,
-    "error_rate_at_0_9": null,
-    "confidence_bias": 0.048143163132142563,
+    "n": 2457,
+    "nll": 0.4501144704897554,
+    "acc": 0.7814407814407814,
+    "ece": 0.0623854079535909,
+    "brier": 0.2631266703069734,
+    "mean_conf": 0.8429047921202532,
+    "confident_error_rate": 0.005698005698005698,
+    "coverage_at_0_9": 0.6353276353276354,
+    "accuracy_at_0_9": 0.9910313901345291,
+    "coverage_at_5pct_error": 0.6935286935286935,
+    "coverage_at_1pct_error": 0.6422466422466423,
+    "aurc": 0.0473000289404362,
+    "error_rate_at_0_9": 0.008968609865470852,
+    "confidence_bias": 0.061464010679471714,
     "top_bins": {
       "0.9": {
-        "n": 0,
-        "errors": 0,
-        "error_rate": null
+        "n": 1561,
+        "errors": 14,
+        "error_rate": 0.008968609865470852
       },
       "0.95": {
-        "n": 0,
-        "errors": 0,
-        "error_rate": null
+        "n": 1522,
+        "errors": 14,
+        "error_rate": 0.009198423127463863
       },
       "0.99": {
-        "n": 0,
-        "errors": 0,
-        "error_rate": null
+        "n": 1466,
+        "errors": 12,
+        "error_rate": 0.008185538881309686
       }
     },
     "selective": {
       "0.5": {
-        "coverage": 0.5010526315789474,
-        "accuracy": 0.35294117647058826,
-        "confidence_cutoff": 0.3749043643474579
+        "coverage": 0.5002035002035002,
+        "accuracy": 0.9910496338486574,
+        "confidence_cutoff": 0.9954625257638747
       },
       "0.8": {
-        "coverage": 0.8,
-        "accuracy": 0.3447368421052632,
-        "confidence_cutoff": 0.3571439278976939
+        "coverage": 0.8001628001628002,
+        "accuracy": 0.8891149542217701,
+        "confidence_cutoff": 0.5730366706848145
       }
     }
   },
   "tasks": {
-    "sev_behavioral_origin": {
-      "n": 475,
-      "nll": 1.1040644052465771,
-      "acc": 0.3452631578947368,
-      "ece": 0.057981423538388685,
-      "brier": 0.6695330728421766,
-      "mean_conf": 0.3934063210268794,
+    "excytin_file": {
+      "n": 80,
+      "nll": 0.0001342531538525746,
+      "acc": 1.0,
+      "ece": 0.0001340575450615722,
+      "brier": 7.773831743030022e-07,
+      "mean_conf": 0.9998659424549384,
       "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.0,
-      "accuracy_at_0_9": null,
-      "coverage_at_5pct_error": 0.0,
-      "coverage_at_1pct_error": 0.0,
-      "aurc": 0.6226976418791688,
-      "error_rate_at_0_9": null,
-      "confidence_bias": 0.048143163132142563,
+      "coverage_at_0_9": 1.0,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.0001340575450615722,
       "top_bins": {
         "0.9": {
-          "n": 0,
+          "n": 80,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.95": {
-          "n": 0,
+          "n": 80,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.99": {
-          "n": 0,
+          "n": 80,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         }
       },
       "selective": {
         "0.5": {
-          "coverage": 0.5010526315789474,
-          "accuracy": 0.35294117647058826,
-          "confidence_cutoff": 0.3749043643474579
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9999710759158646
         },
         "0.8": {
           "coverage": 0.8,
-          "accuracy": 0.3447368421052632,
-          "confidence_cutoff": 0.3571439278976939
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9999066849313115
         }
       }
-    }
-  },
-  "variants": {
-    "clean": {
-      "n": 475,
-      "nll": 1.1040644052465771,
-      "acc": 0.3452631578947368,
-      "ece": 0.057981423538388685,
-      "brier": 0.6695330728421766,
-      "mean_conf": 0.3934063210268794,
+    },
+    "excytin_parent_join": {
+      "n": 78,
+      "nll": 0.0019727127979591126,
+      "acc": 1.0,
+      "ece": 0.0019615703387183547,
+      "brier": 4.023721192309001e-05,
+      "mean_conf": 0.9980384296612816,
       "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.0,
-      "accuracy_at_0_9": null,
-      "coverage_at_5pct_error": 0.0,
-      "coverage_at_1pct_error": 0.0,
-      "aurc": 0.6226976418791688,
-      "error_rate_at_0_9": null,
-      "confidence_bias": 0.048143163132142563,
+      "coverage_at_0_9": 1.0,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.0019615703387183547,
       "top_bins": {
         "0.9": {
-          "n": 0,
+          "n": 78,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.95": {
-          "n": 0,
+          "n": 78,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.99": {
-          "n": 0,
+          "n": 76,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         }
       },
       "selective": {
         "0.5": {
-          "coverage": 0.5010526315789474,
-          "accuracy": 0.35294117647058826,
-          "confidence_cutoff": 0.3749043643474579
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9996318703080018
         },
         "0.8": {
-          "coverage": 0.8,
-          "accuracy": 0.3447368421052632,
-          "confidence_cutoff": 0.3571439278976939
+          "coverage": 0.8076923076923077,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9974582021431176
         }
       }
     },
-    "prefix_loss": {
-      "n": 390,
-      "nll": 1.1076993167917513,
-      "acc": 0.3333333333333333,
-      "ece": 0.06600365992222378,
-      "brier": 0.673099724532684,
-      "mean_conf": 0.39766209170416683,
+    "excytin_process": {
+      "n": 120,
+      "nll": 0.0005180266626209783,
+      "acc": 1.0,
+      "ece": 0.0005174316239614507,
+      "brier": 2.3119235778779676e-06,
+      "mean_conf": 0.9994825683760385,
       "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.0,
-      "accuracy_at_0_9": null,
-      "coverage_at_5pct_error": 0.002564102564102564,
-      "coverage_at_1pct_error": 0.002564102564102564,
-      "aurc": 0.6469893243952387,
-      "error_rate_at_0_9": null,
-      "confidence_bias": 0.06432875837083352,
+      "coverage_at_0_9": 1.0,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.0005174316239614507,
       "top_bins": {
         "0.9": {
-          "n": 0,
+          "n": 120,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.95": {
-          "n": 0,
+          "n": 120,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.99": {
-          "n": 0,
+          "n": 120,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         }
       },
       "selective": {
         "0.5": {
           "coverage": 0.5,
-          "accuracy": 0.3384615384615385,
-          "confidence_cutoff": 0.37823529178605353
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9999134057097643
         },
         "0.8": {
           "coverage": 0.8,
-          "accuracy": 0.32371794871794873,
-          "confidence_cutoff": 0.36099090096381603
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.99901689132107
         }
       }
     },
-    "sampled_loss": {
-      "n": 400,
-      "nll": 1.1039164529781138,
-      "acc": 0.3375,
-      "ece": 0.058967762381204686,
-      "brier": 0.6698679501799927,
-      "mean_conf": 0.38919767867980576,
+    "excytin_signin": {
+      "n": 120,
+      "nll": 0.0013203461429938826,
+      "acc": 1.0,
+      "ece": 0.0012737143193113606,
+      "brier": 0.00013106658103070147,
+      "mean_conf": 0.9987262856806887,
       "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.0,
-      "accuracy_at_0_9": null,
-      "coverage_at_5pct_error": 0.0,
+      "coverage_at_0_9": 0.9916666666666667,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.0012737143193113365,
+      "top_bins": {
+        "0.9": {
+          "n": 119,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.95": {
+          "n": 119,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.99": {
+          "n": 119,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9998413534198065
+        },
+        "0.8": {
+          "coverage": 0.8,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9996110203225037
+        }
+      }
+    },
+    "sev_guide_detector": {
+      "n": 1064,
+      "nll": 0.8276776090882748,
+      "acc": 0.599624060150376,
+      "ece": 0.10621007339433168,
+      "brier": 0.48019952548397726,
+      "mean_conf": 0.7041447618951532,
+      "confident_error_rate": 0.012218045112781954,
+      "coverage_at_0_9": 0.32706766917293234,
+      "accuracy_at_0_9": 0.9626436781609196,
+      "coverage_at_5pct_error": 0.3524436090225564,
       "coverage_at_1pct_error": 0.0,
-      "aurc": 0.6417827102105147,
-      "error_rate_at_0_9": null,
-      "confidence_bias": 0.051697678679805736,
+      "aurc": 0.19304798592127517,
+      "error_rate_at_0_9": 0.03735632183908046,
+      "confidence_bias": 0.10452070174477723,
+      "top_bins": {
+        "0.9": {
+          "n": 348,
+          "errors": 13,
+          "error_rate": 0.03735632183908046
+        },
+        "0.95": {
+          "n": 327,
+          "errors": 13,
+          "error_rate": 0.039755351681957186
+        },
+        "0.99": {
+          "n": 319,
+          "errors": 12,
+          "error_rate": 0.03761755485893417
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5075187969924813,
+          "accuracy": 0.8074074074074075,
+          "confidence_cutoff": 0.6273336504002459
+        },
+        "0.8": {
+          "coverage": 0.8016917293233082,
+          "accuracy": 0.6518171160609613,
+          "confidence_cutoff": 0.48816800774395525
+        }
+      }
+    },
+    "sev_guide_triage": {
+      "n": 231,
+      "nll": 0.8563265185203435,
+      "acc": 0.5627705627705628,
+      "ece": 0.1895007716411903,
+      "brier": 0.5255142828500304,
+      "mean_conf": 0.7444269666893102,
+      "confident_error_rate": 0.0,
+      "coverage_at_0_9": 0.35064935064935066,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 0.36363636363636365,
+      "coverage_at_1pct_error": 0.36363636363636365,
+      "aurc": 0.22468166386345326,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": 0.18165640391874738,
+      "top_bins": {
+        "0.9": {
+          "n": 81,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.95": {
+          "n": 80,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.99": {
+          "n": 79,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5021645021645021,
+          "accuracy": 0.75,
+          "confidence_cutoff": 0.7894512944300596
+        },
+        "0.8": {
+          "coverage": 0.8095238095238095,
+          "accuracy": 0.5775401069518716,
+          "confidence_cutoff": 0.49009219177135965
+        }
+      }
+    },
+    "sev_swarmtraces_static": {
+      "n": 354,
+      "nll": 0.06671770193368179,
+      "acc": 0.9745762711864406,
+      "ece": 0.008417033573491902,
+      "brier": 0.03589778460324393,
+      "mean_conf": 0.9740214622331307,
+      "confident_error_rate": 0.002824858757062147,
+      "coverage_at_0_9": 0.9322033898305084,
+      "accuracy_at_0_9": 0.996969696969697,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 0.9745762711864406,
+      "aurc": 0.0009317569638462815,
+      "error_rate_at_0_9": 0.0030303030303030303,
+      "confidence_bias": -0.0005548089533099487,
+      "top_bins": {
+        "0.9": {
+          "n": 330,
+          "errors": 1,
+          "error_rate": 0.0030303030303030303
+        },
+        "0.95": {
+          "n": 317,
+          "errors": 1,
+          "error_rate": 0.0031545741324921135
+        },
+        "0.99": {
+          "n": 292,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9998280676315141
+        },
+        "0.8": {
+          "coverage": 0.8022598870056498,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9930714713128618
+        }
+      }
+    },
+    "sev_sysmon_native_file": {
+      "n": 62,
+      "nll": 0.010792211612586852,
+      "acc": 1.0,
+      "ece": 0.009954006393107204,
+      "brier": 0.002287040487090685,
+      "mean_conf": 0.9900459936068928,
+      "confident_error_rate": 0.0,
+      "coverage_at_0_9": 0.9838709677419355,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.009954006393107173,
+      "top_bins": {
+        "0.9": {
+          "n": 61,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.95": {
+          "n": 59,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.99": {
+          "n": 54,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9986117420701618
+        },
+        "0.8": {
+          "coverage": 0.8064516129032258,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9936233886125292
+        }
+      }
+    },
+    "sev_sysmon_native_join": {
+      "n": 120,
+      "nll": 0.0008810538663341702,
+      "acc": 1.0,
+      "ece": 0.0008800625080441771,
+      "brier": 3.5216739059086607e-06,
+      "mean_conf": 0.9991199374919558,
+      "confident_error_rate": 0.0,
+      "coverage_at_0_9": 1.0,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.0008800625080441771,
       "top_bins": {
         "0.9": {
-          "n": 0,
+          "n": 120,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.95": {
-          "n": 0,
+          "n": 120,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.99": {
-          "n": 0,
+          "n": 120,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         }
       },
       "selective": {
         "0.5": {
           "coverage": 0.5,
-          "accuracy": 0.34,
-          "confidence_cutoff": 0.3756981715976102
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9994848628990232
         },
         "0.8": {
           "coverage": 0.8,
-          "accuracy": 0.328125,
-          "confidence_cutoff": 0.3595526027546109
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.998641259167216
         }
       }
-    }
-  },
-  "heldout_tasks": {
-    "sev_behavioral_origin": {
-      "n": 279,
-      "nll": 1.1054832638596932,
-      "acc": 0.33691756272401435,
-      "ece": 0.07077264797432041,
-      "brier": 0.6704644833339944,
-      "mean_conf": 0.3860012027442918,
+    },
+    "sev_sysmon_native_other_effect": {
+      "n": 88,
+      "nll": 0.007295006715728625,
+      "acc": 1.0,
+      "ece": 0.006888792886758961,
+      "brier": 0.001385599246512425,
+      "mean_conf": 0.9931112071132411,
       "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.0,
-      "accuracy_at_0_9": null,
-      "coverage_at_5pct_error": 0.0,
-      "coverage_at_1pct_error": 0.0,
-      "aurc": 0.6256505380956604,
-      "error_rate_at_0_9": null,
-      "confidence_bias": 0.04908364002027743,
+      "coverage_at_0_9": 0.9886363636363636,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.006888792886758921,
       "top_bins": {
         "0.9": {
-          "n": 0,
+          "n": 87,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.95": {
-          "n": 0,
+          "n": 86,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
         },
         "0.99": {
-          "n": 0,
+          "n": 79,
           "errors": 0,
-          "error_rate": null
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9993634031772723
+        },
+        "0.8": {
+          "coverage": 0.8068181818181818,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9952397949435474
+        }
+      }
+    },
+    "sev_sysmon_native_process": {
+      "n": 74,
+      "nll": 0.005360095133334598,
+      "acc": 1.0,
+      "ece": 0.005183972707978367,
+      "brier": 0.0006452896028548641,
+      "mean_conf": 0.9948160272920217,
+      "confident_error_rate": 0.0,
+      "coverage_at_0_9": 0.9864864864864865,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 1.0,
+      "aurc": 0.0,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": -0.005183972707978324,
+      "top_bins": {
+        "0.9": {
+          "n": 73,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.95": {
+          "n": 72,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.99": {
+          "n": 67,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9998754065974452
+        },
+        "0.8": {
+          "coverage": 0.8108108108108109,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9964080071736937
+        }
+      }
+    },
+    "sev_sysmon_native_registry": {
+      "n": 66,
+      "nll": 0.02506239179799794,
+      "acc": 0.9848484848484849,
+      "ece": 0.005781042813278618,
+      "brier": 0.017230444731313608,
+      "mean_conf": 0.9868495288218273,
+      "confident_error_rate": 0.0,
+      "coverage_at_0_9": 0.9696969696969697,
+      "accuracy_at_0_9": 1.0,
+      "coverage_at_5pct_error": 1.0,
+      "coverage_at_1pct_error": 0.9696969696969697,
+      "aurc": 0.00046266864448682633,
+      "error_rate_at_0_9": 0.0,
+      "confidence_bias": 0.0020010439733424423,
+      "top_bins": {
+        "0.9": {
+          "n": 64,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.95": {
+          "n": 64,
+          "errors": 0,
+          "error_rate": 0.0
+        },
+        "0.99": {
+          "n": 61,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9996958591947809
+        },
+        "0.8": {
+          "coverage": 0.803030303030303,
+          "accuracy": 1.0,
+          "confidence_cutoff": 0.9967594448763499
+        }
+      }
+    }
+  },
+  "variants": {
+    "clean": {
+      "n": 2457,
+      "nll": 0.4501144704897554,
+      "acc": 0.7814407814407814,
+      "ece": 0.0623854079535909,
+      "brier": 0.2631266703069734,
+      "mean_conf": 0.8429047921202532,
+      "confident_error_rate": 0.005698005698005698,
+      "coverage_at_0_9": 0.6353276353276354,
+      "accuracy_at_0_9": 0.9910313901345291,
+      "coverage_at_5pct_error": 0.6935286935286935,
+      "coverage_at_1pct_error": 0.6422466422466423,
+      "aurc": 0.0473000289404362,
+      "error_rate_at_0_9": 0.008968609865470852,
+      "confidence_bias": 0.061464010679471714,
+      "top_bins": {
+        "0.9": {
+          "n": 1561,
+          "errors": 14,
+          "error_rate": 0.008968609865470852
+        },
+        "0.95": {
+          "n": 1522,
+          "errors": 14,
+          "error_rate": 0.009198423127463863
+        },
+        "0.99": {
+          "n": 1466,
+          "errors": 12,
+          "error_rate": 0.008185538881309686
         }
       },
       "selective": {
         "0.5": {
-          "coverage": 0.5017921146953405,
-          "accuracy": 0.35,
-          "confidence_cutoff": 0.37531410529273435
+          "coverage": 0.5002035002035002,
+          "accuracy": 0.9910496338486574,
+          "confidence_cutoff": 0.9954625257638747
         },
         "0.8": {
-          "coverage": 0.8028673835125448,
-          "accuracy": 0.33482142857142855,
-          "confidence_cutoff": 0.35978143335040114
+          "coverage": 0.8001628001628002,
+          "accuracy": 0.8891149542217701,
+          "confidence_cutoff": 0.5730366706848145
         }
       }
     }
   },
+  "heldout_tasks": {},
   "permutation": {
     "n": 0,
     "mean_max_delta": null,
     "flip_rate": null
   },
-  "temperature": 1.0717734625362931,
+  "temperature": 1.6245047927124707,
   "calibrated_clean": {
-    "n": 475,
-    "nll": 1.1026041370540602,
-    "acc": 0.3452631578947368,
-    "ece": 0.053275370738267415,
-    "brier": 0.6686530034298412,
-    "mean_conf": 0.38936850355603847,
-    "confident_error_rate": 0.0,
-    "coverage_at_0_9": 0.0,
-    "accuracy_at_0_9": null,
-    "coverage_at_5pct_error": 0.0,
-    "coverage_at_1pct_error": 0.0,
-    "aurc": 0.6227258857225079,
-    "error_rate_at_0_9": null,
-    "confidence_bias": 0.04410534566130164,
+    "n": 2457,
+    "nll": 0.43197697165072707,
+    "acc": 0.7814407814407814,
+    "ece": 0.044664429537492924,
+    "brier": 0.2507712462898617,
+    "mean_conf": 0.795027799473369,
+    "confident_error_rate": 0.004884004884004884,
+    "coverage_at_0_9": 0.6084656084656085,
+    "accuracy_at_0_9": 0.9919732441471572,
+    "coverage_at_5pct_error": 0.6935286935286935,
+    "coverage_at_1pct_error": 0.6422466422466423,
+    "aurc": 0.04736115860402151,
+    "error_rate_at_0_9": 0.008026755852842809,
+    "confidence_bias": 0.01358701803258755,
     "top_bins": {
       "0.9": {
-        "n": 0,
-        "errors": 0,
-        "error_rate": null
+        "n": 1495,
+        "errors": 12,
+        "error_rate": 0.008026755852842809
       },
       "0.95": {
-        "n": 0,
-        "errors": 0,
-        "error_rate": null
+        "n": 1247,
+        "errors": 11,
+        "error_rate": 0.008821170809943865
       },
       "0.99": {
-        "n": 0,
+        "n": 705,
         "errors": 0,
-        "error_rate": null
+        "error_rate": 0.0
       }
     },
     "selective": {
       "0.5": {
-        "coverage": 0.5010526315789474,
-        "accuracy": 0.35294117647058826,
-        "confidence_cutoff": 0.37205812028797897
+        "coverage": 0.5002035002035002,
+        "accuracy": 0.9910496338486574,
+        "confidence_cutoff": 0.9547451666401373
       },
       "0.8": {
-        "coverage": 0.8,
-        "accuracy": 0.3447368421052632,
-        "confidence_cutoff": 0.3555337941720841
+        "coverage": 0.8001628001628002,
+        "accuracy": 0.887589013224822,
+        "confidence_cutoff": 0.48481140426518843
       }
     }
   },
@@ -343,28 +702,1702 @@
     "returned_zeros": 0
   },
   "coverage": {
-    "requested_records": 1265,
-    "requested_questions": 1265,
-    "evaluated_records": 1265,
-    "evaluated_questions": 1265,
+    "requested_records": 1793,
+    "requested_questions": 2457,
+    "evaluated_records": 1793,
+    "evaluated_questions": 2457,
     "rejected_records": 0,
     "truncated_records": 0
   },
   "latency_ms": {
-    "median": 78.70266199995513,
-    "p95": 87.40347419995942
+    "median": 60.71334499984005,
+    "p95": 154.8275427999215
   },
   "calibration": {
     "inference_temperature": 1.0,
-    "additional_temperature": 1.0717734625362931,
+    "additional_temperature": 1.6245047927124707,
     "logits_recorded": true
   },
-  "transfer": null,
+  "transfer": {
+    "objective": -0.46662338616523397,
+    "paired_flip": {
+      "pairs": 16,
+      "flip_rate": 0.9375,
+      "both_correct_rate": 0.9375,
+      "invariant_pairs": 8,
+      "invariance_rate": 1.0,
+      "invariant_both_correct_rate": 0.875
+    },
+    "unknowable": null,
+    "clean": {
+      "n": 243,
+      "nll": 0.5657599517457697,
+      "acc": 0.8641975308641975,
+      "ece": 0.09883136412561748,
+      "brier": 0.2329497754098416,
+      "mean_conf": 0.9456019169777928,
+      "confident_error_rate": 0.07818930041152264,
+      "coverage_at_0_9": 0.8518518518518519,
+      "accuracy_at_0_9": 0.9082125603864735,
+      "coverage_at_5pct_error": 0.7119341563786008,
+      "coverage_at_1pct_error": 0.411522633744856,
+      "aurc": 0.033948169133907836,
+      "error_rate_at_0_9": 0.09178743961352658,
+      "confidence_bias": 0.08140438611359535,
+      "top_bins": {
+        "0.9": {
+          "n": 207,
+          "errors": 19,
+          "error_rate": 0.09178743961352658
+        },
+        "0.95": {
+          "n": 179,
+          "errors": 10,
+          "error_rate": 0.055865921787709494
+        },
+        "0.99": {
+          "n": 147,
+          "errors": 4,
+          "error_rate": 0.027210884353741496
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5020576131687243,
+          "accuracy": 0.9836065573770492,
+          "confidence_cutoff": 0.9975663879354146
+        },
+        "0.8": {
+          "coverage": 0.8024691358024691,
+          "accuracy": 0.9179487179487179,
+          "confidence_cutoff": 0.9296758313428735
+        }
+      }
+    },
+    "tasks": {
+      "composition_held_and_or": {
+        "n": 32,
+        "nll": 0.21420351636532084,
+        "acc": 0.90625,
+        "ece": 0.07786484107903366,
+        "brier": 0.14841282029529768,
+        "mean_conf": 0.9841148410790337,
+        "confident_error_rate": 0.03125,
+        "coverage_at_0_9": 0.9375,
+        "accuracy_at_0_9": 0.9666666666666667,
+        "coverage_at_5pct_error": 0.9375,
+        "coverage_at_1pct_error": 0.90625,
+        "aurc": 0.0069010416666666664,
+        "error_rate_at_0_9": 0.03333333333333333,
+        "confidence_bias": 0.07786484107903369,
+        "top_bins": {
+          "0.9": {
+            "n": 30,
+            "errors": 1,
+            "error_rate": 0.03333333333333333
+          },
+          "0.95": {
+            "n": 27,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 26,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9998649716969166
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9993232293186822
+          }
+        }
+      },
+      "contrastive_authorization": {
+        "n": 16,
+        "nll": 3.491797866673952e-05,
+        "acc": 1.0,
+        "ece": 3.491729606275307e-05,
+        "brier": 2.7334442890954754e-09,
+        "mean_conf": 0.9999650827039372,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -3.491729606275307e-05,
+        "top_bins": {
+          "0.9": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999685834101107
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999562785984693
+          }
+        }
+      },
+      "emotion": {
+        "n": 8,
+        "nll": 1.29779922266396,
+        "acc": 0.625,
+        "ece": 0.33113247991180284,
+        "brier": 0.6610413670516024,
+        "mean_conf": 0.956132479911803,
+        "confident_error_rate": 0.25,
+        "coverage_at_0_9": 0.875,
+        "accuracy_at_0_9": 0.7142857142857143,
+        "coverage_at_5pct_error": 0.5,
+        "coverage_at_1pct_error": 0.5,
+        "aurc": 0.12842261904761904,
+        "error_rate_at_0_9": 0.2857142857142857,
+        "confidence_bias": 0.33113247991180295,
+        "top_bins": {
+          "0.9": {
+            "n": 7,
+            "errors": 2,
+            "error_rate": 0.2857142857142857
+          },
+          "0.95": {
+            "n": 5,
+            "errors": 1,
+            "error_rate": 0.2
+          },
+          "0.99": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9816738572788068
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.7142857142857143,
+            "confidence_cutoff": 0.9054981634553654
+          }
+        }
+      },
+      "mmlu": {
+        "n": 8,
+        "nll": 1.270267510803572,
+        "acc": 0.625,
+        "ece": 0.27270807209084075,
+        "brier": 0.6428034740610011,
+        "mean_conf": 0.746352881495524,
+        "confident_error_rate": 0.125,
+        "coverage_at_0_9": 0.5,
+        "accuracy_at_0_9": 0.75,
+        "coverage_at_5pct_error": 0.375,
+        "coverage_at_1pct_error": 0.375,
+        "aurc": 0.22336309523809525,
+        "error_rate_at_0_9": 0.25,
+        "confidence_bias": 0.12135288149552403,
+        "top_bins": {
+          "0.9": {
+            "n": 4,
+            "errors": 1,
+            "error_rate": 0.25
+          },
+          "0.95": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.75,
+            "confidence_cutoff": 0.9282769651923547
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.5714285714285714,
+            "confidence_cutoff": 0.46447665537469063
+          }
+        }
+      },
+      "otrf_sysmon_file": {
+        "n": 16,
+        "nll": 0.012087874921322014,
+        "acc": 1.0,
+        "ece": 0.01177182723418102,
+        "brier": 0.0009653657802209021,
+        "mean_conf": 0.988228172765819,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.01177182723418102,
+        "top_bins": {
+          "0.9": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 14,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 12,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9982843022939405
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9893258214904856
+          }
+        }
+      },
+      "otrf_sysmon_join": {
+        "n": 16,
+        "nll": 0.09339595981110241,
+        "acc": 0.9375,
+        "ece": 0.07064953927615637,
+        "brier": 0.05310446080660936,
+        "mean_conf": 0.9435973784923795,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 0.8125,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 0.9375,
+        "coverage_at_1pct_error": 0.9375,
+        "aurc": 0.00390625,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": 0.006097378492379457,
+        "top_bins": {
+          "0.9": {
+            "n": 13,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 10,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 9,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9971649155189333
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.902828586126225
+          }
+        }
+      },
+      "otrf_sysmon_process": {
+        "n": 24,
+        "nll": 0.14712967386771258,
+        "acc": 0.9583333333333334,
+        "ece": 0.050115936293844,
+        "brier": 0.08546668992178032,
+        "mean_conf": 0.9763794161864038,
+        "confident_error_rate": 0.041666666666666664,
+        "coverage_at_0_9": 0.9583333333333334,
+        "accuracy_at_0_9": 0.9565217391304348,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 0.9166666666666666,
+        "aurc": 0.0035477053140096615,
+        "error_rate_at_0_9": 0.043478260869565216,
+        "confidence_bias": 0.01804608285307041,
+        "top_bins": {
+          "0.9": {
+            "n": 23,
+            "errors": 1,
+            "error_rate": 0.043478260869565216
+          },
+          "0.95": {
+            "n": 22,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 18,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9990212499079042
+          },
+          "0.8": {
+            "coverage": 0.8333333333333334,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9811939267943016
+          }
+        }
+      },
+      "otrf_sysmon_registry": {
+        "n": 8,
+        "nll": 0.01343350224594829,
+        "acc": 1.0,
+        "ece": 0.013163154419292722,
+        "brier": 0.0010226691949093074,
+        "mean_conf": 0.9868368455807073,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.013163154419292722,
+        "top_bins": {
+          "0.9": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 7,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 5,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.996202823022372
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9801659585186474
+          }
+        }
+      },
+      "paws": {
+        "n": 8,
+        "nll": 1.0224593100569639,
+        "acc": 0.75,
+        "ece": 0.16095362351892603,
+        "brier": 0.3818518035200381,
+        "mean_conf": 0.9109536235189261,
+        "confident_error_rate": 0.125,
+        "coverage_at_0_9": 0.75,
+        "accuracy_at_0_9": 0.8333333333333334,
+        "coverage_at_5pct_error": 0.375,
+        "coverage_at_1pct_error": 0.375,
+        "aurc": 0.1261904761904762,
+        "error_rate_at_0_9": 0.16666666666666666,
+        "confidence_bias": 0.16095362351892606,
+        "top_bins": {
+          "0.9": {
+            "n": 6,
+            "errors": 1,
+            "error_rate": 0.16666666666666666
+          },
+          "0.95": {
+            "n": 6,
+            "errors": 1,
+            "error_rate": 0.16666666666666666
+          },
+          "0.99": {
+            "n": 6,
+            "errors": 1,
+            "error_rate": 0.16666666666666666
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.75,
+            "confidence_cutoff": 0.9988025069897564
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.8571428571428571,
+            "confidence_cutoff": 0.6535177425598425
+          }
+        }
+      },
+      "qnli": {
+        "n": 8,
+        "nll": 0.006921467980490048,
+        "acc": 1.0,
+        "ece": 0.006826489572726913,
+        "brier": 0.00037169676338399797,
+        "mean_conf": 0.9931735104272731,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.006826489572726913,
+        "top_bins": {
+          "0.9": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 6,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999038603722666
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9858691572991208
+          }
+        }
+      },
+      "sciq": {
+        "n": 8,
+        "nll": 0.0003138394695547839,
+        "acc": 1.0,
+        "ece": 0.00031363748733115315,
+        "brier": 8.029030123915446e-07,
+        "mean_conf": 0.9996863625126688,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.00031363748733115315,
+        "top_bins": {
+          "0.9": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999823883457734
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9991993823290242
+          }
+        }
+      },
+      "sev_swarmtraces_manual_credential": {
+        "n": 40,
+        "nll": 0.906880605987712,
+        "acc": 0.8,
+        "ece": 0.206070522134782,
+        "brier": 0.3722274315061049,
+        "mean_conf": 0.8993889243102753,
+        "confident_error_rate": 0.15,
+        "coverage_at_0_9": 0.725,
+        "accuracy_at_0_9": 0.7931034482758621,
+        "coverage_at_5pct_error": 0.275,
+        "coverage_at_1pct_error": 0.275,
+        "aurc": 0.10674973976967887,
+        "error_rate_at_0_9": 0.20689655172413793,
+        "confidence_bias": 0.09938892431027524,
+        "top_bins": {
+          "0.9": {
+            "n": 29,
+            "errors": 6,
+            "error_rate": 0.20689655172413793
+          },
+          "0.95": {
+            "n": 21,
+            "errors": 2,
+            "error_rate": 0.09523809523809523
+          },
+          "0.99": {
+            "n": 12,
+            "errors": 1,
+            "error_rate": 0.08333333333333333
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.9,
+            "confidence_cutoff": 0.977808816046421
+          },
+          "0.8": {
+            "coverage": 0.8,
+            "accuracy": 0.8125,
+            "confidence_cutoff": 0.8136632702387824
+          }
+        }
+      },
+      "sev_swarmtraces_manual_flow": {
+        "n": 40,
+        "nll": 1.3789279125484732,
+        "acc": 0.75,
+        "ece": 0.21553934746408088,
+        "brier": 0.43021742389448914,
+        "mean_conf": 0.9146488310871621,
+        "confident_error_rate": 0.15,
+        "coverage_at_0_9": 0.725,
+        "accuracy_at_0_9": 0.7931034482758621,
+        "coverage_at_5pct_error": 0.125,
+        "coverage_at_1pct_error": 0.125,
+        "aurc": 0.1325342131938582,
+        "error_rate_at_0_9": 0.20689655172413793,
+        "confidence_bias": 0.1646488310871621,
+        "top_bins": {
+          "0.9": {
+            "n": 29,
+            "errors": 6,
+            "error_rate": 0.20689655172413793
+          },
+          "0.95": {
+            "n": 27,
+            "errors": 6,
+            "error_rate": 0.2222222222222222
+          },
+          "0.99": {
+            "n": 19,
+            "errors": 2,
+            "error_rate": 0.10526315789473684
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.9,
+            "confidence_cutoff": 0.9892087090945672
+          },
+          "0.8": {
+            "coverage": 0.8,
+            "accuracy": 0.8125,
+            "confidence_cutoff": 0.8501876374234145
+          }
+        }
+      },
+      "sev_swarmtraces_manual_prerequisite": {
+        "n": 3,
+        "nll": 2.3971020760583907e-05,
+        "acc": 1.0,
+        "ece": 2.3970614154600334e-05,
+        "brier": 1.3240221828953951e-09,
+        "mean_conf": 0.9999760293858454,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -2.3970614154600334e-05,
+        "top_bins": {
+          "0.9": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.6666666666666666,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999746394822547
+          },
+          "0.8": {
+            "coverage": 1.0,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999577692345681
+          }
+        }
+      },
+      "tweet_offensive": {
+        "n": 8,
+        "nll": 0.6354715067569514,
+        "acc": 0.75,
+        "ece": 0.20516385594706016,
+        "brier": 0.4183423274934326,
+        "mean_conf": 0.9551638559470601,
+        "confident_error_rate": 0.125,
+        "coverage_at_0_9": 0.875,
+        "accuracy_at_0_9": 0.8571428571428571,
+        "coverage_at_5pct_error": 0.5,
+        "coverage_at_1pct_error": 0.5,
+        "aurc": 0.09494047619047619,
+        "error_rate_at_0_9": 0.14285714285714285,
+        "confidence_bias": 0.2051638559470601,
+        "top_bins": {
+          "0.9": {
+            "n": 7,
+            "errors": 1,
+            "error_rate": 0.14285714285714285
+          },
+          "0.95": {
+            "n": 4,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9829805814759643
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.8571428571428571,
+            "confidence_cutoff": 0.9198208656867042
+          }
+        }
+      }
+    },
+    "variants": {
+      "clean": {
+        "n": 243,
+        "nll": 0.5657599517457697,
+        "acc": 0.8641975308641975,
+        "ece": 0.09883136412561748,
+        "brier": 0.2329497754098416,
+        "mean_conf": 0.9456019169777928,
+        "confident_error_rate": 0.07818930041152264,
+        "coverage_at_0_9": 0.8518518518518519,
+        "accuracy_at_0_9": 0.9082125603864735,
+        "coverage_at_5pct_error": 0.7119341563786008,
+        "coverage_at_1pct_error": 0.411522633744856,
+        "aurc": 0.033948169133907836,
+        "error_rate_at_0_9": 0.09178743961352658,
+        "confidence_bias": 0.08140438611359535,
+        "top_bins": {
+          "0.9": {
+            "n": 207,
+            "errors": 19,
+            "error_rate": 0.09178743961352658
+          },
+          "0.95": {
+            "n": 179,
+            "errors": 10,
+            "error_rate": 0.055865921787709494
+          },
+          "0.99": {
+            "n": 147,
+            "errors": 4,
+            "error_rate": 0.027210884353741496
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5020576131687243,
+            "accuracy": 0.9836065573770492,
+            "confidence_cutoff": 0.9975663879354146
+          },
+          "0.8": {
+            "coverage": 0.8024691358024691,
+            "accuracy": 0.9179487179487179,
+            "confidence_cutoff": 0.9296758313428735
+          }
+        }
+      },
+      "none_absent": {
+        "n": 3,
+        "nll": 1.2952152099288836,
+        "acc": 0.6666666666666666,
+        "ece": 0.23294183559227233,
+        "brier": 0.5788021079044262,
+        "mean_conf": 0.8989412293916658,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 0.3333333333333333,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 0.3333333333333333,
+        "coverage_at_1pct_error": 0.3333333333333333,
+        "aurc": 0.27777777777777773,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": 0.23227456272499913,
+        "top_bins": {
+          "0.9": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.6666666666666666,
+            "accuracy": 0.5,
+            "confidence_cutoff": 0.8518517543949747
+          },
+          "0.8": {
+            "coverage": 1.0,
+            "accuracy": 0.6666666666666666,
+            "confidence_cutoff": 0.8459728430809323
+          }
+        }
+      },
+      "none_present": {
+        "n": 3,
+        "nll": 3.1553040383653848,
+        "acc": 0.0,
+        "ece": 0.7926146688269071,
+        "brier": 1.555251640735374,
+        "mean_conf": 0.7926146688269071,
+        "confident_error_rate": 0.3333333333333333,
+        "coverage_at_0_9": 0.3333333333333333,
+        "accuracy_at_0_9": 0.0,
+        "coverage_at_5pct_error": 0.0,
+        "coverage_at_1pct_error": 0.0,
+        "aurc": 1.0,
+        "error_rate_at_0_9": 1.0,
+        "confidence_bias": 0.7926146688269071,
+        "top_bins": {
+          "0.9": {
+            "n": 1,
+            "errors": 1,
+            "error_rate": 1.0
+          },
+          "0.95": {
+            "n": 1,
+            "errors": 1,
+            "error_rate": 1.0
+          },
+          "0.99": {
+            "n": 0,
+            "errors": 0,
+            "error_rate": null
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.6666666666666666,
+            "accuracy": 0.0,
+            "confidence_cutoff": 0.8341568322128006
+          },
+          "0.8": {
+            "coverage": 1.0,
+            "accuracy": 0.0,
+            "confidence_cutoff": 0.5895972207620483
+          }
+        }
+      },
+      "permuted": {
+        "n": 3,
+        "nll": 1.485986404164769,
+        "acc": 0.3333333333333333,
+        "ece": 0.49707924001582554,
+        "brier": 0.9062019359731462,
+        "mean_conf": 0.8151635960376794,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 0.3333333333333333,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 0.3333333333333333,
+        "coverage_at_1pct_error": 0.3333333333333333,
+        "aurc": 0.38888888888888884,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": 0.4818302627043461,
+        "top_bins": {
+          "0.9": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 0,
+            "errors": 0,
+            "error_rate": null
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.6666666666666666,
+            "accuracy": 0.5,
+            "confidence_cutoff": 0.8281181942730632
+          },
+          "0.8": {
+            "coverage": 1.0,
+            "accuracy": 0.3333333333333333,
+            "confidence_cutoff": 0.6402460598071943
+          }
+        }
+      }
+    },
+    "heldout_tasks": {
+      "composition_held_and_or": {
+        "n": 32,
+        "nll": 0.21420351636532084,
+        "acc": 0.90625,
+        "ece": 0.07786484107903366,
+        "brier": 0.14841282029529768,
+        "mean_conf": 0.9841148410790337,
+        "confident_error_rate": 0.03125,
+        "coverage_at_0_9": 0.9375,
+        "accuracy_at_0_9": 0.9666666666666667,
+        "coverage_at_5pct_error": 0.9375,
+        "coverage_at_1pct_error": 0.90625,
+        "aurc": 0.0069010416666666664,
+        "error_rate_at_0_9": 0.03333333333333333,
+        "confidence_bias": 0.07786484107903369,
+        "top_bins": {
+          "0.9": {
+            "n": 30,
+            "errors": 1,
+            "error_rate": 0.03333333333333333
+          },
+          "0.95": {
+            "n": 27,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 26,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9998649716969166
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9993232293186822
+          }
+        }
+      },
+      "contrastive_authorization": {
+        "n": 16,
+        "nll": 3.491797866673952e-05,
+        "acc": 1.0,
+        "ece": 3.491729606275307e-05,
+        "brier": 2.7334442890954754e-09,
+        "mean_conf": 0.9999650827039372,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -3.491729606275307e-05,
+        "top_bins": {
+          "0.9": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999685834101107
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999562785984693
+          }
+        }
+      },
+      "emotion": {
+        "n": 8,
+        "nll": 1.29779922266396,
+        "acc": 0.625,
+        "ece": 0.33113247991180284,
+        "brier": 0.6610413670516024,
+        "mean_conf": 0.956132479911803,
+        "confident_error_rate": 0.25,
+        "coverage_at_0_9": 0.875,
+        "accuracy_at_0_9": 0.7142857142857143,
+        "coverage_at_5pct_error": 0.5,
+        "coverage_at_1pct_error": 0.5,
+        "aurc": 0.12842261904761904,
+        "error_rate_at_0_9": 0.2857142857142857,
+        "confidence_bias": 0.33113247991180295,
+        "top_bins": {
+          "0.9": {
+            "n": 7,
+            "errors": 2,
+            "error_rate": 0.2857142857142857
+          },
+          "0.95": {
+            "n": 5,
+            "errors": 1,
+            "error_rate": 0.2
+          },
+          "0.99": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9816738572788068
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.7142857142857143,
+            "confidence_cutoff": 0.9054981634553654
+          }
+        }
+      },
+      "mmlu": {
+        "n": 8,
+        "nll": 1.270267510803572,
+        "acc": 0.625,
+        "ece": 0.27270807209084075,
+        "brier": 0.6428034740610011,
+        "mean_conf": 0.746352881495524,
+        "confident_error_rate": 0.125,
+        "coverage_at_0_9": 0.5,
+        "accuracy_at_0_9": 0.75,
+        "coverage_at_5pct_error": 0.375,
+        "coverage_at_1pct_error": 0.375,
+        "aurc": 0.22336309523809525,
+        "error_rate_at_0_9": 0.25,
+        "confidence_bias": 0.12135288149552403,
+        "top_bins": {
+          "0.9": {
+            "n": 4,
+            "errors": 1,
+            "error_rate": 0.25
+          },
+          "0.95": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 1,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.75,
+            "confidence_cutoff": 0.9282769651923547
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.5714285714285714,
+            "confidence_cutoff": 0.46447665537469063
+          }
+        }
+      },
+      "otrf_sysmon_file": {
+        "n": 16,
+        "nll": 0.012087874921322014,
+        "acc": 1.0,
+        "ece": 0.01177182723418102,
+        "brier": 0.0009653657802209021,
+        "mean_conf": 0.988228172765819,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.01177182723418102,
+        "top_bins": {
+          "0.9": {
+            "n": 16,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 14,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 12,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9982843022939405
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9893258214904856
+          }
+        }
+      },
+      "otrf_sysmon_join": {
+        "n": 16,
+        "nll": 0.09339595981110241,
+        "acc": 0.9375,
+        "ece": 0.07064953927615637,
+        "brier": 0.05310446080660936,
+        "mean_conf": 0.9435973784923795,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 0.8125,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 0.9375,
+        "coverage_at_1pct_error": 0.9375,
+        "aurc": 0.00390625,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": 0.006097378492379457,
+        "top_bins": {
+          "0.9": {
+            "n": 13,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 10,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 9,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9971649155189333
+          },
+          "0.8": {
+            "coverage": 0.8125,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.902828586126225
+          }
+        }
+      },
+      "otrf_sysmon_process": {
+        "n": 24,
+        "nll": 0.14712967386771258,
+        "acc": 0.9583333333333334,
+        "ece": 0.050115936293844,
+        "brier": 0.08546668992178032,
+        "mean_conf": 0.9763794161864038,
+        "confident_error_rate": 0.041666666666666664,
+        "coverage_at_0_9": 0.9583333333333334,
+        "accuracy_at_0_9": 0.9565217391304348,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 0.9166666666666666,
+        "aurc": 0.0035477053140096615,
+        "error_rate_at_0_9": 0.043478260869565216,
+        "confidence_bias": 0.01804608285307041,
+        "top_bins": {
+          "0.9": {
+            "n": 23,
+            "errors": 1,
+            "error_rate": 0.043478260869565216
+          },
+          "0.95": {
+            "n": 22,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 18,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9990212499079042
+          },
+          "0.8": {
+            "coverage": 0.8333333333333334,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9811939267943016
+          }
+        }
+      },
+      "otrf_sysmon_registry": {
+        "n": 8,
+        "nll": 0.01343350224594829,
+        "acc": 1.0,
+        "ece": 0.013163154419292722,
+        "brier": 0.0010226691949093074,
+        "mean_conf": 0.9868368455807073,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.013163154419292722,
+        "top_bins": {
+          "0.9": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 7,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 5,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.996202823022372
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9801659585186474
+          }
+        }
+      },
+      "paws": {
+        "n": 8,
+        "nll": 1.0224593100569639,
+        "acc": 0.75,
+        "ece": 0.16095362351892603,
+        "brier": 0.3818518035200381,
+        "mean_conf": 0.9109536235189261,
+        "confident_error_rate": 0.125,
+        "coverage_at_0_9": 0.75,
+        "accuracy_at_0_9": 0.8333333333333334,
+        "coverage_at_5pct_error": 0.375,
+        "coverage_at_1pct_error": 0.375,
+        "aurc": 0.1261904761904762,
+        "error_rate_at_0_9": 0.16666666666666666,
+        "confidence_bias": 0.16095362351892606,
+        "top_bins": {
+          "0.9": {
+            "n": 6,
+            "errors": 1,
+            "error_rate": 0.16666666666666666
+          },
+          "0.95": {
+            "n": 6,
+            "errors": 1,
+            "error_rate": 0.16666666666666666
+          },
+          "0.99": {
+            "n": 6,
+            "errors": 1,
+            "error_rate": 0.16666666666666666
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.75,
+            "confidence_cutoff": 0.9988025069897564
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.8571428571428571,
+            "confidence_cutoff": 0.6535177425598425
+          }
+        }
+      },
+      "qnli": {
+        "n": 8,
+        "nll": 0.006921467980490048,
+        "acc": 1.0,
+        "ece": 0.006826489572726913,
+        "brier": 0.00037169676338399797,
+        "mean_conf": 0.9931735104272731,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.006826489572726913,
+        "top_bins": {
+          "0.9": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 6,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999038603722666
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9858691572991208
+          }
+        }
+      },
+      "sciq": {
+        "n": 8,
+        "nll": 0.0003138394695547839,
+        "acc": 1.0,
+        "ece": 0.00031363748733115315,
+        "brier": 8.029030123915446e-07,
+        "mean_conf": 0.9996863625126688,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -0.00031363748733115315,
+        "top_bins": {
+          "0.9": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 8,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999823883457734
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9991993823290242
+          }
+        }
+      },
+      "sev_swarmtraces_manual_credential": {
+        "n": 40,
+        "nll": 0.906880605987712,
+        "acc": 0.8,
+        "ece": 0.206070522134782,
+        "brier": 0.3722274315061049,
+        "mean_conf": 0.8993889243102753,
+        "confident_error_rate": 0.15,
+        "coverage_at_0_9": 0.725,
+        "accuracy_at_0_9": 0.7931034482758621,
+        "coverage_at_5pct_error": 0.275,
+        "coverage_at_1pct_error": 0.275,
+        "aurc": 0.10674973976967887,
+        "error_rate_at_0_9": 0.20689655172413793,
+        "confidence_bias": 0.09938892431027524,
+        "top_bins": {
+          "0.9": {
+            "n": 29,
+            "errors": 6,
+            "error_rate": 0.20689655172413793
+          },
+          "0.95": {
+            "n": 21,
+            "errors": 2,
+            "error_rate": 0.09523809523809523
+          },
+          "0.99": {
+            "n": 12,
+            "errors": 1,
+            "error_rate": 0.08333333333333333
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.9,
+            "confidence_cutoff": 0.977808816046421
+          },
+          "0.8": {
+            "coverage": 0.8,
+            "accuracy": 0.8125,
+            "confidence_cutoff": 0.8136632702387824
+          }
+        }
+      },
+      "sev_swarmtraces_manual_flow": {
+        "n": 40,
+        "nll": 1.3789279125484732,
+        "acc": 0.75,
+        "ece": 0.21553934746408088,
+        "brier": 0.43021742389448914,
+        "mean_conf": 0.9146488310871621,
+        "confident_error_rate": 0.15,
+        "coverage_at_0_9": 0.725,
+        "accuracy_at_0_9": 0.7931034482758621,
+        "coverage_at_5pct_error": 0.125,
+        "coverage_at_1pct_error": 0.125,
+        "aurc": 0.1325342131938582,
+        "error_rate_at_0_9": 0.20689655172413793,
+        "confidence_bias": 0.1646488310871621,
+        "top_bins": {
+          "0.9": {
+            "n": 29,
+            "errors": 6,
+            "error_rate": 0.20689655172413793
+          },
+          "0.95": {
+            "n": 27,
+            "errors": 6,
+            "error_rate": 0.2222222222222222
+          },
+          "0.99": {
+            "n": 19,
+            "errors": 2,
+            "error_rate": 0.10526315789473684
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 0.9,
+            "confidence_cutoff": 0.9892087090945672
+          },
+          "0.8": {
+            "coverage": 0.8,
+            "accuracy": 0.8125,
+            "confidence_cutoff": 0.8501876374234145
+          }
+        }
+      },
+      "sev_swarmtraces_manual_prerequisite": {
+        "n": 3,
+        "nll": 2.3971020760583907e-05,
+        "acc": 1.0,
+        "ece": 2.3970614154600334e-05,
+        "brier": 1.3240221828953951e-09,
+        "mean_conf": 0.9999760293858454,
+        "confident_error_rate": 0.0,
+        "coverage_at_0_9": 1.0,
+        "accuracy_at_0_9": 1.0,
+        "coverage_at_5pct_error": 1.0,
+        "coverage_at_1pct_error": 1.0,
+        "aurc": 0.0,
+        "error_rate_at_0_9": 0.0,
+        "confidence_bias": -2.3970614154600334e-05,
+        "top_bins": {
+          "0.9": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.95": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.6666666666666666,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999746394822547
+          },
+          "0.8": {
+            "coverage": 1.0,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9999577692345681
+          }
+        }
+      },
+      "tweet_offensive": {
+        "n": 8,
+        "nll": 0.6354715067569514,
+        "acc": 0.75,
+        "ece": 0.20516385594706016,
+        "brier": 0.4183423274934326,
+        "mean_conf": 0.9551638559470601,
+        "confident_error_rate": 0.125,
+        "coverage_at_0_9": 0.875,
+        "accuracy_at_0_9": 0.8571428571428571,
+        "coverage_at_5pct_error": 0.5,
+        "coverage_at_1pct_error": 0.5,
+        "aurc": 0.09494047619047619,
+        "error_rate_at_0_9": 0.14285714285714285,
+        "confidence_bias": 0.2051638559470601,
+        "top_bins": {
+          "0.9": {
+            "n": 7,
+            "errors": 1,
+            "error_rate": 0.14285714285714285
+          },
+          "0.95": {
+            "n": 4,
+            "errors": 0,
+            "error_rate": 0.0
+          },
+          "0.99": {
+            "n": 3,
+            "errors": 0,
+            "error_rate": 0.0
+          }
+        },
+        "selective": {
+          "0.5": {
+            "coverage": 0.5,
+            "accuracy": 1.0,
+            "confidence_cutoff": 0.9829805814759643
+          },
+          "0.8": {
+            "coverage": 0.875,
+            "accuracy": 0.8571428571428571,
+            "confidence_cutoff": 0.9198208656867042
+          }
+        }
+      }
+    },
+    "permutation": {
+      "n": 3,
+      "mean_max_delta": 0.04657707709900826,
+      "flip_rate": 0.0
+    },
+    "temperature": 1.6245047927124707,
+    "calibrated_clean": {
+      "n": 243,
+      "nll": 0.43693910679200926,
+      "acc": 0.8641975308641975,
+      "ece": 0.03675442028315886,
+      "brier": 0.21507438269676163,
+      "mean_conf": 0.8943026059631948,
+      "confident_error_rate": 0.0205761316872428,
+      "coverage_at_0_9": 0.6790123456790124,
+      "accuracy_at_0_9": 0.9696969696969697,
+      "coverage_at_5pct_error": 0.7489711934156379,
+      "coverage_at_1pct_error": 0.41975308641975306,
+      "aurc": 0.03304535542910683,
+      "error_rate_at_0_9": 0.030303030303030304,
+      "confidence_bias": 0.03010507509899729,
+      "top_bins": {
+        "0.9": {
+          "n": 165,
+          "errors": 5,
+          "error_rate": 0.030303030303030304
+        },
+        "0.95": {
+          "n": 137,
+          "errors": 2,
+          "error_rate": 0.014598540145985401
+        },
+        "0.99": {
+          "n": 75,
+          "errors": 0,
+          "error_rate": 0.0
+        }
+      },
+      "selective": {
+        "0.5": {
+          "coverage": 0.5020576131687243,
+          "accuracy": 0.9836065573770492,
+          "confidence_cutoff": 0.969883243235829
+        },
+        "0.8": {
+          "coverage": 0.8024691358024691,
+          "accuracy": 0.9282051282051282,
+          "confidence_cutoff": 0.8019637256887074
+        }
+      }
+    },
+    "metric_policy": {
+      "version": 2,
+      "selective_ties": "whole_confidence_groups",
+      "coverage_at_error": "in-sample maximum over confidence thresholds; not a deployed error guarantee",
+      "aurc": "right-step integral over whole confidence groups",
+      "confident_error_rate": "high-confidence errors divided by all questions",
+      "error_rate_at_0_9": "errors divided by questions accepted at p_max >= 0.9",
+      "nll": "exact from logits when recorded; otherwise from floored probabilities",
+      "nll_floor": 1e-09,
+      "renormalize_returned_probabilities": true,
+      "raw_sums_outside_1e_5": 0,
+      "returned_zeros": 0
+    },
+    "coverage": {
+      "requested_records": 177,
+      "requested_questions": 252,
+      "evaluated_records": 177,
+      "evaluated_questions": 252,
+      "rejected_records": 0,
+      "truncated_records": 0
+    },
+    "latency_ms": {
+      "median": 60.66920900002515,
+      "p95": 191.46798359997774
+    },
+    "calibration": {
+      "inference_temperature": 1.0,
+      "additional_temperature": 1.6245047927124707,
+      "logits_recorded": true
+    },
+    "suite_sha256": "1caf2255850930113bc56b708d0881ed22d3df9ba1868e966f9454595c2cefa7"
+  },
   "provenance": {
     "config": {
       "epochs": 1,
       "seed": 4,
-      "lr": 1e-05,
+      "lr": 2.5e-06,
       "lora": 16,
       "accum": 2,
       "batch": 4,
@@ -385,17 +2418,17 @@
       "focal_gamma": 0.0,
       "base": "Qwen/Qwen3.5-4B-Base",
       "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
-      "init_from": "jaredpalmer/kev-4b@485ace8703592fcf405488b262449990824cfed1",
+      "init_from": "/runs/sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint",
       "lora_targets": "all",
       "head_dim": 256,
       "weights_dtype": "fp32",
-      "option_isolation": 0,
-      "special_embeddings": 0,
       "dtype": "bf16",
-      "checkpointing": 1
+      "checkpointing": 1,
+      "option_isolation": 0,
+      "special_embeddings": 0
     },
-    "config_sha256": "1abcf6795472953561dc92bf54d59e9f10fd5e81d08d558f44197e56299077af",
-    "suite_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
+    "config_sha256": "958d0b20b3ffd1c572e0abf221383a450e0f8e8d3fff3cb745235a371e6a025d",
+    "suite_sha256": "61e3c1898b8a03eb178355d3c8c5050e85bd217285c700df9b4506ec9e3680f8",
     "source_hashes": {
       "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
       "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
@@ -423,28 +2456,28 @@
       "kev/suite.py": "c5e5e27f2fed871b547a021d180b0a7f4a0542a488cce4ee02ad936f433441db",
       "kev/train.py": "a2d85632d73760fb3efe51627ff4fde6af3100ed1c34e75d290c1b00c9899aa9",
       "kev/transfer_v9.py": "0902409742151250a28af1fd8f42258b70fcc161f3deed3ee35fe3f473f3c763",
-      "modal_app.py": "5d55aa2a454aba2ba6e4a392ed77ab2bc849e9823b8298a41ce1b8fe4217344b",
-      "pyproject.toml": "7c17fbe9efcadda3eb488b59adf1b6db5938cd33af4d3bd389ce4359b73fcc7d",
+      "modal_app.py": "639577258f4d581daefa51dad639961280ce92de9da256316ab4124f005a53fb",
+      "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
       "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
     },
-    "git_commit": "d8b1413f428a9a8f4e3d6091acf3d9ed8445c864",
+    "git_commit": "76e180880160721082a065bf1882495e756a643a",
     "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
     "torch": "2.8.0+cu128",
     "device": "cuda",
     "gpu": "NVIDIA H100 80GB HBM3",
     "legacy_checkpoint": false,
     "measured_checkpoint": {
-      "requested": "/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
-      "resolved": "/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
-      "head_sha256": "af6bfd215bf0b7eff92612a83b21cb56754ca87f0bf9f62fdfcfbef275d9eab6",
-      "adapter_sha256": "7fc73ad5a9e219731fba6bd926ba4a683b0c67dd676810d18e1a6018fa3b887c",
+      "requested": "/runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint",
+      "resolved": "/runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint",
+      "head_sha256": "b29abc817f2340aa510fcd4b2c813455ef38102508120ffc35f7917ce359c27e",
+      "adapter_sha256": "b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420",
       "inference_temperature": 1.0
     }
   },
   "mechanism_checks": {
-    "n": 8,
-    "packed_max_delta": 0.0,
-    "sibling_max_delta": 6.73830509185791e-05,
+    "n": 24,
+    "packed_max_delta": 1.430511474609375e-06,
+    "sibling_max_delta": 2.0265579223632812e-06,
     "tolerance": 0.001,
     "passed": true
   },
@@ -452,34 +2485,37 @@
     "passed": true,
     "checks": {
       "complete_coverage": true,
-      "isolation_and_packing": true
+      "isolation_and_packing": true,
+      "transfer_complete": true,
+      "heldout_pairs_at_least_70pct": true,
+      "transfer_confident_errors_below_10pct": true
     },
     "policy": "Research screening only: 0.001 isolation; 5pp task regression; 70% heldout pair correctness; <=10% confident errors; no automatic release."
   },
   "calibration_fit": {
-    "temperature": 1.0717734625362931,
+    "temperature": 1.6245047927124707,
     "aggregation": "micro",
     "objective": "raw-logit NLL",
     "split": "calibration",
-    "rows_sha256": "00909b4d6b838c534d8d403d84a578fea1ddaac5e30a2b488c2d1c138c0d05ab",
-    "suite_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
-    "n": 182
+    "rows_sha256": "5dc5633f59c9da863561e1ae34fbc1c87199053cf1397d4592b921ec5ff69790",
+    "suite_sha256": "61e3c1898b8a03eb178355d3c8c5050e85bd217285c700df9b4506ec9e3680f8",
+    "n": 2534
   },
-  "wall_seconds": 610.9060554900001,
+  "wall_seconds": 1473.325216132,
   "promotable": false,
   "test_evaluated": false,
   "training_resources": {
-    "wall_seconds": 371.20887875556946,
-    "records_seen": 2334,
-    "requested_records": 2334,
+    "wall_seconds": 1043.7473661899567,
+    "records_seen": 6438,
+    "requested_records": 6438,
     "truncated_records": 0,
     "rejected_records": 0,
-    "optimizer_steps": 292,
-    "forward_tokens": 738327,
-    "peak_device_bytes": 18913074688,
+    "optimizer_steps": 805,
+    "forward_tokens": 1447855,
+    "peak_device_bytes": 22227358208,
     "device": "cuda",
     "dtype": "bf16",
     "batch": 4,
-    "peak_rss_bytes": 31561670656
+    "peak_rss_bytes": 31557873664
   }
 }

train.log

@@ -1,35 +1,86 @@
 Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
-delta: warm start from /__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--jaredpalmer--kev-4b/snapshots/485ace8703592fcf405488b262449990824cfed1: 496 adapter tensors and the pointer head loaded
+delta: warm start from /runs/sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint: 496 adapter tensors and the pointer head loaded
 device=cuda trainable params=33.8M
-2334 training requests (holdout=['sev_behavioral_retry_export', 'sev_behavioral_stale_handoff', 'sev_behavioral_reconcile_export']), questions by type {'choice': 2280, 'score': 36, 'noul': 18}
+6438 training requests (holdout=[]), questions by type {'choice': 8312, 'noul': 518, 'score': 348}
 [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
-ep0 step 10/292 loss 1.829 kl 0.000 anchor 0.000 1.503s/rec
-ep0 step 20/292 loss 1.201 kl 0.000 anchor 0.000 0.808s/rec
-ep0 step 30/292 loss 0.990 kl 0.000 anchor 0.000 0.576s/rec
-ep0 step 40/292 loss 0.972 kl 0.000 anchor 0.000 0.461s/rec
-ep0 step 50/292 loss 1.026 kl 0.000 anchor 0.000 0.391s/rec
-ep0 step 60/292 loss 1.032 kl 0.000 anchor 0.000 0.345s/rec
-ep0 step 70/292 loss 1.014 kl 0.000 anchor 0.000 0.311s/rec
-ep0 step 80/292 loss 1.008 kl 0.000 anchor 0.000 0.286s/rec
-ep0 step 90/292 loss 1.031 kl 0.000 anchor 0.000 0.266s/rec
-ep0 step 100/292 loss 0.925 kl 0.000 anchor 0.000 0.250s/rec
-ep0 step 110/292 loss 1.073 kl 0.000 anchor 0.000 0.238s/rec
-ep0 step 120/292 loss 0.969 kl 0.000 anchor 0.000 0.227s/rec
-ep0 step 130/292 loss 1.000 kl 0.000 anchor 0.000 0.218s/rec
-ep0 step 140/292 loss 1.017 kl 0.000 anchor 0.000 0.210s/rec
-ep0 step 150/292 loss 1.058 kl 0.000 anchor 0.000 0.204s/rec
-ep0 step 160/292 loss 0.966 kl 0.000 anchor 0.000 0.198s/rec
-ep0 step 170/292 loss 0.980 kl 0.000 anchor 0.000 0.193s/rec
-ep0 step 180/292 loss 0.891 kl 0.000 anchor 0.000 0.188s/rec
-ep0 step 190/292 loss 1.000 kl 0.000 anchor 0.000 0.185s/rec
-ep0 step 200/292 loss 1.057 kl 0.000 anchor 0.000 0.181s/rec
-ep0 step 210/292 loss 0.975 kl 0.000 anchor 0.000 0.178s/rec
-ep0 step 220/292 loss 0.980 kl 0.000 anchor 0.000 0.175s/rec
-ep0 step 230/292 loss 0.944 kl 0.000 anchor 0.000 0.172s/rec
-ep0 step 240/292 loss 0.952 kl 0.000 anchor 0.000 0.169s/rec
-ep0 step 250/292 loss 0.985 kl 0.000 anchor 0.000 0.167s/rec
-ep0 step 260/292 loss 0.965 kl 0.000 anchor 0.000 0.165s/rec
-ep0 step 270/292 loss 1.043 kl 0.000 anchor 0.000 0.162s/rec
-ep0 step 280/292 loss 1.008 kl 0.000 anchor 0.000 0.160s/rec
-ep0 step 290/292 loss 0.974 kl 0.000 anchor 0.000 0.158s/rec
-saved /runs/sev-r2-research-4b-v1/00-trial-0/checkpoint
+ep0 step 10/805 loss 0.504 kl 0.000 anchor 0.000 2.046s/rec
+ep0 step 20/805 loss 0.420 kl 0.000 anchor 0.000 1.132s/rec
+ep0 step 30/805 loss 0.479 kl 0.000 anchor 0.000 0.791s/rec
+ep0 step 40/805 loss 0.527 kl 0.000 anchor 0.000 0.676s/rec
+ep0 step 50/805 loss 0.388 kl 0.000 anchor 0.000 0.565s/rec
+ep0 step 60/805 loss 0.502 kl 0.000 anchor 0.000 0.490s/rec
+ep0 step 70/805 loss 0.516 kl 0.000 anchor 0.000 0.435s/rec
+ep0 step 80/805 loss 0.345 kl 0.000 anchor 0.000 0.404s/rec
+ep0 step 90/805 loss 0.594 kl 0.000 anchor 0.000 0.375s/rec
+ep0 step 100/805 loss 0.517 kl 0.000 anchor 0.000 0.354s/rec
+ep0 step 110/805 loss 0.513 kl 0.000 anchor 0.000 0.335s/rec
+ep0 step 120/805 loss 0.416 kl 0.000 anchor 0.000 0.319s/rec
+ep0 step 130/805 loss 0.503 kl 0.000 anchor 0.000 0.305s/rec
+ep0 step 140/805 loss 0.671 kl 0.000 anchor 0.000 0.292s/rec
+ep0 step 150/805 loss 0.445 kl 0.000 anchor 0.000 0.281s/rec
+ep0 step 160/805 loss 0.492 kl 0.000 anchor 0.000 0.271s/rec
+ep0 step 170/805 loss 0.645 kl 0.000 anchor 0.000 0.262s/rec
+ep0 step 180/805 loss 0.434 kl 0.000 anchor 0.000 0.254s/rec
+ep0 step 190/805 loss 0.505 kl 0.000 anchor 0.000 0.247s/rec
+ep0 step 200/805 loss 0.408 kl 0.000 anchor 0.000 0.240s/rec
+ep0 step 210/805 loss 0.541 kl 0.000 anchor 0.000 0.235s/rec
+ep0 step 220/805 loss 0.447 kl 0.000 anchor 0.000 0.230s/rec
+ep0 step 230/805 loss 0.562 kl 0.000 anchor 0.000 0.226s/rec
+ep0 step 240/805 loss 0.433 kl 0.000 anchor 0.000 0.221s/rec
+ep0 step 250/805 loss 0.321 kl 0.000 anchor 0.000 0.217s/rec
+ep0 step 260/805 loss 0.495 kl 0.000 anchor 0.000 0.214s/rec
+ep0 step 270/805 loss 0.452 kl 0.000 anchor 0.000 0.210s/rec
+ep0 step 280/805 loss 0.387 kl 0.000 anchor 0.000 0.209s/rec
+ep0 step 290/805 loss 0.345 kl 0.000 anchor 0.000 0.206s/rec
+ep0 step 300/805 loss 0.497 kl 0.000 anchor 0.000 0.204s/rec
+ep0 step 310/805 loss 0.467 kl 0.000 anchor 0.000 0.202s/rec
+ep0 step 320/805 loss 0.409 kl 0.000 anchor 0.000 0.200s/rec
+ep0 step 330/805 loss 0.411 kl 0.000 anchor 0.000 0.198s/rec
+ep0 step 340/805 loss 0.306 kl 0.000 anchor 0.000 0.196s/rec
+ep0 step 350/805 loss 0.456 kl 0.000 anchor 0.000 0.194s/rec
+ep0 step 360/805 loss 0.420 kl 0.000 anchor 0.000 0.193s/rec
+ep0 step 370/805 loss 0.442 kl 0.000 anchor 0.000 0.191s/rec
+ep0 step 380/805 loss 0.433 kl 0.000 anchor 0.000 0.190s/rec
+ep0 step 390/805 loss 0.377 kl 0.000 anchor 0.000 0.188s/rec
+ep0 step 400/805 loss 0.434 kl 0.000 anchor 0.000 0.187s/rec
+ep0 step 410/805 loss 0.515 kl 0.000 anchor 0.000 0.185s/rec
+ep0 step 420/805 loss 0.438 kl 0.000 anchor 0.000 0.184s/rec
+ep0 step 430/805 loss 0.445 kl 0.000 anchor 0.000 0.183s/rec
+ep0 step 440/805 loss 0.396 kl 0.000 anchor 0.000 0.182s/rec
+ep0 step 450/805 loss 0.580 kl 0.000 anchor 0.000 0.180s/rec
+ep0 step 460/805 loss 0.508 kl 0.000 anchor 0.000 0.179s/rec
+ep0 step 470/805 loss 0.414 kl 0.000 anchor 0.000 0.184s/rec
+ep0 step 480/805 loss 0.507 kl 0.000 anchor 0.000 0.182s/rec
+ep0 step 490/805 loss 0.647 kl 0.000 anchor 0.000 0.181s/rec
+ep0 step 500/805 loss 0.340 kl 0.000 anchor 0.000 0.180s/rec
+ep0 step 510/805 loss 0.447 kl 0.000 anchor 0.000 0.179s/rec
+ep0 step 520/805 loss 0.501 kl 0.000 anchor 0.000 0.178s/rec
+ep0 step 530/805 loss 0.543 kl 0.000 anchor 0.000 0.177s/rec
+ep0 step 540/805 loss 0.399 kl 0.000 anchor 0.000 0.177s/rec
+ep0 step 550/805 loss 0.385 kl 0.000 anchor 0.000 0.176s/rec
+ep0 step 560/805 loss 0.450 kl 0.000 anchor 0.000 0.175s/rec
+ep0 step 570/805 loss 0.488 kl 0.000 anchor 0.000 0.174s/rec
+ep0 step 580/805 loss 0.657 kl 0.000 anchor 0.000 0.173s/rec
+ep0 step 590/805 loss 0.508 kl 0.000 anchor 0.000 0.173s/rec
+ep0 step 600/805 loss 0.548 kl 0.000 anchor 0.000 0.172s/rec
+ep0 step 610/805 loss 0.486 kl 0.000 anchor 0.000 0.171s/rec
+ep0 step 620/805 loss 0.411 kl 0.000 anchor 0.000 0.170s/rec
+ep0 step 630/805 loss 0.461 kl 0.000 anchor 0.000 0.170s/rec
+ep0 step 640/805 loss 0.305 kl 0.000 anchor 0.000 0.169s/rec
+ep0 step 650/805 loss 0.539 kl 0.000 anchor 0.000 0.169s/rec
+ep0 step 660/805 loss 0.434 kl 0.000 anchor 0.000 0.168s/rec
+ep0 step 670/805 loss 0.413 kl 0.000 anchor 0.000 0.167s/rec
+ep0 step 680/805 loss 0.421 kl 0.000 anchor 0.000 0.167s/rec
+ep0 step 690/805 loss 0.541 kl 0.000 anchor 0.000 0.166s/rec
+ep0 step 700/805 loss 0.465 kl 0.000 anchor 0.000 0.166s/rec
+ep0 step 710/805 loss 0.526 kl 0.000 anchor 0.000 0.165s/rec
+ep0 step 720/805 loss 0.582 kl 0.000 anchor 0.000 0.165s/rec
+ep0 step 730/805 loss 0.458 kl 0.000 anchor 0.000 0.164s/rec
+ep0 step 740/805 loss 0.450 kl 0.000 anchor 0.000 0.164s/rec
+ep0 step 750/805 loss 0.535 kl 0.000 anchor 0.000 0.164s/rec
+ep0 step 760/805 loss 0.489 kl 0.000 anchor 0.000 0.163s/rec
+ep0 step 770/805 loss 0.512 kl 0.000 anchor 0.000 0.163s/rec
+ep0 step 780/805 loss 0.480 kl 0.000 anchor 0.000 0.162s/rec
+ep0 step 790/805 loss 0.680 kl 0.000 anchor 0.000 0.162s/rec
+ep0 step 800/805 loss 0.529 kl 0.000 anchor 0.000 0.162s/rec
+saved /runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint

training_config.json

@@ -3,7 +3,7 @@
     "base": "Qwen/Qwen3.5-4B-Base",
     "n_per_source": 1000,
     "epochs": 1,
-    "lr": 1e-05,
+    "lr": 2.5e-06,
     "head_lr": 0.0,
     "weight_decay": 0.01,
     "lora": 16,
@@ -15,7 +15,7 @@
     "label_smoothing": 0.0,
     "brier_w": 0.0,
     "focal_gamma": 0.0,
-    "suite": "/root/evals/sev/behavioral-research-v1",
+    "suite": "/root/evals/sev/swarmtraces-curriculum-v1",
     "train_sources": "",
     "device": "cuda",
     "batch": 4,
@@ -36,25 +36,21 @@
     "anchor": "",
     "anchor_w": 0.0,
     "anchor_sources": "",
-    "out": "/runs/sev-r2-research-4b-v1/00-trial-0/checkpoint",
+    "out": "/runs/sev-r2-swarmtraces-4b-retention-v1/00-trial-0/checkpoint",
     "data": "",
     "replay": 0,
-    "init_from": "jaredpalmer/kev-4b@485ace8703592fcf405488b262449990824cfed1",
+    "init_from": "/runs/sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint",
     "seed": 4
   },
-  "suite_sha256": "372d8709cce5fc50262afc5107cd4b3637614ff2b2a7e84a8ccfa4293a4b21c5",
+  "suite_sha256": "61e3c1898b8a03eb178355d3c8c5050e85bd217285c700df9b4506ec9e3680f8",
   "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
   "init_source": {
-    "init_from": "jaredpalmer/kev-4b@485ace8703592fcf405488b262449990824cfed1",
-    "resolved": "/__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--jaredpalmer--kev-4b/snapshots/485ace8703592fcf405488b262449990824cfed1",
-    "adapter_sha256": "9797de69a42188e411b17b7b4fcb66a23374dcebc21d71a7a66f836b5d34df2b",
-    "head_sha256": "d8f796da36ff7bd7c0fb9496b452139bb7851af4fc82b07b500b682d3f721d6a",
+    "init_from": "/runs/sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint",
+    "resolved": "/runs/sev-r2-curriculum-repair-4b-v1/02-trial-2/checkpoint",
+    "adapter_sha256": "0452010abdcc58f7c39a47a1562a1c812bf85f025d4a6c105c93d486051765de",
+    "head_sha256": "5a91ed82f11c6d0c8ab0a910e0c85770119e0034039379aa601d17a3ec7664d6",
     "adapter_tensors": 496
   },
   "ordinal_objective": "ranked_probability_score",
-  "holdout": [
-    "sev_behavioral_retry_export",
-    "sev_behavioral_stale_handoff",
-    "sev_behavioral_reconcile_export"
-  ]
+  "holdout": []
 }

training_metrics.json

@@ -1,14 +1,14 @@
 {
-  "wall_seconds": 371.20887875556946,
-  "records_seen": 2334,
-  "requested_records": 2334,
+  "wall_seconds": 1043.7473661899567,
+  "records_seen": 6438,
+  "requested_records": 6438,
   "truncated_records": 0,
   "rejected_records": 0,
-  "optimizer_steps": 292,
-  "forward_tokens": 738327,
-  "peak_device_bytes": 18913074688,
+  "optimizer_steps": 805,
+  "forward_tokens": 1447855,
+  "peak_device_bytes": 22227358208,
   "device": "cuda",
   "dtype": "bf16",
   "batch": 4,
-  "peak_rss_bytes": 31561670656
+  "peak_rss_bytes": 31557873664
 }