Sev-4B v0.3.1: policy false alerts 5/140, under the 5 percent ceiling

Browse files

Files changed (8) hide show

README.md +42 -47
adapter_model.safetensors +1 -1
head.pt +2 -2
provenance.json +10 -10
result.json +149 -2726
train.log +11 -85
training_config.json +10 -10
training_metrics.json +7 -7

README.md CHANGED Viewed

@@ -19,15 +19,18 @@ tags:
 # Sev-4B: security evidence and response policies
 Sev-4B scores supplied answers to questions about security logs and recovered
-programs. **v0.3.0-response-policy-research** adds explicit authorization-policy
-questions grounded in real SwarmTraces programs. It is a Qwen3.5-4B LoRA adapter
-with a decision head. It does not generate text.
-On 175 policy questions from 35 source programs held out of training, accuracy
-increases from **127/175, 72.6%, to 147/175, 84.0%** compared with the published
-v0.2.0 parent. The user selected this research release with its measured
-tradeoffs. **26 of 28 registered checks pass.** The false-alert ceiling and
-zero-new-DNS-error check fail. Those results are preserved unchanged.
 ## Use
@@ -39,7 +42,7 @@ git clone https://github.com/maceip/Sev.git
 cd Sev
 uv sync --extra serve
 uv run python -m kev.serve \
-  --run macmacmacmac/Sev-4B@v0.3.0-response-policy-research --port 8009
 ```
 Send TypeSafe-shaped requests to `POST /v1/systemone`. Supply the observed
@@ -47,8 +50,11 @@ evidence, the applicable policy, and explicit answer choices. The model reads
 supplied code without executing it. Its scores support analyst review; they do
 not establish that a program ran, identify its human owner, or prove agent origin.
-The checkpoint ships with temperature **1.6817928305074288**. Set
-`KEV_TEMPERATURE=1.0` for raw probabilities. Reported evaluation uses
 `KEV_BACKEND=torch KEV_DTYPE=fp32 KEV_MERGE=1` on CUDA. Accelerated serving can
 differ numerically. The API's `confidence` rescales the largest probability above
 chance; it is not an independently verified correctness probability.
@@ -78,8 +84,8 @@ policies and checked labels. Their source text is real recovered code, not a
 native execution trace. Source-closure groups separate training, calibration
 and development. No locked test was read.
-[Training configuration](training_config.json), [provenance](provenance.json),
-and [data lineage](DATA_PROVENANCE.md) pin the inputs. The original
 [`v0.1.0-research`](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.1.0-research)
 synthetic-origin preview is a separate historical lineage. The previous
 [`v0.2.0-swarmtraces-research`](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.2.0-swarmtraces-research)
@@ -87,32 +93,23 @@ release remains available unchanged.
 ## Matched development results
-| Policy decision | Published parent | This release |
 | --- | ---: | ---: |
-| Correct argmax answers | 127/175 | 147/175 |
-| Required approvals correct at argmax | 29/35 | 35/35 |
-| Permitted answers correct at argmax | 98/140 | 112/140 |
-| Required approvals detected at calibrated threshold | 14/35 | 25/35 |
-| False alerts at that threshold | 12/140 | 14/140 |
-| Calibrated negative log loss | 0.6260 | 0.3743 |
-| Calibrated Brier score | 0.4111 | 0.2394 |
-Each model's alert threshold was selected on calibration only, with an empirical
-5% false-alert budget. The parent threshold is 0.887537 and this release's is
-0.844982. Development false alerts increase from 8.6% to **10.0%**, failing the
-5% ceiling. Required-approval recall rises from 40.0% to 71.4%. These are explicit
 policy decisions on a development panel, not measured detection rates in live
-networks. Always predicting permitted would answer 140/175 correctly and detect
-none of the 35 required approvals.
-All eleven extra threshold detections concern `innerText` or `textContent`.
-Thirteen of the twenty additional correct argmax answers are console-negative
-questions. The panel lacks console-positive and code-execution-positive cases;
-it has one onward-request positive. HTML-policy negatives remain weak at 2/21
-correct argmax answers, and their threshold false alerts increase from 9/21 to
-12/21. The aggregate does not establish every operation's detection quality.
-| Retained panel | Published parent | This release |
 | --- | ---: | ---: |
 | Manual SwarmTraces | 65/83 | 66/83 |
 | Static SwarmTraces | 345/354 | 347/354 |
@@ -128,12 +125,10 @@ individual answers. GUIDE detector calibrated negative log loss worsens by
 0.0116 and Brier by 0.0077. No matched continuation without the new component was
 run, so the isolated causal contribution of SwarmTraces is not established.
-The separate DNS diagnostic remains **21/32**, with one repair and one new error.
-It contains eight authored summaries and controls inspired by one reported
-incident, not raw packet captures or eight independent incidents. It is
-evaluation-only. The new error fails the registered zero-regression criterion.
-[Policy evaluation](evaluation.json), [DNS evaluation](dns-evaluation.json), and
-[registered screen](screen.json) retain the complete comparisons.
 ## Calibration and artifact verification
@@ -145,20 +140,20 @@ validation. Development and DNS observations were excluded from fitting.
 Only temperature metadata changed when assembling the serving copy. Learned
 head tensors, adapter and tokenizer bytes match the evaluated checkpoint.
-[Calibration](calibration.json), [integrity evidence](checkpoint-integrity.json),
-and [SHA256SUMS](SHA256SUMS) identify the release files.
 Research iteration is paused following this release. The 0.8B and 9B checkpoints
 are not updated by this publication.
 ## License and attribution
-Source and adapter/head weights carry Apache-2.0 notices. Preserve [LICENSE](LICENSE),
-[NOTICE](NOTICE) and the Qwen [BASE_LICENSE](BASE_LICENSE). Sev builds on Kev by
 Jared Palmer and Qwen3.5 by the Qwen team.
 Each dataset retains its own terms. Upstream SwarmTraces reuse terms remain
 unverified; the model license does not relicense those artifacts. The collection
 includes metadata-only entries, and membership does not imply training use.
 The historical synthetic source's notice applies only to that source. See
-[data provenance](DATA_PROVENANCE.md).

 # Sev-4B: security evidence and response policies
 Sev-4B scores supplied answers to questions about security logs and recovered
+programs. **v0.3.1-html-contrast-research** continues
+`v0.3.0-response-policy-research`. It is a Qwen3.5-4B LoRA adapter with a
+decision head. It does not generate text.
+On 175 policy questions from 35 source programs held out of training, this
+checkpoint answers **158/175, 90.3%**, up from 147/175 on v0.3.0. At the
+calibration-selected alert cut it catches **35/35** required approvals and
+raises **5/140** false alerts, **3.6%**, under the 5% ceiling. v0.3.0 raised
+14/140 and failed that ceiling. Twelve of those fourteen were allowed HTML
+reads. This pass trains the contrast on the same program: writing `innerText`
+requires approval, and parsing HTML does not. The DNS diagnostic was not rerun.
+The trial's isolation-and-packing check failed. v0.3.0 remains available.
 ## Use
 cd Sev
 uv sync --extra serve
 uv run python -m kev.serve \
+  --run macmacmacmac/Sev-4B@v0.3.1-html-contrast-research --port 8009
 ```
 Send TypeSafe-shaped requests to `POST /v1/systemone`. Supply the observed
 supplied code without executing it. Its scores support analyst review; they do
 not establish that a program ran, identify its human owner, or prove agent origin.
+The checkpoint ships with temperature **1.0**. The false-alert cut for the
+policy panel is **1.0** on that raw approval probability: alert only when the
+approval option is certain. That cut was chosen on the 140 policy calibration
+questions, at a 5% false-alert budget, and then applied to development.
+Reported evaluation uses
 `KEV_BACKEND=torch KEV_DTYPE=fp32 KEV_MERGE=1` on CUDA. Accelerated serving can
 differ numerically. The API's `confidence` rescales the largest probability above
 chance; it is not an independently verified correctness probability.
 native execution trace. Source-closure groups separate training, calibration
 and development. No locked test was read.
+[Training configuration](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/training_config.json), [provenance](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/provenance.json),
+and [data lineage](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/DATA_PROVENANCE.md) pin the inputs. The original
 [`v0.1.0-research`](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.1.0-research)
 synthetic-origin preview is a separate historical lineage. The previous
 [`v0.2.0-swarmtraces-research`](https://huggingface.co/macmacmacmac/Sev-4B/tree/v0.2.0-swarmtraces-research)
 ## Matched development results
+| Policy decision | v0.3.0 | v0.3.1 |
 | --- | ---: | ---: |
+| Correct argmax answers | 147/175 | 158/175 |
+| Required approvals detected at the alert cut | 25/35 | 35/35 |
+| False alerts at that cut | 14/140 | 5/140 |
+| Allowed HTML questions flagged | 12/21 | 0/21 |
+Each alert cut was selected on calibration only, with a 5% false-alert budget.
+v0.3.1's cut is 1.0 on the raw approval probability. The 5 remaining false
+alerts are 4 console questions and 1 plain-text question. These are explicit
 policy decisions on a development panel, not measured detection rates in live
+networks.
+The retained-task table and the DNS result below were measured on v0.3.0.
+They were not rerun for v0.3.1.
+| Retained panel | v0.2.0 | v0.3.0, not rerun here |
 | --- | ---: | ---: |
 | Manual SwarmTraces | 65/83 | 66/83 |
 | Static SwarmTraces | 345/354 | 347/354 |
 0.0116 and Brier by 0.0077. No matched continuation without the new component was
 run, so the isolated causal contribution of SwarmTraces is not established.
+On v0.3.0 the DNS diagnostic was **21/32** and failed the zero-new-error check.
+v0.3.1 did not rerun it.
+[Policy evaluation](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/evaluation.json), [DNS evaluation](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/dns-evaluation.json), and
+[registered screen](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/screen.json) retain the complete comparisons.
 ## Calibration and artifact verification
 Only temperature metadata changed when assembling the serving copy. Learned
 head tensors, adapter and tokenizer bytes match the evaluated checkpoint.
+[Calibration](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/calibration.json), [integrity evidence](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/checkpoint-integrity.json),
+and [SHA256SUMS](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/SHA256SUMS) identify the release files.
 Research iteration is paused following this release. The 0.8B and 9B checkpoints
 are not updated by this publication.
 ## License and attribution
+Source and adapter/head weights carry Apache-2.0 notices. Preserve [LICENSE](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/LICENSE),
+[NOTICE](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/NOTICE) and the Qwen [BASE_LICENSE](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/BASE_LICENSE). Sev builds on Kev by
 Jared Palmer and Qwen3.5 by the Qwen team.
 Each dataset retains its own terms. Upstream SwarmTraces reuse terms remain
 unverified; the model license does not relicense those artifacts. The collection
 includes metadata-only entries, and membership does not imply training use.
 The historical synthetic source's notice applies only to that source. See
+[data provenance](https://huggingface.co/macmacmacmac/Sev-4B/blob/v0.3.0-response-policy-research/DATA_PROVENANCE.md).

adapter_model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fa1916b34fe1eeb10d15c36927c225df43115407d7c97b4f879bba3102c3ed21
 size 129924032

 version https://git-lfs.github.com/spec/v1
+oid sha256:b58b3963175e6c40bd022db5893ebefb6a6f1e1bd1ec11efc6692a089e1dcf51
 size 129924032

head.pt CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:12112262e1822fa4ab2981650bd10ceac46929bb29fbbdd843a39ec9de7dbe82
-size 5251455

 version https://git-lfs.github.com/spec/v1
+oid sha256:2724bebcc54d4846194130594b98d3e3b9c632935c0f1d5077e46e23c48b2dc8
+size 5248831

provenance.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
   "config": {
-    "epochs": 1,
     "seed": 4,
-    "lr": 2.5e-06,
     "lora": 16,
     "accum": 2,
     "batch": 4,
@@ -23,7 +23,7 @@
     "focal_gamma": 0.0,
     "base": "Qwen/Qwen3.5-4B-Base",
     "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
-    "init_from": "macmacmacmac/Sev-4B@a1824aefba305fda86e3503b895a7d9b3871f79a",
     "lora_targets": "all",
     "head_dim": 256,
     "weights_dtype": "fp32",
@@ -32,8 +32,8 @@
     "option_isolation": 0,
     "special_embeddings": 0
   },
-  "config_sha256": "01ce0c9f84977e5c32d40dbd5c012634208c7770615b9e7d9d6f5950c042706f",
-  "suite_sha256": "3b9a80dfd212b6932b66ea520c35ab1a3b19e132a81af1b6b7b306d776a70a02",
   "source_hashes": {
     "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
     "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
@@ -65,17 +65,17 @@
     "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
     "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
   },
-  "git_commit": "740b511bb562cf29430a85457b9e54389a3f21cd",
   "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
   "torch": "2.8.0+cu128",
   "device": "cuda",
   "gpu": "NVIDIA H100 80GB HBM3",
   "legacy_checkpoint": false,
   "measured_checkpoint": {
-    "requested": "/runs/sev-r2-response-policy-4b-v2/00-trial-0/checkpoint",
-    "resolved": "/runs/sev-r2-response-policy-4b-v2/00-trial-0/checkpoint",
-    "head_sha256": "506dcd3587847917a288d95757c497c15a2aef8e2d4bc6523b04ad991c961a2b",
-    "adapter_sha256": "fa1916b34fe1eeb10d15c36927c225df43115407d7c97b4f879bba3102c3ed21",
     "inference_temperature": 1.0
   }
 }

 {
   "config": {
+    "epochs": 5,
     "seed": 4,
+    "lr": 1e-05,
     "lora": 16,
     "accum": 2,
     "batch": 4,
     "focal_gamma": 0.0,
     "base": "Qwen/Qwen3.5-4B-Base",
     "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
+    "init_from": "macmacmacmac/Sev-4B@v0.3.0-response-policy-research",
     "lora_targets": "all",
     "head_dim": 256,
     "weights_dtype": "fp32",
     "option_isolation": 0,
     "special_embeddings": 0
   },
+  "config_sha256": "31d8cf5af214ae4ed0012d1e4fea9fd73e207209cfce1f1112107e4aad65cc9f",
+  "suite_sha256": "40d3a8b3bd416a3754ecc30e9791a9ebb14a26c8f2a4422057f769a2dbce4dee",
   "source_hashes": {
     "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
     "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
     "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
     "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
   },
+  "git_commit": "fcd6756a329f42a515b4b48f83fb8bde9776232f",
   "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
   "torch": "2.8.0+cu128",
   "device": "cuda",
   "gpu": "NVIDIA H100 80GB HBM3",
   "legacy_checkpoint": false,
   "measured_checkpoint": {
+    "requested": "/runs/sev-html-contrast-4b-v1/00-trial-0/checkpoint",
+    "resolved": "/runs/sev-html-contrast-4b-v1/00-trial-0/checkpoint",
+    "head_sha256": "2724bebcc54d4846194130594b98d3e3b9c632935c0f1d5077e46e23c48b2dc8",
+    "adapter_sha256": "b58b3963175e6c40bd022db5893ebefb6a6f1e1bd1ec11efc6692a089e1dcf51",
     "inference_temperature": 1.0
   }
 }

result.json CHANGED Viewed

@@ -1,682 +1,142 @@
 {
-  "objective": -0.17643950559609828,
   "paired_flip": null,
   "unknowable": null,
   "clean": {
-    "n": 2632,
-    "nll": 0.4626010690091542,
-    "acc": 0.7868541033434651,
-    "ece": 0.0822959829967011,
-    "brier": 0.2695405506463609,
-    "mean_conf": 0.8650301490049586,
-    "confident_error_rate": 0.014437689969604863,
-    "coverage_at_0_9": 0.6565349544072948,
-    "accuracy_at_0_9": 0.9780092592592593,
-    "coverage_at_5pct_error": 0.6808510638297872,
-    "coverage_at_1pct_error": 0.5828267477203647,
-    "aurc": 0.04539486327850754,
-    "error_rate_at_0_9": 0.02199074074074074,
-    "confidence_bias": 0.0781760456614935,
     "top_bins": {
       "0.9": {
-        "n": 1728,
-        "errors": 38,
-        "error_rate": 0.02199074074074074
       },
       "0.95": {
-        "n": 1675,
-        "errors": 28,
-        "error_rate": 0.01671641791044776
       },
       "0.99": {
-        "n": 1519,
-        "errors": 15,
-        "error_rate": 0.009874917709019092
       }
     },
     "selective": {
       "0.5": {
-        "coverage": 0.5155775075987842,
-        "accuracy": 0.9918938835666913,
-        "confidence_cutoff": 0.9953955617749282
       },
       "0.8": {
-        "coverage": 0.8028115501519757,
-        "accuracy": 0.8920965451964032,
-        "confidence_cutoff": 0.658356020724118
       }
     }
   },
   "tasks": {
-    "excytin_file": {
-      "n": 80,
-      "nll": 8.416562133899781e-05,
-      "acc": 1.0,
-      "ece": 8.407504508700647e-05,
-      "brier": 3.608306194877327e-07,
-      "mean_conf": 0.999915924954913,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 1.0,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -8.407504508700647e-05,
-      "top_bins": {
-        "0.9": {
-          "n": 80,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 80,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 80,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9999870465451609
-        },
-        "0.8": {
-          "coverage": 0.8,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9999445146924221
-        }
-      }
-    },
-    "excytin_parent_join": {
-      "n": 78,
-      "nll": 0.0010931532029286512,
-      "acc": 1.0,
-      "ece": 0.0010890559164937752,
-      "brier": 1.524968968159761e-05,
-      "mean_conf": 0.9989109440835062,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 1.0,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.0010890559164937752,
-      "top_bins": {
-        "0.9": {
-          "n": 78,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 78,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 77,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9997808486535813
-        },
-        "0.8": {
-          "coverage": 0.8076923076923077,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9985521003494622
-        }
-      }
-    },
-    "excytin_process": {
-      "n": 120,
-      "nll": 0.00038924968205708544,
-      "acc": 1.0,
-      "ece": 0.00038892368944332123,
-      "brier": 1.2813112890567744e-06,
-      "mean_conf": 0.9996110763105567,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 1.0,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.00038892368944332123,
-      "top_bins": {
-        "0.9": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9999547557532524
-        },
-        "0.8": {
-          "coverage": 0.8,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9992749850755753
-        }
-      }
-    },
-    "excytin_signin": {
-      "n": 120,
-      "nll": 0.0009466221977245799,
-      "acc": 1.0,
-      "ece": 0.0009222842581878554,
-      "brier": 7.013154556645113e-05,
-      "mean_conf": 0.9990777157418121,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 1.0,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.0009222842581878554,
-      "top_bins": {
-        "0.9": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 119,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 119,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9999016374480455
-        },
-        "0.8": {
-          "coverage": 0.8,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9997216681533668
-        }
-      }
-    },
-    "sev_guide_detector": {
-      "n": 1064,
-      "nll": 0.8605571101399087,
-      "acc": 0.6005639097744361,
-      "ece": 0.14734385206430384,
-      "brier": 0.5008109312564021,
-      "mean_conf": 0.7382136284198534,
-      "confident_error_rate": 0.017857142857142856,
-      "coverage_at_0_9": 0.33270676691729323,
-      "accuracy_at_0_9": 0.9463276836158192,
-      "coverage_at_5pct_error": 0.3082706766917293,
-      "coverage_at_1pct_error": 0.0,
-      "aurc": 0.19350209200158913,
-      "error_rate_at_0_9": 0.05367231638418079,
-      "confidence_bias": 0.1376497186454173,
-      "top_bins": {
-        "0.9": {
-          "n": 354,
-          "errors": 19,
-          "error_rate": 0.05367231638418079
-        },
-        "0.95": {
-          "n": 327,
-          "errors": 13,
-          "error_rate": 0.039755351681957186
-        },
-        "0.99": {
-          "n": 319,
-          "errors": 12,
-          "error_rate": 0.03761755485893417
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5028195488721805,
-          "accuracy": 0.8205607476635514,
-          "confidence_cutoff": 0.7059991149306561
-        },
-        "0.8": {
-          "coverage": 0.8035714285714286,
-          "accuracy": 0.6596491228070176,
-          "confidence_cutoff": 0.5473316703985754
-        }
-      }
-    },
-    "sev_guide_triage": {
-      "n": 231,
-      "nll": 0.8396428451107271,
-      "acc": 0.5670995670995671,
-      "ece": 0.1889188281653689,
-      "brier": 0.5162927524049745,
-      "mean_conf": 0.7421494516023849,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.3593073593073593,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 0.3722943722943723,
-      "coverage_at_1pct_error": 0.36363636363636365,
-      "aurc": 0.20987091476964936,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": 0.17504988450281778,
-      "top_bins": {
-        "0.9": {
-          "n": 83,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 80,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 14,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5627705627705628,
-          "accuracy": 0.7153846153846154,
-          "confidence_cutoff": 0.7564145922660828
-        },
-        "0.8": {
-          "coverage": 0.8051948051948052,
-          "accuracy": 0.5806451612903226,
-          "confidence_cutoff": 0.51023405030021
-        }
-      }
-    },
     "sev_swarmtraces_response_policy": {
       "n": 175,
-      "nll": 0.49844721765472333,
-      "acc": 0.84,
-      "ece": 0.1173563845984423,
-      "brier": 0.26967228692055817,
-      "mean_conf": 0.955988843320762,
-      "confident_error_rate": 0.10285714285714286,
-      "coverage_at_0_9": 0.8742857142857143,
-      "accuracy_at_0_9": 0.8823529411764706,
-      "coverage_at_5pct_error": 0.6114285714285714,
-      "coverage_at_1pct_error": 0.44,
-      "aurc": 0.044632779737602504,
-      "error_rate_at_0_9": 0.11764705882352941,
-      "confidence_bias": 0.11598884332076198,
       "top_bins": {
         "0.9": {
-          "n": 153,
-          "errors": 18,
-          "error_rate": 0.11764705882352941
         },
         "0.95": {
-          "n": 140,
           "errors": 14,
-          "error_rate": 0.1
         },
         "0.99": {
-          "n": 92,
-          "errors": 3,
-          "error_rate": 0.03260869565217391
         }
       },
       "selective": {
         "0.5": {
           "coverage": 0.5028571428571429,
-          "accuracy": 0.9772727272727273,
-          "confidence_cutoff": 0.9904485381783863
         },
         "0.8": {
           "coverage": 0.8,
-          "accuracy": 0.9,
-          "confidence_cutoff": 0.9516673045977523
-        }
-      }
-    },
-    "sev_swarmtraces_static": {
-      "n": 354,
-      "nll": 0.04944532768977841,
-      "acc": 0.980225988700565,
-      "ece": 0.01487544035693287,
-      "brier": 0.024608535342050798,
-      "mean_conf": 0.9802643779708107,
-      "confident_error_rate": 0.002824858757062147,
-      "coverage_at_0_9": 0.9463276836158192,
-      "accuracy_at_0_9": 0.9970149253731343,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 0.9858757062146892,
-      "aurc": 0.0005032049402250012,
-      "error_rate_at_0_9": 0.0029850746268656717,
-      "confidence_bias": 3.8389270245686724e-05,
-      "top_bins": {
-        "0.9": {
-          "n": 335,
-          "errors": 1,
-          "error_rate": 0.0029850746268656717
-        },
-        "0.95": {
-          "n": 330,
-          "errors": 1,
-          "error_rate": 0.0030303030303030303
-        },
-        "0.99": {
-          "n": 308,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9999276209511099
-        },
-        "0.8": {
-          "coverage": 0.8022598870056498,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.997796467038603
-        }
-      }
-    },
-    "sev_sysmon_native_file": {
-      "n": 62,
-      "nll": 0.008428780795422771,
-      "acc": 1.0,
-      "ece": 0.007810434776248104,
-      "brier": 0.0017443201233063612,
-      "mean_conf": 0.9921895652237519,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.9838709677419355,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.0078104347762480986,
-      "top_bins": {
-        "0.9": {
-          "n": 61,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 59,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 57,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9991686260052061
-        },
-        "0.8": {
-          "coverage": 0.8064516129032258,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9962482122843107
-        }
-      }
-    },
-    "sev_sysmon_native_join": {
-      "n": 120,
-      "nll": 0.00039767294618876087,
-      "acc": 1.0,
-      "ece": 0.0003974740966455892,
-      "brier": 7.310901229709924e-07,
-      "mean_conf": 0.9996025259033544,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 1.0,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.0003974740966455892,
-      "top_bins": {
-        "0.9": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 120,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9997660582574378
-        },
-        "0.8": {
-          "coverage": 0.8,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9994606035447323
-        }
-      }
-    },
-    "sev_sysmon_native_other_effect": {
-      "n": 88,
-      "nll": 0.005466714567741097,
-      "acc": 1.0,
-      "ece": 0.005048372154544871,
-      "brier": 0.0014126265728775101,
-      "mean_conf": 0.994951627845455,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.9886363636363636,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.005048372154545011,
-      "top_bins": {
-        "0.9": {
-          "n": 87,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 86,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 83,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9997677646417501
-        },
-        "0.8": {
-          "coverage": 0.8068181818181818,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9978176224669177
-        }
-      }
-    },
-    "sev_sysmon_native_process": {
-      "n": 74,
-      "nll": 0.004798899209615666,
-      "acc": 1.0,
-      "ece": 0.004644321057477101,
-      "brier": 0.0005716684001551928,
-      "mean_conf": 0.995355678942523,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.9864864864864865,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 1.0,
-      "aurc": 0.0,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": -0.00464432105747703,
-      "top_bins": {
-        "0.9": {
-          "n": 73,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 72,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 68,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9999268900168323
-        },
-        "0.8": {
-          "coverage": 0.8108108108108109,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9973453419434639
-        }
-      }
-    },
-    "sev_sysmon_native_registry": {
-      "n": 66,
-      "nll": 0.024015813931122083,
-      "acc": 0.9848484848484849,
-      "ece": 0.016913465896023366,
-      "brier": 0.01690246747159733,
-      "mean_conf": 0.9866786138448916,
-      "confident_error_rate": 0.0,
-      "coverage_at_0_9": 0.9696969696969697,
-      "accuracy_at_0_9": 1.0,
-      "coverage_at_5pct_error": 1.0,
-      "coverage_at_1pct_error": 0.9696969696969697,
-      "aurc": 0.00046266864448682633,
-      "error_rate_at_0_9": 0.0,
-      "confidence_bias": 0.0018301289964067768,
-      "top_bins": {
-        "0.9": {
-          "n": 64,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.95": {
-          "n": 64,
-          "errors": 0,
-          "error_rate": 0.0
-        },
-        "0.99": {
-          "n": 62,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9998656142719909
-        },
-        "0.8": {
-          "coverage": 0.803030303030303,
-          "accuracy": 1.0,
-          "confidence_cutoff": 0.9982153954384291
         }
       }
     }
   },
   "variants": {
     "clean": {
-      "n": 2632,
-      "nll": 0.4626010690091542,
-      "acc": 0.7868541033434651,
-      "ece": 0.0822959829967011,
-      "brier": 0.2695405506463609,
-      "mean_conf": 0.8650301490049586,
-      "confident_error_rate": 0.014437689969604863,
-      "coverage_at_0_9": 0.6565349544072948,
-      "accuracy_at_0_9": 0.9780092592592593,
-      "coverage_at_5pct_error": 0.6808510638297872,
-      "coverage_at_1pct_error": 0.5828267477203647,
-      "aurc": 0.04539486327850754,
-      "error_rate_at_0_9": 0.02199074074074074,
-      "confidence_bias": 0.0781760456614935,
       "top_bins": {
         "0.9": {
-          "n": 1728,
-          "errors": 38,
-          "error_rate": 0.02199074074074074
         },
         "0.95": {
-          "n": 1675,
-          "errors": 28,
-          "error_rate": 0.01671641791044776
         },
         "0.99": {
-          "n": 1519,
-          "errors": 15,
-          "error_rate": 0.009874917709019092
         }
       },
       "selective": {
         "0.5": {
-          "coverage": 0.5155775075987842,
-          "accuracy": 0.9918938835666913,
-          "confidence_cutoff": 0.9953955617749282
         },
         "0.8": {
-          "coverage": 0.8028115501519757,
-          "accuracy": 0.8920965451964032,
-          "confidence_cutoff": 0.658356020724118
         }
       }
     }
@@ -687,49 +147,49 @@
     "mean_max_delta": null,
     "flip_rate": null
   },
-  "temperature": 1.6817928305074288,
   "calibrated_clean": {
-    "n": 2632,
-    "nll": 0.42892992382965894,
-    "acc": 0.7868541033434651,
-    "ece": 0.04404789769068498,
-    "brier": 0.25086130169182097,
-    "mean_conf": 0.8094973007371603,
-    "confident_error_rate": 0.007978723404255319,
-    "coverage_at_0_9": 0.6196808510638298,
-    "accuracy_at_0_9": 0.9871244635193133,
-    "coverage_at_5pct_error": 0.6918693009118541,
-    "coverage_at_1pct_error": 0.5395136778115501,
-    "aurc": 0.045402363700726256,
-    "error_rate_at_0_9": 0.012875536480686695,
-    "confidence_bias": 0.022643197393695247,
     "top_bins": {
       "0.9": {
-        "n": 1631,
-        "errors": 21,
-        "error_rate": 0.012875536480686695
       },
       "0.95": {
-        "n": 1153,
-        "errors": 0,
-        "error_rate": 0.0
       },
       "0.99": {
-        "n": 776,
         "errors": 0,
-        "error_rate": 0.0
       }
     },
     "selective": {
       "0.5": {
-        "coverage": 0.5208966565349544,
-        "accuracy": 0.9912472647702407,
-        "confidence_cutoff": 0.9486799539412789
       },
       "0.8": {
-        "coverage": 0.8111702127659575,
-        "accuracy": 0.8861826697892272,
-        "confidence_cutoff": 0.52766162421842
       }
     }
   },
@@ -747,2062 +207,28 @@
     "returned_zeros": 0
   },
   "coverage": {
-    "requested_records": 1828,
-    "requested_questions": 2632,
-    "evaluated_records": 1828,
-    "evaluated_questions": 2632,
     "rejected_records": 0,
     "truncated_records": 0
   },
   "latency_ms": {
-    "median": 54.25747899994349,
-    "p95": 161.14951469993457
   },
   "calibration": {
     "inference_temperature": 1.0,
-    "additional_temperature": 1.6817928305074288,
     "logits_recorded": true
   },
-  "transfer": {
-    "objective": -0.4988755241804635,
-    "paired_flip": {
-      "pairs": 16,
-      "flip_rate": 0.9375,
-      "both_correct_rate": 0.9375,
-      "invariant_pairs": 8,
-      "invariance_rate": 1.0,
-      "invariant_both_correct_rate": 0.875
-    },
-    "unknowable": null,
-    "clean": {
-      "n": 243,
-      "nll": 0.6065943824685833,
-      "acc": 0.8683127572016461,
-      "ece": 0.09195653550207036,
-      "brier": 0.23499321102178952,
-      "mean_conf": 0.9525644631029061,
-      "confident_error_rate": 0.09053497942386832,
-      "coverage_at_0_9": 0.8600823045267489,
-      "accuracy_at_0_9": 0.8947368421052632,
-      "coverage_at_5pct_error": 0.6872427983539094,
-      "coverage_at_1pct_error": 0.42386831275720166,
-      "aurc": 0.034793809228677,
-      "error_rate_at_0_9": 0.10526315789473684,
-      "confidence_bias": 0.08425170590125997,
-      "top_bins": {
-        "0.9": {
-          "n": 209,
-          "errors": 22,
-          "error_rate": 0.10526315789473684
-        },
-        "0.95": {
-          "n": 193,
-          "errors": 13,
-          "error_rate": 0.06735751295336788
-        },
-        "0.99": {
-          "n": 154,
-          "errors": 4,
-          "error_rate": 0.025974025974025976
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5020576131687243,
-          "accuracy": 0.9836065573770492,
-          "confidence_cutoff": 0.9985267483650163
-        },
-        "0.8": {
-          "coverage": 0.8024691358024691,
-          "accuracy": 0.9230769230769231,
-          "confidence_cutoff": 0.946113674977154
-        }
-      }
-    },
-    "tasks": {
-      "composition_held_and_or": {
-        "n": 32,
-        "nll": 0.24171629359001318,
-        "acc": 0.90625,
-        "ece": 0.08267706695037225,
-        "brier": 0.1580607438572586,
-        "mean_conf": 0.9889270669503722,
-        "confident_error_rate": 0.09375,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 0.90625,
-        "coverage_at_5pct_error": 0.9375,
-        "coverage_at_1pct_error": 0.90625,
-        "aurc": 0.0069010416666666664,
-        "error_rate_at_0_9": 0.09375,
-        "confidence_bias": 0.08267706695037225,
-        "top_bins": {
-          "0.9": {
-            "n": 32,
-            "errors": 3,
-            "error_rate": 0.09375
-          },
-          "0.95": {
-            "n": 30,
-            "errors": 1,
-            "error_rate": 0.03333333333333333
-          },
-          "0.99": {
-            "n": 26,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999364856833449
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9996934954887139
-          }
-        }
-      },
-      "contrastive_authorization": {
-        "n": 16,
-        "nll": 1.502653141036028e-05,
-        "acc": 1.0,
-        "ece": 1.50264042420023e-05,
-        "brier": 5.139828295030656e-10,
-        "mean_conf": 0.999984973595758,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -1.50264042420023e-05,
-        "top_bins": {
-          "0.9": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999862890472545
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999813198967473
-          }
-        }
-      },
-      "emotion": {
-        "n": 8,
-        "nll": 1.4194273414243654,
-        "acc": 0.625,
-        "ece": 0.3420785503418361,
-        "brier": 0.6845960520300963,
-        "mean_conf": 0.967078550341836,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 0.875,
-        "accuracy_at_0_9": 0.7142857142857143,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.12842261904761904,
-        "error_rate_at_0_9": 0.2857142857142857,
-        "confidence_bias": 0.342078550341836,
-        "top_bins": {
-          "0.9": {
-            "n": 7,
-            "errors": 2,
-            "error_rate": 0.2857142857142857
-          },
-          "0.95": {
-            "n": 5,
-            "errors": 1,
-            "error_rate": 0.2
-          },
-          "0.99": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9869563807629563
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.7142857142857143,
-            "confidence_cutoff": 0.936063568486461
-          }
-        }
-      },
-      "mmlu": {
-        "n": 8,
-        "nll": 1.3129749769552814,
-        "acc": 0.625,
-        "ece": 0.15765322690405312,
-        "brier": 0.6484351711065982,
-        "mean_conf": 0.7567759888850003,
-        "confident_error_rate": 0.125,
-        "coverage_at_0_9": 0.5,
-        "accuracy_at_0_9": 0.75,
-        "coverage_at_5pct_error": 0.375,
-        "coverage_at_1pct_error": 0.375,
-        "aurc": 0.22336309523809525,
-        "error_rate_at_0_9": 0.25,
-        "confidence_bias": 0.1317759888850003,
-        "top_bins": {
-          "0.9": {
-            "n": 4,
-            "errors": 1,
-            "error_rate": 0.25
-          },
-          "0.95": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.75,
-            "confidence_cutoff": 0.9327223999584052
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.5714285714285714,
-            "confidence_cutoff": 0.485476400811401
-          }
-        }
-      },
-      "otrf_sysmon_file": {
-        "n": 16,
-        "nll": 0.008605175298168915,
-        "acc": 1.0,
-        "ece": 0.008437849080092708,
-        "brier": 0.0005211762382657875,
-        "mean_conf": 0.9915621509199073,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.008437849080092708,
-        "top_bins": {
-          "0.9": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 15,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 13,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9989837483137961
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9925042879134804
-          }
-        }
-      },
-      "otrf_sysmon_join": {
-        "n": 16,
-        "nll": 0.12967885980971613,
-        "acc": 0.9375,
-        "ece": 0.0857233928404766,
-        "brier": 0.08146985002971384,
-        "mean_conf": 0.946506108546294,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.6875,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.9375,
-        "coverage_at_1pct_error": 0.9375,
-        "aurc": 0.00390625,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.009006108546294023,
-        "top_bins": {
-          "0.9": {
-            "n": 11,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 11,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 9,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9970209747654696
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.8702734279589676
-          }
-        }
-      },
-      "otrf_sysmon_process": {
-        "n": 24,
-        "nll": 0.16156109121321785,
-        "acc": 0.9583333333333334,
-        "ece": 0.05273169620250518,
-        "brier": 0.0899182770612567,
-        "mean_conf": 0.9768730534919899,
-        "confident_error_rate": 0.041666666666666664,
-        "coverage_at_0_9": 0.9583333333333334,
-        "accuracy_at_0_9": 0.9565217391304348,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 0.9166666666666666,
-        "aurc": 0.0035477053140096615,
-        "error_rate_at_0_9": 0.043478260869565216,
-        "confidence_bias": 0.01853972015865657,
-        "top_bins": {
-          "0.9": {
-            "n": 23,
-            "errors": 1,
-            "error_rate": 0.043478260869565216
-          },
-          "0.95": {
-            "n": 22,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 19,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9994396085787187
-          },
-          "0.8": {
-            "coverage": 0.8333333333333334,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9832554759400715
-          }
-        }
-      },
-      "otrf_sysmon_registry": {
-        "n": 8,
-        "nll": 0.010723606364118954,
-        "acc": 1.0,
-        "ece": 0.010542385937563847,
-        "brier": 0.0006959554565011166,
-        "mean_conf": 0.9894576140624362,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.010542385937563847,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 5,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9972651302471073
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9840692701748682
-          }
-        }
-      },
-      "paws": {
-        "n": 8,
-        "nll": 1.0368545966100273,
-        "acc": 0.75,
-        "ece": 0.22498828525853157,
-        "brier": 0.34163600039007924,
-        "mean_conf": 0.9122532529823204,
-        "confident_error_rate": 0.125,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 0.8333333333333334,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.09494047619047619,
-        "error_rate_at_0_9": 0.16666666666666666,
-        "confidence_bias": 0.1622532529823204,
-        "top_bins": {
-          "0.9": {
-            "n": 6,
-            "errors": 1,
-            "error_rate": 0.16666666666666666
-          },
-          "0.95": {
-            "n": 6,
-            "errors": 1,
-            "error_rate": 0.16666666666666666
-          },
-          "0.99": {
-            "n": 6,
-            "errors": 1,
-            "error_rate": 0.16666666666666666
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9992691391151949
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.8571428571428571,
-            "confidence_cutoff": 0.749059870895155
-          }
-        }
-      },
-      "qnli": {
-        "n": 8,
-        "nll": 0.005181466224742151,
-        "acc": 1.0,
-        "ece": 0.005129334884136472,
-        "brier": 0.00020524982613720548,
-        "mean_conf": 0.9948706651158635,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.005129334884136472,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 6,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.999952371295075
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.98872862957213
-          }
-        }
-      },
-      "sciq": {
-        "n": 8,
-        "nll": 0.00023928943016782004,
-        "acc": 1.0,
-        "ece": 0.0002391650142747892,
-        "brier": 4.955209174865095e-07,
-        "mean_conf": 0.9997608349857252,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.0002391650142747892,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999911919978735
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9994259298933852
-          }
-        }
-      },
-      "sev_swarmtraces_manual_credential": {
-        "n": 40,
-        "nll": 0.9530052125004369,
-        "acc": 0.825,
-        "ece": 0.17316335764162732,
-        "brier": 0.3580423587545788,
-        "mean_conf": 0.9202306277592299,
-        "confident_error_rate": 0.15,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 0.8,
-        "coverage_at_5pct_error": 0.225,
-        "coverage_at_1pct_error": 0.225,
-        "aurc": 0.10718612602417658,
-        "error_rate_at_0_9": 0.2,
-        "confidence_bias": 0.09523062775922997,
-        "top_bins": {
-          "0.9": {
-            "n": 30,
-            "errors": 6,
-            "error_rate": 0.2
-          },
-          "0.95": {
-            "n": 26,
-            "errors": 3,
-            "error_rate": 0.11538461538461539
-          },
-          "0.99": {
-            "n": 15,
-            "errors": 1,
-            "error_rate": 0.06666666666666667
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.9,
-            "confidence_cutoff": 0.9774245184450835
-          },
-          "0.8": {
-            "coverage": 0.8,
-            "accuracy": 0.8125,
-            "confidence_cutoff": 0.8542392185801473
-          }
-        }
-      },
-      "sev_swarmtraces_manual_flow": {
-        "n": 40,
-        "nll": 1.485896746743676,
-        "acc": 0.75,
-        "ece": 0.21977756775377905,
-        "brier": 0.43299049842888415,
-        "mean_conf": 0.9214803040956191,
-        "confident_error_rate": 0.15,
-        "coverage_at_0_9": 0.725,
-        "accuracy_at_0_9": 0.7931034482758621,
-        "coverage_at_5pct_error": 0.1,
-        "coverage_at_1pct_error": 0.1,
-        "aurc": 0.14190456225170725,
-        "error_rate_at_0_9": 0.20689655172413793,
-        "confidence_bias": 0.1714803040956191,
-        "top_bins": {
-          "0.9": {
-            "n": 29,
-            "errors": 6,
-            "error_rate": 0.20689655172413793
-          },
-          "0.95": {
-            "n": 27,
-            "errors": 6,
-            "error_rate": 0.2222222222222222
-          },
-          "0.99": {
-            "n": 21,
-            "errors": 2,
-            "error_rate": 0.09523809523809523
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.9,
-            "confidence_cutoff": 0.9901066734665328
-          },
-          "0.8": {
-            "coverage": 0.8,
-            "accuracy": 0.8125,
-            "confidence_cutoff": 0.8291658959545783
-          }
-        }
-      },
-      "sev_swarmtraces_manual_prerequisite": {
-        "n": 3,
-        "nll": 1.0331257533433365e-05,
-        "acc": 1.0,
-        "ece": 1.0331183196821137e-05,
-        "brier": 2.5195789327524225e-10,
-        "mean_conf": 0.9999896688168032,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -1.0331183196821137e-05,
-        "top_bins": {
-          "0.9": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.6666666666666666,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999897312178962
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999810314632929
-          }
-        }
-      },
-      "tweet_offensive": {
-        "n": 8,
-        "nll": 0.7172428487540772,
-        "acc": 0.75,
-        "ece": 0.2179639808956667,
-        "brier": 0.44120571426799937,
-        "mean_conf": 0.9679639808956667,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 0.75,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.09494047619047619,
-        "error_rate_at_0_9": 0.25,
-        "confidence_bias": 0.2179639808956667,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 2,
-            "error_rate": 0.25
-          },
-          "0.95": {
-            "n": 5,
-            "errors": 1,
-            "error_rate": 0.2
-          },
-          "0.99": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9897168920827885
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.8571428571428571,
-            "confidence_cutoff": 0.940377631626033
-          }
-        }
-      }
-    },
-    "variants": {
-      "authorized_dns_diagnostic": {
-        "n": 4,
-        "nll": 0.2732991829356235,
-        "acc": 0.75,
-        "ece": 0.2014060191596143,
-        "brier": 0.1826209474615037,
-        "mean_conf": 0.8245034642731555,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.5,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.75,
-        "coverage_at_1pct_error": 0.75,
-        "aurc": 0.0625,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.07450346427315546,
-        "top_bins": {
-          "0.9": {
-            "n": 2,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 2,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 2,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9941628768025105
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.75,
-            "confidence_cutoff": 0.5518189668655396
-          }
-        }
-      },
-      "cache_denied_no_reply": {
-        "n": 4,
-        "nll": 0.6290998127898555,
-        "acc": 0.5,
-        "ece": 0.19783432109801047,
-        "brier": 0.45073491547326267,
-        "mean_conf": 0.6842140294824197,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.25,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.25,
-        "coverage_at_1pct_error": 0.25,
-        "aurc": 0.41666666666666663,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.18421402948241972,
-        "top_bins": {
-          "0.9": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 0,
-            "errors": 0,
-            "error_rate": null
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.5,
-            "confidence_cutoff": 0.5997726159351691
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.5,
-            "confidence_cutoff": 0.5818420334307344
-          }
-        }
-      },
-      "clean": {
-        "n": 243,
-        "nll": 0.6065943824685833,
-        "acc": 0.8683127572016461,
-        "ece": 0.09195653550207036,
-        "brier": 0.23499321102178952,
-        "mean_conf": 0.9525644631029061,
-        "confident_error_rate": 0.09053497942386832,
-        "coverage_at_0_9": 0.8600823045267489,
-        "accuracy_at_0_9": 0.8947368421052632,
-        "coverage_at_5pct_error": 0.6872427983539094,
-        "coverage_at_1pct_error": 0.42386831275720166,
-        "aurc": 0.034793809228677,
-        "error_rate_at_0_9": 0.10526315789473684,
-        "confidence_bias": 0.08425170590125997,
-        "top_bins": {
-          "0.9": {
-            "n": 209,
-            "errors": 22,
-            "error_rate": 0.10526315789473684
-          },
-          "0.95": {
-            "n": 193,
-            "errors": 13,
-            "error_rate": 0.06735751295336788
-          },
-          "0.99": {
-            "n": 154,
-            "errors": 4,
-            "error_rate": 0.025974025974025976
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5020576131687243,
-            "accuracy": 0.9836065573770492,
-            "confidence_cutoff": 0.9985267483650163
-          },
-          "0.8": {
-            "coverage": 0.8024691358024691,
-            "accuracy": 0.9230769230769231,
-            "confidence_cutoff": 0.946113674977154
-          }
-        }
-      },
-      "dns_off_task_reply": {
-        "n": 4,
-        "nll": 2.500105858228899,
-        "acc": 0.25,
-        "ece": 0.7354215562262277,
-        "brier": 1.1856881009881057,
-        "mean_conf": 0.7180991063824902,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 0.25,
-        "accuracy_at_0_9": 0.0,
-        "coverage_at_5pct_error": 0.0,
-        "coverage_at_1pct_error": 0.0,
-        "aurc": 0.9375,
-        "error_rate_at_0_9": 1.0,
-        "confidence_bias": 0.4680991063824902,
-        "top_bins": {
-          "0.9": {
-            "n": 1,
-            "errors": 1,
-            "error_rate": 1.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 1,
-            "error_rate": 1.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 1,
-            "error_rate": 1.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.0,
-            "confidence_cutoff": 0.7394568794916744
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.25,
-            "confidence_cutoff": 0.465355100312525
-          }
-        }
-      },
-      "dns_unhelpful_reply": {
-        "n": 4,
-        "nll": 0.6130712887407269,
-        "acc": 0.75,
-        "ece": 0.2694959031277442,
-        "brier": 0.34519095756377294,
-        "mean_conf": 0.7624970557751488,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.5,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.75,
-        "coverage_at_1pct_error": 0.75,
-        "aurc": 0.0625,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.012497055775148813,
-        "top_bins": {
-          "0.9": {
-            "n": 2,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 0,
-            "errors": 0,
-            "error_rate": null
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9232891824527384
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.75,
-            "confidence_cutoff": 0.563985917805786
-          }
-        }
-      },
-      "dns_without_reply": {
-        "n": 4,
-        "nll": 0.6878185272523656,
-        "acc": 0.5,
-        "ece": 0.20221082782282854,
-        "brier": 0.5043315891024525,
-        "mean_conf": 0.7000800429377261,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.25,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.25,
-        "coverage_at_1pct_error": 0.25,
-        "aurc": 0.41666666666666663,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.20008004293772608,
-        "top_bins": {
-          "0.9": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.5,
-            "confidence_cutoff": 0.6943924514728984
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.5,
-            "confidence_cutoff": 0.5490827128017757
-          }
-        }
-      },
-      "none_absent": {
-        "n": 3,
-        "nll": 1.3777335821359413,
-        "acc": 0.6666666666666666,
-        "ece": 0.24885046106665176,
-        "brier": 0.5885968900899531,
-        "mean_conf": 0.9151737396988762,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.3333333333333333,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.6666666666666666,
-        "coverage_at_1pct_error": 0.6666666666666666,
-        "aurc": 0.1111111111111111,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.24850707303220954,
-        "top_bins": {
-          "0.9": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.6666666666666666,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.8739663040318306
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.6666666666666666,
-            "confidence_cutoff": 0.8720699971164613
-          }
-        }
-      },
-      "none_present": {
-        "n": 3,
-        "nll": 3.3971590445247757,
-        "acc": 0.0,
-        "ece": 0.806803931885096,
-        "brier": 1.589050038207996,
-        "mean_conf": 0.8068039318850961,
-        "confident_error_rate": 0.3333333333333333,
-        "coverage_at_0_9": 0.3333333333333333,
-        "accuracy_at_0_9": 0.0,
-        "coverage_at_5pct_error": 0.0,
-        "coverage_at_1pct_error": 0.0,
-        "aurc": 1.0,
-        "error_rate_at_0_9": 1.0,
-        "confidence_bias": 0.8068039318850961,
-        "top_bins": {
-          "0.9": {
-            "n": 1,
-            "errors": 1,
-            "error_rate": 1.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 1,
-            "error_rate": 1.0
-          },
-          "0.99": {
-            "n": 0,
-            "errors": 0,
-            "error_rate": null
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.6666666666666666,
-            "accuracy": 0.0,
-            "confidence_cutoff": 0.8592945698759779
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.0,
-            "confidence_cutoff": 0.5949657569957464
-          }
-        }
-      },
-      "operator_terminated": {
-        "n": 4,
-        "nll": 1.0359158594833722,
-        "acc": 0.75,
-        "ece": 0.2925896655492872,
-        "brier": 0.4933779585120464,
-        "mean_conf": 0.9106235817198012,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 0.6666666666666666,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.14583333333333331,
-        "error_rate_at_0_9": 0.3333333333333333,
-        "confidence_bias": 0.16062358171980118,
-        "top_bins": {
-          "0.9": {
-            "n": 3,
-            "errors": 1,
-            "error_rate": 0.3333333333333333
-          },
-          "0.95": {
-            "n": 2,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9713811910576418
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.75,
-            "confidence_cutoff": 0.736067832341028
-          }
-        }
-      },
-      "ordinary_resolution": {
-        "n": 4,
-        "nll": 0.04811475260764305,
-        "acc": 1.0,
-        "ece": 0.04485818654799875,
-        "brier": 0.011362447845748204,
-        "mean_conf": 0.9551418134520012,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.04485818654799878,
-        "top_bins": {
-          "0.9": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 2,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9973615074301722
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.848997818390719
-          }
-        }
-      },
-      "permuted": {
-        "n": 3,
-        "nll": 1.5849320188552285,
-        "acc": 0.3333333333333333,
-        "ece": 0.5115039138500523,
-        "brier": 0.948500922906351,
-        "mean_conf": 0.834218301404813,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.3333333333333333,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.3333333333333333,
-        "coverage_at_1pct_error": 0.3333333333333333,
-        "aurc": 0.38888888888888884,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.5008849680714798,
-        "top_bins": {
-          "0.9": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 0,
-            "errors": 0,
-            "error_rate": null
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.6666666666666666,
-            "accuracy": 0.5,
-            "confidence_cutoff": 0.8701283299219073
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.3333333333333333,
-            "confidence_cutoff": 0.6484549929603911
-          }
-        }
-      },
-      "review_acknowledged": {
-        "n": 4,
-        "nll": 0.9430886337427746,
-        "acc": 0.75,
-        "ece": 0.26428833978133903,
-        "brier": 0.46621510579801406,
-        "mean_conf": 0.9076209540563065,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 0.6666666666666666,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.14583333333333331,
-        "error_rate_at_0_9": 0.3333333333333333,
-        "confidence_bias": 0.1576209540563065,
-        "top_bins": {
-          "0.9": {
-            "n": 3,
-            "errors": 1,
-            "error_rate": 0.3333333333333333
-          },
-          "0.95": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9371813046912169
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 0.75,
-            "confidence_cutoff": 0.786665228549935
-          }
-        }
-      }
-    },
-    "heldout_tasks": {
-      "composition_held_and_or": {
-        "n": 32,
-        "nll": 0.24171629359001318,
-        "acc": 0.90625,
-        "ece": 0.08267706695037225,
-        "brier": 0.1580607438572586,
-        "mean_conf": 0.9889270669503722,
-        "confident_error_rate": 0.09375,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 0.90625,
-        "coverage_at_5pct_error": 0.9375,
-        "coverage_at_1pct_error": 0.90625,
-        "aurc": 0.0069010416666666664,
-        "error_rate_at_0_9": 0.09375,
-        "confidence_bias": 0.08267706695037225,
-        "top_bins": {
-          "0.9": {
-            "n": 32,
-            "errors": 3,
-            "error_rate": 0.09375
-          },
-          "0.95": {
-            "n": 30,
-            "errors": 1,
-            "error_rate": 0.03333333333333333
-          },
-          "0.99": {
-            "n": 26,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999364856833449
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9996934954887139
-          }
-        }
-      },
-      "contrastive_authorization": {
-        "n": 16,
-        "nll": 1.502653141036028e-05,
-        "acc": 1.0,
-        "ece": 1.50264042420023e-05,
-        "brier": 5.139828295030656e-10,
-        "mean_conf": 0.999984973595758,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -1.50264042420023e-05,
-        "top_bins": {
-          "0.9": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999862890472545
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999813198967473
-          }
-        }
-      },
-      "emotion": {
-        "n": 8,
-        "nll": 1.4194273414243654,
-        "acc": 0.625,
-        "ece": 0.3420785503418361,
-        "brier": 0.6845960520300963,
-        "mean_conf": 0.967078550341836,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 0.875,
-        "accuracy_at_0_9": 0.7142857142857143,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.12842261904761904,
-        "error_rate_at_0_9": 0.2857142857142857,
-        "confidence_bias": 0.342078550341836,
-        "top_bins": {
-          "0.9": {
-            "n": 7,
-            "errors": 2,
-            "error_rate": 0.2857142857142857
-          },
-          "0.95": {
-            "n": 5,
-            "errors": 1,
-            "error_rate": 0.2
-          },
-          "0.99": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9869563807629563
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.7142857142857143,
-            "confidence_cutoff": 0.936063568486461
-          }
-        }
-      },
-      "mmlu": {
-        "n": 8,
-        "nll": 1.3129749769552814,
-        "acc": 0.625,
-        "ece": 0.15765322690405312,
-        "brier": 0.6484351711065982,
-        "mean_conf": 0.7567759888850003,
-        "confident_error_rate": 0.125,
-        "coverage_at_0_9": 0.5,
-        "accuracy_at_0_9": 0.75,
-        "coverage_at_5pct_error": 0.375,
-        "coverage_at_1pct_error": 0.375,
-        "aurc": 0.22336309523809525,
-        "error_rate_at_0_9": 0.25,
-        "confidence_bias": 0.1317759888850003,
-        "top_bins": {
-          "0.9": {
-            "n": 4,
-            "errors": 1,
-            "error_rate": 0.25
-          },
-          "0.95": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 1,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.75,
-            "confidence_cutoff": 0.9327223999584052
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.5714285714285714,
-            "confidence_cutoff": 0.485476400811401
-          }
-        }
-      },
-      "otrf_sysmon_file": {
-        "n": 16,
-        "nll": 0.008605175298168915,
-        "acc": 1.0,
-        "ece": 0.008437849080092708,
-        "brier": 0.0005211762382657875,
-        "mean_conf": 0.9915621509199073,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.008437849080092708,
-        "top_bins": {
-          "0.9": {
-            "n": 16,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 15,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 13,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9989837483137961
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9925042879134804
-          }
-        }
-      },
-      "otrf_sysmon_join": {
-        "n": 16,
-        "nll": 0.12967885980971613,
-        "acc": 0.9375,
-        "ece": 0.0857233928404766,
-        "brier": 0.08146985002971384,
-        "mean_conf": 0.946506108546294,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 0.6875,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 0.9375,
-        "coverage_at_1pct_error": 0.9375,
-        "aurc": 0.00390625,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": 0.009006108546294023,
-        "top_bins": {
-          "0.9": {
-            "n": 11,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 11,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 9,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9970209747654696
-          },
-          "0.8": {
-            "coverage": 0.8125,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.8702734279589676
-          }
-        }
-      },
-      "otrf_sysmon_process": {
-        "n": 24,
-        "nll": 0.16156109121321785,
-        "acc": 0.9583333333333334,
-        "ece": 0.05273169620250518,
-        "brier": 0.0899182770612567,
-        "mean_conf": 0.9768730534919899,
-        "confident_error_rate": 0.041666666666666664,
-        "coverage_at_0_9": 0.9583333333333334,
-        "accuracy_at_0_9": 0.9565217391304348,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 0.9166666666666666,
-        "aurc": 0.0035477053140096615,
-        "error_rate_at_0_9": 0.043478260869565216,
-        "confidence_bias": 0.01853972015865657,
-        "top_bins": {
-          "0.9": {
-            "n": 23,
-            "errors": 1,
-            "error_rate": 0.043478260869565216
-          },
-          "0.95": {
-            "n": 22,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 19,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9994396085787187
-          },
-          "0.8": {
-            "coverage": 0.8333333333333334,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9832554759400715
-          }
-        }
-      },
-      "otrf_sysmon_registry": {
-        "n": 8,
-        "nll": 0.010723606364118954,
-        "acc": 1.0,
-        "ece": 0.010542385937563847,
-        "brier": 0.0006959554565011166,
-        "mean_conf": 0.9894576140624362,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.010542385937563847,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 5,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9972651302471073
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9840692701748682
-          }
-        }
-      },
-      "paws": {
-        "n": 8,
-        "nll": 1.0368545966100273,
-        "acc": 0.75,
-        "ece": 0.22498828525853157,
-        "brier": 0.34163600039007924,
-        "mean_conf": 0.9122532529823204,
-        "confident_error_rate": 0.125,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 0.8333333333333334,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.09494047619047619,
-        "error_rate_at_0_9": 0.16666666666666666,
-        "confidence_bias": 0.1622532529823204,
-        "top_bins": {
-          "0.9": {
-            "n": 6,
-            "errors": 1,
-            "error_rate": 0.16666666666666666
-          },
-          "0.95": {
-            "n": 6,
-            "errors": 1,
-            "error_rate": 0.16666666666666666
-          },
-          "0.99": {
-            "n": 6,
-            "errors": 1,
-            "error_rate": 0.16666666666666666
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9992691391151949
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.8571428571428571,
-            "confidence_cutoff": 0.749059870895155
-          }
-        }
-      },
-      "qnli": {
-        "n": 8,
-        "nll": 0.005181466224742151,
-        "acc": 1.0,
-        "ece": 0.005129334884136472,
-        "brier": 0.00020524982613720548,
-        "mean_conf": 0.9948706651158635,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.005129334884136472,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 6,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.999952371295075
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.98872862957213
-          }
-        }
-      },
-      "sciq": {
-        "n": 8,
-        "nll": 0.00023928943016782004,
-        "acc": 1.0,
-        "ece": 0.0002391650142747892,
-        "brier": 4.955209174865095e-07,
-        "mean_conf": 0.9997608349857252,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -0.0002391650142747892,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 8,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999911919978735
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9994259298933852
-          }
-        }
-      },
-      "sev_swarmtraces_manual_credential": {
-        "n": 40,
-        "nll": 0.9530052125004369,
-        "acc": 0.825,
-        "ece": 0.17316335764162732,
-        "brier": 0.3580423587545788,
-        "mean_conf": 0.9202306277592299,
-        "confident_error_rate": 0.15,
-        "coverage_at_0_9": 0.75,
-        "accuracy_at_0_9": 0.8,
-        "coverage_at_5pct_error": 0.225,
-        "coverage_at_1pct_error": 0.225,
-        "aurc": 0.10718612602417658,
-        "error_rate_at_0_9": 0.2,
-        "confidence_bias": 0.09523062775922997,
-        "top_bins": {
-          "0.9": {
-            "n": 30,
-            "errors": 6,
-            "error_rate": 0.2
-          },
-          "0.95": {
-            "n": 26,
-            "errors": 3,
-            "error_rate": 0.11538461538461539
-          },
-          "0.99": {
-            "n": 15,
-            "errors": 1,
-            "error_rate": 0.06666666666666667
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.9,
-            "confidence_cutoff": 0.9774245184450835
-          },
-          "0.8": {
-            "coverage": 0.8,
-            "accuracy": 0.8125,
-            "confidence_cutoff": 0.8542392185801473
-          }
-        }
-      },
-      "sev_swarmtraces_manual_flow": {
-        "n": 40,
-        "nll": 1.485896746743676,
-        "acc": 0.75,
-        "ece": 0.21977756775377905,
-        "brier": 0.43299049842888415,
-        "mean_conf": 0.9214803040956191,
-        "confident_error_rate": 0.15,
-        "coverage_at_0_9": 0.725,
-        "accuracy_at_0_9": 0.7931034482758621,
-        "coverage_at_5pct_error": 0.1,
-        "coverage_at_1pct_error": 0.1,
-        "aurc": 0.14190456225170725,
-        "error_rate_at_0_9": 0.20689655172413793,
-        "confidence_bias": 0.1714803040956191,
-        "top_bins": {
-          "0.9": {
-            "n": 29,
-            "errors": 6,
-            "error_rate": 0.20689655172413793
-          },
-          "0.95": {
-            "n": 27,
-            "errors": 6,
-            "error_rate": 0.2222222222222222
-          },
-          "0.99": {
-            "n": 21,
-            "errors": 2,
-            "error_rate": 0.09523809523809523
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 0.9,
-            "confidence_cutoff": 0.9901066734665328
-          },
-          "0.8": {
-            "coverage": 0.8,
-            "accuracy": 0.8125,
-            "confidence_cutoff": 0.8291658959545783
-          }
-        }
-      },
-      "sev_swarmtraces_manual_prerequisite": {
-        "n": 3,
-        "nll": 1.0331257533433365e-05,
-        "acc": 1.0,
-        "ece": 1.0331183196821137e-05,
-        "brier": 2.5195789327524225e-10,
-        "mean_conf": 0.9999896688168032,
-        "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 1.0,
-        "coverage_at_5pct_error": 1.0,
-        "coverage_at_1pct_error": 1.0,
-        "aurc": 0.0,
-        "error_rate_at_0_9": 0.0,
-        "confidence_bias": -1.0331183196821137e-05,
-        "top_bins": {
-          "0.9": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.95": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          },
-          "0.99": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.6666666666666666,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999897312178962
-          },
-          "0.8": {
-            "coverage": 1.0,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9999810314632929
-          }
-        }
-      },
-      "tweet_offensive": {
-        "n": 8,
-        "nll": 0.7172428487540772,
-        "acc": 0.75,
-        "ece": 0.2179639808956667,
-        "brier": 0.44120571426799937,
-        "mean_conf": 0.9679639808956667,
-        "confident_error_rate": 0.25,
-        "coverage_at_0_9": 1.0,
-        "accuracy_at_0_9": 0.75,
-        "coverage_at_5pct_error": 0.5,
-        "coverage_at_1pct_error": 0.5,
-        "aurc": 0.09494047619047619,
-        "error_rate_at_0_9": 0.25,
-        "confidence_bias": 0.2179639808956667,
-        "top_bins": {
-          "0.9": {
-            "n": 8,
-            "errors": 2,
-            "error_rate": 0.25
-          },
-          "0.95": {
-            "n": 5,
-            "errors": 1,
-            "error_rate": 0.2
-          },
-          "0.99": {
-            "n": 3,
-            "errors": 0,
-            "error_rate": 0.0
-          }
-        },
-        "selective": {
-          "0.5": {
-            "coverage": 0.5,
-            "accuracy": 1.0,
-            "confidence_cutoff": 0.9897168920827885
-          },
-          "0.8": {
-            "coverage": 0.875,
-            "accuracy": 0.8571428571428571,
-            "confidence_cutoff": 0.940377631626033
-          }
-        }
-      }
-    },
-    "permutation": {
-      "n": 3,
-      "mean_max_delta": 0.04297798881434687,
-      "flip_rate": 0.0
-    },
-    "temperature": 1.6817928305074288,
-    "calibrated_clean": {
-      "n": 243,
-      "nll": 0.4460227427836622,
-      "acc": 0.8683127572016461,
-      "ece": 0.04490914804947624,
-      "brier": 0.21677989484802473,
-      "mean_conf": 0.9016047263613403,
-      "confident_error_rate": 0.024691358024691357,
-      "coverage_at_0_9": 0.6831275720164609,
-      "accuracy_at_0_9": 0.963855421686747,
-      "coverage_at_5pct_error": 0.7489711934156379,
-      "coverage_at_1pct_error": 0.4156378600823045,
-      "aurc": 0.03366327644063936,
-      "error_rate_at_0_9": 0.03614457831325301,
-      "confidence_bias": 0.033291969159694124,
-      "top_bins": {
-        "0.9": {
-          "n": 166,
-          "errors": 6,
-          "error_rate": 0.03614457831325301
-        },
-        "0.95": {
-          "n": 141,
-          "errors": 2,
-          "error_rate": 0.014184397163120567
-        },
-        "0.99": {
-          "n": 81,
-          "errors": 0,
-          "error_rate": 0.0
-        }
-      },
-      "selective": {
-        "0.5": {
-          "coverage": 0.5020576131687243,
-          "accuracy": 0.9836065573770492,
-          "confidence_cutoff": 0.9739727890189261
-        },
-        "0.8": {
-          "coverage": 0.8024691358024691,
-          "accuracy": 0.9282051282051282,
-          "confidence_cutoff": 0.8229542441861438
-        }
-      }
-    },
-    "metric_policy": {
-      "version": 2,
-      "selective_ties": "whole_confidence_groups",
-      "coverage_at_error": "in-sample maximum over confidence thresholds; not a deployed error guarantee",
-      "aurc": "right-step integral over whole confidence groups",
-      "confident_error_rate": "high-confidence errors divided by all questions",
-      "error_rate_at_0_9": "errors divided by questions accepted at p_max >= 0.9",
-      "nll": "exact from logits when recorded; otherwise from floored probabilities",
-      "nll_floor": 1e-09,
-      "renormalize_returned_probabilities": true,
-      "raw_sums_outside_1e_5": 0,
-      "returned_zeros": 0
-    },
-    "coverage": {
-      "requested_records": 185,
-      "requested_questions": 284,
-      "evaluated_records": 185,
-      "evaluated_questions": 284,
-      "rejected_records": 0,
-      "truncated_records": 0
-    },
-    "latency_ms": {
-      "median": 49.463423999895895,
-      "p95": 189.5706840000457
-    },
-    "calibration": {
-      "inference_temperature": 1.0,
-      "additional_temperature": 1.6817928305074288,
-      "logits_recorded": true
-    },
-    "suite_sha256": "adde94227c959c3bfd62ee2ca27d8a73d5b0fff96bf0500c627645e5a7d72f37"
-  },
   "provenance": {
     "config": {
-      "epochs": 1,
       "seed": 4,
-      "lr": 2.5e-06,
       "lora": 16,
       "accum": 2,
       "batch": 4,
@@ -2823,7 +249,7 @@
       "focal_gamma": 0.0,
       "base": "Qwen/Qwen3.5-4B-Base",
       "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
-      "init_from": "macmacmacmac/Sev-4B@a1824aefba305fda86e3503b895a7d9b3871f79a",
       "lora_targets": "all",
       "head_dim": 256,
       "weights_dtype": "fp32",
@@ -2832,8 +258,8 @@
       "option_isolation": 0,
       "special_embeddings": 0
     },
-    "config_sha256": "01ce0c9f84977e5c32d40dbd5c012634208c7770615b9e7d9d6f5950c042706f",
-    "suite_sha256": "3b9a80dfd212b6932b66ea520c35ab1a3b19e132a81af1b6b7b306d776a70a02",
     "source_hashes": {
       "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
       "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
@@ -2865,62 +291,59 @@
       "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
       "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
     },
-    "git_commit": "740b511bb562cf29430a85457b9e54389a3f21cd",
     "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
     "torch": "2.8.0+cu128",
     "device": "cuda",
     "gpu": "NVIDIA H100 80GB HBM3",
     "legacy_checkpoint": false,
     "measured_checkpoint": {
-      "requested": "/runs/sev-r2-response-policy-4b-v2/00-trial-0/checkpoint",
-      "resolved": "/runs/sev-r2-response-policy-4b-v2/00-trial-0/checkpoint",
-      "head_sha256": "506dcd3587847917a288d95757c497c15a2aef8e2d4bc6523b04ad991c961a2b",
-      "adapter_sha256": "fa1916b34fe1eeb10d15c36927c225df43115407d7c97b4f879bba3102c3ed21",
       "inference_temperature": 1.0
     }
   },
   "mechanism_checks": {
-    "n": 24,
-    "packed_max_delta": 1.923413947224617e-06,
-    "sibling_max_delta": 9.5367431640625e-07,
     "tolerance": 0.001,
-    "passed": true
   },
   "gates": {
-    "passed": true,
     "checks": {
       "complete_coverage": true,
-      "isolation_and_packing": true,
-      "transfer_complete": true,
-      "heldout_pairs_at_least_70pct": true,
-      "transfer_confident_errors_below_10pct": true
     },
     "policy": "Research screening only: 0.001 isolation; 5pp task regression; 70% heldout pair correctness; <=10% confident errors; no automatic release."
   },
   "calibration_fit": {
-    "temperature": 1.6817928305074288,
     "aggregation": "micro",
     "objective": "raw-logit NLL",
     "split": "calibration",
-    "rows_sha256": "026b68705b7ac76bc4361bc2fda1cbfe74265b436a807ea68bca77f87d3cf9da",
-    "suite_sha256": "3b9a80dfd212b6932b66ea520c35ab1a3b19e132a81af1b6b7b306d776a70a02",
-    "n": 2674
   },
-  "wall_seconds": 1285.6242614819998,
   "promotable": false,
   "test_evaluated": false,
   "training_resources": {
-    "wall_seconds": 887.5156674385071,
-    "records_seen": 6577,
-    "requested_records": 6577,
     "truncated_records": 0,
     "rejected_records": 0,
-    "optimizer_steps": 823,
-    "forward_tokens": 1505379,
-    "peak_device_bytes": 22225622528,
     "device": "cuda",
     "dtype": "bf16",
     "batch": 4,
-    "peak_rss_bytes": 31556661248
   }
 }

 {
+  "objective": -0.8137446728855184,
   "paired_flip": null,
   "unknowable": null,
   "clean": {
+    "n": 175,
+    "nll": 0.8137446728855184,
+    "acc": 0.9028571428571428,
+    "ece": 0.09141608822222473,
+    "brier": 0.17599452427593348,
+    "mean_conf": 0.9942732310793674,
+    "confident_error_rate": 0.08,
+    "coverage_at_0_9": 0.9828571428571429,
+    "accuracy_at_0_9": 0.9186046511627907,
+    "coverage_at_5pct_error": 0.9142857142857143,
+    "coverage_at_1pct_error": 0.011428571428571429,
+    "aurc": 0.06143423339655433,
+    "error_rate_at_0_9": 0.08139534883720931,
+    "confidence_bias": 0.09141608822222458,
     "top_bins": {
       "0.9": {
+        "n": 172,
+        "errors": 14,
+        "error_rate": 0.08139534883720931
       },
       "0.95": {
+        "n": 172,
+        "errors": 14,
+        "error_rate": 0.08139534883720931
       },
       "0.99": {
+        "n": 168,
+        "errors": 11,
+        "error_rate": 0.06547619047619048
       }
     },
     "selective": {
       "0.5": {
+        "coverage": 0.5028571428571429,
+        "accuracy": 0.9431818181818182,
+        "confidence_cutoff": 0.9999983862993777
       },
       "0.8": {
+        "coverage": 0.8,
+        "accuracy": 0.9571428571428572,
+        "confidence_cutoff": 0.9999883543164546
       }
     }
   },
   "tasks": {
     "sev_swarmtraces_response_policy": {
       "n": 175,
+      "nll": 0.8137446728855184,
+      "acc": 0.9028571428571428,
+      "ece": 0.09141608822222473,
+      "brier": 0.17599452427593348,
+      "mean_conf": 0.9942732310793674,
+      "confident_error_rate": 0.08,
+      "coverage_at_0_9": 0.9828571428571429,
+      "accuracy_at_0_9": 0.9186046511627907,
+      "coverage_at_5pct_error": 0.9142857142857143,
+      "coverage_at_1pct_error": 0.011428571428571429,
+      "aurc": 0.06143423339655433,
+      "error_rate_at_0_9": 0.08139534883720931,
+      "confidence_bias": 0.09141608822222458,
       "top_bins": {
         "0.9": {
+          "n": 172,
+          "errors": 14,
+          "error_rate": 0.08139534883720931
         },
         "0.95": {
+          "n": 172,
           "errors": 14,
+          "error_rate": 0.08139534883720931
         },
         "0.99": {
+          "n": 168,
+          "errors": 11,
+          "error_rate": 0.06547619047619048
         }
       },
       "selective": {
         "0.5": {
           "coverage": 0.5028571428571429,
+          "accuracy": 0.9431818181818182,
+          "confidence_cutoff": 0.9999983862993777
         },
         "0.8": {
           "coverage": 0.8,
+          "accuracy": 0.9571428571428572,
+          "confidence_cutoff": 0.9999883543164546
         }
       }
     }
   },
   "variants": {
     "clean": {
+      "n": 175,
+      "nll": 0.8137446728855184,
+      "acc": 0.9028571428571428,
+      "ece": 0.09141608822222473,
+      "brier": 0.17599452427593348,
+      "mean_conf": 0.9942732310793674,
+      "confident_error_rate": 0.08,
+      "coverage_at_0_9": 0.9828571428571429,
+      "accuracy_at_0_9": 0.9186046511627907,
+      "coverage_at_5pct_error": 0.9142857142857143,
+      "coverage_at_1pct_error": 0.011428571428571429,
+      "aurc": 0.06143423339655433,
+      "error_rate_at_0_9": 0.08139534883720931,
+      "confidence_bias": 0.09141608822222458,
       "top_bins": {
         "0.9": {
+          "n": 172,
+          "errors": 14,
+          "error_rate": 0.08139534883720931
         },
         "0.95": {
+          "n": 172,
+          "errors": 14,
+          "error_rate": 0.08139534883720931
         },
         "0.99": {
+          "n": 168,
+          "errors": 11,
+          "error_rate": 0.06547619047619048
         }
       },
       "selective": {
         "0.5": {
+          "coverage": 0.5028571428571429,
+          "accuracy": 0.9431818181818182,
+          "confidence_cutoff": 0.9999983862993777
         },
         "0.8": {
+          "coverage": 0.8,
+          "accuracy": 0.9571428571428572,
+          "confidence_cutoff": 0.9999883543164546
         }
       }
     }
     "mean_max_delta": null,
     "flip_rate": null
   },
+  "temperature": 4.0,
   "calibrated_clean": {
+    "n": 175,
+    "nll": 0.26348233904029333,
+    "acc": 0.9028571428571428,
+    "ece": 0.04076488602051544,
+    "brier": 0.1426360448938433,
+    "mean_conf": 0.9436220288776582,
+    "confident_error_rate": 0.045714285714285714,
+    "coverage_at_0_9": 0.8971428571428571,
+    "accuracy_at_0_9": 0.9490445859872612,
+    "coverage_at_5pct_error": 0.9142857142857143,
+    "coverage_at_1pct_error": 0.011428571428571429,
+    "aurc": 0.06143423339655433,
+    "error_rate_at_0_9": 0.050955414012738856,
+    "confidence_bias": 0.04076488602051542,
     "top_bins": {
       "0.9": {
+        "n": 157,
+        "errors": 8,
+        "error_rate": 0.050955414012738856
       },
       "0.95": {
+        "n": 131,
+        "errors": 5,
+        "error_rate": 0.03816793893129771
       },
       "0.99": {
+        "n": 0,
         "errors": 0,
+        "error_rate": null
       }
     },
     "selective": {
       "0.5": {
+        "coverage": 0.5028571428571429,
+        "accuracy": 0.9431818181818182,
+        "confidence_cutoff": 0.9655851033024723
       },
       "0.8": {
+        "coverage": 0.8,
+        "accuracy": 0.9571428571428572,
+        "confidence_cutoff": 0.9448068121971666
       }
     }
   },
     "returned_zeros": 0
   },
   "coverage": {
+    "requested_records": 35,
+    "requested_questions": 175,
+    "evaluated_records": 35,
+    "evaluated_questions": 175,
     "rejected_records": 0,
     "truncated_records": 0
   },
   "latency_ms": {
+    "median": 240.12951999998222,
+    "p95": 301.12752519999043
   },
   "calibration": {
     "inference_temperature": 1.0,
+    "additional_temperature": 4.0,
     "logits_recorded": true
   },
+  "transfer": null,
   "provenance": {
     "config": {
+      "epochs": 5,
       "seed": 4,
+      "lr": 1e-05,
       "lora": 16,
       "accum": 2,
       "batch": 4,
       "focal_gamma": 0.0,
       "base": "Qwen/Qwen3.5-4B-Base",
       "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
+      "init_from": "macmacmacmac/Sev-4B@v0.3.0-response-policy-research",
       "lora_targets": "all",
       "head_dim": 256,
       "weights_dtype": "fp32",
       "option_isolation": 0,
       "special_embeddings": 0
     },
+    "config_sha256": "31d8cf5af214ae4ed0012d1e4fea9fd73e207209cfce1f1112107e4aad65cc9f",
+    "suite_sha256": "40d3a8b3bd416a3754ecc30e9791a9ebb14a26c8f2a4422057f769a2dbce4dee",
     "source_hashes": {
       "kev/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
       "kev/anchors.py": "6963eafdb276db5a6c94d939eae675c761555553b8d0f963ffd803448beaabbb",
       "pyproject.toml": "39ec40f14c88598dee7ac98fb859a5a44e22337235a3064ebd5e89036ce81e30",
       "uv.lock": "a9922dbb89acdef78299fd2b4a8c3f7f0fa1b2bc08b55595b6926fa785a9c466"
     },
+    "git_commit": "fcd6756a329f42a515b4b48f83fb8bde9776232f",
     "platform": "Linux-4.19.0-gvisor-x86_64-with-glibc2.36",
     "torch": "2.8.0+cu128",
     "device": "cuda",
     "gpu": "NVIDIA H100 80GB HBM3",
     "legacy_checkpoint": false,
     "measured_checkpoint": {
+      "requested": "/runs/sev-html-contrast-4b-v1/00-trial-0/checkpoint",
+      "resolved": "/runs/sev-html-contrast-4b-v1/00-trial-0/checkpoint",
+      "head_sha256": "2724bebcc54d4846194130594b98d3e3b9c632935c0f1d5077e46e23c48b2dc8",
+      "adapter_sha256": "b58b3963175e6c40bd022db5893ebefb6a6f1e1bd1ec11efc6692a089e1dcf51",
       "inference_temperature": 1.0
     }
   },
   "mechanism_checks": {
+    "n": 40,
+    "packed_max_delta": 0.0015369057655334473,
+    "sibling_max_delta": 0.0007038116455078125,
     "tolerance": 0.001,
+    "passed": false
   },
   "gates": {
+    "passed": false,
     "checks": {
       "complete_coverage": true,
+      "isolation_and_packing": false
     },
     "policy": "Research screening only: 0.001 isolation; 5pp task regression; 70% heldout pair correctness; <=10% confident errors; no automatic release."
   },
   "calibration_fit": {
+    "temperature": 4.0,
     "aggregation": "micro",
     "objective": "raw-logit NLL",
     "split": "calibration",
+    "rows_sha256": "725dc3c48e167e782825490bebc3b3f07b5933660fdca7ac65ad37f13481eb6e",
+    "suite_sha256": "40d3a8b3bd416a3754ecc30e9791a9ebb14a26c8f2a4422057f769a2dbce4dee",
+    "n": 140
   },
+  "wall_seconds": 366.39384709,
   "promotable": false,
   "test_evaluated": false,
   "training_resources": {
+    "wall_seconds": 232.53715324401855,
+    "records_seen": 615,
+    "requested_records": 615,
     "truncated_records": 0,
     "rejected_records": 0,
+    "optimizer_steps": 80,
+    "forward_tokens": 209595,
+    "peak_device_bytes": 20358582784,
     "device": "cuda",
     "dtype": "bf16",
     "batch": 4,
+    "peak_rss_bytes": 31542779904
   }
 }

train.log CHANGED Viewed

@@ -1,88 +1,14 @@
 Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
-delta: warm start from /__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--macmacmacmac--Sev-4B/snapshots/a1824aefba305fda86e3503b895a7d9b3871f79a: 496 adapter tensors and the pointer head loaded
 device=cuda trainable params=33.8M
-6577 training requests (holdout=[]), questions by type {'choice': 8590, 'noul': 518, 'score': 348}
 [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
-ep0 step 10/823 loss 0.397 kl 0.000 anchor 0.000 1.763s/rec
-ep0 step 20/823 loss 0.612 kl 0.000 anchor 0.000 1.071s/rec
-ep0 step 30/823 loss 0.332 kl 0.000 anchor 0.000 0.761s/rec
-ep0 step 40/823 loss 0.428 kl 0.000 anchor 0.000 0.606s/rec
-ep0 step 50/823 loss 0.487 kl 0.000 anchor 0.000 0.508s/rec
-ep0 step 60/823 loss 0.391 kl 0.000 anchor 0.000 0.442s/rec
-ep0 step 70/823 loss 0.304 kl 0.000 anchor 0.000 0.394s/rec
-ep0 step 80/823 loss 0.458 kl 0.000 anchor 0.000 0.358s/rec
-ep0 step 90/823 loss 0.344 kl 0.000 anchor 0.000 0.330s/rec
-ep0 step 100/823 loss 0.675 kl 0.000 anchor 0.000 0.308s/rec
-ep0 step 110/823 loss 0.543 kl 0.000 anchor 0.000 0.290s/rec
-ep0 step 120/823 loss 0.438 kl 0.000 anchor 0.000 0.274s/rec
-ep0 step 130/823 loss 0.481 kl 0.000 anchor 0.000 0.262s/rec
-ep0 step 140/823 loss 0.608 kl 0.000 anchor 0.000 0.250s/rec
-ep0 step 150/823 loss 0.313 kl 0.000 anchor 0.000 0.241s/rec
-ep0 step 160/823 loss 0.419 kl 0.000 anchor 0.000 0.232s/rec
-ep0 step 170/823 loss 0.520 kl 0.000 anchor 0.000 0.225s/rec
-ep0 step 180/823 loss 0.635 kl 0.000 anchor 0.000 0.218s/rec
-ep0 step 190/823 loss 0.429 kl 0.000 anchor 0.000 0.212s/rec
-ep0 step 200/823 loss 0.445 kl 0.000 anchor 0.000 0.207s/rec
-ep0 step 210/823 loss 0.404 kl 0.000 anchor 0.000 0.203s/rec
-ep0 step 220/823 loss 0.472 kl 0.000 anchor 0.000 0.199s/rec
-ep0 step 230/823 loss 0.464 kl 0.000 anchor 0.000 0.195s/rec
-ep0 step 240/823 loss 0.333 kl 0.000 anchor 0.000 0.192s/rec
-ep0 step 250/823 loss 0.507 kl 0.000 anchor 0.000 0.188s/rec
-ep0 step 260/823 loss 0.411 kl 0.000 anchor 0.000 0.185s/rec
-ep0 step 270/823 loss 0.573 kl 0.000 anchor 0.000 0.182s/rec
-ep0 step 280/823 loss 0.492 kl 0.000 anchor 0.000 0.179s/rec
-ep0 step 290/823 loss 0.430 kl 0.000 anchor 0.000 0.177s/rec
-ep0 step 300/823 loss 0.388 kl 0.000 anchor 0.000 0.182s/rec
-ep0 step 310/823 loss 0.426 kl 0.000 anchor 0.000 0.180s/rec
-ep0 step 320/823 loss 0.361 kl 0.000 anchor 0.000 0.177s/rec
-ep0 step 330/823 loss 0.430 kl 0.000 anchor 0.000 0.175s/rec
-ep0 step 340/823 loss 0.510 kl 0.000 anchor 0.000 0.173s/rec
-ep0 step 350/823 loss 0.274 kl 0.000 anchor 0.000 0.171s/rec
-ep0 step 360/823 loss 0.412 kl 0.000 anchor 0.000 0.170s/rec
-ep0 step 370/823 loss 0.459 kl 0.000 anchor 0.000 0.168s/rec
-ep0 step 380/823 loss 0.348 kl 0.000 anchor 0.000 0.166s/rec
-ep0 step 390/823 loss 0.367 kl 0.000 anchor 0.000 0.165s/rec
-ep0 step 400/823 loss 0.523 kl 0.000 anchor 0.000 0.163s/rec
-ep0 step 410/823 loss 0.358 kl 0.000 anchor 0.000 0.162s/rec
-ep0 step 420/823 loss 0.448 kl 0.000 anchor 0.000 0.161s/rec
-ep0 step 430/823 loss 0.474 kl 0.000 anchor 0.000 0.159s/rec
-ep0 step 440/823 loss 0.429 kl 0.000 anchor 0.000 0.158s/rec
-ep0 step 450/823 loss 0.375 kl 0.000 anchor 0.000 0.157s/rec
-ep0 step 460/823 loss 0.413 kl 0.000 anchor 0.000 0.156s/rec
-ep0 step 470/823 loss 0.444 kl 0.000 anchor 0.000 0.154s/rec
-ep0 step 480/823 loss 0.409 kl 0.000 anchor 0.000 0.154s/rec
-ep0 step 490/823 loss 0.272 kl 0.000 anchor 0.000 0.153s/rec
-ep0 step 500/823 loss 0.660 kl 0.000 anchor 0.000 0.152s/rec
-ep0 step 510/823 loss 0.596 kl 0.000 anchor 0.000 0.151s/rec
-ep0 step 520/823 loss 0.453 kl 0.000 anchor 0.000 0.150s/rec
-ep0 step 530/823 loss 0.486 kl 0.000 anchor 0.000 0.150s/rec
-ep0 step 540/823 loss 0.548 kl 0.000 anchor 0.000 0.149s/rec
-ep0 step 550/823 loss 0.412 kl 0.000 anchor 0.000 0.148s/rec
-ep0 step 560/823 loss 0.419 kl 0.000 anchor 0.000 0.147s/rec
-ep0 step 570/823 loss 0.424 kl 0.000 anchor 0.000 0.147s/rec
-ep0 step 580/823 loss 0.378 kl 0.000 anchor 0.000 0.146s/rec
-ep0 step 590/823 loss 0.443 kl 0.000 anchor 0.000 0.145s/rec
-ep0 step 600/823 loss 0.624 kl 0.000 anchor 0.000 0.145s/rec
-ep0 step 610/823 loss 0.536 kl 0.000 anchor 0.000 0.144s/rec
-ep0 step 620/823 loss 0.460 kl 0.000 anchor 0.000 0.143s/rec
-ep0 step 630/823 loss 0.491 kl 0.000 anchor 0.000 0.143s/rec
-ep0 step 640/823 loss 0.472 kl 0.000 anchor 0.000 0.142s/rec
-ep0 step 650/823 loss 0.327 kl 0.000 anchor 0.000 0.142s/rec
-ep0 step 660/823 loss 0.451 kl 0.000 anchor 0.000 0.141s/rec
-ep0 step 670/823 loss 0.380 kl 0.000 anchor 0.000 0.141s/rec
-ep0 step 680/823 loss 0.432 kl 0.000 anchor 0.000 0.140s/rec
-ep0 step 690/823 loss 0.261 kl 0.000 anchor 0.000 0.140s/rec
-ep0 step 700/823 loss 0.610 kl 0.000 anchor 0.000 0.139s/rec
-ep0 step 710/823 loss 0.463 kl 0.000 anchor 0.000 0.139s/rec
-ep0 step 720/823 loss 0.530 kl 0.000 anchor 0.000 0.138s/rec
-ep0 step 730/823 loss 0.495 kl 0.000 anchor 0.000 0.138s/rec
-ep0 step 740/823 loss 0.609 kl 0.000 anchor 0.000 0.137s/rec
-ep0 step 750/823 loss 0.398 kl 0.000 anchor 0.000 0.137s/rec
-ep0 step 760/823 loss 0.536 kl 0.000 anchor 0.000 0.137s/rec
-ep0 step 770/823 loss 0.606 kl 0.000 anchor 0.000 0.136s/rec
-ep0 step 780/823 loss 0.474 kl 0.000 anchor 0.000 0.136s/rec
-ep0 step 790/823 loss 0.423 kl 0.000 anchor 0.000 0.135s/rec
-ep0 step 800/823 loss 0.567 kl 0.000 anchor 0.000 0.135s/rec
-ep0 step 810/823 loss 0.632 kl 0.000 anchor 0.000 0.135s/rec
-ep0 step 820/823 loss 0.528 kl 0.000 anchor 0.000 0.134s/rec
-saved /runs/sev-r2-response-policy-4b-v2/00-trial-0/checkpoint

 Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
+delta: warm start from /__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--macmacmacmac--Sev-4B/snapshots/da0a131c2636b06c16bbca7d8ae84f0dc1e90b7e: 496 adapter tensors and the pointer head loaded
 device=cuda trainable params=33.8M
+123 training requests (holdout=[]), questions by type {'choice': 188}
 [transformers] `causal_conv1d_fn` is falling back to its reference PyTorch implementation because `causal_conv1d` is not installed. This is correct but much slower; install `causal_conv1d` for the optimized kernel.
+ep0 step 10/80 loss 0.521 kl 0.000 anchor 0.000 1.937s/rec
+ep1 step 20/80 loss 0.157 kl 0.000 anchor 0.000 1.087s/rec
+ep1 step 30/80 loss 0.001 kl 0.000 anchor 0.000 0.763s/rec
+ep2 step 40/80 loss 0.000 kl 0.000 anchor 0.000 0.611s/rec
+ep3 step 50/80 loss 0.000 kl 0.000 anchor 0.000 0.518s/rec
+ep3 step 60/80 loss 0.000 kl 0.000 anchor 0.000 0.453s/rec
+ep4 step 70/80 loss 0.000 kl 0.000 anchor 0.000 0.409s/rec
+ep4 step 80/80 loss 0.000 kl 0.000 anchor 0.000 0.377s/rec
+saved /runs/sev-html-contrast-4b-v1/00-trial-0/checkpoint

training_config.json CHANGED Viewed

@@ -2,8 +2,8 @@
   "args": {
     "base": "Qwen/Qwen3.5-4B-Base",
     "n_per_source": 1000,
-    "epochs": 1,
-    "lr": 2.5e-06,
     "head_lr": 0.0,
     "weight_decay": 0.01,
     "lora": 16,
@@ -15,7 +15,7 @@
     "label_smoothing": 0.0,
     "brier_w": 0.0,
     "focal_gamma": 0.0,
-    "suite": "/root/evals/sev/response-policy-curriculum-v2",
     "train_sources": "",
     "device": "cuda",
     "batch": 4,
@@ -36,19 +36,19 @@
     "anchor": "",
     "anchor_w": 0.0,
     "anchor_sources": "",
-    "out": "/runs/sev-r2-response-policy-4b-v2/00-trial-0/checkpoint",
     "data": "",
     "replay": 0,
-    "init_from": "macmacmacmac/Sev-4B@a1824aefba305fda86e3503b895a7d9b3871f79a",
     "seed": 4
   },
-  "suite_sha256": "3b9a80dfd212b6932b66ea520c35ab1a3b19e132a81af1b6b7b306d776a70a02",
   "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
   "init_source": {
-    "init_from": "macmacmacmac/Sev-4B@a1824aefba305fda86e3503b895a7d9b3871f79a",
-    "resolved": "/__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--macmacmacmac--Sev-4B/snapshots/a1824aefba305fda86e3503b895a7d9b3871f79a",
-    "adapter_sha256": "b5afd73584d1098cccb0a1bd42afc40e2de3c3bc3074a5173a642b6d4b880420",
-    "head_sha256": "1d50c37b4db60c22ae8e6bb24f5055056701bb16e332964350d76df57dc82af0",
     "adapter_tensors": 496
   },
   "ordinal_objective": "ranked_probability_score",

   "args": {
     "base": "Qwen/Qwen3.5-4B-Base",
     "n_per_source": 1000,
+    "epochs": 5,
+    "lr": 1e-05,
     "head_lr": 0.0,
     "weight_decay": 0.01,
     "lora": 16,
     "label_smoothing": 0.0,
     "brier_w": 0.0,
     "focal_gamma": 0.0,
+    "suite": "/root/evals/sev/response-policy-html-contrast-v1",
     "train_sources": "",
     "device": "cuda",
     "batch": 4,
     "anchor": "",
     "anchor_w": 0.0,
     "anchor_sources": "",
+    "out": "/runs/sev-html-contrast-4b-v1/00-trial-0/checkpoint",
     "data": "",
     "replay": 0,
+    "init_from": "macmacmacmac/Sev-4B@v0.3.0-response-policy-research",
     "seed": 4
   },
+  "suite_sha256": "40d3a8b3bd416a3754ecc30e9791a9ebb14a26c8f2a4422057f769a2dbce4dee",
   "base_revision": "1001bb4d826a52d1f399e183466143f4da7b741b",
   "init_source": {
+    "init_from": "macmacmacmac/Sev-4B@v0.3.0-response-policy-research",
+    "resolved": "/__modal/volumes/vo-RociVuGycNGtzvUcdOqecz/hub/models--macmacmacmac--Sev-4B/snapshots/da0a131c2636b06c16bbca7d8ae84f0dc1e90b7e",
+    "adapter_sha256": "fa1916b34fe1eeb10d15c36927c225df43115407d7c97b4f879bba3102c3ed21",
+    "head_sha256": "12112262e1822fa4ab2981650bd10ceac46929bb29fbbdd843a39ec9de7dbe82",
     "adapter_tensors": 496
   },
   "ordinal_objective": "ranked_probability_score",

training_metrics.json CHANGED Viewed

@@ -1,14 +1,14 @@
 {
-  "wall_seconds": 887.5156674385071,
-  "records_seen": 6577,
-  "requested_records": 6577,
   "truncated_records": 0,
   "rejected_records": 0,
-  "optimizer_steps": 823,
-  "forward_tokens": 1505379,
-  "peak_device_bytes": 22225622528,
   "device": "cuda",
   "dtype": "bf16",
   "batch": 4,
-  "peak_rss_bytes": 31556661248
 }

 {
+  "wall_seconds": 232.53715324401855,
+  "records_seen": 615,
+  "requested_records": 615,
   "truncated_records": 0,
   "rejected_records": 0,
+  "optimizer_steps": 80,
+  "forward_tokens": 209595,
+  "peak_device_bytes": 20358582784,
   "device": "cuda",
   "dtype": "bf16",
   "batch": 4,
+  "peak_rss_bytes": 31542779904
 }