v0.6.1: routing temperature (use_case_temperatures.routing), same weights as v0.6 (jonpol01/d1a#264)

README.md

@@ -10,6 +10,8 @@ tags: [mlx, gemma4, d1a, decision-model, structured-output, classification, appl
 
 # D1A-E4B v0.6 · MLX 8-bit
 
+**v0.6.1 (2026-10-13)** is v0.6 with one addition, a routing temperature: requests sent with `"use_case": "routing"` are served at T 1.18 instead of 1.95, which brings factory routing's calibration error from 0.093 to 0.026 without changing any answer. The weights and every result below are v0.6's.
+
 **A small decision model for Apple Silicon: give it a document and a few questions, and it picks the answers.** Each question
 is yes/no, multiple choice or a score; D1A returns a calibrated probability for every option in one forward pass, with no
 generated text, so every answer is a valid option. About 0.6 s per request on an M4 Mac mini, 6 GB on disk.
@@ -20,7 +22,7 @@ small pointer head. Questions in English; documents in English or Japanese.
 
 ```python
 from d1a import D1A   # pip install "d1a[serve,media] @ git+https://github.com/jonpol01/d1a@v0.5.0"
-m = D1A.load("mlx-community/d1a-e4b-8bit@v0.6")
+m = D1A.load("mlx-community/d1a-e4b-8bit@v0.6.1")
 m.decide("The customer was charged twice for the same order and wants a refund today.",
          {"team": {"type": "choice", "instructions": "Which team handles this?",
                    "criteria": {"billing": None, "shipping": None, "support": None}},
@@ -134,10 +136,10 @@ On Apple Silicon (the PyTorch checkpoint, for NVIDIA GPUs and CPUs, is [JohnP1/d
 
 ```bash
 pip install "d1a[serve,media] @ git+https://github.com/jonpol01/d1a@v0.5.0"
-python -m d1a.serving.serve --run mlx-community/d1a-e4b-8bit@v0.6 --port 8009
+python -m d1a.serving.serve --run mlx-community/d1a-e4b-8bit@v0.6.1 --port 8009
 ```
 
-or in-process: `from d1a import D1A; D1A.load("mlx-community/d1a-e4b-8bit@v0.6").decide(state, questions)`. For hard-v1-style reasoning or the older PR-label conventions, load `JohnP1/d1a-e4b-mlx-q8@v0.5`.
+or in-process: `from d1a import D1A; D1A.load("mlx-community/d1a-e4b-8bit@v0.6.1").decide(state, questions)`. For hard-v1-style reasoning or the older PR-label conventions, load `JohnP1/d1a-e4b-mlx-q8@v0.5`.
 
 ## Limits
 
@@ -145,7 +147,7 @@ or in-process: `from d1a import D1A; D1A.load("mlx-community/d1a-e4b-8bit@v0.6")
 - **Change type on older PRs is worse**: 85.3% against 89.4% on the frozen English test set, and 72.5% against 76.2% on the Japanese one (all questions, 92 PRs). It calls more PRs type/bug than v0.5 did. On current PRs it is better (above).
 - **Blast radius on the owner's own repositories is worse**: 67% against 74% on 39 hand-checked PRs (1 fix, 4 breaks). v0.6 was not trained for blast radius; on the larger frozen test set it is 62% against 58%.
 - Severity: on the 487 newer PRs, P2 recall falls (0.55 against v0.5's 0.79) while P3 and P0 rise; P0 rests on 7 PRs. Severity on the older development set is 1.8 points lower.
-- One temperature for every question, and no use-case temperatures (routing's is planned for v0.6.1). Calibration error is higher than v0.5's on JGLUE, dates and typesafe-v1.
+- One temperature for every question except routing: from v0.6.1, requests sent with `"use_case": "routing"` are served at T 1.18. Calibration error is higher than v0.5's on JGLUE, dates and typesafe-v1.
 - Smaller losses against v0.5: decision-v7 −1.4, wanli-v1 −2.3, one of 45 hand-labelled routing questions.
 - The recent PR labels come from one large open-source project and a few days of it; if its conventions move again, so will these numbers.
 - Questions are in English; the documents can be English or Japanese.
@@ -161,6 +163,7 @@ Earlier versions (every tag below), the PyTorch checkpoint and the Core ML build
 | `v0.3` | pull-request labeling, English and Japanese |
 | `v0.4` | pull-request labeling round 2: severity, Japanese, less biased blast radius |
 | `v0.5` | Kev's harder skills back (hard, developer-tool and long-document decisions), PR labels held |
+| **`v0.6.1`** | **v0.6's weights with a routing temperature**: `"use_case": "routing"` is served at T 1.18 (others keep 1.95); factory routing calibration error 0.093 → 0.026, answered at ≥ 0.9 confidence 50% → 76%, no answer changed (jonpol01/d1a#264) |
 | **`v0.6`** | **current pull-request labels (severity +4.3, type +3.9 on newer PRs); a release exception: hard decisions −2.7, so pin `v0.5` for those** |
 
 Older tag names (`v0.1-2epoch`, `v0.1.1-2epoch-calibrated`, `v0.2-hybrid`) still work.

d1a_config.json

@@ -1,38 +1,128 @@
 {
-  "format": "d1a-mlx",
-  "format_version": 1,
-  "base": "google/gemma-4-E4B",
-  "base_revision": "411aa17b749aa952df1359d2dcea73917a544d9a",
-  "source": "JohnP1/d1a-e4b",
-  "source_revision": "3ca01b824280cacf027826b0ded8632f242a7e8b",
-  "adapter_sha256": "739ce962c3ae630243ee53b212a822e8b2feee41080b68d5b460fd0ebb19e4ea",
-  "lora": 16,
-  "head_dim": 256,
-  "hidden_size": 2560,
-  "temperature": 1.9543199368684914,
-  "leading_ids": [
-    2
-  ],
-  "bos_id": 2,
-  "delimiter_ids": [
-    6,
-    7,
-    8,
-    9,
-    10
-  ],
-  "pad_id": 0,
-  "dtype": "bfloat16",
-  "quantization": {
-    "group_size": 64,
-    "bits": 8,
-    "mode": "affine",
-    "language_model.model.embed_tokens_per_layer": {
-      "group_size": 64,
-      "bits": 4,
-      "mode": "affine"
-    },
-    "embeddings": true
+ "format": "d1a-mlx",
+ "format_version": 1,
+ "base": "google/gemma-4-E4B",
+ "base_revision": "411aa17b749aa952df1359d2dcea73917a544d9a",
+ "source": "JohnP1/d1a-e4b",
+ "source_revision": "3ca01b824280cacf027826b0ded8632f242a7e8b",
+ "adapter_sha256": "739ce962c3ae630243ee53b212a822e8b2feee41080b68d5b460fd0ebb19e4ea",
+ "lora": 16,
+ "head_dim": 256,
+ "hidden_size": 2560,
+ "temperature": 1.9543199368684914,
+ "leading_ids": [
+  2
+ ],
+ "bos_id": 2,
+ "delimiter_ids": [
+  6,
+  7,
+  8,
+  9,
+  10
+ ],
+ "pad_id": 0,
+ "dtype": "bfloat16",
+ "quantization": {
+  "group_size": 64,
+  "bits": 8,
+  "mode": "affine",
+  "language_model.model.embed_tokens_per_layer": {
+   "group_size": 64,
+   "bits": 4,
+   "mode": "affine"
   },
-  "head": "head.safetensors"
+  "embeddings": true
+ },
+ "head": "head.safetensors",
+ "use_case_temperatures": {
+  "routing": 1.1755479062836087
+ },
+ "use_case_temperature_fits": {
+  "routing": {
+   "rows": [
+    "phase-d3/gate/suites/routing_generic-development/head/rows.json",
+    "phase-d3/gate/suites/routing_handlabelled-45/head/rows.json"
+   ],
+   "n": 315,
+   "method": "min micro mean NLL over a 121-point log grid 0.25..4",
+   "cross_validation": {
+    "raw": {
+     "n": 315,
+     "ece": 0.005376748980220513,
+     "brier": 0.02213792789102334,
+     "nll": 0.037784826756766074,
+     "confident_error_rate": 0.0031746031746031746,
+     "coverage_at_5pct_error": 1.0
+    },
+    "out_of_fold": {
+     "n": 315,
+     "ece": 0.008392239537266366,
+     "brier": 0.021994761576082954,
+     "nll": 0.0385561510340784,
+     "confident_error_rate": 0.0031746031746031746,
+     "coverage_at_5pct_error": 1.0
+    },
+    "ece_ci95": {
+     "raw": [
+      0.003303220868466807,
+      0.018871250910842064
+     ],
+     "out_of_fold": [
+      0.0058853472476678124,
+      0.022409565392811873
+     ],
+     "delta": [
+      -0.001788803325547818,
+      0.007955193851064562
+     ]
+    },
+    "separated": false,
+    "temperatures": [
+     1.2311444133449163,
+     1.148698354997035,
+     1.2893703084395791,
+     1.2311444133449163,
+     1.0233738919967748
+    ],
+    "folds": 5,
+    "seed": 0,
+    "samples": 1000,
+    "groups": 315,
+    "unit": "source-stratified (source, group); sibling questions and variants stay together"
+   },
+   "interval": {
+    "level": 0.9,
+    "samples": 2000,
+    "seed": 0,
+    "lower": 0.6744039221992122,
+    "upper": 1.5157165665103978,
+    "questions": 315
+   },
+   "fit_rows": [
+    {
+     "rows": "phase-d3/gate/suites/routing_generic-development/head/rows.json",
+     "suite": "evals/d1a/routing",
+     "split": "generic-development",
+     "questions": 270
+    },
+    {
+     "rows": "phase-d3/gate/suites/routing_handlabelled-45/head/rows.json",
+     "suite": "evals/d1a/routing",
+     "split": "handlabelled-45",
+     "questions": 45
+    }
+   ],
+   "training_suites": [
+    "evals/v7/decision-v7"
+   ],
+   "training_partitions": [],
+   "in_distribution": {
+    "allowed": true,
+    "problems": [
+     "cannot list the sources of --data train.jsonl (outside evals/, no sidecar train.jsonl.json naming its sources), training data of routing-temp-v061/stub/d1a-e4b"
+    ]
+   }
+  }
+ }
 }