v0.6.1: routing temperature (use_case_temperatures.routing), same weights as v0.6 (jonpol01/d1a#264)

README.md

@@ -10,6 +10,8 @@ tags: [mlx, gemma4, d1a, decision-model, structured-output, classification, appl
 
 # D1A-E2B v0.6 · MLX 8-bit
 
+**v0.6.1 (2026-10-13)** is v0.6 with one addition, a routing temperature: requests sent with `"use_case": "routing"` are served at T 0.95 instead of 1.45, which brings factory routing's calibration error from 0.091 to 0.035 without changing any answer. The weights and every result below are v0.6's.
+
 **A small decision model for Apple Silicon: give it a document and a few questions, and it picks the answers.** Each question
 is yes/no, multiple choice or a score; D1A returns a calibrated probability for every option in one forward pass, with no
 generated text, so every answer is a valid option. About 0.2 s per request on an M4 Mac mini, 3.5 GB on disk.
@@ -20,7 +22,7 @@ small pointer head. Questions in English; documents in English or Japanese.
 
 ```python
 from d1a import D1A   # pip install "d1a[serve,media] @ git+https://github.com/jonpol01/d1a@v0.5.0"
-m = D1A.load("mlx-community/d1a-e2b-8bit@v0.6")
+m = D1A.load("mlx-community/d1a-e2b-8bit@v0.6.1")
 m.decide("The customer was charged twice for the same order and wants a refund today.",
          {"team": {"type": "choice", "instructions": "Which team handles this?",
                    "criteria": {"billing": None, "shipping": None, "support": None}},
@@ -130,10 +132,10 @@ The [PR-labeler recipe](https://github.com/jonpol01/d1a/tree/main/recipes/pr-lab
 
 ```bash
 pip install "d1a[serve,media] @ git+https://github.com/jonpol01/d1a@v0.5.0"
-python -m d1a.serving.serve --run mlx-community/d1a-e2b-8bit@v0.6 --port 8009
+python -m d1a.serving.serve --run mlx-community/d1a-e2b-8bit@v0.6.1 --port 8009
 ```
 
-or in-process: `from d1a import D1A; D1A.load("mlx-community/d1a-e2b-8bit@v0.6").decide(state, questions)`. The PyTorch checkpoint is [JohnP1/d1a-e2b](https://huggingface.co/JohnP1/d1a-e2b) `v0.6`. For general decisions only, `@v0.2.1-2epoch-calibrated` is 1.4 points better on decision-v7.
+or in-process: `from d1a import D1A; D1A.load("mlx-community/d1a-e2b-8bit@v0.6.1").decide(state, questions)`. The PyTorch checkpoint is [JohnP1/d1a-e2b](https://huggingface.co/JohnP1/d1a-e2b) `v0.6`. For general decisions only, `@v0.2.1-2epoch-calibrated` is 1.4 points better on decision-v7.
 
 ## Limits
 
@@ -145,7 +147,7 @@ or in-process: `from d1a import D1A; D1A.load("mlx-community/d1a-e2b-8bit@v0.6")
 - **Media**: zero-shot, measured on 14 demo requests only. It calls 2 of 3 intact photos damaged, more often than E4B v0.6 does (above).
 - **The playground's evals demo** grades answers less sharply than v0.2.1 did. For the English "good answer", error goes from 0.088 to 0.153 and quality from 3.81 to 3.51 (out of 5).
 - On the owner's review-bot replay it calls security 5 times, where v0.2.1 called it once.
-- One temperature for every question, and no use-case temperatures.
+- One temperature for every question except routing: from v0.6.1, requests sent with `"use_case": "routing"` are served at T 0.95.
 - The recent PR labels come from one large open-source project and a few days of it. If its conventions move again, so will these numbers.
 - Questions are in English. The documents can be English or Japanese.
 
@@ -158,6 +160,7 @@ Earlier versions (every tag below), the PyTorch checkpoint and the Core ML build
 | `v0.1`, `v0.1-1epoch` | general decisions (decision-v7, 1 epoch) |
 | `v0.2`, `v0.2-2epoch` | general decisions, 2 epochs |
 | `v0.2.1-2epoch-calibrated` | the same weights with a calibration temperature (1.52) |
+| **`v0.6.1`** | **v0.6's weights with a routing temperature**: `"use_case": "routing"` is served at T 0.95 (others keep 1.45); factory routing calibration error 0.091 → 0.035, answered at ≥ 0.9 confidence 34% → 55%, no answer changed (jonpol01/d1a#264) |
 | **`v0.6`** | **every later D1A skill in one run: PR labels, hard, developer-tool and long-document decisions, Japanese, agent routing; a release exception (decision-v7 −1.4)** |
 
 ## License and data

d1a_config.json

@@ -1,142 +1,246 @@
 {
-  "format": "d1a-mlx",
-  "format_version": 1,
-  "base": "google/gemma-4-E2B",
-  "base_revision": "d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f",
-  "source": "JohnP1/d1a-e2b",
-  "source_revision": "5517e8213504de567175e0b644b9c674c91af771",
-  "adapter_sha256": "247f222314c8ceee158140d38a0be1c304da81719034ef6e71ee02506e334c1b",
-  "lora": 16,
-  "head_dim": 256,
-  "hidden_size": 1536,
-  "temperature": 1.447269237440378,
-  "leading_ids": [
-    2
+ "format": "d1a-mlx",
+ "format_version": 1,
+ "base": "google/gemma-4-E2B",
+ "base_revision": "d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f",
+ "source": "JohnP1/d1a-e2b",
+ "source_revision": "5517e8213504de567175e0b644b9c674c91af771",
+ "adapter_sha256": "247f222314c8ceee158140d38a0be1c304da81719034ef6e71ee02506e334c1b",
+ "lora": 16,
+ "head_dim": 256,
+ "hidden_size": 1536,
+ "temperature": 1.447269237440378,
+ "leading_ids": [
+  2
+ ],
+ "bos_id": 2,
+ "delimiter_ids": [
+  6,
+  7,
+  8,
+  9,
+  10
+ ],
+ "pad_id": 0,
+ "dtype": "bfloat16",
+ "quantization": {
+  "group_size": 64,
+  "bits": 8,
+  "mode": "affine",
+  "language_model.model.embed_tokens_per_layer": {
+   "group_size": 64,
+   "bits": 4,
+   "mode": "affine"
+  },
+  "embeddings": true
+ },
+ "head": "head.safetensors",
+ "temperature_fit": {
+  "rows": [
+   "cal/rows.json",
+   "calpr/rows.json"
+  ],
+  "n": 3178,
+  "method": "min micro mean NLL over a 121-point log grid 0.25..4",
+  "cross_validation": {
+   "raw": {
+    "n": 3178,
+    "ece": 0.0505561988092814,
+    "brier": 0.2952905648331223,
+    "nll": 0.5648973549274169,
+    "confident_error_rate": 0.02957835116425425,
+    "coverage_at_5pct_error": 0.5613593455003146
+   },
+   "out_of_fold": {
+    "n": 3178,
+    "ece": 0.023704548579686884,
+    "brier": 0.2896952220920997,
+    "nll": 0.5324486179012022,
+    "confident_error_rate": 0.017935808684707365,
+    "coverage_at_5pct_error": 0.5556954059156702
+   },
+   "ece_ci95": {
+    "raw": [
+     0.04102750015575881,
+     0.06377306530700594
+    ],
+    "out_of_fold": [
+     0.016511069955034523,
+     0.03860363764230206
+    ],
+    "delta": [
+     -0.04016350379193325,
+     -0.010727297381358629
+    ]
+   },
+   "separated": true,
+   "temperatures": [
+    1.447269237440378,
+    1.447269237440378,
+    1.4142135623730947,
+    1.4810975522865641,
+    1.447269237440378
+   ],
+   "folds": 5,
+   "seed": 0,
+   "samples": 1000,
+   "groups": 1670,
+   "unit": "source-stratified (source, group); sibling questions and variants stay together"
+  },
+  "interval": {
+   "level": 0.9,
+   "samples": 2000,
+   "seed": 0,
+   "lower": 1.381912879967776,
+   "upper": 1.5157165665103978,
+   "questions": 3178
+  },
+  "fit_rows": [
+   {
+    "rows": "cal/rows.json",
+    "suite": "evals/v7/decision-v7",
+    "split": "calibration",
+    "questions": 1148
+   },
+   {
+    "rows": "calpr/rows.json",
+    "suite": "evals/d1a/pr-labels",
+    "split": "development",
+    "questions": 2030
+   }
+  ],
+  "training_suites": [
+   "evals/devtools-v1",
+   "evals/documents-v1",
+   "evals/hard-v1",
+   "evals/v7/decision-v7"
   ],
-  "bos_id": 2,
-  "delimiter_ids": [
-    6,
-    7,
-    8,
-    9,
-    10
+  "training_partitions": [
+   "evals/d1a/ja-jglue:train",
+   "evals/d1a/pr-labels:train",
+   "evals/d1a/pr-labels:train-blast",
+   "evals/d1a/pr-labels:train-ja",
+   "evals/d1a/routing:factory-train",
+   "evals/d1a/routing:generic-train"
   ],
-  "pad_id": 0,
-  "dtype": "bfloat16",
-  "quantization": {
-    "group_size": 64,
-    "bits": 8,
-    "mode": "affine",
-    "language_model.model.embed_tokens_per_layer": {
-      "group_size": 64,
-      "bits": 4,
-      "mode": "affine"
+  "same_corpus": [
+   "calpr/rows.json: SAME CORPUS: evals/d1a/pr-labels:development is an eval partition of evals/d1a/pr-labels, whose train partitions ['evals/d1a/pr-labels:train', 'evals/d1a/pr-labels:train-blast', 'evals/d1a/pr-labels:train-ja'] the checkpoint trained on"
+  ],
+  "in_distribution": {
+   "allowed": true,
+   "problems": [
+    "cal/rows.json: evals/v7/decision-v7 is training data of the checkpoint",
+    "cal/rows.json: it pools 12 training source(s) ['agnews', 'amazon', 'banking77', 'boolq', 'compositional', 'dbpedia14', 'imdb', 'legacy_policy'] ...",
+    "cal/rows.json: it reads the calibration partition of evals/v7/decision-v7, a training corpus",
+    "calpr/rows.json: SAME CORPUS: evals/d1a/pr-labels:development is an eval partition of evals/d1a/pr-labels, whose train partitions ['evals/d1a/pr-labels:train', 'evals/d1a/pr-labels:train-blast', 'evals/d1a/pr-labels:train-ja'] the checkpoint trained on",
+    "cannot list the sources of --data data/train-e2b-v06.jsonl (outside evals/, no sidecar train-e2b-v06.jsonl.json naming its sources), training data of d1"
+   ]
+  }
+ },
+ "use_case_temperatures": {
+  "routing": 0.9548416039104165
+ },
+ "use_case_temperature_fits": {
+  "routing": {
+   "rows": [
+    "e2b-v06/gate/suites/routing_generic-development/head/rows.json",
+    "e2b-v06/gate/suites/routing_handlabelled-45/head/rows.json"
+   ],
+   "n": 315,
+   "method": "min micro mean NLL over a 121-point log grid 0.25..4",
+   "cross_validation": {
+    "raw": {
+     "n": 315,
+     "ece": 0.018499680832099685,
+     "brier": 0.03799074208801305,
+     "nll": 0.06040660176682812,
+     "confident_error_rate": 0.0,
+     "coverage_at_5pct_error": 1.0
     },
-    "embeddings": true
-  },
-  "head": "head.safetensors",
-  "temperature_fit": {
-    "rows": [
-      "cal/rows.json",
-      "calpr/rows.json"
-    ],
-    "n": 3178,
-    "method": "min micro mean NLL over a 121-point log grid 0.25..4",
-    "cross_validation": {
-      "raw": {
-        "n": 3178,
-        "ece": 0.0505561988092814,
-        "brier": 0.2952905648331223,
-        "nll": 0.5648973549274169,
-        "confident_error_rate": 0.02957835116425425,
-        "coverage_at_5pct_error": 0.5613593455003146
-      },
-      "out_of_fold": {
-        "n": 3178,
-        "ece": 0.023704548579686884,
-        "brier": 0.2896952220920997,
-        "nll": 0.5324486179012022,
-        "confident_error_rate": 0.017935808684707365,
-        "coverage_at_5pct_error": 0.5556954059156702
-      },
-      "ece_ci95": {
-        "raw": [
-          0.04102750015575881,
-          0.06377306530700594
-        ],
-        "out_of_fold": [
-          0.016511069955034523,
-          0.03860363764230206
-        ],
-        "delta": [
-          -0.04016350379193325,
-          -0.010727297381358629
-        ]
-      },
-      "separated": true,
-      "temperatures": [
-        1.447269237440378,
-        1.447269237440378,
-        1.4142135623730947,
-        1.4810975522865641,
-        1.447269237440378
-      ],
-      "folds": 5,
-      "seed": 0,
-      "samples": 1000,
-      "groups": 1670,
-      "unit": "source-stratified (source, group); sibling questions and variants stay together"
+    "out_of_fold": {
+     "n": 315,
+     "ece": 0.013612898878925516,
+     "brier": 0.03937697895309676,
+     "nll": 0.06225521527784288,
+     "confident_error_rate": 0.0031746031746031746,
+     "coverage_at_5pct_error": 1.0
     },
-    "interval": {
-      "level": 0.9,
-      "samples": 2000,
-      "seed": 0,
-      "lower": 1.381912879967776,
-      "upper": 1.5157165665103978,
-      "questions": 3178
+    "ece_ci95": {
+     "raw": [
+      0.01077328946933042,
+      0.034931159728294615
+     ],
+     "out_of_fold": [
+      0.0067492580757402216,
+      0.03106535692064003
+     ],
+     "delta": [
+      -0.010430770740017884,
+      0.002236456299991558
+     ]
     },
-    "fit_rows": [
-      {
-        "rows": "cal/rows.json",
-        "suite": "evals/v7/decision-v7",
-        "split": "calibration",
-        "questions": 1148
-      },
-      {
-        "rows": "calpr/rows.json",
-        "suite": "evals/d1a/pr-labels",
-        "split": "development",
-        "questions": 2030
-      }
-    ],
-    "training_suites": [
-      "evals/devtools-v1",
-      "evals/documents-v1",
-      "evals/hard-v1",
-      "evals/v7/decision-v7"
+    "separated": false,
+    "temperatures": [
+     1.0,
+     0.8705505632961241,
+     0.9771599684342458,
+     0.8705505632961241,
+     1.0
     ],
-    "training_partitions": [
-      "evals/d1a/ja-jglue:train",
-      "evals/d1a/pr-labels:train",
-      "evals/d1a/pr-labels:train-blast",
-      "evals/d1a/pr-labels:train-ja",
-      "evals/d1a/routing:factory-train",
-      "evals/d1a/routing:generic-train"
-    ],
-    "same_corpus": [
-      "calpr/rows.json: SAME CORPUS: evals/d1a/pr-labels:development is an eval partition of evals/d1a/pr-labels, whose train partitions ['evals/d1a/pr-labels:train', 'evals/d1a/pr-labels:train-blast', 'evals/d1a/pr-labels:train-ja'] the checkpoint trained on"
-    ],
-    "in_distribution": {
-      "allowed": true,
-      "problems": [
-        "cal/rows.json: evals/v7/decision-v7 is training data of the checkpoint",
-        "cal/rows.json: it pools 12 training source(s) ['agnews', 'amazon', 'banking77', 'boolq', 'compositional', 'dbpedia14', 'imdb', 'legacy_policy'] ...",
-        "cal/rows.json: it reads the calibration partition of evals/v7/decision-v7, a training corpus",
-        "calpr/rows.json: SAME CORPUS: evals/d1a/pr-labels:development is an eval partition of evals/d1a/pr-labels, whose train partitions ['evals/d1a/pr-labels:train', 'evals/d1a/pr-labels:train-blast', 'evals/d1a/pr-labels:train-ja'] the checkpoint trained on",
-        "cannot list the sources of --data data/train-e2b-v06.jsonl (outside evals/, no sidecar train-e2b-v06.jsonl.json naming its sources), training data of d1"
-      ]
+    "folds": 5,
+    "seed": 0,
+    "samples": 1000,
+    "groups": 315,
+    "unit": "source-stratified (source, group); sibling questions and variants stay together"
+   },
+   "interval": {
+    "level": 0.9,
+    "samples": 2000,
+    "seed": 0,
+    "lower": 0.7405487761432822,
+    "upper": 1.1224620483093728,
+    "questions": 315
+   },
+   "fit_rows": [
+    {
+     "rows": "e2b-v06/gate/suites/routing_generic-development/head/rows.json",
+     "suite": "evals/d1a/routing",
+     "split": "generic-development",
+     "questions": 270
+    },
+    {
+     "rows": "e2b-v06/gate/suites/routing_handlabelled-45/head/rows.json",
+     "suite": "evals/d1a/routing",
+     "split": "handlabelled-45",
+     "questions": 45
     }
-  },
-  "use_case_temperatures": {},
-  "use_case_temperature_fits": {}
+   ],
+   "training_suites": [
+    "evals/devtools-v1",
+    "evals/documents-v1",
+    "evals/hard-v1",
+    "evals/v7/decision-v7"
+   ],
+   "training_partitions": [
+    "evals/d1a/ja-jglue:train",
+    "evals/d1a/pr-labels:train",
+    "evals/d1a/pr-labels:train-blast",
+    "evals/d1a/pr-labels:train-ja",
+    "evals/d1a/routing:factory-train",
+    "evals/d1a/routing:generic-train"
+   ],
+   "same_corpus": [
+    "e2b-v06/gate/suites/routing_generic-development/head/rows.json: SAME CORPUS: evals/d1a/routing:generic-development is an eval partition of evals/d1a/routing, whose train partitions ['evals/d1a/routing:factory-train', 'evals/d1a/routing:generic-train'] the checkpoint trained on",
+    "e2b-v06/gate/suites/routing_handlabelled-45/head/rows.json: SAME CORPUS: evals/d1a/routing:handlabelled-45 is an eval partition of evals/d1a/routing, whose train partitions ['evals/d1a/routing:factory-train', 'evals/d1a/routing:generic-train'] the checkpoint trained on"
+   ],
+   "in_distribution": {
+    "allowed": true,
+    "problems": [
+     "e2b-v06/gate/suites/routing_generic-development/head/rows.json: SAME CORPUS: evals/d1a/routing:generic-development is an eval partition of evals/d1a/routing, whose train partitions ['evals/d1a/routing:factory-train', 'evals/d1a/routing:generic-train'] the checkpoint trained on",
+     "e2b-v06/gate/suites/routing_handlabelled-45/head/rows.json: SAME CORPUS: evals/d1a/routing:handlabelled-45 is an eval partition of evals/d1a/routing, whose train partitions ['evals/d1a/routing:factory-train', 'evals/d1a/routing:generic-train'] the checkpoint trained on",
+     "cannot list the sources of --data data/train-e2b-v06.jsonl (outside evals/, no sidecar train-e2b-v06.jsonl.json naming its sources), training data of routing-temp-v061/stub/d1a-e2b"
+    ]
+   }
+  }
+ }
 }