Describe available model formats and measured results directly

README.md

@@ -134,50 +134,28 @@ Embeddings, the vocabulary head, and other non-projection tensors retain BF16 pr
 
 ## Evaluation
 
-### NVFP4 quantization validation
+### NVFP4 validation
 
-The current NVFP4 weights use NVIDIA Model Optimizer. This comparison uses 1,024 validation questions and the same merged BF16 source model.
-Calibration uses separate training examples. Validation accuracy helped select the quantization recipe, so these results are not an independent test.
+The NVFP4 model scores **78.52% accuracy**, **0.2569 Brier error**, and **0.9667 cross-entropy** on 1,024 validation questions.
+Quantization uses 256 separate training examples, with no shared question IDs or source groups between calibration and validation.
+Validation guided quantization selection, so these results are not an independent test.
+[Validation details](evaluation/modelopt-validation.json) identify the weights, task mix, and scoring results.
 
-| Quantization | Accuracy | Brier error ↓ | Cross-entropy ↓ |
-|---|---:|---:|---:|
-| Previous weight-MSE export | 76.17% | 0.2779 | 0.9928 |
-| **Current ModelOpt export** | **78.52%** | **0.2569** | **0.9667** |
-
-[Validation results and weight hashes](evaluation/modelopt-validation.json) identify both exports.
-Quantization calibration selects weight scales. It does not calibrate the returned answer probabilities, which remain uncalibrated.
-
-### Published 512-question comparisons
-
-**The NVFP4 results in the following tables and chart describe the previous weight-MSE export, not the current ModelOpt weights.**
-The Q8_0 weights and their results are unchanged. The replacement NVFP4 weights have not yet been scored on this 512-question set.
+### Q8_0 test and hosted-model comparisons
 
 The [512-question test set](evaluation/questions-512.json) includes the exact inputs, answer options, and target distributions used in all tables below.
 [Scoring rules and source attribution](evaluation/README.md) explain how to reproduce the results.
 
-Both base-versus-fine-tune comparisons use the same 512 questions, two recurrent loops, the same typed-decision prompts, and no probability calibration.
-Within each comparison, base Dory and the fine-tune use the same quantization and runtime.
+The base-versus-fine-tune comparison uses Q8_0 weights, two recurrent loops, identical typed-decision prompts, and no probability calibration.
+Both models run through llama.cpp.
 
 The model was selected using a separate 1,024-question validation set before the test questions were scored.
 
-### NVFP4 — evaluated with Eider
-
-Base Dory is the original NVIDIA BF16 model quantized with the same NVFP4 recipe used for this fine-tune.
-These results measure decisions with the supplied prompt format and allowed-answer scoring. They do not measure the base model with other prompting methods.
-[Comparison results and hashes](evaluation/base-comparison.json) record the matched setup.
-
-| Test task | Questions | Base Dory accuracy | Fine-tuned accuracy | Base Brier ↓ | Fine-tuned Brier ↓ |
-|---|---:|---:|---:|---:|---:|
-| SCONE state tracking | 192 | 29.69% | 75.52% | 1.0003 | 0.3779 |
-| Block-stacking and delivery planning | 192 | 42.71% | 92.19% | 0.6045 | 0.1909 |
-| HelpSteer2 human judgements | 128 | 47.66% | 50.78% | 0.3385 | 0.2290 |
-| **Combined** | 512 | 39.06% | 75.59% | 0.6865 | 0.2706 |
-
 ### Q8_0 GGUF — evaluated with llama.cpp
 
 The native `/v1/systemone` endpoint scored both models with Q8_0 weights.
-All 512 questions use identical prompt tokens and answer labels across both runtimes.
-This measures fine-tuning under llama.cpp; differences from the NVFP4 table can reflect both quantization and runtime arithmetic.
+Base and fine-tuned Dory receive identical prompt tokens and answer labels.
+These results measure the Q8_0 model with the supplied decision template and allowed-answer scoring.
 
 | Test task | Questions | Base Dory accuracy | Fine-tuned accuracy | Base Brier ↓ | Fine-tuned Brier ↓ |
 |---|---:|---:|---:|---:|---:|
@@ -192,7 +170,7 @@ Accuracy counts any answer tied for the highest target probability as correct.
 Brier error compares the complete predicted and target distributions.
 These are adapted decision tasks, not scores under the source datasets' original benchmark protocols.
 
-The BF16 weights have not received the same accuracy evaluation.
+These 512-question test results apply to Q8_0. BF16 and NVFP4 have not been measured on this test set.
 
 ### Comparison with hosted decision models
 
@@ -205,16 +183,14 @@ Hosted models used their providers' prompt templates and probability settings; n
 | Model | State tracking | Planning | Human judgements | Combined | Brier ↓ |
 |---|---:|---:|---:|---:|---:|
 | Base Dory — Q8_0 / llama.cpp | 31.77% | 38.02% | 49.22% | 38.48% | 0.6736 |
-| Base Dory — NVFP4 / Eider | 29.69% | 42.71% | 47.66% | 39.06% | 0.6865 |
 | Dory Typed Decisions — Q8_0 / llama.cpp | **76.04%** | **96.88%** | 51.56% | **77.73%** | **0.2599** |
-| Dory Typed Decisions — NVFP4 / Eider | 75.52% | 92.19% | 50.78% | 75.59% | 0.2706 |
 | [TypeSafe Jev 1.14](https://openrouter.ai/typesafe/jev-1.14) | 64.58% | 74.48% | 55.47% | 66.02% | 0.3843 |
 | [Kev 4B](https://openrouter.ai/jaredpalmer/kev-4b) | 38.54% | 36.46% | 57.81% | 42.58% | 0.5134 |
 | [Liquid d1](https://openrouter.ai/liquid/d1) | 58.33% | 79.17% | **58.59%** | 66.21% | 0.4156 |
 | [Perplexity Decider 1.2](https://openrouter.ai/perplexity/pplx-decider-v1.2) | 72.92% | 88.02% | 57.03% | 74.61% | 0.3178 |
 
 Accuracy columns show percentages; **bold** marks the best result in each column. Combined accuracy weights all 512 questions equally.
-All four hosted models score higher than this fine-tune on human judgements. Jev scores 55.47%, compared with 51.56% for Q8_0 and 50.78% for NVFP4.
+All four hosted models score higher than this fine-tune on human judgements. Jev scores 55.47%, compared with 51.56% for this model in Q8_0.
 The fine-tune scores higher on state tracking and planning.
 
 This test measures new examples from task families included in this fine-tune's training. It does not establish a general ranking of decision models.

evaluation/README.md

@@ -1,8 +1,7 @@
 # Decision evaluation: 512 questions
 
-The NVFP4 scores and chart in this directory describe the previous weight-MSE export.
-The current ModelOpt export has a separate [1,024-question validation report](modelopt-validation.json).
-Its 512-question test results are not yet available. Q8_0 and hosted-model results are unchanged.
+The model card compares Q8_0 Dory with hosted models on these questions.
+[NVFP4 validation results](modelopt-validation.json) use a separate 1,024-question set.
 
 [Download the questions and targets](questions-512.json).
 This is the exact file used for the base Dory, fine-tuned Dory, and hosted-model comparisons in the model card.
@@ -103,6 +102,5 @@ Collection and question-format adaptations are provided under CC BY-SA 4.0; the
 
 ## Results
 
-- [Base versus fine-tuned Dory, NVFP4 / Eider](base-comparison.json)
 - [Base versus fine-tuned Dory, Q8_0 / llama.cpp](llama-comparison.json)
 - [Hosted decision models on the same questions](openrouter-comparison.json)

evaluation/modelopt-validation.json

@@ -6,70 +6,31 @@
   "probability_calibration": false,
   "calibration_examples": 256,
   "calibration_source": "Training examples disjoint from validation by ID and source group",
-  "weights_sha256": {
-    "previous": "9bc834f4dce29a48499b6614c11772d189710138b29c637a384f753b1326b4c0",
-    "modelopt": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4"
-  },
+  "weights_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
   "metrics": {
-    "current": {
-      "questions": 1024,
-      "correct": 0.76171875,
-      "brier": 0.27786017825450965,
-      "cross_entropy": 0.9927829125866763
-    },
-    "modelopt": {
-      "questions": 1024,
-      "correct": 0.78515625,
-      "brier": 0.256938595885194,
-      "cross_entropy": 0.9667092188545374
-    }
-  },
-  "paired": {
-    "modelopt_only_correct": 58,
-    "current_only_correct": 34
+    "questions": 1024,
+    "correct": 0.78515625,
+    "brier": 0.256938595885194,
+    "cross_entropy": 0.9667092188545374
   },
   "by_kind": {
     "choice": {
-      "current": {
-        "questions": 665,
-        "correct": 0.7654135338345864,
-        "brier": 0.3189465668614991,
-        "cross_entropy": 1.0534065611001757
-      },
-      "modelopt": {
-        "questions": 665,
-        "correct": 0.7819548872180451,
-        "brier": 0.29931089665593197,
-        "cross_entropy": 1.021091694949712
-      }
+      "questions": 665,
+      "correct": 0.7819548872180451,
+      "brier": 0.29931089665593197,
+      "cross_entropy": 1.021091694949712
     },
     "noul": {
-      "current": {
-        "questions": 144,
-        "correct": 0.8958333333333334,
-        "brier": 0.16471210037099535,
-        "cross_entropy": 0.39482479972079665
-      },
-      "modelopt": {
-        "questions": 144,
-        "correct": 0.9097222222222222,
-        "brier": 0.146691427234091,
-        "cross_entropy": 0.4264544071293922
-      }
+      "questions": 144,
+      "correct": 0.9097222222222222,
+      "brier": 0.146691427234091,
+      "cross_entropy": 0.4264544071293922
     },
     "score": {
-      "current": {
-        "questions": 215,
-        "correct": 0.6604651162790698,
-        "brier": 0.226561921471152,
-        "cross_entropy": 1.2057654334760235
-      },
-      "modelopt": {
-        "questions": 215,
-        "correct": 0.7116279069767442,
-        "brier": 0.19972004831876652,
-        "cross_entropy": 1.160348038785374
-      }
+      "questions": 215,
+      "correct": 0.7116279069767442,
+      "brier": 0.19972004831876652,
+      "cross_entropy": 1.160348038785374
     }
   }
 }

nvfp4/evaluation/README.md

@@ -1,4 +1,4 @@
-# Evaluation scope
+# NVFP4 evaluation
 
-These reports describe the previous NVFP4 export, with SHA-256 `9bc834f4dce29a48499b6614c11772d189710138b29c637a384f753b1326b4c0`.
-For the current weights, see [ModelOpt validation](../../evaluation/modelopt-validation.json) and [export metadata](../export.json).
+[Validation results](../../evaluation/modelopt-validation.json) report accuracy and probability error for these weights.
+[Export metadata](../export.json) records the weight hashes and quantization settings.