Publish Decisive Dory comparisons and JevMan charts
Browse files- .gitattributes +1 -0
- README.md +46 -9
- adapter/README.md +1 -1
- evaluation/README.md +2 -1
- evaluation/base-comparison.json +72 -43
- evaluation/decision-comparison.png +2 -2
- evaluation/decision-comparison.svg +304 -214
- evaluation/export.json +0 -0
- evaluation/jevman-comparison.json +232 -0
- evaluation/jevman-comparison.png +3 -0
- evaluation/jevman-comparison.svg +341 -0
- evaluation/native-serving.json +12 -36
- evaluation/nvfp4-comparison.json +153 -2011
- evaluation/openrouter-comparison.json +41 -21
- evaluation/runtime.json +11 -33
- nvfp4/adapter/README.md +1 -1
- nvfp4/evaluation/README.md +2 -1
- nvfp4/evaluation/export.json +0 -0
- nvfp4/evaluation/native-serving.json +12 -36
- nvfp4/evaluation/nvfp4-comparison.json +153 -2011
- nvfp4/evaluation/runtime.json +11 -33
- nvfp4/provenance.json +5 -6
- provenance.json +5 -4
.gitattributes
CHANGED
|
@@ -37,3 +37,4 @@ dory-decisions-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
|
| 37 |
nvfp4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 38 |
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 39 |
evaluation/decision-comparison.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 37 |
nvfp4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 38 |
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 39 |
evaluation/decision-comparison.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
evaluation/jevman-comparison.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -31,7 +31,7 @@ datasets:
|
|
| 31 |
- nvidia/HelpSteer2
|
| 32 |
---
|
| 33 |
|
| 34 |
-
#
|
| 35 |
|
| 36 |
A fine-tune of [NVIDIA Dory 1.0 4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) for decisions over application-defined answers.
|
| 37 |
Supply context, a question, and the allowed answers. The model scores those answers without generating an explanation.
|
|
@@ -141,16 +141,31 @@ Quantization uses 256 separate training examples, with no shared question IDs or
|
|
| 141 |
Validation guided quantization selection, so these results are not an independent test.
|
| 142 |
[Validation details](evaluation/modelopt-validation.json) identify the weights, task mix, and scoring results.
|
| 143 |
|
| 144 |
-
###
|
| 145 |
|
| 146 |
The [512-question test set](evaluation/questions-512.json) includes the exact inputs, answer options, and target distributions used in all tables below.
|
| 147 |
[Scoring rules and source attribution](evaluation/README.md) explain how to reproduce the results.
|
| 148 |
|
| 149 |
-
|
| 150 |
-
|
| 151 |
|
| 152 |
The model was selected using a separate 1,024-question validation set before the test questions were scored.
|
| 153 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
### Q8_0 GGUF — evaluated with llama.cpp
|
| 155 |
|
| 156 |
The native `/v1/systemone` endpoint scored both models with Q8_0 weights.
|
|
@@ -170,7 +185,7 @@ Accuracy counts any answer tied for the highest target probability as correct.
|
|
| 170 |
Brier error compares the complete predicted and target distributions.
|
| 171 |
These are adapted decision tasks, not scores under the source datasets' original benchmark protocols.
|
| 172 |
|
| 173 |
-
|
| 174 |
|
| 175 |
### Comparison with hosted decision models
|
| 176 |
|
|
@@ -183,15 +198,17 @@ Hosted models used their providers' prompt templates and probability settings; n
|
|
| 183 |
| Model | State tracking | Planning | Human judgements | Combined | Brier ↓ |
|
| 184 |
|---|---:|---:|---:|---:|---:|
|
| 185 |
| Base Dory — Q8_0 / llama.cpp | 31.77% | 38.02% | 49.22% | 38.48% | 0.6736 |
|
| 186 |
-
| Dory
|
|
|
|
|
|
|
| 187 |
| [TypeSafe Jev 1.14](https://openrouter.ai/typesafe/jev-1.14) | 64.58% | 74.48% | 55.47% | 66.02% | 0.3843 |
|
| 188 |
| [Kev 4B](https://openrouter.ai/jaredpalmer/kev-4b) | 38.54% | 36.46% | 57.81% | 42.58% | 0.5134 |
|
| 189 |
| [Liquid d1](https://openrouter.ai/liquid/d1) | 58.33% | 79.17% | **58.59%** | 66.21% | 0.4156 |
|
| 190 |
| [Perplexity Decider 1.2](https://openrouter.ai/perplexity/pplx-decider-v1.2) | 72.92% | 88.02% | 57.03% | 74.61% | 0.3178 |
|
| 191 |
|
| 192 |
Accuracy columns show percentages; **bold** marks the best result in each column. Combined accuracy weights all 512 questions equally.
|
| 193 |
-
|
| 194 |
-
|
| 195 |
|
| 196 |
This test measures new examples from task families included in this fine-tune's training. It does not establish a general ranking of decision models.
|
| 197 |
The hosted models' training overlap with these sources is unknown.
|
|
@@ -199,6 +216,26 @@ Small rounding errors in returned distributions were normalized before Brier sco
|
|
| 199 |
[Comparison details](evaluation/openrouter-comparison.json) include exact served model versions, per-task scores, and normalization counts.
|
| 200 |
Latency is not ranked here: remote requests include network and provider overhead, while the local models ran on DGX Spark.
|
| 201 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 202 |
## Limitations
|
| 203 |
|
| 204 |
Probabilities are normalized over the supplied answers and have no fitted calibration profile.
|
|
@@ -221,7 +258,7 @@ Install the Hugging Face CLI to download the weights.
|
|
| 221 |
Download only the format you need:
|
| 222 |
|
| 223 |
```sh
|
| 224 |
-
hf download rdaum/
|
| 225 |
dory-decisions-Q8_0.gguf --local-dir ./dory-decisions
|
| 226 |
```
|
| 227 |
|
|
|
|
| 31 |
- nvidia/HelpSteer2
|
| 32 |
---
|
| 33 |
|
| 34 |
+
# Decisive Dory
|
| 35 |
|
| 36 |
A fine-tune of [NVIDIA Dory 1.0 4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) for decisions over application-defined answers.
|
| 37 |
Supply context, a question, and the allowed answers. The model scores those answers without generating an explanation.
|
|
|
|
| 141 |
Validation guided quantization selection, so these results are not an independent test.
|
| 142 |
[Validation details](evaluation/modelopt-validation.json) identify the weights, task mix, and scoring results.
|
| 143 |
|
| 144 |
+
### Test and hosted-model comparisons
|
| 145 |
|
| 146 |
The [512-question test set](evaluation/questions-512.json) includes the exact inputs, answer options, and target distributions used in all tables below.
|
| 147 |
[Scoring rules and source attribution](evaluation/README.md) explain how to reproduce the results.
|
| 148 |
|
| 149 |
+
Within each base-versus-fine-tune comparison, both models use the same quantization recipe and runtime.
|
| 150 |
+
All local models use two recurrent loops, identical typed-decision prompts, and no probability calibration.
|
| 151 |
|
| 152 |
The model was selected using a separate 1,024-question validation set before the test questions were scored.
|
| 153 |
|
| 154 |
+
### NVFP4 — evaluated with Eider
|
| 155 |
+
|
| 156 |
+
Base Dory and the fine-tune use NVIDIA Model Optimizer with local-Hessian calibration on the same 256 training examples.
|
| 157 |
+
Both models use the same questions, prompt tokens, answer labels, two recurrent loops, and Eider executable.
|
| 158 |
+
No fitted probability calibration is applied.
|
| 159 |
+
|
| 160 |
+
| Test task | Questions | Base Dory accuracy | Fine-tuned accuracy | Base Brier ↓ | Fine-tuned Brier ↓ |
|
| 161 |
+
|---|---:|---:|---:|---:|---:|
|
| 162 |
+
| SCONE state tracking | 192 | 31.25% | 75.52% | 0.9245 | 0.3981 |
|
| 163 |
+
| Block-stacking and delivery planning | 192 | 36.98% | 96.35% | 0.6013 | 0.1558 |
|
| 164 |
+
| HelpSteer2 human judgements | 128 | 48.44% | 50.78% | 0.3162 | 0.2278 |
|
| 165 |
+
| **Combined** | 512 | 37.70% | 77.15% | 0.6512 | 0.2646 |
|
| 166 |
+
|
| 167 |
+
[Results and weight hashes](evaluation/base-comparison.json) identify the models and test configuration.
|
| 168 |
+
|
| 169 |
### Q8_0 GGUF — evaluated with llama.cpp
|
| 170 |
|
| 171 |
The native `/v1/systemone` endpoint scored both models with Q8_0 weights.
|
|
|
|
| 185 |
Brier error compares the complete predicted and target distributions.
|
| 186 |
These are adapted decision tasks, not scores under the source datasets' original benchmark protocols.
|
| 187 |
|
| 188 |
+
The merged BF16 weights have not been measured on this 512-question test set.
|
| 189 |
|
| 190 |
### Comparison with hosted decision models
|
| 191 |
|
|
|
|
| 198 |
| Model | State tracking | Planning | Human judgements | Combined | Brier ↓ |
|
| 199 |
|---|---:|---:|---:|---:|---:|
|
| 200 |
| Base Dory — Q8_0 / llama.cpp | 31.77% | 38.02% | 49.22% | 38.48% | 0.6736 |
|
| 201 |
+
| Base Dory — NVFP4 / Eider | 31.25% | 36.98% | 48.44% | 37.70% | 0.6512 |
|
| 202 |
+
| Decisive Dory — Q8_0 / llama.cpp | **76.04%** | **96.88%** | 51.56% | **77.73%** | **0.2599** |
|
| 203 |
+
| Decisive Dory — NVFP4 / Eider | 75.52% | 96.35% | 50.78% | 77.15% | 0.2646 |
|
| 204 |
| [TypeSafe Jev 1.14](https://openrouter.ai/typesafe/jev-1.14) | 64.58% | 74.48% | 55.47% | 66.02% | 0.3843 |
|
| 205 |
| [Kev 4B](https://openrouter.ai/jaredpalmer/kev-4b) | 38.54% | 36.46% | 57.81% | 42.58% | 0.5134 |
|
| 206 |
| [Liquid d1](https://openrouter.ai/liquid/d1) | 58.33% | 79.17% | **58.59%** | 66.21% | 0.4156 |
|
| 207 |
| [Perplexity Decider 1.2](https://openrouter.ai/perplexity/pplx-decider-v1.2) | 72.92% | 88.02% | 57.03% | 74.61% | 0.3178 |
|
| 208 |
|
| 209 |
Accuracy columns show percentages; **bold** marks the best result in each column. Combined accuracy weights all 512 questions equally.
|
| 210 |
+
On human judgements, Jev scores 55.47% and Liquid d1 scores 58.59%.
|
| 211 |
+
This model scores 50.78% with NVFP4 and 51.56% with Q8_0.
|
| 212 |
|
| 213 |
This test measures new examples from task families included in this fine-tune's training. It does not establish a general ranking of decision models.
|
| 214 |
The hosted models' training overlap with these sources is unknown.
|
|
|
|
| 216 |
[Comparison details](evaluation/openrouter-comparison.json) include exact served model versions, per-task scores, and normalization counts.
|
| 217 |
Latency is not ranked here: remote requests include network and provider overhead, while the local models ran on DGX Spark.
|
| 218 |
|
| 219 |
+
## JevMan gameplay
|
| 220 |
+
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
| Model / runtime | Games | Mean score ↑ | Median score | Mean decision latency ↓ |
|
| 224 |
+
|---|---:|---:|---:|---:|
|
| 225 |
+
| Decisive Dory — NVFP4 · Eider | 6 | 2,338 | 2,280 | 163 ms |
|
| 226 |
+
| Decisive Dory — Q8_0 · llama.cpp | 6 | 2,362 | 2,265 | 342 ms |
|
| 227 |
+
| Jev 1.14 — OpenRouter | 12 | 3,704 | 3,190 | 202 ms |
|
| 228 |
+
|
| 229 |
+
Games use the original fixed map, a 300-second gameplay cap, one game at a time, and no artificial latency.
|
| 230 |
+
Dots show individual game scores. The Jev result includes all 12 games from two runs with the same protocol.
|
| 231 |
+
Local models run on NVIDIA DGX Spark. Jev runs through OpenRouter, so its request latency includes network and provider overhead.
|
| 232 |
+
Each local game starts a fresh server. Server startup is excluded from gameplay and decision latency.
|
| 233 |
+
All completed replays passed the upstream verifier.
|
| 234 |
+
|
| 235 |
+
Jev scores higher in this sample. Dory has similar scores through both runtimes, with lower request latency through Eider.
|
| 236 |
+
These repeated real-time games do not establish performance across other maps or games.
|
| 237 |
+
[Per-game results and configuration](evaluation/jevman-comparison.json) provide the measurements behind the chart.
|
| 238 |
+
|
| 239 |
## Limitations
|
| 240 |
|
| 241 |
Probabilities are normalized over the supplied answers and have no fitted calibration profile.
|
|
|
|
| 258 |
Download only the format you need:
|
| 259 |
|
| 260 |
```sh
|
| 261 |
+
hf download rdaum/decisive-dory \
|
| 262 |
dory-decisions-Q8_0.gguf --local-dir ./dory-decisions
|
| 263 |
```
|
| 264 |
|
adapter/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Dory
|
| 2 |
|
| 3 |
This rank-8 LoRA adapter uses the tensor format supported by the included `dory_lora.py` loader. It is not in PEFT format.
|
| 4 |
The BF16, NVFP4, and GGUF model files already include the adaptation. Use this standalone adapter only with the original NVIDIA BF16 weights.
|
|
|
|
| 1 |
+
# Decisive Dory — LoRA adapter
|
| 2 |
|
| 3 |
This rank-8 LoRA adapter uses the tensor format supported by the included `dory_lora.py` loader. It is not in PEFT format.
|
| 4 |
The BF16, NVFP4, and GGUF model files already include the adaptation. Use this standalone adapter only with the original NVIDIA BF16 weights.
|
evaluation/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Decision evaluation: 512 questions
|
| 2 |
|
| 3 |
-
The model card compares Q8_0 Dory with hosted models on these questions.
|
| 4 |
[NVFP4 validation results](modelopt-validation.json) use a separate 1,024-question set.
|
| 5 |
|
| 6 |
[Download the questions and targets](questions-512.json).
|
|
@@ -102,5 +102,6 @@ Collection and question-format adaptations are provided under CC BY-SA 4.0; the
|
|
| 102 |
|
| 103 |
## Results
|
| 104 |
|
|
|
|
| 105 |
- [Base versus fine-tuned Dory, Q8_0 / llama.cpp](llama-comparison.json)
|
| 106 |
- [Hosted decision models on the same questions](openrouter-comparison.json)
|
|
|
|
| 1 |
# Decision evaluation: 512 questions
|
| 2 |
|
| 3 |
+
The model card compares NVFP4 and Q8_0 Dory with hosted models on these questions.
|
| 4 |
[NVFP4 validation results](modelopt-validation.json) use a separate 1,024-question set.
|
| 5 |
|
| 6 |
[Download the questions and targets](questions-512.json).
|
|
|
|
| 102 |
|
| 103 |
## Results
|
| 104 |
|
| 105 |
+
- [Base versus fine-tuned Dory, NVFP4 / Eider](base-comparison.json)
|
| 106 |
- [Base versus fine-tuned Dory, Q8_0 / llama.cpp](llama-comparison.json)
|
| 107 |
- [Hosted decision models on the same questions](openrouter-comparison.json)
|
evaluation/base-comparison.json
CHANGED
|
@@ -3,104 +3,133 @@
|
|
| 3 |
"base_model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16",
|
| 4 |
"base_model_revision": "b366b87f762ea334ba3eb55912031baba1515810",
|
| 5 |
"format": "NVFP4",
|
| 6 |
-
"quantization_recipe": "
|
|
|
|
| 7 |
"loops": 2,
|
| 8 |
"calibrated": false,
|
| 9 |
-
"prompt": "
|
| 10 |
-
"base_weight_sha256": "602a1bc6a69c1a5c41253df785f335cea77eeeb854b985ab11c125bf1d08a547",
|
| 11 |
-
"fine_tuned_weight_sha256": "9bc834f4dce29a48499b6614c11772d189710138b29c637a384f753b1326b4c0",
|
| 12 |
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 13 |
-
"scoring_script_sha256": "
|
| 14 |
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
|
|
|
| 15 |
"base": {
|
| 16 |
"questions": 512,
|
| 17 |
-
"modal_accuracy": 0.
|
| 18 |
-
"brier": 0.
|
| 19 |
-
"cross_entropy":
|
|
|
|
|
|
|
| 20 |
"by_kind": {
|
| 21 |
"choice": {
|
| 22 |
"questions": 241,
|
| 23 |
-
"modal_accuracy": 0.
|
| 24 |
-
"brier": 0.
|
| 25 |
-
"cross_entropy": 2.
|
|
|
|
|
|
|
| 26 |
},
|
| 27 |
"noul": {
|
| 28 |
"questions": 94,
|
| 29 |
"modal_accuracy": 0.5,
|
| 30 |
-
"brier": 0.
|
| 31 |
-
"cross_entropy":
|
|
|
|
|
|
|
| 32 |
},
|
| 33 |
"score": {
|
| 34 |
"questions": 177,
|
| 35 |
-
"modal_accuracy": 0.
|
| 36 |
-
"brier": 0.
|
| 37 |
-
"cross_entropy": 2.
|
|
|
|
|
|
|
| 38 |
}
|
| 39 |
},
|
| 40 |
"by_source": {
|
| 41 |
"helpsteer2": {
|
| 42 |
"questions": 128,
|
| 43 |
-
"modal_accuracy": 0.
|
| 44 |
-
"brier": 0.
|
| 45 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 46 |
},
|
| 47 |
"planning": {
|
| 48 |
"questions": 192,
|
| 49 |
-
"modal_accuracy": 0.
|
| 50 |
-
"brier": 0.
|
| 51 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 52 |
},
|
| 53 |
"scone": {
|
| 54 |
"questions": 192,
|
| 55 |
-
"modal_accuracy": 0.
|
| 56 |
-
"brier":
|
| 57 |
-
"cross_entropy": 2.
|
|
|
|
|
|
|
| 58 |
}
|
| 59 |
}
|
| 60 |
},
|
|
|
|
| 61 |
"fine_tuned": {
|
| 62 |
"questions": 512,
|
| 63 |
-
"modal_accuracy": 0.
|
| 64 |
-
"brier": 0.
|
| 65 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 66 |
"by_kind": {
|
| 67 |
"choice": {
|
| 68 |
"questions": 241,
|
| 69 |
"modal_accuracy": 0.8215767634854771,
|
| 70 |
-
"brier": 0.
|
| 71 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 72 |
},
|
| 73 |
"noul": {
|
| 74 |
"questions": 94,
|
| 75 |
-
"modal_accuracy": 0.
|
| 76 |
-
"brier": 0.
|
| 77 |
-
"cross_entropy": 0.
|
|
|
|
|
|
|
| 78 |
},
|
| 79 |
"score": {
|
| 80 |
"questions": 177,
|
| 81 |
-
"modal_accuracy": 0.
|
| 82 |
-
"brier": 0.
|
| 83 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 84 |
}
|
| 85 |
},
|
| 86 |
"by_source": {
|
| 87 |
"helpsteer2": {
|
| 88 |
"questions": 128,
|
| 89 |
"modal_accuracy": 0.5078125,
|
| 90 |
-
"brier": 0.
|
| 91 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 92 |
},
|
| 93 |
"planning": {
|
| 94 |
"questions": 192,
|
| 95 |
-
"modal_accuracy": 0.
|
| 96 |
-
"brier": 0.
|
| 97 |
-
"cross_entropy": 0.
|
|
|
|
|
|
|
| 98 |
},
|
| 99 |
"scone": {
|
| 100 |
"questions": 192,
|
| 101 |
"modal_accuracy": 0.7552083333333334,
|
| 102 |
-
"brier": 0.
|
| 103 |
-
"cross_entropy":
|
|
|
|
|
|
|
| 104 |
}
|
| 105 |
}
|
| 106 |
}
|
|
|
|
| 3 |
"base_model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16",
|
| 4 |
"base_model_revision": "b366b87f762ea334ba3eb55912031baba1515810",
|
| 5 |
"format": "NVFP4",
|
| 6 |
+
"quantization_recipe": "NVIDIA Model Optimizer local_hessian",
|
| 7 |
+
"calibration_examples": 256,
|
| 8 |
"loops": 2,
|
| 9 |
"calibrated": false,
|
| 10 |
+
"prompt": "Identical typed-decision prompts and allowed-answer scoring; no generated reasoning.",
|
|
|
|
|
|
|
| 11 |
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 12 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 13 |
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 14 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 15 |
"base": {
|
| 16 |
"questions": 512,
|
| 17 |
+
"modal_accuracy": 0.376953125,
|
| 18 |
+
"brier": 0.6512325676171807,
|
| 19 |
+
"cross_entropy": 1.9877935454851494,
|
| 20 |
+
"mean_request_ms": 81.26816220556066,
|
| 21 |
+
"median_request_ms": 72.5804881658405,
|
| 22 |
"by_kind": {
|
| 23 |
"choice": {
|
| 24 |
"questions": 241,
|
| 25 |
+
"modal_accuracy": 0.2863070539419087,
|
| 26 |
+
"brier": 0.742723819745487,
|
| 27 |
+
"cross_entropy": 2.2877653477997058,
|
| 28 |
+
"mean_request_ms": 75.63938830179235,
|
| 29 |
+
"median_request_ms": 72.81611207872629
|
| 30 |
},
|
| 31 |
"noul": {
|
| 32 |
"questions": 94,
|
| 33 |
"modal_accuracy": 0.5,
|
| 34 |
+
"brier": 0.670176294758546,
|
| 35 |
+
"cross_entropy": 0.9175096541022592,
|
| 36 |
+
"mean_request_ms": 68.15949506423574,
|
| 37 |
+
"median_request_ms": 68.27253755182028
|
| 38 |
},
|
| 39 |
"score": {
|
| 40 |
"questions": 177,
|
| 41 |
+
"modal_accuracy": 0.4350282485875706,
|
| 42 |
+
"brier": 0.5165992223391576,
|
| 43 |
+
"cross_entropy": 2.1477567174183902,
|
| 44 |
+
"mean_request_ms": 95.89386402529344,
|
| 45 |
+
"median_request_ms": 77.44529517367482
|
| 46 |
}
|
| 47 |
},
|
| 48 |
"by_source": {
|
| 49 |
"helpsteer2": {
|
| 50 |
"questions": 128,
|
| 51 |
+
"modal_accuracy": 0.484375,
|
| 52 |
+
"brier": 0.31624743248922543,
|
| 53 |
+
"cross_entropy": 1.797596837336767,
|
| 54 |
+
"mean_request_ms": 107.6381878774555,
|
| 55 |
+
"median_request_ms": 96.75377863459289
|
| 56 |
},
|
| 57 |
"planning": {
|
| 58 |
"questions": 192,
|
| 59 |
+
"modal_accuracy": 0.3697916666666667,
|
| 60 |
+
"brier": 0.6013039965397795,
|
| 61 |
+
"cross_entropy": 1.8274480710869951,
|
| 62 |
+
"mean_request_ms": 75.0043233759546,
|
| 63 |
+
"median_request_ms": 73.60281050205231
|
| 64 |
},
|
| 65 |
"scone": {
|
| 66 |
"questions": 192,
|
| 67 |
+
"modal_accuracy": 0.3125,
|
| 68 |
+
"brier": 0.9244845621132188,
|
| 69 |
+
"cross_entropy": 2.274936825315559,
|
| 70 |
+
"mean_request_ms": 69.95198392057016,
|
| 71 |
+
"median_request_ms": 69.4849812425673
|
| 72 |
}
|
| 73 |
}
|
| 74 |
},
|
| 75 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 76 |
"fine_tuned": {
|
| 77 |
"questions": 512,
|
| 78 |
+
"modal_accuracy": 0.771484375,
|
| 79 |
+
"brier": 0.26463106057883456,
|
| 80 |
+
"cross_entropy": 1.075109941121327,
|
| 81 |
+
"mean_request_ms": 80.6613628164996,
|
| 82 |
+
"median_request_ms": 71.86161540448666,
|
| 83 |
"by_kind": {
|
| 84 |
"choice": {
|
| 85 |
"questions": 241,
|
| 86 |
"modal_accuracy": 0.8215767634854771,
|
| 87 |
+
"brier": 0.34044989428658823,
|
| 88 |
+
"cross_entropy": 1.1801933058267577,
|
| 89 |
+
"mean_request_ms": 75.06443578865827,
|
| 90 |
+
"median_request_ms": 72.3452023230493
|
| 91 |
},
|
| 92 |
"noul": {
|
| 93 |
"questions": 94,
|
| 94 |
+
"modal_accuracy": 0.9574468085106383,
|
| 95 |
+
"brier": 0.09152335422286362,
|
| 96 |
+
"cross_entropy": 0.2881069286288201,
|
| 97 |
+
"mean_request_ms": 67.15274397581022,
|
| 98 |
+
"median_request_ms": 67.46977800503373
|
| 99 |
},
|
| 100 |
"score": {
|
| 101 |
"questions": 177,
|
| 102 |
+
"modal_accuracy": 0.6045197740112994,
|
| 103 |
+
"brier": 0.2533304135386799,
|
| 104 |
+
"cross_entropy": 1.3499867336653204,
|
| 105 |
+
"mean_request_ms": 95.45610623307905,
|
| 106 |
+
"median_request_ms": 77.96838786453009
|
| 107 |
}
|
| 108 |
},
|
| 109 |
"by_source": {
|
| 110 |
"helpsteer2": {
|
| 111 |
"questions": 128,
|
| 112 |
"modal_accuracy": 0.5078125,
|
| 113 |
+
"brier": 0.22775570941129228,
|
| 114 |
+
"cross_entropy": 1.5323848707632575,
|
| 115 |
+
"mean_request_ms": 107.58943439577706,
|
| 116 |
+
"median_request_ms": 97.03774400986731
|
| 117 |
},
|
| 118 |
"planning": {
|
| 119 |
"questions": 192,
|
| 120 |
+
"modal_accuracy": 0.9635416666666666,
|
| 121 |
+
"brier": 0.155754142490034,
|
| 122 |
+
"cross_entropy": 0.834348251030471,
|
| 123 |
+
"mean_request_ms": 74.49704928634067,
|
| 124 |
+
"median_request_ms": 73.17394809797406
|
| 125 |
},
|
| 126 |
"scone": {
|
| 127 |
"questions": 192,
|
| 128 |
"modal_accuracy": 0.7552083333333334,
|
| 129 |
+
"brier": 0.3980915461126633,
|
| 130 |
+
"cross_entropy": 1.0110216781175627,
|
| 131 |
+
"mean_request_ms": 68.87362862714023,
|
| 132 |
+
"median_request_ms": 68.2264359202236
|
| 133 |
}
|
| 134 |
}
|
| 135 |
}
|
evaluation/decision-comparison.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
evaluation/decision-comparison.svg
CHANGED
|
|
evaluation/export.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
evaluation/jevman-comparison.json
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"protocol": {
|
| 3 |
+
"benchmark": "https://github.com/opper-ai/jevman-benchmark",
|
| 4 |
+
"revision": "412ed72484ffeb3bd205a64be2d2ea5e233aac88",
|
| 5 |
+
"map": "Original fixed map",
|
| 6 |
+
"max_gameplay_seconds": 300,
|
| 7 |
+
"parallel_games": 1,
|
| 8 |
+
"added_latency_ms": 0,
|
| 9 |
+
"local_hardware": "NVIDIA DGX Spark / GB10",
|
| 10 |
+
"local_loops": 2,
|
| 11 |
+
"local_probability_calibration": false,
|
| 12 |
+
"local_server": "Fresh server per game; startup excluded from gameplay",
|
| 13 |
+
"hosted_route": "OpenRouter; HTTP latency includes network and provider overhead",
|
| 14 |
+
"sampling": "Repeated real-time games; not independent maps or paired random seeds",
|
| 15 |
+
"jev_sampling": "All 12 completed games from two six-game runs using the same protocol",
|
| 16 |
+
"replay_verification": "All completed replays passed the upstream verifier"
|
| 17 |
+
},
|
| 18 |
+
"conditions": [
|
| 19 |
+
{
|
| 20 |
+
"model": "Decisive Dory",
|
| 21 |
+
"route": "NVFP4 \u00b7 Eider",
|
| 22 |
+
"games": [
|
| 23 |
+
{
|
| 24 |
+
"score": 2300,
|
| 25 |
+
"survived_seconds": 92.66160000000217,
|
| 26 |
+
"calls": 191,
|
| 27 |
+
"latency_ms_sum": 30752,
|
| 28 |
+
"replay_sha256": "ab452665d336a32a32a02f92b1fbe2412afb0107f7d6a664d98a8a6ba8d5afaf"
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"score": 2040,
|
| 32 |
+
"survived_seconds": 97.0343000000044,
|
| 33 |
+
"calls": 212,
|
| 34 |
+
"latency_ms_sum": 34994,
|
| 35 |
+
"replay_sha256": "962baf1ab3627b119e0884158bbb7eb9f47285ea3f40c49179f7cdf376239a22"
|
| 36 |
+
},
|
| 37 |
+
{
|
| 38 |
+
"score": 2830,
|
| 39 |
+
"survived_seconds": 96.17950000000396,
|
| 40 |
+
"calls": 195,
|
| 41 |
+
"latency_ms_sum": 30941,
|
| 42 |
+
"replay_sha256": "77f0471014575ad5c2f27d9878d71297300a9d5038562951f5f7047526fd43e7"
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"score": 2260,
|
| 46 |
+
"survived_seconds": 76.8986000000027,
|
| 47 |
+
"calls": 150,
|
| 48 |
+
"latency_ms_sum": 24841,
|
| 49 |
+
"replay_sha256": "226832364e02185418f46e0b2ae7051fe692f2eb51283c72e2322f42d9a09c15"
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"score": 2690,
|
| 53 |
+
"survived_seconds": 81.84360000000173,
|
| 54 |
+
"calls": 159,
|
| 55 |
+
"latency_ms_sum": 24517,
|
| 56 |
+
"replay_sha256": "6f58daea7fa64b0eb14520b820a30ff6fa1ab21b89add0a31649f3e36899b332"
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"score": 1910,
|
| 60 |
+
"survived_seconds": 67.23850000000128,
|
| 61 |
+
"calls": 143,
|
| 62 |
+
"latency_ms_sum": 25136,
|
| 63 |
+
"replay_sha256": "fd5b20647dd61a04aa71565e5540e3ac0b8d2f25a8878a742790f3af33271982"
|
| 64 |
+
}
|
| 65 |
+
],
|
| 66 |
+
"game_count": 6,
|
| 67 |
+
"mean_score": 2338.3333333333335,
|
| 68 |
+
"median_score": 2280.0,
|
| 69 |
+
"mean_decision_ms": 163.02952380952382,
|
| 70 |
+
"scheduler_errors": 0,
|
| 71 |
+
"resolved_models": [
|
| 72 |
+
"local-dory"
|
| 73 |
+
]
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"model": "Decisive Dory",
|
| 77 |
+
"route": "Q8_0 \u00b7 llama.cpp",
|
| 78 |
+
"games": [
|
| 79 |
+
{
|
| 80 |
+
"score": 2220,
|
| 81 |
+
"survived_seconds": 90.00120000000175,
|
| 82 |
+
"calls": 151,
|
| 83 |
+
"latency_ms_sum": 48905,
|
| 84 |
+
"replay_sha256": "97f4bcc396b43e9e8848bd0093977b531aa8193c1871d3f26917c0ea290acb39"
|
| 85 |
+
},
|
| 86 |
+
{
|
| 87 |
+
"score": 2310,
|
| 88 |
+
"survived_seconds": 65.34389999999954,
|
| 89 |
+
"calls": 114,
|
| 90 |
+
"latency_ms_sum": 40117,
|
| 91 |
+
"replay_sha256": "1e9b0d7114f5fa33d5719e8d20bfb89ffdc96d2753e4a128ee31e94797f6ca8f"
|
| 92 |
+
},
|
| 93 |
+
{
|
| 94 |
+
"score": 2310,
|
| 95 |
+
"survived_seconds": 64.91600000000018,
|
| 96 |
+
"calls": 114,
|
| 97 |
+
"latency_ms_sum": 40300,
|
| 98 |
+
"replay_sha256": "efc1ee7b3c96469a866e5aadd94708104a76dbc6520f69dc3247a32a03af412f"
|
| 99 |
+
},
|
| 100 |
+
{
|
| 101 |
+
"score": 2130,
|
| 102 |
+
"survived_seconds": 58.775600000001006,
|
| 103 |
+
"calls": 102,
|
| 104 |
+
"latency_ms_sum": 35172,
|
| 105 |
+
"replay_sha256": "c2ddfdb40daf0a7ca943d6c012690bef8b5ac49f9589681c7e09e9483ad0d002"
|
| 106 |
+
},
|
| 107 |
+
{
|
| 108 |
+
"score": 2220,
|
| 109 |
+
"survived_seconds": 90.04169999999984,
|
| 110 |
+
"calls": 153,
|
| 111 |
+
"latency_ms_sum": 51135,
|
| 112 |
+
"replay_sha256": "c0b76b482c1aabec20cd11b314051537517fedc3012f998995d9a3f0bf0fcd6d"
|
| 113 |
+
},
|
| 114 |
+
{
|
| 115 |
+
"score": 2980,
|
| 116 |
+
"survived_seconds": 87.8558000000031,
|
| 117 |
+
"calls": 152,
|
| 118 |
+
"latency_ms_sum": 52981,
|
| 119 |
+
"replay_sha256": "287970b9664eea214c9c28dd092d1a6dc942430eabc8394a2970360bb357fce5"
|
| 120 |
+
}
|
| 121 |
+
],
|
| 122 |
+
"game_count": 6,
|
| 123 |
+
"mean_score": 2361.6666666666665,
|
| 124 |
+
"median_score": 2265.0,
|
| 125 |
+
"mean_decision_ms": 341.74300254452925,
|
| 126 |
+
"scheduler_errors": 0,
|
| 127 |
+
"resolved_models": [
|
| 128 |
+
"dory-jevman-q8"
|
| 129 |
+
]
|
| 130 |
+
},
|
| 131 |
+
{
|
| 132 |
+
"model": "Jev 1.14",
|
| 133 |
+
"route": "OpenRouter",
|
| 134 |
+
"games": [
|
| 135 |
+
{
|
| 136 |
+
"score": 4770,
|
| 137 |
+
"survived_seconds": 66.7314000000021,
|
| 138 |
+
"calls": 135,
|
| 139 |
+
"latency_ms_sum": 33957,
|
| 140 |
+
"replay_sha256": "f33346b8d20393805575649c416a2c4343e9d7c4e6011009188dc5c3b055e662"
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"score": 2460,
|
| 144 |
+
"survived_seconds": 46.36650000000062,
|
| 145 |
+
"calls": 95,
|
| 146 |
+
"latency_ms_sum": 16876,
|
| 147 |
+
"replay_sha256": "2a295eb5464d7ce30be8323147cd936ce40a85473f3a110e7ff5c5c2960827ce"
|
| 148 |
+
},
|
| 149 |
+
{
|
| 150 |
+
"score": 2540,
|
| 151 |
+
"survived_seconds": 45.76800000000065,
|
| 152 |
+
"calls": 93,
|
| 153 |
+
"latency_ms_sum": 17012,
|
| 154 |
+
"replay_sha256": "f8d9967634a0b6a067e158213785c57e51af9ad2763b457e925a28d360110ea5"
|
| 155 |
+
},
|
| 156 |
+
{
|
| 157 |
+
"score": 6050,
|
| 158 |
+
"survived_seconds": 105.78640000000006,
|
| 159 |
+
"calls": 204,
|
| 160 |
+
"latency_ms_sum": 35635,
|
| 161 |
+
"replay_sha256": "b299862de3623ebb332c36646d3d202cda701b0d02495dba118141eb050c12d4"
|
| 162 |
+
},
|
| 163 |
+
{
|
| 164 |
+
"score": 3020,
|
| 165 |
+
"survived_seconds": 65.42200000000238,
|
| 166 |
+
"calls": 124,
|
| 167 |
+
"latency_ms_sum": 22516,
|
| 168 |
+
"replay_sha256": "817ef7cf8f250fcde6d8ef31ac5c511977cc7822e4c80938fbc5a7f9a607e367"
|
| 169 |
+
},
|
| 170 |
+
{
|
| 171 |
+
"score": 5170,
|
| 172 |
+
"survived_seconds": 113.07670000000334,
|
| 173 |
+
"calls": 211,
|
| 174 |
+
"latency_ms_sum": 41874,
|
| 175 |
+
"replay_sha256": "dca22fa9a3b7e7e7adf20fb7350ed04f5b75869e0c7eebecfb9cdef00dc8471e"
|
| 176 |
+
},
|
| 177 |
+
{
|
| 178 |
+
"score": 3360,
|
| 179 |
+
"survived_seconds": 65.91620000000175,
|
| 180 |
+
"calls": 115,
|
| 181 |
+
"latency_ms_sum": 27609,
|
| 182 |
+
"replay_sha256": "f3f2a114c9c196317d76daad98ffeb7760f6ebfeedaec2910369274670d9fcb0"
|
| 183 |
+
},
|
| 184 |
+
{
|
| 185 |
+
"score": 2510,
|
| 186 |
+
"survived_seconds": 56.38020000000047,
|
| 187 |
+
"calls": 110,
|
| 188 |
+
"latency_ms_sum": 20643,
|
| 189 |
+
"replay_sha256": "405afa6aa1e283518d90e4b4e2105be5e79e9c726d2c9d758431e4d0226c231c"
|
| 190 |
+
},
|
| 191 |
+
{
|
| 192 |
+
"score": 6610,
|
| 193 |
+
"survived_seconds": 100.2825000000033,
|
| 194 |
+
"calls": 196,
|
| 195 |
+
"latency_ms_sum": 33076,
|
| 196 |
+
"replay_sha256": "ee10c68d2ef0f7081379508d658bc45ab926cb86a36d5be53105d249ac8f9026"
|
| 197 |
+
},
|
| 198 |
+
{
|
| 199 |
+
"score": 2400,
|
| 200 |
+
"survived_seconds": 50.01130000000068,
|
| 201 |
+
"calls": 95,
|
| 202 |
+
"latency_ms_sum": 23380,
|
| 203 |
+
"replay_sha256": "425ff1f1258f15bad585284621e1ae135d6ab50092b0fedace4915bc9a945365"
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"score": 2190,
|
| 207 |
+
"survived_seconds": 50.25400000000027,
|
| 208 |
+
"calls": 88,
|
| 209 |
+
"latency_ms_sum": 22038,
|
| 210 |
+
"replay_sha256": "49f635dabfa938af30377099bdaa9e43666402997c8a08153b34ce31d570d325"
|
| 211 |
+
},
|
| 212 |
+
{
|
| 213 |
+
"score": 3370,
|
| 214 |
+
"survived_seconds": 78.94970000000323,
|
| 215 |
+
"calls": 138,
|
| 216 |
+
"latency_ms_sum": 29063,
|
| 217 |
+
"replay_sha256": "3cb6f1436d33144467147d9e4ec45fc8d4ce23eb43d571ba3e3cbfc8762ace4a"
|
| 218 |
+
}
|
| 219 |
+
],
|
| 220 |
+
"game_count": 12,
|
| 221 |
+
"mean_score": 3704.1666666666665,
|
| 222 |
+
"median_score": 3190.0,
|
| 223 |
+
"mean_decision_ms": 201.79488778054863,
|
| 224 |
+
"scheduler_errors": 4,
|
| 225 |
+
"resolved_models": [
|
| 226 |
+
"typesafe/jev-1.14-20261014"
|
| 227 |
+
]
|
| 228 |
+
}
|
| 229 |
+
],
|
| 230 |
+
"eider_weights_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 231 |
+
"llama_q8_weights_sha256": "bb36b8ed263ce633649ccfad9c6e1dc065e9c3d333624fc27028f496089a38b8"
|
| 232 |
+
}
|
evaluation/jevman-comparison.png
ADDED
|
Git LFS Details
|
evaluation/jevman-comparison.svg
ADDED
|
evaluation/native-serving.json
CHANGED
|
@@ -1,38 +1,14 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
"127.0.0.1:18089",
|
| 15 |
-
"--max-active-sequences",
|
| 16 |
-
"1",
|
| 17 |
-
"--decode-capacity",
|
| 18 |
-
"1",
|
| 19 |
-
"--prefill-sequence-capacity",
|
| 20 |
-
"1",
|
| 21 |
-
"--prefill-token-capacity",
|
| 22 |
-
"2048",
|
| 23 |
-
"--decision-branch-capacity",
|
| 24 |
-
"8",
|
| 25 |
-
"--max-context-tokens",
|
| 26 |
-
"4096",
|
| 27 |
-
"--retained-prefix-gib",
|
| 28 |
-
"0"
|
| 29 |
-
],
|
| 30 |
-
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 31 |
-
"script_sha256": "cbd6c833654b1fff319abf1d8f3cb491af5d96b78d541e593a283b684268da04",
|
| 32 |
-
"evaluation_data_dir": "data",
|
| 33 |
-
"evaluation_data_sha256": {
|
| 34 |
-
"new_test": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 35 |
-
"test": "ace4bd84b914cee4c9ad8658d934386b045b7c643d490283d4f178f037acdba5",
|
| 36 |
-
"transfer": "3c5a77ed25fc7d0c2d1419151aa816bab9d12975b1e59de681934c114834d8e7"
|
| 37 |
-
}
|
| 38 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"server_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 3 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 4 |
+
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 5 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 6 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 7 |
+
"loops": 2,
|
| 8 |
+
"probability_calibration": false,
|
| 9 |
+
"max_context_tokens": 8192,
|
| 10 |
+
"prefill_token_capacity": 2048,
|
| 11 |
+
"decision_branch_capacity": 8,
|
| 12 |
+
"retained_prefix_gib": 0,
|
| 13 |
+
"max_active_sequences": 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
}
|
evaluation/nvfp4-comparison.json
CHANGED
|
@@ -1,2033 +1,175 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
"
|
| 5 |
-
"
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
"
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
"cross_entropy": 1.050083253299498
|
| 48 |
-
},
|
| 49 |
-
"babyai": {
|
| 50 |
-
"questions": 58,
|
| 51 |
-
"modal_accuracy": 0.39655172413793105,
|
| 52 |
-
"brier": 0.7305371209519513,
|
| 53 |
-
"cross_entropy": 1.4973851876562216
|
| 54 |
-
},
|
| 55 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 56 |
-
"questions": 2,
|
| 57 |
-
"modal_accuracy": 1.0,
|
| 58 |
-
"brier": 4.986938069711992e-05,
|
| 59 |
-
"cross_entropy": 0.003831617325445817
|
| 60 |
-
},
|
| 61 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 62 |
-
"questions": 5,
|
| 63 |
-
"modal_accuracy": 0.6,
|
| 64 |
-
"brier": 0.7805844658893524,
|
| 65 |
-
"cross_entropy": 1.7801497473019683
|
| 66 |
-
},
|
| 67 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 68 |
-
"questions": 7,
|
| 69 |
-
"modal_accuracy": 0.7142857142857143,
|
| 70 |
-
"brier": 0.4790055889595874,
|
| 71 |
-
"cross_entropy": 1.1121963454176031
|
| 72 |
-
},
|
| 73 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 74 |
-
"questions": 12,
|
| 75 |
-
"modal_accuracy": 0.6666666666666666,
|
| 76 |
-
"brier": 0.6101325803838552,
|
| 77 |
-
"cross_entropy": 2.0033591272403792
|
| 78 |
-
},
|
| 79 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 80 |
-
"questions": 10,
|
| 81 |
-
"modal_accuracy": 0.7,
|
| 82 |
-
"brier": 0.6003855550573081,
|
| 83 |
-
"cross_entropy": 1.611779150841159
|
| 84 |
-
},
|
| 85 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 86 |
-
"questions": 4,
|
| 87 |
-
"modal_accuracy": 0.75,
|
| 88 |
-
"brier": 0.42100449664852624,
|
| 89 |
-
"cross_entropy": 1.9128633478153523
|
| 90 |
-
},
|
| 91 |
-
"connect_four": {
|
| 92 |
-
"questions": 56,
|
| 93 |
-
"modal_accuracy": 0.5357142857142857,
|
| 94 |
-
"brier": 0.2741592752209492,
|
| 95 |
-
"cross_entropy": 1.860460665385586
|
| 96 |
-
},
|
| 97 |
-
"customer-control-v1": {
|
| 98 |
-
"questions": 7,
|
| 99 |
-
"modal_accuracy": 1.0,
|
| 100 |
-
"brier": 2.0002025677443218e-06,
|
| 101 |
-
"cross_entropy": 0.0004476462766485558
|
| 102 |
-
},
|
| 103 |
-
"devign": {
|
| 104 |
-
"questions": 2,
|
| 105 |
-
"modal_accuracy": 0.5,
|
| 106 |
-
"brier": 0.41023068518803085,
|
| 107 |
-
"cross_entropy": 0.6006732852714238
|
| 108 |
-
},
|
| 109 |
-
"gsm8k": {
|
| 110 |
-
"questions": 4,
|
| 111 |
-
"modal_accuracy": 1.0,
|
| 112 |
-
"brier": 0.0013119361198151522,
|
| 113 |
-
"cross_entropy": 0.02163855006976333
|
| 114 |
-
},
|
| 115 |
-
"helpsteer2": {
|
| 116 |
-
"questions": 128,
|
| 117 |
-
"modal_accuracy": 0.4296875,
|
| 118 |
-
"brier": 0.5887451086780798,
|
| 119 |
-
"cross_entropy": 2.801737422463417
|
| 120 |
-
},
|
| 121 |
-
"planning": {
|
| 122 |
-
"questions": 192,
|
| 123 |
-
"modal_accuracy": 0.6041666666666666,
|
| 124 |
-
"brier": 0.5293956848932907,
|
| 125 |
-
"cross_entropy": 1.6646787560472525
|
| 126 |
-
},
|
| 127 |
-
"race": {
|
| 128 |
-
"questions": 4,
|
| 129 |
-
"modal_accuracy": 0.5,
|
| 130 |
-
"brier": 0.8826525379691635,
|
| 131 |
-
"cross_entropy": 2.2968173663260414
|
| 132 |
-
},
|
| 133 |
-
"ragtruth": {
|
| 134 |
-
"questions": 6,
|
| 135 |
-
"modal_accuracy": 0.6666666666666666,
|
| 136 |
-
"brier": 0.31120510035647425,
|
| 137 |
-
"cross_entropy": 0.4402550888479215
|
| 138 |
-
},
|
| 139 |
-
"scienceworld": {
|
| 140 |
-
"questions": 128,
|
| 141 |
-
"modal_accuracy": 0.6953125,
|
| 142 |
-
"brier": 0.4588827899272394,
|
| 143 |
-
"cross_entropy": 1.3847298212396886
|
| 144 |
-
},
|
| 145 |
-
"scone": {
|
| 146 |
-
"questions": 192,
|
| 147 |
-
"modal_accuracy": 0.4739583333333333,
|
| 148 |
-
"brier": 0.8022954836937664,
|
| 149 |
-
"cross_entropy": 2.0042890400598963
|
| 150 |
-
},
|
| 151 |
-
"toolace": {
|
| 152 |
-
"questions": 2,
|
| 153 |
-
"modal_accuracy": 1.0,
|
| 154 |
-
"brier": 2.2949296518888497e-08,
|
| 155 |
-
"cross_entropy": 9.582785411797238e-05
|
| 156 |
-
},
|
| 157 |
-
"yuri_v3": {
|
| 158 |
-
"questions": 53,
|
| 159 |
-
"modal_accuracy": 0.7358490566037735,
|
| 160 |
-
"brier": 0.0564251534824245,
|
| 161 |
-
"cross_entropy": 0.8602018822761758
|
| 162 |
-
}
|
| 163 |
-
}
|
| 164 |
-
}
|
| 165 |
-
},
|
| 166 |
-
{
|
| 167 |
-
"step": 2000,
|
| 168 |
-
"examples_seen": 8000,
|
| 169 |
-
"checkpoint": "step-002000",
|
| 170 |
-
"export": "step-002000",
|
| 171 |
-
"validation": {
|
| 172 |
-
"questions": 1024,
|
| 173 |
-
"modal_accuracy": 0.7275390625,
|
| 174 |
-
"brier": 0.31872868889688055,
|
| 175 |
-
"cross_entropy": 1.116584448395983,
|
| 176 |
-
"by_kind": {
|
| 177 |
-
"choice": {
|
| 178 |
-
"questions": 665,
|
| 179 |
-
"modal_accuracy": 0.7398496240601504,
|
| 180 |
-
"brier": 0.3486314543321744,
|
| 181 |
-
"cross_entropy": 1.1847615210242326
|
| 182 |
-
},
|
| 183 |
-
"noul": {
|
| 184 |
-
"questions": 144,
|
| 185 |
-
"modal_accuracy": 0.8472222222222222,
|
| 186 |
-
"brier": 0.24375927242920462,
|
| 187 |
-
"cross_entropy": 0.5036143361404846
|
| 188 |
-
},
|
| 189 |
-
"score": {
|
| 190 |
-
"questions": 215,
|
| 191 |
-
"modal_accuracy": 0.6093023255813953,
|
| 192 |
-
"brier": 0.27645081427769413,
|
| 193 |
-
"cross_entropy": 1.3162586012657769
|
| 194 |
-
}
|
| 195 |
-
},
|
| 196 |
-
"by_source": {
|
| 197 |
-
"alfworld": {
|
| 198 |
-
"questions": 128,
|
| 199 |
-
"modal_accuracy": 0.8515625,
|
| 200 |
-
"brier": 0.20112173562767713,
|
| 201 |
-
"cross_entropy": 0.7568937670273396
|
| 202 |
-
},
|
| 203 |
-
"arithmetic": {
|
| 204 |
-
"questions": 24,
|
| 205 |
-
"modal_accuracy": 0.75,
|
| 206 |
-
"brier": 0.3778614792863498,
|
| 207 |
-
"cross_entropy": 1.0070228429929422
|
| 208 |
-
},
|
| 209 |
-
"babyai": {
|
| 210 |
-
"questions": 58,
|
| 211 |
-
"modal_accuracy": 0.5517241379310345,
|
| 212 |
-
"brier": 0.6998687564446888,
|
| 213 |
-
"cross_entropy": 1.449144847710849
|
| 214 |
-
},
|
| 215 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 216 |
-
"questions": 2,
|
| 217 |
-
"modal_accuracy": 1.0,
|
| 218 |
-
"brier": 8.502797038864707e-06,
|
| 219 |
-
"cross_entropy": 0.0017491505823356403
|
| 220 |
-
},
|
| 221 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 222 |
-
"questions": 5,
|
| 223 |
-
"modal_accuracy": 0.6,
|
| 224 |
-
"brier": 0.64090513735759,
|
| 225 |
-
"cross_entropy": 1.2842974245060323
|
| 226 |
-
},
|
| 227 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 228 |
-
"questions": 7,
|
| 229 |
-
"modal_accuracy": 0.7142857142857143,
|
| 230 |
-
"brier": 0.5732229366022509,
|
| 231 |
-
"cross_entropy": 1.3498112142920367
|
| 232 |
-
},
|
| 233 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 234 |
-
"questions": 12,
|
| 235 |
-
"modal_accuracy": 0.75,
|
| 236 |
-
"brier": 0.45820962916227576,
|
| 237 |
-
"cross_entropy": 1.7893452530760972
|
| 238 |
-
},
|
| 239 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 240 |
-
"questions": 10,
|
| 241 |
-
"modal_accuracy": 0.6,
|
| 242 |
-
"brier": 0.7878584523917267,
|
| 243 |
-
"cross_entropy": 2.649907510922513
|
| 244 |
-
},
|
| 245 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 246 |
-
"questions": 4,
|
| 247 |
-
"modal_accuracy": 0.5,
|
| 248 |
-
"brier": 0.8062445012416534,
|
| 249 |
-
"cross_entropy": 1.8181549807503623
|
| 250 |
-
},
|
| 251 |
-
"connect_four": {
|
| 252 |
-
"questions": 56,
|
| 253 |
-
"modal_accuracy": 0.5714285714285714,
|
| 254 |
-
"brier": 0.268346840698743,
|
| 255 |
-
"cross_entropy": 1.8491581077937622
|
| 256 |
-
},
|
| 257 |
-
"customer-control-v1": {
|
| 258 |
-
"questions": 7,
|
| 259 |
-
"modal_accuracy": 1.0,
|
| 260 |
-
"brier": 4.014360934536871e-05,
|
| 261 |
-
"cross_entropy": 0.0018805145026462434
|
| 262 |
-
},
|
| 263 |
-
"devign": {
|
| 264 |
-
"questions": 2,
|
| 265 |
-
"modal_accuracy": 1.0,
|
| 266 |
-
"brier": 0.43785613217853164,
|
| 267 |
-
"cross_entropy": 0.6306845393838758
|
| 268 |
-
},
|
| 269 |
-
"gsm8k": {
|
| 270 |
-
"questions": 4,
|
| 271 |
-
"modal_accuracy": 1.0,
|
| 272 |
-
"brier": 0.0023963047501729595,
|
| 273 |
-
"cross_entropy": 0.02626187987425209
|
| 274 |
-
},
|
| 275 |
-
"helpsteer2": {
|
| 276 |
-
"questions": 128,
|
| 277 |
-
"modal_accuracy": 0.546875,
|
| 278 |
-
"brier": 0.20973734203719346,
|
| 279 |
-
"cross_entropy": 1.5167245986871056
|
| 280 |
-
},
|
| 281 |
-
"planning": {
|
| 282 |
-
"questions": 192,
|
| 283 |
-
"modal_accuracy": 0.9114583333333334,
|
| 284 |
-
"brier": 0.1816467064252445,
|
| 285 |
-
"cross_entropy": 0.8230249970733392
|
| 286 |
-
},
|
| 287 |
-
"race": {
|
| 288 |
-
"questions": 4,
|
| 289 |
-
"modal_accuracy": 0.5,
|
| 290 |
-
"brier": 0.7315860423730358,
|
| 291 |
-
"cross_entropy": 2.0558892146843153
|
| 292 |
-
},
|
| 293 |
-
"ragtruth": {
|
| 294 |
-
"questions": 6,
|
| 295 |
-
"modal_accuracy": 0.5,
|
| 296 |
-
"brier": 0.6525781536661275,
|
| 297 |
-
"cross_entropy": 0.9808095278526777
|
| 298 |
-
},
|
| 299 |
-
"scienceworld": {
|
| 300 |
-
"questions": 128,
|
| 301 |
-
"modal_accuracy": 0.6953125,
|
| 302 |
-
"brier": 0.430457617431226,
|
| 303 |
-
"cross_entropy": 1.3267025517668618
|
| 304 |
-
},
|
| 305 |
-
"scone": {
|
| 306 |
-
"questions": 192,
|
| 307 |
-
"modal_accuracy": 0.6979166666666666,
|
| 308 |
-
"brier": 0.44132333714721916,
|
| 309 |
-
"cross_entropy": 0.939129528604241
|
| 310 |
-
},
|
| 311 |
-
"toolace": {
|
| 312 |
-
"questions": 2,
|
| 313 |
-
"modal_accuracy": 1.0,
|
| 314 |
-
"brier": 1.4302336637850583e-07,
|
| 315 |
-
"cross_entropy": 0.00026036870363595557
|
| 316 |
-
},
|
| 317 |
-
"yuri_v3": {
|
| 318 |
-
"questions": 53,
|
| 319 |
-
"modal_accuracy": 0.7358490566037735,
|
| 320 |
-
"brier": 0.05366664986435064,
|
| 321 |
-
"cross_entropy": 0.8647152527990684
|
| 322 |
-
}
|
| 323 |
-
}
|
| 324 |
-
}
|
| 325 |
-
},
|
| 326 |
-
{
|
| 327 |
-
"step": 4000,
|
| 328 |
-
"examples_seen": 16000,
|
| 329 |
-
"checkpoint": "step-004000",
|
| 330 |
-
"export": "step-004000",
|
| 331 |
-
"validation": {
|
| 332 |
-
"questions": 1024,
|
| 333 |
-
"modal_accuracy": 0.740234375,
|
| 334 |
-
"brier": 0.31259603426070215,
|
| 335 |
-
"cross_entropy": 1.0641685621781927,
|
| 336 |
-
"by_kind": {
|
| 337 |
-
"choice": {
|
| 338 |
-
"questions": 665,
|
| 339 |
-
"modal_accuracy": 0.7413533834586467,
|
| 340 |
-
"brier": 0.35520899641938364,
|
| 341 |
-
"cross_entropy": 1.150404749554445
|
| 342 |
-
},
|
| 343 |
-
"noul": {
|
| 344 |
-
"questions": 144,
|
| 345 |
-
"modal_accuracy": 0.8333333333333334,
|
| 346 |
-
"brier": 0.2587362169867197,
|
| 347 |
-
"cross_entropy": 0.527162702406588
|
| 348 |
-
},
|
| 349 |
-
"score": {
|
| 350 |
-
"questions": 215,
|
| 351 |
-
"modal_accuracy": 0.6744186046511628,
|
| 352 |
-
"brier": 0.2168667033394475,
|
| 353 |
-
"cross_entropy": 1.157107070094022
|
| 354 |
-
}
|
| 355 |
-
},
|
| 356 |
-
"by_source": {
|
| 357 |
-
"alfworld": {
|
| 358 |
-
"questions": 128,
|
| 359 |
-
"modal_accuracy": 0.859375,
|
| 360 |
-
"brier": 0.1875144540034567,
|
| 361 |
-
"cross_entropy": 0.6708904667559872
|
| 362 |
-
},
|
| 363 |
-
"arithmetic": {
|
| 364 |
-
"questions": 24,
|
| 365 |
-
"modal_accuracy": 0.7083333333333334,
|
| 366 |
-
"brier": 0.4071248120462074,
|
| 367 |
-
"cross_entropy": 1.072867837361791
|
| 368 |
-
},
|
| 369 |
-
"babyai": {
|
| 370 |
-
"questions": 58,
|
| 371 |
-
"modal_accuracy": 0.4827586206896552,
|
| 372 |
-
"brier": 0.7226287558174715,
|
| 373 |
-
"cross_entropy": 1.4270151869584489
|
| 374 |
-
},
|
| 375 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 376 |
-
"questions": 2,
|
| 377 |
-
"modal_accuracy": 1.0,
|
| 378 |
-
"brier": 6.599285085529234e-06,
|
| 379 |
-
"cross_entropy": 0.0016185379703032327
|
| 380 |
-
},
|
| 381 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 382 |
-
"questions": 5,
|
| 383 |
-
"modal_accuracy": 0.8,
|
| 384 |
-
"brier": 0.39998067070790544,
|
| 385 |
-
"cross_entropy": 1.08609476980073
|
| 386 |
-
},
|
| 387 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 388 |
-
"questions": 7,
|
| 389 |
-
"modal_accuracy": 0.8571428571428571,
|
| 390 |
-
"brier": 0.28687468365522234,
|
| 391 |
-
"cross_entropy": 0.7248149597143485
|
| 392 |
-
},
|
| 393 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 394 |
-
"questions": 12,
|
| 395 |
-
"modal_accuracy": 0.75,
|
| 396 |
-
"brier": 0.477438586205062,
|
| 397 |
-
"cross_entropy": 1.9109389754190291
|
| 398 |
-
},
|
| 399 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 400 |
-
"questions": 10,
|
| 401 |
-
"modal_accuracy": 0.6,
|
| 402 |
-
"brier": 0.7693115419877218,
|
| 403 |
-
"cross_entropy": 2.051549980800976
|
| 404 |
-
},
|
| 405 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 406 |
-
"questions": 4,
|
| 407 |
-
"modal_accuracy": 0.25,
|
| 408 |
-
"brier": 1.1149087692116901,
|
| 409 |
-
"cross_entropy": 2.466424832470031
|
| 410 |
-
},
|
| 411 |
-
"connect_four": {
|
| 412 |
-
"questions": 56,
|
| 413 |
-
"modal_accuracy": 0.6428571428571429,
|
| 414 |
-
"brier": 0.26805081416207754,
|
| 415 |
-
"cross_entropy": 1.8479820614570504
|
| 416 |
-
},
|
| 417 |
-
"customer-control-v1": {
|
| 418 |
-
"questions": 7,
|
| 419 |
-
"modal_accuracy": 1.0,
|
| 420 |
-
"brier": 1.6518035601985627e-05,
|
| 421 |
-
"cross_entropy": 0.0012598198888345246
|
| 422 |
-
},
|
| 423 |
-
"devign": {
|
| 424 |
-
"questions": 2,
|
| 425 |
-
"modal_accuracy": 0.5,
|
| 426 |
-
"brier": 0.43338634881169164,
|
| 427 |
-
"cross_entropy": 0.6242116836662074
|
| 428 |
-
},
|
| 429 |
-
"gsm8k": {
|
| 430 |
-
"questions": 4,
|
| 431 |
-
"modal_accuracy": 1.0,
|
| 432 |
-
"brier": 0.0005199031989910008,
|
| 433 |
-
"cross_entropy": 0.013970904245707016
|
| 434 |
-
},
|
| 435 |
-
"helpsteer2": {
|
| 436 |
-
"questions": 128,
|
| 437 |
-
"modal_accuracy": 0.5625,
|
| 438 |
-
"brier": 0.2123470190045601,
|
| 439 |
-
"cross_entropy": 1.5161004021555795
|
| 440 |
-
},
|
| 441 |
-
"planning": {
|
| 442 |
-
"questions": 192,
|
| 443 |
-
"modal_accuracy": 0.9010416666666666,
|
| 444 |
-
"brier": 0.18652079389211781,
|
| 445 |
-
"cross_entropy": 0.7723674627217165
|
| 446 |
-
},
|
| 447 |
-
"race": {
|
| 448 |
-
"questions": 4,
|
| 449 |
-
"modal_accuracy": 0.5,
|
| 450 |
-
"brier": 0.6806357215652038,
|
| 451 |
-
"cross_entropy": 1.9339496668179939
|
| 452 |
-
},
|
| 453 |
-
"ragtruth": {
|
| 454 |
-
"questions": 6,
|
| 455 |
-
"modal_accuracy": 0.6666666666666666,
|
| 456 |
-
"brier": 0.5780328053534197,
|
| 457 |
-
"cross_entropy": 0.9064917649110132
|
| 458 |
-
},
|
| 459 |
-
"scienceworld": {
|
| 460 |
-
"questions": 128,
|
| 461 |
-
"modal_accuracy": 0.6796875,
|
| 462 |
-
"brier": 0.49342345206712573,
|
| 463 |
-
"cross_entropy": 1.4302781643246851
|
| 464 |
-
},
|
| 465 |
-
"scone": {
|
| 466 |
-
"questions": 192,
|
| 467 |
-
"modal_accuracy": 0.765625,
|
| 468 |
-
"brier": 0.3713316872108095,
|
| 469 |
-
"cross_entropy": 0.7410961018920553
|
| 470 |
-
},
|
| 471 |
-
"toolace": {
|
| 472 |
-
"questions": 2,
|
| 473 |
-
"modal_accuracy": 1.0,
|
| 474 |
-
"brier": 3.664008183371336e-07,
|
| 475 |
-
"cross_entropy": 0.00041208643515311584
|
| 476 |
-
},
|
| 477 |
-
"yuri_v3": {
|
| 478 |
-
"questions": 53,
|
| 479 |
-
"modal_accuracy": 0.7547169811320755,
|
| 480 |
-
"brier": 0.056719285382028715,
|
| 481 |
-
"cross_entropy": 0.8641864337643139
|
| 482 |
-
}
|
| 483 |
-
}
|
| 484 |
}
|
| 485 |
},
|
| 486 |
-
{
|
| 487 |
-
"
|
| 488 |
-
|
| 489 |
-
|
| 490 |
-
|
| 491 |
-
|
| 492 |
-
"
|
| 493 |
-
"
|
| 494 |
-
|
| 495 |
-
|
| 496 |
-
"
|
| 497 |
-
|
| 498 |
-
|
| 499 |
-
|
| 500 |
-
|
| 501 |
-
|
| 502 |
-
},
|
| 503 |
-
"noul": {
|
| 504 |
-
"questions": 144,
|
| 505 |
-
"modal_accuracy": 0.8958333333333334,
|
| 506 |
-
"brier": 0.16471210037099535,
|
| 507 |
-
"cross_entropy": 0.39482479972079665
|
| 508 |
-
},
|
| 509 |
-
"score": {
|
| 510 |
-
"questions": 215,
|
| 511 |
-
"modal_accuracy": 0.6604651162790698,
|
| 512 |
-
"brier": 0.22656192147115203,
|
| 513 |
-
"cross_entropy": 1.2057654334760237
|
| 514 |
-
}
|
| 515 |
-
},
|
| 516 |
-
"by_source": {
|
| 517 |
-
"alfworld": {
|
| 518 |
-
"questions": 128,
|
| 519 |
-
"modal_accuracy": 0.8671875,
|
| 520 |
-
"brier": 0.1790825849971539,
|
| 521 |
-
"cross_entropy": 0.6080971490404414
|
| 522 |
-
},
|
| 523 |
-
"arithmetic": {
|
| 524 |
-
"questions": 24,
|
| 525 |
-
"modal_accuracy": 0.6666666666666666,
|
| 526 |
-
"brier": 0.5263100827359132,
|
| 527 |
-
"cross_entropy": 1.403012675529207
|
| 528 |
-
},
|
| 529 |
-
"babyai": {
|
| 530 |
-
"questions": 58,
|
| 531 |
-
"modal_accuracy": 0.5517241379310345,
|
| 532 |
-
"brier": 0.684833327087956,
|
| 533 |
-
"cross_entropy": 1.5818612127101548
|
| 534 |
-
},
|
| 535 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 536 |
-
"questions": 2,
|
| 537 |
-
"modal_accuracy": 1.0,
|
| 538 |
-
"brier": 8.422208826818114e-06,
|
| 539 |
-
"cross_entropy": 0.0016334122276107624
|
| 540 |
-
},
|
| 541 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 542 |
-
"questions": 5,
|
| 543 |
-
"modal_accuracy": 0.8,
|
| 544 |
-
"brier": 0.39072800981047207,
|
| 545 |
-
"cross_entropy": 0.8926558622999824
|
| 546 |
-
},
|
| 547 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 548 |
-
"questions": 7,
|
| 549 |
-
"modal_accuracy": 0.8571428571428571,
|
| 550 |
-
"brier": 0.28439123582360026,
|
| 551 |
-
"cross_entropy": 0.7928804085918381
|
| 552 |
-
},
|
| 553 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 554 |
-
"questions": 12,
|
| 555 |
-
"modal_accuracy": 0.6666666666666666,
|
| 556 |
-
"brier": 0.6286826358235,
|
| 557 |
-
"cross_entropy": 2.0004531092707
|
| 558 |
-
},
|
| 559 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 560 |
-
"questions": 10,
|
| 561 |
-
"modal_accuracy": 0.7,
|
| 562 |
-
"brier": 0.44538602593083176,
|
| 563 |
-
"cross_entropy": 1.117689014223367
|
| 564 |
-
},
|
| 565 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 566 |
-
"questions": 4,
|
| 567 |
-
"modal_accuracy": 0.25,
|
| 568 |
-
"brier": 1.4342464951649092,
|
| 569 |
-
"cross_entropy": 3.381933827755728
|
| 570 |
-
},
|
| 571 |
-
"connect_four": {
|
| 572 |
-
"questions": 56,
|
| 573 |
-
"modal_accuracy": 0.5535714285714286,
|
| 574 |
-
"brier": 0.2622696208064482,
|
| 575 |
-
"cross_entropy": 1.8219529101323455
|
| 576 |
-
},
|
| 577 |
-
"customer-control-v1": {
|
| 578 |
-
"questions": 7,
|
| 579 |
-
"modal_accuracy": 1.0,
|
| 580 |
-
"brier": 2.417104630113183e-06,
|
| 581 |
-
"cross_entropy": 0.0005900334430327367
|
| 582 |
-
},
|
| 583 |
-
"devign": {
|
| 584 |
-
"questions": 2,
|
| 585 |
-
"modal_accuracy": 1.0,
|
| 586 |
-
"brier": 0.43780762834608883,
|
| 587 |
-
"cross_entropy": 0.6306076256145883
|
| 588 |
-
},
|
| 589 |
-
"gsm8k": {
|
| 590 |
-
"questions": 4,
|
| 591 |
-
"modal_accuracy": 1.0,
|
| 592 |
-
"brier": 0.0003744313065983422,
|
| 593 |
-
"cross_entropy": 0.011988960485765076
|
| 594 |
-
},
|
| 595 |
-
"helpsteer2": {
|
| 596 |
-
"questions": 128,
|
| 597 |
-
"modal_accuracy": 0.5859375,
|
| 598 |
-
"brier": 0.20697473621325052,
|
| 599 |
-
"cross_entropy": 1.5034032587243362
|
| 600 |
-
},
|
| 601 |
-
"planning": {
|
| 602 |
-
"questions": 192,
|
| 603 |
-
"modal_accuracy": 0.9635416666666666,
|
| 604 |
-
"brier": 0.11802009440200961,
|
| 605 |
-
"cross_entropy": 0.6579051066101222
|
| 606 |
-
},
|
| 607 |
-
"race": {
|
| 608 |
-
"questions": 4,
|
| 609 |
-
"modal_accuracy": 0.5,
|
| 610 |
-
"brier": 0.7315329429054658,
|
| 611 |
-
"cross_entropy": 2.0647249708287245
|
| 612 |
-
},
|
| 613 |
-
"ragtruth": {
|
| 614 |
-
"questions": 6,
|
| 615 |
-
"modal_accuracy": 0.6666666666666666,
|
| 616 |
-
"brier": 0.609860991064251,
|
| 617 |
-
"cross_entropy": 0.9741778978250341
|
| 618 |
-
},
|
| 619 |
-
"scienceworld": {
|
| 620 |
-
"questions": 128,
|
| 621 |
-
"modal_accuracy": 0.703125,
|
| 622 |
-
"brier": 0.44470820194957994,
|
| 623 |
-
"cross_entropy": 1.2102900099661185
|
| 624 |
-
},
|
| 625 |
-
"scone": {
|
| 626 |
-
"questions": 192,
|
| 627 |
-
"modal_accuracy": 0.796875,
|
| 628 |
-
"brier": 0.29338767561802165,
|
| 629 |
-
"cross_entropy": 0.6140028830315581
|
| 630 |
-
},
|
| 631 |
-
"toolace": {
|
| 632 |
-
"questions": 2,
|
| 633 |
-
"modal_accuracy": 1.0,
|
| 634 |
-
"brier": 2.1804417353162663e-07,
|
| 635 |
-
"cross_entropy": 0.0003234946827488933
|
| 636 |
-
},
|
| 637 |
-
"yuri_v3": {
|
| 638 |
-
"questions": 53,
|
| 639 |
-
"modal_accuracy": 0.7169811320754716,
|
| 640 |
-
"brier": 0.056977519481840835,
|
| 641 |
-
"cross_entropy": 0.860752644211149
|
| 642 |
-
}
|
| 643 |
-
}
|
| 644 |
-
}
|
| 645 |
-
}
|
| 646 |
-
],
|
| 647 |
-
"winner": {
|
| 648 |
-
"step": 8000,
|
| 649 |
-
"examples_seen": 32000,
|
| 650 |
-
"checkpoint": "step-008000",
|
| 651 |
-
"export": "step-008000",
|
| 652 |
-
"validation": {
|
| 653 |
-
"questions": 1024,
|
| 654 |
-
"modal_accuracy": 0.76171875,
|
| 655 |
-
"brier": 0.27786017825450965,
|
| 656 |
-
"cross_entropy": 0.9927829125866763,
|
| 657 |
-
"by_kind": {
|
| 658 |
-
"choice": {
|
| 659 |
-
"questions": 665,
|
| 660 |
-
"modal_accuracy": 0.7654135338345864,
|
| 661 |
-
"brier": 0.3189465668614991,
|
| 662 |
-
"cross_entropy": 1.0534065611001757
|
| 663 |
-
},
|
| 664 |
-
"noul": {
|
| 665 |
-
"questions": 144,
|
| 666 |
-
"modal_accuracy": 0.8958333333333334,
|
| 667 |
-
"brier": 0.16471210037099535,
|
| 668 |
-
"cross_entropy": 0.39482479972079665
|
| 669 |
-
},
|
| 670 |
-
"score": {
|
| 671 |
-
"questions": 215,
|
| 672 |
-
"modal_accuracy": 0.6604651162790698,
|
| 673 |
-
"brier": 0.22656192147115203,
|
| 674 |
-
"cross_entropy": 1.2057654334760237
|
| 675 |
-
}
|
| 676 |
},
|
| 677 |
-
"
|
| 678 |
-
"
|
| 679 |
-
|
| 680 |
-
|
| 681 |
-
|
| 682 |
-
|
| 683 |
-
|
| 684 |
-
"arithmetic": {
|
| 685 |
-
"questions": 24,
|
| 686 |
-
"modal_accuracy": 0.6666666666666666,
|
| 687 |
-
"brier": 0.5263100827359132,
|
| 688 |
-
"cross_entropy": 1.403012675529207
|
| 689 |
-
},
|
| 690 |
-
"babyai": {
|
| 691 |
-
"questions": 58,
|
| 692 |
-
"modal_accuracy": 0.5517241379310345,
|
| 693 |
-
"brier": 0.684833327087956,
|
| 694 |
-
"cross_entropy": 1.5818612127101548
|
| 695 |
-
},
|
| 696 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 697 |
-
"questions": 2,
|
| 698 |
-
"modal_accuracy": 1.0,
|
| 699 |
-
"brier": 8.422208826818114e-06,
|
| 700 |
-
"cross_entropy": 0.0016334122276107624
|
| 701 |
-
},
|
| 702 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 703 |
-
"questions": 5,
|
| 704 |
-
"modal_accuracy": 0.8,
|
| 705 |
-
"brier": 0.39072800981047207,
|
| 706 |
-
"cross_entropy": 0.8926558622999824
|
| 707 |
-
},
|
| 708 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 709 |
-
"questions": 7,
|
| 710 |
-
"modal_accuracy": 0.8571428571428571,
|
| 711 |
-
"brier": 0.28439123582360026,
|
| 712 |
-
"cross_entropy": 0.7928804085918381
|
| 713 |
-
},
|
| 714 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 715 |
-
"questions": 12,
|
| 716 |
-
"modal_accuracy": 0.6666666666666666,
|
| 717 |
-
"brier": 0.6286826358235,
|
| 718 |
-
"cross_entropy": 2.0004531092707
|
| 719 |
-
},
|
| 720 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 721 |
-
"questions": 10,
|
| 722 |
-
"modal_accuracy": 0.7,
|
| 723 |
-
"brier": 0.44538602593083176,
|
| 724 |
-
"cross_entropy": 1.117689014223367
|
| 725 |
-
},
|
| 726 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 727 |
-
"questions": 4,
|
| 728 |
-
"modal_accuracy": 0.25,
|
| 729 |
-
"brier": 1.4342464951649092,
|
| 730 |
-
"cross_entropy": 3.381933827755728
|
| 731 |
-
},
|
| 732 |
-
"connect_four": {
|
| 733 |
-
"questions": 56,
|
| 734 |
-
"modal_accuracy": 0.5535714285714286,
|
| 735 |
-
"brier": 0.2622696208064482,
|
| 736 |
-
"cross_entropy": 1.8219529101323455
|
| 737 |
-
},
|
| 738 |
-
"customer-control-v1": {
|
| 739 |
-
"questions": 7,
|
| 740 |
-
"modal_accuracy": 1.0,
|
| 741 |
-
"brier": 2.417104630113183e-06,
|
| 742 |
-
"cross_entropy": 0.0005900334430327367
|
| 743 |
-
},
|
| 744 |
-
"devign": {
|
| 745 |
-
"questions": 2,
|
| 746 |
-
"modal_accuracy": 1.0,
|
| 747 |
-
"brier": 0.43780762834608883,
|
| 748 |
-
"cross_entropy": 0.6306076256145883
|
| 749 |
-
},
|
| 750 |
-
"gsm8k": {
|
| 751 |
-
"questions": 4,
|
| 752 |
-
"modal_accuracy": 1.0,
|
| 753 |
-
"brier": 0.0003744313065983422,
|
| 754 |
-
"cross_entropy": 0.011988960485765076
|
| 755 |
-
},
|
| 756 |
-
"helpsteer2": {
|
| 757 |
-
"questions": 128,
|
| 758 |
-
"modal_accuracy": 0.5859375,
|
| 759 |
-
"brier": 0.20697473621325052,
|
| 760 |
-
"cross_entropy": 1.5034032587243362
|
| 761 |
-
},
|
| 762 |
-
"planning": {
|
| 763 |
-
"questions": 192,
|
| 764 |
-
"modal_accuracy": 0.9635416666666666,
|
| 765 |
-
"brier": 0.11802009440200961,
|
| 766 |
-
"cross_entropy": 0.6579051066101222
|
| 767 |
-
},
|
| 768 |
-
"race": {
|
| 769 |
-
"questions": 4,
|
| 770 |
-
"modal_accuracy": 0.5,
|
| 771 |
-
"brier": 0.7315329429054658,
|
| 772 |
-
"cross_entropy": 2.0647249708287245
|
| 773 |
-
},
|
| 774 |
-
"ragtruth": {
|
| 775 |
-
"questions": 6,
|
| 776 |
-
"modal_accuracy": 0.6666666666666666,
|
| 777 |
-
"brier": 0.609860991064251,
|
| 778 |
-
"cross_entropy": 0.9741778978250341
|
| 779 |
-
},
|
| 780 |
-
"scienceworld": {
|
| 781 |
-
"questions": 128,
|
| 782 |
-
"modal_accuracy": 0.703125,
|
| 783 |
-
"brier": 0.44470820194957994,
|
| 784 |
-
"cross_entropy": 1.2102900099661185
|
| 785 |
-
},
|
| 786 |
-
"scone": {
|
| 787 |
-
"questions": 192,
|
| 788 |
-
"modal_accuracy": 0.796875,
|
| 789 |
-
"brier": 0.29338767561802165,
|
| 790 |
-
"cross_entropy": 0.6140028830315581
|
| 791 |
-
},
|
| 792 |
-
"toolace": {
|
| 793 |
-
"questions": 2,
|
| 794 |
-
"modal_accuracy": 1.0,
|
| 795 |
-
"brier": 2.1804417353162663e-07,
|
| 796 |
-
"cross_entropy": 0.0003234946827488933
|
| 797 |
-
},
|
| 798 |
-
"yuri_v3": {
|
| 799 |
-
"questions": 53,
|
| 800 |
-
"modal_accuracy": 0.7169811320754716,
|
| 801 |
-
"brier": 0.056977519481840835,
|
| 802 |
-
"cross_entropy": 0.860752644211149
|
| 803 |
-
}
|
| 804 |
}
|
| 805 |
}
|
| 806 |
},
|
| 807 |
-
"
|
| 808 |
-
|
| 809 |
-
|
| 810 |
-
|
| 811 |
-
"
|
| 812 |
-
|
| 813 |
-
|
| 814 |
-
|
| 815 |
-
|
| 816 |
-
"
|
| 817 |
-
"
|
| 818 |
-
|
| 819 |
-
|
| 820 |
-
|
| 821 |
-
|
| 822 |
-
|
| 823 |
-
"noul": {
|
| 824 |
-
"questions": 144,
|
| 825 |
-
"modal_accuracy": 0.7361111111111112,
|
| 826 |
-
"brier": 0.4178836877295325,
|
| 827 |
-
"cross_entropy": 0.9008511909421006
|
| 828 |
-
},
|
| 829 |
-
"score": {
|
| 830 |
-
"questions": 215,
|
| 831 |
-
"modal_accuracy": 0.5023255813953489,
|
| 832 |
-
"brier": 0.5745442767822497,
|
| 833 |
-
"cross_entropy": 2.347590209909279
|
| 834 |
-
}
|
| 835 |
},
|
| 836 |
-
"
|
| 837 |
-
"
|
| 838 |
-
|
| 839 |
-
|
| 840 |
-
|
| 841 |
-
|
| 842 |
-
|
| 843 |
-
"arithmetic": {
|
| 844 |
-
"questions": 24,
|
| 845 |
-
"modal_accuracy": 0.7083333333333334,
|
| 846 |
-
"brier": 0.428241328947602,
|
| 847 |
-
"cross_entropy": 1.050083253299498
|
| 848 |
-
},
|
| 849 |
-
"babyai": {
|
| 850 |
-
"questions": 58,
|
| 851 |
-
"modal_accuracy": 0.39655172413793105,
|
| 852 |
-
"brier": 0.7305371209519513,
|
| 853 |
-
"cross_entropy": 1.4973851876562216
|
| 854 |
-
},
|
| 855 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 856 |
-
"questions": 2,
|
| 857 |
-
"modal_accuracy": 1.0,
|
| 858 |
-
"brier": 4.986938069711992e-05,
|
| 859 |
-
"cross_entropy": 0.003831617325445817
|
| 860 |
-
},
|
| 861 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 862 |
-
"questions": 5,
|
| 863 |
-
"modal_accuracy": 0.6,
|
| 864 |
-
"brier": 0.7805844658893524,
|
| 865 |
-
"cross_entropy": 1.7801497473019683
|
| 866 |
-
},
|
| 867 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 868 |
-
"questions": 7,
|
| 869 |
-
"modal_accuracy": 0.7142857142857143,
|
| 870 |
-
"brier": 0.4790055889595874,
|
| 871 |
-
"cross_entropy": 1.1121963454176031
|
| 872 |
-
},
|
| 873 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 874 |
-
"questions": 12,
|
| 875 |
-
"modal_accuracy": 0.6666666666666666,
|
| 876 |
-
"brier": 0.6101325803838552,
|
| 877 |
-
"cross_entropy": 2.0033591272403792
|
| 878 |
-
},
|
| 879 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 880 |
-
"questions": 10,
|
| 881 |
-
"modal_accuracy": 0.7,
|
| 882 |
-
"brier": 0.6003855550573081,
|
| 883 |
-
"cross_entropy": 1.611779150841159
|
| 884 |
-
},
|
| 885 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 886 |
-
"questions": 4,
|
| 887 |
-
"modal_accuracy": 0.75,
|
| 888 |
-
"brier": 0.42100449664852624,
|
| 889 |
-
"cross_entropy": 1.9128633478153523
|
| 890 |
-
},
|
| 891 |
-
"connect_four": {
|
| 892 |
-
"questions": 56,
|
| 893 |
-
"modal_accuracy": 0.5357142857142857,
|
| 894 |
-
"brier": 0.2741592752209492,
|
| 895 |
-
"cross_entropy": 1.860460665385586
|
| 896 |
-
},
|
| 897 |
-
"customer-control-v1": {
|
| 898 |
-
"questions": 7,
|
| 899 |
-
"modal_accuracy": 1.0,
|
| 900 |
-
"brier": 2.0002025677443218e-06,
|
| 901 |
-
"cross_entropy": 0.0004476462766485558
|
| 902 |
-
},
|
| 903 |
-
"devign": {
|
| 904 |
-
"questions": 2,
|
| 905 |
-
"modal_accuracy": 0.5,
|
| 906 |
-
"brier": 0.41023068518803085,
|
| 907 |
-
"cross_entropy": 0.6006732852714238
|
| 908 |
-
},
|
| 909 |
-
"gsm8k": {
|
| 910 |
-
"questions": 4,
|
| 911 |
-
"modal_accuracy": 1.0,
|
| 912 |
-
"brier": 0.0013119361198151522,
|
| 913 |
-
"cross_entropy": 0.02163855006976333
|
| 914 |
-
},
|
| 915 |
-
"helpsteer2": {
|
| 916 |
-
"questions": 128,
|
| 917 |
-
"modal_accuracy": 0.4296875,
|
| 918 |
-
"brier": 0.5887451086780798,
|
| 919 |
-
"cross_entropy": 2.801737422463417
|
| 920 |
-
},
|
| 921 |
-
"planning": {
|
| 922 |
-
"questions": 192,
|
| 923 |
-
"modal_accuracy": 0.6041666666666666,
|
| 924 |
-
"brier": 0.5293956848932907,
|
| 925 |
-
"cross_entropy": 1.6646787560472525
|
| 926 |
-
},
|
| 927 |
-
"race": {
|
| 928 |
-
"questions": 4,
|
| 929 |
-
"modal_accuracy": 0.5,
|
| 930 |
-
"brier": 0.8826525379691635,
|
| 931 |
-
"cross_entropy": 2.2968173663260414
|
| 932 |
-
},
|
| 933 |
-
"ragtruth": {
|
| 934 |
-
"questions": 6,
|
| 935 |
-
"modal_accuracy": 0.6666666666666666,
|
| 936 |
-
"brier": 0.31120510035647425,
|
| 937 |
-
"cross_entropy": 0.4402550888479215
|
| 938 |
-
},
|
| 939 |
-
"scienceworld": {
|
| 940 |
-
"questions": 128,
|
| 941 |
-
"modal_accuracy": 0.6953125,
|
| 942 |
-
"brier": 0.4588827899272394,
|
| 943 |
-
"cross_entropy": 1.3847298212396886
|
| 944 |
-
},
|
| 945 |
-
"scone": {
|
| 946 |
-
"questions": 192,
|
| 947 |
-
"modal_accuracy": 0.4739583333333333,
|
| 948 |
-
"brier": 0.8022954836937664,
|
| 949 |
-
"cross_entropy": 2.0042890400598963
|
| 950 |
-
},
|
| 951 |
-
"toolace": {
|
| 952 |
-
"questions": 2,
|
| 953 |
-
"modal_accuracy": 1.0,
|
| 954 |
-
"brier": 2.2949296518888497e-08,
|
| 955 |
-
"cross_entropy": 9.582785411797238e-05
|
| 956 |
-
},
|
| 957 |
-
"yuri_v3": {
|
| 958 |
-
"questions": 53,
|
| 959 |
-
"modal_accuracy": 0.7358490566037735,
|
| 960 |
-
"brier": 0.0564251534824245,
|
| 961 |
-
"cross_entropy": 0.8602018822761758
|
| 962 |
-
}
|
| 963 |
-
}
|
| 964 |
-
},
|
| 965 |
-
"new_test": {
|
| 966 |
-
"questions": 512,
|
| 967 |
-
"modal_accuracy": 0.46484375,
|
| 968 |
-
"brier": 0.7260609345715332,
|
| 969 |
-
"cross_entropy": 2.283754099438618,
|
| 970 |
-
"by_kind": {
|
| 971 |
-
"choice": {
|
| 972 |
-
"questions": 241,
|
| 973 |
-
"modal_accuracy": 0.3941908713692946,
|
| 974 |
-
"brier": 0.814820900706993,
|
| 975 |
-
"cross_entropy": 2.425967860593386
|
| 976 |
-
},
|
| 977 |
-
"noul": {
|
| 978 |
-
"questions": 94,
|
| 979 |
-
"modal_accuracy": 0.5851063829787234,
|
| 980 |
-
"brier": 0.7046781668962778,
|
| 981 |
-
"cross_entropy": 1.348769508075005
|
| 982 |
-
},
|
| 983 |
-
"score": {
|
| 984 |
-
"questions": 177,
|
| 985 |
-
"modal_accuracy": 0.4971751412429379,
|
| 986 |
-
"brier": 0.6165627895027659,
|
| 987 |
-
"cross_entropy": 2.586663902545287
|
| 988 |
-
}
|
| 989 |
},
|
| 990 |
-
"
|
| 991 |
-
"
|
| 992 |
-
|
| 993 |
-
|
| 994 |
-
|
| 995 |
-
|
| 996 |
-
|
| 997 |
-
"planning": {
|
| 998 |
-
"questions": 192,
|
| 999 |
-
"modal_accuracy": 0.484375,
|
| 1000 |
-
"brier": 0.695581487251716,
|
| 1001 |
-
"cross_entropy": 2.128417640206426
|
| 1002 |
-
},
|
| 1003 |
-
"scone": {
|
| 1004 |
-
"questions": 192,
|
| 1005 |
-
"modal_accuracy": 0.421875,
|
| 1006 |
-
"brier": 0.8903884009045245,
|
| 1007 |
-
"cross_entropy": 2.1523295323408504
|
| 1008 |
-
}
|
| 1009 |
}
|
| 1010 |
},
|
| 1011 |
-
"
|
| 1012 |
-
"
|
| 1013 |
-
|
| 1014 |
-
|
| 1015 |
-
|
| 1016 |
-
|
| 1017 |
-
"
|
| 1018 |
-
|
| 1019 |
-
"modal_accuracy": 0.7176368375325803,
|
| 1020 |
-
"brier": 0.34765282650467216,
|
| 1021 |
-
"cross_entropy": 1.0022488366720779
|
| 1022 |
-
},
|
| 1023 |
-
"noul": {
|
| 1024 |
-
"questions": 649,
|
| 1025 |
-
"modal_accuracy": 0.9090909090909091,
|
| 1026 |
-
"brier": 0.1294082456235435,
|
| 1027 |
-
"cross_entropy": 0.3089760935689536
|
| 1028 |
-
},
|
| 1029 |
-
"score": {
|
| 1030 |
-
"questions": 248,
|
| 1031 |
-
"modal_accuracy": 0.8588709677419355,
|
| 1032 |
-
"brier": 0.1295168598982062,
|
| 1033 |
-
"cross_entropy": 0.6581264080729309
|
| 1034 |
-
}
|
| 1035 |
},
|
| 1036 |
-
"
|
| 1037 |
-
"
|
| 1038 |
-
|
| 1039 |
-
|
| 1040 |
-
|
| 1041 |
-
|
| 1042 |
-
|
| 1043 |
-
"arithmetic": {
|
| 1044 |
-
"questions": 30,
|
| 1045 |
-
"modal_accuracy": 0.8,
|
| 1046 |
-
"brier": 0.365987202749238,
|
| 1047 |
-
"cross_entropy": 0.874987503758623
|
| 1048 |
-
},
|
| 1049 |
-
"babyai": {
|
| 1050 |
-
"questions": 128,
|
| 1051 |
-
"modal_accuracy": 0.4140625,
|
| 1052 |
-
"brier": 0.7052731307744691,
|
| 1053 |
-
"cross_entropy": 1.4919952201710827
|
| 1054 |
-
},
|
| 1055 |
-
"community-diversity-v2/browser_tools/goal_completion": {
|
| 1056 |
-
"questions": 24,
|
| 1057 |
-
"modal_accuracy": 0.9583333333333334,
|
| 1058 |
-
"brier": 0.0757580121004977,
|
| 1059 |
-
"cross_entropy": 0.10499604055191543
|
| 1060 |
-
},
|
| 1061 |
-
"community-diversity-v2/contracts/exclusivity_exception": {
|
| 1062 |
-
"questions": 23,
|
| 1063 |
-
"modal_accuracy": 0.8695652173913043,
|
| 1064 |
-
"brier": 0.17625150922502755,
|
| 1065 |
-
"cross_entropy": 0.3819911750055886
|
| 1066 |
-
},
|
| 1067 |
-
"community-diversity-v2/games/pit_stop": {
|
| 1068 |
-
"questions": 13,
|
| 1069 |
-
"modal_accuracy": 0.46153846153846156,
|
| 1070 |
-
"brier": 0.8058966314340112,
|
| 1071 |
-
"cross_entropy": 1.7033569197410379
|
| 1072 |
-
},
|
| 1073 |
-
"community-diversity-v2/ir/warranty_transfer": {
|
| 1074 |
-
"questions": 15,
|
| 1075 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1076 |
-
"brier": 0.13580214677968178,
|
| 1077 |
-
"cross_entropy": 0.36657722104104
|
| 1078 |
-
},
|
| 1079 |
-
"community-diversity-v2/rag/independent_sources": {
|
| 1080 |
-
"questions": 21,
|
| 1081 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1082 |
-
"brier": 0.5103346094744698,
|
| 1083 |
-
"cross_entropy": 0.9877849897459301
|
| 1084 |
-
},
|
| 1085 |
-
"community-diversity-v2/rubric_judge/balanced_tradeoff": {
|
| 1086 |
-
"questions": 12,
|
| 1087 |
-
"modal_accuracy": 0.8333333333333334,
|
| 1088 |
-
"brier": 0.31827468282287835,
|
| 1089 |
-
"cross_entropy": 0.7584622273231553
|
| 1090 |
-
},
|
| 1091 |
-
"community-diversity-v2/shell_history/incremental_copy": {
|
| 1092 |
-
"questions": 17,
|
| 1093 |
-
"modal_accuracy": 0.8823529411764706,
|
| 1094 |
-
"brier": 0.23477636259692025,
|
| 1095 |
-
"cross_entropy": 0.5888464829632447
|
| 1096 |
-
},
|
| 1097 |
-
"community-diversity-v2/support/outage_compensation": {
|
| 1098 |
-
"questions": 3,
|
| 1099 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1100 |
-
"brier": 0.46034493119811687,
|
| 1101 |
-
"cross_entropy": 1.0544000105768865
|
| 1102 |
-
},
|
| 1103 |
-
"connect_four": {
|
| 1104 |
-
"questions": 128,
|
| 1105 |
-
"modal_accuracy": 0.5234375,
|
| 1106 |
-
"brier": 0.2605442517143562,
|
| 1107 |
-
"cross_entropy": 1.8198101893170386
|
| 1108 |
-
},
|
| 1109 |
-
"customer-control-v1": {
|
| 1110 |
-
"questions": 45,
|
| 1111 |
-
"modal_accuracy": 1.0,
|
| 1112 |
-
"brier": 0.0003903831787449739,
|
| 1113 |
-
"cross_entropy": 0.01599108917033031
|
| 1114 |
-
},
|
| 1115 |
-
"devign": {
|
| 1116 |
-
"questions": 46,
|
| 1117 |
-
"modal_accuracy": 0.5434782608695652,
|
| 1118 |
-
"brier": 0.5038973081034425,
|
| 1119 |
-
"cross_entropy": 0.6986062066022048
|
| 1120 |
-
},
|
| 1121 |
-
"entity_belief_tracking": {
|
| 1122 |
-
"questions": 19,
|
| 1123 |
-
"modal_accuracy": 1.0,
|
| 1124 |
-
"brier": 0.003996155280999412,
|
| 1125 |
-
"cross_entropy": 0.018453305162397043
|
| 1126 |
-
},
|
| 1127 |
-
"event_state_reconstruction": {
|
| 1128 |
-
"questions": 30,
|
| 1129 |
-
"modal_accuracy": 1.0,
|
| 1130 |
-
"brier": 0.000632084906836889,
|
| 1131 |
-
"cross_entropy": 0.0070833207931613705
|
| 1132 |
-
},
|
| 1133 |
-
"event_triggered_execution": {
|
| 1134 |
-
"questions": 18,
|
| 1135 |
-
"modal_accuracy": 0.8888888888888888,
|
| 1136 |
-
"brier": 0.20065607784122677,
|
| 1137 |
-
"cross_entropy": 0.3866436913361192
|
| 1138 |
-
},
|
| 1139 |
-
"evidence_sufficiency": {
|
| 1140 |
-
"questions": 13,
|
| 1141 |
-
"modal_accuracy": 0.7692307692307693,
|
| 1142 |
-
"brier": 0.3885272040680112,
|
| 1143 |
-
"cross_entropy": 0.7354956714769301
|
| 1144 |
-
},
|
| 1145 |
-
"goal_directed_planning": {
|
| 1146 |
-
"questions": 19,
|
| 1147 |
-
"modal_accuracy": 0.8421052631578947,
|
| 1148 |
-
"brier": 0.24661119154460764,
|
| 1149 |
-
"cross_entropy": 0.80410976975599
|
| 1150 |
-
},
|
| 1151 |
-
"graph_navigation": {
|
| 1152 |
-
"questions": 6,
|
| 1153 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1154 |
-
"brier": 0.5177916242601774,
|
| 1155 |
-
"cross_entropy": 1.158343409907232
|
| 1156 |
-
},
|
| 1157 |
-
"gsm8k": {
|
| 1158 |
-
"questions": 106,
|
| 1159 |
-
"modal_accuracy": 0.7547169811320755,
|
| 1160 |
-
"brier": 0.38874049122073956,
|
| 1161 |
-
"cross_entropy": 0.7555982932600139
|
| 1162 |
-
},
|
| 1163 |
-
"instruction_execution": {
|
| 1164 |
-
"questions": 12,
|
| 1165 |
-
"modal_accuracy": 0.9166666666666666,
|
| 1166 |
-
"brier": 0.16643936427800027,
|
| 1167 |
-
"cross_entropy": 0.5910813947667087
|
| 1168 |
-
},
|
| 1169 |
-
"multi_view_adjudication": {
|
| 1170 |
-
"questions": 20,
|
| 1171 |
-
"modal_accuracy": 0.95,
|
| 1172 |
-
"brier": 0.07032918266188046,
|
| 1173 |
-
"cross_entropy": 0.09175669834651938
|
| 1174 |
-
},
|
| 1175 |
-
"needle_retrieval": {
|
| 1176 |
-
"questions": 5,
|
| 1177 |
-
"modal_accuracy": 1.0,
|
| 1178 |
-
"brier": 1.0762412043050325e-08,
|
| 1179 |
-
"cross_entropy": 4.8760052363298926e-05
|
| 1180 |
-
},
|
| 1181 |
-
"partial_observation_calibration": {
|
| 1182 |
-
"questions": 5,
|
| 1183 |
-
"modal_accuracy": 0.8,
|
| 1184 |
-
"brier": 0.02878926642066244,
|
| 1185 |
-
"cross_entropy": 0.5423652505807739
|
| 1186 |
-
},
|
| 1187 |
-
"policy_applicability": {
|
| 1188 |
-
"questions": 17,
|
| 1189 |
-
"modal_accuracy": 1.0,
|
| 1190 |
-
"brier": 0.000332556851194119,
|
| 1191 |
-
"cross_entropy": 0.006342650317050527
|
| 1192 |
-
},
|
| 1193 |
-
"policy_under_uncertainty": {
|
| 1194 |
-
"questions": 8,
|
| 1195 |
-
"modal_accuracy": 1.0,
|
| 1196 |
-
"brier": 0.055895947994874184,
|
| 1197 |
-
"cross_entropy": 0.5045127678250978
|
| 1198 |
-
},
|
| 1199 |
-
"race": {
|
| 1200 |
-
"questions": 104,
|
| 1201 |
-
"modal_accuracy": 0.9134615384615384,
|
| 1202 |
-
"brier": 0.15016846124700772,
|
| 1203 |
-
"cross_entropy": 0.3446495798522507
|
| 1204 |
-
},
|
| 1205 |
-
"ragtruth": {
|
| 1206 |
-
"questions": 128,
|
| 1207 |
-
"modal_accuracy": 0.84375,
|
| 1208 |
-
"brier": 0.2673585181473687,
|
| 1209 |
-
"cross_entropy": 0.4660759647283446
|
| 1210 |
-
},
|
| 1211 |
-
"reasoning-control-v1": {
|
| 1212 |
-
"questions": 45,
|
| 1213 |
-
"modal_accuracy": 0.9111111111111111,
|
| 1214 |
-
"brier": 0.11092651727785302,
|
| 1215 |
-
"cross_entropy": 0.2669770296039986
|
| 1216 |
-
},
|
| 1217 |
-
"record_aggregation": {
|
| 1218 |
-
"questions": 14,
|
| 1219 |
-
"modal_accuracy": 0.7857142857142857,
|
| 1220 |
-
"brier": 0.29174772725112047,
|
| 1221 |
-
"cross_entropy": 0.45793008149173803
|
| 1222 |
-
},
|
| 1223 |
-
"record_eligibility": {
|
| 1224 |
-
"questions": 8,
|
| 1225 |
-
"modal_accuracy": 0.875,
|
| 1226 |
-
"brier": 0.10770378005071689,
|
| 1227 |
-
"cross_entropy": 0.13562242943700845
|
| 1228 |
-
},
|
| 1229 |
-
"scienceworld": {
|
| 1230 |
-
"questions": 128,
|
| 1231 |
-
"modal_accuracy": 0.5859375,
|
| 1232 |
-
"brier": 0.6006833831415513,
|
| 1233 |
-
"cross_entropy": 1.7746254397586039
|
| 1234 |
-
},
|
| 1235 |
-
"snake-v1": {
|
| 1236 |
-
"questions": 30,
|
| 1237 |
-
"modal_accuracy": 0.9666666666666667,
|
| 1238 |
-
"brier": 0.04547834255423711,
|
| 1239 |
-
"cross_entropy": 0.25335150127501904
|
| 1240 |
-
},
|
| 1241 |
-
"state_perturbation": {
|
| 1242 |
-
"questions": 11,
|
| 1243 |
-
"modal_accuracy": 0.8181818181818182,
|
| 1244 |
-
"brier": 0.3589651830631727,
|
| 1245 |
-
"cross_entropy": 0.9294419729230835
|
| 1246 |
-
},
|
| 1247 |
-
"table_lookup": {
|
| 1248 |
-
"questions": 13,
|
| 1249 |
-
"modal_accuracy": 0.8461538461538461,
|
| 1250 |
-
"brier": 0.1995349559267647,
|
| 1251 |
-
"cross_entropy": 0.7888754577823849
|
| 1252 |
-
},
|
| 1253 |
-
"taxonomy_routing": {
|
| 1254 |
-
"questions": 8,
|
| 1255 |
-
"modal_accuracy": 1.0,
|
| 1256 |
-
"brier": 4.5835879900615156e-05,
|
| 1257 |
-
"cross_entropy": 0.002082040171618654
|
| 1258 |
-
},
|
| 1259 |
-
"tic_tac_toe-v1": {
|
| 1260 |
-
"questions": 15,
|
| 1261 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1262 |
-
"brier": 0.2455337950398313,
|
| 1263 |
-
"cross_entropy": 1.1492408686263713
|
| 1264 |
-
},
|
| 1265 |
-
"tile_platformer-v1": {
|
| 1266 |
-
"questions": 15,
|
| 1267 |
-
"modal_accuracy": 1.0,
|
| 1268 |
-
"brier": 0.023323220046764518,
|
| 1269 |
-
"cross_entropy": 0.08534179201524755
|
| 1270 |
-
},
|
| 1271 |
-
"toolace": {
|
| 1272 |
-
"questions": 128,
|
| 1273 |
-
"modal_accuracy": 0.9921875,
|
| 1274 |
-
"brier": 0.015580959620615913,
|
| 1275 |
-
"cross_entropy": 0.043558856167122205
|
| 1276 |
-
},
|
| 1277 |
-
"trex_runner-v1": {
|
| 1278 |
-
"questions": 2,
|
| 1279 |
-
"modal_accuracy": 1.0,
|
| 1280 |
-
"brier": 0.01809983422643395,
|
| 1281 |
-
"cross_entropy": 0.11356731402761971
|
| 1282 |
-
},
|
| 1283 |
-
"vizdoom-basic-v1": {
|
| 1284 |
-
"questions": 45,
|
| 1285 |
-
"modal_accuracy": 0.9555555555555556,
|
| 1286 |
-
"brier": 0.08476731050032603,
|
| 1287 |
-
"cross_entropy": 0.18352009286167528
|
| 1288 |
-
},
|
| 1289 |
-
"wanli-decisions-v1": {
|
| 1290 |
-
"questions": 128,
|
| 1291 |
-
"modal_accuracy": 0.6953125,
|
| 1292 |
-
"brier": 0.5342434192250224,
|
| 1293 |
-
"cross_entropy": 1.027889201322489
|
| 1294 |
-
},
|
| 1295 |
-
"workflow-controls-v1/agent_trace_observability": {
|
| 1296 |
-
"questions": 15,
|
| 1297 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1298 |
-
"brier": 0.13236608755793375,
|
| 1299 |
-
"cross_entropy": 0.37446550855959687
|
| 1300 |
-
},
|
| 1301 |
-
"workflow-controls-v1/customer_service": {
|
| 1302 |
-
"questions": 15,
|
| 1303 |
-
"modal_accuracy": 1.0,
|
| 1304 |
-
"brier": 9.801834989730725e-07,
|
| 1305 |
-
"cross_entropy": 0.00023351213246618595
|
| 1306 |
-
},
|
| 1307 |
-
"workflow-controls-v1/invoice_processing": {
|
| 1308 |
-
"questions": 15,
|
| 1309 |
-
"modal_accuracy": 1.0,
|
| 1310 |
-
"brier": 4.168477226094833e-07,
|
| 1311 |
-
"cross_entropy": 0.00024165434698863794
|
| 1312 |
-
},
|
| 1313 |
-
"workflow-controls-v1/security_incidents": {
|
| 1314 |
-
"questions": 14,
|
| 1315 |
-
"modal_accuracy": 1.0,
|
| 1316 |
-
"brier": 2.5913148618252294e-09,
|
| 1317 |
-
"cross_entropy": 1.9618643712043932e-05
|
| 1318 |
-
},
|
| 1319 |
-
"yuri_v3": {
|
| 1320 |
-
"questions": 256,
|
| 1321 |
-
"modal_accuracy": 0.8671875,
|
| 1322 |
-
"brier": 0.03336715127934959,
|
| 1323 |
-
"cross_entropy": 0.8210925298273019
|
| 1324 |
-
}
|
| 1325 |
-
}
|
| 1326 |
-
},
|
| 1327 |
-
"transfer": {
|
| 1328 |
-
"questions": 89,
|
| 1329 |
-
"modal_accuracy": 0.6404494382022472,
|
| 1330 |
-
"brier": 0.49972145070841484,
|
| 1331 |
-
"cross_entropy": 1.069735656698559,
|
| 1332 |
-
"by_kind": {
|
| 1333 |
-
"choice": {
|
| 1334 |
-
"questions": 33,
|
| 1335 |
-
"modal_accuracy": 0.6060606060606061,
|
| 1336 |
-
"brier": 0.5839105237267139,
|
| 1337 |
-
"cross_entropy": 1.3151504789912494
|
| 1338 |
-
},
|
| 1339 |
-
"noul": {
|
| 1340 |
-
"questions": 41,
|
| 1341 |
-
"modal_accuracy": 0.7560975609756098,
|
| 1342 |
-
"brier": 0.34728857025217713,
|
| 1343 |
-
"cross_entropy": 0.6781416340204177
|
| 1344 |
-
},
|
| 1345 |
-
"score": {
|
| 1346 |
-
"questions": 15,
|
| 1347 |
-
"modal_accuracy": 0.4,
|
| 1348 |
-
"brier": 0.7311553633152064,
|
| 1349 |
-
"cross_entropy": 1.6001800429748936
|
| 1350 |
-
}
|
| 1351 |
},
|
| 1352 |
-
"
|
| 1353 |
-
"
|
| 1354 |
-
|
| 1355 |
-
|
| 1356 |
-
|
| 1357 |
-
|
| 1358 |
-
|
| 1359 |
-
"code": {
|
| 1360 |
-
"questions": 6,
|
| 1361 |
-
"modal_accuracy": 0.5,
|
| 1362 |
-
"brier": 0.6355124607251713,
|
| 1363 |
-
"cross_entropy": 0.9367448318704151
|
| 1364 |
-
},
|
| 1365 |
-
"dialog": {
|
| 1366 |
-
"questions": 12,
|
| 1367 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1368 |
-
"brier": 0.4233092995278946,
|
| 1369 |
-
"cross_entropy": 1.4220052117820898
|
| 1370 |
-
},
|
| 1371 |
-
"judge": {
|
| 1372 |
-
"questions": 8,
|
| 1373 |
-
"modal_accuracy": 0.75,
|
| 1374 |
-
"brier": 0.4294109009179358,
|
| 1375 |
-
"cross_entropy": 1.1813213005452772
|
| 1376 |
-
},
|
| 1377 |
-
"know": {
|
| 1378 |
-
"questions": 10,
|
| 1379 |
-
"modal_accuracy": 0.8,
|
| 1380 |
-
"brier": 0.22602505333734763,
|
| 1381 |
-
"cross_entropy": 0.39225738115288933
|
| 1382 |
-
},
|
| 1383 |
-
"ret": {
|
| 1384 |
-
"questions": 10,
|
| 1385 |
-
"modal_accuracy": 0.4,
|
| 1386 |
-
"brier": 0.9045065751061194,
|
| 1387 |
-
"cross_entropy": 1.655086462390568
|
| 1388 |
-
},
|
| 1389 |
-
"route": {
|
| 1390 |
-
"questions": 9,
|
| 1391 |
-
"modal_accuracy": 0.4444444444444444,
|
| 1392 |
-
"brier": 0.6393115918944736,
|
| 1393 |
-
"cross_entropy": 1.1139742544661488
|
| 1394 |
-
},
|
| 1395 |
-
"safety": {
|
| 1396 |
-
"questions": 12,
|
| 1397 |
-
"modal_accuracy": 0.9166666666666666,
|
| 1398 |
-
"brier": 0.17156815175055062,
|
| 1399 |
-
"cross_entropy": 0.4272056920862004
|
| 1400 |
-
},
|
| 1401 |
-
"step": {
|
| 1402 |
-
"questions": 8,
|
| 1403 |
-
"modal_accuracy": 0.625,
|
| 1404 |
-
"brier": 0.710965126411655,
|
| 1405 |
-
"cross_entropy": 1.9533480392831426
|
| 1406 |
-
},
|
| 1407 |
-
"struct": {
|
| 1408 |
-
"questions": 2,
|
| 1409 |
-
"modal_accuracy": 0.5,
|
| 1410 |
-
"brier": 0.8092234753736499,
|
| 1411 |
-
"cross_entropy": 1.1517337483418988
|
| 1412 |
-
}
|
| 1413 |
}
|
| 1414 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1415 |
},
|
| 1416 |
-
"
|
| 1417 |
-
"
|
| 1418 |
-
"questions":
|
| 1419 |
-
"
|
| 1420 |
-
"brier": 0.
|
| 1421 |
-
"cross_entropy":
|
| 1422 |
-
"by_kind": {
|
| 1423 |
-
"choice": {
|
| 1424 |
-
"questions": 665,
|
| 1425 |
-
"modal_accuracy": 0.7654135338345864,
|
| 1426 |
-
"brier": 0.3189465668614991,
|
| 1427 |
-
"cross_entropy": 1.0534065611001757
|
| 1428 |
-
},
|
| 1429 |
-
"noul": {
|
| 1430 |
-
"questions": 144,
|
| 1431 |
-
"modal_accuracy": 0.8958333333333334,
|
| 1432 |
-
"brier": 0.16471210037099535,
|
| 1433 |
-
"cross_entropy": 0.39482479972079665
|
| 1434 |
-
},
|
| 1435 |
-
"score": {
|
| 1436 |
-
"questions": 215,
|
| 1437 |
-
"modal_accuracy": 0.6604651162790698,
|
| 1438 |
-
"brier": 0.22656192147115203,
|
| 1439 |
-
"cross_entropy": 1.2057654334760237
|
| 1440 |
-
}
|
| 1441 |
-
},
|
| 1442 |
-
"by_source": {
|
| 1443 |
-
"alfworld": {
|
| 1444 |
-
"questions": 128,
|
| 1445 |
-
"modal_accuracy": 0.8671875,
|
| 1446 |
-
"brier": 0.1790825849971539,
|
| 1447 |
-
"cross_entropy": 0.6080971490404414
|
| 1448 |
-
},
|
| 1449 |
-
"arithmetic": {
|
| 1450 |
-
"questions": 24,
|
| 1451 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1452 |
-
"brier": 0.5263100827359132,
|
| 1453 |
-
"cross_entropy": 1.403012675529207
|
| 1454 |
-
},
|
| 1455 |
-
"babyai": {
|
| 1456 |
-
"questions": 58,
|
| 1457 |
-
"modal_accuracy": 0.5517241379310345,
|
| 1458 |
-
"brier": 0.684833327087956,
|
| 1459 |
-
"cross_entropy": 1.5818612127101548
|
| 1460 |
-
},
|
| 1461 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 1462 |
-
"questions": 2,
|
| 1463 |
-
"modal_accuracy": 1.0,
|
| 1464 |
-
"brier": 8.422208826818114e-06,
|
| 1465 |
-
"cross_entropy": 0.0016334122276107624
|
| 1466 |
-
},
|
| 1467 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 1468 |
-
"questions": 5,
|
| 1469 |
-
"modal_accuracy": 0.8,
|
| 1470 |
-
"brier": 0.39072800981047207,
|
| 1471 |
-
"cross_entropy": 0.8926558622999824
|
| 1472 |
-
},
|
| 1473 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 1474 |
-
"questions": 7,
|
| 1475 |
-
"modal_accuracy": 0.8571428571428571,
|
| 1476 |
-
"brier": 0.28439123582360026,
|
| 1477 |
-
"cross_entropy": 0.7928804085918381
|
| 1478 |
-
},
|
| 1479 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 1480 |
-
"questions": 12,
|
| 1481 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1482 |
-
"brier": 0.6286826358235,
|
| 1483 |
-
"cross_entropy": 2.0004531092707
|
| 1484 |
-
},
|
| 1485 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 1486 |
-
"questions": 10,
|
| 1487 |
-
"modal_accuracy": 0.7,
|
| 1488 |
-
"brier": 0.44538602593083176,
|
| 1489 |
-
"cross_entropy": 1.117689014223367
|
| 1490 |
-
},
|
| 1491 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 1492 |
-
"questions": 4,
|
| 1493 |
-
"modal_accuracy": 0.25,
|
| 1494 |
-
"brier": 1.4342464951649092,
|
| 1495 |
-
"cross_entropy": 3.381933827755728
|
| 1496 |
-
},
|
| 1497 |
-
"connect_four": {
|
| 1498 |
-
"questions": 56,
|
| 1499 |
-
"modal_accuracy": 0.5535714285714286,
|
| 1500 |
-
"brier": 0.2622696208064482,
|
| 1501 |
-
"cross_entropy": 1.8219529101323455
|
| 1502 |
-
},
|
| 1503 |
-
"customer-control-v1": {
|
| 1504 |
-
"questions": 7,
|
| 1505 |
-
"modal_accuracy": 1.0,
|
| 1506 |
-
"brier": 2.417104630113183e-06,
|
| 1507 |
-
"cross_entropy": 0.0005900334430327367
|
| 1508 |
-
},
|
| 1509 |
-
"devign": {
|
| 1510 |
-
"questions": 2,
|
| 1511 |
-
"modal_accuracy": 1.0,
|
| 1512 |
-
"brier": 0.43780762834608883,
|
| 1513 |
-
"cross_entropy": 0.6306076256145883
|
| 1514 |
-
},
|
| 1515 |
-
"gsm8k": {
|
| 1516 |
-
"questions": 4,
|
| 1517 |
-
"modal_accuracy": 1.0,
|
| 1518 |
-
"brier": 0.0003744313065983422,
|
| 1519 |
-
"cross_entropy": 0.011988960485765076
|
| 1520 |
-
},
|
| 1521 |
-
"helpsteer2": {
|
| 1522 |
-
"questions": 128,
|
| 1523 |
-
"modal_accuracy": 0.5859375,
|
| 1524 |
-
"brier": 0.20697473621325052,
|
| 1525 |
-
"cross_entropy": 1.5034032587243362
|
| 1526 |
-
},
|
| 1527 |
-
"planning": {
|
| 1528 |
-
"questions": 192,
|
| 1529 |
-
"modal_accuracy": 0.9635416666666666,
|
| 1530 |
-
"brier": 0.11802009440200961,
|
| 1531 |
-
"cross_entropy": 0.6579051066101222
|
| 1532 |
-
},
|
| 1533 |
-
"race": {
|
| 1534 |
-
"questions": 4,
|
| 1535 |
-
"modal_accuracy": 0.5,
|
| 1536 |
-
"brier": 0.7315329429054658,
|
| 1537 |
-
"cross_entropy": 2.0647249708287245
|
| 1538 |
-
},
|
| 1539 |
-
"ragtruth": {
|
| 1540 |
-
"questions": 6,
|
| 1541 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1542 |
-
"brier": 0.609860991064251,
|
| 1543 |
-
"cross_entropy": 0.9741778978250341
|
| 1544 |
-
},
|
| 1545 |
-
"scienceworld": {
|
| 1546 |
-
"questions": 128,
|
| 1547 |
-
"modal_accuracy": 0.703125,
|
| 1548 |
-
"brier": 0.44470820194957994,
|
| 1549 |
-
"cross_entropy": 1.2102900099661185
|
| 1550 |
-
},
|
| 1551 |
-
"scone": {
|
| 1552 |
-
"questions": 192,
|
| 1553 |
-
"modal_accuracy": 0.796875,
|
| 1554 |
-
"brier": 0.29338767561802165,
|
| 1555 |
-
"cross_entropy": 0.6140028830315581
|
| 1556 |
-
},
|
| 1557 |
-
"toolace": {
|
| 1558 |
-
"questions": 2,
|
| 1559 |
-
"modal_accuracy": 1.0,
|
| 1560 |
-
"brier": 2.1804417353162663e-07,
|
| 1561 |
-
"cross_entropy": 0.0003234946827488933
|
| 1562 |
-
},
|
| 1563 |
-
"yuri_v3": {
|
| 1564 |
-
"questions": 53,
|
| 1565 |
-
"modal_accuracy": 0.7169811320754716,
|
| 1566 |
-
"brier": 0.056977519481840835,
|
| 1567 |
-
"cross_entropy": 0.860752644211149
|
| 1568 |
-
}
|
| 1569 |
-
}
|
| 1570 |
-
},
|
| 1571 |
-
"new_test": {
|
| 1572 |
-
"questions": 512,
|
| 1573 |
-
"modal_accuracy": 0.755859375,
|
| 1574 |
-
"brier": 0.2705710538872297,
|
| 1575 |
-
"cross_entropy": 1.0354930422480668,
|
| 1576 |
-
"by_kind": {
|
| 1577 |
-
"choice": {
|
| 1578 |
-
"questions": 241,
|
| 1579 |
-
"modal_accuracy": 0.8215767634854771,
|
| 1580 |
-
"brier": 0.30593746240810665,
|
| 1581 |
-
"cross_entropy": 1.0288481446263855
|
| 1582 |
-
},
|
| 1583 |
-
"noul": {
|
| 1584 |
-
"questions": 94,
|
| 1585 |
-
"modal_accuracy": 0.8936170212765957,
|
| 1586 |
-
"brier": 0.1853453372075174,
|
| 1587 |
-
"cross_entropy": 0.4718470608079094
|
| 1588 |
-
},
|
| 1589 |
-
"score": {
|
| 1590 |
-
"questions": 177,
|
| 1591 |
-
"modal_accuracy": 0.5932203389830508,
|
| 1592 |
-
"brier": 0.26767790651074164,
|
| 1593 |
-
"cross_entropy": 1.34387802858818
|
| 1594 |
-
}
|
| 1595 |
-
},
|
| 1596 |
-
"by_source": {
|
| 1597 |
-
"helpsteer2": {
|
| 1598 |
-
"questions": 128,
|
| 1599 |
-
"modal_accuracy": 0.5078125,
|
| 1600 |
-
"brier": 0.22898458428820745,
|
| 1601 |
-
"cross_entropy": 1.5346420909767997
|
| 1602 |
-
},
|
| 1603 |
-
"planning": {
|
| 1604 |
-
"questions": 192,
|
| 1605 |
-
"modal_accuracy": 0.921875,
|
| 1606 |
-
"brier": 0.1909474730913934,
|
| 1607 |
-
"cross_entropy": 0.8915435249464682
|
| 1608 |
-
},
|
| 1609 |
-
"scone": {
|
| 1610 |
-
"questions": 192,
|
| 1611 |
-
"modal_accuracy": 0.7552083333333334,
|
| 1612 |
-
"brier": 0.3779189477490808,
|
| 1613 |
-
"cross_entropy": 0.8466765270638437
|
| 1614 |
-
}
|
| 1615 |
-
}
|
| 1616 |
},
|
| 1617 |
-
"
|
| 1618 |
-
"questions":
|
| 1619 |
-
"
|
| 1620 |
-
"brier": 0.
|
| 1621 |
-
"cross_entropy": 0.
|
| 1622 |
-
"by_kind": {
|
| 1623 |
-
"choice": {
|
| 1624 |
-
"questions": 1151,
|
| 1625 |
-
"modal_accuracy": 0.7193744569939183,
|
| 1626 |
-
"brier": 0.36420584491384933,
|
| 1627 |
-
"cross_entropy": 1.0683460554623134
|
| 1628 |
-
},
|
| 1629 |
-
"noul": {
|
| 1630 |
-
"questions": 649,
|
| 1631 |
-
"modal_accuracy": 0.9090909090909091,
|
| 1632 |
-
"brier": 0.12795487547550322,
|
| 1633 |
-
"cross_entropy": 0.3201729560342784
|
| 1634 |
-
},
|
| 1635 |
-
"score": {
|
| 1636 |
-
"questions": 248,
|
| 1637 |
-
"modal_accuracy": 0.8346774193548387,
|
| 1638 |
-
"brier": 0.1562044216890414,
|
| 1639 |
-
"cross_entropy": 0.6832748093410428
|
| 1640 |
-
}
|
| 1641 |
-
},
|
| 1642 |
-
"by_source": {
|
| 1643 |
-
"alfworld": {
|
| 1644 |
-
"questions": 128,
|
| 1645 |
-
"modal_accuracy": 0.8671875,
|
| 1646 |
-
"brier": 0.1662207666853679,
|
| 1647 |
-
"cross_entropy": 0.4678064926902018
|
| 1648 |
-
},
|
| 1649 |
-
"arithmetic": {
|
| 1650 |
-
"questions": 30,
|
| 1651 |
-
"modal_accuracy": 0.7333333333333333,
|
| 1652 |
-
"brier": 0.5001721787123533,
|
| 1653 |
-
"cross_entropy": 1.2885683809129926
|
| 1654 |
-
},
|
| 1655 |
-
"babyai": {
|
| 1656 |
-
"questions": 128,
|
| 1657 |
-
"modal_accuracy": 0.484375,
|
| 1658 |
-
"brier": 0.7067526312424577,
|
| 1659 |
-
"cross_entropy": 1.6676958019283898
|
| 1660 |
-
},
|
| 1661 |
-
"community-diversity-v2/browser_tools/goal_completion": {
|
| 1662 |
-
"questions": 24,
|
| 1663 |
-
"modal_accuracy": 0.9583333333333334,
|
| 1664 |
-
"brier": 0.06921012297338076,
|
| 1665 |
-
"cross_entropy": 0.10122484092336533
|
| 1666 |
-
},
|
| 1667 |
-
"community-diversity-v2/contracts/exclusivity_exception": {
|
| 1668 |
-
"questions": 23,
|
| 1669 |
-
"modal_accuracy": 0.8695652173913043,
|
| 1670 |
-
"brier": 0.24976225264440965,
|
| 1671 |
-
"cross_entropy": 0.5909170782174602
|
| 1672 |
-
},
|
| 1673 |
-
"community-diversity-v2/games/pit_stop": {
|
| 1674 |
-
"questions": 13,
|
| 1675 |
-
"modal_accuracy": 0.7692307692307693,
|
| 1676 |
-
"brier": 0.4785595907160497,
|
| 1677 |
-
"cross_entropy": 1.2971969110502473
|
| 1678 |
-
},
|
| 1679 |
-
"community-diversity-v2/ir/warranty_transfer": {
|
| 1680 |
-
"questions": 15,
|
| 1681 |
-
"modal_accuracy": 0.8666666666666667,
|
| 1682 |
-
"brier": 0.14068735459076726,
|
| 1683 |
-
"cross_entropy": 0.17808536247316048
|
| 1684 |
-
},
|
| 1685 |
-
"community-diversity-v2/rag/independent_sources": {
|
| 1686 |
-
"questions": 21,
|
| 1687 |
-
"modal_accuracy": 0.47619047619047616,
|
| 1688 |
-
"brier": 0.9915299974666058,
|
| 1689 |
-
"cross_entropy": 2.310007596806081
|
| 1690 |
-
},
|
| 1691 |
-
"community-diversity-v2/rubric_judge/balanced_tradeoff": {
|
| 1692 |
-
"questions": 12,
|
| 1693 |
-
"modal_accuracy": 0.75,
|
| 1694 |
-
"brier": 0.4814089394948157,
|
| 1695 |
-
"cross_entropy": 1.102602184903888
|
| 1696 |
-
},
|
| 1697 |
-
"community-diversity-v2/shell_history/incremental_copy": {
|
| 1698 |
-
"questions": 17,
|
| 1699 |
-
"modal_accuracy": 0.8823529411764706,
|
| 1700 |
-
"brier": 0.20441354684881466,
|
| 1701 |
-
"cross_entropy": 0.5009642986702474
|
| 1702 |
-
},
|
| 1703 |
-
"community-diversity-v2/support/outage_compensation": {
|
| 1704 |
-
"questions": 3,
|
| 1705 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1706 |
-
"brier": 0.4884574890213706,
|
| 1707 |
-
"cross_entropy": 1.3886592311282289
|
| 1708 |
-
},
|
| 1709 |
-
"connect_four": {
|
| 1710 |
-
"questions": 128,
|
| 1711 |
-
"modal_accuracy": 0.59375,
|
| 1712 |
-
"brier": 0.2561712647462938,
|
| 1713 |
-
"cross_entropy": 1.802544508268942
|
| 1714 |
-
},
|
| 1715 |
-
"customer-control-v1": {
|
| 1716 |
-
"questions": 45,
|
| 1717 |
-
"modal_accuracy": 1.0,
|
| 1718 |
-
"brier": 0.00019936847019355491,
|
| 1719 |
-
"cross_entropy": 0.015971454914726523
|
| 1720 |
-
},
|
| 1721 |
-
"devign": {
|
| 1722 |
-
"questions": 46,
|
| 1723 |
-
"modal_accuracy": 0.5217391304347826,
|
| 1724 |
-
"brier": 0.5225763998633566,
|
| 1725 |
-
"cross_entropy": 0.722933623300632
|
| 1726 |
-
},
|
| 1727 |
-
"entity_belief_tracking": {
|
| 1728 |
-
"questions": 19,
|
| 1729 |
-
"modal_accuracy": 0.8421052631578947,
|
| 1730 |
-
"brier": 0.259415578060587,
|
| 1731 |
-
"cross_entropy": 0.6433967485445069
|
| 1732 |
-
},
|
| 1733 |
-
"event_state_reconstruction": {
|
| 1734 |
-
"questions": 30,
|
| 1735 |
-
"modal_accuracy": 1.0,
|
| 1736 |
-
"brier": 0.015897166327449284,
|
| 1737 |
-
"cross_entropy": 0.02589146516518676
|
| 1738 |
-
},
|
| 1739 |
-
"event_triggered_execution": {
|
| 1740 |
-
"questions": 18,
|
| 1741 |
-
"modal_accuracy": 0.8888888888888888,
|
| 1742 |
-
"brier": 0.16562186759207273,
|
| 1743 |
-
"cross_entropy": 0.2757516689442136
|
| 1744 |
-
},
|
| 1745 |
-
"evidence_sufficiency": {
|
| 1746 |
-
"questions": 13,
|
| 1747 |
-
"modal_accuracy": 0.6923076923076923,
|
| 1748 |
-
"brier": 0.5880021120988826,
|
| 1749 |
-
"cross_entropy": 1.4889127617030355
|
| 1750 |
-
},
|
| 1751 |
-
"goal_directed_planning": {
|
| 1752 |
-
"questions": 19,
|
| 1753 |
-
"modal_accuracy": 0.8421052631578947,
|
| 1754 |
-
"brier": 0.2509895082794504,
|
| 1755 |
-
"cross_entropy": 0.4540842796343451
|
| 1756 |
-
},
|
| 1757 |
-
"graph_navigation": {
|
| 1758 |
-
"questions": 6,
|
| 1759 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1760 |
-
"brier": 0.44283093090962034,
|
| 1761 |
-
"cross_entropy": 1.3194420415465578
|
| 1762 |
-
},
|
| 1763 |
-
"gsm8k": {
|
| 1764 |
-
"questions": 106,
|
| 1765 |
-
"modal_accuracy": 0.7075471698113207,
|
| 1766 |
-
"brier": 0.43472841661725453,
|
| 1767 |
-
"cross_entropy": 0.8890682888838121
|
| 1768 |
-
},
|
| 1769 |
-
"instruction_execution": {
|
| 1770 |
-
"questions": 12,
|
| 1771 |
-
"modal_accuracy": 0.9166666666666666,
|
| 1772 |
-
"brier": 0.16555489236402793,
|
| 1773 |
-
"cross_entropy": 0.49206138807591987
|
| 1774 |
-
},
|
| 1775 |
-
"multi_view_adjudication": {
|
| 1776 |
-
"questions": 20,
|
| 1777 |
-
"modal_accuracy": 1.0,
|
| 1778 |
-
"brier": 5.2326944475445606e-05,
|
| 1779 |
-
"cross_entropy": 0.0019525076494628021
|
| 1780 |
-
},
|
| 1781 |
-
"needle_retrieval": {
|
| 1782 |
-
"questions": 5,
|
| 1783 |
-
"modal_accuracy": 1.0,
|
| 1784 |
-
"brier": 3.199473123353177e-08,
|
| 1785 |
-
"cross_entropy": 8.974482553742138e-05
|
| 1786 |
-
},
|
| 1787 |
-
"partial_observation_calibration": {
|
| 1788 |
-
"questions": 5,
|
| 1789 |
-
"modal_accuracy": 0.8,
|
| 1790 |
-
"brier": 0.02661607292190903,
|
| 1791 |
-
"cross_entropy": 0.5399351074049796
|
| 1792 |
-
},
|
| 1793 |
-
"policy_applicability": {
|
| 1794 |
-
"questions": 17,
|
| 1795 |
-
"modal_accuracy": 1.0,
|
| 1796 |
-
"brier": 6.106492112719508e-05,
|
| 1797 |
-
"cross_entropy": 0.003675418740395231
|
| 1798 |
-
},
|
| 1799 |
-
"policy_under_uncertainty": {
|
| 1800 |
-
"questions": 8,
|
| 1801 |
-
"modal_accuracy": 0.875,
|
| 1802 |
-
"brier": 0.10177113114450002,
|
| 1803 |
-
"cross_entropy": 0.560772453887988
|
| 1804 |
-
},
|
| 1805 |
-
"race": {
|
| 1806 |
-
"questions": 104,
|
| 1807 |
-
"modal_accuracy": 0.9134615384615384,
|
| 1808 |
-
"brier": 0.16990537457587326,
|
| 1809 |
-
"cross_entropy": 0.3822661135973287
|
| 1810 |
-
},
|
| 1811 |
-
"ragtruth": {
|
| 1812 |
-
"questions": 128,
|
| 1813 |
-
"modal_accuracy": 0.8515625,
|
| 1814 |
-
"brier": 0.2317892915096781,
|
| 1815 |
-
"cross_entropy": 0.39549112820971377
|
| 1816 |
-
},
|
| 1817 |
-
"reasoning-control-v1": {
|
| 1818 |
-
"questions": 45,
|
| 1819 |
-
"modal_accuracy": 0.9111111111111111,
|
| 1820 |
-
"brier": 0.1285355661042637,
|
| 1821 |
-
"cross_entropy": 0.30974376864438463
|
| 1822 |
-
},
|
| 1823 |
-
"record_aggregation": {
|
| 1824 |
-
"questions": 14,
|
| 1825 |
-
"modal_accuracy": 0.7857142857142857,
|
| 1826 |
-
"brier": 0.35331876706309845,
|
| 1827 |
-
"cross_entropy": 0.696535349171323
|
| 1828 |
-
},
|
| 1829 |
-
"record_eligibility": {
|
| 1830 |
-
"questions": 8,
|
| 1831 |
-
"modal_accuracy": 1.0,
|
| 1832 |
-
"brier": 0.00021253785775029812,
|
| 1833 |
-
"cross_entropy": 0.0043067743592292455
|
| 1834 |
-
},
|
| 1835 |
-
"scienceworld": {
|
| 1836 |
-
"questions": 128,
|
| 1837 |
-
"modal_accuracy": 0.5625,
|
| 1838 |
-
"brier": 0.6406304488569265,
|
| 1839 |
-
"cross_entropy": 1.8806145121011755
|
| 1840 |
-
},
|
| 1841 |
-
"snake-v1": {
|
| 1842 |
-
"questions": 30,
|
| 1843 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1844 |
-
"brier": 0.04564391684280246,
|
| 1845 |
-
"cross_entropy": 0.2431433189091872
|
| 1846 |
-
},
|
| 1847 |
-
"state_perturbation": {
|
| 1848 |
-
"questions": 11,
|
| 1849 |
-
"modal_accuracy": 1.0,
|
| 1850 |
-
"brier": 0.0002344655717187281,
|
| 1851 |
-
"cross_entropy": 0.005994309073216685
|
| 1852 |
-
},
|
| 1853 |
-
"table_lookup": {
|
| 1854 |
-
"questions": 13,
|
| 1855 |
-
"modal_accuracy": 0.9230769230769231,
|
| 1856 |
-
"brier": 0.1498702753732909,
|
| 1857 |
-
"cross_entropy": 0.3551783229099786
|
| 1858 |
-
},
|
| 1859 |
-
"taxonomy_routing": {
|
| 1860 |
-
"questions": 8,
|
| 1861 |
-
"modal_accuracy": 1.0,
|
| 1862 |
-
"brier": 0.045424091680658744,
|
| 1863 |
-
"cross_entropy": 0.07014486556652395
|
| 1864 |
-
},
|
| 1865 |
-
"tic_tac_toe-v1": {
|
| 1866 |
-
"questions": 15,
|
| 1867 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1868 |
-
"brier": 0.26386162320957124,
|
| 1869 |
-
"cross_entropy": 1.1837003824426573
|
| 1870 |
-
},
|
| 1871 |
-
"tile_platformer-v1": {
|
| 1872 |
-
"questions": 15,
|
| 1873 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1874 |
-
"brier": 0.10164899419557688,
|
| 1875 |
-
"cross_entropy": 0.220570822391115
|
| 1876 |
-
},
|
| 1877 |
-
"toolace": {
|
| 1878 |
-
"questions": 128,
|
| 1879 |
-
"modal_accuracy": 0.9921875,
|
| 1880 |
-
"brier": 0.01562046345535589,
|
| 1881 |
-
"cross_entropy": 0.04367168294967396
|
| 1882 |
-
},
|
| 1883 |
-
"trex_runner-v1": {
|
| 1884 |
-
"questions": 2,
|
| 1885 |
-
"modal_accuracy": 0.5,
|
| 1886 |
-
"brier": 0.36032773502188475,
|
| 1887 |
-
"cross_entropy": 0.5817032440936637
|
| 1888 |
-
},
|
| 1889 |
-
"vizdoom-basic-v1": {
|
| 1890 |
-
"questions": 45,
|
| 1891 |
-
"modal_accuracy": 0.9555555555555556,
|
| 1892 |
-
"brier": 0.08693939517676995,
|
| 1893 |
-
"cross_entropy": 0.20638551041670192
|
| 1894 |
-
},
|
| 1895 |
-
"wanli-decisions-v1": {
|
| 1896 |
-
"questions": 128,
|
| 1897 |
-
"modal_accuracy": 0.6875,
|
| 1898 |
-
"brier": 0.5529247926341857,
|
| 1899 |
-
"cross_entropy": 1.0802996209215674
|
| 1900 |
-
},
|
| 1901 |
-
"workflow-controls-v1/agent_trace_observability": {
|
| 1902 |
-
"questions": 15,
|
| 1903 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1904 |
-
"brier": 0.13168080686035147,
|
| 1905 |
-
"cross_entropy": 0.3395590286213042
|
| 1906 |
-
},
|
| 1907 |
-
"workflow-controls-v1/customer_service": {
|
| 1908 |
-
"questions": 15,
|
| 1909 |
-
"modal_accuracy": 1.0,
|
| 1910 |
-
"brier": 7.55085699524451e-06,
|
| 1911 |
-
"cross_entropy": 0.0007668466946998275
|
| 1912 |
-
},
|
| 1913 |
-
"workflow-controls-v1/invoice_processing": {
|
| 1914 |
-
"questions": 15,
|
| 1915 |
-
"modal_accuracy": 1.0,
|
| 1916 |
-
"brier": 1.1169422484519878e-06,
|
| 1917 |
-
"cross_entropy": 0.0002747598457579768
|
| 1918 |
-
},
|
| 1919 |
-
"workflow-controls-v1/security_incidents": {
|
| 1920 |
-
"questions": 14,
|
| 1921 |
-
"modal_accuracy": 1.0,
|
| 1922 |
-
"brier": 2.4204779837314167e-09,
|
| 1923 |
-
"cross_entropy": 2.2446460647594408e-05
|
| 1924 |
-
},
|
| 1925 |
-
"yuri_v3": {
|
| 1926 |
-
"questions": 256,
|
| 1927 |
-
"modal_accuracy": 0.83203125,
|
| 1928 |
-
"brier": 0.03526590762889966,
|
| 1929 |
-
"cross_entropy": 0.8245288048118414
|
| 1930 |
-
}
|
| 1931 |
-
}
|
| 1932 |
},
|
| 1933 |
-
"
|
| 1934 |
-
"questions":
|
| 1935 |
-
"
|
| 1936 |
-
"brier": 0.
|
| 1937 |
-
"cross_entropy": 1.
|
| 1938 |
-
"by_kind": {
|
| 1939 |
-
"choice": {
|
| 1940 |
-
"questions": 33,
|
| 1941 |
-
"modal_accuracy": 0.6363636363636364,
|
| 1942 |
-
"brier": 0.5172571933930417,
|
| 1943 |
-
"cross_entropy": 1.2048737574594803
|
| 1944 |
-
},
|
| 1945 |
-
"noul": {
|
| 1946 |
-
"questions": 41,
|
| 1947 |
-
"modal_accuracy": 0.7317073170731707,
|
| 1948 |
-
"brier": 0.3731759473027374,
|
| 1949 |
-
"cross_entropy": 0.7052377430670229
|
| 1950 |
-
},
|
| 1951 |
-
"score": {
|
| 1952 |
-
"questions": 15,
|
| 1953 |
-
"modal_accuracy": 0.4666666666666667,
|
| 1954 |
-
"brier": 0.6849531954881848,
|
| 1955 |
-
"cross_entropy": 1.381039887344188
|
| 1956 |
-
}
|
| 1957 |
-
},
|
| 1958 |
-
"by_source": {
|
| 1959 |
-
"basic_cls": {
|
| 1960 |
-
"questions": 12,
|
| 1961 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1962 |
-
"brier": 0.4038371458185728,
|
| 1963 |
-
"cross_entropy": 0.7355992645221435
|
| 1964 |
-
},
|
| 1965 |
-
"code": {
|
| 1966 |
-
"questions": 6,
|
| 1967 |
-
"modal_accuracy": 0.3333333333333333,
|
| 1968 |
-
"brier": 0.8287228425681598,
|
| 1969 |
-
"cross_entropy": 1.262918501594028
|
| 1970 |
-
},
|
| 1971 |
-
"dialog": {
|
| 1972 |
-
"questions": 12,
|
| 1973 |
-
"modal_accuracy": 0.5833333333333334,
|
| 1974 |
-
"brier": 0.30448197088852874,
|
| 1975 |
-
"cross_entropy": 1.1186203509870534
|
| 1976 |
-
},
|
| 1977 |
-
"judge": {
|
| 1978 |
-
"questions": 8,
|
| 1979 |
-
"modal_accuracy": 0.875,
|
| 1980 |
-
"brier": 0.3292730875017608,
|
| 1981 |
-
"cross_entropy": 0.7769559856794094
|
| 1982 |
-
},
|
| 1983 |
-
"know": {
|
| 1984 |
-
"questions": 10,
|
| 1985 |
-
"modal_accuracy": 0.8,
|
| 1986 |
-
"brier": 0.26539280354851125,
|
| 1987 |
-
"cross_entropy": 0.4839723017070307
|
| 1988 |
-
},
|
| 1989 |
-
"ret": {
|
| 1990 |
-
"questions": 10,
|
| 1991 |
-
"modal_accuracy": 0.5,
|
| 1992 |
-
"brier": 0.814367705138209,
|
| 1993 |
-
"cross_entropy": 1.6245878852220568
|
| 1994 |
-
},
|
| 1995 |
-
"route": {
|
| 1996 |
-
"questions": 9,
|
| 1997 |
-
"modal_accuracy": 0.4444444444444444,
|
| 1998 |
-
"brier": 0.7471958010258476,
|
| 1999 |
-
"cross_entropy": 1.3369414656592875
|
| 2000 |
-
},
|
| 2001 |
-
"safety": {
|
| 2002 |
-
"questions": 12,
|
| 2003 |
-
"modal_accuracy": 0.9166666666666666,
|
| 2004 |
-
"brier": 0.1672822055997213,
|
| 2005 |
-
"cross_entropy": 0.43712494222128856
|
| 2006 |
-
},
|
| 2007 |
-
"step": {
|
| 2008 |
-
"questions": 8,
|
| 2009 |
-
"modal_accuracy": 0.625,
|
| 2010 |
-
"brier": 0.6510657904776266,
|
| 2011 |
-
"cross_entropy": 1.5015904290656892
|
| 2012 |
-
},
|
| 2013 |
-
"struct": {
|
| 2014 |
-
"questions": 2,
|
| 2015 |
-
"modal_accuracy": 0.5,
|
| 2016 |
-
"brier": 0.8996839553398099,
|
| 2017 |
-
"cross_entropy": 1.4855438457791856
|
| 2018 |
-
}
|
| 2019 |
-
}
|
| 2020 |
}
|
| 2021 |
}
|
| 2022 |
},
|
| 2023 |
-
"scope":
|
| 2024 |
-
"new_validation": "1024 selection questions: 512 new and 512 previous; all four candidate checkpoints scored",
|
| 2025 |
-
"new_test": "512 reserved new questions; first scored after checkpoint selection",
|
| 2026 |
-
"test": "2048 earlier questions; reused to check retention of earlier abilities",
|
| 2027 |
-
"transfer": "89 general questions; small reused development check",
|
| 2028 |
-
"parent": "Previous actions-48k adapter, re-exported to byte-identical NVFP4 and rescored with the same Eider binary",
|
| 2029 |
-
"calibration": "All scores are uncalibrated. No fitted profile for the new weights.",
|
| 2030 |
-
"bf16": "Matching merged BF16 weights are included but were not independently scored in this native evaluation.",
|
| 2031 |
-
"base_model": "The unfine-tuned NVIDIA model was not rescored in this evaluation."
|
| 2032 |
-
}
|
| 2033 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"test": {
|
| 3 |
+
"schema": "dory-base-vs-typed-decisions/v1",
|
| 4 |
+
"base_model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16",
|
| 5 |
+
"base_model_revision": "b366b87f762ea334ba3eb55912031baba1515810",
|
| 6 |
+
"format": "NVFP4",
|
| 7 |
+
"quantization_recipe": "NVIDIA Model Optimizer local_hessian",
|
| 8 |
+
"calibration_examples": 256,
|
| 9 |
+
"loops": 2,
|
| 10 |
+
"calibrated": false,
|
| 11 |
+
"prompt": "Identical typed-decision prompts and allowed-answer scoring; no generated reasoning.",
|
| 12 |
+
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 13 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 14 |
+
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 15 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 16 |
+
"base": {
|
| 17 |
+
"questions": 512,
|
| 18 |
+
"modal_accuracy": 0.376953125,
|
| 19 |
+
"brier": 0.6512325676171807,
|
| 20 |
+
"cross_entropy": 1.9877935454851494,
|
| 21 |
+
"mean_request_ms": 81.26816220556066,
|
| 22 |
+
"median_request_ms": 72.5804881658405,
|
| 23 |
+
"by_kind": {
|
| 24 |
+
"choice": {
|
| 25 |
+
"questions": 241,
|
| 26 |
+
"modal_accuracy": 0.2863070539419087,
|
| 27 |
+
"brier": 0.742723819745487,
|
| 28 |
+
"cross_entropy": 2.2877653477997058,
|
| 29 |
+
"mean_request_ms": 75.63938830179235,
|
| 30 |
+
"median_request_ms": 72.81611207872629
|
| 31 |
+
},
|
| 32 |
+
"noul": {
|
| 33 |
+
"questions": 94,
|
| 34 |
+
"modal_accuracy": 0.5,
|
| 35 |
+
"brier": 0.670176294758546,
|
| 36 |
+
"cross_entropy": 0.9175096541022592,
|
| 37 |
+
"mean_request_ms": 68.15949506423574,
|
| 38 |
+
"median_request_ms": 68.27253755182028
|
| 39 |
+
},
|
| 40 |
+
"score": {
|
| 41 |
+
"questions": 177,
|
| 42 |
+
"modal_accuracy": 0.4350282485875706,
|
| 43 |
+
"brier": 0.5165992223391576,
|
| 44 |
+
"cross_entropy": 2.1477567174183902,
|
| 45 |
+
"mean_request_ms": 95.89386402529344,
|
| 46 |
+
"median_request_ms": 77.44529517367482
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
}
|
| 48 |
},
|
| 49 |
+
"by_source": {
|
| 50 |
+
"helpsteer2": {
|
| 51 |
+
"questions": 128,
|
| 52 |
+
"modal_accuracy": 0.484375,
|
| 53 |
+
"brier": 0.31624743248922543,
|
| 54 |
+
"cross_entropy": 1.797596837336767,
|
| 55 |
+
"mean_request_ms": 107.6381878774555,
|
| 56 |
+
"median_request_ms": 96.75377863459289
|
| 57 |
+
},
|
| 58 |
+
"planning": {
|
| 59 |
+
"questions": 192,
|
| 60 |
+
"modal_accuracy": 0.3697916666666667,
|
| 61 |
+
"brier": 0.6013039965397795,
|
| 62 |
+
"cross_entropy": 1.8274480710869951,
|
| 63 |
+
"mean_request_ms": 75.0043233759546,
|
| 64 |
+
"median_request_ms": 73.60281050205231
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
},
|
| 66 |
+
"scone": {
|
| 67 |
+
"questions": 192,
|
| 68 |
+
"modal_accuracy": 0.3125,
|
| 69 |
+
"brier": 0.9244845621132188,
|
| 70 |
+
"cross_entropy": 2.274936825315559,
|
| 71 |
+
"mean_request_ms": 69.95198392057016,
|
| 72 |
+
"median_request_ms": 69.4849812425673
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
}
|
| 74 |
}
|
| 75 |
},
|
| 76 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 77 |
+
"fine_tuned": {
|
| 78 |
+
"questions": 512,
|
| 79 |
+
"modal_accuracy": 0.771484375,
|
| 80 |
+
"brier": 0.26463106057883456,
|
| 81 |
+
"cross_entropy": 1.075109941121327,
|
| 82 |
+
"mean_request_ms": 80.6613628164996,
|
| 83 |
+
"median_request_ms": 71.86161540448666,
|
| 84 |
+
"by_kind": {
|
| 85 |
+
"choice": {
|
| 86 |
+
"questions": 241,
|
| 87 |
+
"modal_accuracy": 0.8215767634854771,
|
| 88 |
+
"brier": 0.34044989428658823,
|
| 89 |
+
"cross_entropy": 1.1801933058267577,
|
| 90 |
+
"mean_request_ms": 75.06443578865827,
|
| 91 |
+
"median_request_ms": 72.3452023230493
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
},
|
| 93 |
+
"noul": {
|
| 94 |
+
"questions": 94,
|
| 95 |
+
"modal_accuracy": 0.9574468085106383,
|
| 96 |
+
"brier": 0.09152335422286362,
|
| 97 |
+
"cross_entropy": 0.2881069286288201,
|
| 98 |
+
"mean_request_ms": 67.15274397581022,
|
| 99 |
+
"median_request_ms": 67.46977800503373
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
},
|
| 101 |
+
"score": {
|
| 102 |
+
"questions": 177,
|
| 103 |
+
"modal_accuracy": 0.6045197740112994,
|
| 104 |
+
"brier": 0.2533304135386799,
|
| 105 |
+
"cross_entropy": 1.3499867336653204,
|
| 106 |
+
"mean_request_ms": 95.45610623307905,
|
| 107 |
+
"median_request_ms": 77.96838786453009
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
}
|
| 109 |
},
|
| 110 |
+
"by_source": {
|
| 111 |
+
"helpsteer2": {
|
| 112 |
+
"questions": 128,
|
| 113 |
+
"modal_accuracy": 0.5078125,
|
| 114 |
+
"brier": 0.22775570941129228,
|
| 115 |
+
"cross_entropy": 1.5323848707632575,
|
| 116 |
+
"mean_request_ms": 107.58943439577706,
|
| 117 |
+
"median_request_ms": 97.03774400986731
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
},
|
| 119 |
+
"planning": {
|
| 120 |
+
"questions": 192,
|
| 121 |
+
"modal_accuracy": 0.9635416666666666,
|
| 122 |
+
"brier": 0.155754142490034,
|
| 123 |
+
"cross_entropy": 0.834348251030471,
|
| 124 |
+
"mean_request_ms": 74.49704928634067,
|
| 125 |
+
"median_request_ms": 73.17394809797406
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
},
|
| 127 |
+
"scone": {
|
| 128 |
+
"questions": 192,
|
| 129 |
+
"modal_accuracy": 0.7552083333333334,
|
| 130 |
+
"brier": 0.3980915461126633,
|
| 131 |
+
"cross_entropy": 1.0110216781175627,
|
| 132 |
+
"mean_request_ms": 68.87362862714023,
|
| 133 |
+
"median_request_ms": 68.2264359202236
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
}
|
| 135 |
}
|
| 136 |
+
}
|
| 137 |
+
},
|
| 138 |
+
"validation": {
|
| 139 |
+
"questions": 1024,
|
| 140 |
+
"split": "validation",
|
| 141 |
+
"selection_use": "Quantization recipe comparison; not an independent test",
|
| 142 |
+
"loops": 2,
|
| 143 |
+
"probability_calibration": false,
|
| 144 |
+
"calibration_examples": 256,
|
| 145 |
+
"calibration_source": "Training examples disjoint from validation by ID and source group",
|
| 146 |
+
"weights_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 147 |
+
"metrics": {
|
| 148 |
+
"questions": 1024,
|
| 149 |
+
"correct": 0.78515625,
|
| 150 |
+
"brier": 0.256938595885194,
|
| 151 |
+
"cross_entropy": 0.9667092188545374
|
| 152 |
},
|
| 153 |
+
"by_kind": {
|
| 154 |
+
"choice": {
|
| 155 |
+
"questions": 665,
|
| 156 |
+
"correct": 0.7819548872180451,
|
| 157 |
+
"brier": 0.29931089665593197,
|
| 158 |
+
"cross_entropy": 1.021091694949712
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
},
|
| 160 |
+
"noul": {
|
| 161 |
+
"questions": 144,
|
| 162 |
+
"correct": 0.9097222222222222,
|
| 163 |
+
"brier": 0.146691427234091,
|
| 164 |
+
"cross_entropy": 0.4264544071293922
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 165 |
},
|
| 166 |
+
"score": {
|
| 167 |
+
"questions": 215,
|
| 168 |
+
"correct": 0.7116279069767442,
|
| 169 |
+
"brier": 0.19972004831876652,
|
| 170 |
+
"cross_entropy": 1.160348038785374
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
}
|
| 172 |
}
|
| 173 |
},
|
| 174 |
+
"scope": "Test questions are disjoint from calibration training examples. Validation was used for quantization selection."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 175 |
}
|
evaluation/openrouter-comparison.json
CHANGED
|
@@ -256,7 +256,7 @@
|
|
| 256 |
"errors": {},
|
| 257 |
"total_cost_usd": 0.02509872800000002,
|
| 258 |
"local_results": {
|
| 259 |
-
"Dory
|
| 260 |
"questions": 512,
|
| 261 |
"modal_accuracy": 0.77734375,
|
| 262 |
"brier": 0.2599162990600157,
|
|
@@ -340,51 +340,71 @@
|
|
| 340 |
"--no-warmup"
|
| 341 |
]
|
| 342 |
},
|
| 343 |
-
"Dory
|
| 344 |
"questions": 512,
|
| 345 |
-
"modal_accuracy": 0.
|
| 346 |
-
"brier": 0.
|
| 347 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 348 |
"by_kind": {
|
| 349 |
"choice": {
|
| 350 |
"questions": 241,
|
| 351 |
"modal_accuracy": 0.8215767634854771,
|
| 352 |
-
"brier": 0.
|
| 353 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 354 |
},
|
| 355 |
"noul": {
|
| 356 |
"questions": 94,
|
| 357 |
-
"modal_accuracy": 0.
|
| 358 |
-
"brier": 0.
|
| 359 |
-
"cross_entropy": 0.
|
|
|
|
|
|
|
| 360 |
},
|
| 361 |
"score": {
|
| 362 |
"questions": 177,
|
| 363 |
-
"modal_accuracy": 0.
|
| 364 |
-
"brier": 0.
|
| 365 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 366 |
}
|
| 367 |
},
|
| 368 |
"by_source": {
|
| 369 |
"helpsteer2": {
|
| 370 |
"questions": 128,
|
| 371 |
"modal_accuracy": 0.5078125,
|
| 372 |
-
"brier": 0.
|
| 373 |
-
"cross_entropy": 1.
|
|
|
|
|
|
|
| 374 |
},
|
| 375 |
"planning": {
|
| 376 |
"questions": 192,
|
| 377 |
-
"modal_accuracy": 0.
|
| 378 |
-
"brier": 0.
|
| 379 |
-
"cross_entropy": 0.
|
|
|
|
|
|
|
| 380 |
},
|
| 381 |
"scone": {
|
| 382 |
"questions": 192,
|
| 383 |
"modal_accuracy": 0.7552083333333334,
|
| 384 |
-
"brier": 0.
|
| 385 |
-
"cross_entropy":
|
|
|
|
|
|
|
| 386 |
}
|
| 387 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 388 |
}
|
| 389 |
},
|
| 390 |
"scope": "512 test questions withheld from this fine-tune. Task families are represented in its training. Provider training overlap is unknown. This is not an official source benchmark or a general model ranking.",
|
|
|
|
| 256 |
"errors": {},
|
| 257 |
"total_cost_usd": 0.02509872800000002,
|
| 258 |
"local_results": {
|
| 259 |
+
"Decisive Dory \u2014 Q8_0 / llama.cpp": {
|
| 260 |
"questions": 512,
|
| 261 |
"modal_accuracy": 0.77734375,
|
| 262 |
"brier": 0.2599162990600157,
|
|
|
|
| 340 |
"--no-warmup"
|
| 341 |
]
|
| 342 |
},
|
| 343 |
+
"Decisive Dory \u2014 NVFP4 / Eider": {
|
| 344 |
"questions": 512,
|
| 345 |
+
"modal_accuracy": 0.771484375,
|
| 346 |
+
"brier": 0.26463106057883456,
|
| 347 |
+
"cross_entropy": 1.075109941121327,
|
| 348 |
+
"mean_request_ms": 80.6613628164996,
|
| 349 |
+
"median_request_ms": 71.86161540448666,
|
| 350 |
"by_kind": {
|
| 351 |
"choice": {
|
| 352 |
"questions": 241,
|
| 353 |
"modal_accuracy": 0.8215767634854771,
|
| 354 |
+
"brier": 0.34044989428658823,
|
| 355 |
+
"cross_entropy": 1.1801933058267577,
|
| 356 |
+
"mean_request_ms": 75.06443578865827,
|
| 357 |
+
"median_request_ms": 72.3452023230493
|
| 358 |
},
|
| 359 |
"noul": {
|
| 360 |
"questions": 94,
|
| 361 |
+
"modal_accuracy": 0.9574468085106383,
|
| 362 |
+
"brier": 0.09152335422286362,
|
| 363 |
+
"cross_entropy": 0.2881069286288201,
|
| 364 |
+
"mean_request_ms": 67.15274397581022,
|
| 365 |
+
"median_request_ms": 67.46977800503373
|
| 366 |
},
|
| 367 |
"score": {
|
| 368 |
"questions": 177,
|
| 369 |
+
"modal_accuracy": 0.6045197740112994,
|
| 370 |
+
"brier": 0.2533304135386799,
|
| 371 |
+
"cross_entropy": 1.3499867336653204,
|
| 372 |
+
"mean_request_ms": 95.45610623307905,
|
| 373 |
+
"median_request_ms": 77.96838786453009
|
| 374 |
}
|
| 375 |
},
|
| 376 |
"by_source": {
|
| 377 |
"helpsteer2": {
|
| 378 |
"questions": 128,
|
| 379 |
"modal_accuracy": 0.5078125,
|
| 380 |
+
"brier": 0.22775570941129228,
|
| 381 |
+
"cross_entropy": 1.5323848707632575,
|
| 382 |
+
"mean_request_ms": 107.58943439577706,
|
| 383 |
+
"median_request_ms": 97.03774400986731
|
| 384 |
},
|
| 385 |
"planning": {
|
| 386 |
"questions": 192,
|
| 387 |
+
"modal_accuracy": 0.9635416666666666,
|
| 388 |
+
"brier": 0.155754142490034,
|
| 389 |
+
"cross_entropy": 0.834348251030471,
|
| 390 |
+
"mean_request_ms": 74.49704928634067,
|
| 391 |
+
"median_request_ms": 73.17394809797406
|
| 392 |
},
|
| 393 |
"scone": {
|
| 394 |
"questions": 192,
|
| 395 |
"modal_accuracy": 0.7552083333333334,
|
| 396 |
+
"brier": 0.3980915461126633,
|
| 397 |
+
"cross_entropy": 1.0110216781175627,
|
| 398 |
+
"mean_request_ms": 68.87362862714023,
|
| 399 |
+
"median_request_ms": 68.2264359202236
|
| 400 |
}
|
| 401 |
+
},
|
| 402 |
+
"model_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 403 |
+
"loops": 2,
|
| 404 |
+
"calibrated": false,
|
| 405 |
+
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 406 |
+
"quantization_recipe": "NVIDIA Model Optimizer local_hessian",
|
| 407 |
+
"calibration_examples": 256
|
| 408 |
}
|
| 409 |
},
|
| 410 |
"scope": "512 test questions withheld from this fine-tune. Task families are represented in its training. Provider training overlap is unknown. This is not an official source benchmark or a general model ranking.",
|
evaluation/runtime.json
CHANGED
|
@@ -1,36 +1,14 @@
|
|
| 1 |
{
|
| 2 |
"server_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 3 |
-
"
|
| 4 |
-
"
|
| 5 |
-
"
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
"dory-decision-targeted.py": "f78f2633a35fa96967a956ee8060cafb5861e4d65e449c3edb5e7dcbc84ded82",
|
| 15 |
-
"dory-decision-continuation.py": "b044299772c69184e256c593aa97cb33fc2d79a8f0bb1bab4c6230aedd46ba40",
|
| 16 |
-
"decision-eval.py": "15cf0b44d2d21ae2484e97dc298b4b3e25e541bedf653436b0c5f7431a982956",
|
| 17 |
-
"run-dory-decision-overnight.sh": "f12d624177c78a28a255b4b910e24b92d4883662076ddaf2afe3561288dd76f7",
|
| 18 |
-
"dory-decision-overnight.py": "0ffa9a9353edf2cb77ce8b9254182a1753a88a036517b1b9f199f8d4981a9c1a",
|
| 19 |
-
"dory_pytorch.py": "51582e13b8a8742f0597bf1cb6c3aaf2b1fa03b3d3f8a2e02a1577b0b58f6d0d",
|
| 20 |
-
"dory-decision-actions.py": "37e23a2d72b5738b5a977bd1625a385cdee9b014173ca6165d47d7b250c1b735"
|
| 21 |
-
},
|
| 22 |
-
"validation_split": "new_validation",
|
| 23 |
-
"test_splits": [
|
| 24 |
-
"new_test",
|
| 25 |
-
"test",
|
| 26 |
-
"transfer"
|
| 27 |
-
],
|
| 28 |
-
"calibration": "uncalibrated",
|
| 29 |
-
"candidate_steps": [
|
| 30 |
-
0,
|
| 31 |
-
2000,
|
| 32 |
-
4000,
|
| 33 |
-
8000
|
| 34 |
-
],
|
| 35 |
-
"parent_comparison_step": 0
|
| 36 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"server_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 3 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 4 |
+
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 5 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 6 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 7 |
+
"loops": 2,
|
| 8 |
+
"probability_calibration": false,
|
| 9 |
+
"max_context_tokens": 8192,
|
| 10 |
+
"prefill_token_capacity": 2048,
|
| 11 |
+
"decision_branch_capacity": 8,
|
| 12 |
+
"retained_prefix_gib": 0,
|
| 13 |
+
"max_active_sequences": 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
}
|
nvfp4/adapter/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Dory
|
| 2 |
|
| 3 |
This rank-8 LoRA adapter uses the tensor format supported by the included `dory_lora.py` loader. It is not in PEFT format.
|
| 4 |
The BF16, NVFP4, and GGUF model files already include the adaptation. Use this standalone adapter only with the original NVIDIA BF16 weights.
|
|
|
|
| 1 |
+
# Decisive Dory — LoRA adapter
|
| 2 |
|
| 3 |
This rank-8 LoRA adapter uses the tensor format supported by the included `dory_lora.py` loader. It is not in PEFT format.
|
| 4 |
The BF16, NVFP4, and GGUF model files already include the adaptation. Use this standalone adapter only with the original NVIDIA BF16 weights.
|
nvfp4/evaluation/README.md
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
# NVFP4 evaluation
|
| 2 |
|
| 3 |
-
[
|
|
|
|
| 4 |
[Export metadata](../export.json) records the weight hashes and quantization settings.
|
|
|
|
| 1 |
# NVFP4 evaluation
|
| 2 |
|
| 3 |
+
[Test results](../../evaluation/base-comparison.json) compare base Dory and these fine-tuned weights on 512 questions.
|
| 4 |
+
[Validation results](../../evaluation/modelopt-validation.json) cover a separate 1,024-question set.
|
| 5 |
[Export metadata](../export.json) records the weight hashes and quantization settings.
|
nvfp4/evaluation/export.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
nvfp4/evaluation/native-serving.json
CHANGED
|
@@ -1,38 +1,14 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
"127.0.0.1:18089",
|
| 15 |
-
"--max-active-sequences",
|
| 16 |
-
"1",
|
| 17 |
-
"--decode-capacity",
|
| 18 |
-
"1",
|
| 19 |
-
"--prefill-sequence-capacity",
|
| 20 |
-
"1",
|
| 21 |
-
"--prefill-token-capacity",
|
| 22 |
-
"2048",
|
| 23 |
-
"--decision-branch-capacity",
|
| 24 |
-
"8",
|
| 25 |
-
"--max-context-tokens",
|
| 26 |
-
"4096",
|
| 27 |
-
"--retained-prefix-gib",
|
| 28 |
-
"0"
|
| 29 |
-
],
|
| 30 |
-
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 31 |
-
"script_sha256": "cbd6c833654b1fff319abf1d8f3cb491af5d96b78d541e593a283b684268da04",
|
| 32 |
-
"evaluation_data_dir": "data",
|
| 33 |
-
"evaluation_data_sha256": {
|
| 34 |
-
"new_test": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 35 |
-
"test": "ace4bd84b914cee4c9ad8658d934386b045b7c643d490283d4f178f037acdba5",
|
| 36 |
-
"transfer": "3c5a77ed25fc7d0c2d1419151aa816bab9d12975b1e59de681934c114834d8e7"
|
| 37 |
-
}
|
| 38 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"server_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 3 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 4 |
+
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 5 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 6 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 7 |
+
"loops": 2,
|
| 8 |
+
"probability_calibration": false,
|
| 9 |
+
"max_context_tokens": 8192,
|
| 10 |
+
"prefill_token_capacity": 2048,
|
| 11 |
+
"decision_branch_capacity": 8,
|
| 12 |
+
"retained_prefix_gib": 0,
|
| 13 |
+
"max_active_sequences": 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
}
|
nvfp4/evaluation/nvfp4-comparison.json
CHANGED
|
@@ -1,2033 +1,175 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
"
|
| 5 |
-
"
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
"
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
"cross_entropy": 1.050083253299498
|
| 48 |
-
},
|
| 49 |
-
"babyai": {
|
| 50 |
-
"questions": 58,
|
| 51 |
-
"modal_accuracy": 0.39655172413793105,
|
| 52 |
-
"brier": 0.7305371209519513,
|
| 53 |
-
"cross_entropy": 1.4973851876562216
|
| 54 |
-
},
|
| 55 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 56 |
-
"questions": 2,
|
| 57 |
-
"modal_accuracy": 1.0,
|
| 58 |
-
"brier": 4.986938069711992e-05,
|
| 59 |
-
"cross_entropy": 0.003831617325445817
|
| 60 |
-
},
|
| 61 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 62 |
-
"questions": 5,
|
| 63 |
-
"modal_accuracy": 0.6,
|
| 64 |
-
"brier": 0.7805844658893524,
|
| 65 |
-
"cross_entropy": 1.7801497473019683
|
| 66 |
-
},
|
| 67 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 68 |
-
"questions": 7,
|
| 69 |
-
"modal_accuracy": 0.7142857142857143,
|
| 70 |
-
"brier": 0.4790055889595874,
|
| 71 |
-
"cross_entropy": 1.1121963454176031
|
| 72 |
-
},
|
| 73 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 74 |
-
"questions": 12,
|
| 75 |
-
"modal_accuracy": 0.6666666666666666,
|
| 76 |
-
"brier": 0.6101325803838552,
|
| 77 |
-
"cross_entropy": 2.0033591272403792
|
| 78 |
-
},
|
| 79 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 80 |
-
"questions": 10,
|
| 81 |
-
"modal_accuracy": 0.7,
|
| 82 |
-
"brier": 0.6003855550573081,
|
| 83 |
-
"cross_entropy": 1.611779150841159
|
| 84 |
-
},
|
| 85 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 86 |
-
"questions": 4,
|
| 87 |
-
"modal_accuracy": 0.75,
|
| 88 |
-
"brier": 0.42100449664852624,
|
| 89 |
-
"cross_entropy": 1.9128633478153523
|
| 90 |
-
},
|
| 91 |
-
"connect_four": {
|
| 92 |
-
"questions": 56,
|
| 93 |
-
"modal_accuracy": 0.5357142857142857,
|
| 94 |
-
"brier": 0.2741592752209492,
|
| 95 |
-
"cross_entropy": 1.860460665385586
|
| 96 |
-
},
|
| 97 |
-
"customer-control-v1": {
|
| 98 |
-
"questions": 7,
|
| 99 |
-
"modal_accuracy": 1.0,
|
| 100 |
-
"brier": 2.0002025677443218e-06,
|
| 101 |
-
"cross_entropy": 0.0004476462766485558
|
| 102 |
-
},
|
| 103 |
-
"devign": {
|
| 104 |
-
"questions": 2,
|
| 105 |
-
"modal_accuracy": 0.5,
|
| 106 |
-
"brier": 0.41023068518803085,
|
| 107 |
-
"cross_entropy": 0.6006732852714238
|
| 108 |
-
},
|
| 109 |
-
"gsm8k": {
|
| 110 |
-
"questions": 4,
|
| 111 |
-
"modal_accuracy": 1.0,
|
| 112 |
-
"brier": 0.0013119361198151522,
|
| 113 |
-
"cross_entropy": 0.02163855006976333
|
| 114 |
-
},
|
| 115 |
-
"helpsteer2": {
|
| 116 |
-
"questions": 128,
|
| 117 |
-
"modal_accuracy": 0.4296875,
|
| 118 |
-
"brier": 0.5887451086780798,
|
| 119 |
-
"cross_entropy": 2.801737422463417
|
| 120 |
-
},
|
| 121 |
-
"planning": {
|
| 122 |
-
"questions": 192,
|
| 123 |
-
"modal_accuracy": 0.6041666666666666,
|
| 124 |
-
"brier": 0.5293956848932907,
|
| 125 |
-
"cross_entropy": 1.6646787560472525
|
| 126 |
-
},
|
| 127 |
-
"race": {
|
| 128 |
-
"questions": 4,
|
| 129 |
-
"modal_accuracy": 0.5,
|
| 130 |
-
"brier": 0.8826525379691635,
|
| 131 |
-
"cross_entropy": 2.2968173663260414
|
| 132 |
-
},
|
| 133 |
-
"ragtruth": {
|
| 134 |
-
"questions": 6,
|
| 135 |
-
"modal_accuracy": 0.6666666666666666,
|
| 136 |
-
"brier": 0.31120510035647425,
|
| 137 |
-
"cross_entropy": 0.4402550888479215
|
| 138 |
-
},
|
| 139 |
-
"scienceworld": {
|
| 140 |
-
"questions": 128,
|
| 141 |
-
"modal_accuracy": 0.6953125,
|
| 142 |
-
"brier": 0.4588827899272394,
|
| 143 |
-
"cross_entropy": 1.3847298212396886
|
| 144 |
-
},
|
| 145 |
-
"scone": {
|
| 146 |
-
"questions": 192,
|
| 147 |
-
"modal_accuracy": 0.4739583333333333,
|
| 148 |
-
"brier": 0.8022954836937664,
|
| 149 |
-
"cross_entropy": 2.0042890400598963
|
| 150 |
-
},
|
| 151 |
-
"toolace": {
|
| 152 |
-
"questions": 2,
|
| 153 |
-
"modal_accuracy": 1.0,
|
| 154 |
-
"brier": 2.2949296518888497e-08,
|
| 155 |
-
"cross_entropy": 9.582785411797238e-05
|
| 156 |
-
},
|
| 157 |
-
"yuri_v3": {
|
| 158 |
-
"questions": 53,
|
| 159 |
-
"modal_accuracy": 0.7358490566037735,
|
| 160 |
-
"brier": 0.0564251534824245,
|
| 161 |
-
"cross_entropy": 0.8602018822761758
|
| 162 |
-
}
|
| 163 |
-
}
|
| 164 |
-
}
|
| 165 |
-
},
|
| 166 |
-
{
|
| 167 |
-
"step": 2000,
|
| 168 |
-
"examples_seen": 8000,
|
| 169 |
-
"checkpoint": "step-002000",
|
| 170 |
-
"export": "step-002000",
|
| 171 |
-
"validation": {
|
| 172 |
-
"questions": 1024,
|
| 173 |
-
"modal_accuracy": 0.7275390625,
|
| 174 |
-
"brier": 0.31872868889688055,
|
| 175 |
-
"cross_entropy": 1.116584448395983,
|
| 176 |
-
"by_kind": {
|
| 177 |
-
"choice": {
|
| 178 |
-
"questions": 665,
|
| 179 |
-
"modal_accuracy": 0.7398496240601504,
|
| 180 |
-
"brier": 0.3486314543321744,
|
| 181 |
-
"cross_entropy": 1.1847615210242326
|
| 182 |
-
},
|
| 183 |
-
"noul": {
|
| 184 |
-
"questions": 144,
|
| 185 |
-
"modal_accuracy": 0.8472222222222222,
|
| 186 |
-
"brier": 0.24375927242920462,
|
| 187 |
-
"cross_entropy": 0.5036143361404846
|
| 188 |
-
},
|
| 189 |
-
"score": {
|
| 190 |
-
"questions": 215,
|
| 191 |
-
"modal_accuracy": 0.6093023255813953,
|
| 192 |
-
"brier": 0.27645081427769413,
|
| 193 |
-
"cross_entropy": 1.3162586012657769
|
| 194 |
-
}
|
| 195 |
-
},
|
| 196 |
-
"by_source": {
|
| 197 |
-
"alfworld": {
|
| 198 |
-
"questions": 128,
|
| 199 |
-
"modal_accuracy": 0.8515625,
|
| 200 |
-
"brier": 0.20112173562767713,
|
| 201 |
-
"cross_entropy": 0.7568937670273396
|
| 202 |
-
},
|
| 203 |
-
"arithmetic": {
|
| 204 |
-
"questions": 24,
|
| 205 |
-
"modal_accuracy": 0.75,
|
| 206 |
-
"brier": 0.3778614792863498,
|
| 207 |
-
"cross_entropy": 1.0070228429929422
|
| 208 |
-
},
|
| 209 |
-
"babyai": {
|
| 210 |
-
"questions": 58,
|
| 211 |
-
"modal_accuracy": 0.5517241379310345,
|
| 212 |
-
"brier": 0.6998687564446888,
|
| 213 |
-
"cross_entropy": 1.449144847710849
|
| 214 |
-
},
|
| 215 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 216 |
-
"questions": 2,
|
| 217 |
-
"modal_accuracy": 1.0,
|
| 218 |
-
"brier": 8.502797038864707e-06,
|
| 219 |
-
"cross_entropy": 0.0017491505823356403
|
| 220 |
-
},
|
| 221 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 222 |
-
"questions": 5,
|
| 223 |
-
"modal_accuracy": 0.6,
|
| 224 |
-
"brier": 0.64090513735759,
|
| 225 |
-
"cross_entropy": 1.2842974245060323
|
| 226 |
-
},
|
| 227 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 228 |
-
"questions": 7,
|
| 229 |
-
"modal_accuracy": 0.7142857142857143,
|
| 230 |
-
"brier": 0.5732229366022509,
|
| 231 |
-
"cross_entropy": 1.3498112142920367
|
| 232 |
-
},
|
| 233 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 234 |
-
"questions": 12,
|
| 235 |
-
"modal_accuracy": 0.75,
|
| 236 |
-
"brier": 0.45820962916227576,
|
| 237 |
-
"cross_entropy": 1.7893452530760972
|
| 238 |
-
},
|
| 239 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 240 |
-
"questions": 10,
|
| 241 |
-
"modal_accuracy": 0.6,
|
| 242 |
-
"brier": 0.7878584523917267,
|
| 243 |
-
"cross_entropy": 2.649907510922513
|
| 244 |
-
},
|
| 245 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 246 |
-
"questions": 4,
|
| 247 |
-
"modal_accuracy": 0.5,
|
| 248 |
-
"brier": 0.8062445012416534,
|
| 249 |
-
"cross_entropy": 1.8181549807503623
|
| 250 |
-
},
|
| 251 |
-
"connect_four": {
|
| 252 |
-
"questions": 56,
|
| 253 |
-
"modal_accuracy": 0.5714285714285714,
|
| 254 |
-
"brier": 0.268346840698743,
|
| 255 |
-
"cross_entropy": 1.8491581077937622
|
| 256 |
-
},
|
| 257 |
-
"customer-control-v1": {
|
| 258 |
-
"questions": 7,
|
| 259 |
-
"modal_accuracy": 1.0,
|
| 260 |
-
"brier": 4.014360934536871e-05,
|
| 261 |
-
"cross_entropy": 0.0018805145026462434
|
| 262 |
-
},
|
| 263 |
-
"devign": {
|
| 264 |
-
"questions": 2,
|
| 265 |
-
"modal_accuracy": 1.0,
|
| 266 |
-
"brier": 0.43785613217853164,
|
| 267 |
-
"cross_entropy": 0.6306845393838758
|
| 268 |
-
},
|
| 269 |
-
"gsm8k": {
|
| 270 |
-
"questions": 4,
|
| 271 |
-
"modal_accuracy": 1.0,
|
| 272 |
-
"brier": 0.0023963047501729595,
|
| 273 |
-
"cross_entropy": 0.02626187987425209
|
| 274 |
-
},
|
| 275 |
-
"helpsteer2": {
|
| 276 |
-
"questions": 128,
|
| 277 |
-
"modal_accuracy": 0.546875,
|
| 278 |
-
"brier": 0.20973734203719346,
|
| 279 |
-
"cross_entropy": 1.5167245986871056
|
| 280 |
-
},
|
| 281 |
-
"planning": {
|
| 282 |
-
"questions": 192,
|
| 283 |
-
"modal_accuracy": 0.9114583333333334,
|
| 284 |
-
"brier": 0.1816467064252445,
|
| 285 |
-
"cross_entropy": 0.8230249970733392
|
| 286 |
-
},
|
| 287 |
-
"race": {
|
| 288 |
-
"questions": 4,
|
| 289 |
-
"modal_accuracy": 0.5,
|
| 290 |
-
"brier": 0.7315860423730358,
|
| 291 |
-
"cross_entropy": 2.0558892146843153
|
| 292 |
-
},
|
| 293 |
-
"ragtruth": {
|
| 294 |
-
"questions": 6,
|
| 295 |
-
"modal_accuracy": 0.5,
|
| 296 |
-
"brier": 0.6525781536661275,
|
| 297 |
-
"cross_entropy": 0.9808095278526777
|
| 298 |
-
},
|
| 299 |
-
"scienceworld": {
|
| 300 |
-
"questions": 128,
|
| 301 |
-
"modal_accuracy": 0.6953125,
|
| 302 |
-
"brier": 0.430457617431226,
|
| 303 |
-
"cross_entropy": 1.3267025517668618
|
| 304 |
-
},
|
| 305 |
-
"scone": {
|
| 306 |
-
"questions": 192,
|
| 307 |
-
"modal_accuracy": 0.6979166666666666,
|
| 308 |
-
"brier": 0.44132333714721916,
|
| 309 |
-
"cross_entropy": 0.939129528604241
|
| 310 |
-
},
|
| 311 |
-
"toolace": {
|
| 312 |
-
"questions": 2,
|
| 313 |
-
"modal_accuracy": 1.0,
|
| 314 |
-
"brier": 1.4302336637850583e-07,
|
| 315 |
-
"cross_entropy": 0.00026036870363595557
|
| 316 |
-
},
|
| 317 |
-
"yuri_v3": {
|
| 318 |
-
"questions": 53,
|
| 319 |
-
"modal_accuracy": 0.7358490566037735,
|
| 320 |
-
"brier": 0.05366664986435064,
|
| 321 |
-
"cross_entropy": 0.8647152527990684
|
| 322 |
-
}
|
| 323 |
-
}
|
| 324 |
-
}
|
| 325 |
-
},
|
| 326 |
-
{
|
| 327 |
-
"step": 4000,
|
| 328 |
-
"examples_seen": 16000,
|
| 329 |
-
"checkpoint": "step-004000",
|
| 330 |
-
"export": "step-004000",
|
| 331 |
-
"validation": {
|
| 332 |
-
"questions": 1024,
|
| 333 |
-
"modal_accuracy": 0.740234375,
|
| 334 |
-
"brier": 0.31259603426070215,
|
| 335 |
-
"cross_entropy": 1.0641685621781927,
|
| 336 |
-
"by_kind": {
|
| 337 |
-
"choice": {
|
| 338 |
-
"questions": 665,
|
| 339 |
-
"modal_accuracy": 0.7413533834586467,
|
| 340 |
-
"brier": 0.35520899641938364,
|
| 341 |
-
"cross_entropy": 1.150404749554445
|
| 342 |
-
},
|
| 343 |
-
"noul": {
|
| 344 |
-
"questions": 144,
|
| 345 |
-
"modal_accuracy": 0.8333333333333334,
|
| 346 |
-
"brier": 0.2587362169867197,
|
| 347 |
-
"cross_entropy": 0.527162702406588
|
| 348 |
-
},
|
| 349 |
-
"score": {
|
| 350 |
-
"questions": 215,
|
| 351 |
-
"modal_accuracy": 0.6744186046511628,
|
| 352 |
-
"brier": 0.2168667033394475,
|
| 353 |
-
"cross_entropy": 1.157107070094022
|
| 354 |
-
}
|
| 355 |
-
},
|
| 356 |
-
"by_source": {
|
| 357 |
-
"alfworld": {
|
| 358 |
-
"questions": 128,
|
| 359 |
-
"modal_accuracy": 0.859375,
|
| 360 |
-
"brier": 0.1875144540034567,
|
| 361 |
-
"cross_entropy": 0.6708904667559872
|
| 362 |
-
},
|
| 363 |
-
"arithmetic": {
|
| 364 |
-
"questions": 24,
|
| 365 |
-
"modal_accuracy": 0.7083333333333334,
|
| 366 |
-
"brier": 0.4071248120462074,
|
| 367 |
-
"cross_entropy": 1.072867837361791
|
| 368 |
-
},
|
| 369 |
-
"babyai": {
|
| 370 |
-
"questions": 58,
|
| 371 |
-
"modal_accuracy": 0.4827586206896552,
|
| 372 |
-
"brier": 0.7226287558174715,
|
| 373 |
-
"cross_entropy": 1.4270151869584489
|
| 374 |
-
},
|
| 375 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 376 |
-
"questions": 2,
|
| 377 |
-
"modal_accuracy": 1.0,
|
| 378 |
-
"brier": 6.599285085529234e-06,
|
| 379 |
-
"cross_entropy": 0.0016185379703032327
|
| 380 |
-
},
|
| 381 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 382 |
-
"questions": 5,
|
| 383 |
-
"modal_accuracy": 0.8,
|
| 384 |
-
"brier": 0.39998067070790544,
|
| 385 |
-
"cross_entropy": 1.08609476980073
|
| 386 |
-
},
|
| 387 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 388 |
-
"questions": 7,
|
| 389 |
-
"modal_accuracy": 0.8571428571428571,
|
| 390 |
-
"brier": 0.28687468365522234,
|
| 391 |
-
"cross_entropy": 0.7248149597143485
|
| 392 |
-
},
|
| 393 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 394 |
-
"questions": 12,
|
| 395 |
-
"modal_accuracy": 0.75,
|
| 396 |
-
"brier": 0.477438586205062,
|
| 397 |
-
"cross_entropy": 1.9109389754190291
|
| 398 |
-
},
|
| 399 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 400 |
-
"questions": 10,
|
| 401 |
-
"modal_accuracy": 0.6,
|
| 402 |
-
"brier": 0.7693115419877218,
|
| 403 |
-
"cross_entropy": 2.051549980800976
|
| 404 |
-
},
|
| 405 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 406 |
-
"questions": 4,
|
| 407 |
-
"modal_accuracy": 0.25,
|
| 408 |
-
"brier": 1.1149087692116901,
|
| 409 |
-
"cross_entropy": 2.466424832470031
|
| 410 |
-
},
|
| 411 |
-
"connect_four": {
|
| 412 |
-
"questions": 56,
|
| 413 |
-
"modal_accuracy": 0.6428571428571429,
|
| 414 |
-
"brier": 0.26805081416207754,
|
| 415 |
-
"cross_entropy": 1.8479820614570504
|
| 416 |
-
},
|
| 417 |
-
"customer-control-v1": {
|
| 418 |
-
"questions": 7,
|
| 419 |
-
"modal_accuracy": 1.0,
|
| 420 |
-
"brier": 1.6518035601985627e-05,
|
| 421 |
-
"cross_entropy": 0.0012598198888345246
|
| 422 |
-
},
|
| 423 |
-
"devign": {
|
| 424 |
-
"questions": 2,
|
| 425 |
-
"modal_accuracy": 0.5,
|
| 426 |
-
"brier": 0.43338634881169164,
|
| 427 |
-
"cross_entropy": 0.6242116836662074
|
| 428 |
-
},
|
| 429 |
-
"gsm8k": {
|
| 430 |
-
"questions": 4,
|
| 431 |
-
"modal_accuracy": 1.0,
|
| 432 |
-
"brier": 0.0005199031989910008,
|
| 433 |
-
"cross_entropy": 0.013970904245707016
|
| 434 |
-
},
|
| 435 |
-
"helpsteer2": {
|
| 436 |
-
"questions": 128,
|
| 437 |
-
"modal_accuracy": 0.5625,
|
| 438 |
-
"brier": 0.2123470190045601,
|
| 439 |
-
"cross_entropy": 1.5161004021555795
|
| 440 |
-
},
|
| 441 |
-
"planning": {
|
| 442 |
-
"questions": 192,
|
| 443 |
-
"modal_accuracy": 0.9010416666666666,
|
| 444 |
-
"brier": 0.18652079389211781,
|
| 445 |
-
"cross_entropy": 0.7723674627217165
|
| 446 |
-
},
|
| 447 |
-
"race": {
|
| 448 |
-
"questions": 4,
|
| 449 |
-
"modal_accuracy": 0.5,
|
| 450 |
-
"brier": 0.6806357215652038,
|
| 451 |
-
"cross_entropy": 1.9339496668179939
|
| 452 |
-
},
|
| 453 |
-
"ragtruth": {
|
| 454 |
-
"questions": 6,
|
| 455 |
-
"modal_accuracy": 0.6666666666666666,
|
| 456 |
-
"brier": 0.5780328053534197,
|
| 457 |
-
"cross_entropy": 0.9064917649110132
|
| 458 |
-
},
|
| 459 |
-
"scienceworld": {
|
| 460 |
-
"questions": 128,
|
| 461 |
-
"modal_accuracy": 0.6796875,
|
| 462 |
-
"brier": 0.49342345206712573,
|
| 463 |
-
"cross_entropy": 1.4302781643246851
|
| 464 |
-
},
|
| 465 |
-
"scone": {
|
| 466 |
-
"questions": 192,
|
| 467 |
-
"modal_accuracy": 0.765625,
|
| 468 |
-
"brier": 0.3713316872108095,
|
| 469 |
-
"cross_entropy": 0.7410961018920553
|
| 470 |
-
},
|
| 471 |
-
"toolace": {
|
| 472 |
-
"questions": 2,
|
| 473 |
-
"modal_accuracy": 1.0,
|
| 474 |
-
"brier": 3.664008183371336e-07,
|
| 475 |
-
"cross_entropy": 0.00041208643515311584
|
| 476 |
-
},
|
| 477 |
-
"yuri_v3": {
|
| 478 |
-
"questions": 53,
|
| 479 |
-
"modal_accuracy": 0.7547169811320755,
|
| 480 |
-
"brier": 0.056719285382028715,
|
| 481 |
-
"cross_entropy": 0.8641864337643139
|
| 482 |
-
}
|
| 483 |
-
}
|
| 484 |
}
|
| 485 |
},
|
| 486 |
-
{
|
| 487 |
-
"
|
| 488 |
-
|
| 489 |
-
|
| 490 |
-
|
| 491 |
-
|
| 492 |
-
"
|
| 493 |
-
"
|
| 494 |
-
|
| 495 |
-
|
| 496 |
-
"
|
| 497 |
-
|
| 498 |
-
|
| 499 |
-
|
| 500 |
-
|
| 501 |
-
|
| 502 |
-
},
|
| 503 |
-
"noul": {
|
| 504 |
-
"questions": 144,
|
| 505 |
-
"modal_accuracy": 0.8958333333333334,
|
| 506 |
-
"brier": 0.16471210037099535,
|
| 507 |
-
"cross_entropy": 0.39482479972079665
|
| 508 |
-
},
|
| 509 |
-
"score": {
|
| 510 |
-
"questions": 215,
|
| 511 |
-
"modal_accuracy": 0.6604651162790698,
|
| 512 |
-
"brier": 0.22656192147115203,
|
| 513 |
-
"cross_entropy": 1.2057654334760237
|
| 514 |
-
}
|
| 515 |
-
},
|
| 516 |
-
"by_source": {
|
| 517 |
-
"alfworld": {
|
| 518 |
-
"questions": 128,
|
| 519 |
-
"modal_accuracy": 0.8671875,
|
| 520 |
-
"brier": 0.1790825849971539,
|
| 521 |
-
"cross_entropy": 0.6080971490404414
|
| 522 |
-
},
|
| 523 |
-
"arithmetic": {
|
| 524 |
-
"questions": 24,
|
| 525 |
-
"modal_accuracy": 0.6666666666666666,
|
| 526 |
-
"brier": 0.5263100827359132,
|
| 527 |
-
"cross_entropy": 1.403012675529207
|
| 528 |
-
},
|
| 529 |
-
"babyai": {
|
| 530 |
-
"questions": 58,
|
| 531 |
-
"modal_accuracy": 0.5517241379310345,
|
| 532 |
-
"brier": 0.684833327087956,
|
| 533 |
-
"cross_entropy": 1.5818612127101548
|
| 534 |
-
},
|
| 535 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 536 |
-
"questions": 2,
|
| 537 |
-
"modal_accuracy": 1.0,
|
| 538 |
-
"brier": 8.422208826818114e-06,
|
| 539 |
-
"cross_entropy": 0.0016334122276107624
|
| 540 |
-
},
|
| 541 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 542 |
-
"questions": 5,
|
| 543 |
-
"modal_accuracy": 0.8,
|
| 544 |
-
"brier": 0.39072800981047207,
|
| 545 |
-
"cross_entropy": 0.8926558622999824
|
| 546 |
-
},
|
| 547 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 548 |
-
"questions": 7,
|
| 549 |
-
"modal_accuracy": 0.8571428571428571,
|
| 550 |
-
"brier": 0.28439123582360026,
|
| 551 |
-
"cross_entropy": 0.7928804085918381
|
| 552 |
-
},
|
| 553 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 554 |
-
"questions": 12,
|
| 555 |
-
"modal_accuracy": 0.6666666666666666,
|
| 556 |
-
"brier": 0.6286826358235,
|
| 557 |
-
"cross_entropy": 2.0004531092707
|
| 558 |
-
},
|
| 559 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 560 |
-
"questions": 10,
|
| 561 |
-
"modal_accuracy": 0.7,
|
| 562 |
-
"brier": 0.44538602593083176,
|
| 563 |
-
"cross_entropy": 1.117689014223367
|
| 564 |
-
},
|
| 565 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 566 |
-
"questions": 4,
|
| 567 |
-
"modal_accuracy": 0.25,
|
| 568 |
-
"brier": 1.4342464951649092,
|
| 569 |
-
"cross_entropy": 3.381933827755728
|
| 570 |
-
},
|
| 571 |
-
"connect_four": {
|
| 572 |
-
"questions": 56,
|
| 573 |
-
"modal_accuracy": 0.5535714285714286,
|
| 574 |
-
"brier": 0.2622696208064482,
|
| 575 |
-
"cross_entropy": 1.8219529101323455
|
| 576 |
-
},
|
| 577 |
-
"customer-control-v1": {
|
| 578 |
-
"questions": 7,
|
| 579 |
-
"modal_accuracy": 1.0,
|
| 580 |
-
"brier": 2.417104630113183e-06,
|
| 581 |
-
"cross_entropy": 0.0005900334430327367
|
| 582 |
-
},
|
| 583 |
-
"devign": {
|
| 584 |
-
"questions": 2,
|
| 585 |
-
"modal_accuracy": 1.0,
|
| 586 |
-
"brier": 0.43780762834608883,
|
| 587 |
-
"cross_entropy": 0.6306076256145883
|
| 588 |
-
},
|
| 589 |
-
"gsm8k": {
|
| 590 |
-
"questions": 4,
|
| 591 |
-
"modal_accuracy": 1.0,
|
| 592 |
-
"brier": 0.0003744313065983422,
|
| 593 |
-
"cross_entropy": 0.011988960485765076
|
| 594 |
-
},
|
| 595 |
-
"helpsteer2": {
|
| 596 |
-
"questions": 128,
|
| 597 |
-
"modal_accuracy": 0.5859375,
|
| 598 |
-
"brier": 0.20697473621325052,
|
| 599 |
-
"cross_entropy": 1.5034032587243362
|
| 600 |
-
},
|
| 601 |
-
"planning": {
|
| 602 |
-
"questions": 192,
|
| 603 |
-
"modal_accuracy": 0.9635416666666666,
|
| 604 |
-
"brier": 0.11802009440200961,
|
| 605 |
-
"cross_entropy": 0.6579051066101222
|
| 606 |
-
},
|
| 607 |
-
"race": {
|
| 608 |
-
"questions": 4,
|
| 609 |
-
"modal_accuracy": 0.5,
|
| 610 |
-
"brier": 0.7315329429054658,
|
| 611 |
-
"cross_entropy": 2.0647249708287245
|
| 612 |
-
},
|
| 613 |
-
"ragtruth": {
|
| 614 |
-
"questions": 6,
|
| 615 |
-
"modal_accuracy": 0.6666666666666666,
|
| 616 |
-
"brier": 0.609860991064251,
|
| 617 |
-
"cross_entropy": 0.9741778978250341
|
| 618 |
-
},
|
| 619 |
-
"scienceworld": {
|
| 620 |
-
"questions": 128,
|
| 621 |
-
"modal_accuracy": 0.703125,
|
| 622 |
-
"brier": 0.44470820194957994,
|
| 623 |
-
"cross_entropy": 1.2102900099661185
|
| 624 |
-
},
|
| 625 |
-
"scone": {
|
| 626 |
-
"questions": 192,
|
| 627 |
-
"modal_accuracy": 0.796875,
|
| 628 |
-
"brier": 0.29338767561802165,
|
| 629 |
-
"cross_entropy": 0.6140028830315581
|
| 630 |
-
},
|
| 631 |
-
"toolace": {
|
| 632 |
-
"questions": 2,
|
| 633 |
-
"modal_accuracy": 1.0,
|
| 634 |
-
"brier": 2.1804417353162663e-07,
|
| 635 |
-
"cross_entropy": 0.0003234946827488933
|
| 636 |
-
},
|
| 637 |
-
"yuri_v3": {
|
| 638 |
-
"questions": 53,
|
| 639 |
-
"modal_accuracy": 0.7169811320754716,
|
| 640 |
-
"brier": 0.056977519481840835,
|
| 641 |
-
"cross_entropy": 0.860752644211149
|
| 642 |
-
}
|
| 643 |
-
}
|
| 644 |
-
}
|
| 645 |
-
}
|
| 646 |
-
],
|
| 647 |
-
"winner": {
|
| 648 |
-
"step": 8000,
|
| 649 |
-
"examples_seen": 32000,
|
| 650 |
-
"checkpoint": "step-008000",
|
| 651 |
-
"export": "step-008000",
|
| 652 |
-
"validation": {
|
| 653 |
-
"questions": 1024,
|
| 654 |
-
"modal_accuracy": 0.76171875,
|
| 655 |
-
"brier": 0.27786017825450965,
|
| 656 |
-
"cross_entropy": 0.9927829125866763,
|
| 657 |
-
"by_kind": {
|
| 658 |
-
"choice": {
|
| 659 |
-
"questions": 665,
|
| 660 |
-
"modal_accuracy": 0.7654135338345864,
|
| 661 |
-
"brier": 0.3189465668614991,
|
| 662 |
-
"cross_entropy": 1.0534065611001757
|
| 663 |
-
},
|
| 664 |
-
"noul": {
|
| 665 |
-
"questions": 144,
|
| 666 |
-
"modal_accuracy": 0.8958333333333334,
|
| 667 |
-
"brier": 0.16471210037099535,
|
| 668 |
-
"cross_entropy": 0.39482479972079665
|
| 669 |
-
},
|
| 670 |
-
"score": {
|
| 671 |
-
"questions": 215,
|
| 672 |
-
"modal_accuracy": 0.6604651162790698,
|
| 673 |
-
"brier": 0.22656192147115203,
|
| 674 |
-
"cross_entropy": 1.2057654334760237
|
| 675 |
-
}
|
| 676 |
},
|
| 677 |
-
"
|
| 678 |
-
"
|
| 679 |
-
|
| 680 |
-
|
| 681 |
-
|
| 682 |
-
|
| 683 |
-
|
| 684 |
-
"arithmetic": {
|
| 685 |
-
"questions": 24,
|
| 686 |
-
"modal_accuracy": 0.6666666666666666,
|
| 687 |
-
"brier": 0.5263100827359132,
|
| 688 |
-
"cross_entropy": 1.403012675529207
|
| 689 |
-
},
|
| 690 |
-
"babyai": {
|
| 691 |
-
"questions": 58,
|
| 692 |
-
"modal_accuracy": 0.5517241379310345,
|
| 693 |
-
"brier": 0.684833327087956,
|
| 694 |
-
"cross_entropy": 1.5818612127101548
|
| 695 |
-
},
|
| 696 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 697 |
-
"questions": 2,
|
| 698 |
-
"modal_accuracy": 1.0,
|
| 699 |
-
"brier": 8.422208826818114e-06,
|
| 700 |
-
"cross_entropy": 0.0016334122276107624
|
| 701 |
-
},
|
| 702 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 703 |
-
"questions": 5,
|
| 704 |
-
"modal_accuracy": 0.8,
|
| 705 |
-
"brier": 0.39072800981047207,
|
| 706 |
-
"cross_entropy": 0.8926558622999824
|
| 707 |
-
},
|
| 708 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 709 |
-
"questions": 7,
|
| 710 |
-
"modal_accuracy": 0.8571428571428571,
|
| 711 |
-
"brier": 0.28439123582360026,
|
| 712 |
-
"cross_entropy": 0.7928804085918381
|
| 713 |
-
},
|
| 714 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 715 |
-
"questions": 12,
|
| 716 |
-
"modal_accuracy": 0.6666666666666666,
|
| 717 |
-
"brier": 0.6286826358235,
|
| 718 |
-
"cross_entropy": 2.0004531092707
|
| 719 |
-
},
|
| 720 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 721 |
-
"questions": 10,
|
| 722 |
-
"modal_accuracy": 0.7,
|
| 723 |
-
"brier": 0.44538602593083176,
|
| 724 |
-
"cross_entropy": 1.117689014223367
|
| 725 |
-
},
|
| 726 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 727 |
-
"questions": 4,
|
| 728 |
-
"modal_accuracy": 0.25,
|
| 729 |
-
"brier": 1.4342464951649092,
|
| 730 |
-
"cross_entropy": 3.381933827755728
|
| 731 |
-
},
|
| 732 |
-
"connect_four": {
|
| 733 |
-
"questions": 56,
|
| 734 |
-
"modal_accuracy": 0.5535714285714286,
|
| 735 |
-
"brier": 0.2622696208064482,
|
| 736 |
-
"cross_entropy": 1.8219529101323455
|
| 737 |
-
},
|
| 738 |
-
"customer-control-v1": {
|
| 739 |
-
"questions": 7,
|
| 740 |
-
"modal_accuracy": 1.0,
|
| 741 |
-
"brier": 2.417104630113183e-06,
|
| 742 |
-
"cross_entropy": 0.0005900334430327367
|
| 743 |
-
},
|
| 744 |
-
"devign": {
|
| 745 |
-
"questions": 2,
|
| 746 |
-
"modal_accuracy": 1.0,
|
| 747 |
-
"brier": 0.43780762834608883,
|
| 748 |
-
"cross_entropy": 0.6306076256145883
|
| 749 |
-
},
|
| 750 |
-
"gsm8k": {
|
| 751 |
-
"questions": 4,
|
| 752 |
-
"modal_accuracy": 1.0,
|
| 753 |
-
"brier": 0.0003744313065983422,
|
| 754 |
-
"cross_entropy": 0.011988960485765076
|
| 755 |
-
},
|
| 756 |
-
"helpsteer2": {
|
| 757 |
-
"questions": 128,
|
| 758 |
-
"modal_accuracy": 0.5859375,
|
| 759 |
-
"brier": 0.20697473621325052,
|
| 760 |
-
"cross_entropy": 1.5034032587243362
|
| 761 |
-
},
|
| 762 |
-
"planning": {
|
| 763 |
-
"questions": 192,
|
| 764 |
-
"modal_accuracy": 0.9635416666666666,
|
| 765 |
-
"brier": 0.11802009440200961,
|
| 766 |
-
"cross_entropy": 0.6579051066101222
|
| 767 |
-
},
|
| 768 |
-
"race": {
|
| 769 |
-
"questions": 4,
|
| 770 |
-
"modal_accuracy": 0.5,
|
| 771 |
-
"brier": 0.7315329429054658,
|
| 772 |
-
"cross_entropy": 2.0647249708287245
|
| 773 |
-
},
|
| 774 |
-
"ragtruth": {
|
| 775 |
-
"questions": 6,
|
| 776 |
-
"modal_accuracy": 0.6666666666666666,
|
| 777 |
-
"brier": 0.609860991064251,
|
| 778 |
-
"cross_entropy": 0.9741778978250341
|
| 779 |
-
},
|
| 780 |
-
"scienceworld": {
|
| 781 |
-
"questions": 128,
|
| 782 |
-
"modal_accuracy": 0.703125,
|
| 783 |
-
"brier": 0.44470820194957994,
|
| 784 |
-
"cross_entropy": 1.2102900099661185
|
| 785 |
-
},
|
| 786 |
-
"scone": {
|
| 787 |
-
"questions": 192,
|
| 788 |
-
"modal_accuracy": 0.796875,
|
| 789 |
-
"brier": 0.29338767561802165,
|
| 790 |
-
"cross_entropy": 0.6140028830315581
|
| 791 |
-
},
|
| 792 |
-
"toolace": {
|
| 793 |
-
"questions": 2,
|
| 794 |
-
"modal_accuracy": 1.0,
|
| 795 |
-
"brier": 2.1804417353162663e-07,
|
| 796 |
-
"cross_entropy": 0.0003234946827488933
|
| 797 |
-
},
|
| 798 |
-
"yuri_v3": {
|
| 799 |
-
"questions": 53,
|
| 800 |
-
"modal_accuracy": 0.7169811320754716,
|
| 801 |
-
"brier": 0.056977519481840835,
|
| 802 |
-
"cross_entropy": 0.860752644211149
|
| 803 |
-
}
|
| 804 |
}
|
| 805 |
}
|
| 806 |
},
|
| 807 |
-
"
|
| 808 |
-
|
| 809 |
-
|
| 810 |
-
|
| 811 |
-
"
|
| 812 |
-
|
| 813 |
-
|
| 814 |
-
|
| 815 |
-
|
| 816 |
-
"
|
| 817 |
-
"
|
| 818 |
-
|
| 819 |
-
|
| 820 |
-
|
| 821 |
-
|
| 822 |
-
|
| 823 |
-
"noul": {
|
| 824 |
-
"questions": 144,
|
| 825 |
-
"modal_accuracy": 0.7361111111111112,
|
| 826 |
-
"brier": 0.4178836877295325,
|
| 827 |
-
"cross_entropy": 0.9008511909421006
|
| 828 |
-
},
|
| 829 |
-
"score": {
|
| 830 |
-
"questions": 215,
|
| 831 |
-
"modal_accuracy": 0.5023255813953489,
|
| 832 |
-
"brier": 0.5745442767822497,
|
| 833 |
-
"cross_entropy": 2.347590209909279
|
| 834 |
-
}
|
| 835 |
},
|
| 836 |
-
"
|
| 837 |
-
"
|
| 838 |
-
|
| 839 |
-
|
| 840 |
-
|
| 841 |
-
|
| 842 |
-
|
| 843 |
-
"arithmetic": {
|
| 844 |
-
"questions": 24,
|
| 845 |
-
"modal_accuracy": 0.7083333333333334,
|
| 846 |
-
"brier": 0.428241328947602,
|
| 847 |
-
"cross_entropy": 1.050083253299498
|
| 848 |
-
},
|
| 849 |
-
"babyai": {
|
| 850 |
-
"questions": 58,
|
| 851 |
-
"modal_accuracy": 0.39655172413793105,
|
| 852 |
-
"brier": 0.7305371209519513,
|
| 853 |
-
"cross_entropy": 1.4973851876562216
|
| 854 |
-
},
|
| 855 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 856 |
-
"questions": 2,
|
| 857 |
-
"modal_accuracy": 1.0,
|
| 858 |
-
"brier": 4.986938069711992e-05,
|
| 859 |
-
"cross_entropy": 0.003831617325445817
|
| 860 |
-
},
|
| 861 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 862 |
-
"questions": 5,
|
| 863 |
-
"modal_accuracy": 0.6,
|
| 864 |
-
"brier": 0.7805844658893524,
|
| 865 |
-
"cross_entropy": 1.7801497473019683
|
| 866 |
-
},
|
| 867 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 868 |
-
"questions": 7,
|
| 869 |
-
"modal_accuracy": 0.7142857142857143,
|
| 870 |
-
"brier": 0.4790055889595874,
|
| 871 |
-
"cross_entropy": 1.1121963454176031
|
| 872 |
-
},
|
| 873 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 874 |
-
"questions": 12,
|
| 875 |
-
"modal_accuracy": 0.6666666666666666,
|
| 876 |
-
"brier": 0.6101325803838552,
|
| 877 |
-
"cross_entropy": 2.0033591272403792
|
| 878 |
-
},
|
| 879 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 880 |
-
"questions": 10,
|
| 881 |
-
"modal_accuracy": 0.7,
|
| 882 |
-
"brier": 0.6003855550573081,
|
| 883 |
-
"cross_entropy": 1.611779150841159
|
| 884 |
-
},
|
| 885 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 886 |
-
"questions": 4,
|
| 887 |
-
"modal_accuracy": 0.75,
|
| 888 |
-
"brier": 0.42100449664852624,
|
| 889 |
-
"cross_entropy": 1.9128633478153523
|
| 890 |
-
},
|
| 891 |
-
"connect_four": {
|
| 892 |
-
"questions": 56,
|
| 893 |
-
"modal_accuracy": 0.5357142857142857,
|
| 894 |
-
"brier": 0.2741592752209492,
|
| 895 |
-
"cross_entropy": 1.860460665385586
|
| 896 |
-
},
|
| 897 |
-
"customer-control-v1": {
|
| 898 |
-
"questions": 7,
|
| 899 |
-
"modal_accuracy": 1.0,
|
| 900 |
-
"brier": 2.0002025677443218e-06,
|
| 901 |
-
"cross_entropy": 0.0004476462766485558
|
| 902 |
-
},
|
| 903 |
-
"devign": {
|
| 904 |
-
"questions": 2,
|
| 905 |
-
"modal_accuracy": 0.5,
|
| 906 |
-
"brier": 0.41023068518803085,
|
| 907 |
-
"cross_entropy": 0.6006732852714238
|
| 908 |
-
},
|
| 909 |
-
"gsm8k": {
|
| 910 |
-
"questions": 4,
|
| 911 |
-
"modal_accuracy": 1.0,
|
| 912 |
-
"brier": 0.0013119361198151522,
|
| 913 |
-
"cross_entropy": 0.02163855006976333
|
| 914 |
-
},
|
| 915 |
-
"helpsteer2": {
|
| 916 |
-
"questions": 128,
|
| 917 |
-
"modal_accuracy": 0.4296875,
|
| 918 |
-
"brier": 0.5887451086780798,
|
| 919 |
-
"cross_entropy": 2.801737422463417
|
| 920 |
-
},
|
| 921 |
-
"planning": {
|
| 922 |
-
"questions": 192,
|
| 923 |
-
"modal_accuracy": 0.6041666666666666,
|
| 924 |
-
"brier": 0.5293956848932907,
|
| 925 |
-
"cross_entropy": 1.6646787560472525
|
| 926 |
-
},
|
| 927 |
-
"race": {
|
| 928 |
-
"questions": 4,
|
| 929 |
-
"modal_accuracy": 0.5,
|
| 930 |
-
"brier": 0.8826525379691635,
|
| 931 |
-
"cross_entropy": 2.2968173663260414
|
| 932 |
-
},
|
| 933 |
-
"ragtruth": {
|
| 934 |
-
"questions": 6,
|
| 935 |
-
"modal_accuracy": 0.6666666666666666,
|
| 936 |
-
"brier": 0.31120510035647425,
|
| 937 |
-
"cross_entropy": 0.4402550888479215
|
| 938 |
-
},
|
| 939 |
-
"scienceworld": {
|
| 940 |
-
"questions": 128,
|
| 941 |
-
"modal_accuracy": 0.6953125,
|
| 942 |
-
"brier": 0.4588827899272394,
|
| 943 |
-
"cross_entropy": 1.3847298212396886
|
| 944 |
-
},
|
| 945 |
-
"scone": {
|
| 946 |
-
"questions": 192,
|
| 947 |
-
"modal_accuracy": 0.4739583333333333,
|
| 948 |
-
"brier": 0.8022954836937664,
|
| 949 |
-
"cross_entropy": 2.0042890400598963
|
| 950 |
-
},
|
| 951 |
-
"toolace": {
|
| 952 |
-
"questions": 2,
|
| 953 |
-
"modal_accuracy": 1.0,
|
| 954 |
-
"brier": 2.2949296518888497e-08,
|
| 955 |
-
"cross_entropy": 9.582785411797238e-05
|
| 956 |
-
},
|
| 957 |
-
"yuri_v3": {
|
| 958 |
-
"questions": 53,
|
| 959 |
-
"modal_accuracy": 0.7358490566037735,
|
| 960 |
-
"brier": 0.0564251534824245,
|
| 961 |
-
"cross_entropy": 0.8602018822761758
|
| 962 |
-
}
|
| 963 |
-
}
|
| 964 |
-
},
|
| 965 |
-
"new_test": {
|
| 966 |
-
"questions": 512,
|
| 967 |
-
"modal_accuracy": 0.46484375,
|
| 968 |
-
"brier": 0.7260609345715332,
|
| 969 |
-
"cross_entropy": 2.283754099438618,
|
| 970 |
-
"by_kind": {
|
| 971 |
-
"choice": {
|
| 972 |
-
"questions": 241,
|
| 973 |
-
"modal_accuracy": 0.3941908713692946,
|
| 974 |
-
"brier": 0.814820900706993,
|
| 975 |
-
"cross_entropy": 2.425967860593386
|
| 976 |
-
},
|
| 977 |
-
"noul": {
|
| 978 |
-
"questions": 94,
|
| 979 |
-
"modal_accuracy": 0.5851063829787234,
|
| 980 |
-
"brier": 0.7046781668962778,
|
| 981 |
-
"cross_entropy": 1.348769508075005
|
| 982 |
-
},
|
| 983 |
-
"score": {
|
| 984 |
-
"questions": 177,
|
| 985 |
-
"modal_accuracy": 0.4971751412429379,
|
| 986 |
-
"brier": 0.6165627895027659,
|
| 987 |
-
"cross_entropy": 2.586663902545287
|
| 988 |
-
}
|
| 989 |
},
|
| 990 |
-
"
|
| 991 |
-
"
|
| 992 |
-
|
| 993 |
-
|
| 994 |
-
|
| 995 |
-
|
| 996 |
-
|
| 997 |
-
"planning": {
|
| 998 |
-
"questions": 192,
|
| 999 |
-
"modal_accuracy": 0.484375,
|
| 1000 |
-
"brier": 0.695581487251716,
|
| 1001 |
-
"cross_entropy": 2.128417640206426
|
| 1002 |
-
},
|
| 1003 |
-
"scone": {
|
| 1004 |
-
"questions": 192,
|
| 1005 |
-
"modal_accuracy": 0.421875,
|
| 1006 |
-
"brier": 0.8903884009045245,
|
| 1007 |
-
"cross_entropy": 2.1523295323408504
|
| 1008 |
-
}
|
| 1009 |
}
|
| 1010 |
},
|
| 1011 |
-
"
|
| 1012 |
-
"
|
| 1013 |
-
|
| 1014 |
-
|
| 1015 |
-
|
| 1016 |
-
|
| 1017 |
-
"
|
| 1018 |
-
|
| 1019 |
-
"modal_accuracy": 0.7176368375325803,
|
| 1020 |
-
"brier": 0.34765282650467216,
|
| 1021 |
-
"cross_entropy": 1.0022488366720779
|
| 1022 |
-
},
|
| 1023 |
-
"noul": {
|
| 1024 |
-
"questions": 649,
|
| 1025 |
-
"modal_accuracy": 0.9090909090909091,
|
| 1026 |
-
"brier": 0.1294082456235435,
|
| 1027 |
-
"cross_entropy": 0.3089760935689536
|
| 1028 |
-
},
|
| 1029 |
-
"score": {
|
| 1030 |
-
"questions": 248,
|
| 1031 |
-
"modal_accuracy": 0.8588709677419355,
|
| 1032 |
-
"brier": 0.1295168598982062,
|
| 1033 |
-
"cross_entropy": 0.6581264080729309
|
| 1034 |
-
}
|
| 1035 |
},
|
| 1036 |
-
"
|
| 1037 |
-
"
|
| 1038 |
-
|
| 1039 |
-
|
| 1040 |
-
|
| 1041 |
-
|
| 1042 |
-
|
| 1043 |
-
"arithmetic": {
|
| 1044 |
-
"questions": 30,
|
| 1045 |
-
"modal_accuracy": 0.8,
|
| 1046 |
-
"brier": 0.365987202749238,
|
| 1047 |
-
"cross_entropy": 0.874987503758623
|
| 1048 |
-
},
|
| 1049 |
-
"babyai": {
|
| 1050 |
-
"questions": 128,
|
| 1051 |
-
"modal_accuracy": 0.4140625,
|
| 1052 |
-
"brier": 0.7052731307744691,
|
| 1053 |
-
"cross_entropy": 1.4919952201710827
|
| 1054 |
-
},
|
| 1055 |
-
"community-diversity-v2/browser_tools/goal_completion": {
|
| 1056 |
-
"questions": 24,
|
| 1057 |
-
"modal_accuracy": 0.9583333333333334,
|
| 1058 |
-
"brier": 0.0757580121004977,
|
| 1059 |
-
"cross_entropy": 0.10499604055191543
|
| 1060 |
-
},
|
| 1061 |
-
"community-diversity-v2/contracts/exclusivity_exception": {
|
| 1062 |
-
"questions": 23,
|
| 1063 |
-
"modal_accuracy": 0.8695652173913043,
|
| 1064 |
-
"brier": 0.17625150922502755,
|
| 1065 |
-
"cross_entropy": 0.3819911750055886
|
| 1066 |
-
},
|
| 1067 |
-
"community-diversity-v2/games/pit_stop": {
|
| 1068 |
-
"questions": 13,
|
| 1069 |
-
"modal_accuracy": 0.46153846153846156,
|
| 1070 |
-
"brier": 0.8058966314340112,
|
| 1071 |
-
"cross_entropy": 1.7033569197410379
|
| 1072 |
-
},
|
| 1073 |
-
"community-diversity-v2/ir/warranty_transfer": {
|
| 1074 |
-
"questions": 15,
|
| 1075 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1076 |
-
"brier": 0.13580214677968178,
|
| 1077 |
-
"cross_entropy": 0.36657722104104
|
| 1078 |
-
},
|
| 1079 |
-
"community-diversity-v2/rag/independent_sources": {
|
| 1080 |
-
"questions": 21,
|
| 1081 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1082 |
-
"brier": 0.5103346094744698,
|
| 1083 |
-
"cross_entropy": 0.9877849897459301
|
| 1084 |
-
},
|
| 1085 |
-
"community-diversity-v2/rubric_judge/balanced_tradeoff": {
|
| 1086 |
-
"questions": 12,
|
| 1087 |
-
"modal_accuracy": 0.8333333333333334,
|
| 1088 |
-
"brier": 0.31827468282287835,
|
| 1089 |
-
"cross_entropy": 0.7584622273231553
|
| 1090 |
-
},
|
| 1091 |
-
"community-diversity-v2/shell_history/incremental_copy": {
|
| 1092 |
-
"questions": 17,
|
| 1093 |
-
"modal_accuracy": 0.8823529411764706,
|
| 1094 |
-
"brier": 0.23477636259692025,
|
| 1095 |
-
"cross_entropy": 0.5888464829632447
|
| 1096 |
-
},
|
| 1097 |
-
"community-diversity-v2/support/outage_compensation": {
|
| 1098 |
-
"questions": 3,
|
| 1099 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1100 |
-
"brier": 0.46034493119811687,
|
| 1101 |
-
"cross_entropy": 1.0544000105768865
|
| 1102 |
-
},
|
| 1103 |
-
"connect_four": {
|
| 1104 |
-
"questions": 128,
|
| 1105 |
-
"modal_accuracy": 0.5234375,
|
| 1106 |
-
"brier": 0.2605442517143562,
|
| 1107 |
-
"cross_entropy": 1.8198101893170386
|
| 1108 |
-
},
|
| 1109 |
-
"customer-control-v1": {
|
| 1110 |
-
"questions": 45,
|
| 1111 |
-
"modal_accuracy": 1.0,
|
| 1112 |
-
"brier": 0.0003903831787449739,
|
| 1113 |
-
"cross_entropy": 0.01599108917033031
|
| 1114 |
-
},
|
| 1115 |
-
"devign": {
|
| 1116 |
-
"questions": 46,
|
| 1117 |
-
"modal_accuracy": 0.5434782608695652,
|
| 1118 |
-
"brier": 0.5038973081034425,
|
| 1119 |
-
"cross_entropy": 0.6986062066022048
|
| 1120 |
-
},
|
| 1121 |
-
"entity_belief_tracking": {
|
| 1122 |
-
"questions": 19,
|
| 1123 |
-
"modal_accuracy": 1.0,
|
| 1124 |
-
"brier": 0.003996155280999412,
|
| 1125 |
-
"cross_entropy": 0.018453305162397043
|
| 1126 |
-
},
|
| 1127 |
-
"event_state_reconstruction": {
|
| 1128 |
-
"questions": 30,
|
| 1129 |
-
"modal_accuracy": 1.0,
|
| 1130 |
-
"brier": 0.000632084906836889,
|
| 1131 |
-
"cross_entropy": 0.0070833207931613705
|
| 1132 |
-
},
|
| 1133 |
-
"event_triggered_execution": {
|
| 1134 |
-
"questions": 18,
|
| 1135 |
-
"modal_accuracy": 0.8888888888888888,
|
| 1136 |
-
"brier": 0.20065607784122677,
|
| 1137 |
-
"cross_entropy": 0.3866436913361192
|
| 1138 |
-
},
|
| 1139 |
-
"evidence_sufficiency": {
|
| 1140 |
-
"questions": 13,
|
| 1141 |
-
"modal_accuracy": 0.7692307692307693,
|
| 1142 |
-
"brier": 0.3885272040680112,
|
| 1143 |
-
"cross_entropy": 0.7354956714769301
|
| 1144 |
-
},
|
| 1145 |
-
"goal_directed_planning": {
|
| 1146 |
-
"questions": 19,
|
| 1147 |
-
"modal_accuracy": 0.8421052631578947,
|
| 1148 |
-
"brier": 0.24661119154460764,
|
| 1149 |
-
"cross_entropy": 0.80410976975599
|
| 1150 |
-
},
|
| 1151 |
-
"graph_navigation": {
|
| 1152 |
-
"questions": 6,
|
| 1153 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1154 |
-
"brier": 0.5177916242601774,
|
| 1155 |
-
"cross_entropy": 1.158343409907232
|
| 1156 |
-
},
|
| 1157 |
-
"gsm8k": {
|
| 1158 |
-
"questions": 106,
|
| 1159 |
-
"modal_accuracy": 0.7547169811320755,
|
| 1160 |
-
"brier": 0.38874049122073956,
|
| 1161 |
-
"cross_entropy": 0.7555982932600139
|
| 1162 |
-
},
|
| 1163 |
-
"instruction_execution": {
|
| 1164 |
-
"questions": 12,
|
| 1165 |
-
"modal_accuracy": 0.9166666666666666,
|
| 1166 |
-
"brier": 0.16643936427800027,
|
| 1167 |
-
"cross_entropy": 0.5910813947667087
|
| 1168 |
-
},
|
| 1169 |
-
"multi_view_adjudication": {
|
| 1170 |
-
"questions": 20,
|
| 1171 |
-
"modal_accuracy": 0.95,
|
| 1172 |
-
"brier": 0.07032918266188046,
|
| 1173 |
-
"cross_entropy": 0.09175669834651938
|
| 1174 |
-
},
|
| 1175 |
-
"needle_retrieval": {
|
| 1176 |
-
"questions": 5,
|
| 1177 |
-
"modal_accuracy": 1.0,
|
| 1178 |
-
"brier": 1.0762412043050325e-08,
|
| 1179 |
-
"cross_entropy": 4.8760052363298926e-05
|
| 1180 |
-
},
|
| 1181 |
-
"partial_observation_calibration": {
|
| 1182 |
-
"questions": 5,
|
| 1183 |
-
"modal_accuracy": 0.8,
|
| 1184 |
-
"brier": 0.02878926642066244,
|
| 1185 |
-
"cross_entropy": 0.5423652505807739
|
| 1186 |
-
},
|
| 1187 |
-
"policy_applicability": {
|
| 1188 |
-
"questions": 17,
|
| 1189 |
-
"modal_accuracy": 1.0,
|
| 1190 |
-
"brier": 0.000332556851194119,
|
| 1191 |
-
"cross_entropy": 0.006342650317050527
|
| 1192 |
-
},
|
| 1193 |
-
"policy_under_uncertainty": {
|
| 1194 |
-
"questions": 8,
|
| 1195 |
-
"modal_accuracy": 1.0,
|
| 1196 |
-
"brier": 0.055895947994874184,
|
| 1197 |
-
"cross_entropy": 0.5045127678250978
|
| 1198 |
-
},
|
| 1199 |
-
"race": {
|
| 1200 |
-
"questions": 104,
|
| 1201 |
-
"modal_accuracy": 0.9134615384615384,
|
| 1202 |
-
"brier": 0.15016846124700772,
|
| 1203 |
-
"cross_entropy": 0.3446495798522507
|
| 1204 |
-
},
|
| 1205 |
-
"ragtruth": {
|
| 1206 |
-
"questions": 128,
|
| 1207 |
-
"modal_accuracy": 0.84375,
|
| 1208 |
-
"brier": 0.2673585181473687,
|
| 1209 |
-
"cross_entropy": 0.4660759647283446
|
| 1210 |
-
},
|
| 1211 |
-
"reasoning-control-v1": {
|
| 1212 |
-
"questions": 45,
|
| 1213 |
-
"modal_accuracy": 0.9111111111111111,
|
| 1214 |
-
"brier": 0.11092651727785302,
|
| 1215 |
-
"cross_entropy": 0.2669770296039986
|
| 1216 |
-
},
|
| 1217 |
-
"record_aggregation": {
|
| 1218 |
-
"questions": 14,
|
| 1219 |
-
"modal_accuracy": 0.7857142857142857,
|
| 1220 |
-
"brier": 0.29174772725112047,
|
| 1221 |
-
"cross_entropy": 0.45793008149173803
|
| 1222 |
-
},
|
| 1223 |
-
"record_eligibility": {
|
| 1224 |
-
"questions": 8,
|
| 1225 |
-
"modal_accuracy": 0.875,
|
| 1226 |
-
"brier": 0.10770378005071689,
|
| 1227 |
-
"cross_entropy": 0.13562242943700845
|
| 1228 |
-
},
|
| 1229 |
-
"scienceworld": {
|
| 1230 |
-
"questions": 128,
|
| 1231 |
-
"modal_accuracy": 0.5859375,
|
| 1232 |
-
"brier": 0.6006833831415513,
|
| 1233 |
-
"cross_entropy": 1.7746254397586039
|
| 1234 |
-
},
|
| 1235 |
-
"snake-v1": {
|
| 1236 |
-
"questions": 30,
|
| 1237 |
-
"modal_accuracy": 0.9666666666666667,
|
| 1238 |
-
"brier": 0.04547834255423711,
|
| 1239 |
-
"cross_entropy": 0.25335150127501904
|
| 1240 |
-
},
|
| 1241 |
-
"state_perturbation": {
|
| 1242 |
-
"questions": 11,
|
| 1243 |
-
"modal_accuracy": 0.8181818181818182,
|
| 1244 |
-
"brier": 0.3589651830631727,
|
| 1245 |
-
"cross_entropy": 0.9294419729230835
|
| 1246 |
-
},
|
| 1247 |
-
"table_lookup": {
|
| 1248 |
-
"questions": 13,
|
| 1249 |
-
"modal_accuracy": 0.8461538461538461,
|
| 1250 |
-
"brier": 0.1995349559267647,
|
| 1251 |
-
"cross_entropy": 0.7888754577823849
|
| 1252 |
-
},
|
| 1253 |
-
"taxonomy_routing": {
|
| 1254 |
-
"questions": 8,
|
| 1255 |
-
"modal_accuracy": 1.0,
|
| 1256 |
-
"brier": 4.5835879900615156e-05,
|
| 1257 |
-
"cross_entropy": 0.002082040171618654
|
| 1258 |
-
},
|
| 1259 |
-
"tic_tac_toe-v1": {
|
| 1260 |
-
"questions": 15,
|
| 1261 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1262 |
-
"brier": 0.2455337950398313,
|
| 1263 |
-
"cross_entropy": 1.1492408686263713
|
| 1264 |
-
},
|
| 1265 |
-
"tile_platformer-v1": {
|
| 1266 |
-
"questions": 15,
|
| 1267 |
-
"modal_accuracy": 1.0,
|
| 1268 |
-
"brier": 0.023323220046764518,
|
| 1269 |
-
"cross_entropy": 0.08534179201524755
|
| 1270 |
-
},
|
| 1271 |
-
"toolace": {
|
| 1272 |
-
"questions": 128,
|
| 1273 |
-
"modal_accuracy": 0.9921875,
|
| 1274 |
-
"brier": 0.015580959620615913,
|
| 1275 |
-
"cross_entropy": 0.043558856167122205
|
| 1276 |
-
},
|
| 1277 |
-
"trex_runner-v1": {
|
| 1278 |
-
"questions": 2,
|
| 1279 |
-
"modal_accuracy": 1.0,
|
| 1280 |
-
"brier": 0.01809983422643395,
|
| 1281 |
-
"cross_entropy": 0.11356731402761971
|
| 1282 |
-
},
|
| 1283 |
-
"vizdoom-basic-v1": {
|
| 1284 |
-
"questions": 45,
|
| 1285 |
-
"modal_accuracy": 0.9555555555555556,
|
| 1286 |
-
"brier": 0.08476731050032603,
|
| 1287 |
-
"cross_entropy": 0.18352009286167528
|
| 1288 |
-
},
|
| 1289 |
-
"wanli-decisions-v1": {
|
| 1290 |
-
"questions": 128,
|
| 1291 |
-
"modal_accuracy": 0.6953125,
|
| 1292 |
-
"brier": 0.5342434192250224,
|
| 1293 |
-
"cross_entropy": 1.027889201322489
|
| 1294 |
-
},
|
| 1295 |
-
"workflow-controls-v1/agent_trace_observability": {
|
| 1296 |
-
"questions": 15,
|
| 1297 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1298 |
-
"brier": 0.13236608755793375,
|
| 1299 |
-
"cross_entropy": 0.37446550855959687
|
| 1300 |
-
},
|
| 1301 |
-
"workflow-controls-v1/customer_service": {
|
| 1302 |
-
"questions": 15,
|
| 1303 |
-
"modal_accuracy": 1.0,
|
| 1304 |
-
"brier": 9.801834989730725e-07,
|
| 1305 |
-
"cross_entropy": 0.00023351213246618595
|
| 1306 |
-
},
|
| 1307 |
-
"workflow-controls-v1/invoice_processing": {
|
| 1308 |
-
"questions": 15,
|
| 1309 |
-
"modal_accuracy": 1.0,
|
| 1310 |
-
"brier": 4.168477226094833e-07,
|
| 1311 |
-
"cross_entropy": 0.00024165434698863794
|
| 1312 |
-
},
|
| 1313 |
-
"workflow-controls-v1/security_incidents": {
|
| 1314 |
-
"questions": 14,
|
| 1315 |
-
"modal_accuracy": 1.0,
|
| 1316 |
-
"brier": 2.5913148618252294e-09,
|
| 1317 |
-
"cross_entropy": 1.9618643712043932e-05
|
| 1318 |
-
},
|
| 1319 |
-
"yuri_v3": {
|
| 1320 |
-
"questions": 256,
|
| 1321 |
-
"modal_accuracy": 0.8671875,
|
| 1322 |
-
"brier": 0.03336715127934959,
|
| 1323 |
-
"cross_entropy": 0.8210925298273019
|
| 1324 |
-
}
|
| 1325 |
-
}
|
| 1326 |
-
},
|
| 1327 |
-
"transfer": {
|
| 1328 |
-
"questions": 89,
|
| 1329 |
-
"modal_accuracy": 0.6404494382022472,
|
| 1330 |
-
"brier": 0.49972145070841484,
|
| 1331 |
-
"cross_entropy": 1.069735656698559,
|
| 1332 |
-
"by_kind": {
|
| 1333 |
-
"choice": {
|
| 1334 |
-
"questions": 33,
|
| 1335 |
-
"modal_accuracy": 0.6060606060606061,
|
| 1336 |
-
"brier": 0.5839105237267139,
|
| 1337 |
-
"cross_entropy": 1.3151504789912494
|
| 1338 |
-
},
|
| 1339 |
-
"noul": {
|
| 1340 |
-
"questions": 41,
|
| 1341 |
-
"modal_accuracy": 0.7560975609756098,
|
| 1342 |
-
"brier": 0.34728857025217713,
|
| 1343 |
-
"cross_entropy": 0.6781416340204177
|
| 1344 |
-
},
|
| 1345 |
-
"score": {
|
| 1346 |
-
"questions": 15,
|
| 1347 |
-
"modal_accuracy": 0.4,
|
| 1348 |
-
"brier": 0.7311553633152064,
|
| 1349 |
-
"cross_entropy": 1.6001800429748936
|
| 1350 |
-
}
|
| 1351 |
},
|
| 1352 |
-
"
|
| 1353 |
-
"
|
| 1354 |
-
|
| 1355 |
-
|
| 1356 |
-
|
| 1357 |
-
|
| 1358 |
-
|
| 1359 |
-
"code": {
|
| 1360 |
-
"questions": 6,
|
| 1361 |
-
"modal_accuracy": 0.5,
|
| 1362 |
-
"brier": 0.6355124607251713,
|
| 1363 |
-
"cross_entropy": 0.9367448318704151
|
| 1364 |
-
},
|
| 1365 |
-
"dialog": {
|
| 1366 |
-
"questions": 12,
|
| 1367 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1368 |
-
"brier": 0.4233092995278946,
|
| 1369 |
-
"cross_entropy": 1.4220052117820898
|
| 1370 |
-
},
|
| 1371 |
-
"judge": {
|
| 1372 |
-
"questions": 8,
|
| 1373 |
-
"modal_accuracy": 0.75,
|
| 1374 |
-
"brier": 0.4294109009179358,
|
| 1375 |
-
"cross_entropy": 1.1813213005452772
|
| 1376 |
-
},
|
| 1377 |
-
"know": {
|
| 1378 |
-
"questions": 10,
|
| 1379 |
-
"modal_accuracy": 0.8,
|
| 1380 |
-
"brier": 0.22602505333734763,
|
| 1381 |
-
"cross_entropy": 0.39225738115288933
|
| 1382 |
-
},
|
| 1383 |
-
"ret": {
|
| 1384 |
-
"questions": 10,
|
| 1385 |
-
"modal_accuracy": 0.4,
|
| 1386 |
-
"brier": 0.9045065751061194,
|
| 1387 |
-
"cross_entropy": 1.655086462390568
|
| 1388 |
-
},
|
| 1389 |
-
"route": {
|
| 1390 |
-
"questions": 9,
|
| 1391 |
-
"modal_accuracy": 0.4444444444444444,
|
| 1392 |
-
"brier": 0.6393115918944736,
|
| 1393 |
-
"cross_entropy": 1.1139742544661488
|
| 1394 |
-
},
|
| 1395 |
-
"safety": {
|
| 1396 |
-
"questions": 12,
|
| 1397 |
-
"modal_accuracy": 0.9166666666666666,
|
| 1398 |
-
"brier": 0.17156815175055062,
|
| 1399 |
-
"cross_entropy": 0.4272056920862004
|
| 1400 |
-
},
|
| 1401 |
-
"step": {
|
| 1402 |
-
"questions": 8,
|
| 1403 |
-
"modal_accuracy": 0.625,
|
| 1404 |
-
"brier": 0.710965126411655,
|
| 1405 |
-
"cross_entropy": 1.9533480392831426
|
| 1406 |
-
},
|
| 1407 |
-
"struct": {
|
| 1408 |
-
"questions": 2,
|
| 1409 |
-
"modal_accuracy": 0.5,
|
| 1410 |
-
"brier": 0.8092234753736499,
|
| 1411 |
-
"cross_entropy": 1.1517337483418988
|
| 1412 |
-
}
|
| 1413 |
}
|
| 1414 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1415 |
},
|
| 1416 |
-
"
|
| 1417 |
-
"
|
| 1418 |
-
"questions":
|
| 1419 |
-
"
|
| 1420 |
-
"brier": 0.
|
| 1421 |
-
"cross_entropy":
|
| 1422 |
-
"by_kind": {
|
| 1423 |
-
"choice": {
|
| 1424 |
-
"questions": 665,
|
| 1425 |
-
"modal_accuracy": 0.7654135338345864,
|
| 1426 |
-
"brier": 0.3189465668614991,
|
| 1427 |
-
"cross_entropy": 1.0534065611001757
|
| 1428 |
-
},
|
| 1429 |
-
"noul": {
|
| 1430 |
-
"questions": 144,
|
| 1431 |
-
"modal_accuracy": 0.8958333333333334,
|
| 1432 |
-
"brier": 0.16471210037099535,
|
| 1433 |
-
"cross_entropy": 0.39482479972079665
|
| 1434 |
-
},
|
| 1435 |
-
"score": {
|
| 1436 |
-
"questions": 215,
|
| 1437 |
-
"modal_accuracy": 0.6604651162790698,
|
| 1438 |
-
"brier": 0.22656192147115203,
|
| 1439 |
-
"cross_entropy": 1.2057654334760237
|
| 1440 |
-
}
|
| 1441 |
-
},
|
| 1442 |
-
"by_source": {
|
| 1443 |
-
"alfworld": {
|
| 1444 |
-
"questions": 128,
|
| 1445 |
-
"modal_accuracy": 0.8671875,
|
| 1446 |
-
"brier": 0.1790825849971539,
|
| 1447 |
-
"cross_entropy": 0.6080971490404414
|
| 1448 |
-
},
|
| 1449 |
-
"arithmetic": {
|
| 1450 |
-
"questions": 24,
|
| 1451 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1452 |
-
"brier": 0.5263100827359132,
|
| 1453 |
-
"cross_entropy": 1.403012675529207
|
| 1454 |
-
},
|
| 1455 |
-
"babyai": {
|
| 1456 |
-
"questions": 58,
|
| 1457 |
-
"modal_accuracy": 0.5517241379310345,
|
| 1458 |
-
"brier": 0.684833327087956,
|
| 1459 |
-
"cross_entropy": 1.5818612127101548
|
| 1460 |
-
},
|
| 1461 |
-
"community-diversity-v2/browser_tools/idempotent_retry": {
|
| 1462 |
-
"questions": 2,
|
| 1463 |
-
"modal_accuracy": 1.0,
|
| 1464 |
-
"brier": 8.422208826818114e-06,
|
| 1465 |
-
"cross_entropy": 0.0016334122276107624
|
| 1466 |
-
},
|
| 1467 |
-
"community-diversity-v2/contracts/governing_law_venue": {
|
| 1468 |
-
"questions": 5,
|
| 1469 |
-
"modal_accuracy": 0.8,
|
| 1470 |
-
"brier": 0.39072800981047207,
|
| 1471 |
-
"cross_entropy": 0.8926558622999824
|
| 1472 |
-
},
|
| 1473 |
-
"community-diversity-v2/games/racing_overtake": {
|
| 1474 |
-
"questions": 7,
|
| 1475 |
-
"modal_accuracy": 0.8571428571428571,
|
| 1476 |
-
"brier": 0.28439123582360026,
|
| 1477 |
-
"cross_entropy": 0.7928804085918381
|
| 1478 |
-
},
|
| 1479 |
-
"community-diversity-v2/ir/access_schedule": {
|
| 1480 |
-
"questions": 12,
|
| 1481 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1482 |
-
"brier": 0.6286826358235,
|
| 1483 |
-
"cross_entropy": 2.0004531092707
|
| 1484 |
-
},
|
| 1485 |
-
"community-diversity-v2/rag/policy_exception": {
|
| 1486 |
-
"questions": 10,
|
| 1487 |
-
"modal_accuracy": 0.7,
|
| 1488 |
-
"brier": 0.44538602593083176,
|
| 1489 |
-
"cross_entropy": 1.117689014223367
|
| 1490 |
-
},
|
| 1491 |
-
"community-diversity-v2/rubric_judge/trace_consistency": {
|
| 1492 |
-
"questions": 4,
|
| 1493 |
-
"modal_accuracy": 0.25,
|
| 1494 |
-
"brier": 1.4342464951649092,
|
| 1495 |
-
"cross_entropy": 3.381933827755728
|
| 1496 |
-
},
|
| 1497 |
-
"connect_four": {
|
| 1498 |
-
"questions": 56,
|
| 1499 |
-
"modal_accuracy": 0.5535714285714286,
|
| 1500 |
-
"brier": 0.2622696208064482,
|
| 1501 |
-
"cross_entropy": 1.8219529101323455
|
| 1502 |
-
},
|
| 1503 |
-
"customer-control-v1": {
|
| 1504 |
-
"questions": 7,
|
| 1505 |
-
"modal_accuracy": 1.0,
|
| 1506 |
-
"brier": 2.417104630113183e-06,
|
| 1507 |
-
"cross_entropy": 0.0005900334430327367
|
| 1508 |
-
},
|
| 1509 |
-
"devign": {
|
| 1510 |
-
"questions": 2,
|
| 1511 |
-
"modal_accuracy": 1.0,
|
| 1512 |
-
"brier": 0.43780762834608883,
|
| 1513 |
-
"cross_entropy": 0.6306076256145883
|
| 1514 |
-
},
|
| 1515 |
-
"gsm8k": {
|
| 1516 |
-
"questions": 4,
|
| 1517 |
-
"modal_accuracy": 1.0,
|
| 1518 |
-
"brier": 0.0003744313065983422,
|
| 1519 |
-
"cross_entropy": 0.011988960485765076
|
| 1520 |
-
},
|
| 1521 |
-
"helpsteer2": {
|
| 1522 |
-
"questions": 128,
|
| 1523 |
-
"modal_accuracy": 0.5859375,
|
| 1524 |
-
"brier": 0.20697473621325052,
|
| 1525 |
-
"cross_entropy": 1.5034032587243362
|
| 1526 |
-
},
|
| 1527 |
-
"planning": {
|
| 1528 |
-
"questions": 192,
|
| 1529 |
-
"modal_accuracy": 0.9635416666666666,
|
| 1530 |
-
"brier": 0.11802009440200961,
|
| 1531 |
-
"cross_entropy": 0.6579051066101222
|
| 1532 |
-
},
|
| 1533 |
-
"race": {
|
| 1534 |
-
"questions": 4,
|
| 1535 |
-
"modal_accuracy": 0.5,
|
| 1536 |
-
"brier": 0.7315329429054658,
|
| 1537 |
-
"cross_entropy": 2.0647249708287245
|
| 1538 |
-
},
|
| 1539 |
-
"ragtruth": {
|
| 1540 |
-
"questions": 6,
|
| 1541 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1542 |
-
"brier": 0.609860991064251,
|
| 1543 |
-
"cross_entropy": 0.9741778978250341
|
| 1544 |
-
},
|
| 1545 |
-
"scienceworld": {
|
| 1546 |
-
"questions": 128,
|
| 1547 |
-
"modal_accuracy": 0.703125,
|
| 1548 |
-
"brier": 0.44470820194957994,
|
| 1549 |
-
"cross_entropy": 1.2102900099661185
|
| 1550 |
-
},
|
| 1551 |
-
"scone": {
|
| 1552 |
-
"questions": 192,
|
| 1553 |
-
"modal_accuracy": 0.796875,
|
| 1554 |
-
"brier": 0.29338767561802165,
|
| 1555 |
-
"cross_entropy": 0.6140028830315581
|
| 1556 |
-
},
|
| 1557 |
-
"toolace": {
|
| 1558 |
-
"questions": 2,
|
| 1559 |
-
"modal_accuracy": 1.0,
|
| 1560 |
-
"brier": 2.1804417353162663e-07,
|
| 1561 |
-
"cross_entropy": 0.0003234946827488933
|
| 1562 |
-
},
|
| 1563 |
-
"yuri_v3": {
|
| 1564 |
-
"questions": 53,
|
| 1565 |
-
"modal_accuracy": 0.7169811320754716,
|
| 1566 |
-
"brier": 0.056977519481840835,
|
| 1567 |
-
"cross_entropy": 0.860752644211149
|
| 1568 |
-
}
|
| 1569 |
-
}
|
| 1570 |
-
},
|
| 1571 |
-
"new_test": {
|
| 1572 |
-
"questions": 512,
|
| 1573 |
-
"modal_accuracy": 0.755859375,
|
| 1574 |
-
"brier": 0.2705710538872297,
|
| 1575 |
-
"cross_entropy": 1.0354930422480668,
|
| 1576 |
-
"by_kind": {
|
| 1577 |
-
"choice": {
|
| 1578 |
-
"questions": 241,
|
| 1579 |
-
"modal_accuracy": 0.8215767634854771,
|
| 1580 |
-
"brier": 0.30593746240810665,
|
| 1581 |
-
"cross_entropy": 1.0288481446263855
|
| 1582 |
-
},
|
| 1583 |
-
"noul": {
|
| 1584 |
-
"questions": 94,
|
| 1585 |
-
"modal_accuracy": 0.8936170212765957,
|
| 1586 |
-
"brier": 0.1853453372075174,
|
| 1587 |
-
"cross_entropy": 0.4718470608079094
|
| 1588 |
-
},
|
| 1589 |
-
"score": {
|
| 1590 |
-
"questions": 177,
|
| 1591 |
-
"modal_accuracy": 0.5932203389830508,
|
| 1592 |
-
"brier": 0.26767790651074164,
|
| 1593 |
-
"cross_entropy": 1.34387802858818
|
| 1594 |
-
}
|
| 1595 |
-
},
|
| 1596 |
-
"by_source": {
|
| 1597 |
-
"helpsteer2": {
|
| 1598 |
-
"questions": 128,
|
| 1599 |
-
"modal_accuracy": 0.5078125,
|
| 1600 |
-
"brier": 0.22898458428820745,
|
| 1601 |
-
"cross_entropy": 1.5346420909767997
|
| 1602 |
-
},
|
| 1603 |
-
"planning": {
|
| 1604 |
-
"questions": 192,
|
| 1605 |
-
"modal_accuracy": 0.921875,
|
| 1606 |
-
"brier": 0.1909474730913934,
|
| 1607 |
-
"cross_entropy": 0.8915435249464682
|
| 1608 |
-
},
|
| 1609 |
-
"scone": {
|
| 1610 |
-
"questions": 192,
|
| 1611 |
-
"modal_accuracy": 0.7552083333333334,
|
| 1612 |
-
"brier": 0.3779189477490808,
|
| 1613 |
-
"cross_entropy": 0.8466765270638437
|
| 1614 |
-
}
|
| 1615 |
-
}
|
| 1616 |
},
|
| 1617 |
-
"
|
| 1618 |
-
"questions":
|
| 1619 |
-
"
|
| 1620 |
-
"brier": 0.
|
| 1621 |
-
"cross_entropy": 0.
|
| 1622 |
-
"by_kind": {
|
| 1623 |
-
"choice": {
|
| 1624 |
-
"questions": 1151,
|
| 1625 |
-
"modal_accuracy": 0.7193744569939183,
|
| 1626 |
-
"brier": 0.36420584491384933,
|
| 1627 |
-
"cross_entropy": 1.0683460554623134
|
| 1628 |
-
},
|
| 1629 |
-
"noul": {
|
| 1630 |
-
"questions": 649,
|
| 1631 |
-
"modal_accuracy": 0.9090909090909091,
|
| 1632 |
-
"brier": 0.12795487547550322,
|
| 1633 |
-
"cross_entropy": 0.3201729560342784
|
| 1634 |
-
},
|
| 1635 |
-
"score": {
|
| 1636 |
-
"questions": 248,
|
| 1637 |
-
"modal_accuracy": 0.8346774193548387,
|
| 1638 |
-
"brier": 0.1562044216890414,
|
| 1639 |
-
"cross_entropy": 0.6832748093410428
|
| 1640 |
-
}
|
| 1641 |
-
},
|
| 1642 |
-
"by_source": {
|
| 1643 |
-
"alfworld": {
|
| 1644 |
-
"questions": 128,
|
| 1645 |
-
"modal_accuracy": 0.8671875,
|
| 1646 |
-
"brier": 0.1662207666853679,
|
| 1647 |
-
"cross_entropy": 0.4678064926902018
|
| 1648 |
-
},
|
| 1649 |
-
"arithmetic": {
|
| 1650 |
-
"questions": 30,
|
| 1651 |
-
"modal_accuracy": 0.7333333333333333,
|
| 1652 |
-
"brier": 0.5001721787123533,
|
| 1653 |
-
"cross_entropy": 1.2885683809129926
|
| 1654 |
-
},
|
| 1655 |
-
"babyai": {
|
| 1656 |
-
"questions": 128,
|
| 1657 |
-
"modal_accuracy": 0.484375,
|
| 1658 |
-
"brier": 0.7067526312424577,
|
| 1659 |
-
"cross_entropy": 1.6676958019283898
|
| 1660 |
-
},
|
| 1661 |
-
"community-diversity-v2/browser_tools/goal_completion": {
|
| 1662 |
-
"questions": 24,
|
| 1663 |
-
"modal_accuracy": 0.9583333333333334,
|
| 1664 |
-
"brier": 0.06921012297338076,
|
| 1665 |
-
"cross_entropy": 0.10122484092336533
|
| 1666 |
-
},
|
| 1667 |
-
"community-diversity-v2/contracts/exclusivity_exception": {
|
| 1668 |
-
"questions": 23,
|
| 1669 |
-
"modal_accuracy": 0.8695652173913043,
|
| 1670 |
-
"brier": 0.24976225264440965,
|
| 1671 |
-
"cross_entropy": 0.5909170782174602
|
| 1672 |
-
},
|
| 1673 |
-
"community-diversity-v2/games/pit_stop": {
|
| 1674 |
-
"questions": 13,
|
| 1675 |
-
"modal_accuracy": 0.7692307692307693,
|
| 1676 |
-
"brier": 0.4785595907160497,
|
| 1677 |
-
"cross_entropy": 1.2971969110502473
|
| 1678 |
-
},
|
| 1679 |
-
"community-diversity-v2/ir/warranty_transfer": {
|
| 1680 |
-
"questions": 15,
|
| 1681 |
-
"modal_accuracy": 0.8666666666666667,
|
| 1682 |
-
"brier": 0.14068735459076726,
|
| 1683 |
-
"cross_entropy": 0.17808536247316048
|
| 1684 |
-
},
|
| 1685 |
-
"community-diversity-v2/rag/independent_sources": {
|
| 1686 |
-
"questions": 21,
|
| 1687 |
-
"modal_accuracy": 0.47619047619047616,
|
| 1688 |
-
"brier": 0.9915299974666058,
|
| 1689 |
-
"cross_entropy": 2.310007596806081
|
| 1690 |
-
},
|
| 1691 |
-
"community-diversity-v2/rubric_judge/balanced_tradeoff": {
|
| 1692 |
-
"questions": 12,
|
| 1693 |
-
"modal_accuracy": 0.75,
|
| 1694 |
-
"brier": 0.4814089394948157,
|
| 1695 |
-
"cross_entropy": 1.102602184903888
|
| 1696 |
-
},
|
| 1697 |
-
"community-diversity-v2/shell_history/incremental_copy": {
|
| 1698 |
-
"questions": 17,
|
| 1699 |
-
"modal_accuracy": 0.8823529411764706,
|
| 1700 |
-
"brier": 0.20441354684881466,
|
| 1701 |
-
"cross_entropy": 0.5009642986702474
|
| 1702 |
-
},
|
| 1703 |
-
"community-diversity-v2/support/outage_compensation": {
|
| 1704 |
-
"questions": 3,
|
| 1705 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1706 |
-
"brier": 0.4884574890213706,
|
| 1707 |
-
"cross_entropy": 1.3886592311282289
|
| 1708 |
-
},
|
| 1709 |
-
"connect_four": {
|
| 1710 |
-
"questions": 128,
|
| 1711 |
-
"modal_accuracy": 0.59375,
|
| 1712 |
-
"brier": 0.2561712647462938,
|
| 1713 |
-
"cross_entropy": 1.802544508268942
|
| 1714 |
-
},
|
| 1715 |
-
"customer-control-v1": {
|
| 1716 |
-
"questions": 45,
|
| 1717 |
-
"modal_accuracy": 1.0,
|
| 1718 |
-
"brier": 0.00019936847019355491,
|
| 1719 |
-
"cross_entropy": 0.015971454914726523
|
| 1720 |
-
},
|
| 1721 |
-
"devign": {
|
| 1722 |
-
"questions": 46,
|
| 1723 |
-
"modal_accuracy": 0.5217391304347826,
|
| 1724 |
-
"brier": 0.5225763998633566,
|
| 1725 |
-
"cross_entropy": 0.722933623300632
|
| 1726 |
-
},
|
| 1727 |
-
"entity_belief_tracking": {
|
| 1728 |
-
"questions": 19,
|
| 1729 |
-
"modal_accuracy": 0.8421052631578947,
|
| 1730 |
-
"brier": 0.259415578060587,
|
| 1731 |
-
"cross_entropy": 0.6433967485445069
|
| 1732 |
-
},
|
| 1733 |
-
"event_state_reconstruction": {
|
| 1734 |
-
"questions": 30,
|
| 1735 |
-
"modal_accuracy": 1.0,
|
| 1736 |
-
"brier": 0.015897166327449284,
|
| 1737 |
-
"cross_entropy": 0.02589146516518676
|
| 1738 |
-
},
|
| 1739 |
-
"event_triggered_execution": {
|
| 1740 |
-
"questions": 18,
|
| 1741 |
-
"modal_accuracy": 0.8888888888888888,
|
| 1742 |
-
"brier": 0.16562186759207273,
|
| 1743 |
-
"cross_entropy": 0.2757516689442136
|
| 1744 |
-
},
|
| 1745 |
-
"evidence_sufficiency": {
|
| 1746 |
-
"questions": 13,
|
| 1747 |
-
"modal_accuracy": 0.6923076923076923,
|
| 1748 |
-
"brier": 0.5880021120988826,
|
| 1749 |
-
"cross_entropy": 1.4889127617030355
|
| 1750 |
-
},
|
| 1751 |
-
"goal_directed_planning": {
|
| 1752 |
-
"questions": 19,
|
| 1753 |
-
"modal_accuracy": 0.8421052631578947,
|
| 1754 |
-
"brier": 0.2509895082794504,
|
| 1755 |
-
"cross_entropy": 0.4540842796343451
|
| 1756 |
-
},
|
| 1757 |
-
"graph_navigation": {
|
| 1758 |
-
"questions": 6,
|
| 1759 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1760 |
-
"brier": 0.44283093090962034,
|
| 1761 |
-
"cross_entropy": 1.3194420415465578
|
| 1762 |
-
},
|
| 1763 |
-
"gsm8k": {
|
| 1764 |
-
"questions": 106,
|
| 1765 |
-
"modal_accuracy": 0.7075471698113207,
|
| 1766 |
-
"brier": 0.43472841661725453,
|
| 1767 |
-
"cross_entropy": 0.8890682888838121
|
| 1768 |
-
},
|
| 1769 |
-
"instruction_execution": {
|
| 1770 |
-
"questions": 12,
|
| 1771 |
-
"modal_accuracy": 0.9166666666666666,
|
| 1772 |
-
"brier": 0.16555489236402793,
|
| 1773 |
-
"cross_entropy": 0.49206138807591987
|
| 1774 |
-
},
|
| 1775 |
-
"multi_view_adjudication": {
|
| 1776 |
-
"questions": 20,
|
| 1777 |
-
"modal_accuracy": 1.0,
|
| 1778 |
-
"brier": 5.2326944475445606e-05,
|
| 1779 |
-
"cross_entropy": 0.0019525076494628021
|
| 1780 |
-
},
|
| 1781 |
-
"needle_retrieval": {
|
| 1782 |
-
"questions": 5,
|
| 1783 |
-
"modal_accuracy": 1.0,
|
| 1784 |
-
"brier": 3.199473123353177e-08,
|
| 1785 |
-
"cross_entropy": 8.974482553742138e-05
|
| 1786 |
-
},
|
| 1787 |
-
"partial_observation_calibration": {
|
| 1788 |
-
"questions": 5,
|
| 1789 |
-
"modal_accuracy": 0.8,
|
| 1790 |
-
"brier": 0.02661607292190903,
|
| 1791 |
-
"cross_entropy": 0.5399351074049796
|
| 1792 |
-
},
|
| 1793 |
-
"policy_applicability": {
|
| 1794 |
-
"questions": 17,
|
| 1795 |
-
"modal_accuracy": 1.0,
|
| 1796 |
-
"brier": 6.106492112719508e-05,
|
| 1797 |
-
"cross_entropy": 0.003675418740395231
|
| 1798 |
-
},
|
| 1799 |
-
"policy_under_uncertainty": {
|
| 1800 |
-
"questions": 8,
|
| 1801 |
-
"modal_accuracy": 0.875,
|
| 1802 |
-
"brier": 0.10177113114450002,
|
| 1803 |
-
"cross_entropy": 0.560772453887988
|
| 1804 |
-
},
|
| 1805 |
-
"race": {
|
| 1806 |
-
"questions": 104,
|
| 1807 |
-
"modal_accuracy": 0.9134615384615384,
|
| 1808 |
-
"brier": 0.16990537457587326,
|
| 1809 |
-
"cross_entropy": 0.3822661135973287
|
| 1810 |
-
},
|
| 1811 |
-
"ragtruth": {
|
| 1812 |
-
"questions": 128,
|
| 1813 |
-
"modal_accuracy": 0.8515625,
|
| 1814 |
-
"brier": 0.2317892915096781,
|
| 1815 |
-
"cross_entropy": 0.39549112820971377
|
| 1816 |
-
},
|
| 1817 |
-
"reasoning-control-v1": {
|
| 1818 |
-
"questions": 45,
|
| 1819 |
-
"modal_accuracy": 0.9111111111111111,
|
| 1820 |
-
"brier": 0.1285355661042637,
|
| 1821 |
-
"cross_entropy": 0.30974376864438463
|
| 1822 |
-
},
|
| 1823 |
-
"record_aggregation": {
|
| 1824 |
-
"questions": 14,
|
| 1825 |
-
"modal_accuracy": 0.7857142857142857,
|
| 1826 |
-
"brier": 0.35331876706309845,
|
| 1827 |
-
"cross_entropy": 0.696535349171323
|
| 1828 |
-
},
|
| 1829 |
-
"record_eligibility": {
|
| 1830 |
-
"questions": 8,
|
| 1831 |
-
"modal_accuracy": 1.0,
|
| 1832 |
-
"brier": 0.00021253785775029812,
|
| 1833 |
-
"cross_entropy": 0.0043067743592292455
|
| 1834 |
-
},
|
| 1835 |
-
"scienceworld": {
|
| 1836 |
-
"questions": 128,
|
| 1837 |
-
"modal_accuracy": 0.5625,
|
| 1838 |
-
"brier": 0.6406304488569265,
|
| 1839 |
-
"cross_entropy": 1.8806145121011755
|
| 1840 |
-
},
|
| 1841 |
-
"snake-v1": {
|
| 1842 |
-
"questions": 30,
|
| 1843 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1844 |
-
"brier": 0.04564391684280246,
|
| 1845 |
-
"cross_entropy": 0.2431433189091872
|
| 1846 |
-
},
|
| 1847 |
-
"state_perturbation": {
|
| 1848 |
-
"questions": 11,
|
| 1849 |
-
"modal_accuracy": 1.0,
|
| 1850 |
-
"brier": 0.0002344655717187281,
|
| 1851 |
-
"cross_entropy": 0.005994309073216685
|
| 1852 |
-
},
|
| 1853 |
-
"table_lookup": {
|
| 1854 |
-
"questions": 13,
|
| 1855 |
-
"modal_accuracy": 0.9230769230769231,
|
| 1856 |
-
"brier": 0.1498702753732909,
|
| 1857 |
-
"cross_entropy": 0.3551783229099786
|
| 1858 |
-
},
|
| 1859 |
-
"taxonomy_routing": {
|
| 1860 |
-
"questions": 8,
|
| 1861 |
-
"modal_accuracy": 1.0,
|
| 1862 |
-
"brier": 0.045424091680658744,
|
| 1863 |
-
"cross_entropy": 0.07014486556652395
|
| 1864 |
-
},
|
| 1865 |
-
"tic_tac_toe-v1": {
|
| 1866 |
-
"questions": 15,
|
| 1867 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1868 |
-
"brier": 0.26386162320957124,
|
| 1869 |
-
"cross_entropy": 1.1837003824426573
|
| 1870 |
-
},
|
| 1871 |
-
"tile_platformer-v1": {
|
| 1872 |
-
"questions": 15,
|
| 1873 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1874 |
-
"brier": 0.10164899419557688,
|
| 1875 |
-
"cross_entropy": 0.220570822391115
|
| 1876 |
-
},
|
| 1877 |
-
"toolace": {
|
| 1878 |
-
"questions": 128,
|
| 1879 |
-
"modal_accuracy": 0.9921875,
|
| 1880 |
-
"brier": 0.01562046345535589,
|
| 1881 |
-
"cross_entropy": 0.04367168294967396
|
| 1882 |
-
},
|
| 1883 |
-
"trex_runner-v1": {
|
| 1884 |
-
"questions": 2,
|
| 1885 |
-
"modal_accuracy": 0.5,
|
| 1886 |
-
"brier": 0.36032773502188475,
|
| 1887 |
-
"cross_entropy": 0.5817032440936637
|
| 1888 |
-
},
|
| 1889 |
-
"vizdoom-basic-v1": {
|
| 1890 |
-
"questions": 45,
|
| 1891 |
-
"modal_accuracy": 0.9555555555555556,
|
| 1892 |
-
"brier": 0.08693939517676995,
|
| 1893 |
-
"cross_entropy": 0.20638551041670192
|
| 1894 |
-
},
|
| 1895 |
-
"wanli-decisions-v1": {
|
| 1896 |
-
"questions": 128,
|
| 1897 |
-
"modal_accuracy": 0.6875,
|
| 1898 |
-
"brier": 0.5529247926341857,
|
| 1899 |
-
"cross_entropy": 1.0802996209215674
|
| 1900 |
-
},
|
| 1901 |
-
"workflow-controls-v1/agent_trace_observability": {
|
| 1902 |
-
"questions": 15,
|
| 1903 |
-
"modal_accuracy": 0.9333333333333333,
|
| 1904 |
-
"brier": 0.13168080686035147,
|
| 1905 |
-
"cross_entropy": 0.3395590286213042
|
| 1906 |
-
},
|
| 1907 |
-
"workflow-controls-v1/customer_service": {
|
| 1908 |
-
"questions": 15,
|
| 1909 |
-
"modal_accuracy": 1.0,
|
| 1910 |
-
"brier": 7.55085699524451e-06,
|
| 1911 |
-
"cross_entropy": 0.0007668466946998275
|
| 1912 |
-
},
|
| 1913 |
-
"workflow-controls-v1/invoice_processing": {
|
| 1914 |
-
"questions": 15,
|
| 1915 |
-
"modal_accuracy": 1.0,
|
| 1916 |
-
"brier": 1.1169422484519878e-06,
|
| 1917 |
-
"cross_entropy": 0.0002747598457579768
|
| 1918 |
-
},
|
| 1919 |
-
"workflow-controls-v1/security_incidents": {
|
| 1920 |
-
"questions": 14,
|
| 1921 |
-
"modal_accuracy": 1.0,
|
| 1922 |
-
"brier": 2.4204779837314167e-09,
|
| 1923 |
-
"cross_entropy": 2.2446460647594408e-05
|
| 1924 |
-
},
|
| 1925 |
-
"yuri_v3": {
|
| 1926 |
-
"questions": 256,
|
| 1927 |
-
"modal_accuracy": 0.83203125,
|
| 1928 |
-
"brier": 0.03526590762889966,
|
| 1929 |
-
"cross_entropy": 0.8245288048118414
|
| 1930 |
-
}
|
| 1931 |
-
}
|
| 1932 |
},
|
| 1933 |
-
"
|
| 1934 |
-
"questions":
|
| 1935 |
-
"
|
| 1936 |
-
"brier": 0.
|
| 1937 |
-
"cross_entropy": 1.
|
| 1938 |
-
"by_kind": {
|
| 1939 |
-
"choice": {
|
| 1940 |
-
"questions": 33,
|
| 1941 |
-
"modal_accuracy": 0.6363636363636364,
|
| 1942 |
-
"brier": 0.5172571933930417,
|
| 1943 |
-
"cross_entropy": 1.2048737574594803
|
| 1944 |
-
},
|
| 1945 |
-
"noul": {
|
| 1946 |
-
"questions": 41,
|
| 1947 |
-
"modal_accuracy": 0.7317073170731707,
|
| 1948 |
-
"brier": 0.3731759473027374,
|
| 1949 |
-
"cross_entropy": 0.7052377430670229
|
| 1950 |
-
},
|
| 1951 |
-
"score": {
|
| 1952 |
-
"questions": 15,
|
| 1953 |
-
"modal_accuracy": 0.4666666666666667,
|
| 1954 |
-
"brier": 0.6849531954881848,
|
| 1955 |
-
"cross_entropy": 1.381039887344188
|
| 1956 |
-
}
|
| 1957 |
-
},
|
| 1958 |
-
"by_source": {
|
| 1959 |
-
"basic_cls": {
|
| 1960 |
-
"questions": 12,
|
| 1961 |
-
"modal_accuracy": 0.6666666666666666,
|
| 1962 |
-
"brier": 0.4038371458185728,
|
| 1963 |
-
"cross_entropy": 0.7355992645221435
|
| 1964 |
-
},
|
| 1965 |
-
"code": {
|
| 1966 |
-
"questions": 6,
|
| 1967 |
-
"modal_accuracy": 0.3333333333333333,
|
| 1968 |
-
"brier": 0.8287228425681598,
|
| 1969 |
-
"cross_entropy": 1.262918501594028
|
| 1970 |
-
},
|
| 1971 |
-
"dialog": {
|
| 1972 |
-
"questions": 12,
|
| 1973 |
-
"modal_accuracy": 0.5833333333333334,
|
| 1974 |
-
"brier": 0.30448197088852874,
|
| 1975 |
-
"cross_entropy": 1.1186203509870534
|
| 1976 |
-
},
|
| 1977 |
-
"judge": {
|
| 1978 |
-
"questions": 8,
|
| 1979 |
-
"modal_accuracy": 0.875,
|
| 1980 |
-
"brier": 0.3292730875017608,
|
| 1981 |
-
"cross_entropy": 0.7769559856794094
|
| 1982 |
-
},
|
| 1983 |
-
"know": {
|
| 1984 |
-
"questions": 10,
|
| 1985 |
-
"modal_accuracy": 0.8,
|
| 1986 |
-
"brier": 0.26539280354851125,
|
| 1987 |
-
"cross_entropy": 0.4839723017070307
|
| 1988 |
-
},
|
| 1989 |
-
"ret": {
|
| 1990 |
-
"questions": 10,
|
| 1991 |
-
"modal_accuracy": 0.5,
|
| 1992 |
-
"brier": 0.814367705138209,
|
| 1993 |
-
"cross_entropy": 1.6245878852220568
|
| 1994 |
-
},
|
| 1995 |
-
"route": {
|
| 1996 |
-
"questions": 9,
|
| 1997 |
-
"modal_accuracy": 0.4444444444444444,
|
| 1998 |
-
"brier": 0.7471958010258476,
|
| 1999 |
-
"cross_entropy": 1.3369414656592875
|
| 2000 |
-
},
|
| 2001 |
-
"safety": {
|
| 2002 |
-
"questions": 12,
|
| 2003 |
-
"modal_accuracy": 0.9166666666666666,
|
| 2004 |
-
"brier": 0.1672822055997213,
|
| 2005 |
-
"cross_entropy": 0.43712494222128856
|
| 2006 |
-
},
|
| 2007 |
-
"step": {
|
| 2008 |
-
"questions": 8,
|
| 2009 |
-
"modal_accuracy": 0.625,
|
| 2010 |
-
"brier": 0.6510657904776266,
|
| 2011 |
-
"cross_entropy": 1.5015904290656892
|
| 2012 |
-
},
|
| 2013 |
-
"struct": {
|
| 2014 |
-
"questions": 2,
|
| 2015 |
-
"modal_accuracy": 0.5,
|
| 2016 |
-
"brier": 0.8996839553398099,
|
| 2017 |
-
"cross_entropy": 1.4855438457791856
|
| 2018 |
-
}
|
| 2019 |
-
}
|
| 2020 |
}
|
| 2021 |
}
|
| 2022 |
},
|
| 2023 |
-
"scope":
|
| 2024 |
-
"new_validation": "1024 selection questions: 512 new and 512 previous; all four candidate checkpoints scored",
|
| 2025 |
-
"new_test": "512 reserved new questions; first scored after checkpoint selection",
|
| 2026 |
-
"test": "2048 earlier questions; reused to check retention of earlier abilities",
|
| 2027 |
-
"transfer": "89 general questions; small reused development check",
|
| 2028 |
-
"parent": "Previous actions-48k adapter, re-exported to byte-identical NVFP4 and rescored with the same Eider binary",
|
| 2029 |
-
"calibration": "All scores are uncalibrated. No fitted profile for the new weights.",
|
| 2030 |
-
"bf16": "Matching merged BF16 weights are included but were not independently scored in this native evaluation.",
|
| 2031 |
-
"base_model": "The unfine-tuned NVIDIA model was not rescored in this evaluation."
|
| 2032 |
-
}
|
| 2033 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"test": {
|
| 3 |
+
"schema": "dory-base-vs-typed-decisions/v1",
|
| 4 |
+
"base_model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16",
|
| 5 |
+
"base_model_revision": "b366b87f762ea334ba3eb55912031baba1515810",
|
| 6 |
+
"format": "NVFP4",
|
| 7 |
+
"quantization_recipe": "NVIDIA Model Optimizer local_hessian",
|
| 8 |
+
"calibration_examples": 256,
|
| 9 |
+
"loops": 2,
|
| 10 |
+
"calibrated": false,
|
| 11 |
+
"prompt": "Identical typed-decision prompts and allowed-answer scoring; no generated reasoning.",
|
| 12 |
+
"binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 13 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 14 |
+
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 15 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 16 |
+
"base": {
|
| 17 |
+
"questions": 512,
|
| 18 |
+
"modal_accuracy": 0.376953125,
|
| 19 |
+
"brier": 0.6512325676171807,
|
| 20 |
+
"cross_entropy": 1.9877935454851494,
|
| 21 |
+
"mean_request_ms": 81.26816220556066,
|
| 22 |
+
"median_request_ms": 72.5804881658405,
|
| 23 |
+
"by_kind": {
|
| 24 |
+
"choice": {
|
| 25 |
+
"questions": 241,
|
| 26 |
+
"modal_accuracy": 0.2863070539419087,
|
| 27 |
+
"brier": 0.742723819745487,
|
| 28 |
+
"cross_entropy": 2.2877653477997058,
|
| 29 |
+
"mean_request_ms": 75.63938830179235,
|
| 30 |
+
"median_request_ms": 72.81611207872629
|
| 31 |
+
},
|
| 32 |
+
"noul": {
|
| 33 |
+
"questions": 94,
|
| 34 |
+
"modal_accuracy": 0.5,
|
| 35 |
+
"brier": 0.670176294758546,
|
| 36 |
+
"cross_entropy": 0.9175096541022592,
|
| 37 |
+
"mean_request_ms": 68.15949506423574,
|
| 38 |
+
"median_request_ms": 68.27253755182028
|
| 39 |
+
},
|
| 40 |
+
"score": {
|
| 41 |
+
"questions": 177,
|
| 42 |
+
"modal_accuracy": 0.4350282485875706,
|
| 43 |
+
"brier": 0.5165992223391576,
|
| 44 |
+
"cross_entropy": 2.1477567174183902,
|
| 45 |
+
"mean_request_ms": 95.89386402529344,
|
| 46 |
+
"median_request_ms": 77.44529517367482
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
}
|
| 48 |
},
|
| 49 |
+
"by_source": {
|
| 50 |
+
"helpsteer2": {
|
| 51 |
+
"questions": 128,
|
| 52 |
+
"modal_accuracy": 0.484375,
|
| 53 |
+
"brier": 0.31624743248922543,
|
| 54 |
+
"cross_entropy": 1.797596837336767,
|
| 55 |
+
"mean_request_ms": 107.6381878774555,
|
| 56 |
+
"median_request_ms": 96.75377863459289
|
| 57 |
+
},
|
| 58 |
+
"planning": {
|
| 59 |
+
"questions": 192,
|
| 60 |
+
"modal_accuracy": 0.3697916666666667,
|
| 61 |
+
"brier": 0.6013039965397795,
|
| 62 |
+
"cross_entropy": 1.8274480710869951,
|
| 63 |
+
"mean_request_ms": 75.0043233759546,
|
| 64 |
+
"median_request_ms": 73.60281050205231
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
},
|
| 66 |
+
"scone": {
|
| 67 |
+
"questions": 192,
|
| 68 |
+
"modal_accuracy": 0.3125,
|
| 69 |
+
"brier": 0.9244845621132188,
|
| 70 |
+
"cross_entropy": 2.274936825315559,
|
| 71 |
+
"mean_request_ms": 69.95198392057016,
|
| 72 |
+
"median_request_ms": 69.4849812425673
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
}
|
| 74 |
}
|
| 75 |
},
|
| 76 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 77 |
+
"fine_tuned": {
|
| 78 |
+
"questions": 512,
|
| 79 |
+
"modal_accuracy": 0.771484375,
|
| 80 |
+
"brier": 0.26463106057883456,
|
| 81 |
+
"cross_entropy": 1.075109941121327,
|
| 82 |
+
"mean_request_ms": 80.6613628164996,
|
| 83 |
+
"median_request_ms": 71.86161540448666,
|
| 84 |
+
"by_kind": {
|
| 85 |
+
"choice": {
|
| 86 |
+
"questions": 241,
|
| 87 |
+
"modal_accuracy": 0.8215767634854771,
|
| 88 |
+
"brier": 0.34044989428658823,
|
| 89 |
+
"cross_entropy": 1.1801933058267577,
|
| 90 |
+
"mean_request_ms": 75.06443578865827,
|
| 91 |
+
"median_request_ms": 72.3452023230493
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
},
|
| 93 |
+
"noul": {
|
| 94 |
+
"questions": 94,
|
| 95 |
+
"modal_accuracy": 0.9574468085106383,
|
| 96 |
+
"brier": 0.09152335422286362,
|
| 97 |
+
"cross_entropy": 0.2881069286288201,
|
| 98 |
+
"mean_request_ms": 67.15274397581022,
|
| 99 |
+
"median_request_ms": 67.46977800503373
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
},
|
| 101 |
+
"score": {
|
| 102 |
+
"questions": 177,
|
| 103 |
+
"modal_accuracy": 0.6045197740112994,
|
| 104 |
+
"brier": 0.2533304135386799,
|
| 105 |
+
"cross_entropy": 1.3499867336653204,
|
| 106 |
+
"mean_request_ms": 95.45610623307905,
|
| 107 |
+
"median_request_ms": 77.96838786453009
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
}
|
| 109 |
},
|
| 110 |
+
"by_source": {
|
| 111 |
+
"helpsteer2": {
|
| 112 |
+
"questions": 128,
|
| 113 |
+
"modal_accuracy": 0.5078125,
|
| 114 |
+
"brier": 0.22775570941129228,
|
| 115 |
+
"cross_entropy": 1.5323848707632575,
|
| 116 |
+
"mean_request_ms": 107.58943439577706,
|
| 117 |
+
"median_request_ms": 97.03774400986731
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
},
|
| 119 |
+
"planning": {
|
| 120 |
+
"questions": 192,
|
| 121 |
+
"modal_accuracy": 0.9635416666666666,
|
| 122 |
+
"brier": 0.155754142490034,
|
| 123 |
+
"cross_entropy": 0.834348251030471,
|
| 124 |
+
"mean_request_ms": 74.49704928634067,
|
| 125 |
+
"median_request_ms": 73.17394809797406
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
},
|
| 127 |
+
"scone": {
|
| 128 |
+
"questions": 192,
|
| 129 |
+
"modal_accuracy": 0.7552083333333334,
|
| 130 |
+
"brier": 0.3980915461126633,
|
| 131 |
+
"cross_entropy": 1.0110216781175627,
|
| 132 |
+
"mean_request_ms": 68.87362862714023,
|
| 133 |
+
"median_request_ms": 68.2264359202236
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
}
|
| 135 |
}
|
| 136 |
+
}
|
| 137 |
+
},
|
| 138 |
+
"validation": {
|
| 139 |
+
"questions": 1024,
|
| 140 |
+
"split": "validation",
|
| 141 |
+
"selection_use": "Quantization recipe comparison; not an independent test",
|
| 142 |
+
"loops": 2,
|
| 143 |
+
"probability_calibration": false,
|
| 144 |
+
"calibration_examples": 256,
|
| 145 |
+
"calibration_source": "Training examples disjoint from validation by ID and source group",
|
| 146 |
+
"weights_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 147 |
+
"metrics": {
|
| 148 |
+
"questions": 1024,
|
| 149 |
+
"correct": 0.78515625,
|
| 150 |
+
"brier": 0.256938595885194,
|
| 151 |
+
"cross_entropy": 0.9667092188545374
|
| 152 |
},
|
| 153 |
+
"by_kind": {
|
| 154 |
+
"choice": {
|
| 155 |
+
"questions": 665,
|
| 156 |
+
"correct": 0.7819548872180451,
|
| 157 |
+
"brier": 0.29931089665593197,
|
| 158 |
+
"cross_entropy": 1.021091694949712
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
},
|
| 160 |
+
"noul": {
|
| 161 |
+
"questions": 144,
|
| 162 |
+
"correct": 0.9097222222222222,
|
| 163 |
+
"brier": 0.146691427234091,
|
| 164 |
+
"cross_entropy": 0.4264544071293922
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 165 |
},
|
| 166 |
+
"score": {
|
| 167 |
+
"questions": 215,
|
| 168 |
+
"correct": 0.7116279069767442,
|
| 169 |
+
"brier": 0.19972004831876652,
|
| 170 |
+
"cross_entropy": 1.160348038785374
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
}
|
| 172 |
}
|
| 173 |
},
|
| 174 |
+
"scope": "Test questions are disjoint from calibration training examples. Validation was used for quantization selection."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 175 |
}
|
nvfp4/evaluation/runtime.json
CHANGED
|
@@ -1,36 +1,14 @@
|
|
| 1 |
{
|
| 2 |
"server_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 3 |
-
"
|
| 4 |
-
"
|
| 5 |
-
"
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
"dory-decision-targeted.py": "f78f2633a35fa96967a956ee8060cafb5861e4d65e449c3edb5e7dcbc84ded82",
|
| 15 |
-
"dory-decision-continuation.py": "b044299772c69184e256c593aa97cb33fc2d79a8f0bb1bab4c6230aedd46ba40",
|
| 16 |
-
"decision-eval.py": "15cf0b44d2d21ae2484e97dc298b4b3e25e541bedf653436b0c5f7431a982956",
|
| 17 |
-
"run-dory-decision-overnight.sh": "f12d624177c78a28a255b4b910e24b92d4883662076ddaf2afe3561288dd76f7",
|
| 18 |
-
"dory-decision-overnight.py": "0ffa9a9353edf2cb77ce8b9254182a1753a88a036517b1b9f199f8d4981a9c1a",
|
| 19 |
-
"dory_pytorch.py": "51582e13b8a8742f0597bf1cb6c3aaf2b1fa03b3d3f8a2e02a1577b0b58f6d0d",
|
| 20 |
-
"dory-decision-actions.py": "37e23a2d72b5738b5a977bd1625a385cdee9b014173ca6165d47d7b250c1b735"
|
| 21 |
-
},
|
| 22 |
-
"validation_split": "new_validation",
|
| 23 |
-
"test_splits": [
|
| 24 |
-
"new_test",
|
| 25 |
-
"test",
|
| 26 |
-
"transfer"
|
| 27 |
-
],
|
| 28 |
-
"calibration": "uncalibrated",
|
| 29 |
-
"candidate_steps": [
|
| 30 |
-
0,
|
| 31 |
-
2000,
|
| 32 |
-
4000,
|
| 33 |
-
8000
|
| 34 |
-
],
|
| 35 |
-
"parent_comparison_step": 0
|
| 36 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"server_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
|
| 3 |
+
"scoring_script_sha256": "1a8acf16b2fd060500689ad2580e3c0709ec07647265067fe57e168764b56b42",
|
| 4 |
+
"test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
|
| 5 |
+
"fine_tuned_weight_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 6 |
+
"base_weight_sha256": "2dc8446f2c52d3249eac5bd2154f1f4a8f206152e3b66d7077965b9ad991bbf7",
|
| 7 |
+
"loops": 2,
|
| 8 |
+
"probability_calibration": false,
|
| 9 |
+
"max_context_tokens": 8192,
|
| 10 |
+
"prefill_token_capacity": 2048,
|
| 11 |
+
"decision_branch_capacity": 8,
|
| 12 |
+
"retained_prefix_gib": 0,
|
| 13 |
+
"max_active_sequences": 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
}
|
nvfp4/provenance.json
CHANGED
|
@@ -882,11 +882,10 @@
|
|
| 882 |
},
|
| 883 |
"export_metadata": "export.json"
|
| 884 |
},
|
| 885 |
-
"
|
| 886 |
-
|
| 887 |
-
"
|
| 888 |
-
|
| 889 |
-
|
| 890 |
-
}
|
| 891 |
}
|
| 892 |
}
|
|
|
|
| 882 |
},
|
| 883 |
"export_metadata": "export.json"
|
| 884 |
},
|
| 885 |
+
"published_nvfp4_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 886 |
+
"evaluation": {
|
| 887 |
+
"test": "../evaluation/base-comparison.json",
|
| 888 |
+
"validation": "../evaluation/modelopt-validation.json",
|
| 889 |
+
"jevman": "../evaluation/jevman-comparison.json"
|
|
|
|
| 890 |
}
|
| 891 |
}
|
provenance.json
CHANGED
|
@@ -18,7 +18,7 @@
|
|
| 18 |
"model-00008-of-00008.safetensors": "09dde4e45f4cc217f00222f3c57d8b8656d12a8f215e09d300a6f98d434fe80e",
|
| 19 |
"adapter/adapter.safetensors": "60723ac470780ac7e8bfe7785c23665be2d5c5d6653cbd2fad653523214aea7d"
|
| 20 |
},
|
| 21 |
-
"evaluated_nvfp4_sha256": "
|
| 22 |
"accuracy_evaluation_precision": "nvfp4",
|
| 23 |
"bf16_merge": {
|
| 24 |
"method": "W + (alpha/rank) * BF16(B) @ BF16(A), accumulated F32, stored BF16",
|
|
@@ -879,8 +879,9 @@
|
|
| 879 |
"previous_profile": "Removed from main because it applies to actions-48k weights; retained in actions-48k history."
|
| 880 |
},
|
| 881 |
"published_nvfp4_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 882 |
-
"
|
| 883 |
-
"
|
| 884 |
-
"
|
|
|
|
| 885 |
}
|
| 886 |
}
|
|
|
|
| 18 |
"model-00008-of-00008.safetensors": "09dde4e45f4cc217f00222f3c57d8b8656d12a8f215e09d300a6f98d434fe80e",
|
| 19 |
"adapter/adapter.safetensors": "60723ac470780ac7e8bfe7785c23665be2d5c5d6653cbd2fad653523214aea7d"
|
| 20 |
},
|
| 21 |
+
"evaluated_nvfp4_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 22 |
"accuracy_evaluation_precision": "nvfp4",
|
| 23 |
"bf16_merge": {
|
| 24 |
"method": "W + (alpha/rank) * BF16(B) @ BF16(A), accumulated F32, stored BF16",
|
|
|
|
| 879 |
"previous_profile": "Removed from main because it applies to actions-48k weights; retained in actions-48k history."
|
| 880 |
},
|
| 881 |
"published_nvfp4_sha256": "62bc20a6fbafaf157154650dfb4e9ec32a19d67e8e1f60ddcc3318fd37d6ccb4",
|
| 882 |
+
"evaluation": {
|
| 883 |
+
"test": "evaluation/base-comparison.json",
|
| 884 |
+
"validation": "evaluation/modelopt-validation.json",
|
| 885 |
+
"jevman": "evaluation/jevman-comparison.json"
|
| 886 |
}
|
| 887 |
}
|