Hugging Face's logo

mlboydaisuke
/
decider-2b-coreai-ft

Text Classification
Transformers
Safetensors
English
qwen3_5_text
text-generation
decision-model
system-one
typed-decisions
calibrated
decider
Model card Files
xet
 Community
mlboydaisuke commited on
Commit
284dfd3
·
verified ·
1 Parent(s): 33b2b24

v3: decider-2b v11 + 6,362 verified typed-decision items incl. 543 long documents (delta recipe, 3 epochs, lr 8e-6); card numbers from the 2026-09-26 run

Browse files
Files changed (7) hide show
  1. README.md +12 -9
  2. eval/hist.json +16 -16
  3. eval/jevbench_public231_baseline_decider-2b.json +200 -200
  4. eval/jevbench_public231_previous_delta-5k.json +3646 -0
  5. eval/jevbench_public231_this_model.json +954 -954
  6. eval/train.log +82 -65
  7. model.safetensors +1 -1
README.md CHANGED
@@ -14,32 +14,35 @@ language:
14
  - en
15
  ---
16
 
17
- # decider-2b-coreai-ft — Mapika/decider-2b fine-tuned on 5,819 verified hard decision items
18
 
19
  A [decider](https://github.com/Mapika/decider) checkpoint: it reads a state (text or JSON) and typed questions (choice, score, yes/no) and returns a probability for every option from the letter logits at the answer slot. It never generates text. Same architecture, tokenizer, readout and config as [Mapika/decider-2b](https://huggingface.co/Mapika/decider-2b) v11; only the weights changed.
20
 
21
- What changed: three epochs over 5,819 typed-decision items that we authored and verified for the hard end of the task (long policies, multi-hop rules, dates and numbers, traps). On the 111 public hard items of [JevBench](https://github.com/fstandhartinger/jevbench) it answers 74 correctly against 63 for the v11 checkpoint, measured in the same run with the same code (table below). Easy and standard items are unchanged.
22
 
23
  ## Results
24
 
25
- JevBench public 231 items (easy 48 / standard 72 / hard 111), one question per item, single pass, letter readout through `decider.infer.Decider.system_one` with `use_graphs=False`, bf16 on an A100, 2026-09-25. Accuracy = argmax of the returned probabilities.
26
 
27
  | checkpoint | easy 48 | standard 72 | hard 111 |
28
  |---|---|---|---|
29
  | Mapika/decider-2b v11 (baseline, same run) | 1.0000 | 0.8889 | 0.5676 |
30
- | this model | 1.0000 | 0.8889 | 0.6667 |
 
31
 
32
- This is an independent measurement, not an official JevBench result. The public leaderboard scores include sealed items we cannot run, so the numbers above are not comparable to leaderboard rows. Mapika's own card reports the v11 checkpoint on their serving stack; the baseline row here is our re-measurement in eager mode. Per-item results for both rows are in `eval/`. The previous checkpoint of this repo (1,399 items, 73 of 111 hard in its own run) is revision `5517eb3c`.
 
 
33
 
34
  Two things to know before relying on the probabilities:
35
 
36
- - The fine-tune makes the model more confident on hard items, wrong answers included: mean max-probability on the 111 hard items 0.751 → 0.856, 10-bin ECE 0.183 → 0.189 (from `eval/`). The temperature in `decider_config.json` is v11's; it was not refitted.
37
- - Training loss reaches about 0.1 by the third epoch on 5,819 items, so the checkpoint has largely memorised its training set. Held-out accuracy on 100 items from the same authoring pipeline: 0.83 (chance 0.30).
38
 
39
  ## Training data and recipe
40
 
41
- - Data: 5,819 items written by Qwen3.8-27B (FP8) from family/topic/length specs, then verified by the same model in five chain-of-thought passes with shuffled options; an item is kept when at least four passes agree with the authored answer. Three authoring rounds (1,065 + 334 + 4,420 items). No JevBench item, public or sealed, is in the training data; the JevBench items were used for evaluation only.
42
- - Recipe: `decider.train` from Mapika/decider at commit `15ab28e`, their delta settings: full fine-tune in bf16, cross-entropy on the letter logits, 3 epochs, lr 8e-6, warmup 20, 12,288 tokens per step with accumulation 2, options shuffled per tokenization, no replay of Mapika's own mixture. 1,218 optimizer steps, about 54 minutes on one A100.
43
 
44
  ## Use
45
 
 
14
  - en
15
  ---
16
 
17
+ # decider-2b-coreai-ft — Mapika/decider-2b fine-tuned on 6,362 verified hard decision items
18
 
19
  A [decider](https://github.com/Mapika/decider) checkpoint: it reads a state (text or JSON) and typed questions (choice, score, yes/no) and returns a probability for every option from the letter logits at the answer slot. It never generates text. Same architecture, tokenizer, readout and config as [Mapika/decider-2b](https://huggingface.co/Mapika/decider-2b) v11; only the weights changed.
20
 
21
+ What changed: three epochs over 6,362 typed-decision items that we authored and verified for the hard end of the task (long policies, multi-hop rules, dates and numbers, traps). On the 111 public hard items of [JevBench](https://github.com/fstandhartinger/jevbench) it answers 77 correctly against 63 for the v11 checkpoint, measured in the same run with the same code (table below). Easy items are unchanged. On the 72 standard items it answers 62 correctly against 64 for v11.
22
 
23
  ## Results
24
 
25
+ JevBench public 231 items (easy 48 / standard 72 / hard 111), one question per item, single pass, letter readout through `decider.infer.Decider.system_one` with `use_graphs=False`, bf16 on an A100, 2026-09-26. Accuracy = argmax of the returned probabilities.
26
 
27
  | checkpoint | easy 48 | standard 72 | hard 111 |
28
  |---|---|---|---|
29
  | Mapika/decider-2b v11 (baseline, same run) | 1.0000 | 0.8889 | 0.5676 |
30
+ | previous version of this repo (5,819 items, revision `33b2b241`, same run) | 1.0000 | 0.8889 | 0.6667 |
31
+ | this model | 1.0000 | 0.8611 | 0.6937 |
32
 
33
+ This is an independent measurement, not an official JevBench result. The public leaderboard scores include sealed items we cannot run, so the numbers above are not comparable to leaderboard rows. Mapika's own card reports the v11 checkpoint on their serving stack; the baseline row here is our re-measurement in eager mode. Per-item results for every row are in `eval/`. Earlier checkpoints of this repo: 5,819 items at revision `33b2b241`, 1,399 items at revision `5517eb3c`.
34
+
35
+ The fourth authoring round added 543 long documents (long policies and multi-hop rules, median state about 11,800 characters) aimed at JevBench's longest items. On the 12 hard items whose state is 12,000 characters or longer, this model answers 6 correctly, the previous version 5 and v11 6.
36
 
37
  Two things to know before relying on the probabilities:
38
 
39
+ - The fine-tune makes the model more confident on hard items, wrong answers included: mean max-probability on the 111 hard items 0.751 → 0.871, 10-bin ECE 0.183 → 0.181 (from `eval/`). The temperature in `decider_config.json` is v11's; it was not refitted.
40
+ - Training loss reaches about 0.1 by the third epoch on 6,362 items, so the checkpoint has largely memorised its training set. Held-out accuracy on 100 items from the same authoring pipeline: 0.81 (chance 0.30).
41
 
42
  ## Training data and recipe
43
 
44
+ - Data: 6,362 items written by Qwen3.8-27B (FP8) from family/topic/length specs, then verified by the same model in five chain-of-thought passes with shuffled options; an item is kept when at least four passes agree with the authored answer. Four authoring rounds (1,065 + 334 + 4,420 + 543 items); the fourth round is long documents only. No JevBench item, public or sealed, is in the training data; the JevBench items were used for evaluation only.
45
+ - Recipe: `decider.train` from Mapika/decider at commit `15ab28e`, their delta settings: full fine-tune in bf16, cross-entropy on the letter logits, 3 epochs, lr 8e-6, warmup 20, 12,288 tokens per step with accumulation 2, options shuffled per tokenization, no replay of Mapika's own mixture. 1,545 optimizer steps, about 70 minutes on one A100.
46
 
47
  ## Use
48
 
eval/hist.json CHANGED
@@ -1,14 +1,14 @@
1
  [
2
  {
3
- "step": 1218,
4
  "agg": {
5
  "in_task": {
6
- "acc": 0.83,
7
- "nll": 0.806352436542511,
8
- "brier": 0.3144715130329132,
9
- "ece": 0.14947944998741147,
10
- "aurc": 0.05629271727862271,
11
- "acc_at_80": 0.875,
12
  "chance": 0.29995238095238097
13
  },
14
  "heldout": {},
@@ -18,16 +18,16 @@
18
  "results": {
19
  "coreai_v2_heldout": {
20
  "n": 100,
21
- "acc": 0.83,
22
- "nll": 0.806352436542511,
23
- "brier": 0.3144715130329132,
24
- "ece": 0.14947944998741147,
25
- "aurc": 0.05629271727862271,
26
- "acc_at_80": 0.875,
27
- "acc_at_50": 0.96,
28
  "chance": 0.29995238095238097,
29
- "mean_conf": 0.9653506278991699,
30
- "sec": 26.1,
31
  "heldout": false
32
  }
33
  }
 
1
  [
2
  {
3
+ "step": 1545,
4
  "agg": {
5
  "in_task": {
6
+ "acc": 0.81,
7
+ "nll": 0.9282295107841492,
8
+ "brier": 0.3245173692703247,
9
+ "ece": 0.1702162915468216,
10
+ "aurc": 0.06609745285128044,
11
+ "acc_at_80": 0.8625,
12
  "chance": 0.29995238095238097
13
  },
14
  "heldout": {},
 
18
  "results": {
19
  "coreai_v2_heldout": {
20
  "n": 100,
21
+ "acc": 0.81,
22
+ "nll": 0.9282295107841492,
23
+ "brier": 0.3245173692703247,
24
+ "ece": 0.1702162915468216,
25
+ "aurc": 0.06609745285128044,
26
+ "acc_at_80": 0.8625,
27
+ "acc_at_50": 0.94,
28
  "chance": 0.29995238095238097,
29
+ "mean_conf": 0.9573179483413696,
30
+ "sec": 27.1,
31
  "heldout": false
32
  }
33
  }
eval/jevbench_public231_baseline_decider-2b.json CHANGED
@@ -5,19 +5,19 @@
5
  "n": 48,
6
  "accuracy": 1.0,
7
  "invalid": 0,
8
- "p50_s": 0.067
9
  },
10
  "standard": {
11
  "n": 72,
12
  "accuracy": 0.8889,
13
  "invalid": 0,
14
- "p50_s": 0.067
15
  },
16
  "hard": {
17
  "n": 111,
18
  "accuracy": 0.5676,
19
  "invalid": 0,
20
- "p50_s": 0.071
21
  }
22
  },
23
  "rows": [
@@ -36,7 +36,7 @@
36
  "report_damage": 0.0001,
37
  "track_order": 0.9993
38
  },
39
- "latency_s": 48.697
40
  },
41
  {
42
  "id": "easy-intent-01",
@@ -53,7 +53,7 @@
53
  "report_damage": 0.0001,
54
  "track_order": 0.0001
55
  },
56
- "latency_s": 0.07
57
  },
58
  {
59
  "id": "easy-intent-02",
@@ -70,7 +70,7 @@
70
  "report_damage": 0.0002,
71
  "track_order": 0.0002
72
  },
73
- "latency_s": 0.068
74
  },
75
  {
76
  "id": "easy-intent-03",
@@ -87,7 +87,7 @@
87
  "report_damage": 0.9926,
88
  "track_order": 0.0007
89
  },
90
- "latency_s": 0.07
91
  },
92
  {
93
  "id": "easy-intent-04",
@@ -104,7 +104,7 @@
104
  "report_damage": 0.0016,
105
  "track_order": 0.0016
106
  },
107
- "latency_s": 0.067
108
  },
109
  {
110
  "id": "easy-intent-05",
@@ -121,7 +121,7 @@
121
  "report_damage": 0.0002,
122
  "track_order": 0.9983
123
  },
124
- "latency_s": 0.067
125
  },
126
  {
127
  "id": "easy-intent-06",
@@ -138,7 +138,7 @@
138
  "turn_off_lights": 0.0004,
139
  "weather": 0.0002
140
  },
141
- "latency_s": 0.067
142
  },
143
  {
144
  "id": "easy-intent-07",
@@ -155,7 +155,7 @@
155
  "turn_off_lights": 0.001,
156
  "weather": 0.0008
157
  },
158
- "latency_s": 0.067
159
  },
160
  {
161
  "id": "easy-intent-08",
@@ -172,7 +172,7 @@
172
  "turn_off_lights": 0.0,
173
  "weather": 0.9996
174
  },
175
- "latency_s": 0.067
176
  },
177
  {
178
  "id": "easy-intent-09",
@@ -189,7 +189,7 @@
189
  "turn_off_lights": 0.0011,
190
  "weather": 0.0013
191
  },
192
- "latency_s": 0.067
193
  },
194
  {
195
  "id": "easy-intent-10",
@@ -206,7 +206,7 @@
206
  "turn_off_lights": 0.9972,
207
  "weather": 0.0004
208
  },
209
- "latency_s": 0.067
210
  },
211
  {
212
  "id": "easy-intent-11",
@@ -223,7 +223,7 @@
223
  "turn_off_lights": 0.0003,
224
  "weather": 0.0002
225
  },
226
- "latency_s": 0.067
227
  },
228
  {
229
  "id": "easy-fact-00",
@@ -237,7 +237,7 @@
237
  "yes": 0.9527,
238
  "no": 0.0473
239
  },
240
- "latency_s": 0.066
241
  },
242
  {
243
  "id": "easy-fact-01",
@@ -251,7 +251,7 @@
251
  "yes": 0.0186,
252
  "no": 0.9814
253
  },
254
- "latency_s": 0.067
255
  },
256
  {
257
  "id": "easy-fact-02",
@@ -265,7 +265,7 @@
265
  "yes": 0.966,
266
  "no": 0.034
267
  },
268
- "latency_s": 0.067
269
  },
270
  {
271
  "id": "easy-fact-03",
@@ -279,7 +279,7 @@
279
  "yes": 0.0094,
280
  "no": 0.9906
281
  },
282
- "latency_s": 0.067
283
  },
284
  {
285
  "id": "easy-fact-04",
@@ -293,7 +293,7 @@
293
  "yes": 0.9685,
294
  "no": 0.0315
295
  },
296
- "latency_s": 0.067
297
  },
298
  {
299
  "id": "easy-fact-05",
@@ -307,7 +307,7 @@
307
  "yes": 0.038,
308
  "no": 0.962
309
  },
310
- "latency_s": 0.068
311
  },
312
  {
313
  "id": "easy-fact-06",
@@ -321,7 +321,7 @@
321
  "yes": 0.9472,
322
  "no": 0.0528
323
  },
324
- "latency_s": 0.066
325
  },
326
  {
327
  "id": "easy-fact-07",
@@ -335,7 +335,7 @@
335
  "yes": 0.0217,
336
  "no": 0.9783
337
  },
338
- "latency_s": 0.065
339
  },
340
  {
341
  "id": "easy-fact-08",
@@ -349,7 +349,7 @@
349
  "yes": 0.9757,
350
  "no": 0.0243
351
  },
352
- "latency_s": 0.073
353
  },
354
  {
355
  "id": "easy-fact-09",
@@ -363,7 +363,7 @@
363
  "yes": 0.0179,
364
  "no": 0.9821
365
  },
366
- "latency_s": 0.065
367
  },
368
  {
369
  "id": "easy-fact-10",
@@ -377,7 +377,7 @@
377
  "yes": 0.9269,
378
  "no": 0.0731
379
  },
380
- "latency_s": 0.066
381
  },
382
  {
383
  "id": "easy-fact-11",
@@ -391,7 +391,7 @@
391
  "yes": 0.0424,
392
  "no": 0.9576
393
  },
394
- "latency_s": 0.066
395
  },
396
  {
397
  "id": "easy-extraction-00",
@@ -407,7 +407,7 @@
407
  "credit_card": 0.0015,
408
  "paypal": 0.9976
409
  },
410
- "latency_s": 0.077
411
  },
412
  {
413
  "id": "easy-extraction-01",
@@ -423,7 +423,7 @@
423
  "credit_card": 0.0054,
424
  "paypal": 0.0007
425
  },
426
- "latency_s": 0.066
427
  },
428
  {
429
  "id": "easy-extraction-02",
@@ -439,7 +439,7 @@
439
  "credit_card": 0.0014,
440
  "paypal": 0.001
441
  },
442
- "latency_s": 0.066
443
  },
444
  {
445
  "id": "easy-extraction-03",
@@ -455,7 +455,7 @@
455
  "medium": 0.0172,
456
  "urgent": 0.0624
457
  },
458
- "latency_s": 0.066
459
  },
460
  {
461
  "id": "easy-extraction-04",
@@ -471,7 +471,7 @@
471
  "medium": 0.0134,
472
  "urgent": 0.0016
473
  },
474
- "latency_s": 0.066
475
  },
476
  {
477
  "id": "easy-extraction-05",
@@ -487,7 +487,7 @@
487
  "medium": 0.0033,
488
  "urgent": 0.9838
489
  },
490
- "latency_s": 0.066
491
  },
492
  {
493
  "id": "easy-extraction-06",
@@ -503,7 +503,7 @@
503
  "S": 0.0021,
504
  "XL": 0.0015
505
  },
506
- "latency_s": 0.066
507
  },
508
  {
509
  "id": "easy-extraction-07",
@@ -519,7 +519,7 @@
519
  "S": 0.0009,
520
  "XL": 0.9929
521
  },
522
- "latency_s": 0.066
523
  },
524
  {
525
  "id": "easy-extraction-08",
@@ -536,7 +536,7 @@
536
  "tuesday": 0.0017,
537
  "wednesday": 0.0013
538
  },
539
- "latency_s": 0.066
540
  },
541
  {
542
  "id": "easy-extraction-09",
@@ -553,7 +553,7 @@
553
  "tuesday": 0.0275,
554
  "wednesday": 0.017
555
  },
556
- "latency_s": 0.066
557
  },
558
  {
559
  "id": "easy-extraction-10",
@@ -570,7 +570,7 @@
570
  "tuesday": 0.0053,
571
  "wednesday": 0.0051
572
  },
573
- "latency_s": 0.064
574
  },
575
  {
576
  "id": "easy-extraction-11",
@@ -586,7 +586,7 @@
586
  "credit_card": 0.9978,
587
  "paypal": 0.001
588
  },
589
- "latency_s": 0.065
590
  },
591
  {
592
  "id": "easy-tool_selection-00",
@@ -603,7 +603,7 @@
603
  "send_email": 0.0002,
604
  "translate_text": 0.0003
605
  },
606
- "latency_s": 0.066
607
  },
608
  {
609
  "id": "easy-tool_selection-01",
@@ -620,7 +620,7 @@
620
  "send_email": 0.9983,
621
  "translate_text": 0.0006
622
  },
623
- "latency_s": 0.066
624
  },
625
  {
626
  "id": "easy-tool_selection-02",
@@ -637,7 +637,7 @@
637
  "send_email": 0.0003,
638
  "translate_text": 0.0004
639
  },
640
- "latency_s": 0.066
641
  },
642
  {
643
  "id": "easy-tool_selection-03",
@@ -654,7 +654,7 @@
654
  "send_email": 0.0004,
655
  "translate_text": 0.0007
656
  },
657
- "latency_s": 0.067
658
  },
659
  {
660
  "id": "easy-tool_selection-04",
@@ -671,7 +671,7 @@
671
  "send_email": 0.0001,
672
  "translate_text": 0.9993
673
  },
674
- "latency_s": 0.068
675
  },
676
  {
677
  "id": "easy-tool_selection-05",
@@ -688,7 +688,7 @@
688
  "send_email": 0.0007,
689
  "translate_text": 0.0004
690
  },
691
- "latency_s": 0.068
692
  },
693
  {
694
  "id": "easy-tool_selection-06",
@@ -705,7 +705,7 @@
705
  "search_flights": 0.9993,
706
  "set_reminder": 0.0003
707
  },
708
- "latency_s": 0.069
709
  },
710
  {
711
  "id": "easy-tool_selection-07",
@@ -722,7 +722,7 @@
722
  "search_flights": 0.0006,
723
  "set_reminder": 0.003
724
  },
725
- "latency_s": 0.069
726
  },
727
  {
728
  "id": "easy-tool_selection-08",
@@ -739,7 +739,7 @@
739
  "search_flights": 0.0002,
740
  "set_reminder": 0.999
741
  },
742
- "latency_s": 0.069
743
  },
744
  {
745
  "id": "easy-tool_selection-09",
@@ -756,7 +756,7 @@
756
  "search_flights": 0.0002,
757
  "set_reminder": 0.0003
758
  },
759
- "latency_s": 0.068
760
  },
761
  {
762
  "id": "easy-tool_selection-10",
@@ -773,7 +773,7 @@
773
  "search_flights": 0.0006,
774
  "set_reminder": 0.0012
775
  },
776
- "latency_s": 0.068
777
  },
778
  {
779
  "id": "easy-tool_selection-11",
@@ -790,7 +790,7 @@
790
  "search_flights": 0.0001,
791
  "set_reminder": 0.9994
792
  },
793
- "latency_s": 0.068
794
  },
795
  {
796
  "id": "original-policy-01-0",
@@ -804,7 +804,7 @@
804
  "yes": 0.0456,
805
  "no": 0.9544
806
  },
807
- "latency_s": 0.068
808
  },
809
  {
810
  "id": "original-policy-01-1",
@@ -818,7 +818,7 @@
818
  "yes": 0.0473,
819
  "no": 0.9527
820
  },
821
- "latency_s": 0.068
822
  },
823
  {
824
  "id": "original-policy-02-0",
@@ -832,7 +832,7 @@
832
  "yes": 0.8683,
833
  "no": 0.1317
834
  },
835
- "latency_s": 0.068
836
  },
837
  {
838
  "id": "original-policy-02-1",
@@ -846,7 +846,7 @@
846
  "yes": 0.2921,
847
  "no": 0.7079
848
  },
849
- "latency_s": 0.068
850
  },
851
  {
852
  "id": "original-policy-03-0",
@@ -860,7 +860,7 @@
860
  "yes": 0.5192,
861
  "no": 0.4808
862
  },
863
- "latency_s": 0.068
864
  },
865
  {
866
  "id": "original-policy-03-1",
@@ -874,7 +874,7 @@
874
  "yes": 0.3957,
875
  "no": 0.6043
876
  },
877
- "latency_s": 0.068
878
  },
879
  {
880
  "id": "original-policy-04-0",
@@ -888,7 +888,7 @@
888
  "yes": 0.1605,
889
  "no": 0.8395
890
  },
891
- "latency_s": 0.068
892
  },
893
  {
894
  "id": "original-policy-04-1",
@@ -902,7 +902,7 @@
902
  "yes": 0.0785,
903
  "no": 0.9215
904
  },
905
- "latency_s": 0.068
906
  },
907
  {
908
  "id": "original-policy-05-0",
@@ -916,7 +916,7 @@
916
  "yes": 0.3249,
917
  "no": 0.6751
918
  },
919
- "latency_s": 0.068
920
  },
921
  {
922
  "id": "original-policy-05-1",
@@ -944,7 +944,7 @@
944
  "yes": 0.8234,
945
  "no": 0.1766
946
  },
947
- "latency_s": 0.069
948
  },
949
  {
950
  "id": "original-policy-06-1",
@@ -975,7 +975,7 @@
975
  "refund": 0.9348,
976
  "status": 0.0006
977
  },
978
- "latency_s": 0.068
979
  },
980
  {
981
  "id": "original-intent-01-1",
@@ -1009,7 +1009,7 @@
1009
  "refund": 0.0025,
1010
  "status": 0.7357
1011
  },
1012
- "latency_s": 0.068
1013
  },
1014
  {
1015
  "id": "original-intent-02-1",
@@ -1026,7 +1026,7 @@
1026
  "refund": 0.0003,
1027
  "status": 0.9801
1028
  },
1029
- "latency_s": 0.067
1030
  },
1031
  {
1032
  "id": "original-intent-03-0",
@@ -1043,7 +1043,7 @@
1043
  "refund": 0.0073,
1044
  "status": 0.0073
1045
  },
1046
- "latency_s": 0.068
1047
  },
1048
  {
1049
  "id": "original-intent-03-1",
@@ -1060,7 +1060,7 @@
1060
  "refund": 0.0007,
1061
  "status": 0.0003
1062
  },
1063
- "latency_s": 0.069
1064
  },
1065
  {
1066
  "id": "original-intent-04-0",
@@ -1077,7 +1077,7 @@
1077
  "refund": 0.0129,
1078
  "status": 0.0007
1079
  },
1080
- "latency_s": 0.068
1081
  },
1082
  {
1083
  "id": "original-intent-04-1",
@@ -1094,7 +1094,7 @@
1094
  "refund": 0.0047,
1095
  "status": 0.0006
1096
  },
1097
- "latency_s": 0.066
1098
  },
1099
  {
1100
  "id": "original-intent-05-0",
@@ -1111,7 +1111,7 @@
1111
  "refund": 0.2237,
1112
  "status": 0.0017
1113
  },
1114
- "latency_s": 0.066
1115
  },
1116
  {
1117
  "id": "original-intent-05-1",
@@ -1128,7 +1128,7 @@
1128
  "refund": 0.3631,
1129
  "status": 0.0019
1130
  },
1131
- "latency_s": 0.066
1132
  },
1133
  {
1134
  "id": "original-intent-06-0",
@@ -1145,7 +1145,7 @@
1145
  "refund": 0.0009,
1146
  "status": 0.9703
1147
  },
1148
- "latency_s": 0.066
1149
  },
1150
  {
1151
  "id": "original-intent-06-1",
@@ -1162,7 +1162,7 @@
1162
  "refund": 0.0007,
1163
  "status": 0.9817
1164
  },
1165
- "latency_s": 0.066
1166
  },
1167
  {
1168
  "id": "original-ordinal-01-0",
@@ -1178,7 +1178,7 @@
1178
  "2": 0.0062,
1179
  "3": 0.0032
1180
  },
1181
- "latency_s": 0.801
1182
  },
1183
  {
1184
  "id": "original-ordinal-01-1",
@@ -1194,7 +1194,7 @@
1194
  "2": 0.01,
1195
  "3": 0.0031
1196
  },
1197
- "latency_s": 0.068
1198
  },
1199
  {
1200
  "id": "original-ordinal-02-0",
@@ -1210,7 +1210,7 @@
1210
  "2": 0.1319,
1211
  "3": 0.008
1212
  },
1213
- "latency_s": 0.068
1214
  },
1215
  {
1216
  "id": "original-ordinal-02-1",
@@ -1226,7 +1226,7 @@
1226
  "2": 0.0164,
1227
  "3": 0.0095
1228
  },
1229
- "latency_s": 0.068
1230
  },
1231
  {
1232
  "id": "original-ordinal-03-0",
@@ -1242,7 +1242,7 @@
1242
  "2": 0.7779,
1243
  "3": 0.0051
1244
  },
1245
- "latency_s": 0.068
1246
  },
1247
  {
1248
  "id": "original-ordinal-03-1",
@@ -1258,7 +1258,7 @@
1258
  "2": 0.7353,
1259
  "3": 0.0151
1260
  },
1261
- "latency_s": 0.068
1262
  },
1263
  {
1264
  "id": "original-ordinal-04-0",
@@ -1274,7 +1274,7 @@
1274
  "2": 0.0429,
1275
  "3": 0.9247
1276
  },
1277
- "latency_s": 0.067
1278
  },
1279
  {
1280
  "id": "original-ordinal-04-1",
@@ -1290,7 +1290,7 @@
1290
  "2": 0.1234,
1291
  "3": 0.824
1292
  },
1293
- "latency_s": 0.067
1294
  },
1295
  {
1296
  "id": "original-ordinal-05-0",
@@ -1306,7 +1306,7 @@
1306
  "2": 0.0323,
1307
  "3": 0.0052
1308
  },
1309
- "latency_s": 0.067
1310
  },
1311
  {
1312
  "id": "original-ordinal-05-1",
@@ -1322,7 +1322,7 @@
1322
  "2": 0.1294,
1323
  "3": 0.021
1324
  },
1325
- "latency_s": 0.067
1326
  },
1327
  {
1328
  "id": "original-ordinal-06-0",
@@ -1338,7 +1338,7 @@
1338
  "2": 0.7511,
1339
  "3": 0.0022
1340
  },
1341
- "latency_s": 0.068
1342
  },
1343
  {
1344
  "id": "original-ordinal-06-1",
@@ -1354,7 +1354,7 @@
1354
  "2": 0.7949,
1355
  "3": 0.0026
1356
  },
1357
- "latency_s": 0.068
1358
  },
1359
  {
1360
  "id": "original-extraction-01-0",
@@ -1370,7 +1370,7 @@
1370
  "post": 0.0063,
1371
  "unknown": 0.0929
1372
  },
1373
- "latency_s": 0.066
1374
  },
1375
  {
1376
  "id": "original-extraction-01-1",
@@ -1386,7 +1386,7 @@
1386
  "post": 0.0065,
1387
  "unknown": 0.081
1388
  },
1389
- "latency_s": 0.066
1390
  },
1391
  {
1392
  "id": "original-extraction-02-0",
@@ -1402,7 +1402,7 @@
1402
  "post": 0.1995,
1403
  "unknown": 0.7637
1404
  },
1405
- "latency_s": 0.066
1406
  },
1407
  {
1408
  "id": "original-extraction-02-1",
@@ -1418,7 +1418,7 @@
1418
  "post": 0.0732,
1419
  "unknown": 0.9133
1420
  },
1421
- "latency_s": 0.066
1422
  },
1423
  {
1424
  "id": "original-extraction-03-0",
@@ -1434,7 +1434,7 @@
1434
  "post": 0.0344,
1435
  "unknown": 0.295
1436
  },
1437
- "latency_s": 0.066
1438
  },
1439
  {
1440
  "id": "original-extraction-03-1",
@@ -1450,7 +1450,7 @@
1450
  "post": 0.0051,
1451
  "unknown": 0.0092
1452
  },
1453
- "latency_s": 0.066
1454
  },
1455
  {
1456
  "id": "original-extraction-04-0",
@@ -1466,7 +1466,7 @@
1466
  "post": 0.9733,
1467
  "unknown": 0.0227
1468
  },
1469
- "latency_s": 0.066
1470
  },
1471
  {
1472
  "id": "original-extraction-04-1",
@@ -1482,7 +1482,7 @@
1482
  "post": 0.9545,
1483
  "unknown": 0.0361
1484
  },
1485
- "latency_s": 0.067
1486
  },
1487
  {
1488
  "id": "original-extraction-05-0",
@@ -1498,7 +1498,7 @@
1498
  "post": 0.0037,
1499
  "unknown": 0.9846
1500
  },
1501
- "latency_s": 0.067
1502
  },
1503
  {
1504
  "id": "original-extraction-05-1",
@@ -1514,7 +1514,7 @@
1514
  "post": 0.0181,
1515
  "unknown": 0.6251
1516
  },
1517
- "latency_s": 0.066
1518
  },
1519
  {
1520
  "id": "original-extraction-06-0",
@@ -1530,7 +1530,7 @@
1530
  "post": 0.0077,
1531
  "unknown": 0.0225
1532
  },
1533
- "latency_s": 0.066
1534
  },
1535
  {
1536
  "id": "original-extraction-06-1",
@@ -1546,7 +1546,7 @@
1546
  "post": 0.0048,
1547
  "unknown": 0.0204
1548
  },
1549
- "latency_s": 0.066
1550
  },
1551
  {
1552
  "id": "original-adequacy-01-0",
@@ -1560,7 +1560,7 @@
1560
  "yes": 0.8447,
1561
  "no": 0.1553
1562
  },
1563
- "latency_s": 0.066
1564
  },
1565
  {
1566
  "id": "original-adequacy-01-1",
@@ -1574,7 +1574,7 @@
1574
  "yes": 0.746,
1575
  "no": 0.254
1576
  },
1577
- "latency_s": 0.066
1578
  },
1579
  {
1580
  "id": "original-adequacy-02-0",
@@ -1588,7 +1588,7 @@
1588
  "yes": 0.1362,
1589
  "no": 0.8638
1590
  },
1591
- "latency_s": 0.067
1592
  },
1593
  {
1594
  "id": "original-adequacy-02-1",
@@ -1602,7 +1602,7 @@
1602
  "yes": 0.1605,
1603
  "no": 0.8395
1604
  },
1605
- "latency_s": 0.066
1606
  },
1607
  {
1608
  "id": "original-adequacy-03-0",
@@ -1616,7 +1616,7 @@
1616
  "yes": 0.7873,
1617
  "no": 0.2127
1618
  },
1619
- "latency_s": 0.066
1620
  },
1621
  {
1622
  "id": "original-adequacy-03-1",
@@ -1630,7 +1630,7 @@
1630
  "yes": 0.6999,
1631
  "no": 0.3001
1632
  },
1633
- "latency_s": 0.067
1634
  },
1635
  {
1636
  "id": "original-adequacy-04-0",
@@ -1644,7 +1644,7 @@
1644
  "yes": 0.8343,
1645
  "no": 0.1657
1646
  },
1647
- "latency_s": 0.066
1648
  },
1649
  {
1650
  "id": "original-adequacy-04-1",
@@ -1658,7 +1658,7 @@
1658
  "yes": 0.6751,
1659
  "no": 0.3249
1660
  },
1661
- "latency_s": 0.067
1662
  },
1663
  {
1664
  "id": "original-adequacy-05-0",
@@ -1672,7 +1672,7 @@
1672
  "yes": 0.548,
1673
  "no": 0.452
1674
  },
1675
- "latency_s": 0.066
1676
  },
1677
  {
1678
  "id": "original-adequacy-05-1",
@@ -1686,7 +1686,7 @@
1686
  "yes": 0.6917,
1687
  "no": 0.3083
1688
  },
1689
- "latency_s": 0.066
1690
  },
1691
  {
1692
  "id": "original-adequacy-06-0",
@@ -1700,7 +1700,7 @@
1700
  "yes": 0.548,
1701
  "no": 0.452
1702
  },
1703
- "latency_s": 0.066
1704
  },
1705
  {
1706
  "id": "original-adequacy-06-1",
@@ -1714,7 +1714,7 @@
1714
  "yes": 0.595,
1715
  "no": 0.405
1716
  },
1717
- "latency_s": 0.067
1718
  },
1719
  {
1720
  "id": "original-routing-01-0",
@@ -1732,7 +1732,7 @@
1732
  "math": 0.9149,
1733
  "tools": 0.0056
1734
  },
1735
- "latency_s": 0.067
1736
  },
1737
  {
1738
  "id": "original-routing-01-1",
@@ -1750,7 +1750,7 @@
1750
  "math": 0.9137,
1751
  "tools": 0.0036
1752
  },
1753
- "latency_s": 0.067
1754
  },
1755
  {
1756
  "id": "original-routing-02-0",
@@ -1768,7 +1768,7 @@
1768
  "math": 0.0023,
1769
  "tools": 0.003
1770
  },
1771
- "latency_s": 0.067
1772
  },
1773
  {
1774
  "id": "original-routing-02-1",
@@ -1786,7 +1786,7 @@
1786
  "math": 0.0035,
1787
  "tools": 0.0024
1788
  },
1789
- "latency_s": 0.067
1790
  },
1791
  {
1792
  "id": "original-routing-03-0",
@@ -1804,7 +1804,7 @@
1804
  "math": 0.0008,
1805
  "tools": 0.0045
1806
  },
1807
- "latency_s": 0.066
1808
  },
1809
  {
1810
  "id": "original-routing-03-1",
@@ -1822,7 +1822,7 @@
1822
  "math": 0.0007,
1823
  "tools": 0.0047
1824
  },
1825
- "latency_s": 0.067
1826
  },
1827
  {
1828
  "id": "original-routing-04-0",
@@ -1840,7 +1840,7 @@
1840
  "math": 0.002,
1841
  "tools": 0.0166
1842
  },
1843
- "latency_s": 0.067
1844
  },
1845
  {
1846
  "id": "original-routing-04-1",
@@ -1858,7 +1858,7 @@
1858
  "math": 0.0016,
1859
  "tools": 0.0119
1860
  },
1861
- "latency_s": 0.067
1862
  },
1863
  {
1864
  "id": "original-routing-05-0",
@@ -1876,7 +1876,7 @@
1876
  "math": 0.0018,
1877
  "tools": 0.4325
1878
  },
1879
- "latency_s": 0.067
1880
  },
1881
  {
1882
  "id": "original-routing-05-1",
@@ -1894,7 +1894,7 @@
1894
  "math": 0.0012,
1895
  "tools": 0.1126
1896
  },
1897
- "latency_s": 0.066
1898
  },
1899
  {
1900
  "id": "original-routing-06-0",
@@ -1912,7 +1912,7 @@
1912
  "math": 0.0012,
1913
  "tools": 0.0025
1914
  },
1915
- "latency_s": 0.067
1916
  },
1917
  {
1918
  "id": "original-routing-06-1",
@@ -1930,7 +1930,7 @@
1930
  "math": 0.0012,
1931
  "tools": 0.0036
1932
  },
1933
- "latency_s": 0.066
1934
  },
1935
  {
1936
  "id": "hard-opus-a-long_policy-01",
@@ -1947,7 +1947,7 @@
1947
  "pay_subject_to_10000_sublimit": 0.0233,
1948
  "pay_subject_to_15000_sublimit": 0.949
1949
  },
1950
- "latency_s": 0.161
1951
  },
1952
  {
1953
  "id": "hard-opus-a-long_policy-04",
@@ -1964,7 +1964,7 @@
1964
  "executive_committee": 0.0028,
1965
  "vp_and_finance_director": 0.6234
1966
  },
1967
- "latency_s": 0.104
1968
  },
1969
  {
1970
  "id": "hard-opus-a-long_policy-08",
@@ -1996,7 +1996,7 @@
1996
  "no_license_required": 0.1142,
1997
  "shipment_prohibited": 0.0194
1998
  },
1999
- "latency_s": 0.088
2000
  },
2001
  {
2002
  "id": "hard-opus-a-long_policy-11",
@@ -2010,7 +2010,7 @@
2010
  "yes": 0.0681,
2011
  "no": 0.9319
2012
  },
2013
- "latency_s": 0.083
2014
  },
2015
  {
2016
  "id": "hard-opus-a-long_policy-13",
@@ -2024,7 +2024,7 @@
2024
  "yes": 0.3596,
2025
  "no": 0.6404
2026
  },
2027
- "latency_s": 0.084
2028
  },
2029
  {
2030
  "id": "hard-opus-a-long_policy-17",
@@ -2041,7 +2041,7 @@
2041
  "security_incident_response": 0.0054,
2042
  "vendor_support": 0.06
2043
  },
2044
- "latency_s": 0.083
2045
  },
2046
  {
2047
  "id": "hard-opus-a-long_policy-19",
@@ -2084,7 +2084,7 @@
2084
  "late": 0.1989,
2085
  "on_time": 0.7614
2086
  },
2087
- "latency_s": 0.073
2088
  },
2089
  {
2090
  "id": "hard-opus-a-probability-07",
@@ -2099,7 +2099,7 @@
2099
  "reopened": 0.0377,
2100
  "resolved_first_contact": 0.1993
2101
  },
2102
- "latency_s": 0.073
2103
  },
2104
  {
2105
  "id": "hard-opus-a-probability-08",
@@ -2113,7 +2113,7 @@
2113
  "yes": 0.3957,
2114
  "no": 0.6043
2115
  },
2116
- "latency_s": 0.069
2117
  },
2118
  {
2119
  "id": "hard-opus-a-temporal_numeric-03",
@@ -2127,7 +2127,7 @@
2127
  "yes": 0.3165,
2128
  "no": 0.6835
2129
  },
2130
- "latency_s": 0.069
2131
  },
2132
  {
2133
  "id": "hard-opus-a-temporal_numeric-06",
@@ -2172,7 +2172,7 @@
2172
  "yes": 0.548,
2173
  "no": 0.452
2174
  },
2175
- "latency_s": 0.069
2176
  },
2177
  {
2178
  "id": "hard-opus-a-temporal_numeric-12",
@@ -2188,7 +2188,7 @@
2188
  "2": 0.4681,
2189
  "3": 0.0113
2190
  },
2191
- "latency_s": 0.075
2192
  },
2193
  {
2194
  "id": "hard-opus-b-ambiguous-02",
@@ -2203,7 +2203,7 @@
2203
  "false_positive": 0.4813,
2204
  "real_incident": 0.4561
2205
  },
2206
- "latency_s": 0.07
2207
  },
2208
  {
2209
  "id": "hard-opus-b-ambiguous-03",
@@ -2218,7 +2218,7 @@
2218
  "time_barred": 0.5371,
2219
  "within_limitation": 0.4333
2220
  },
2221
- "latency_s": 0.069
2222
  },
2223
  {
2224
  "id": "hard-opus-b-ambiguous-07",
@@ -2233,7 +2233,7 @@
2233
  "over_limit": 0.1475,
2234
  "within_policy": 0.0734
2235
  },
2236
- "latency_s": 0.069
2237
  },
2238
  {
2239
  "id": "hard-opus-b-ambiguous-09",
@@ -2248,7 +2248,7 @@
2248
  "inconclusive": 0.0149,
2249
  "legitimate": 0.4398
2250
  },
2251
- "latency_s": 0.071
2252
  },
2253
  {
2254
  "id": "hard-opus-b-ambiguous-10",
@@ -2263,7 +2263,7 @@
2263
  "send_to_alex_moreno": 0.7646,
2264
  "send_to_alexandra_lind": 0.0246
2265
  },
2266
- "latency_s": 0.069
2267
  },
2268
  {
2269
  "id": "hard-opus-b-ambiguous-11",
@@ -2278,7 +2278,7 @@
2278
  "grade_correct": 0.0157,
2279
  "grade_should_be_higher": 0.0048
2280
  },
2281
- "latency_s": 0.07
2282
  },
2283
  {
2284
  "id": "hard-opus-b-ambiguous-13",
@@ -2294,7 +2294,7 @@
2294
  "charge_platform_team": 0.0051,
2295
  "split_50_50": 0.0057
2296
  },
2297
- "latency_s": 0.07
2298
  },
2299
  {
2300
  "id": "hard-opus-b-multi_hop-03",
@@ -2328,7 +2328,7 @@
2328
  "esra": 0.0055,
2329
  "no_page_ticket_only": 0.8158
2330
  },
2331
- "latency_s": 0.083
2332
  },
2333
  {
2334
  "id": "hard-opus-b-multi_hop-05",
@@ -2345,7 +2345,7 @@
2345
  "3": 0.0922,
2346
  "4": 0.0447
2347
  },
2348
- "latency_s": 11.634
2349
  },
2350
  {
2351
  "id": "hard-opus-b-multi_hop-07",
@@ -2407,7 +2407,7 @@
2407
  "database_hardware": 0.0108,
2408
  "upstream_provider": 0.317
2409
  },
2410
- "latency_s": 0.071
2411
  },
2412
  {
2413
  "id": "hard-opus-b-probability-03",
@@ -2421,7 +2421,7 @@
2421
  "yes": 0.6666,
2422
  "no": 0.3334
2423
  },
2424
- "latency_s": 0.072
2425
  },
2426
  {
2427
  "id": "hard-opus-b-probability-04",
@@ -2468,7 +2468,7 @@
2468
  "process_refund": 0.0061,
2469
  "send_kb_article": 0.0018
2470
  },
2471
- "latency_s": 0.072
2472
  },
2473
  {
2474
  "id": "hard-opus-b-tradeoff-03",
@@ -2484,7 +2484,7 @@
2484
  "order_c": 0.0186,
2485
  "order_d": 0.0374
2486
  },
2487
- "latency_s": 0.072
2488
  },
2489
  {
2490
  "id": "hard-opus-b-tradeoff-06",
@@ -2498,7 +2498,7 @@
2498
  "yes": 0.1408,
2499
  "no": 0.8592
2500
  },
2501
- "latency_s": 0.071
2502
  },
2503
  {
2504
  "id": "hard-opus-b-tradeoff-07",
@@ -2515,7 +2515,7 @@
2515
  "p2_fix_30d": 0.2351,
2516
  "p3_backlog": 0.0014
2517
  },
2518
- "latency_s": 0.071
2519
  },
2520
  {
2521
  "id": "hard-opus-b-tradeoff-08",
@@ -2529,7 +2529,7 @@
2529
  "yes": 0.1408,
2530
  "no": 0.8592
2531
  },
2532
- "latency_s": 0.07
2533
  },
2534
  {
2535
  "id": "hard-opus-b-tradeoff-12",
@@ -2546,7 +2546,7 @@
2546
  "refund_now": 0.9697,
2547
  "wait_for_inspection": 0.0096
2548
  },
2549
- "latency_s": 0.07
2550
  },
2551
  {
2552
  "id": "hard-opus-c-long_policy-02",
@@ -2581,7 +2581,7 @@
2581
  "eur_400": 0.179,
2582
  "eur_600": 0.0357
2583
  },
2584
- "latency_s": 0.121
2585
  },
2586
  {
2587
  "id": "hard-opus-c-long_policy-04",
@@ -2611,7 +2611,7 @@
2611
  "2": 0.4746,
2612
  "3": 0.0444
2613
  },
2614
- "latency_s": 0.386
2615
  },
2616
  {
2617
  "id": "hard-opus-c-long_policy-08",
@@ -2627,7 +2627,7 @@
2627
  "2": 0.1642,
2628
  "3": 0.0503
2629
  },
2630
- "latency_s": 0.392
2631
  },
2632
  {
2633
  "id": "hard-opus-c-long_policy-10",
@@ -2643,7 +2643,7 @@
2643
  "reject_return": 0.0882,
2644
  "repair_or_replace_only": 0.143
2645
  },
2646
- "latency_s": 0.126
2647
  },
2648
  {
2649
  "id": "hard-opus-c-long_policy-11",
@@ -2686,7 +2686,7 @@
2686
  "yes": 0.1823,
2687
  "no": 0.8177
2688
  },
2689
- "latency_s": 0.074
2690
  },
2691
  {
2692
  "id": "hard-opus-c-temporal_numeric-03",
@@ -2720,7 +2720,7 @@
2720
  "usd_2335_00": 0.0994,
2721
  "usd_2340_00": 0.0289
2722
  },
2723
- "latency_s": 0.07
2724
  },
2725
  {
2726
  "id": "hard-opus-c-temporal_numeric-06",
@@ -2770,7 +2770,7 @@
2770
  "expired_24_month_term": 0.0958,
2771
  "expired_30_month_cap": 0.8658
2772
  },
2773
- "latency_s": 0.071
2774
  },
2775
  {
2776
  "id": "hard-sol-a-adversarial-01",
@@ -2786,7 +2786,7 @@
2786
  "grant_access": 0.0039,
2787
  "grant_owner_exception": 0.0149
2788
  },
2789
- "latency_s": 0.069
2790
  },
2791
  {
2792
  "id": "hard-sol-a-adversarial-06",
@@ -2800,7 +2800,7 @@
2800
  "yes": 0.0252,
2801
  "no": 0.9748
2802
  },
2803
- "latency_s": 0.069
2804
  },
2805
  {
2806
  "id": "hard-sol-a-adversarial-07",
@@ -2816,7 +2816,7 @@
2816
  "retain_indefinitely": 0.0014,
2817
  "retain_until_2031": 0.9634
2818
  },
2819
- "latency_s": 0.069
2820
  },
2821
  {
2822
  "id": "hard-sol-a-adversarial-08",
@@ -2830,7 +2830,7 @@
2830
  "yes": 0.9757,
2831
  "no": 0.0243
2832
  },
2833
- "latency_s": 0.068
2834
  },
2835
  {
2836
  "id": "hard-sol-a-adversarial-09",
@@ -2846,7 +2846,7 @@
2846
  "2": 0.7697,
2847
  "3": 0.167
2848
  },
2849
- "latency_s": 0.072
2850
  },
2851
  {
2852
  "id": "hard-sol-a-adversarial-11",
@@ -2862,7 +2862,7 @@
2862
  "manual_hold": 0.0041,
2863
  "standard_research": 0.6779
2864
  },
2865
- "latency_s": 0.069
2866
  },
2867
  {
2868
  "id": "hard-sol-a-multi_hop-01",
@@ -2878,7 +2878,7 @@
2878
  "reject_tier": 0.1175,
2879
  "reroute_s3": 0.0498
2880
  },
2881
- "latency_s": 0.086
2882
  },
2883
  {
2884
  "id": "hard-sol-a-multi_hop-02",
@@ -2894,7 +2894,7 @@
2894
  "insufficient_information": 0.0294,
2895
  "security_primary": 0.6623
2896
  },
2897
- "latency_s": 0.086
2898
  },
2899
  {
2900
  "id": "hard-sol-a-multi_hop-05",
@@ -2910,7 +2910,7 @@
2910
  "require_director": 0.7348,
2911
  "require_finance_board": 0.0034
2912
  },
2913
- "latency_s": 0.085
2914
  },
2915
  {
2916
  "id": "hard-sol-a-multi_hop-07",
@@ -2926,7 +2926,7 @@
2926
  "deny_score": 0.0837,
2927
  "manual_equivalency": 0.1872
2928
  },
2929
- "latency_s": 0.084
2930
  },
2931
  {
2932
  "id": "hard-sol-a-multi_hop-08",
@@ -2942,7 +2942,7 @@
2942
  "local_maintenance": 0.1563,
2943
  "safety_shutdown": 0.005
2944
  },
2945
- "latency_s": 0.085
2946
  },
2947
  {
2948
  "id": "hard-sol-a-multi_hop-09",
@@ -2974,7 +2974,7 @@
2974
  "needs_field_review": 0.0198,
2975
  "reduce_to_30_days": 0.4218
2976
  },
2977
- "latency_s": 0.084
2978
  },
2979
  {
2980
  "id": "hard-sol-a-multi_hop-12",
@@ -3004,7 +3004,7 @@
3004
  "yes": 0.0105,
3005
  "no": 0.9895
3006
  },
3007
- "latency_s": 0.072
3008
  },
3009
  {
3010
  "id": "hard-sol-a-trap-04",
@@ -3018,7 +3018,7 @@
3018
  "yes": 0.0813,
3019
  "no": 0.9187
3020
  },
3021
- "latency_s": 0.072
3022
  },
3023
  {
3024
  "id": "hard-sol-a-trap-05",
@@ -3034,7 +3034,7 @@
3034
  "2": 0.3736,
3035
  "3": 0.007
3036
  },
3037
- "latency_s": 0.076
3038
  },
3039
  {
3040
  "id": "hard-sol-a-trap-06",
@@ -3066,7 +3066,7 @@
3066
  "late": 0.9767,
3067
  "on_time": 0.0228
3068
  },
3069
- "latency_s": 0.071
3070
  },
3071
  {
3072
  "id": "hard-sol-a-trap-10",
@@ -3082,7 +3082,7 @@
3082
  "ready_one_review": 0.0087,
3083
  "ready_two_reviews": 0.0542
3084
  },
3085
- "latency_s": 0.071
3086
  },
3087
  {
3088
  "id": "hard-sol-a-trap-13",
@@ -3096,7 +3096,7 @@
3096
  "yes": 0.044,
3097
  "no": 0.956
3098
  },
3099
- "latency_s": 0.071
3100
  },
3101
  {
3102
  "id": "hard-sol-a-trap-15",
@@ -3110,7 +3110,7 @@
3110
  "yes": 0.3596,
3111
  "no": 0.6404
3112
  },
3113
- "latency_s": 0.071
3114
  },
3115
  {
3116
  "id": "hard-sol-b-judge_hard-01",
@@ -3124,7 +3124,7 @@
3124
  "yes": 0.8119,
3125
  "no": 0.1881
3126
  },
3127
- "latency_s": 0.07
3128
  },
3129
  {
3130
  "id": "hard-sol-b-judge_hard-02",
@@ -3152,7 +3152,7 @@
3152
  "yes": 0.8726,
3153
  "no": 0.1274
3154
  },
3155
- "latency_s": 0.067
3156
  },
3157
  {
3158
  "id": "hard-sol-b-judge_hard-08",
@@ -3166,7 +3166,7 @@
3166
  "yes": 0.8925,
3167
  "no": 0.1075
3168
  },
3169
- "latency_s": 0.067
3170
  },
3171
  {
3172
  "id": "hard-sol-b-judge_hard-10",
@@ -3180,7 +3180,7 @@
3180
  "yes": 0.4712,
3181
  "no": 0.5288
3182
  },
3183
- "latency_s": 0.067
3184
  },
3185
  {
3186
  "id": "hard-sol-b-judge_hard-14",
@@ -3222,7 +3222,7 @@
3222
  "yes": 0.9343,
3223
  "no": 0.0657
3224
  },
3225
- "latency_s": 0.068
3226
  },
3227
  {
3228
  "id": "hard-sol-b-long_policy-01",
@@ -3254,7 +3254,7 @@
3254
  "privacy_and_security": 0.0669,
3255
  "privacy_only": 0.835
3256
  },
3257
- "latency_s": 0.091
3258
  },
3259
  {
3260
  "id": "hard-sol-b-long_policy-05",
@@ -3286,7 +3286,7 @@
3286
  "no_award": 0.063,
3287
  "river_arts": 0.8743
3288
  },
3289
- "latency_s": 0.091
3290
  },
3291
  {
3292
  "id": "hard-sol-b-routing_hard-01",
@@ -3303,7 +3303,7 @@
3303
  "office_file_repair": 0.987,
3304
  "spreadsheet_formula": 0.0092
3305
  },
3306
- "latency_s": 0.07
3307
  },
3308
  {
3309
  "id": "hard-sol-b-routing_hard-02",
@@ -3320,7 +3320,7 @@
3320
  "mobile_app": 0.001,
3321
  "payment_disputes": 0.0051
3322
  },
3323
- "latency_s": 0.069
3324
  },
3325
  {
3326
  "id": "hard-sol-b-routing_hard-03",
@@ -3337,7 +3337,7 @@
3337
  "network_dns": 0.9525,
3338
  "tls_certificates": 0.0017
3339
  },
3340
- "latency_s": 0.069
3341
  },
3342
  {
3343
  "id": "hard-sol-b-routing_hard-07",
@@ -3354,7 +3354,7 @@
3354
  "macos_runtime": 0.0017,
3355
  "release_notarization": 0.9902
3356
  },
3357
- "latency_s": 0.068
3358
  },
3359
  {
3360
  "id": "hard-sol-b-routing_hard-09",
@@ -3371,7 +3371,7 @@
3371
  "source_access": 0.9916,
3372
  "vpn_network": 0.0023
3373
  },
3374
- "latency_s": 0.068
3375
  },
3376
  {
3377
  "id": "hard-sol-b-temporal_numeric-01",
@@ -3387,7 +3387,7 @@
3387
  "late_by_under_2h": 0.1855,
3388
  "within_window": 0.6043
3389
  },
3390
- "latency_s": 0.068
3391
  },
3392
  {
3393
  "id": "hard-sol-b-temporal_numeric-02",
@@ -3403,7 +3403,7 @@
3403
  "over_limit": 0.1269,
3404
  "under_100gb_below": 0.541
3405
  },
3406
- "latency_s": 0.068
3407
  },
3408
  {
3409
  "id": "hard-sol-b-temporal_numeric-03",
@@ -3419,7 +3419,7 @@
3419
  "6_credits": 0.1571,
3420
  "9_credits": 0.6016
3421
  },
3422
- "latency_s": 0.067
3423
  },
3424
  {
3425
  "id": "hard-sol-b-temporal_numeric-04",
@@ -3434,7 +3434,7 @@
3434
  "narrow_fail": 0.5702,
3435
  "pass": 0.2548
3436
  },
3437
- "latency_s": 0.068
3438
  },
3439
  {
3440
  "id": "hard-sol-c-judge_hard-03",
@@ -3448,7 +3448,7 @@
3448
  "yes": 0.806,
3449
  "no": 0.194
3450
  },
3451
- "latency_s": 0.069
3452
  },
3453
  {
3454
  "id": "hard-sol-c-judge_hard-05",
@@ -3476,7 +3476,7 @@
3476
  "yes": 0.658,
3477
  "no": 0.342
3478
  },
3479
- "latency_s": 0.068
3480
  },
3481
  {
3482
  "id": "hard-sol-c-judge_hard-08",
@@ -3490,7 +3490,7 @@
3490
  "yes": 0.9158,
3491
  "no": 0.0842
3492
  },
3493
- "latency_s": 0.068
3494
  },
3495
  {
3496
  "id": "hard-sol-c-judge_hard-09",
@@ -3504,7 +3504,7 @@
3504
  "yes": 0.8809,
3505
  "no": 0.1191
3506
  },
3507
- "latency_s": 0.068
3508
  },
3509
  {
3510
  "id": "hard-sol-c-judge_hard-10",
@@ -3518,7 +3518,7 @@
3518
  "yes": 0.8289,
3519
  "no": 0.1711
3520
  },
3521
- "latency_s": 0.069
3522
  },
3523
  {
3524
  "id": "hard-sol-c-judge_hard-11",
@@ -3560,7 +3560,7 @@
3560
  "yes": 0.9432,
3561
  "no": 0.0568
3562
  },
3563
- "latency_s": 0.069
3564
  },
3565
  {
3566
  "id": "hard-sol-c-multi_hop-07",
@@ -3592,7 +3592,7 @@
3592
  "pass_fen_cross": 0.5778,
3593
  "terminate_oakmere": 0.0354
3594
  },
3595
- "latency_s": 0.086
3596
  },
3597
  {
3598
  "id": "hard-sol-c-multi_hop-11",
@@ -3608,7 +3608,7 @@
3608
  "coverage_excluded": 0.5061,
3609
  "documentation_needed": 0.0531
3610
  },
3611
- "latency_s": 0.089
3612
  },
3613
  {
3614
  "id": "hard-sol-c-multi_hop-12",
@@ -3624,7 +3624,7 @@
3624
  "digital_copy_only": 0.1412,
3625
  "supervised_delivery": 0.635
3626
  },
3627
- "latency_s": 0.07
3628
  },
3629
  {
3630
  "id": "hard-sol-c-multi_hop-13",
 
5
  "n": 48,
6
  "accuracy": 1.0,
7
  "invalid": 0,
8
+ "p50_s": 0.069
9
  },
10
  "standard": {
11
  "n": 72,
12
  "accuracy": 0.8889,
13
  "invalid": 0,
14
+ "p50_s": 0.07
15
  },
16
  "hard": {
17
  "n": 111,
18
  "accuracy": 0.5676,
19
  "invalid": 0,
20
+ "p50_s": 0.072
21
  }
22
  },
23
  "rows": [
 
36
  "report_damage": 0.0001,
37
  "track_order": 0.9993
38
  },
39
+ "latency_s": 50.159
40
  },
41
  {
42
  "id": "easy-intent-01",
 
53
  "report_damage": 0.0001,
54
  "track_order": 0.0001
55
  },
56
+ "latency_s": 0.071
57
  },
58
  {
59
  "id": "easy-intent-02",
 
70
  "report_damage": 0.0002,
71
  "track_order": 0.0002
72
  },
73
+ "latency_s": 0.074
74
  },
75
  {
76
  "id": "easy-intent-03",
 
87
  "report_damage": 0.9926,
88
  "track_order": 0.0007
89
  },
90
+ "latency_s": 0.069
91
  },
92
  {
93
  "id": "easy-intent-04",
 
104
  "report_damage": 0.0016,
105
  "track_order": 0.0016
106
  },
107
+ "latency_s": 0.069
108
  },
109
  {
110
  "id": "easy-intent-05",
 
121
  "report_damage": 0.0002,
122
  "track_order": 0.9983
123
  },
124
+ "latency_s": 0.068
125
  },
126
  {
127
  "id": "easy-intent-06",
 
138
  "turn_off_lights": 0.0004,
139
  "weather": 0.0002
140
  },
141
+ "latency_s": 0.068
142
  },
143
  {
144
  "id": "easy-intent-07",
 
155
  "turn_off_lights": 0.001,
156
  "weather": 0.0008
157
  },
158
+ "latency_s": 0.068
159
  },
160
  {
161
  "id": "easy-intent-08",
 
172
  "turn_off_lights": 0.0,
173
  "weather": 0.9996
174
  },
175
+ "latency_s": 0.068
176
  },
177
  {
178
  "id": "easy-intent-09",
 
189
  "turn_off_lights": 0.0011,
190
  "weather": 0.0013
191
  },
192
+ "latency_s": 0.069
193
  },
194
  {
195
  "id": "easy-intent-10",
 
206
  "turn_off_lights": 0.9972,
207
  "weather": 0.0004
208
  },
209
+ "latency_s": 0.069
210
  },
211
  {
212
  "id": "easy-intent-11",
 
223
  "turn_off_lights": 0.0003,
224
  "weather": 0.0002
225
  },
226
+ "latency_s": 0.068
227
  },
228
  {
229
  "id": "easy-fact-00",
 
237
  "yes": 0.9527,
238
  "no": 0.0473
239
  },
240
+ "latency_s": 0.068
241
  },
242
  {
243
  "id": "easy-fact-01",
 
251
  "yes": 0.0186,
252
  "no": 0.9814
253
  },
254
+ "latency_s": 0.068
255
  },
256
  {
257
  "id": "easy-fact-02",
 
265
  "yes": 0.966,
266
  "no": 0.034
267
  },
268
+ "latency_s": 0.069
269
  },
270
  {
271
  "id": "easy-fact-03",
 
279
  "yes": 0.0094,
280
  "no": 0.9906
281
  },
282
+ "latency_s": 0.068
283
  },
284
  {
285
  "id": "easy-fact-04",
 
293
  "yes": 0.9685,
294
  "no": 0.0315
295
  },
296
+ "latency_s": 0.069
297
  },
298
  {
299
  "id": "easy-fact-05",
 
307
  "yes": 0.038,
308
  "no": 0.962
309
  },
310
+ "latency_s": 0.069
311
  },
312
  {
313
  "id": "easy-fact-06",
 
321
  "yes": 0.9472,
322
  "no": 0.0528
323
  },
324
+ "latency_s": 0.069
325
  },
326
  {
327
  "id": "easy-fact-07",
 
335
  "yes": 0.0217,
336
  "no": 0.9783
337
  },
338
+ "latency_s": 0.068
339
  },
340
  {
341
  "id": "easy-fact-08",
 
349
  "yes": 0.9757,
350
  "no": 0.0243
351
  },
352
+ "latency_s": 0.077
353
  },
354
  {
355
  "id": "easy-fact-09",
 
363
  "yes": 0.0179,
364
  "no": 0.9821
365
  },
366
+ "latency_s": 0.069
367
  },
368
  {
369
  "id": "easy-fact-10",
 
377
  "yes": 0.9269,
378
  "no": 0.0731
379
  },
380
+ "latency_s": 0.068
381
  },
382
  {
383
  "id": "easy-fact-11",
 
391
  "yes": 0.0424,
392
  "no": 0.9576
393
  },
394
+ "latency_s": 0.068
395
  },
396
  {
397
  "id": "easy-extraction-00",
 
407
  "credit_card": 0.0015,
408
  "paypal": 0.9976
409
  },
410
+ "latency_s": 0.082
411
  },
412
  {
413
  "id": "easy-extraction-01",
 
423
  "credit_card": 0.0054,
424
  "paypal": 0.0007
425
  },
426
+ "latency_s": 0.07
427
  },
428
  {
429
  "id": "easy-extraction-02",
 
439
  "credit_card": 0.0014,
440
  "paypal": 0.001
441
  },
442
+ "latency_s": 0.069
443
  },
444
  {
445
  "id": "easy-extraction-03",
 
455
  "medium": 0.0172,
456
  "urgent": 0.0624
457
  },
458
+ "latency_s": 0.068
459
  },
460
  {
461
  "id": "easy-extraction-04",
 
471
  "medium": 0.0134,
472
  "urgent": 0.0016
473
  },
474
+ "latency_s": 0.068
475
  },
476
  {
477
  "id": "easy-extraction-05",
 
487
  "medium": 0.0033,
488
  "urgent": 0.9838
489
  },
490
+ "latency_s": 0.069
491
  },
492
  {
493
  "id": "easy-extraction-06",
 
503
  "S": 0.0021,
504
  "XL": 0.0015
505
  },
506
+ "latency_s": 0.068
507
  },
508
  {
509
  "id": "easy-extraction-07",
 
519
  "S": 0.0009,
520
  "XL": 0.9929
521
  },
522
+ "latency_s": 0.068
523
  },
524
  {
525
  "id": "easy-extraction-08",
 
536
  "tuesday": 0.0017,
537
  "wednesday": 0.0013
538
  },
539
+ "latency_s": 0.068
540
  },
541
  {
542
  "id": "easy-extraction-09",
 
553
  "tuesday": 0.0275,
554
  "wednesday": 0.017
555
  },
556
+ "latency_s": 0.071
557
  },
558
  {
559
  "id": "easy-extraction-10",
 
570
  "tuesday": 0.0053,
571
  "wednesday": 0.0051
572
  },
573
+ "latency_s": 0.066
574
  },
575
  {
576
  "id": "easy-extraction-11",
 
586
  "credit_card": 0.9978,
587
  "paypal": 0.001
588
  },
589
+ "latency_s": 0.068
590
  },
591
  {
592
  "id": "easy-tool_selection-00",
 
603
  "send_email": 0.0002,
604
  "translate_text": 0.0003
605
  },
606
+ "latency_s": 0.069
607
  },
608
  {
609
  "id": "easy-tool_selection-01",
 
620
  "send_email": 0.9983,
621
  "translate_text": 0.0006
622
  },
623
+ "latency_s": 0.07
624
  },
625
  {
626
  "id": "easy-tool_selection-02",
 
637
  "send_email": 0.0003,
638
  "translate_text": 0.0004
639
  },
640
+ "latency_s": 0.07
641
  },
642
  {
643
  "id": "easy-tool_selection-03",
 
654
  "send_email": 0.0004,
655
  "translate_text": 0.0007
656
  },
657
+ "latency_s": 0.069
658
  },
659
  {
660
  "id": "easy-tool_selection-04",
 
671
  "send_email": 0.0001,
672
  "translate_text": 0.9993
673
  },
674
+ "latency_s": 0.069
675
  },
676
  {
677
  "id": "easy-tool_selection-05",
 
688
  "send_email": 0.0007,
689
  "translate_text": 0.0004
690
  },
691
+ "latency_s": 0.069
692
  },
693
  {
694
  "id": "easy-tool_selection-06",
 
705
  "search_flights": 0.9993,
706
  "set_reminder": 0.0003
707
  },
708
+ "latency_s": 0.07
709
  },
710
  {
711
  "id": "easy-tool_selection-07",
 
722
  "search_flights": 0.0006,
723
  "set_reminder": 0.003
724
  },
725
+ "latency_s": 0.07
726
  },
727
  {
728
  "id": "easy-tool_selection-08",
 
739
  "search_flights": 0.0002,
740
  "set_reminder": 0.999
741
  },
742
+ "latency_s": 0.071
743
  },
744
  {
745
  "id": "easy-tool_selection-09",
 
756
  "search_flights": 0.0002,
757
  "set_reminder": 0.0003
758
  },
759
+ "latency_s": 0.07
760
  },
761
  {
762
  "id": "easy-tool_selection-10",
 
773
  "search_flights": 0.0006,
774
  "set_reminder": 0.0012
775
  },
776
+ "latency_s": 0.07
777
  },
778
  {
779
  "id": "easy-tool_selection-11",
 
790
  "search_flights": 0.0001,
791
  "set_reminder": 0.9994
792
  },
793
+ "latency_s": 0.07
794
  },
795
  {
796
  "id": "original-policy-01-0",
 
804
  "yes": 0.0456,
805
  "no": 0.9544
806
  },
807
+ "latency_s": 0.07
808
  },
809
  {
810
  "id": "original-policy-01-1",
 
818
  "yes": 0.0473,
819
  "no": 0.9527
820
  },
821
+ "latency_s": 0.069
822
  },
823
  {
824
  "id": "original-policy-02-0",
 
832
  "yes": 0.8683,
833
  "no": 0.1317
834
  },
835
+ "latency_s": 0.069
836
  },
837
  {
838
  "id": "original-policy-02-1",
 
846
  "yes": 0.2921,
847
  "no": 0.7079
848
  },
849
+ "latency_s": 0.069
850
  },
851
  {
852
  "id": "original-policy-03-0",
 
860
  "yes": 0.5192,
861
  "no": 0.4808
862
  },
863
+ "latency_s": 0.069
864
  },
865
  {
866
  "id": "original-policy-03-1",
 
874
  "yes": 0.3957,
875
  "no": 0.6043
876
  },
877
+ "latency_s": 0.07
878
  },
879
  {
880
  "id": "original-policy-04-0",
 
888
  "yes": 0.1605,
889
  "no": 0.8395
890
  },
891
+ "latency_s": 0.076
892
  },
893
  {
894
  "id": "original-policy-04-1",
 
902
  "yes": 0.0785,
903
  "no": 0.9215
904
  },
905
+ "latency_s": 0.069
906
  },
907
  {
908
  "id": "original-policy-05-0",
 
916
  "yes": 0.3249,
917
  "no": 0.6751
918
  },
919
+ "latency_s": 0.069
920
  },
921
  {
922
  "id": "original-policy-05-1",
 
944
  "yes": 0.8234,
945
  "no": 0.1766
946
  },
947
+ "latency_s": 0.068
948
  },
949
  {
950
  "id": "original-policy-06-1",
 
975
  "refund": 0.9348,
976
  "status": 0.0006
977
  },
978
+ "latency_s": 0.069
979
  },
980
  {
981
  "id": "original-intent-01-1",
 
1009
  "refund": 0.0025,
1010
  "status": 0.7357
1011
  },
1012
+ "latency_s": 0.069
1013
  },
1014
  {
1015
  "id": "original-intent-02-1",
 
1026
  "refund": 0.0003,
1027
  "status": 0.9801
1028
  },
1029
+ "latency_s": 0.069
1030
  },
1031
  {
1032
  "id": "original-intent-03-0",
 
1043
  "refund": 0.0073,
1044
  "status": 0.0073
1045
  },
1046
+ "latency_s": 0.069
1047
  },
1048
  {
1049
  "id": "original-intent-03-1",
 
1060
  "refund": 0.0007,
1061
  "status": 0.0003
1062
  },
1063
+ "latency_s": 0.081
1064
  },
1065
  {
1066
  "id": "original-intent-04-0",
 
1077
  "refund": 0.0129,
1078
  "status": 0.0007
1079
  },
1080
+ "latency_s": 0.071
1081
  },
1082
  {
1083
  "id": "original-intent-04-1",
 
1094
  "refund": 0.0047,
1095
  "status": 0.0006
1096
  },
1097
+ "latency_s": 0.07
1098
  },
1099
  {
1100
  "id": "original-intent-05-0",
 
1111
  "refund": 0.2237,
1112
  "status": 0.0017
1113
  },
1114
+ "latency_s": 0.07
1115
  },
1116
  {
1117
  "id": "original-intent-05-1",
 
1128
  "refund": 0.3631,
1129
  "status": 0.0019
1130
  },
1131
+ "latency_s": 0.07
1132
  },
1133
  {
1134
  "id": "original-intent-06-0",
 
1145
  "refund": 0.0009,
1146
  "status": 0.9703
1147
  },
1148
+ "latency_s": 0.069
1149
  },
1150
  {
1151
  "id": "original-intent-06-1",
 
1162
  "refund": 0.0007,
1163
  "status": 0.9817
1164
  },
1165
+ "latency_s": 0.072
1166
  },
1167
  {
1168
  "id": "original-ordinal-01-0",
 
1178
  "2": 0.0062,
1179
  "3": 0.0032
1180
  },
1181
+ "latency_s": 0.823
1182
  },
1183
  {
1184
  "id": "original-ordinal-01-1",
 
1194
  "2": 0.01,
1195
  "3": 0.0031
1196
  },
1197
+ "latency_s": 0.073
1198
  },
1199
  {
1200
  "id": "original-ordinal-02-0",
 
1210
  "2": 0.1319,
1211
  "3": 0.008
1212
  },
1213
+ "latency_s": 0.074
1214
  },
1215
  {
1216
  "id": "original-ordinal-02-1",
 
1226
  "2": 0.0164,
1227
  "3": 0.0095
1228
  },
1229
+ "latency_s": 0.074
1230
  },
1231
  {
1232
  "id": "original-ordinal-03-0",
 
1242
  "2": 0.7779,
1243
  "3": 0.0051
1244
  },
1245
+ "latency_s": 0.072
1246
  },
1247
  {
1248
  "id": "original-ordinal-03-1",
 
1258
  "2": 0.7353,
1259
  "3": 0.0151
1260
  },
1261
+ "latency_s": 0.072
1262
  },
1263
  {
1264
  "id": "original-ordinal-04-0",
 
1274
  "2": 0.0429,
1275
  "3": 0.9247
1276
  },
1277
+ "latency_s": 0.073
1278
  },
1279
  {
1280
  "id": "original-ordinal-04-1",
 
1290
  "2": 0.1234,
1291
  "3": 0.824
1292
  },
1293
+ "latency_s": 0.073
1294
  },
1295
  {
1296
  "id": "original-ordinal-05-0",
 
1306
  "2": 0.0323,
1307
  "3": 0.0052
1308
  },
1309
+ "latency_s": 0.071
1310
  },
1311
  {
1312
  "id": "original-ordinal-05-1",
 
1322
  "2": 0.1294,
1323
  "3": 0.021
1324
  },
1325
+ "latency_s": 0.071
1326
  },
1327
  {
1328
  "id": "original-ordinal-06-0",
 
1338
  "2": 0.7511,
1339
  "3": 0.0022
1340
  },
1341
+ "latency_s": 0.071
1342
  },
1343
  {
1344
  "id": "original-ordinal-06-1",
 
1354
  "2": 0.7949,
1355
  "3": 0.0026
1356
  },
1357
+ "latency_s": 0.071
1358
  },
1359
  {
1360
  "id": "original-extraction-01-0",
 
1370
  "post": 0.0063,
1371
  "unknown": 0.0929
1372
  },
1373
+ "latency_s": 0.07
1374
  },
1375
  {
1376
  "id": "original-extraction-01-1",
 
1386
  "post": 0.0065,
1387
  "unknown": 0.081
1388
  },
1389
+ "latency_s": 0.07
1390
  },
1391
  {
1392
  "id": "original-extraction-02-0",
 
1402
  "post": 0.1995,
1403
  "unknown": 0.7637
1404
  },
1405
+ "latency_s": 0.072
1406
  },
1407
  {
1408
  "id": "original-extraction-02-1",
 
1418
  "post": 0.0732,
1419
  "unknown": 0.9133
1420
  },
1421
+ "latency_s": 0.076
1422
  },
1423
  {
1424
  "id": "original-extraction-03-0",
 
1434
  "post": 0.0344,
1435
  "unknown": 0.295
1436
  },
1437
+ "latency_s": 0.074
1438
  },
1439
  {
1440
  "id": "original-extraction-03-1",
 
1450
  "post": 0.0051,
1451
  "unknown": 0.0092
1452
  },
1453
+ "latency_s": 0.072
1454
  },
1455
  {
1456
  "id": "original-extraction-04-0",
 
1466
  "post": 0.9733,
1467
  "unknown": 0.0227
1468
  },
1469
+ "latency_s": 0.071
1470
  },
1471
  {
1472
  "id": "original-extraction-04-1",
 
1482
  "post": 0.9545,
1483
  "unknown": 0.0361
1484
  },
1485
+ "latency_s": 0.071
1486
  },
1487
  {
1488
  "id": "original-extraction-05-0",
 
1498
  "post": 0.0037,
1499
  "unknown": 0.9846
1500
  },
1501
+ "latency_s": 0.071
1502
  },
1503
  {
1504
  "id": "original-extraction-05-1",
 
1514
  "post": 0.0181,
1515
  "unknown": 0.6251
1516
  },
1517
+ "latency_s": 0.071
1518
  },
1519
  {
1520
  "id": "original-extraction-06-0",
 
1530
  "post": 0.0077,
1531
  "unknown": 0.0225
1532
  },
1533
+ "latency_s": 0.071
1534
  },
1535
  {
1536
  "id": "original-extraction-06-1",
 
1546
  "post": 0.0048,
1547
  "unknown": 0.0204
1548
  },
1549
+ "latency_s": 0.072
1550
  },
1551
  {
1552
  "id": "original-adequacy-01-0",
 
1560
  "yes": 0.8447,
1561
  "no": 0.1553
1562
  },
1563
+ "latency_s": 0.069
1564
  },
1565
  {
1566
  "id": "original-adequacy-01-1",
 
1574
  "yes": 0.746,
1575
  "no": 0.254
1576
  },
1577
+ "latency_s": 0.069
1578
  },
1579
  {
1580
  "id": "original-adequacy-02-0",
 
1588
  "yes": 0.1362,
1589
  "no": 0.8638
1590
  },
1591
+ "latency_s": 0.068
1592
  },
1593
  {
1594
  "id": "original-adequacy-02-1",
 
1602
  "yes": 0.1605,
1603
  "no": 0.8395
1604
  },
1605
+ "latency_s": 0.069
1606
  },
1607
  {
1608
  "id": "original-adequacy-03-0",
 
1616
  "yes": 0.7873,
1617
  "no": 0.2127
1618
  },
1619
+ "latency_s": 0.068
1620
  },
1621
  {
1622
  "id": "original-adequacy-03-1",
 
1630
  "yes": 0.6999,
1631
  "no": 0.3001
1632
  },
1633
+ "latency_s": 0.068
1634
  },
1635
  {
1636
  "id": "original-adequacy-04-0",
 
1644
  "yes": 0.8343,
1645
  "no": 0.1657
1646
  },
1647
+ "latency_s": 0.069
1648
  },
1649
  {
1650
  "id": "original-adequacy-04-1",
 
1658
  "yes": 0.6751,
1659
  "no": 0.3249
1660
  },
1661
+ "latency_s": 0.069
1662
  },
1663
  {
1664
  "id": "original-adequacy-05-0",
 
1672
  "yes": 0.548,
1673
  "no": 0.452
1674
  },
1675
+ "latency_s": 0.069
1676
  },
1677
  {
1678
  "id": "original-adequacy-05-1",
 
1686
  "yes": 0.6917,
1687
  "no": 0.3083
1688
  },
1689
+ "latency_s": 0.069
1690
  },
1691
  {
1692
  "id": "original-adequacy-06-0",
 
1700
  "yes": 0.548,
1701
  "no": 0.452
1702
  },
1703
+ "latency_s": 0.077
1704
  },
1705
  {
1706
  "id": "original-adequacy-06-1",
 
1714
  "yes": 0.595,
1715
  "no": 0.405
1716
  },
1717
+ "latency_s": 0.071
1718
  },
1719
  {
1720
  "id": "original-routing-01-0",
 
1732
  "math": 0.9149,
1733
  "tools": 0.0056
1734
  },
1735
+ "latency_s": 0.071
1736
  },
1737
  {
1738
  "id": "original-routing-01-1",
 
1750
  "math": 0.9137,
1751
  "tools": 0.0036
1752
  },
1753
+ "latency_s": 0.073
1754
  },
1755
  {
1756
  "id": "original-routing-02-0",
 
1768
  "math": 0.0023,
1769
  "tools": 0.003
1770
  },
1771
+ "latency_s": 0.072
1772
  },
1773
  {
1774
  "id": "original-routing-02-1",
 
1786
  "math": 0.0035,
1787
  "tools": 0.0024
1788
  },
1789
+ "latency_s": 0.07
1790
  },
1791
  {
1792
  "id": "original-routing-03-0",
 
1804
  "math": 0.0008,
1805
  "tools": 0.0045
1806
  },
1807
+ "latency_s": 0.07
1808
  },
1809
  {
1810
  "id": "original-routing-03-1",
 
1822
  "math": 0.0007,
1823
  "tools": 0.0047
1824
  },
1825
+ "latency_s": 0.07
1826
  },
1827
  {
1828
  "id": "original-routing-04-0",
 
1840
  "math": 0.002,
1841
  "tools": 0.0166
1842
  },
1843
+ "latency_s": 0.069
1844
  },
1845
  {
1846
  "id": "original-routing-04-1",
 
1858
  "math": 0.0016,
1859
  "tools": 0.0119
1860
  },
1861
+ "latency_s": 0.069
1862
  },
1863
  {
1864
  "id": "original-routing-05-0",
 
1876
  "math": 0.0018,
1877
  "tools": 0.4325
1878
  },
1879
+ "latency_s": 0.069
1880
  },
1881
  {
1882
  "id": "original-routing-05-1",
 
1894
  "math": 0.0012,
1895
  "tools": 0.1126
1896
  },
1897
+ "latency_s": 0.068
1898
  },
1899
  {
1900
  "id": "original-routing-06-0",
 
1912
  "math": 0.0012,
1913
  "tools": 0.0025
1914
  },
1915
+ "latency_s": 0.069
1916
  },
1917
  {
1918
  "id": "original-routing-06-1",
 
1930
  "math": 0.0012,
1931
  "tools": 0.0036
1932
  },
1933
+ "latency_s": 0.068
1934
  },
1935
  {
1936
  "id": "hard-opus-a-long_policy-01",
 
1947
  "pay_subject_to_10000_sublimit": 0.0233,
1948
  "pay_subject_to_15000_sublimit": 0.949
1949
  },
1950
+ "latency_s": 0.165
1951
  },
1952
  {
1953
  "id": "hard-opus-a-long_policy-04",
 
1964
  "executive_committee": 0.0028,
1965
  "vp_and_finance_director": 0.6234
1966
  },
1967
+ "latency_s": 0.105
1968
  },
1969
  {
1970
  "id": "hard-opus-a-long_policy-08",
 
1996
  "no_license_required": 0.1142,
1997
  "shipment_prohibited": 0.0194
1998
  },
1999
+ "latency_s": 0.089
2000
  },
2001
  {
2002
  "id": "hard-opus-a-long_policy-11",
 
2010
  "yes": 0.0681,
2011
  "no": 0.9319
2012
  },
2013
+ "latency_s": 0.084
2014
  },
2015
  {
2016
  "id": "hard-opus-a-long_policy-13",
 
2024
  "yes": 0.3596,
2025
  "no": 0.6404
2026
  },
2027
+ "latency_s": 0.087
2028
  },
2029
  {
2030
  "id": "hard-opus-a-long_policy-17",
 
2041
  "security_incident_response": 0.0054,
2042
  "vendor_support": 0.06
2043
  },
2044
+ "latency_s": 0.085
2045
  },
2046
  {
2047
  "id": "hard-opus-a-long_policy-19",
 
2084
  "late": 0.1989,
2085
  "on_time": 0.7614
2086
  },
2087
+ "latency_s": 0.075
2088
  },
2089
  {
2090
  "id": "hard-opus-a-probability-07",
 
2099
  "reopened": 0.0377,
2100
  "resolved_first_contact": 0.1993
2101
  },
2102
+ "latency_s": 0.074
2103
  },
2104
  {
2105
  "id": "hard-opus-a-probability-08",
 
2113
  "yes": 0.3957,
2114
  "no": 0.6043
2115
  },
2116
+ "latency_s": 0.071
2117
  },
2118
  {
2119
  "id": "hard-opus-a-temporal_numeric-03",
 
2127
  "yes": 0.3165,
2128
  "no": 0.6835
2129
  },
2130
+ "latency_s": 0.071
2131
  },
2132
  {
2133
  "id": "hard-opus-a-temporal_numeric-06",
 
2172
  "yes": 0.548,
2173
  "no": 0.452
2174
  },
2175
+ "latency_s": 0.07
2176
  },
2177
  {
2178
  "id": "hard-opus-a-temporal_numeric-12",
 
2188
  "2": 0.4681,
2189
  "3": 0.0113
2190
  },
2191
+ "latency_s": 0.083
2192
  },
2193
  {
2194
  "id": "hard-opus-b-ambiguous-02",
 
2203
  "false_positive": 0.4813,
2204
  "real_incident": 0.4561
2205
  },
2206
+ "latency_s": 0.075
2207
  },
2208
  {
2209
  "id": "hard-opus-b-ambiguous-03",
 
2218
  "time_barred": 0.5371,
2219
  "within_limitation": 0.4333
2220
  },
2221
+ "latency_s": 0.072
2222
  },
2223
  {
2224
  "id": "hard-opus-b-ambiguous-07",
 
2233
  "over_limit": 0.1475,
2234
  "within_policy": 0.0734
2235
  },
2236
+ "latency_s": 0.071
2237
  },
2238
  {
2239
  "id": "hard-opus-b-ambiguous-09",
 
2248
  "inconclusive": 0.0149,
2249
  "legitimate": 0.4398
2250
  },
2251
+ "latency_s": 0.073
2252
  },
2253
  {
2254
  "id": "hard-opus-b-ambiguous-10",
 
2263
  "send_to_alex_moreno": 0.7646,
2264
  "send_to_alexandra_lind": 0.0246
2265
  },
2266
+ "latency_s": 0.072
2267
  },
2268
  {
2269
  "id": "hard-opus-b-ambiguous-11",
 
2278
  "grade_correct": 0.0157,
2279
  "grade_should_be_higher": 0.0048
2280
  },
2281
+ "latency_s": 0.071
2282
  },
2283
  {
2284
  "id": "hard-opus-b-ambiguous-13",
 
2294
  "charge_platform_team": 0.0051,
2295
  "split_50_50": 0.0057
2296
  },
2297
+ "latency_s": 0.072
2298
  },
2299
  {
2300
  "id": "hard-opus-b-multi_hop-03",
 
2328
  "esra": 0.0055,
2329
  "no_page_ticket_only": 0.8158
2330
  },
2331
+ "latency_s": 0.084
2332
  },
2333
  {
2334
  "id": "hard-opus-b-multi_hop-05",
 
2345
  "3": 0.0922,
2346
  "4": 0.0447
2347
  },
2348
+ "latency_s": 12.113
2349
  },
2350
  {
2351
  "id": "hard-opus-b-multi_hop-07",
 
2407
  "database_hardware": 0.0108,
2408
  "upstream_provider": 0.317
2409
  },
2410
+ "latency_s": 0.07
2411
  },
2412
  {
2413
  "id": "hard-opus-b-probability-03",
 
2421
  "yes": 0.6666,
2422
  "no": 0.3334
2423
  },
2424
+ "latency_s": 0.073
2425
  },
2426
  {
2427
  "id": "hard-opus-b-probability-04",
 
2468
  "process_refund": 0.0061,
2469
  "send_kb_article": 0.0018
2470
  },
2471
+ "latency_s": 0.074
2472
  },
2473
  {
2474
  "id": "hard-opus-b-tradeoff-03",
 
2484
  "order_c": 0.0186,
2485
  "order_d": 0.0374
2486
  },
2487
+ "latency_s": 0.075
2488
  },
2489
  {
2490
  "id": "hard-opus-b-tradeoff-06",
 
2498
  "yes": 0.1408,
2499
  "no": 0.8592
2500
  },
2501
+ "latency_s": 0.075
2502
  },
2503
  {
2504
  "id": "hard-opus-b-tradeoff-07",
 
2515
  "p2_fix_30d": 0.2351,
2516
  "p3_backlog": 0.0014
2517
  },
2518
+ "latency_s": 0.074
2519
  },
2520
  {
2521
  "id": "hard-opus-b-tradeoff-08",
 
2529
  "yes": 0.1408,
2530
  "no": 0.8592
2531
  },
2532
+ "latency_s": 0.072
2533
  },
2534
  {
2535
  "id": "hard-opus-b-tradeoff-12",
 
2546
  "refund_now": 0.9697,
2547
  "wait_for_inspection": 0.0096
2548
  },
2549
+ "latency_s": 0.071
2550
  },
2551
  {
2552
  "id": "hard-opus-c-long_policy-02",
 
2581
  "eur_400": 0.179,
2582
  "eur_600": 0.0357
2583
  },
2584
+ "latency_s": 0.122
2585
  },
2586
  {
2587
  "id": "hard-opus-c-long_policy-04",
 
2611
  "2": 0.4746,
2612
  "3": 0.0444
2613
  },
2614
+ "latency_s": 0.39
2615
  },
2616
  {
2617
  "id": "hard-opus-c-long_policy-08",
 
2627
  "2": 0.1642,
2628
  "3": 0.0503
2629
  },
2630
+ "latency_s": 0.393
2631
  },
2632
  {
2633
  "id": "hard-opus-c-long_policy-10",
 
2643
  "reject_return": 0.0882,
2644
  "repair_or_replace_only": 0.143
2645
  },
2646
+ "latency_s": 0.125
2647
  },
2648
  {
2649
  "id": "hard-opus-c-long_policy-11",
 
2686
  "yes": 0.1823,
2687
  "no": 0.8177
2688
  },
2689
+ "latency_s": 0.072
2690
  },
2691
  {
2692
  "id": "hard-opus-c-temporal_numeric-03",
 
2720
  "usd_2335_00": 0.0994,
2721
  "usd_2340_00": 0.0289
2722
  },
2723
+ "latency_s": 0.071
2724
  },
2725
  {
2726
  "id": "hard-opus-c-temporal_numeric-06",
 
2770
  "expired_24_month_term": 0.0958,
2771
  "expired_30_month_cap": 0.8658
2772
  },
2773
+ "latency_s": 0.072
2774
  },
2775
  {
2776
  "id": "hard-sol-a-adversarial-01",
 
2786
  "grant_access": 0.0039,
2787
  "grant_owner_exception": 0.0149
2788
  },
2789
+ "latency_s": 0.071
2790
  },
2791
  {
2792
  "id": "hard-sol-a-adversarial-06",
 
2800
  "yes": 0.0252,
2801
  "no": 0.9748
2802
  },
2803
+ "latency_s": 0.07
2804
  },
2805
  {
2806
  "id": "hard-sol-a-adversarial-07",
 
2816
  "retain_indefinitely": 0.0014,
2817
  "retain_until_2031": 0.9634
2818
  },
2819
+ "latency_s": 0.071
2820
  },
2821
  {
2822
  "id": "hard-sol-a-adversarial-08",
 
2830
  "yes": 0.9757,
2831
  "no": 0.0243
2832
  },
2833
+ "latency_s": 0.071
2834
  },
2835
  {
2836
  "id": "hard-sol-a-adversarial-09",
 
2846
  "2": 0.7697,
2847
  "3": 0.167
2848
  },
2849
+ "latency_s": 0.073
2850
  },
2851
  {
2852
  "id": "hard-sol-a-adversarial-11",
 
2862
  "manual_hold": 0.0041,
2863
  "standard_research": 0.6779
2864
  },
2865
+ "latency_s": 0.07
2866
  },
2867
  {
2868
  "id": "hard-sol-a-multi_hop-01",
 
2878
  "reject_tier": 0.1175,
2879
  "reroute_s3": 0.0498
2880
  },
2881
+ "latency_s": 0.087
2882
  },
2883
  {
2884
  "id": "hard-sol-a-multi_hop-02",
 
2894
  "insufficient_information": 0.0294,
2895
  "security_primary": 0.6623
2896
  },
2897
+ "latency_s": 0.088
2898
  },
2899
  {
2900
  "id": "hard-sol-a-multi_hop-05",
 
2910
  "require_director": 0.7348,
2911
  "require_finance_board": 0.0034
2912
  },
2913
+ "latency_s": 0.086
2914
  },
2915
  {
2916
  "id": "hard-sol-a-multi_hop-07",
 
2926
  "deny_score": 0.0837,
2927
  "manual_equivalency": 0.1872
2928
  },
2929
+ "latency_s": 0.086
2930
  },
2931
  {
2932
  "id": "hard-sol-a-multi_hop-08",
 
2942
  "local_maintenance": 0.1563,
2943
  "safety_shutdown": 0.005
2944
  },
2945
+ "latency_s": 0.086
2946
  },
2947
  {
2948
  "id": "hard-sol-a-multi_hop-09",
 
2974
  "needs_field_review": 0.0198,
2975
  "reduce_to_30_days": 0.4218
2976
  },
2977
+ "latency_s": 0.086
2978
  },
2979
  {
2980
  "id": "hard-sol-a-multi_hop-12",
 
3004
  "yes": 0.0105,
3005
  "no": 0.9895
3006
  },
3007
+ "latency_s": 0.07
3008
  },
3009
  {
3010
  "id": "hard-sol-a-trap-04",
 
3018
  "yes": 0.0813,
3019
  "no": 0.9187
3020
  },
3021
+ "latency_s": 0.07
3022
  },
3023
  {
3024
  "id": "hard-sol-a-trap-05",
 
3034
  "2": 0.3736,
3035
  "3": 0.007
3036
  },
3037
+ "latency_s": 0.075
3038
  },
3039
  {
3040
  "id": "hard-sol-a-trap-06",
 
3066
  "late": 0.9767,
3067
  "on_time": 0.0228
3068
  },
3069
+ "latency_s": 0.069
3070
  },
3071
  {
3072
  "id": "hard-sol-a-trap-10",
 
3082
  "ready_one_review": 0.0087,
3083
  "ready_two_reviews": 0.0542
3084
  },
3085
+ "latency_s": 0.069
3086
  },
3087
  {
3088
  "id": "hard-sol-a-trap-13",
 
3096
  "yes": 0.044,
3097
  "no": 0.956
3098
  },
3099
+ "latency_s": 0.069
3100
  },
3101
  {
3102
  "id": "hard-sol-a-trap-15",
 
3110
  "yes": 0.3596,
3111
  "no": 0.6404
3112
  },
3113
+ "latency_s": 0.069
3114
  },
3115
  {
3116
  "id": "hard-sol-b-judge_hard-01",
 
3124
  "yes": 0.8119,
3125
  "no": 0.1881
3126
  },
3127
+ "latency_s": 0.069
3128
  },
3129
  {
3130
  "id": "hard-sol-b-judge_hard-02",
 
3152
  "yes": 0.8726,
3153
  "no": 0.1274
3154
  },
3155
+ "latency_s": 0.068
3156
  },
3157
  {
3158
  "id": "hard-sol-b-judge_hard-08",
 
3166
  "yes": 0.8925,
3167
  "no": 0.1075
3168
  },
3169
+ "latency_s": 0.069
3170
  },
3171
  {
3172
  "id": "hard-sol-b-judge_hard-10",
 
3180
  "yes": 0.4712,
3181
  "no": 0.5288
3182
  },
3183
+ "latency_s": 0.068
3184
  },
3185
  {
3186
  "id": "hard-sol-b-judge_hard-14",
 
3222
  "yes": 0.9343,
3223
  "no": 0.0657
3224
  },
3225
+ "latency_s": 0.069
3226
  },
3227
  {
3228
  "id": "hard-sol-b-long_policy-01",
 
3254
  "privacy_and_security": 0.0669,
3255
  "privacy_only": 0.835
3256
  },
3257
+ "latency_s": 0.092
3258
  },
3259
  {
3260
  "id": "hard-sol-b-long_policy-05",
 
3286
  "no_award": 0.063,
3287
  "river_arts": 0.8743
3288
  },
3289
+ "latency_s": 0.092
3290
  },
3291
  {
3292
  "id": "hard-sol-b-routing_hard-01",
 
3303
  "office_file_repair": 0.987,
3304
  "spreadsheet_formula": 0.0092
3305
  },
3306
+ "latency_s": 0.071
3307
  },
3308
  {
3309
  "id": "hard-sol-b-routing_hard-02",
 
3320
  "mobile_app": 0.001,
3321
  "payment_disputes": 0.0051
3322
  },
3323
+ "latency_s": 0.07
3324
  },
3325
  {
3326
  "id": "hard-sol-b-routing_hard-03",
 
3337
  "network_dns": 0.9525,
3338
  "tls_certificates": 0.0017
3339
  },
3340
+ "latency_s": 0.071
3341
  },
3342
  {
3343
  "id": "hard-sol-b-routing_hard-07",
 
3354
  "macos_runtime": 0.0017,
3355
  "release_notarization": 0.9902
3356
  },
3357
+ "latency_s": 0.07
3358
  },
3359
  {
3360
  "id": "hard-sol-b-routing_hard-09",
 
3371
  "source_access": 0.9916,
3372
  "vpn_network": 0.0023
3373
  },
3374
+ "latency_s": 0.07
3375
  },
3376
  {
3377
  "id": "hard-sol-b-temporal_numeric-01",
 
3387
  "late_by_under_2h": 0.1855,
3388
  "within_window": 0.6043
3389
  },
3390
+ "latency_s": 0.07
3391
  },
3392
  {
3393
  "id": "hard-sol-b-temporal_numeric-02",
 
3403
  "over_limit": 0.1269,
3404
  "under_100gb_below": 0.541
3405
  },
3406
+ "latency_s": 0.07
3407
  },
3408
  {
3409
  "id": "hard-sol-b-temporal_numeric-03",
 
3419
  "6_credits": 0.1571,
3420
  "9_credits": 0.6016
3421
  },
3422
+ "latency_s": 0.069
3423
  },
3424
  {
3425
  "id": "hard-sol-b-temporal_numeric-04",
 
3434
  "narrow_fail": 0.5702,
3435
  "pass": 0.2548
3436
  },
3437
+ "latency_s": 0.069
3438
  },
3439
  {
3440
  "id": "hard-sol-c-judge_hard-03",
 
3448
  "yes": 0.806,
3449
  "no": 0.194
3450
  },
3451
+ "latency_s": 0.07
3452
  },
3453
  {
3454
  "id": "hard-sol-c-judge_hard-05",
 
3476
  "yes": 0.658,
3477
  "no": 0.342
3478
  },
3479
+ "latency_s": 0.069
3480
  },
3481
  {
3482
  "id": "hard-sol-c-judge_hard-08",
 
3490
  "yes": 0.9158,
3491
  "no": 0.0842
3492
  },
3493
+ "latency_s": 0.07
3494
  },
3495
  {
3496
  "id": "hard-sol-c-judge_hard-09",
 
3504
  "yes": 0.8809,
3505
  "no": 0.1191
3506
  },
3507
+ "latency_s": 0.07
3508
  },
3509
  {
3510
  "id": "hard-sol-c-judge_hard-10",
 
3518
  "yes": 0.8289,
3519
  "no": 0.1711
3520
  },
3521
+ "latency_s": 0.07
3522
  },
3523
  {
3524
  "id": "hard-sol-c-judge_hard-11",
 
3560
  "yes": 0.9432,
3561
  "no": 0.0568
3562
  },
3563
+ "latency_s": 0.07
3564
  },
3565
  {
3566
  "id": "hard-sol-c-multi_hop-07",
 
3592
  "pass_fen_cross": 0.5778,
3593
  "terminate_oakmere": 0.0354
3594
  },
3595
+ "latency_s": 0.088
3596
  },
3597
  {
3598
  "id": "hard-sol-c-multi_hop-11",
 
3608
  "coverage_excluded": 0.5061,
3609
  "documentation_needed": 0.0531
3610
  },
3611
+ "latency_s": 0.091
3612
  },
3613
  {
3614
  "id": "hard-sol-c-multi_hop-12",
 
3624
  "digital_copy_only": 0.1412,
3625
  "supervised_delivery": 0.635
3626
  },
3627
+ "latency_s": 0.071
3628
  },
3629
  {
3630
  "id": "hard-sol-c-multi_hop-13",
eval/jevbench_public231_previous_delta-5k.json ADDED
@@ -0,0 +1,3646 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "/content/delta5k",
3
+ "summary": {
4
+ "easy": {
5
+ "n": 48,
6
+ "accuracy": 1.0,
7
+ "invalid": 0,
8
+ "p50_s": 0.068
9
+ },
10
+ "standard": {
11
+ "n": 72,
12
+ "accuracy": 0.8889,
13
+ "invalid": 0,
14
+ "p50_s": 0.068
15
+ },
16
+ "hard": {
17
+ "n": 111,
18
+ "accuracy": 0.6667,
19
+ "invalid": 0,
20
+ "p50_s": 0.073
21
+ }
22
+ },
23
+ "rows": [
24
+ {
25
+ "id": "easy-intent-00",
26
+ "tier": "easy",
27
+ "expected": "track_order",
28
+ "predicted": "track_order",
29
+ "correct": true,
30
+ "valid": true,
31
+ "error": null,
32
+ "probs": {
33
+ "billing_question": 0.0,
34
+ "cancel_order": 0.0,
35
+ "change_address": 0.0,
36
+ "report_damage": 0.0,
37
+ "track_order": 0.9999
38
+ },
39
+ "latency_s": 1.986
40
+ },
41
+ {
42
+ "id": "easy-intent-01",
43
+ "tier": "easy",
44
+ "expected": "cancel_order",
45
+ "predicted": "cancel_order",
46
+ "correct": true,
47
+ "valid": true,
48
+ "error": null,
49
+ "probs": {
50
+ "billing_question": 0.0,
51
+ "cancel_order": 0.9999,
52
+ "change_address": 0.0,
53
+ "report_damage": 0.0,
54
+ "track_order": 0.0
55
+ },
56
+ "latency_s": 0.072
57
+ },
58
+ {
59
+ "id": "easy-intent-02",
60
+ "tier": "easy",
61
+ "expected": "change_address",
62
+ "predicted": "change_address",
63
+ "correct": true,
64
+ "valid": true,
65
+ "error": null,
66
+ "probs": {
67
+ "billing_question": 0.0,
68
+ "cancel_order": 0.0001,
69
+ "change_address": 0.9998,
70
+ "report_damage": 0.0001,
71
+ "track_order": 0.0
72
+ },
73
+ "latency_s": 0.068
74
+ },
75
+ {
76
+ "id": "easy-intent-03",
77
+ "tier": "easy",
78
+ "expected": "report_damage",
79
+ "predicted": "report_damage",
80
+ "correct": true,
81
+ "valid": true,
82
+ "error": null,
83
+ "probs": {
84
+ "billing_question": 0.0,
85
+ "cancel_order": 0.0004,
86
+ "change_address": 0.0001,
87
+ "report_damage": 0.9995,
88
+ "track_order": 0.0001
89
+ },
90
+ "latency_s": 0.068
91
+ },
92
+ {
93
+ "id": "easy-intent-04",
94
+ "tier": "easy",
95
+ "expected": "billing_question",
96
+ "predicted": "billing_question",
97
+ "correct": true,
98
+ "valid": true,
99
+ "error": null,
100
+ "probs": {
101
+ "billing_question": 0.998,
102
+ "cancel_order": 0.001,
103
+ "change_address": 0.0004,
104
+ "report_damage": 0.0003,
105
+ "track_order": 0.0002
106
+ },
107
+ "latency_s": 0.068
108
+ },
109
+ {
110
+ "id": "easy-intent-05",
111
+ "tier": "easy",
112
+ "expected": "track_order",
113
+ "predicted": "track_order",
114
+ "correct": true,
115
+ "valid": true,
116
+ "error": null,
117
+ "probs": {
118
+ "billing_question": 0.0,
119
+ "cancel_order": 0.0001,
120
+ "change_address": 0.0001,
121
+ "report_damage": 0.0001,
122
+ "track_order": 0.9997
123
+ },
124
+ "latency_s": 0.068
125
+ },
126
+ {
127
+ "id": "easy-intent-06",
128
+ "tier": "easy",
129
+ "expected": "set_alarm",
130
+ "predicted": "set_alarm",
131
+ "correct": true,
132
+ "valid": true,
133
+ "error": null,
134
+ "probs": {
135
+ "play_music": 0.0,
136
+ "send_message": 0.0001,
137
+ "set_alarm": 0.9997,
138
+ "turn_off_lights": 0.0001,
139
+ "weather": 0.0001
140
+ },
141
+ "latency_s": 0.069
142
+ },
143
+ {
144
+ "id": "easy-intent-07",
145
+ "tier": "easy",
146
+ "expected": "play_music",
147
+ "predicted": "play_music",
148
+ "correct": true,
149
+ "valid": true,
150
+ "error": null,
151
+ "probs": {
152
+ "play_music": 0.9994,
153
+ "send_message": 0.0002,
154
+ "set_alarm": 0.0002,
155
+ "turn_off_lights": 0.0001,
156
+ "weather": 0.0001
157
+ },
158
+ "latency_s": 0.069
159
+ },
160
+ {
161
+ "id": "easy-intent-08",
162
+ "tier": "easy",
163
+ "expected": "weather",
164
+ "predicted": "weather",
165
+ "correct": true,
166
+ "valid": true,
167
+ "error": null,
168
+ "probs": {
169
+ "play_music": 0.0,
170
+ "send_message": 0.0,
171
+ "set_alarm": 0.0,
172
+ "turn_off_lights": 0.0,
173
+ "weather": 1.0
174
+ },
175
+ "latency_s": 0.069
176
+ },
177
+ {
178
+ "id": "easy-intent-09",
179
+ "tier": "easy",
180
+ "expected": "send_message",
181
+ "predicted": "send_message",
182
+ "correct": true,
183
+ "valid": true,
184
+ "error": null,
185
+ "probs": {
186
+ "play_music": 0.0001,
187
+ "send_message": 0.9993,
188
+ "set_alarm": 0.0002,
189
+ "turn_off_lights": 0.0002,
190
+ "weather": 0.0002
191
+ },
192
+ "latency_s": 0.068
193
+ },
194
+ {
195
+ "id": "easy-intent-10",
196
+ "tier": "easy",
197
+ "expected": "turn_off_lights",
198
+ "predicted": "turn_off_lights",
199
+ "correct": true,
200
+ "valid": true,
201
+ "error": null,
202
+ "probs": {
203
+ "play_music": 0.0001,
204
+ "send_message": 0.0001,
205
+ "set_alarm": 0.0002,
206
+ "turn_off_lights": 0.9994,
207
+ "weather": 0.0001
208
+ },
209
+ "latency_s": 0.068
210
+ },
211
+ {
212
+ "id": "easy-intent-11",
213
+ "tier": "easy",
214
+ "expected": "set_alarm",
215
+ "predicted": "set_alarm",
216
+ "correct": true,
217
+ "valid": true,
218
+ "error": null,
219
+ "probs": {
220
+ "play_music": 0.0,
221
+ "send_message": 0.0001,
222
+ "set_alarm": 0.9998,
223
+ "turn_off_lights": 0.0001,
224
+ "weather": 0.0001
225
+ },
226
+ "latency_s": 0.068
227
+ },
228
+ {
229
+ "id": "easy-fact-00",
230
+ "tier": "easy",
231
+ "expected": "yes",
232
+ "predicted": "yes",
233
+ "correct": true,
234
+ "valid": true,
235
+ "error": null,
236
+ "probs": {
237
+ "yes": 0.984,
238
+ "no": 0.016
239
+ },
240
+ "latency_s": 0.068
241
+ },
242
+ {
243
+ "id": "easy-fact-01",
244
+ "tier": "easy",
245
+ "expected": "no",
246
+ "predicted": "no",
247
+ "correct": true,
248
+ "valid": true,
249
+ "error": null,
250
+ "probs": {
251
+ "yes": 0.0057,
252
+ "no": 0.9943
253
+ },
254
+ "latency_s": 0.068
255
+ },
256
+ {
257
+ "id": "easy-fact-02",
258
+ "tier": "easy",
259
+ "expected": "yes",
260
+ "predicted": "yes",
261
+ "correct": true,
262
+ "valid": true,
263
+ "error": null,
264
+ "probs": {
265
+ "yes": 0.9834,
266
+ "no": 0.0166
267
+ },
268
+ "latency_s": 0.072
269
+ },
270
+ {
271
+ "id": "easy-fact-03",
272
+ "tier": "easy",
273
+ "expected": "no",
274
+ "predicted": "no",
275
+ "correct": true,
276
+ "valid": true,
277
+ "error": null,
278
+ "probs": {
279
+ "yes": 0.0026,
280
+ "no": 0.9974
281
+ },
282
+ "latency_s": 0.067
283
+ },
284
+ {
285
+ "id": "easy-fact-04",
286
+ "tier": "easy",
287
+ "expected": "yes",
288
+ "predicted": "yes",
289
+ "correct": true,
290
+ "valid": true,
291
+ "error": null,
292
+ "probs": {
293
+ "yes": 0.9783,
294
+ "no": 0.0217
295
+ },
296
+ "latency_s": 0.066
297
+ },
298
+ {
299
+ "id": "easy-fact-05",
300
+ "tier": "easy",
301
+ "expected": "no",
302
+ "predicted": "no",
303
+ "correct": true,
304
+ "valid": true,
305
+ "error": null,
306
+ "probs": {
307
+ "yes": 0.011,
308
+ "no": 0.989
309
+ },
310
+ "latency_s": 0.068
311
+ },
312
+ {
313
+ "id": "easy-fact-06",
314
+ "tier": "easy",
315
+ "expected": "yes",
316
+ "predicted": "yes",
317
+ "correct": true,
318
+ "valid": true,
319
+ "error": null,
320
+ "probs": {
321
+ "yes": 0.9591,
322
+ "no": 0.0409
323
+ },
324
+ "latency_s": 0.068
325
+ },
326
+ {
327
+ "id": "easy-fact-07",
328
+ "tier": "easy",
329
+ "expected": "no",
330
+ "predicted": "no",
331
+ "correct": true,
332
+ "valid": true,
333
+ "error": null,
334
+ "probs": {
335
+ "yes": 0.0087,
336
+ "no": 0.9913
337
+ },
338
+ "latency_s": 0.068
339
+ },
340
+ {
341
+ "id": "easy-fact-08",
342
+ "tier": "easy",
343
+ "expected": "yes",
344
+ "predicted": "yes",
345
+ "correct": true,
346
+ "valid": true,
347
+ "error": null,
348
+ "probs": {
349
+ "yes": 0.9827,
350
+ "no": 0.0173
351
+ },
352
+ "latency_s": 0.073
353
+ },
354
+ {
355
+ "id": "easy-fact-09",
356
+ "tier": "easy",
357
+ "expected": "no",
358
+ "predicted": "no",
359
+ "correct": true,
360
+ "valid": true,
361
+ "error": null,
362
+ "probs": {
363
+ "yes": 0.0069,
364
+ "no": 0.9931
365
+ },
366
+ "latency_s": 0.067
367
+ },
368
+ {
369
+ "id": "easy-fact-10",
370
+ "tier": "easy",
371
+ "expected": "yes",
372
+ "predicted": "yes",
373
+ "correct": true,
374
+ "valid": true,
375
+ "error": null,
376
+ "probs": {
377
+ "yes": 0.9411,
378
+ "no": 0.0589
379
+ },
380
+ "latency_s": 0.067
381
+ },
382
+ {
383
+ "id": "easy-fact-11",
384
+ "tier": "easy",
385
+ "expected": "no",
386
+ "predicted": "no",
387
+ "correct": true,
388
+ "valid": true,
389
+ "error": null,
390
+ "probs": {
391
+ "yes": 0.0154,
392
+ "no": 0.9846
393
+ },
394
+ "latency_s": 0.067
395
+ },
396
+ {
397
+ "id": "easy-extraction-00",
398
+ "tier": "easy",
399
+ "expected": "paypal",
400
+ "predicted": "paypal",
401
+ "correct": true,
402
+ "valid": true,
403
+ "error": null,
404
+ "probs": {
405
+ "bank_transfer": 0.0,
406
+ "cash": 0.0,
407
+ "credit_card": 0.0001,
408
+ "paypal": 0.9998
409
+ },
410
+ "latency_s": 0.068
411
+ },
412
+ {
413
+ "id": "easy-extraction-01",
414
+ "tier": "easy",
415
+ "expected": "cash",
416
+ "predicted": "cash",
417
+ "correct": true,
418
+ "valid": true,
419
+ "error": null,
420
+ "probs": {
421
+ "bank_transfer": 0.0001,
422
+ "cash": 0.9996,
423
+ "credit_card": 0.0003,
424
+ "paypal": 0.0001
425
+ },
426
+ "latency_s": 0.067
427
+ },
428
+ {
429
+ "id": "easy-extraction-02",
430
+ "tier": "easy",
431
+ "expected": "bank_transfer",
432
+ "predicted": "bank_transfer",
433
+ "correct": true,
434
+ "valid": true,
435
+ "error": null,
436
+ "probs": {
437
+ "bank_transfer": 0.9997,
438
+ "cash": 0.0001,
439
+ "credit_card": 0.0001,
440
+ "paypal": 0.0001
441
+ },
442
+ "latency_s": 0.067
443
+ },
444
+ {
445
+ "id": "easy-extraction-03",
446
+ "tier": "easy",
447
+ "expected": "high",
448
+ "predicted": "high",
449
+ "correct": true,
450
+ "valid": true,
451
+ "error": null,
452
+ "probs": {
453
+ "high": 0.966,
454
+ "low": 0.0016,
455
+ "medium": 0.0029,
456
+ "urgent": 0.0295
457
+ },
458
+ "latency_s": 0.067
459
+ },
460
+ {
461
+ "id": "easy-extraction-04",
462
+ "tier": "easy",
463
+ "expected": "low",
464
+ "predicted": "low",
465
+ "correct": true,
466
+ "valid": true,
467
+ "error": null,
468
+ "probs": {
469
+ "high": 0.001,
470
+ "low": 0.9971,
471
+ "medium": 0.0016,
472
+ "urgent": 0.0003
473
+ },
474
+ "latency_s": 0.067
475
+ },
476
+ {
477
+ "id": "easy-extraction-05",
478
+ "tier": "easy",
479
+ "expected": "urgent",
480
+ "predicted": "urgent",
481
+ "correct": true,
482
+ "valid": true,
483
+ "error": null,
484
+ "probs": {
485
+ "high": 0.0011,
486
+ "low": 0.0002,
487
+ "medium": 0.0006,
488
+ "urgent": 0.998
489
+ },
490
+ "latency_s": 0.067
491
+ },
492
+ {
493
+ "id": "easy-extraction-06",
494
+ "tier": "easy",
495
+ "expected": "M",
496
+ "predicted": "M",
497
+ "correct": true,
498
+ "valid": true,
499
+ "error": null,
500
+ "probs": {
501
+ "L": 0.0007,
502
+ "M": 0.999,
503
+ "S": 0.0002,
504
+ "XL": 0.0001
505
+ },
506
+ "latency_s": 0.067
507
+ },
508
+ {
509
+ "id": "easy-extraction-07",
510
+ "tier": "easy",
511
+ "expected": "XL",
512
+ "predicted": "XL",
513
+ "correct": true,
514
+ "valid": true,
515
+ "error": null,
516
+ "probs": {
517
+ "L": 0.0004,
518
+ "M": 0.0001,
519
+ "S": 0.0001,
520
+ "XL": 0.9994
521
+ },
522
+ "latency_s": 0.066
523
+ },
524
+ {
525
+ "id": "easy-extraction-08",
526
+ "tier": "easy",
527
+ "expected": "thursday",
528
+ "predicted": "thursday",
529
+ "correct": true,
530
+ "valid": true,
531
+ "error": null,
532
+ "probs": {
533
+ "friday": 0.0007,
534
+ "monday": 0.0009,
535
+ "thursday": 0.9976,
536
+ "tuesday": 0.0004,
537
+ "wednesday": 0.0004
538
+ },
539
+ "latency_s": 0.067
540
+ },
541
+ {
542
+ "id": "easy-extraction-09",
543
+ "tier": "easy",
544
+ "expected": "monday",
545
+ "predicted": "monday",
546
+ "correct": true,
547
+ "valid": true,
548
+ "error": null,
549
+ "probs": {
550
+ "friday": 0.0096,
551
+ "monday": 0.9764,
552
+ "thursday": 0.0059,
553
+ "tuesday": 0.0043,
554
+ "wednesday": 0.0037
555
+ },
556
+ "latency_s": 0.067
557
+ },
558
+ {
559
+ "id": "easy-extraction-10",
560
+ "tier": "easy",
561
+ "expected": "friday",
562
+ "predicted": "friday",
563
+ "correct": true,
564
+ "valid": true,
565
+ "error": null,
566
+ "probs": {
567
+ "friday": 0.9964,
568
+ "monday": 0.0011,
569
+ "thursday": 0.0009,
570
+ "tuesday": 0.0007,
571
+ "wednesday": 0.0008
572
+ },
573
+ "latency_s": 0.065
574
+ },
575
+ {
576
+ "id": "easy-extraction-11",
577
+ "tier": "easy",
578
+ "expected": "credit_card",
579
+ "predicted": "credit_card",
580
+ "correct": true,
581
+ "valid": true,
582
+ "error": null,
583
+ "probs": {
584
+ "bank_transfer": 0.0001,
585
+ "cash": 0.0001,
586
+ "credit_card": 0.9997,
587
+ "paypal": 0.0001
588
+ },
589
+ "latency_s": 0.067
590
+ },
591
+ {
592
+ "id": "easy-tool_selection-00",
593
+ "tier": "easy",
594
+ "expected": "get_weather",
595
+ "predicted": "get_weather",
596
+ "correct": true,
597
+ "valid": true,
598
+ "error": null,
599
+ "probs": {
600
+ "convert_currency": 0.0,
601
+ "create_calendar_event": 0.0,
602
+ "get_weather": 0.9998,
603
+ "send_email": 0.0001,
604
+ "translate_text": 0.0
605
+ },
606
+ "latency_s": 0.067
607
+ },
608
+ {
609
+ "id": "easy-tool_selection-01",
610
+ "tier": "easy",
611
+ "expected": "send_email",
612
+ "predicted": "send_email",
613
+ "correct": true,
614
+ "valid": true,
615
+ "error": null,
616
+ "probs": {
617
+ "convert_currency": 0.0,
618
+ "create_calendar_event": 0.0,
619
+ "get_weather": 0.0001,
620
+ "send_email": 0.9998,
621
+ "translate_text": 0.0001
622
+ },
623
+ "latency_s": 0.067
624
+ },
625
+ {
626
+ "id": "easy-tool_selection-02",
627
+ "tier": "easy",
628
+ "expected": "create_calendar_event",
629
+ "predicted": "create_calendar_event",
630
+ "correct": true,
631
+ "valid": true,
632
+ "error": null,
633
+ "probs": {
634
+ "convert_currency": 0.0,
635
+ "create_calendar_event": 0.9999,
636
+ "get_weather": 0.0,
637
+ "send_email": 0.0,
638
+ "translate_text": 0.0
639
+ },
640
+ "latency_s": 0.068
641
+ },
642
+ {
643
+ "id": "easy-tool_selection-03",
644
+ "tier": "easy",
645
+ "expected": "convert_currency",
646
+ "predicted": "convert_currency",
647
+ "correct": true,
648
+ "valid": true,
649
+ "error": null,
650
+ "probs": {
651
+ "convert_currency": 0.9997,
652
+ "create_calendar_event": 0.0001,
653
+ "get_weather": 0.0001,
654
+ "send_email": 0.0001,
655
+ "translate_text": 0.0001
656
+ },
657
+ "latency_s": 0.068
658
+ },
659
+ {
660
+ "id": "easy-tool_selection-04",
661
+ "tier": "easy",
662
+ "expected": "translate_text",
663
+ "predicted": "translate_text",
664
+ "correct": true,
665
+ "valid": true,
666
+ "error": null,
667
+ "probs": {
668
+ "convert_currency": 0.0,
669
+ "create_calendar_event": 0.0,
670
+ "get_weather": 0.0,
671
+ "send_email": 0.0,
672
+ "translate_text": 0.9999
673
+ },
674
+ "latency_s": 0.068
675
+ },
676
+ {
677
+ "id": "easy-tool_selection-05",
678
+ "tier": "easy",
679
+ "expected": "create_calendar_event",
680
+ "predicted": "create_calendar_event",
681
+ "correct": true,
682
+ "valid": true,
683
+ "error": null,
684
+ "probs": {
685
+ "convert_currency": 0.0,
686
+ "create_calendar_event": 0.9998,
687
+ "get_weather": 0.0001,
688
+ "send_email": 0.0,
689
+ "translate_text": 0.0
690
+ },
691
+ "latency_s": 0.067
692
+ },
693
+ {
694
+ "id": "easy-tool_selection-06",
695
+ "tier": "easy",
696
+ "expected": "search_flights",
697
+ "predicted": "search_flights",
698
+ "correct": true,
699
+ "valid": true,
700
+ "error": null,
701
+ "probs": {
702
+ "book_restaurant": 0.0,
703
+ "call_taxi": 0.0,
704
+ "get_stock_price": 0.0,
705
+ "search_flights": 0.9999,
706
+ "set_reminder": 0.0001
707
+ },
708
+ "latency_s": 0.073
709
+ },
710
+ {
711
+ "id": "easy-tool_selection-07",
712
+ "tier": "easy",
713
+ "expected": "book_restaurant",
714
+ "predicted": "book_restaurant",
715
+ "correct": true,
716
+ "valid": true,
717
+ "error": null,
718
+ "probs": {
719
+ "book_restaurant": 0.9994,
720
+ "call_taxi": 0.0001,
721
+ "get_stock_price": 0.0001,
722
+ "search_flights": 0.0001,
723
+ "set_reminder": 0.0003
724
+ },
725
+ "latency_s": 0.068
726
+ },
727
+ {
728
+ "id": "easy-tool_selection-08",
729
+ "tier": "easy",
730
+ "expected": "set_reminder",
731
+ "predicted": "set_reminder",
732
+ "correct": true,
733
+ "valid": true,
734
+ "error": null,
735
+ "probs": {
736
+ "book_restaurant": 0.0,
737
+ "call_taxi": 0.0,
738
+ "get_stock_price": 0.0001,
739
+ "search_flights": 0.0,
740
+ "set_reminder": 0.9998
741
+ },
742
+ "latency_s": 0.068
743
+ },
744
+ {
745
+ "id": "easy-tool_selection-09",
746
+ "tier": "easy",
747
+ "expected": "get_stock_price",
748
+ "predicted": "get_stock_price",
749
+ "correct": true,
750
+ "valid": true,
751
+ "error": null,
752
+ "probs": {
753
+ "book_restaurant": 0.0,
754
+ "call_taxi": 0.0001,
755
+ "get_stock_price": 0.9998,
756
+ "search_flights": 0.0,
757
+ "set_reminder": 0.0001
758
+ },
759
+ "latency_s": 0.07
760
+ },
761
+ {
762
+ "id": "easy-tool_selection-10",
763
+ "tier": "easy",
764
+ "expected": "call_taxi",
765
+ "predicted": "call_taxi",
766
+ "correct": true,
767
+ "valid": true,
768
+ "error": null,
769
+ "probs": {
770
+ "book_restaurant": 0.0001,
771
+ "call_taxi": 0.9995,
772
+ "get_stock_price": 0.0001,
773
+ "search_flights": 0.0001,
774
+ "set_reminder": 0.0002
775
+ },
776
+ "latency_s": 0.07
777
+ },
778
+ {
779
+ "id": "easy-tool_selection-11",
780
+ "tier": "easy",
781
+ "expected": "set_reminder",
782
+ "predicted": "set_reminder",
783
+ "correct": true,
784
+ "valid": true,
785
+ "error": null,
786
+ "probs": {
787
+ "book_restaurant": 0.0,
788
+ "call_taxi": 0.0,
789
+ "get_stock_price": 0.0,
790
+ "search_flights": 0.0,
791
+ "set_reminder": 0.9999
792
+ },
793
+ "latency_s": 0.068
794
+ },
795
+ {
796
+ "id": "original-policy-01-0",
797
+ "tier": "standard",
798
+ "expected": "no",
799
+ "predicted": "no",
800
+ "correct": true,
801
+ "valid": true,
802
+ "error": null,
803
+ "probs": {
804
+ "yes": 0.0173,
805
+ "no": 0.9827
806
+ },
807
+ "latency_s": 0.067
808
+ },
809
+ {
810
+ "id": "original-policy-01-1",
811
+ "tier": "standard",
812
+ "expected": "no",
813
+ "predicted": "no",
814
+ "correct": true,
815
+ "valid": true,
816
+ "error": null,
817
+ "probs": {
818
+ "yes": 0.0271,
819
+ "no": 0.9729
820
+ },
821
+ "latency_s": 0.068
822
+ },
823
+ {
824
+ "id": "original-policy-02-0",
825
+ "tier": "standard",
826
+ "expected": "yes",
827
+ "predicted": "yes",
828
+ "correct": true,
829
+ "valid": true,
830
+ "error": null,
831
+ "probs": {
832
+ "yes": 0.9411,
833
+ "no": 0.0589
834
+ },
835
+ "latency_s": 0.069
836
+ },
837
+ {
838
+ "id": "original-policy-02-1",
839
+ "tier": "standard",
840
+ "expected": "yes",
841
+ "predicted": "no",
842
+ "correct": false,
843
+ "valid": true,
844
+ "error": null,
845
+ "probs": {
846
+ "yes": 0.2327,
847
+ "no": 0.7673
848
+ },
849
+ "latency_s": 0.068
850
+ },
851
+ {
852
+ "id": "original-policy-03-0",
853
+ "tier": "standard",
854
+ "expected": "yes",
855
+ "predicted": "yes",
856
+ "correct": true,
857
+ "valid": true,
858
+ "error": null,
859
+ "probs": {
860
+ "yes": 0.5669,
861
+ "no": 0.4331
862
+ },
863
+ "latency_s": 0.068
864
+ },
865
+ {
866
+ "id": "original-policy-03-1",
867
+ "tier": "standard",
868
+ "expected": "yes",
869
+ "predicted": "yes",
870
+ "correct": true,
871
+ "valid": true,
872
+ "error": null,
873
+ "probs": {
874
+ "yes": 0.6043,
875
+ "no": 0.3957
876
+ },
877
+ "latency_s": 0.068
878
+ },
879
+ {
880
+ "id": "original-policy-04-0",
881
+ "tier": "standard",
882
+ "expected": "no",
883
+ "predicted": "no",
884
+ "correct": true,
885
+ "valid": true,
886
+ "error": null,
887
+ "probs": {
888
+ "yes": 0.0758,
889
+ "no": 0.9242
890
+ },
891
+ "latency_s": 0.068
892
+ },
893
+ {
894
+ "id": "original-policy-04-1",
895
+ "tier": "standard",
896
+ "expected": "no",
897
+ "predicted": "no",
898
+ "correct": true,
899
+ "valid": true,
900
+ "error": null,
901
+ "probs": {
902
+ "yes": 0.0133,
903
+ "no": 0.9867
904
+ },
905
+ "latency_s": 0.068
906
+ },
907
+ {
908
+ "id": "original-policy-05-0",
909
+ "tier": "standard",
910
+ "expected": "no",
911
+ "predicted": "no",
912
+ "correct": true,
913
+ "valid": true,
914
+ "error": null,
915
+ "probs": {
916
+ "yes": 0.1151,
917
+ "no": 0.8849
918
+ },
919
+ "latency_s": 0.068
920
+ },
921
+ {
922
+ "id": "original-policy-05-1",
923
+ "tier": "standard",
924
+ "expected": "no",
925
+ "predicted": "no",
926
+ "correct": true,
927
+ "valid": true,
928
+ "error": null,
929
+ "probs": {
930
+ "yes": 0.0758,
931
+ "no": 0.9242
932
+ },
933
+ "latency_s": 0.067
934
+ },
935
+ {
936
+ "id": "original-policy-06-0",
937
+ "tier": "standard",
938
+ "expected": "yes",
939
+ "predicted": "yes",
940
+ "correct": true,
941
+ "valid": true,
942
+ "error": null,
943
+ "probs": {
944
+ "yes": 0.9606,
945
+ "no": 0.0394
946
+ },
947
+ "latency_s": 0.068
948
+ },
949
+ {
950
+ "id": "original-policy-06-1",
951
+ "tier": "standard",
952
+ "expected": "yes",
953
+ "predicted": "yes",
954
+ "correct": true,
955
+ "valid": true,
956
+ "error": null,
957
+ "probs": {
958
+ "yes": 0.8177,
959
+ "no": 0.1823
960
+ },
961
+ "latency_s": 0.069
962
+ },
963
+ {
964
+ "id": "original-intent-01-0",
965
+ "tier": "standard",
966
+ "expected": "refund",
967
+ "predicted": "refund",
968
+ "correct": true,
969
+ "valid": true,
970
+ "error": null,
971
+ "probs": {
972
+ "cancel": 0.0008,
973
+ "change_address": 0.0003,
974
+ "other": 0.0039,
975
+ "refund": 0.9948,
976
+ "status": 0.0002
977
+ },
978
+ "latency_s": 0.068
979
+ },
980
+ {
981
+ "id": "original-intent-01-1",
982
+ "tier": "standard",
983
+ "expected": "refund",
984
+ "predicted": "refund",
985
+ "correct": true,
986
+ "valid": true,
987
+ "error": null,
988
+ "probs": {
989
+ "cancel": 0.0357,
990
+ "change_address": 0.0012,
991
+ "other": 0.0178,
992
+ "refund": 0.9446,
993
+ "status": 0.0007
994
+ },
995
+ "latency_s": 0.071
996
+ },
997
+ {
998
+ "id": "original-intent-02-0",
999
+ "tier": "standard",
1000
+ "expected": "status",
1001
+ "predicted": "status",
1002
+ "correct": true,
1003
+ "valid": true,
1004
+ "error": null,
1005
+ "probs": {
1006
+ "cancel": 0.0011,
1007
+ "change_address": 0.0039,
1008
+ "other": 0.4431,
1009
+ "refund": 0.0027,
1010
+ "status": 0.5492
1011
+ },
1012
+ "latency_s": 0.07
1013
+ },
1014
+ {
1015
+ "id": "original-intent-02-1",
1016
+ "tier": "standard",
1017
+ "expected": "status",
1018
+ "predicted": "status",
1019
+ "correct": true,
1020
+ "valid": true,
1021
+ "error": null,
1022
+ "probs": {
1023
+ "cancel": 0.0001,
1024
+ "change_address": 0.0001,
1025
+ "other": 0.0017,
1026
+ "refund": 0.0001,
1027
+ "status": 0.9981
1028
+ },
1029
+ "latency_s": 0.068
1030
+ },
1031
+ {
1032
+ "id": "original-intent-03-0",
1033
+ "tier": "standard",
1034
+ "expected": "change_address",
1035
+ "predicted": "change_address",
1036
+ "correct": true,
1037
+ "valid": true,
1038
+ "error": null,
1039
+ "probs": {
1040
+ "cancel": 0.0006,
1041
+ "change_address": 0.9693,
1042
+ "other": 0.028,
1043
+ "refund": 0.0015,
1044
+ "status": 0.0006
1045
+ },
1046
+ "latency_s": 0.069
1047
+ },
1048
+ {
1049
+ "id": "original-intent-03-1",
1050
+ "tier": "standard",
1051
+ "expected": "change_address",
1052
+ "predicted": "change_address",
1053
+ "correct": true,
1054
+ "valid": true,
1055
+ "error": null,
1056
+ "probs": {
1057
+ "cancel": 0.0,
1058
+ "change_address": 0.9992,
1059
+ "other": 0.0006,
1060
+ "refund": 0.0001,
1061
+ "status": 0.0
1062
+ },
1063
+ "latency_s": 0.068
1064
+ },
1065
+ {
1066
+ "id": "original-intent-04-0",
1067
+ "tier": "standard",
1068
+ "expected": "cancel",
1069
+ "predicted": "cancel",
1070
+ "correct": true,
1071
+ "valid": true,
1072
+ "error": null,
1073
+ "probs": {
1074
+ "cancel": 0.9957,
1075
+ "change_address": 0.0004,
1076
+ "other": 0.0032,
1077
+ "refund": 0.0006,
1078
+ "status": 0.0001
1079
+ },
1080
+ "latency_s": 0.068
1081
+ },
1082
+ {
1083
+ "id": "original-intent-04-1",
1084
+ "tier": "standard",
1085
+ "expected": "cancel",
1086
+ "predicted": "cancel",
1087
+ "correct": true,
1088
+ "valid": true,
1089
+ "error": null,
1090
+ "probs": {
1091
+ "cancel": 0.9957,
1092
+ "change_address": 0.0004,
1093
+ "other": 0.0034,
1094
+ "refund": 0.0004,
1095
+ "status": 0.0002
1096
+ },
1097
+ "latency_s": 0.068
1098
+ },
1099
+ {
1100
+ "id": "original-intent-05-0",
1101
+ "tier": "standard",
1102
+ "expected": "other",
1103
+ "predicted": "other",
1104
+ "correct": true,
1105
+ "valid": true,
1106
+ "error": null,
1107
+ "probs": {
1108
+ "cancel": 0.0015,
1109
+ "change_address": 0.0027,
1110
+ "other": 0.8433,
1111
+ "refund": 0.1513,
1112
+ "status": 0.0013
1113
+ },
1114
+ "latency_s": 0.068
1115
+ },
1116
+ {
1117
+ "id": "original-intent-05-1",
1118
+ "tier": "standard",
1119
+ "expected": "other",
1120
+ "predicted": "other",
1121
+ "correct": true,
1122
+ "valid": true,
1123
+ "error": null,
1124
+ "probs": {
1125
+ "cancel": 0.0027,
1126
+ "change_address": 0.003,
1127
+ "other": 0.7087,
1128
+ "refund": 0.2845,
1129
+ "status": 0.0013
1130
+ },
1131
+ "latency_s": 0.069
1132
+ },
1133
+ {
1134
+ "id": "original-intent-06-0",
1135
+ "tier": "standard",
1136
+ "expected": "status",
1137
+ "predicted": "status",
1138
+ "correct": true,
1139
+ "valid": true,
1140
+ "error": null,
1141
+ "probs": {
1142
+ "cancel": 0.0001,
1143
+ "change_address": 0.0001,
1144
+ "other": 0.003,
1145
+ "refund": 0.0001,
1146
+ "status": 0.9966
1147
+ },
1148
+ "latency_s": 0.068
1149
+ },
1150
+ {
1151
+ "id": "original-intent-06-1",
1152
+ "tier": "standard",
1153
+ "expected": "status",
1154
+ "predicted": "status",
1155
+ "correct": true,
1156
+ "valid": true,
1157
+ "error": null,
1158
+ "probs": {
1159
+ "cancel": 0.0001,
1160
+ "change_address": 0.0,
1161
+ "other": 0.0014,
1162
+ "refund": 0.0001,
1163
+ "status": 0.9985
1164
+ },
1165
+ "latency_s": 0.068
1166
+ },
1167
+ {
1168
+ "id": "original-ordinal-01-0",
1169
+ "tier": "standard",
1170
+ "expected": "0",
1171
+ "predicted": "0",
1172
+ "correct": true,
1173
+ "valid": true,
1174
+ "error": null,
1175
+ "probs": {
1176
+ "0": 0.8812,
1177
+ "1": 0.1156,
1178
+ "2": 0.0021,
1179
+ "3": 0.0011
1180
+ },
1181
+ "latency_s": 0.075
1182
+ },
1183
+ {
1184
+ "id": "original-ordinal-01-1",
1185
+ "tier": "standard",
1186
+ "expected": "0",
1187
+ "predicted": "0",
1188
+ "correct": true,
1189
+ "valid": true,
1190
+ "error": null,
1191
+ "probs": {
1192
+ "0": 0.807,
1193
+ "1": 0.1897,
1194
+ "2": 0.0024,
1195
+ "3": 0.0009
1196
+ },
1197
+ "latency_s": 0.069
1198
+ },
1199
+ {
1200
+ "id": "original-ordinal-02-0",
1201
+ "tier": "standard",
1202
+ "expected": "1",
1203
+ "predicted": "1",
1204
+ "correct": true,
1205
+ "valid": true,
1206
+ "error": null,
1207
+ "probs": {
1208
+ "0": 0.0171,
1209
+ "1": 0.9198,
1210
+ "2": 0.0599,
1211
+ "3": 0.0032
1212
+ },
1213
+ "latency_s": 0.07
1214
+ },
1215
+ {
1216
+ "id": "original-ordinal-02-1",
1217
+ "tier": "standard",
1218
+ "expected": "1",
1219
+ "predicted": "1",
1220
+ "correct": true,
1221
+ "valid": true,
1222
+ "error": null,
1223
+ "probs": {
1224
+ "0": 0.0131,
1225
+ "1": 0.9794,
1226
+ "2": 0.0051,
1227
+ "3": 0.0024
1228
+ },
1229
+ "latency_s": 0.07
1230
+ },
1231
+ {
1232
+ "id": "original-ordinal-03-0",
1233
+ "tier": "standard",
1234
+ "expected": "2",
1235
+ "predicted": "2",
1236
+ "correct": true,
1237
+ "valid": true,
1238
+ "error": null,
1239
+ "probs": {
1240
+ "0": 0.0241,
1241
+ "1": 0.0734,
1242
+ "2": 0.9013,
1243
+ "3": 0.0012
1244
+ },
1245
+ "latency_s": 0.071
1246
+ },
1247
+ {
1248
+ "id": "original-ordinal-03-1",
1249
+ "tier": "standard",
1250
+ "expected": "2",
1251
+ "predicted": "2",
1252
+ "correct": true,
1253
+ "valid": true,
1254
+ "error": null,
1255
+ "probs": {
1256
+ "0": 0.0182,
1257
+ "1": 0.0959,
1258
+ "2": 0.8829,
1259
+ "3": 0.0029
1260
+ },
1261
+ "latency_s": 0.07
1262
+ },
1263
+ {
1264
+ "id": "original-ordinal-04-0",
1265
+ "tier": "standard",
1266
+ "expected": "3",
1267
+ "predicted": "3",
1268
+ "correct": true,
1269
+ "valid": true,
1270
+ "error": null,
1271
+ "probs": {
1272
+ "0": 0.0041,
1273
+ "1": 0.002,
1274
+ "2": 0.0051,
1275
+ "3": 0.9888
1276
+ },
1277
+ "latency_s": 0.069
1278
+ },
1279
+ {
1280
+ "id": "original-ordinal-04-1",
1281
+ "tier": "standard",
1282
+ "expected": "3",
1283
+ "predicted": "3",
1284
+ "correct": true,
1285
+ "valid": true,
1286
+ "error": null,
1287
+ "probs": {
1288
+ "0": 0.0057,
1289
+ "1": 0.0046,
1290
+ "2": 0.0392,
1291
+ "3": 0.9505
1292
+ },
1293
+ "latency_s": 0.069
1294
+ },
1295
+ {
1296
+ "id": "original-ordinal-05-0",
1297
+ "tier": "standard",
1298
+ "expected": "0",
1299
+ "predicted": "0",
1300
+ "correct": true,
1301
+ "valid": true,
1302
+ "error": null,
1303
+ "probs": {
1304
+ "0": 0.9805,
1305
+ "1": 0.0145,
1306
+ "2": 0.004,
1307
+ "3": 0.001
1308
+ },
1309
+ "latency_s": 0.069
1310
+ },
1311
+ {
1312
+ "id": "original-ordinal-05-1",
1313
+ "tier": "standard",
1314
+ "expected": "0",
1315
+ "predicted": "0",
1316
+ "correct": true,
1317
+ "valid": true,
1318
+ "error": null,
1319
+ "probs": {
1320
+ "0": 0.6784,
1321
+ "1": 0.1454,
1322
+ "2": 0.1454,
1323
+ "3": 0.0308
1324
+ },
1325
+ "latency_s": 0.069
1326
+ },
1327
+ {
1328
+ "id": "original-ordinal-06-0",
1329
+ "tier": "standard",
1330
+ "expected": "2",
1331
+ "predicted": "2",
1332
+ "correct": true,
1333
+ "valid": true,
1334
+ "error": null,
1335
+ "probs": {
1336
+ "0": 0.0144,
1337
+ "1": 0.0793,
1338
+ "2": 0.9057,
1339
+ "3": 0.0006
1340
+ },
1341
+ "latency_s": 0.069
1342
+ },
1343
+ {
1344
+ "id": "original-ordinal-06-1",
1345
+ "tier": "standard",
1346
+ "expected": "2",
1347
+ "predicted": "2",
1348
+ "correct": true,
1349
+ "valid": true,
1350
+ "error": null,
1351
+ "probs": {
1352
+ "0": 0.0055,
1353
+ "1": 0.0809,
1354
+ "2": 0.9129,
1355
+ "3": 0.0008
1356
+ },
1357
+ "latency_s": 0.071
1358
+ },
1359
+ {
1360
+ "id": "original-extraction-01-0",
1361
+ "tier": "standard",
1362
+ "expected": "pickup",
1363
+ "predicted": "pickup",
1364
+ "correct": true,
1365
+ "valid": true,
1366
+ "error": null,
1367
+ "probs": {
1368
+ "courier": 0.0012,
1369
+ "pickup": 0.9882,
1370
+ "post": 0.0009,
1371
+ "unknown": 0.0098
1372
+ },
1373
+ "latency_s": 0.069
1374
+ },
1375
+ {
1376
+ "id": "original-extraction-01-1",
1377
+ "tier": "standard",
1378
+ "expected": "pickup",
1379
+ "predicted": "pickup",
1380
+ "correct": true,
1381
+ "valid": true,
1382
+ "error": null,
1383
+ "probs": {
1384
+ "courier": 0.0004,
1385
+ "pickup": 0.9926,
1386
+ "post": 0.0007,
1387
+ "unknown": 0.0064
1388
+ },
1389
+ "latency_s": 0.068
1390
+ },
1391
+ {
1392
+ "id": "original-extraction-02-0",
1393
+ "tier": "standard",
1394
+ "expected": "unknown",
1395
+ "predicted": "unknown",
1396
+ "correct": true,
1397
+ "valid": true,
1398
+ "error": null,
1399
+ "probs": {
1400
+ "courier": 0.0106,
1401
+ "pickup": 0.0073,
1402
+ "post": 0.3872,
1403
+ "unknown": 0.5949
1404
+ },
1405
+ "latency_s": 0.067
1406
+ },
1407
+ {
1408
+ "id": "original-extraction-02-1",
1409
+ "tier": "standard",
1410
+ "expected": "unknown",
1411
+ "predicted": "unknown",
1412
+ "correct": true,
1413
+ "valid": true,
1414
+ "error": null,
1415
+ "probs": {
1416
+ "courier": 0.0022,
1417
+ "pickup": 0.0015,
1418
+ "post": 0.0141,
1419
+ "unknown": 0.9822
1420
+ },
1421
+ "latency_s": 0.068
1422
+ },
1423
+ {
1424
+ "id": "original-extraction-03-0",
1425
+ "tier": "standard",
1426
+ "expected": "courier",
1427
+ "predicted": "courier",
1428
+ "correct": true,
1429
+ "valid": true,
1430
+ "error": null,
1431
+ "probs": {
1432
+ "courier": 0.9283,
1433
+ "pickup": 0.007,
1434
+ "post": 0.0078,
1435
+ "unknown": 0.0569
1436
+ },
1437
+ "latency_s": 0.068
1438
+ },
1439
+ {
1440
+ "id": "original-extraction-03-1",
1441
+ "tier": "standard",
1442
+ "expected": "courier",
1443
+ "predicted": "courier",
1444
+ "correct": true,
1445
+ "valid": true,
1446
+ "error": null,
1447
+ "probs": {
1448
+ "courier": 0.9994,
1449
+ "pickup": 0.0002,
1450
+ "post": 0.0002,
1451
+ "unknown": 0.0001
1452
+ },
1453
+ "latency_s": 0.068
1454
+ },
1455
+ {
1456
+ "id": "original-extraction-04-0",
1457
+ "tier": "standard",
1458
+ "expected": "post",
1459
+ "predicted": "post",
1460
+ "correct": true,
1461
+ "valid": true,
1462
+ "error": null,
1463
+ "probs": {
1464
+ "courier": 0.0002,
1465
+ "pickup": 0.0004,
1466
+ "post": 0.9976,
1467
+ "unknown": 0.0018
1468
+ },
1469
+ "latency_s": 0.068
1470
+ },
1471
+ {
1472
+ "id": "original-extraction-04-1",
1473
+ "tier": "standard",
1474
+ "expected": "post",
1475
+ "predicted": "post",
1476
+ "correct": true,
1477
+ "valid": true,
1478
+ "error": null,
1479
+ "probs": {
1480
+ "courier": 0.0007,
1481
+ "pickup": 0.0006,
1482
+ "post": 0.9974,
1483
+ "unknown": 0.0012
1484
+ },
1485
+ "latency_s": 0.067
1486
+ },
1487
+ {
1488
+ "id": "original-extraction-05-0",
1489
+ "tier": "standard",
1490
+ "expected": "unknown",
1491
+ "predicted": "unknown",
1492
+ "correct": true,
1493
+ "valid": true,
1494
+ "error": null,
1495
+ "probs": {
1496
+ "courier": 0.0006,
1497
+ "pickup": 0.0044,
1498
+ "post": 0.0015,
1499
+ "unknown": 0.9935
1500
+ },
1501
+ "latency_s": 0.067
1502
+ },
1503
+ {
1504
+ "id": "original-extraction-05-1",
1505
+ "tier": "standard",
1506
+ "expected": "unknown",
1507
+ "predicted": "courier",
1508
+ "correct": false,
1509
+ "valid": true,
1510
+ "error": null,
1511
+ "probs": {
1512
+ "courier": 0.7409,
1513
+ "pickup": 0.0096,
1514
+ "post": 0.0096,
1515
+ "unknown": 0.2399
1516
+ },
1517
+ "latency_s": 0.067
1518
+ },
1519
+ {
1520
+ "id": "original-extraction-06-0",
1521
+ "tier": "standard",
1522
+ "expected": "courier",
1523
+ "predicted": "courier",
1524
+ "correct": true,
1525
+ "valid": true,
1526
+ "error": null,
1527
+ "probs": {
1528
+ "courier": 0.9991,
1529
+ "pickup": 0.0002,
1530
+ "post": 0.0003,
1531
+ "unknown": 0.0004
1532
+ },
1533
+ "latency_s": 0.067
1534
+ },
1535
+ {
1536
+ "id": "original-extraction-06-1",
1537
+ "tier": "standard",
1538
+ "expected": "courier",
1539
+ "predicted": "courier",
1540
+ "correct": true,
1541
+ "valid": true,
1542
+ "error": null,
1543
+ "probs": {
1544
+ "courier": 0.9988,
1545
+ "pickup": 0.0003,
1546
+ "post": 0.0003,
1547
+ "unknown": 0.0007
1548
+ },
1549
+ "latency_s": 0.067
1550
+ },
1551
+ {
1552
+ "id": "original-adequacy-01-0",
1553
+ "tier": "standard",
1554
+ "expected": "yes",
1555
+ "predicted": "yes",
1556
+ "correct": true,
1557
+ "valid": true,
1558
+ "error": null,
1559
+ "probs": {
1560
+ "yes": 0.8343,
1561
+ "no": 0.1657
1562
+ },
1563
+ "latency_s": 0.067
1564
+ },
1565
+ {
1566
+ "id": "original-adequacy-01-1",
1567
+ "tier": "standard",
1568
+ "expected": "yes",
1569
+ "predicted": "yes",
1570
+ "correct": true,
1571
+ "valid": true,
1572
+ "error": null,
1573
+ "probs": {
1574
+ "yes": 0.7235,
1575
+ "no": 0.2765
1576
+ },
1577
+ "latency_s": 0.067
1578
+ },
1579
+ {
1580
+ "id": "original-adequacy-02-0",
1581
+ "tier": "standard",
1582
+ "expected": "no",
1583
+ "predicted": "no",
1584
+ "correct": true,
1585
+ "valid": true,
1586
+ "error": null,
1587
+ "probs": {
1588
+ "yes": 0.0657,
1589
+ "no": 0.9343
1590
+ },
1591
+ "latency_s": 0.068
1592
+ },
1593
+ {
1594
+ "id": "original-adequacy-02-1",
1595
+ "tier": "standard",
1596
+ "expected": "no",
1597
+ "predicted": "no",
1598
+ "correct": true,
1599
+ "valid": true,
1600
+ "error": null,
1601
+ "probs": {
1602
+ "yes": 0.0904,
1603
+ "no": 0.9096
1604
+ },
1605
+ "latency_s": 0.067
1606
+ },
1607
+ {
1608
+ "id": "original-adequacy-03-0",
1609
+ "tier": "standard",
1610
+ "expected": "no",
1611
+ "predicted": "yes",
1612
+ "correct": false,
1613
+ "valid": true,
1614
+ "error": null,
1615
+ "probs": {
1616
+ "yes": 0.7673,
1617
+ "no": 0.2327
1618
+ },
1619
+ "latency_s": 0.067
1620
+ },
1621
+ {
1622
+ "id": "original-adequacy-03-1",
1623
+ "tier": "standard",
1624
+ "expected": "no",
1625
+ "predicted": "yes",
1626
+ "correct": false,
1627
+ "valid": true,
1628
+ "error": null,
1629
+ "probs": {
1630
+ "yes": 0.658,
1631
+ "no": 0.342
1632
+ },
1633
+ "latency_s": 0.066
1634
+ },
1635
+ {
1636
+ "id": "original-adequacy-04-0",
1637
+ "tier": "standard",
1638
+ "expected": "yes",
1639
+ "predicted": "yes",
1640
+ "correct": true,
1641
+ "valid": true,
1642
+ "error": null,
1643
+ "probs": {
1644
+ "yes": 0.8395,
1645
+ "no": 0.1605
1646
+ },
1647
+ "latency_s": 0.067
1648
+ },
1649
+ {
1650
+ "id": "original-adequacy-04-1",
1651
+ "tier": "standard",
1652
+ "expected": "yes",
1653
+ "predicted": "yes",
1654
+ "correct": true,
1655
+ "valid": true,
1656
+ "error": null,
1657
+ "probs": {
1658
+ "yes": 0.6134,
1659
+ "no": 0.3866
1660
+ },
1661
+ "latency_s": 0.067
1662
+ },
1663
+ {
1664
+ "id": "original-adequacy-05-0",
1665
+ "tier": "standard",
1666
+ "expected": "no",
1667
+ "predicted": "yes",
1668
+ "correct": false,
1669
+ "valid": true,
1670
+ "error": null,
1671
+ "probs": {
1672
+ "yes": 0.5,
1673
+ "no": 0.5
1674
+ },
1675
+ "latency_s": 0.068
1676
+ },
1677
+ {
1678
+ "id": "original-adequacy-05-1",
1679
+ "tier": "standard",
1680
+ "expected": "no",
1681
+ "predicted": "yes",
1682
+ "correct": false,
1683
+ "valid": true,
1684
+ "error": null,
1685
+ "probs": {
1686
+ "yes": 0.6225,
1687
+ "no": 0.3775
1688
+ },
1689
+ "latency_s": 0.068
1690
+ },
1691
+ {
1692
+ "id": "original-adequacy-06-0",
1693
+ "tier": "standard",
1694
+ "expected": "yes",
1695
+ "predicted": "yes",
1696
+ "correct": true,
1697
+ "valid": true,
1698
+ "error": null,
1699
+ "probs": {
1700
+ "yes": 0.5288,
1701
+ "no": 0.4712
1702
+ },
1703
+ "latency_s": 0.069
1704
+ },
1705
+ {
1706
+ "id": "original-adequacy-06-1",
1707
+ "tier": "standard",
1708
+ "expected": "yes",
1709
+ "predicted": "yes",
1710
+ "correct": true,
1711
+ "valid": true,
1712
+ "error": null,
1713
+ "probs": {
1714
+ "yes": 0.5,
1715
+ "no": 0.5
1716
+ },
1717
+ "latency_s": 0.068
1718
+ },
1719
+ {
1720
+ "id": "original-routing-01-0",
1721
+ "tier": "standard",
1722
+ "expected": "math",
1723
+ "predicted": "math",
1724
+ "correct": true,
1725
+ "valid": true,
1726
+ "error": null,
1727
+ "probs": {
1728
+ "coding": 0.012,
1729
+ "coding_agent": 0.0009,
1730
+ "document": 0.0028,
1731
+ "general": 0.0512,
1732
+ "math": 0.9296,
1733
+ "tools": 0.0035
1734
+ },
1735
+ "latency_s": 0.073
1736
+ },
1737
+ {
1738
+ "id": "original-routing-01-1",
1739
+ "tier": "standard",
1740
+ "expected": "math",
1741
+ "predicted": "math",
1742
+ "correct": true,
1743
+ "valid": true,
1744
+ "error": null,
1745
+ "probs": {
1746
+ "coding": 0.0093,
1747
+ "coding_agent": 0.0007,
1748
+ "document": 0.0033,
1749
+ "general": 0.0464,
1750
+ "math": 0.938,
1751
+ "tools": 0.0023
1752
+ },
1753
+ "latency_s": 0.069
1754
+ },
1755
+ {
1756
+ "id": "original-routing-02-0",
1757
+ "tier": "standard",
1758
+ "expected": "coding",
1759
+ "predicted": "coding",
1760
+ "correct": true,
1761
+ "valid": true,
1762
+ "error": null,
1763
+ "probs": {
1764
+ "coding": 0.9776,
1765
+ "coding_agent": 0.0113,
1766
+ "document": 0.0014,
1767
+ "general": 0.0082,
1768
+ "math": 0.0006,
1769
+ "tools": 0.0008
1770
+ },
1771
+ "latency_s": 0.069
1772
+ },
1773
+ {
1774
+ "id": "original-routing-02-1",
1775
+ "tier": "standard",
1776
+ "expected": "coding",
1777
+ "predicted": "coding",
1778
+ "correct": true,
1779
+ "valid": true,
1780
+ "error": null,
1781
+ "probs": {
1782
+ "coding": 0.9851,
1783
+ "coding_agent": 0.0054,
1784
+ "document": 0.0013,
1785
+ "general": 0.007,
1786
+ "math": 0.0005,
1787
+ "tools": 0.0006
1788
+ },
1789
+ "latency_s": 0.069
1790
+ },
1791
+ {
1792
+ "id": "original-routing-03-0",
1793
+ "tier": "standard",
1794
+ "expected": "coding_agent",
1795
+ "predicted": "coding_agent",
1796
+ "correct": true,
1797
+ "valid": true,
1798
+ "error": null,
1799
+ "probs": {
1800
+ "coding": 0.0011,
1801
+ "coding_agent": 0.9965,
1802
+ "document": 0.0005,
1803
+ "general": 0.0011,
1804
+ "math": 0.0002,
1805
+ "tools": 0.0005
1806
+ },
1807
+ "latency_s": 0.068
1808
+ },
1809
+ {
1810
+ "id": "original-routing-03-1",
1811
+ "tier": "standard",
1812
+ "expected": "coding_agent",
1813
+ "predicted": "coding_agent",
1814
+ "correct": true,
1815
+ "valid": true,
1816
+ "error": null,
1817
+ "probs": {
1818
+ "coding": 0.0007,
1819
+ "coding_agent": 0.9959,
1820
+ "document": 0.0006,
1821
+ "general": 0.0021,
1822
+ "math": 0.0002,
1823
+ "tools": 0.0005
1824
+ },
1825
+ "latency_s": 0.069
1826
+ },
1827
+ {
1828
+ "id": "original-routing-04-0",
1829
+ "tier": "standard",
1830
+ "expected": "document",
1831
+ "predicted": "general",
1832
+ "correct": false,
1833
+ "valid": true,
1834
+ "error": null,
1835
+ "probs": {
1836
+ "coding": 0.0029,
1837
+ "coding_agent": 0.004,
1838
+ "document": 0.4781,
1839
+ "general": 0.5045,
1840
+ "math": 0.002,
1841
+ "tools": 0.0085
1842
+ },
1843
+ "latency_s": 0.068
1844
+ },
1845
+ {
1846
+ "id": "original-routing-04-1",
1847
+ "tier": "standard",
1848
+ "expected": "document",
1849
+ "predicted": "document",
1850
+ "correct": true,
1851
+ "valid": true,
1852
+ "error": null,
1853
+ "probs": {
1854
+ "coding": 0.0017,
1855
+ "coding_agent": 0.0022,
1856
+ "document": 0.8249,
1857
+ "general": 0.1648,
1858
+ "math": 0.0011,
1859
+ "tools": 0.0053
1860
+ },
1861
+ "latency_s": 0.068
1862
+ },
1863
+ {
1864
+ "id": "original-routing-05-0",
1865
+ "tier": "standard",
1866
+ "expected": "tools",
1867
+ "predicted": "tools",
1868
+ "correct": true,
1869
+ "valid": true,
1870
+ "error": null,
1871
+ "probs": {
1872
+ "coding": 0.0022,
1873
+ "coding_agent": 0.0014,
1874
+ "document": 0.0065,
1875
+ "general": 0.4283,
1876
+ "math": 0.0013,
1877
+ "tools": 0.5602
1878
+ },
1879
+ "latency_s": 0.068
1880
+ },
1881
+ {
1882
+ "id": "original-routing-05-1",
1883
+ "tier": "standard",
1884
+ "expected": "tools",
1885
+ "predicted": "general",
1886
+ "correct": false,
1887
+ "valid": true,
1888
+ "error": null,
1889
+ "probs": {
1890
+ "coding": 0.001,
1891
+ "coding_agent": 0.0009,
1892
+ "document": 0.0029,
1893
+ "general": 0.9046,
1894
+ "math": 0.0007,
1895
+ "tools": 0.0899
1896
+ },
1897
+ "latency_s": 0.068
1898
+ },
1899
+ {
1900
+ "id": "original-routing-06-0",
1901
+ "tier": "standard",
1902
+ "expected": "general",
1903
+ "predicted": "general",
1904
+ "correct": true,
1905
+ "valid": true,
1906
+ "error": null,
1907
+ "probs": {
1908
+ "coding": 0.0007,
1909
+ "coding_agent": 0.0008,
1910
+ "document": 0.0051,
1911
+ "general": 0.9912,
1912
+ "math": 0.0007,
1913
+ "tools": 0.0013
1914
+ },
1915
+ "latency_s": 0.068
1916
+ },
1917
+ {
1918
+ "id": "original-routing-06-1",
1919
+ "tier": "standard",
1920
+ "expected": "general",
1921
+ "predicted": "general",
1922
+ "correct": true,
1923
+ "valid": true,
1924
+ "error": null,
1925
+ "probs": {
1926
+ "coding": 0.0009,
1927
+ "coding_agent": 0.0008,
1928
+ "document": 0.0046,
1929
+ "general": 0.9914,
1930
+ "math": 0.0007,
1931
+ "tools": 0.0016
1932
+ },
1933
+ "latency_s": 0.067
1934
+ },
1935
+ {
1936
+ "id": "hard-opus-a-long_policy-01",
1937
+ "tier": "hard",
1938
+ "expected": "pay_subject_to_15000_sublimit",
1939
+ "predicted": "pay_subject_to_15000_sublimit",
1940
+ "correct": true,
1941
+ "valid": true,
1942
+ "error": null,
1943
+ "probs": {
1944
+ "deny_repeated_seepage": 0.0007,
1945
+ "deny_vacancy_exclusion": 0.0011,
1946
+ "pay_full_estimate_less_deductible": 0.0011,
1947
+ "pay_subject_to_10000_sublimit": 0.0327,
1948
+ "pay_subject_to_15000_sublimit": 0.9644
1949
+ },
1950
+ "latency_s": 0.154
1951
+ },
1952
+ {
1953
+ "id": "hard-opus-a-long_policy-04",
1954
+ "tier": "hard",
1955
+ "expected": "cfo",
1956
+ "predicted": "vp_and_finance_director",
1957
+ "correct": false,
1958
+ "valid": true,
1959
+ "error": null,
1960
+ "probs": {
1961
+ "budget_holder": 0.0005,
1962
+ "cfo": 0.0009,
1963
+ "department_head": 0.0029,
1964
+ "executive_committee": 0.0007,
1965
+ "vp_and_finance_director": 0.995
1966
+ },
1967
+ "latency_s": 0.101
1968
+ },
1969
+ {
1970
+ "id": "hard-opus-a-long_policy-08",
1971
+ "tier": "hard",
1972
+ "expected": "parts_covered_labour_charged",
1973
+ "predicted": "parts_covered_labour_charged",
1974
+ "correct": true,
1975
+ "valid": true,
1976
+ "error": null,
1977
+ "probs": {
1978
+ "full_repair_no_charge": 0.0127,
1979
+ "not_covered": 0.0046,
1980
+ "parts_covered_labour_charged": 0.9797,
1981
+ "pro_rata_replacement_credit": 0.0031
1982
+ },
1983
+ "latency_s": 0.083
1984
+ },
1985
+ {
1986
+ "id": "hard-opus-a-long_policy-09",
1987
+ "tier": "hard",
1988
+ "expected": "license_required",
1989
+ "predicted": "license_required",
1990
+ "correct": true,
1991
+ "valid": true,
1992
+ "error": null,
1993
+ "probs": {
1994
+ "license_exception_lvs": 0.0002,
1995
+ "license_required": 0.9993,
1996
+ "no_license_required": 0.0003,
1997
+ "shipment_prohibited": 0.0002
1998
+ },
1999
+ "latency_s": 0.086
2000
+ },
2001
+ {
2002
+ "id": "hard-opus-a-long_policy-11",
2003
+ "tier": "hard",
2004
+ "expected": "yes",
2005
+ "predicted": "no",
2006
+ "correct": false,
2007
+ "valid": true,
2008
+ "error": null,
2009
+ "probs": {
2010
+ "yes": 0.0049,
2011
+ "no": 0.9951
2012
+ },
2013
+ "latency_s": 0.082
2014
+ },
2015
+ {
2016
+ "id": "hard-opus-a-long_policy-13",
2017
+ "tier": "hard",
2018
+ "expected": "yes",
2019
+ "predicted": "no",
2020
+ "correct": false,
2021
+ "valid": true,
2022
+ "error": null,
2023
+ "probs": {
2024
+ "yes": 0.2127,
2025
+ "no": 0.7873
2026
+ },
2027
+ "latency_s": 0.083
2028
+ },
2029
+ {
2030
+ "id": "hard-opus-a-long_policy-17",
2031
+ "tier": "hard",
2032
+ "expected": "database_oncall",
2033
+ "predicted": "database_oncall",
2034
+ "correct": true,
2035
+ "valid": true,
2036
+ "error": null,
2037
+ "probs": {
2038
+ "database_oncall": 0.9962,
2039
+ "no_page_ticket_only": 0.0012,
2040
+ "platform_oncall": 0.0013,
2041
+ "security_incident_response": 0.0005,
2042
+ "vendor_support": 0.0008
2043
+ },
2044
+ "latency_s": 0.082
2045
+ },
2046
+ {
2047
+ "id": "hard-opus-a-long_policy-19",
2048
+ "tier": "hard",
2049
+ "expected": "no",
2050
+ "predicted": "no",
2051
+ "correct": true,
2052
+ "valid": true,
2053
+ "error": null,
2054
+ "probs": {
2055
+ "yes": 0.0081,
2056
+ "no": 0.9919
2057
+ },
2058
+ "latency_s": 0.081
2059
+ },
2060
+ {
2061
+ "id": "hard-opus-a-probability-03",
2062
+ "tier": "hard",
2063
+ "expected": "yes",
2064
+ "predicted": "no",
2065
+ "correct": false,
2066
+ "valid": true,
2067
+ "error": null,
2068
+ "probs": {
2069
+ "yes": 0.4425,
2070
+ "no": 0.5575
2071
+ },
2072
+ "latency_s": 0.08
2073
+ },
2074
+ {
2075
+ "id": "hard-opus-a-probability-04",
2076
+ "tier": "hard",
2077
+ "expected": "on_time",
2078
+ "predicted": "on_time",
2079
+ "correct": true,
2080
+ "valid": true,
2081
+ "error": null,
2082
+ "probs": {
2083
+ "early": 0.0011,
2084
+ "late": 0.0364,
2085
+ "on_time": 0.9625
2086
+ },
2087
+ "latency_s": 0.086
2088
+ },
2089
+ {
2090
+ "id": "hard-opus-a-probability-07",
2091
+ "tier": "hard",
2092
+ "expected": "escalated",
2093
+ "predicted": "escalated",
2094
+ "correct": true,
2095
+ "valid": true,
2096
+ "error": null,
2097
+ "probs": {
2098
+ "escalated": 0.9957,
2099
+ "reopened": 0.001,
2100
+ "resolved_first_contact": 0.0034
2101
+ },
2102
+ "latency_s": 0.073
2103
+ },
2104
+ {
2105
+ "id": "hard-opus-a-probability-08",
2106
+ "tier": "hard",
2107
+ "expected": "no",
2108
+ "predicted": "no",
2109
+ "correct": true,
2110
+ "valid": true,
2111
+ "error": null,
2112
+ "probs": {
2113
+ "yes": 0.4616,
2114
+ "no": 0.5384
2115
+ },
2116
+ "latency_s": 0.07
2117
+ },
2118
+ {
2119
+ "id": "hard-opus-a-temporal_numeric-03",
2120
+ "tier": "hard",
2121
+ "expected": "yes",
2122
+ "predicted": "yes",
2123
+ "correct": true,
2124
+ "valid": true,
2125
+ "error": null,
2126
+ "probs": {
2127
+ "yes": 0.7312,
2128
+ "no": 0.2688
2129
+ },
2130
+ "latency_s": 0.069
2131
+ },
2132
+ {
2133
+ "id": "hard-opus-a-temporal_numeric-06",
2134
+ "tier": "hard",
2135
+ "expected": "no",
2136
+ "predicted": "no",
2137
+ "correct": true,
2138
+ "valid": true,
2139
+ "error": null,
2140
+ "probs": {
2141
+ "yes": 0.038,
2142
+ "no": 0.962
2143
+ },
2144
+ "latency_s": 0.069
2145
+ },
2146
+ {
2147
+ "id": "hard-opus-a-temporal_numeric-07",
2148
+ "tier": "hard",
2149
+ "expected": "sep_26",
2150
+ "predicted": "sep_28",
2151
+ "correct": false,
2152
+ "valid": true,
2153
+ "error": null,
2154
+ "probs": {
2155
+ "sep_24": 0.3247,
2156
+ "sep_25": 0.0116,
2157
+ "sep_26": 0.003,
2158
+ "sep_27": 0.008,
2159
+ "sep_28": 0.6526
2160
+ },
2161
+ "latency_s": 0.072
2162
+ },
2163
+ {
2164
+ "id": "hard-opus-a-temporal_numeric-09",
2165
+ "tier": "hard",
2166
+ "expected": "no",
2167
+ "predicted": "no",
2168
+ "correct": true,
2169
+ "valid": true,
2170
+ "error": null,
2171
+ "probs": {
2172
+ "yes": 0.1605,
2173
+ "no": 0.8395
2174
+ },
2175
+ "latency_s": 0.072
2176
+ },
2177
+ {
2178
+ "id": "hard-opus-a-temporal_numeric-12",
2179
+ "tier": "hard",
2180
+ "expected": "1",
2181
+ "predicted": "1",
2182
+ "correct": true,
2183
+ "valid": true,
2184
+ "error": null,
2185
+ "probs": {
2186
+ "0": 0.0918,
2187
+ "1": 0.519,
2188
+ "2": 0.3855,
2189
+ "3": 0.0036
2190
+ },
2191
+ "latency_s": 0.078
2192
+ },
2193
+ {
2194
+ "id": "hard-opus-b-ambiguous-02",
2195
+ "tier": "hard",
2196
+ "expected": "real_incident",
2197
+ "predicted": "real_incident",
2198
+ "correct": true,
2199
+ "valid": true,
2200
+ "error": null,
2201
+ "probs": {
2202
+ "cannot_determine": 0.0096,
2203
+ "false_positive": 0.0183,
2204
+ "real_incident": 0.9721
2205
+ },
2206
+ "latency_s": 0.073
2207
+ },
2208
+ {
2209
+ "id": "hard-opus-b-ambiguous-03",
2210
+ "tier": "hard",
2211
+ "expected": "time_barred",
2212
+ "predicted": "within_limitation",
2213
+ "correct": false,
2214
+ "valid": true,
2215
+ "error": null,
2216
+ "probs": {
2217
+ "insufficient_information": 0.0015,
2218
+ "time_barred": 0.3559,
2219
+ "within_limitation": 0.6425
2220
+ },
2221
+ "latency_s": 0.07
2222
+ },
2223
+ {
2224
+ "id": "hard-opus-b-ambiguous-07",
2225
+ "tier": "hard",
2226
+ "expected": "within_policy",
2227
+ "predicted": "within_policy",
2228
+ "correct": true,
2229
+ "valid": true,
2230
+ "error": null,
2231
+ "probs": {
2232
+ "insufficient_information": 0.096,
2233
+ "over_limit": 0.0817,
2234
+ "within_policy": 0.8223
2235
+ },
2236
+ "latency_s": 0.07
2237
+ },
2238
+ {
2239
+ "id": "hard-opus-b-ambiguous-09",
2240
+ "tier": "hard",
2241
+ "expected": "legitimate",
2242
+ "predicted": "legitimate",
2243
+ "correct": true,
2244
+ "valid": true,
2245
+ "error": null,
2246
+ "probs": {
2247
+ "compromised": 0.2069,
2248
+ "inconclusive": 0.001,
2249
+ "legitimate": 0.7921
2250
+ },
2251
+ "latency_s": 0.072
2252
+ },
2253
+ {
2254
+ "id": "hard-opus-b-ambiguous-10",
2255
+ "tier": "hard",
2256
+ "expected": "ask_user_which_alex",
2257
+ "predicted": "send_to_alex_moreno",
2258
+ "correct": false,
2259
+ "valid": true,
2260
+ "error": null,
2261
+ "probs": {
2262
+ "ask_user_which_alex": 0.0985,
2263
+ "send_to_alex_moreno": 0.8906,
2264
+ "send_to_alexandra_lind": 0.0109
2265
+ },
2266
+ "latency_s": 0.07
2267
+ },
2268
+ {
2269
+ "id": "hard-opus-b-ambiguous-11",
2270
+ "tier": "hard",
2271
+ "expected": "cannot_determine",
2272
+ "predicted": "cannot_determine",
2273
+ "correct": true,
2274
+ "valid": true,
2275
+ "error": null,
2276
+ "probs": {
2277
+ "cannot_determine": 0.9923,
2278
+ "grade_correct": 0.0067,
2279
+ "grade_should_be_higher": 0.001
2280
+ },
2281
+ "latency_s": 0.073
2282
+ },
2283
+ {
2284
+ "id": "hard-opus-b-ambiguous-13",
2285
+ "tier": "hard",
2286
+ "expected": "cannot_determine",
2287
+ "predicted": "cannot_determine",
2288
+ "correct": true,
2289
+ "valid": true,
2290
+ "error": null,
2291
+ "probs": {
2292
+ "cannot_determine": 0.9787,
2293
+ "charge_data_team": 0.0078,
2294
+ "charge_platform_team": 0.0092,
2295
+ "split_50_50": 0.0043
2296
+ },
2297
+ "latency_s": 0.07
2298
+ },
2299
+ {
2300
+ "id": "hard-opus-b-multi_hop-03",
2301
+ "tier": "hard",
2302
+ "expected": "tier3_cfo",
2303
+ "predicted": "tier3_cfo",
2304
+ "correct": true,
2305
+ "valid": true,
2306
+ "error": null,
2307
+ "probs": {
2308
+ "tier1_team_lead": 0.0675,
2309
+ "tier2_department_head": 0.0394,
2310
+ "tier3_cfo": 0.888,
2311
+ "tier4_board": 0.0051
2312
+ },
2313
+ "latency_s": 0.09
2314
+ },
2315
+ {
2316
+ "id": "hard-opus-b-multi_hop-04",
2317
+ "tier": "hard",
2318
+ "expected": "bjorn",
2319
+ "predicted": "chen",
2320
+ "correct": false,
2321
+ "valid": true,
2322
+ "error": null,
2323
+ "probs": {
2324
+ "ana": 0.0552,
2325
+ "bjorn": 0.1235,
2326
+ "chen": 0.6185,
2327
+ "dmitri": 0.0099,
2328
+ "esra": 0.013,
2329
+ "no_page_ticket_only": 0.1799
2330
+ },
2331
+ "latency_s": 0.081
2332
+ },
2333
+ {
2334
+ "id": "hard-opus-b-multi_hop-05",
2335
+ "tier": "hard",
2336
+ "expected": "2",
2337
+ "predicted": "2",
2338
+ "correct": true,
2339
+ "valid": true,
2340
+ "error": null,
2341
+ "probs": {
2342
+ "0": 0.0009,
2343
+ "1": 0.0067,
2344
+ "2": 0.9879,
2345
+ "3": 0.0022,
2346
+ "4": 0.0022
2347
+ },
2348
+ "latency_s": 0.369
2349
+ },
2350
+ {
2351
+ "id": "hard-opus-b-multi_hop-07",
2352
+ "tier": "hard",
2353
+ "expected": "credit_20_percent",
2354
+ "predicted": "credit_50_percent",
2355
+ "correct": false,
2356
+ "valid": true,
2357
+ "error": null,
2358
+ "probs": {
2359
+ "credit_10_percent": 0.0045,
2360
+ "credit_20_percent": 0.0072,
2361
+ "credit_25_percent": 0.1625,
2362
+ "credit_50_percent": 0.6562,
2363
+ "credit_full_month": 0.1625,
2364
+ "no_credit": 0.0072
2365
+ },
2366
+ "latency_s": 0.088
2367
+ },
2368
+ {
2369
+ "id": "hard-opus-b-multi_hop-08",
2370
+ "tier": "hard",
2371
+ "expected": "no",
2372
+ "predicted": "no",
2373
+ "correct": true,
2374
+ "valid": true,
2375
+ "error": null,
2376
+ "probs": {
2377
+ "yes": 0.0366,
2378
+ "no": 0.9634
2379
+ },
2380
+ "latency_s": 0.077
2381
+ },
2382
+ {
2383
+ "id": "hard-opus-b-probability-01",
2384
+ "tier": "hard",
2385
+ "expected": "merchant_wins",
2386
+ "predicted": "merchant_wins",
2387
+ "correct": true,
2388
+ "valid": true,
2389
+ "error": null,
2390
+ "probs": {
2391
+ "customer_wins": 0.0006,
2392
+ "customer_withdraws": 0.0004,
2393
+ "merchant_wins": 0.9991
2394
+ },
2395
+ "latency_s": 0.074
2396
+ },
2397
+ {
2398
+ "id": "hard-opus-b-probability-02",
2399
+ "tier": "hard",
2400
+ "expected": "upstream_provider",
2401
+ "predicted": "upstream_provider",
2402
+ "correct": true,
2403
+ "valid": true,
2404
+ "error": null,
2405
+ "probs": {
2406
+ "bad_push": 0.0149,
2407
+ "database_hardware": 0.0021,
2408
+ "upstream_provider": 0.983
2409
+ },
2410
+ "latency_s": 0.07
2411
+ },
2412
+ {
2413
+ "id": "hard-opus-b-probability-03",
2414
+ "tier": "hard",
2415
+ "expected": "yes",
2416
+ "predicted": "yes",
2417
+ "correct": true,
2418
+ "valid": true,
2419
+ "error": null,
2420
+ "probs": {
2421
+ "yes": 0.9269,
2422
+ "no": 0.0731
2423
+ },
2424
+ "latency_s": 0.072
2425
+ },
2426
+ {
2427
+ "id": "hard-opus-b-probability-04",
2428
+ "tier": "hard",
2429
+ "expected": "attended",
2430
+ "predicted": "attended",
2431
+ "correct": true,
2432
+ "valid": true,
2433
+ "error": null,
2434
+ "probs": {
2435
+ "attended": 0.9984,
2436
+ "late_cancellation": 0.001,
2437
+ "no_show": 0.0006
2438
+ },
2439
+ "latency_s": 0.072
2440
+ },
2441
+ {
2442
+ "id": "hard-opus-b-probability-06",
2443
+ "tier": "hard",
2444
+ "expected": "upheld",
2445
+ "predicted": "upheld",
2446
+ "correct": true,
2447
+ "valid": true,
2448
+ "error": null,
2449
+ "probs": {
2450
+ "modified": 0.0001,
2451
+ "overturned": 0.0006,
2452
+ "upheld": 0.9993
2453
+ },
2454
+ "latency_s": 0.072
2455
+ },
2456
+ {
2457
+ "id": "hard-opus-b-tradeoff-01",
2458
+ "tier": "hard",
2459
+ "expected": "escalate_to_engineering",
2460
+ "predicted": "escalate_to_engineering",
2461
+ "correct": true,
2462
+ "valid": true,
2463
+ "error": null,
2464
+ "probs": {
2465
+ "escalate_to_engineering": 0.8595,
2466
+ "escalate_to_security": 0.1385,
2467
+ "page_oncall": 0.0009,
2468
+ "process_refund": 0.0007,
2469
+ "send_kb_article": 0.0004
2470
+ },
2471
+ "latency_s": 0.073
2472
+ },
2473
+ {
2474
+ "id": "hard-opus-b-tradeoff-03",
2475
+ "tier": "hard",
2476
+ "expected": "order_b",
2477
+ "predicted": "order_b",
2478
+ "correct": true,
2479
+ "valid": true,
2480
+ "error": null,
2481
+ "probs": {
2482
+ "order_a": 0.003,
2483
+ "order_b": 0.9929,
2484
+ "order_c": 0.0018,
2485
+ "order_d": 0.0023
2486
+ },
2487
+ "latency_s": 0.072
2488
+ },
2489
+ {
2490
+ "id": "hard-opus-b-tradeoff-06",
2491
+ "tier": "hard",
2492
+ "expected": "yes",
2493
+ "predicted": "yes",
2494
+ "correct": true,
2495
+ "valid": true,
2496
+ "error": null,
2497
+ "probs": {
2498
+ "yes": 0.966,
2499
+ "no": 0.034
2500
+ },
2501
+ "latency_s": 0.071
2502
+ },
2503
+ {
2504
+ "id": "hard-opus-b-tradeoff-07",
2505
+ "tier": "hard",
2506
+ "expected": "p0_fix_24h",
2507
+ "predicted": "p0_fix_24h",
2508
+ "correct": true,
2509
+ "valid": true,
2510
+ "error": null,
2511
+ "probs": {
2512
+ "out_of_scope": 0.0001,
2513
+ "p0_fix_24h": 0.9994,
2514
+ "p1_fix_7d": 0.0003,
2515
+ "p2_fix_30d": 0.0001,
2516
+ "p3_backlog": 0.0001
2517
+ },
2518
+ "latency_s": 0.073
2519
+ },
2520
+ {
2521
+ "id": "hard-opus-b-tradeoff-08",
2522
+ "tier": "hard",
2523
+ "expected": "yes",
2524
+ "predicted": "no",
2525
+ "correct": false,
2526
+ "valid": true,
2527
+ "error": null,
2528
+ "probs": {
2529
+ "yes": 0.4904,
2530
+ "no": 0.5096
2531
+ },
2532
+ "latency_s": 0.072
2533
+ },
2534
+ {
2535
+ "id": "hard-opus-b-tradeoff-12",
2536
+ "tier": "hard",
2537
+ "expected": "refund_now",
2538
+ "predicted": "refund_now",
2539
+ "correct": true,
2540
+ "valid": true,
2541
+ "error": null,
2542
+ "probs": {
2543
+ "deny_refund": 0.0001,
2544
+ "escalate_to_fraud_team": 0.0001,
2545
+ "offer_store_credit": 0.0002,
2546
+ "refund_now": 0.9996,
2547
+ "wait_for_inspection": 0.0001
2548
+ },
2549
+ "latency_s": 0.072
2550
+ },
2551
+ {
2552
+ "id": "hard-opus-c-long_policy-02",
2553
+ "tier": "hard",
2554
+ "expected": "erase_but_retain_transaction_records",
2555
+ "predicted": "restrict_all_pending_legal_hold",
2556
+ "correct": false,
2557
+ "valid": true,
2558
+ "error": null,
2559
+ "probs": {
2560
+ "erase_all": 0.0004,
2561
+ "erase_but_retain_transaction_records": 0.0216,
2562
+ "erase_marketing_only": 0.0004,
2563
+ "reject_unverified_request": 0.0002,
2564
+ "restrict_all_pending_legal_hold": 0.9773
2565
+ },
2566
+ "latency_s": 0.118
2567
+ },
2568
+ {
2569
+ "id": "hard-opus-c-long_policy-03",
2570
+ "tier": "hard",
2571
+ "expected": "eur_400",
2572
+ "predicted": "eur_150",
2573
+ "correct": false,
2574
+ "valid": true,
2575
+ "error": null,
2576
+ "probs": {
2577
+ "eur_0": 0.0407,
2578
+ "eur_150": 0.4322,
2579
+ "eur_200": 0.215,
2580
+ "eur_300": 0.0593,
2581
+ "eur_400": 0.1399,
2582
+ "eur_600": 0.1129
2583
+ },
2584
+ "latency_s": 0.117
2585
+ },
2586
+ {
2587
+ "id": "hard-opus-c-long_policy-04",
2588
+ "tier": "hard",
2589
+ "expected": "yes",
2590
+ "predicted": "yes",
2591
+ "correct": true,
2592
+ "valid": true,
2593
+ "error": null,
2594
+ "probs": {
2595
+ "yes": 0.6134,
2596
+ "no": 0.3866
2597
+ },
2598
+ "latency_s": 0.105
2599
+ },
2600
+ {
2601
+ "id": "hard-opus-c-long_policy-05",
2602
+ "tier": "hard",
2603
+ "expected": "1",
2604
+ "predicted": "2",
2605
+ "correct": false,
2606
+ "valid": true,
2607
+ "error": null,
2608
+ "probs": {
2609
+ "0": 0.0088,
2610
+ "1": 0.3272,
2611
+ "2": 0.6427,
2612
+ "3": 0.0212
2613
+ },
2614
+ "latency_s": 0.373
2615
+ },
2616
+ {
2617
+ "id": "hard-opus-c-long_policy-08",
2618
+ "tier": "hard",
2619
+ "expected": "1",
2620
+ "predicted": "1",
2621
+ "correct": true,
2622
+ "valid": true,
2623
+ "error": null,
2624
+ "probs": {
2625
+ "0": 0.0013,
2626
+ "1": 0.9959,
2627
+ "2": 0.0012,
2628
+ "3": 0.0017
2629
+ },
2630
+ "latency_s": 0.377
2631
+ },
2632
+ {
2633
+ "id": "hard-opus-c-long_policy-10",
2634
+ "tier": "hard",
2635
+ "expected": "credit_minus_restocking_fee",
2636
+ "predicted": "credit_minus_restocking_fee",
2637
+ "correct": true,
2638
+ "valid": true,
2639
+ "error": null,
2640
+ "probs": {
2641
+ "credit_minus_restocking_fee": 0.9975,
2642
+ "full_credit": 0.001,
2643
+ "reject_return": 0.0011,
2644
+ "repair_or_replace_only": 0.0004
2645
+ },
2646
+ "latency_s": 0.121
2647
+ },
2648
+ {
2649
+ "id": "hard-opus-c-long_policy-11",
2650
+ "tier": "hard",
2651
+ "expected": "no",
2652
+ "predicted": "yes",
2653
+ "correct": false,
2654
+ "valid": true,
2655
+ "error": null,
2656
+ "probs": {
2657
+ "yes": 0.7235,
2658
+ "no": 0.2765
2659
+ },
2660
+ "latency_s": 0.106
2661
+ },
2662
+ {
2663
+ "id": "hard-opus-c-probability-03",
2664
+ "tier": "hard",
2665
+ "expected": "kuznets_technik",
2666
+ "predicted": "halvorsen_polymer",
2667
+ "correct": false,
2668
+ "valid": true,
2669
+ "error": null,
2670
+ "probs": {
2671
+ "halvorsen_polymer": 0.9962,
2672
+ "kuznets_technik": 0.0018,
2673
+ "ortega_seal_works": 0.0021
2674
+ },
2675
+ "latency_s": 0.073
2676
+ },
2677
+ {
2678
+ "id": "hard-opus-c-temporal_numeric-02",
2679
+ "tier": "hard",
2680
+ "expected": "yes",
2681
+ "predicted": "no",
2682
+ "correct": false,
2683
+ "valid": true,
2684
+ "error": null,
2685
+ "probs": {
2686
+ "yes": 0.1766,
2687
+ "no": 0.8234
2688
+ },
2689
+ "latency_s": 0.073
2690
+ },
2691
+ {
2692
+ "id": "hard-opus-c-temporal_numeric-03",
2693
+ "tier": "hard",
2694
+ "expected": "eur_692_00",
2695
+ "predicted": "eur_693_90",
2696
+ "correct": false,
2697
+ "valid": true,
2698
+ "error": null,
2699
+ "probs": {
2700
+ "eur_617_00": 0.0165,
2701
+ "eur_688_00": 0.0538,
2702
+ "eur_692_00": 0.2553,
2703
+ "eur_693_90": 0.6712,
2704
+ "eur_812_00": 0.0031
2705
+ },
2706
+ "latency_s": 0.075
2707
+ },
2708
+ {
2709
+ "id": "hard-opus-c-temporal_numeric-04",
2710
+ "tier": "hard",
2711
+ "expected": "usd_2335_00",
2712
+ "predicted": "usd_2303_01",
2713
+ "correct": false,
2714
+ "valid": true,
2715
+ "error": null,
2716
+ "probs": {
2717
+ "usd_2222_50": 0.0251,
2718
+ "usd_2303_01": 0.597,
2719
+ "usd_2303_33": 0.3134,
2720
+ "usd_2335_00": 0.0533,
2721
+ "usd_2340_00": 0.0112
2722
+ },
2723
+ "latency_s": 0.074
2724
+ },
2725
+ {
2726
+ "id": "hard-opus-c-temporal_numeric-06",
2727
+ "tier": "hard",
2728
+ "expected": "sep_18_book_set",
2729
+ "predicted": "none_declined",
2730
+ "correct": false,
2731
+ "valid": true,
2732
+ "error": null,
2733
+ "probs": {
2734
+ "none_declined": 0.6977,
2735
+ "sep_15_office_chair": 0.101,
2736
+ "sep_16_taxi": 0.0202,
2737
+ "sep_17_parking": 0.1252,
2738
+ "sep_18_book_set": 0.0405,
2739
+ "sep_21_client_dinner": 0.0154
2740
+ },
2741
+ "latency_s": 0.074
2742
+ },
2743
+ {
2744
+ "id": "hard-opus-c-temporal_numeric-08",
2745
+ "tier": "hard",
2746
+ "expected": "eur_498_81",
2747
+ "predicted": "eur_513_69",
2748
+ "correct": false,
2749
+ "valid": true,
2750
+ "error": null,
2751
+ "probs": {
2752
+ "eur_497_40": 0.0582,
2753
+ "eur_498_81": 0.2482,
2754
+ "eur_506_06": 0.1451,
2755
+ "eur_513_69": 0.4989,
2756
+ "eur_532_01": 0.0496
2757
+ },
2758
+ "latency_s": 0.074
2759
+ },
2760
+ {
2761
+ "id": "hard-opus-c-temporal_numeric-12",
2762
+ "tier": "hard",
2763
+ "expected": "expired_30_month_cap",
2764
+ "predicted": "expired_30_month_cap",
2765
+ "correct": true,
2766
+ "valid": true,
2767
+ "error": null,
2768
+ "probs": {
2769
+ "covered": 0.0038,
2770
+ "expired_24_month_term": 0.2964,
2771
+ "expired_30_month_cap": 0.6998
2772
+ },
2773
+ "latency_s": 0.073
2774
+ },
2775
+ {
2776
+ "id": "hard-sol-a-adversarial-01",
2777
+ "tier": "hard",
2778
+ "expected": "deny_missing_approval",
2779
+ "predicted": "deny_missing_approval",
2780
+ "correct": true,
2781
+ "valid": true,
2782
+ "error": null,
2783
+ "probs": {
2784
+ "deny_inactive_user": 0.0001,
2785
+ "deny_missing_approval": 0.9991,
2786
+ "grant_access": 0.0005,
2787
+ "grant_owner_exception": 0.0003
2788
+ },
2789
+ "latency_s": 0.071
2790
+ },
2791
+ {
2792
+ "id": "hard-sol-a-adversarial-06",
2793
+ "tier": "hard",
2794
+ "expected": "no",
2795
+ "predicted": "no",
2796
+ "correct": true,
2797
+ "valid": true,
2798
+ "error": null,
2799
+ "probs": {
2800
+ "yes": 0.0039,
2801
+ "no": 0.9961
2802
+ },
2803
+ "latency_s": 0.07
2804
+ },
2805
+ {
2806
+ "id": "hard-sol-a-adversarial-07",
2807
+ "tier": "hard",
2808
+ "expected": "retain_until_2031",
2809
+ "predicted": "retain_until_2031",
2810
+ "correct": true,
2811
+ "valid": true,
2812
+ "error": null,
2813
+ "probs": {
2814
+ "delete_now": 0.0004,
2815
+ "needs_signature_review": 0.0001,
2816
+ "retain_indefinitely": 0.0002,
2817
+ "retain_until_2031": 0.9993
2818
+ },
2819
+ "latency_s": 0.07
2820
+ },
2821
+ {
2822
+ "id": "hard-sol-a-adversarial-08",
2823
+ "tier": "hard",
2824
+ "expected": "yes",
2825
+ "predicted": "yes",
2826
+ "correct": true,
2827
+ "valid": true,
2828
+ "error": null,
2829
+ "probs": {
2830
+ "yes": 0.9959,
2831
+ "no": 0.0041
2832
+ },
2833
+ "latency_s": 0.07
2834
+ },
2835
+ {
2836
+ "id": "hard-sol-a-adversarial-09",
2837
+ "tier": "hard",
2838
+ "expected": "2",
2839
+ "predicted": "2",
2840
+ "correct": true,
2841
+ "valid": true,
2842
+ "error": null,
2843
+ "probs": {
2844
+ "0": 0.0189,
2845
+ "1": 0.0945,
2846
+ "2": 0.8729,
2847
+ "3": 0.0136
2848
+ },
2849
+ "latency_s": 0.073
2850
+ },
2851
+ {
2852
+ "id": "hard-sol-a-adversarial-11",
2853
+ "tier": "hard",
2854
+ "expected": "standard_research",
2855
+ "predicted": "ethics_review",
2856
+ "correct": false,
2857
+ "valid": true,
2858
+ "error": null,
2859
+ "probs": {
2860
+ "ethics_review": 0.6565,
2861
+ "major_grants": 0.0137,
2862
+ "manual_hold": 0.0031,
2863
+ "standard_research": 0.3266
2864
+ },
2865
+ "latency_s": 0.07
2866
+ },
2867
+ {
2868
+ "id": "hard-sol-a-multi_hop-01",
2869
+ "tier": "hard",
2870
+ "expected": "accept_s2_dusk",
2871
+ "predicted": "accept_s2_dusk",
2872
+ "correct": true,
2873
+ "valid": true,
2874
+ "error": null,
2875
+ "probs": {
2876
+ "accept_s2_dusk": 0.6402,
2877
+ "reject_cutoff": 0.3546,
2878
+ "reject_tier": 0.003,
2879
+ "reroute_s3": 0.0022
2880
+ },
2881
+ "latency_s": 0.085
2882
+ },
2883
+ {
2884
+ "id": "hard-sol-a-multi_hop-02",
2885
+ "tier": "hard",
2886
+ "expected": "security_primary",
2887
+ "predicted": "executive_bridge",
2888
+ "correct": false,
2889
+ "valid": true,
2890
+ "error": null,
2891
+ "probs": {
2892
+ "executive_bridge": 0.9964,
2893
+ "heron_q2": 0.0013,
2894
+ "insufficient_information": 0.0005,
2895
+ "security_primary": 0.0018
2896
+ },
2897
+ "latency_s": 0.084
2898
+ },
2899
+ {
2900
+ "id": "hard-sol-a-multi_hop-05",
2901
+ "tier": "hard",
2902
+ "expected": "require_director",
2903
+ "predicted": "require_director",
2904
+ "correct": true,
2905
+ "valid": true,
2906
+ "error": null,
2907
+ "probs": {
2908
+ "pay_manager_approved": 0.0133,
2909
+ "reject_category": 0.0074,
2910
+ "require_director": 0.9778,
2911
+ "require_finance_board": 0.0015
2912
+ },
2913
+ "latency_s": 0.083
2914
+ },
2915
+ {
2916
+ "id": "hard-sol-a-multi_hop-07",
2917
+ "tier": "hard",
2918
+ "expected": "admit",
2919
+ "predicted": "deny_prerequisite",
2920
+ "correct": false,
2921
+ "valid": true,
2922
+ "error": null,
2923
+ "probs": {
2924
+ "admit": 0.2409,
2925
+ "deny_prerequisite": 0.7438,
2926
+ "deny_score": 0.0091,
2927
+ "manual_equivalency": 0.0063
2928
+ },
2929
+ "latency_s": 0.082
2930
+ },
2931
+ {
2932
+ "id": "hard-sol-a-multi_hop-08",
2933
+ "tier": "hard",
2934
+ "expected": "central_engineering",
2935
+ "predicted": "central_engineering",
2936
+ "correct": true,
2937
+ "valid": true,
2938
+ "error": null,
2939
+ "probs": {
2940
+ "central_engineering": 0.9135,
2941
+ "insufficient_log": 0.0073,
2942
+ "local_maintenance": 0.0773,
2943
+ "safety_shutdown": 0.002
2944
+ },
2945
+ "latency_s": 0.082
2946
+ },
2947
+ {
2948
+ "id": "hard-sol-a-multi_hop-09",
2949
+ "tier": "hard",
2950
+ "expected": "require_steward",
2951
+ "predicted": "require_steward",
2952
+ "correct": true,
2953
+ "valid": true,
2954
+ "error": null,
2955
+ "probs": {
2956
+ "confirm_l3": 0.0035,
2957
+ "reject_capacity": 0.0204,
2958
+ "require_steward": 0.9748,
2959
+ "switch_l4": 0.0014
2960
+ },
2961
+ "latency_s": 0.082
2962
+ },
2963
+ {
2964
+ "id": "hard-sol-a-multi_hop-10",
2965
+ "tier": "hard",
2966
+ "expected": "approve_45_days",
2967
+ "predicted": "approve_45_days",
2968
+ "correct": true,
2969
+ "valid": true,
2970
+ "error": null,
2971
+ "probs": {
2972
+ "approve_45_days": 0.9661,
2973
+ "deny_sensitive": 0.0106,
2974
+ "needs_field_review": 0.0069,
2975
+ "reduce_to_30_days": 0.0163
2976
+ },
2977
+ "latency_s": 0.083
2978
+ },
2979
+ {
2980
+ "id": "hard-sol-a-multi_hop-12",
2981
+ "tier": "hard",
2982
+ "expected": "needs_authentication",
2983
+ "predicted": "covered_recurrence",
2984
+ "correct": false,
2985
+ "valid": true,
2986
+ "error": null,
2987
+ "probs": {
2988
+ "covered_recurrence": 0.7149,
2989
+ "expired": 0.177,
2990
+ "needs_authentication": 0.0981,
2991
+ "wrong_issue_class": 0.01
2992
+ },
2993
+ "latency_s": 0.083
2994
+ },
2995
+ {
2996
+ "id": "hard-sol-a-trap-02",
2997
+ "tier": "hard",
2998
+ "expected": "no",
2999
+ "predicted": "no",
3000
+ "correct": true,
3001
+ "valid": true,
3002
+ "error": null,
3003
+ "probs": {
3004
+ "yes": 0.0018,
3005
+ "no": 0.9982
3006
+ },
3007
+ "latency_s": 0.071
3008
+ },
3009
+ {
3010
+ "id": "hard-sol-a-trap-04",
3011
+ "tier": "hard",
3012
+ "expected": "no",
3013
+ "predicted": "no",
3014
+ "correct": true,
3015
+ "valid": true,
3016
+ "error": null,
3017
+ "probs": {
3018
+ "yes": 0.0133,
3019
+ "no": 0.9867
3020
+ },
3021
+ "latency_s": 0.071
3022
+ },
3023
+ {
3024
+ "id": "hard-sol-a-trap-05",
3025
+ "tier": "hard",
3026
+ "expected": "0",
3027
+ "predicted": "0",
3028
+ "correct": true,
3029
+ "valid": true,
3030
+ "error": null,
3031
+ "probs": {
3032
+ "0": 0.9908,
3033
+ "1": 0.002,
3034
+ "2": 0.0067,
3035
+ "3": 0.0005
3036
+ },
3037
+ "latency_s": 0.075
3038
+ },
3039
+ {
3040
+ "id": "hard-sol-a-trap-06",
3041
+ "tier": "hard",
3042
+ "expected": "keep_open_unresolved",
3043
+ "predicted": "keep_open_unresolved",
3044
+ "correct": true,
3045
+ "valid": true,
3046
+ "error": null,
3047
+ "probs": {
3048
+ "await_internal_metrics": 0.0002,
3049
+ "close_agent_verified": 0.0002,
3050
+ "close_confirmed": 0.0005,
3051
+ "keep_open_unresolved": 0.9992
3052
+ },
3053
+ "latency_s": 0.074
3054
+ },
3055
+ {
3056
+ "id": "hard-sol-a-trap-08",
3057
+ "tier": "hard",
3058
+ "expected": "late",
3059
+ "predicted": "late",
3060
+ "correct": true,
3061
+ "valid": true,
3062
+ "error": null,
3063
+ "probs": {
3064
+ "canceled": 0.0,
3065
+ "indeterminate": 0.0001,
3066
+ "late": 0.9998,
3067
+ "on_time": 0.0001
3068
+ },
3069
+ "latency_s": 0.07
3070
+ },
3071
+ {
3072
+ "id": "hard-sol-a-trap-10",
3073
+ "tier": "hard",
3074
+ "expected": "blocked_second_review",
3075
+ "predicted": "blocked_second_review",
3076
+ "correct": true,
3077
+ "valid": true,
3078
+ "error": null,
3079
+ "probs": {
3080
+ "blocked_failed_test": 0.0007,
3081
+ "blocked_second_review": 0.994,
3082
+ "ready_one_review": 0.0016,
3083
+ "ready_two_reviews": 0.0037
3084
+ },
3085
+ "latency_s": 0.071
3086
+ },
3087
+ {
3088
+ "id": "hard-sol-a-trap-13",
3089
+ "tier": "hard",
3090
+ "expected": "no",
3091
+ "predicted": "no",
3092
+ "correct": true,
3093
+ "valid": true,
3094
+ "error": null,
3095
+ "probs": {
3096
+ "yes": 0.0053,
3097
+ "no": 0.9947
3098
+ },
3099
+ "latency_s": 0.07
3100
+ },
3101
+ {
3102
+ "id": "hard-sol-a-trap-15",
3103
+ "tier": "hard",
3104
+ "expected": "no",
3105
+ "predicted": "no",
3106
+ "correct": true,
3107
+ "valid": true,
3108
+ "error": null,
3109
+ "probs": {
3110
+ "yes": 0.1823,
3111
+ "no": 0.8177
3112
+ },
3113
+ "latency_s": 0.07
3114
+ },
3115
+ {
3116
+ "id": "hard-sol-b-judge_hard-01",
3117
+ "tier": "hard",
3118
+ "expected": "yes",
3119
+ "predicted": "yes",
3120
+ "correct": true,
3121
+ "valid": true,
3122
+ "error": null,
3123
+ "probs": {
3124
+ "yes": 0.7079,
3125
+ "no": 0.2921
3126
+ },
3127
+ "latency_s": 0.069
3128
+ },
3129
+ {
3130
+ "id": "hard-sol-b-judge_hard-02",
3131
+ "tier": "hard",
3132
+ "expected": "no",
3133
+ "predicted": "yes",
3134
+ "correct": false,
3135
+ "valid": true,
3136
+ "error": null,
3137
+ "probs": {
3138
+ "yes": 0.8925,
3139
+ "no": 0.1075
3140
+ },
3141
+ "latency_s": 0.069
3142
+ },
3143
+ {
3144
+ "id": "hard-sol-b-judge_hard-05",
3145
+ "tier": "hard",
3146
+ "expected": "yes",
3147
+ "predicted": "yes",
3148
+ "correct": true,
3149
+ "valid": true,
3150
+ "error": null,
3151
+ "probs": {
3152
+ "yes": 0.7387,
3153
+ "no": 0.2613
3154
+ },
3155
+ "latency_s": 0.07
3156
+ },
3157
+ {
3158
+ "id": "hard-sol-b-judge_hard-08",
3159
+ "tier": "hard",
3160
+ "expected": "no",
3161
+ "predicted": "yes",
3162
+ "correct": false,
3163
+ "valid": true,
3164
+ "error": null,
3165
+ "probs": {
3166
+ "yes": 0.9452,
3167
+ "no": 0.0548
3168
+ },
3169
+ "latency_s": 0.07
3170
+ },
3171
+ {
3172
+ "id": "hard-sol-b-judge_hard-10",
3173
+ "tier": "hard",
3174
+ "expected": "no",
3175
+ "predicted": "no",
3176
+ "correct": true,
3177
+ "valid": true,
3178
+ "error": null,
3179
+ "probs": {
3180
+ "yes": 0.3249,
3181
+ "no": 0.6751
3182
+ },
3183
+ "latency_s": 0.069
3184
+ },
3185
+ {
3186
+ "id": "hard-sol-b-judge_hard-14",
3187
+ "tier": "hard",
3188
+ "expected": "no",
3189
+ "predicted": "no",
3190
+ "correct": true,
3191
+ "valid": true,
3192
+ "error": null,
3193
+ "probs": {
3194
+ "yes": 0.0138,
3195
+ "no": 0.9862
3196
+ },
3197
+ "latency_s": 0.069
3198
+ },
3199
+ {
3200
+ "id": "hard-sol-b-judge_hard-15",
3201
+ "tier": "hard",
3202
+ "expected": "yes",
3203
+ "predicted": "yes",
3204
+ "correct": true,
3205
+ "valid": true,
3206
+ "error": null,
3207
+ "probs": {
3208
+ "yes": 0.8849,
3209
+ "no": 0.1151
3210
+ },
3211
+ "latency_s": 0.07
3212
+ },
3213
+ {
3214
+ "id": "hard-sol-b-judge_hard-18",
3215
+ "tier": "hard",
3216
+ "expected": "no",
3217
+ "predicted": "yes",
3218
+ "correct": false,
3219
+ "valid": true,
3220
+ "error": null,
3221
+ "probs": {
3222
+ "yes": 0.9799,
3223
+ "no": 0.0201
3224
+ },
3225
+ "latency_s": 0.07
3226
+ },
3227
+ {
3228
+ "id": "hard-sol-b-long_policy-01",
3229
+ "tier": "hard",
3230
+ "expected": "reject_entire_claim",
3231
+ "predicted": "reject_entire_claim",
3232
+ "correct": true,
3233
+ "valid": true,
3234
+ "error": null,
3235
+ "probs": {
3236
+ "approve_all": 0.004,
3237
+ "approve_lodging_only": 0.0248,
3238
+ "approve_meals_only": 0.0145,
3239
+ "reject_entire_claim": 0.9566
3240
+ },
3241
+ "latency_s": 0.091
3242
+ },
3243
+ {
3244
+ "id": "hard-sol-b-long_policy-02",
3245
+ "tier": "hard",
3246
+ "expected": "deny_current_export",
3247
+ "predicted": "privacy_only",
3248
+ "correct": false,
3249
+ "valid": true,
3250
+ "error": null,
3251
+ "probs": {
3252
+ "analytics_owner_only": 0.006,
3253
+ "deny_current_export": 0.4371,
3254
+ "privacy_and_security": 0.0434,
3255
+ "privacy_only": 0.5135
3256
+ },
3257
+ "latency_s": 0.088
3258
+ },
3259
+ {
3260
+ "id": "hard-sol-b-long_policy-05",
3261
+ "tier": "hard",
3262
+ "expected": "delete_profile_omar_april_and_table",
3263
+ "predicted": "delete_nothing",
3264
+ "correct": false,
3265
+ "valid": true,
3266
+ "error": null,
3267
+ "probs": {
3268
+ "delete_everything": 0.0006,
3269
+ "delete_nothing": 0.9697,
3270
+ "delete_only_profile": 0.028,
3271
+ "delete_profile_omar_april_and_table": 0.0016
3272
+ },
3273
+ "latency_s": 0.086
3274
+ },
3275
+ {
3276
+ "id": "hard-sol-b-long_policy-06",
3277
+ "tier": "hard",
3278
+ "expected": "east_ward",
3279
+ "predicted": "river_arts",
3280
+ "correct": false,
3281
+ "valid": true,
3282
+ "error": null,
3283
+ "probs": {
3284
+ "east_ward": 0.0067,
3285
+ "harbor_neighbors": 0.0012,
3286
+ "no_award": 0.0024,
3287
+ "river_arts": 0.9897
3288
+ },
3289
+ "latency_s": 0.088
3290
+ },
3291
+ {
3292
+ "id": "hard-sol-b-routing_hard-01",
3293
+ "tier": "hard",
3294
+ "expected": "office_file_repair",
3295
+ "predicted": "office_file_repair",
3296
+ "correct": true,
3297
+ "valid": true,
3298
+ "error": null,
3299
+ "probs": {
3300
+ "data_pipeline": 0.0,
3301
+ "desktop_support": 0.0001,
3302
+ "finance_review": 0.0001,
3303
+ "office_file_repair": 0.9995,
3304
+ "spreadsheet_formula": 0.0003
3305
+ },
3306
+ "latency_s": 0.071
3307
+ },
3308
+ {
3309
+ "id": "hard-sol-b-routing_hard-02",
3310
+ "tier": "hard",
3311
+ "expected": "ledger_reconciliation",
3312
+ "predicted": "ledger_reconciliation",
3313
+ "correct": true,
3314
+ "valid": true,
3315
+ "error": null,
3316
+ "probs": {
3317
+ "customer_support": 0.0001,
3318
+ "fraud_review": 0.0001,
3319
+ "ledger_reconciliation": 0.9996,
3320
+ "mobile_app": 0.0001,
3321
+ "payment_disputes": 0.0002
3322
+ },
3323
+ "latency_s": 0.071
3324
+ },
3325
+ {
3326
+ "id": "hard-sol-b-routing_hard-03",
3327
+ "tier": "hard",
3328
+ "expected": "network_dns",
3329
+ "predicted": "network_dns",
3330
+ "correct": true,
3331
+ "valid": true,
3332
+ "error": null,
3333
+ "probs": {
3334
+ "api_platform": 0.0003,
3335
+ "authoritative_dns": 0.0006,
3336
+ "endpoint_devices": 0.0032,
3337
+ "network_dns": 0.9956,
3338
+ "tls_certificates": 0.0003
3339
+ },
3340
+ "latency_s": 0.075
3341
+ },
3342
+ {
3343
+ "id": "hard-sol-b-routing_hard-07",
3344
+ "tier": "hard",
3345
+ "expected": "release_notarization",
3346
+ "predicted": "release_notarization",
3347
+ "correct": true,
3348
+ "valid": true,
3349
+ "error": null,
3350
+ "probs": {
3351
+ "ci_infrastructure": 0.0001,
3352
+ "code_signing": 0.0003,
3353
+ "endpoint_security": 0.0002,
3354
+ "macos_runtime": 0.0002,
3355
+ "release_notarization": 0.9993
3356
+ },
3357
+ "latency_s": 0.071
3358
+ },
3359
+ {
3360
+ "id": "hard-sol-b-routing_hard-09",
3361
+ "tier": "hard",
3362
+ "expected": "source_access",
3363
+ "predicted": "source_access",
3364
+ "correct": true,
3365
+ "valid": true,
3366
+ "error": null,
3367
+ "probs": {
3368
+ "git_client": 0.0001,
3369
+ "identity_login": 0.0001,
3370
+ "repository_admin": 0.0001,
3371
+ "source_access": 0.9996,
3372
+ "vpn_network": 0.0002
3373
+ },
3374
+ "latency_s": 0.071
3375
+ },
3376
+ {
3377
+ "id": "hard-sol-b-temporal_numeric-01",
3378
+ "tier": "hard",
3379
+ "expected": "late_by_under_2h",
3380
+ "predicted": "within_window",
3381
+ "correct": false,
3382
+ "valid": true,
3383
+ "error": null,
3384
+ "probs": {
3385
+ "before_open": 0.0166,
3386
+ "late_by_2h_or_more": 0.0877,
3387
+ "late_by_under_2h": 0.2566,
3388
+ "within_window": 0.6392
3389
+ },
3390
+ "latency_s": 0.071
3391
+ },
3392
+ {
3393
+ "id": "hard-sol-b-temporal_numeric-02",
3394
+ "tier": "hard",
3395
+ "expected": "over_limit",
3396
+ "predicted": "under_100gb_below",
3397
+ "correct": false,
3398
+ "valid": true,
3399
+ "error": null,
3400
+ "probs": {
3401
+ "at_least_100gb_below": 0.2542,
3402
+ "exactly_equal": 0.041,
3403
+ "over_limit": 0.0368,
3404
+ "under_100gb_below": 0.6681
3405
+ },
3406
+ "latency_s": 0.071
3407
+ },
3408
+ {
3409
+ "id": "hard-sol-b-temporal_numeric-03",
3410
+ "tier": "hard",
3411
+ "expected": "9_credits",
3412
+ "predicted": "9_credits",
3413
+ "correct": true,
3414
+ "valid": true,
3415
+ "error": null,
3416
+ "probs": {
3417
+ "12_credits": 0.1823,
3418
+ "15_credits": 0.059,
3419
+ "6_credits": 0.1321,
3420
+ "9_credits": 0.6267
3421
+ },
3422
+ "latency_s": 0.07
3423
+ },
3424
+ {
3425
+ "id": "hard-sol-b-temporal_numeric-04",
3426
+ "tier": "hard",
3427
+ "expected": "pass",
3428
+ "predicted": "narrow_fail",
3429
+ "correct": false,
3430
+ "valid": true,
3431
+ "error": null,
3432
+ "probs": {
3433
+ "clear_fail": 0.1427,
3434
+ "narrow_fail": 0.7539,
3435
+ "pass": 0.1034
3436
+ },
3437
+ "latency_s": 0.07
3438
+ },
3439
+ {
3440
+ "id": "hard-sol-c-judge_hard-03",
3441
+ "tier": "hard",
3442
+ "expected": "no",
3443
+ "predicted": "no",
3444
+ "correct": true,
3445
+ "valid": true,
3446
+ "error": null,
3447
+ "probs": {
3448
+ "yes": 0.4616,
3449
+ "no": 0.5384
3450
+ },
3451
+ "latency_s": 0.071
3452
+ },
3453
+ {
3454
+ "id": "hard-sol-c-judge_hard-05",
3455
+ "tier": "hard",
3456
+ "expected": "yes",
3457
+ "predicted": "yes",
3458
+ "correct": true,
3459
+ "valid": true,
3460
+ "error": null,
3461
+ "probs": {
3462
+ "yes": 0.5764,
3463
+ "no": 0.4236
3464
+ },
3465
+ "latency_s": 0.071
3466
+ },
3467
+ {
3468
+ "id": "hard-sol-c-judge_hard-07",
3469
+ "tier": "hard",
3470
+ "expected": "no",
3471
+ "predicted": "no",
3472
+ "correct": true,
3473
+ "valid": true,
3474
+ "error": null,
3475
+ "probs": {
3476
+ "yes": 0.2688,
3477
+ "no": 0.7312
3478
+ },
3479
+ "latency_s": 0.07
3480
+ },
3481
+ {
3482
+ "id": "hard-sol-c-judge_hard-08",
3483
+ "tier": "hard",
3484
+ "expected": "no",
3485
+ "predicted": "yes",
3486
+ "correct": false,
3487
+ "valid": true,
3488
+ "error": null,
3489
+ "probs": {
3490
+ "yes": 0.9389,
3491
+ "no": 0.0611
3492
+ },
3493
+ "latency_s": 0.07
3494
+ },
3495
+ {
3496
+ "id": "hard-sol-c-judge_hard-09",
3497
+ "tier": "hard",
3498
+ "expected": "yes",
3499
+ "predicted": "yes",
3500
+ "correct": true,
3501
+ "valid": true,
3502
+ "error": null,
3503
+ "probs": {
3504
+ "yes": 0.9127,
3505
+ "no": 0.0873
3506
+ },
3507
+ "latency_s": 0.07
3508
+ },
3509
+ {
3510
+ "id": "hard-sol-c-judge_hard-10",
3511
+ "tier": "hard",
3512
+ "expected": "yes",
3513
+ "predicted": "yes",
3514
+ "correct": true,
3515
+ "valid": true,
3516
+ "error": null,
3517
+ "probs": {
3518
+ "yes": 0.7603,
3519
+ "no": 0.2397
3520
+ },
3521
+ "latency_s": 0.071
3522
+ },
3523
+ {
3524
+ "id": "hard-sol-c-judge_hard-11",
3525
+ "tier": "hard",
3526
+ "expected": "no",
3527
+ "predicted": "no",
3528
+ "correct": true,
3529
+ "valid": true,
3530
+ "error": null,
3531
+ "probs": {
3532
+ "yes": 0.0194,
3533
+ "no": 0.9806
3534
+ },
3535
+ "latency_s": 0.07
3536
+ },
3537
+ {
3538
+ "id": "hard-sol-c-judge_hard-13",
3539
+ "tier": "hard",
3540
+ "expected": "no",
3541
+ "predicted": "yes",
3542
+ "correct": false,
3543
+ "valid": true,
3544
+ "error": null,
3545
+ "probs": {
3546
+ "yes": 0.5192,
3547
+ "no": 0.4808
3548
+ },
3549
+ "latency_s": 0.07
3550
+ },
3551
+ {
3552
+ "id": "hard-sol-c-judge_hard-15",
3553
+ "tier": "hard",
3554
+ "expected": "yes",
3555
+ "predicted": "yes",
3556
+ "correct": true,
3557
+ "valid": true,
3558
+ "error": null,
3559
+ "probs": {
3560
+ "yes": 0.9606,
3561
+ "no": 0.0394
3562
+ },
3563
+ "latency_s": 0.07
3564
+ },
3565
+ {
3566
+ "id": "hard-sol-c-multi_hop-07",
3567
+ "tier": "hard",
3568
+ "expected": "programme_review",
3569
+ "predicted": "release_draw",
3570
+ "correct": false,
3571
+ "valid": true,
3572
+ "error": null,
3573
+ "probs": {
3574
+ "partial_release": 0.0027,
3575
+ "programme_review": 0.0108,
3576
+ "reject_draw": 0.002,
3577
+ "release_draw": 0.9844
3578
+ },
3579
+ "latency_s": 0.084
3580
+ },
3581
+ {
3582
+ "id": "hard-sol-c-multi_hop-09",
3583
+ "tier": "hard",
3584
+ "expected": "bus_substitution",
3585
+ "predicted": "pass_fen_cross",
3586
+ "correct": false,
3587
+ "valid": true,
3588
+ "error": null,
3589
+ "probs": {
3590
+ "bus_substitution": 0.0289,
3591
+ "call_fen_cross": 0.0444,
3592
+ "pass_fen_cross": 0.8508,
3593
+ "terminate_oakmere": 0.0759
3594
+ },
3595
+ "latency_s": 0.084
3596
+ },
3597
+ {
3598
+ "id": "hard-sol-c-multi_hop-11",
3599
+ "tier": "hard",
3600
+ "expected": "coverage_excluded",
3601
+ "predicted": "coverage_excluded",
3602
+ "correct": true,
3603
+ "valid": true,
3604
+ "error": null,
3605
+ "probs": {
3606
+ "adjuster_referral": 0.0013,
3607
+ "coverage_confirmed": 0.1088,
3608
+ "coverage_excluded": 0.8832,
3609
+ "documentation_needed": 0.0067
3610
+ },
3611
+ "latency_s": 0.087
3612
+ },
3613
+ {
3614
+ "id": "hard-sol-c-multi_hop-12",
3615
+ "tier": "hard",
3616
+ "expected": "conservation_hold",
3617
+ "predicted": "conservation_hold",
3618
+ "correct": true,
3619
+ "valid": true,
3620
+ "error": null,
3621
+ "probs": {
3622
+ "conservation_hold": 0.9577,
3623
+ "deny_request": 0.0138,
3624
+ "digital_copy_only": 0.0223,
3625
+ "supervised_delivery": 0.0062
3626
+ },
3627
+ "latency_s": 0.073
3628
+ },
3629
+ {
3630
+ "id": "hard-sol-c-multi_hop-13",
3631
+ "tier": "hard",
3632
+ "expected": "normal_release",
3633
+ "predicted": "normal_release",
3634
+ "correct": true,
3635
+ "valid": true,
3636
+ "error": null,
3637
+ "probs": {
3638
+ "engineering_review": 0.0,
3639
+ "normal_release": 0.9999,
3640
+ "rework": 0.0001,
3641
+ "scrap_lot": 0.0001
3642
+ },
3643
+ "latency_s": 0.086
3644
+ }
3645
+ ]
3646
+ }
eval/jevbench_public231_this_model.json CHANGED
@@ -5,19 +5,19 @@
5
  "n": 48,
6
  "accuracy": 1.0,
7
  "invalid": 0,
8
- "p50_s": 0.067
9
  },
10
  "standard": {
11
  "n": 72,
12
- "accuracy": 0.8889,
13
  "invalid": 0,
14
- "p50_s": 0.067
15
  },
16
  "hard": {
17
  "n": 111,
18
- "accuracy": 0.6667,
19
  "invalid": 0,
20
- "p50_s": 0.07
21
  }
22
  },
23
  "rows": [
@@ -34,9 +34,9 @@
34
  "cancel_order": 0.0,
35
  "change_address": 0.0,
36
  "report_damage": 0.0,
37
- "track_order": 0.9999
38
  },
39
- "latency_s": 1.952
40
  },
41
  {
42
  "id": "easy-intent-01",
@@ -53,7 +53,7 @@
53
  "report_damage": 0.0,
54
  "track_order": 0.0
55
  },
56
- "latency_s": 0.069
57
  },
58
  {
59
  "id": "easy-intent-02",
@@ -65,12 +65,12 @@
65
  "error": null,
66
  "probs": {
67
  "billing_question": 0.0,
68
- "cancel_order": 0.0001,
69
  "change_address": 0.9998,
70
- "report_damage": 0.0001,
71
  "track_order": 0.0
72
  },
73
- "latency_s": 0.066
74
  },
75
  {
76
  "id": "easy-intent-03",
@@ -82,12 +82,12 @@
82
  "error": null,
83
  "probs": {
84
  "billing_question": 0.0,
85
- "cancel_order": 0.0004,
86
  "change_address": 0.0001,
87
- "report_damage": 0.9995,
88
  "track_order": 0.0001
89
  },
90
- "latency_s": 0.066
91
  },
92
  {
93
  "id": "easy-intent-04",
@@ -98,13 +98,13 @@
98
  "valid": true,
99
  "error": null,
100
  "probs": {
101
- "billing_question": 0.998,
102
- "cancel_order": 0.001,
103
- "change_address": 0.0004,
104
- "report_damage": 0.0003,
105
  "track_order": 0.0002
106
  },
107
- "latency_s": 0.067
108
  },
109
  {
110
  "id": "easy-intent-05",
@@ -118,10 +118,10 @@
118
  "billing_question": 0.0,
119
  "cancel_order": 0.0001,
120
  "change_address": 0.0001,
121
- "report_damage": 0.0001,
122
- "track_order": 0.9997
123
  },
124
- "latency_s": 0.066
125
  },
126
  {
127
  "id": "easy-intent-06",
@@ -134,11 +134,11 @@
134
  "probs": {
135
  "play_music": 0.0,
136
  "send_message": 0.0001,
137
- "set_alarm": 0.9997,
138
  "turn_off_lights": 0.0001,
139
  "weather": 0.0001
140
  },
141
- "latency_s": 0.066
142
  },
143
  {
144
  "id": "easy-intent-07",
@@ -149,13 +149,13 @@
149
  "valid": true,
150
  "error": null,
151
  "probs": {
152
- "play_music": 0.9994,
153
- "send_message": 0.0002,
154
- "set_alarm": 0.0002,
155
  "turn_off_lights": 0.0001,
156
  "weather": 0.0001
157
  },
158
- "latency_s": 0.066
159
  },
160
  {
161
  "id": "easy-intent-08",
@@ -172,7 +172,7 @@
172
  "turn_off_lights": 0.0,
173
  "weather": 1.0
174
  },
175
- "latency_s": 0.066
176
  },
177
  {
178
  "id": "easy-intent-09",
@@ -185,11 +185,11 @@
185
  "probs": {
186
  "play_music": 0.0001,
187
  "send_message": 0.9993,
188
- "set_alarm": 0.0002,
189
- "turn_off_lights": 0.0002,
190
- "weather": 0.0002
191
  },
192
- "latency_s": 0.066
193
  },
194
  {
195
  "id": "easy-intent-10",
@@ -204,9 +204,9 @@
204
  "send_message": 0.0001,
205
  "set_alarm": 0.0002,
206
  "turn_off_lights": 0.9994,
207
- "weather": 0.0001
208
  },
209
- "latency_s": 0.066
210
  },
211
  {
212
  "id": "easy-intent-11",
@@ -218,12 +218,12 @@
218
  "error": null,
219
  "probs": {
220
  "play_music": 0.0,
221
- "send_message": 0.0001,
222
- "set_alarm": 0.9998,
223
- "turn_off_lights": 0.0001,
224
- "weather": 0.0001
225
  },
226
- "latency_s": 0.07
227
  },
228
  {
229
  "id": "easy-fact-00",
@@ -234,10 +234,10 @@
234
  "valid": true,
235
  "error": null,
236
  "probs": {
237
- "yes": 0.984,
238
- "no": 0.016
239
  },
240
- "latency_s": 0.067
241
  },
242
  {
243
  "id": "easy-fact-01",
@@ -248,10 +248,10 @@
248
  "valid": true,
249
  "error": null,
250
  "probs": {
251
- "yes": 0.0057,
252
- "no": 0.9943
253
  },
254
- "latency_s": 0.067
255
  },
256
  {
257
  "id": "easy-fact-02",
@@ -262,10 +262,10 @@
262
  "valid": true,
263
  "error": null,
264
  "probs": {
265
- "yes": 0.9834,
266
- "no": 0.0166
267
  },
268
- "latency_s": 0.067
269
  },
270
  {
271
  "id": "easy-fact-03",
@@ -276,10 +276,10 @@
276
  "valid": true,
277
  "error": null,
278
  "probs": {
279
- "yes": 0.0026,
280
- "no": 0.9974
281
  },
282
- "latency_s": 0.067
283
  },
284
  {
285
  "id": "easy-fact-04",
@@ -290,10 +290,10 @@
290
  "valid": true,
291
  "error": null,
292
  "probs": {
293
- "yes": 0.9783,
294
- "no": 0.0217
295
  },
296
- "latency_s": 0.078
297
  },
298
  {
299
  "id": "easy-fact-05",
@@ -304,10 +304,10 @@
304
  "valid": true,
305
  "error": null,
306
  "probs": {
307
- "yes": 0.011,
308
- "no": 0.989
309
  },
310
- "latency_s": 0.066
311
  },
312
  {
313
  "id": "easy-fact-06",
@@ -318,10 +318,10 @@
318
  "valid": true,
319
  "error": null,
320
  "probs": {
321
- "yes": 0.9591,
322
- "no": 0.0409
323
  },
324
- "latency_s": 0.066
325
  },
326
  {
327
  "id": "easy-fact-07",
@@ -332,10 +332,10 @@
332
  "valid": true,
333
  "error": null,
334
  "probs": {
335
- "yes": 0.0087,
336
- "no": 0.9913
337
  },
338
- "latency_s": 0.066
339
  },
340
  {
341
  "id": "easy-fact-08",
@@ -346,10 +346,10 @@
346
  "valid": true,
347
  "error": null,
348
  "probs": {
349
- "yes": 0.9827,
350
- "no": 0.0173
351
  },
352
- "latency_s": 0.071
353
  },
354
  {
355
  "id": "easy-fact-09",
@@ -360,10 +360,10 @@
360
  "valid": true,
361
  "error": null,
362
  "probs": {
363
- "yes": 0.0069,
364
- "no": 0.9931
365
  },
366
- "latency_s": 0.066
367
  },
368
  {
369
  "id": "easy-fact-10",
@@ -374,10 +374,10 @@
374
  "valid": true,
375
  "error": null,
376
  "probs": {
377
- "yes": 0.9411,
378
- "no": 0.0589
379
  },
380
- "latency_s": 0.066
381
  },
382
  {
383
  "id": "easy-fact-11",
@@ -388,10 +388,10 @@
388
  "valid": true,
389
  "error": null,
390
  "probs": {
391
- "yes": 0.0154,
392
- "no": 0.9846
393
  },
394
- "latency_s": 0.066
395
  },
396
  {
397
  "id": "easy-extraction-00",
@@ -402,12 +402,12 @@
402
  "valid": true,
403
  "error": null,
404
  "probs": {
405
- "bank_transfer": 0.0,
406
  "cash": 0.0,
407
  "credit_card": 0.0001,
408
  "paypal": 0.9998
409
  },
410
- "latency_s": 0.066
411
  },
412
  {
413
  "id": "easy-extraction-01",
@@ -420,10 +420,10 @@
420
  "probs": {
421
  "bank_transfer": 0.0001,
422
  "cash": 0.9996,
423
- "credit_card": 0.0003,
424
  "paypal": 0.0001
425
  },
426
- "latency_s": 0.066
427
  },
428
  {
429
  "id": "easy-extraction-02",
@@ -434,12 +434,12 @@
434
  "valid": true,
435
  "error": null,
436
  "probs": {
437
- "bank_transfer": 0.9997,
438
- "cash": 0.0001,
439
  "credit_card": 0.0001,
440
- "paypal": 0.0001
441
  },
442
- "latency_s": 0.066
443
  },
444
  {
445
  "id": "easy-extraction-03",
@@ -450,12 +450,12 @@
450
  "valid": true,
451
  "error": null,
452
  "probs": {
453
- "high": 0.966,
454
- "low": 0.0016,
455
- "medium": 0.0029,
456
- "urgent": 0.0295
457
  },
458
- "latency_s": 0.066
459
  },
460
  {
461
  "id": "easy-extraction-04",
@@ -466,12 +466,12 @@
466
  "valid": true,
467
  "error": null,
468
  "probs": {
469
- "high": 0.001,
470
- "low": 0.9971,
471
- "medium": 0.0016,
472
- "urgent": 0.0003
473
  },
474
- "latency_s": 0.066
475
  },
476
  {
477
  "id": "easy-extraction-05",
@@ -482,12 +482,12 @@
482
  "valid": true,
483
  "error": null,
484
  "probs": {
485
- "high": 0.0011,
486
  "low": 0.0002,
487
  "medium": 0.0006,
488
- "urgent": 0.998
489
  },
490
- "latency_s": 0.067
491
  },
492
  {
493
  "id": "easy-extraction-06",
@@ -498,12 +498,12 @@
498
  "valid": true,
499
  "error": null,
500
  "probs": {
501
- "L": 0.0007,
502
- "M": 0.999,
503
  "S": 0.0002,
504
- "XL": 0.0001
505
  },
506
- "latency_s": 0.067
507
  },
508
  {
509
  "id": "easy-extraction-07",
@@ -514,12 +514,12 @@
514
  "valid": true,
515
  "error": null,
516
  "probs": {
517
- "L": 0.0004,
518
  "M": 0.0001,
519
  "S": 0.0001,
520
- "XL": 0.9994
521
  },
522
- "latency_s": 0.067
523
  },
524
  {
525
  "id": "easy-extraction-08",
@@ -530,13 +530,13 @@
530
  "valid": true,
531
  "error": null,
532
  "probs": {
533
- "friday": 0.0007,
534
- "monday": 0.0009,
535
- "thursday": 0.9976,
536
- "tuesday": 0.0004,
537
- "wednesday": 0.0004
538
  },
539
- "latency_s": 0.067
540
  },
541
  {
542
  "id": "easy-extraction-09",
@@ -547,13 +547,13 @@
547
  "valid": true,
548
  "error": null,
549
  "probs": {
550
- "friday": 0.0096,
551
- "monday": 0.9764,
552
- "thursday": 0.0059,
553
- "tuesday": 0.0043,
554
- "wednesday": 0.0037
555
  },
556
- "latency_s": 0.067
557
  },
558
  {
559
  "id": "easy-extraction-10",
@@ -564,13 +564,13 @@
564
  "valid": true,
565
  "error": null,
566
  "probs": {
567
- "friday": 0.9964,
568
- "monday": 0.0011,
569
- "thursday": 0.0009,
570
- "tuesday": 0.0007,
571
- "wednesday": 0.0008
572
  },
573
- "latency_s": 0.065
574
  },
575
  {
576
  "id": "easy-extraction-11",
@@ -583,10 +583,10 @@
583
  "probs": {
584
  "bank_transfer": 0.0001,
585
  "cash": 0.0001,
586
- "credit_card": 0.9997,
587
  "paypal": 0.0001
588
  },
589
- "latency_s": 0.067
590
  },
591
  {
592
  "id": "easy-tool_selection-00",
@@ -599,11 +599,11 @@
599
  "probs": {
600
  "convert_currency": 0.0,
601
  "create_calendar_event": 0.0,
602
- "get_weather": 0.9998,
603
- "send_email": 0.0001,
604
  "translate_text": 0.0
605
  },
606
- "latency_s": 0.068
607
  },
608
  {
609
  "id": "easy-tool_selection-01",
@@ -620,7 +620,7 @@
620
  "send_email": 0.9998,
621
  "translate_text": 0.0001
622
  },
623
- "latency_s": 0.068
624
  },
625
  {
626
  "id": "easy-tool_selection-02",
@@ -637,7 +637,7 @@
637
  "send_email": 0.0,
638
  "translate_text": 0.0
639
  },
640
- "latency_s": 0.067
641
  },
642
  {
643
  "id": "easy-tool_selection-03",
@@ -648,13 +648,13 @@
648
  "valid": true,
649
  "error": null,
650
  "probs": {
651
- "convert_currency": 0.9997,
652
- "create_calendar_event": 0.0001,
653
- "get_weather": 0.0001,
654
- "send_email": 0.0001,
655
  "translate_text": 0.0001
656
  },
657
- "latency_s": 0.067
658
  },
659
  {
660
  "id": "easy-tool_selection-04",
@@ -671,7 +671,7 @@
671
  "send_email": 0.0,
672
  "translate_text": 0.9999
673
  },
674
- "latency_s": 0.067
675
  },
676
  {
677
  "id": "easy-tool_selection-05",
@@ -684,11 +684,11 @@
684
  "probs": {
685
  "convert_currency": 0.0,
686
  "create_calendar_event": 0.9998,
687
- "get_weather": 0.0001,
688
  "send_email": 0.0,
689
- "translate_text": 0.0
690
  },
691
- "latency_s": 0.067
692
  },
693
  {
694
  "id": "easy-tool_selection-06",
@@ -705,7 +705,7 @@
705
  "search_flights": 0.9999,
706
  "set_reminder": 0.0001
707
  },
708
- "latency_s": 0.067
709
  },
710
  {
711
  "id": "easy-tool_selection-07",
@@ -716,13 +716,13 @@
716
  "valid": true,
717
  "error": null,
718
  "probs": {
719
- "book_restaurant": 0.9994,
720
- "call_taxi": 0.0001,
721
  "get_stock_price": 0.0001,
722
- "search_flights": 0.0001,
723
  "set_reminder": 0.0003
724
  },
725
- "latency_s": 0.067
726
  },
727
  {
728
  "id": "easy-tool_selection-08",
@@ -737,9 +737,9 @@
737
  "call_taxi": 0.0,
738
  "get_stock_price": 0.0001,
739
  "search_flights": 0.0,
740
- "set_reminder": 0.9998
741
  },
742
- "latency_s": 0.067
743
  },
744
  {
745
  "id": "easy-tool_selection-09",
@@ -751,12 +751,12 @@
751
  "error": null,
752
  "probs": {
753
  "book_restaurant": 0.0,
754
- "call_taxi": 0.0001,
755
- "get_stock_price": 0.9998,
756
  "search_flights": 0.0,
757
  "set_reminder": 0.0001
758
  },
759
- "latency_s": 0.067
760
  },
761
  {
762
  "id": "easy-tool_selection-10",
@@ -771,9 +771,9 @@
771
  "call_taxi": 0.9995,
772
  "get_stock_price": 0.0001,
773
  "search_flights": 0.0001,
774
- "set_reminder": 0.0002
775
  },
776
- "latency_s": 0.066
777
  },
778
  {
779
  "id": "easy-tool_selection-11",
@@ -790,7 +790,7 @@
790
  "search_flights": 0.0,
791
  "set_reminder": 0.9999
792
  },
793
- "latency_s": 0.066
794
  },
795
  {
796
  "id": "original-policy-01-0",
@@ -801,10 +801,10 @@
801
  "valid": true,
802
  "error": null,
803
  "probs": {
804
- "yes": 0.0173,
805
- "no": 0.9827
806
  },
807
- "latency_s": 0.066
808
  },
809
  {
810
  "id": "original-policy-01-1",
@@ -815,10 +815,10 @@
815
  "valid": true,
816
  "error": null,
817
  "probs": {
818
- "yes": 0.0271,
819
- "no": 0.9729
820
  },
821
- "latency_s": 0.067
822
  },
823
  {
824
  "id": "original-policy-02-0",
@@ -829,10 +829,10 @@
829
  "valid": true,
830
  "error": null,
831
  "probs": {
832
- "yes": 0.9411,
833
- "no": 0.0589
834
  },
835
- "latency_s": 0.067
836
  },
837
  {
838
  "id": "original-policy-02-1",
@@ -843,38 +843,38 @@
843
  "valid": true,
844
  "error": null,
845
  "probs": {
846
- "yes": 0.2327,
847
- "no": 0.7673
848
  },
849
- "latency_s": 0.066
850
  },
851
  {
852
  "id": "original-policy-03-0",
853
  "tier": "standard",
854
  "expected": "yes",
855
- "predicted": "yes",
856
- "correct": true,
857
  "valid": true,
858
  "error": null,
859
  "probs": {
860
- "yes": 0.5669,
861
- "no": 0.4331
862
  },
863
- "latency_s": 0.067
864
  },
865
  {
866
  "id": "original-policy-03-1",
867
  "tier": "standard",
868
  "expected": "yes",
869
- "predicted": "yes",
870
- "correct": true,
871
  "valid": true,
872
  "error": null,
873
  "probs": {
874
- "yes": 0.6043,
875
- "no": 0.3957
876
  },
877
- "latency_s": 0.066
878
  },
879
  {
880
  "id": "original-policy-04-0",
@@ -885,10 +885,10 @@
885
  "valid": true,
886
  "error": null,
887
  "probs": {
888
- "yes": 0.0758,
889
- "no": 0.9242
890
  },
891
- "latency_s": 0.066
892
  },
893
  {
894
  "id": "original-policy-04-1",
@@ -899,10 +899,10 @@
899
  "valid": true,
900
  "error": null,
901
  "probs": {
902
- "yes": 0.0133,
903
- "no": 0.9867
904
  },
905
- "latency_s": 0.067
906
  },
907
  {
908
  "id": "original-policy-05-0",
@@ -913,10 +913,10 @@
913
  "valid": true,
914
  "error": null,
915
  "probs": {
916
- "yes": 0.1151,
917
- "no": 0.8849
918
  },
919
- "latency_s": 0.066
920
  },
921
  {
922
  "id": "original-policy-05-1",
@@ -927,10 +927,10 @@
927
  "valid": true,
928
  "error": null,
929
  "probs": {
930
- "yes": 0.0758,
931
- "no": 0.9242
932
  },
933
- "latency_s": 0.066
934
  },
935
  {
936
  "id": "original-policy-06-0",
@@ -941,10 +941,10 @@
941
  "valid": true,
942
  "error": null,
943
  "probs": {
944
- "yes": 0.9606,
945
- "no": 0.0394
946
  },
947
- "latency_s": 0.066
948
  },
949
  {
950
  "id": "original-policy-06-1",
@@ -955,10 +955,10 @@
955
  "valid": true,
956
  "error": null,
957
  "probs": {
958
- "yes": 0.8177,
959
- "no": 0.1823
960
  },
961
- "latency_s": 0.066
962
  },
963
  {
964
  "id": "original-intent-01-0",
@@ -969,13 +969,13 @@
969
  "valid": true,
970
  "error": null,
971
  "probs": {
972
- "cancel": 0.0008,
973
- "change_address": 0.0003,
974
- "other": 0.0039,
975
- "refund": 0.9948,
976
- "status": 0.0002
977
  },
978
- "latency_s": 0.066
979
  },
980
  {
981
  "id": "original-intent-01-1",
@@ -986,13 +986,13 @@
986
  "valid": true,
987
  "error": null,
988
  "probs": {
989
- "cancel": 0.0357,
990
  "change_address": 0.0012,
991
- "other": 0.0178,
992
- "refund": 0.9446,
993
- "status": 0.0007
994
  },
995
- "latency_s": 0.066
996
  },
997
  {
998
  "id": "original-intent-02-0",
@@ -1003,13 +1003,13 @@
1003
  "valid": true,
1004
  "error": null,
1005
  "probs": {
1006
- "cancel": 0.0011,
1007
- "change_address": 0.0039,
1008
- "other": 0.4431,
1009
- "refund": 0.0027,
1010
- "status": 0.5492
1011
  },
1012
- "latency_s": 0.066
1013
  },
1014
  {
1015
  "id": "original-intent-02-1",
@@ -1022,11 +1022,11 @@
1022
  "probs": {
1023
  "cancel": 0.0001,
1024
  "change_address": 0.0001,
1025
- "other": 0.0017,
1026
  "refund": 0.0001,
1027
- "status": 0.9981
1028
  },
1029
- "latency_s": 0.066
1030
  },
1031
  {
1032
  "id": "original-intent-03-0",
@@ -1037,13 +1037,13 @@
1037
  "valid": true,
1038
  "error": null,
1039
  "probs": {
1040
- "cancel": 0.0006,
1041
- "change_address": 0.9693,
1042
- "other": 0.028,
1043
- "refund": 0.0015,
1044
- "status": 0.0006
1045
  },
1046
- "latency_s": 0.066
1047
  },
1048
  {
1049
  "id": "original-intent-03-1",
@@ -1055,12 +1055,12 @@
1055
  "error": null,
1056
  "probs": {
1057
  "cancel": 0.0,
1058
- "change_address": 0.9992,
1059
- "other": 0.0006,
1060
  "refund": 0.0001,
1061
- "status": 0.0
1062
  },
1063
- "latency_s": 0.066
1064
  },
1065
  {
1066
  "id": "original-intent-04-0",
@@ -1071,13 +1071,13 @@
1071
  "valid": true,
1072
  "error": null,
1073
  "probs": {
1074
- "cancel": 0.9957,
1075
- "change_address": 0.0004,
1076
- "other": 0.0032,
1077
- "refund": 0.0006,
1078
- "status": 0.0001
1079
  },
1080
- "latency_s": 0.066
1081
  },
1082
  {
1083
  "id": "original-intent-04-1",
@@ -1088,13 +1088,13 @@
1088
  "valid": true,
1089
  "error": null,
1090
  "probs": {
1091
- "cancel": 0.9957,
1092
- "change_address": 0.0004,
1093
- "other": 0.0034,
1094
- "refund": 0.0004,
1095
  "status": 0.0002
1096
  },
1097
- "latency_s": 0.066
1098
  },
1099
  {
1100
  "id": "original-intent-05-0",
@@ -1105,13 +1105,13 @@
1105
  "valid": true,
1106
  "error": null,
1107
  "probs": {
1108
- "cancel": 0.0015,
1109
- "change_address": 0.0027,
1110
- "other": 0.8433,
1111
- "refund": 0.1513,
1112
- "status": 0.0013
1113
  },
1114
- "latency_s": 0.066
1115
  },
1116
  {
1117
  "id": "original-intent-05-1",
@@ -1122,13 +1122,13 @@
1122
  "valid": true,
1123
  "error": null,
1124
  "probs": {
1125
- "cancel": 0.0027,
1126
- "change_address": 0.003,
1127
- "other": 0.7087,
1128
- "refund": 0.2845,
1129
- "status": 0.0013
1130
  },
1131
- "latency_s": 0.066
1132
  },
1133
  {
1134
  "id": "original-intent-06-0",
@@ -1141,11 +1141,11 @@
1141
  "probs": {
1142
  "cancel": 0.0001,
1143
  "change_address": 0.0001,
1144
- "other": 0.003,
1145
  "refund": 0.0001,
1146
- "status": 0.9966
1147
  },
1148
- "latency_s": 0.066
1149
  },
1150
  {
1151
  "id": "original-intent-06-1",
@@ -1156,13 +1156,13 @@
1156
  "valid": true,
1157
  "error": null,
1158
  "probs": {
1159
- "cancel": 0.0001,
1160
  "change_address": 0.0,
1161
- "other": 0.0014,
1162
- "refund": 0.0001,
1163
- "status": 0.9985
1164
  },
1165
- "latency_s": 0.066
1166
  },
1167
  {
1168
  "id": "original-ordinal-01-0",
@@ -1173,12 +1173,12 @@
1173
  "valid": true,
1174
  "error": null,
1175
  "probs": {
1176
- "0": 0.8812,
1177
- "1": 0.1156,
1178
- "2": 0.0021,
1179
- "3": 0.0011
1180
  },
1181
- "latency_s": 0.072
1182
  },
1183
  {
1184
  "id": "original-ordinal-01-1",
@@ -1189,12 +1189,12 @@
1189
  "valid": true,
1190
  "error": null,
1191
  "probs": {
1192
- "0": 0.807,
1193
- "1": 0.1897,
1194
- "2": 0.0024,
1195
- "3": 0.0009
1196
  },
1197
- "latency_s": 0.067
1198
  },
1199
  {
1200
  "id": "original-ordinal-02-0",
@@ -1205,12 +1205,12 @@
1205
  "valid": true,
1206
  "error": null,
1207
  "probs": {
1208
- "0": 0.0171,
1209
- "1": 0.9198,
1210
- "2": 0.0599,
1211
- "3": 0.0032
1212
  },
1213
- "latency_s": 0.067
1214
  },
1215
  {
1216
  "id": "original-ordinal-02-1",
@@ -1221,12 +1221,12 @@
1221
  "valid": true,
1222
  "error": null,
1223
  "probs": {
1224
- "0": 0.0131,
1225
- "1": 0.9794,
1226
- "2": 0.0051,
1227
- "3": 0.0024
1228
  },
1229
- "latency_s": 0.067
1230
  },
1231
  {
1232
  "id": "original-ordinal-03-0",
@@ -1237,12 +1237,12 @@
1237
  "valid": true,
1238
  "error": null,
1239
  "probs": {
1240
- "0": 0.0241,
1241
- "1": 0.0734,
1242
- "2": 0.9013,
1243
- "3": 0.0012
1244
  },
1245
- "latency_s": 0.067
1246
  },
1247
  {
1248
  "id": "original-ordinal-03-1",
@@ -1253,12 +1253,12 @@
1253
  "valid": true,
1254
  "error": null,
1255
  "probs": {
1256
- "0": 0.0182,
1257
- "1": 0.0959,
1258
- "2": 0.8829,
1259
- "3": 0.0029
1260
  },
1261
- "latency_s": 0.067
1262
  },
1263
  {
1264
  "id": "original-ordinal-04-0",
@@ -1269,12 +1269,12 @@
1269
  "valid": true,
1270
  "error": null,
1271
  "probs": {
1272
- "0": 0.0041,
1273
- "1": 0.002,
1274
- "2": 0.0051,
1275
- "3": 0.9888
1276
  },
1277
- "latency_s": 0.067
1278
  },
1279
  {
1280
  "id": "original-ordinal-04-1",
@@ -1285,12 +1285,12 @@
1285
  "valid": true,
1286
  "error": null,
1287
  "probs": {
1288
- "0": 0.0057,
1289
- "1": 0.0046,
1290
- "2": 0.0392,
1291
- "3": 0.9505
1292
  },
1293
- "latency_s": 0.067
1294
  },
1295
  {
1296
  "id": "original-ordinal-05-0",
@@ -1301,12 +1301,12 @@
1301
  "valid": true,
1302
  "error": null,
1303
  "probs": {
1304
- "0": 0.9805,
1305
- "1": 0.0145,
1306
- "2": 0.004,
1307
- "3": 0.001
1308
  },
1309
- "latency_s": 0.067
1310
  },
1311
  {
1312
  "id": "original-ordinal-05-1",
@@ -1317,12 +1317,12 @@
1317
  "valid": true,
1318
  "error": null,
1319
  "probs": {
1320
- "0": 0.6784,
1321
- "1": 0.1454,
1322
- "2": 0.1454,
1323
- "3": 0.0308
1324
  },
1325
- "latency_s": 0.067
1326
  },
1327
  {
1328
  "id": "original-ordinal-06-0",
@@ -1333,12 +1333,12 @@
1333
  "valid": true,
1334
  "error": null,
1335
  "probs": {
1336
- "0": 0.0144,
1337
- "1": 0.0793,
1338
- "2": 0.9057,
1339
- "3": 0.0006
1340
  },
1341
- "latency_s": 0.067
1342
  },
1343
  {
1344
  "id": "original-ordinal-06-1",
@@ -1349,12 +1349,12 @@
1349
  "valid": true,
1350
  "error": null,
1351
  "probs": {
1352
- "0": 0.0055,
1353
- "1": 0.0809,
1354
- "2": 0.9129,
1355
- "3": 0.0008
1356
  },
1357
- "latency_s": 0.067
1358
  },
1359
  {
1360
  "id": "original-extraction-01-0",
@@ -1365,12 +1365,12 @@
1365
  "valid": true,
1366
  "error": null,
1367
  "probs": {
1368
- "courier": 0.0012,
1369
- "pickup": 0.9882,
1370
- "post": 0.0009,
1371
- "unknown": 0.0098
1372
  },
1373
- "latency_s": 0.066
1374
  },
1375
  {
1376
  "id": "original-extraction-01-1",
@@ -1381,12 +1381,12 @@
1381
  "valid": true,
1382
  "error": null,
1383
  "probs": {
1384
- "courier": 0.0004,
1385
- "pickup": 0.9926,
1386
- "post": 0.0007,
1387
- "unknown": 0.0064
1388
  },
1389
- "latency_s": 0.066
1390
  },
1391
  {
1392
  "id": "original-extraction-02-0",
@@ -1397,12 +1397,12 @@
1397
  "valid": true,
1398
  "error": null,
1399
  "probs": {
1400
- "courier": 0.0106,
1401
- "pickup": 0.0073,
1402
- "post": 0.3872,
1403
- "unknown": 0.5949
1404
  },
1405
- "latency_s": 0.066
1406
  },
1407
  {
1408
  "id": "original-extraction-02-1",
@@ -1413,12 +1413,12 @@
1413
  "valid": true,
1414
  "error": null,
1415
  "probs": {
1416
- "courier": 0.0022,
1417
- "pickup": 0.0015,
1418
- "post": 0.0141,
1419
- "unknown": 0.9822
1420
  },
1421
- "latency_s": 0.067
1422
  },
1423
  {
1424
  "id": "original-extraction-03-0",
@@ -1429,12 +1429,12 @@
1429
  "valid": true,
1430
  "error": null,
1431
  "probs": {
1432
- "courier": 0.9283,
1433
- "pickup": 0.007,
1434
- "post": 0.0078,
1435
- "unknown": 0.0569
1436
  },
1437
- "latency_s": 0.066
1438
  },
1439
  {
1440
  "id": "original-extraction-03-1",
@@ -1445,12 +1445,12 @@
1445
  "valid": true,
1446
  "error": null,
1447
  "probs": {
1448
- "courier": 0.9994,
1449
- "pickup": 0.0002,
1450
  "post": 0.0002,
1451
  "unknown": 0.0001
1452
  },
1453
- "latency_s": 0.066
1454
  },
1455
  {
1456
  "id": "original-extraction-04-0",
@@ -1461,12 +1461,12 @@
1461
  "valid": true,
1462
  "error": null,
1463
  "probs": {
1464
- "courier": 0.0002,
1465
  "pickup": 0.0004,
1466
- "post": 0.9976,
1467
- "unknown": 0.0018
1468
  },
1469
- "latency_s": 0.07
1470
  },
1471
  {
1472
  "id": "original-extraction-04-1",
@@ -1477,12 +1477,12 @@
1477
  "valid": true,
1478
  "error": null,
1479
  "probs": {
1480
- "courier": 0.0007,
1481
- "pickup": 0.0006,
1482
- "post": 0.9974,
1483
- "unknown": 0.0012
1484
  },
1485
- "latency_s": 0.066
1486
  },
1487
  {
1488
  "id": "original-extraction-05-0",
@@ -1493,12 +1493,12 @@
1493
  "valid": true,
1494
  "error": null,
1495
  "probs": {
1496
- "courier": 0.0006,
1497
- "pickup": 0.0044,
1498
- "post": 0.0015,
1499
- "unknown": 0.9935
1500
  },
1501
- "latency_s": 0.066
1502
  },
1503
  {
1504
  "id": "original-extraction-05-1",
@@ -1509,12 +1509,12 @@
1509
  "valid": true,
1510
  "error": null,
1511
  "probs": {
1512
- "courier": 0.7409,
1513
- "pickup": 0.0096,
1514
- "post": 0.0096,
1515
- "unknown": 0.2399
1516
  },
1517
- "latency_s": 0.067
1518
  },
1519
  {
1520
  "id": "original-extraction-06-0",
@@ -1525,12 +1525,12 @@
1525
  "valid": true,
1526
  "error": null,
1527
  "probs": {
1528
- "courier": 0.9991,
1529
- "pickup": 0.0002,
1530
- "post": 0.0003,
1531
- "unknown": 0.0004
1532
  },
1533
- "latency_s": 0.072
1534
  },
1535
  {
1536
  "id": "original-extraction-06-1",
@@ -1541,12 +1541,12 @@
1541
  "valid": true,
1542
  "error": null,
1543
  "probs": {
1544
- "courier": 0.9988,
1545
- "pickup": 0.0003,
1546
- "post": 0.0003,
1547
- "unknown": 0.0007
1548
  },
1549
- "latency_s": 0.068
1550
  },
1551
  {
1552
  "id": "original-adequacy-01-0",
@@ -1557,10 +1557,10 @@
1557
  "valid": true,
1558
  "error": null,
1559
  "probs": {
1560
- "yes": 0.8343,
1561
- "no": 0.1657
1562
  },
1563
- "latency_s": 0.067
1564
  },
1565
  {
1566
  "id": "original-adequacy-01-1",
@@ -1571,10 +1571,10 @@
1571
  "valid": true,
1572
  "error": null,
1573
  "probs": {
1574
- "yes": 0.7235,
1575
- "no": 0.2765
1576
  },
1577
- "latency_s": 0.067
1578
  },
1579
  {
1580
  "id": "original-adequacy-02-0",
@@ -1585,10 +1585,10 @@
1585
  "valid": true,
1586
  "error": null,
1587
  "probs": {
1588
- "yes": 0.0657,
1589
- "no": 0.9343
1590
  },
1591
- "latency_s": 0.067
1592
  },
1593
  {
1594
  "id": "original-adequacy-02-1",
@@ -1599,10 +1599,10 @@
1599
  "valid": true,
1600
  "error": null,
1601
  "probs": {
1602
- "yes": 0.0904,
1603
- "no": 0.9096
1604
  },
1605
- "latency_s": 0.067
1606
  },
1607
  {
1608
  "id": "original-adequacy-03-0",
@@ -1613,10 +1613,10 @@
1613
  "valid": true,
1614
  "error": null,
1615
  "probs": {
1616
- "yes": 0.7673,
1617
- "no": 0.2327
1618
  },
1619
- "latency_s": 0.067
1620
  },
1621
  {
1622
  "id": "original-adequacy-03-1",
@@ -1627,10 +1627,10 @@
1627
  "valid": true,
1628
  "error": null,
1629
  "probs": {
1630
- "yes": 0.658,
1631
- "no": 0.342
1632
  },
1633
- "latency_s": 0.067
1634
  },
1635
  {
1636
  "id": "original-adequacy-04-0",
@@ -1641,10 +1641,10 @@
1641
  "valid": true,
1642
  "error": null,
1643
  "probs": {
1644
- "yes": 0.8395,
1645
- "no": 0.1605
1646
  },
1647
- "latency_s": 0.067
1648
  },
1649
  {
1650
  "id": "original-adequacy-04-1",
@@ -1655,24 +1655,24 @@
1655
  "valid": true,
1656
  "error": null,
1657
  "probs": {
1658
- "yes": 0.6134,
1659
- "no": 0.3866
1660
  },
1661
- "latency_s": 0.067
1662
  },
1663
  {
1664
  "id": "original-adequacy-05-0",
1665
  "tier": "standard",
1666
  "expected": "no",
1667
- "predicted": "yes",
1668
- "correct": false,
1669
  "valid": true,
1670
  "error": null,
1671
  "probs": {
1672
- "yes": 0.5,
1673
- "no": 0.5
1674
  },
1675
- "latency_s": 0.067
1676
  },
1677
  {
1678
  "id": "original-adequacy-05-1",
@@ -1683,38 +1683,38 @@
1683
  "valid": true,
1684
  "error": null,
1685
  "probs": {
1686
- "yes": 0.6225,
1687
- "no": 0.3775
1688
  },
1689
- "latency_s": 0.068
1690
  },
1691
  {
1692
  "id": "original-adequacy-06-0",
1693
  "tier": "standard",
1694
  "expected": "yes",
1695
- "predicted": "yes",
1696
- "correct": true,
1697
  "valid": true,
1698
  "error": null,
1699
  "probs": {
1700
- "yes": 0.5288,
1701
- "no": 0.4712
1702
  },
1703
- "latency_s": 0.067
1704
  },
1705
  {
1706
  "id": "original-adequacy-06-1",
1707
  "tier": "standard",
1708
  "expected": "yes",
1709
- "predicted": "yes",
1710
- "correct": true,
1711
  "valid": true,
1712
  "error": null,
1713
  "probs": {
1714
- "yes": 0.5,
1715
- "no": 0.5
1716
  },
1717
- "latency_s": 0.067
1718
  },
1719
  {
1720
  "id": "original-routing-01-0",
@@ -1725,14 +1725,14 @@
1725
  "valid": true,
1726
  "error": null,
1727
  "probs": {
1728
- "coding": 0.012,
1729
- "coding_agent": 0.0009,
1730
- "document": 0.0028,
1731
- "general": 0.0512,
1732
- "math": 0.9296,
1733
- "tools": 0.0035
1734
  },
1735
- "latency_s": 0.067
1736
  },
1737
  {
1738
  "id": "original-routing-01-1",
@@ -1743,14 +1743,14 @@
1743
  "valid": true,
1744
  "error": null,
1745
  "probs": {
1746
- "coding": 0.0093,
1747
- "coding_agent": 0.0007,
1748
- "document": 0.0033,
1749
- "general": 0.0464,
1750
- "math": 0.938,
1751
- "tools": 0.0023
1752
  },
1753
- "latency_s": 0.068
1754
  },
1755
  {
1756
  "id": "original-routing-02-0",
@@ -1761,12 +1761,12 @@
1761
  "valid": true,
1762
  "error": null,
1763
  "probs": {
1764
- "coding": 0.9776,
1765
- "coding_agent": 0.0113,
1766
- "document": 0.0014,
1767
- "general": 0.0082,
1768
- "math": 0.0006,
1769
- "tools": 0.0008
1770
  },
1771
  "latency_s": 0.07
1772
  },
@@ -1779,12 +1779,12 @@
1779
  "valid": true,
1780
  "error": null,
1781
  "probs": {
1782
- "coding": 0.9851,
1783
- "coding_agent": 0.0054,
1784
- "document": 0.0013,
1785
- "general": 0.007,
1786
- "math": 0.0005,
1787
- "tools": 0.0006
1788
  },
1789
  "latency_s": 0.069
1790
  },
@@ -1797,12 +1797,12 @@
1797
  "valid": true,
1798
  "error": null,
1799
  "probs": {
1800
- "coding": 0.0011,
1801
  "coding_agent": 0.9965,
1802
- "document": 0.0005,
1803
  "general": 0.0011,
1804
  "math": 0.0002,
1805
- "tools": 0.0005
1806
  },
1807
  "latency_s": 0.069
1808
  },
@@ -1817,30 +1817,30 @@
1817
  "probs": {
1818
  "coding": 0.0007,
1819
  "coding_agent": 0.9959,
1820
- "document": 0.0006,
1821
- "general": 0.0021,
1822
  "math": 0.0002,
1823
- "tools": 0.0005
1824
  },
1825
- "latency_s": 0.067
1826
  },
1827
  {
1828
  "id": "original-routing-04-0",
1829
  "tier": "standard",
1830
- "expected": "document",
1831
- "predicted": "general",
1832
- "correct": false,
1833
  "valid": true,
1834
  "error": null,
1835
  "probs": {
1836
- "coding": 0.0029,
1837
  "coding_agent": 0.004,
1838
- "document": 0.4781,
1839
- "general": 0.5045,
1840
- "math": 0.002,
1841
- "tools": 0.0085
1842
  },
1843
- "latency_s": 0.068
1844
  },
1845
  {
1846
  "id": "original-routing-04-1",
@@ -1851,14 +1851,14 @@
1851
  "valid": true,
1852
  "error": null,
1853
  "probs": {
1854
- "coding": 0.0017,
1855
- "coding_agent": 0.0022,
1856
- "document": 0.8249,
1857
- "general": 0.1648,
1858
- "math": 0.0011,
1859
- "tools": 0.0053
1860
  },
1861
- "latency_s": 0.069
1862
  },
1863
  {
1864
  "id": "original-routing-05-0",
@@ -1869,14 +1869,14 @@
1869
  "valid": true,
1870
  "error": null,
1871
  "probs": {
1872
- "coding": 0.0022,
1873
  "coding_agent": 0.0014,
1874
- "document": 0.0065,
1875
- "general": 0.4283,
1876
- "math": 0.0013,
1877
- "tools": 0.5602
1878
  },
1879
- "latency_s": 0.069
1880
  },
1881
  {
1882
  "id": "original-routing-05-1",
@@ -1887,14 +1887,14 @@
1887
  "valid": true,
1888
  "error": null,
1889
  "probs": {
1890
- "coding": 0.001,
1891
  "coding_agent": 0.0009,
1892
- "document": 0.0029,
1893
- "general": 0.9046,
1894
- "math": 0.0007,
1895
- "tools": 0.0899
1896
  },
1897
- "latency_s": 0.067
1898
  },
1899
  {
1900
  "id": "original-routing-06-0",
@@ -1905,14 +1905,14 @@
1905
  "valid": true,
1906
  "error": null,
1907
  "probs": {
1908
- "coding": 0.0007,
1909
- "coding_agent": 0.0008,
1910
- "document": 0.0051,
1911
- "general": 0.9912,
1912
- "math": 0.0007,
1913
  "tools": 0.0013
1914
  },
1915
- "latency_s": 0.067
1916
  },
1917
  {
1918
  "id": "original-routing-06-1",
@@ -1923,14 +1923,14 @@
1923
  "valid": true,
1924
  "error": null,
1925
  "probs": {
1926
- "coding": 0.0009,
1927
- "coding_agent": 0.0008,
1928
- "document": 0.0046,
1929
- "general": 0.9914,
1930
- "math": 0.0007,
1931
- "tools": 0.0016
1932
  },
1933
- "latency_s": 0.066
1934
  },
1935
  {
1936
  "id": "hard-opus-a-long_policy-01",
@@ -1941,13 +1941,13 @@
1941
  "valid": true,
1942
  "error": null,
1943
  "probs": {
1944
- "deny_repeated_seepage": 0.0007,
1945
- "deny_vacancy_exclusion": 0.0011,
1946
- "pay_full_estimate_less_deductible": 0.0011,
1947
- "pay_subject_to_10000_sublimit": 0.0327,
1948
- "pay_subject_to_15000_sublimit": 0.9644
1949
  },
1950
- "latency_s": 0.151
1951
  },
1952
  {
1953
  "id": "hard-opus-a-long_policy-04",
@@ -1958,13 +1958,13 @@
1958
  "valid": true,
1959
  "error": null,
1960
  "probs": {
1961
- "budget_holder": 0.0005,
1962
- "cfo": 0.0009,
1963
- "department_head": 0.0029,
1964
- "executive_committee": 0.0007,
1965
- "vp_and_finance_director": 0.995
1966
  },
1967
- "latency_s": 0.088
1968
  },
1969
  {
1970
  "id": "hard-opus-a-long_policy-08",
@@ -1975,12 +1975,12 @@
1975
  "valid": true,
1976
  "error": null,
1977
  "probs": {
1978
- "full_repair_no_charge": 0.0127,
1979
- "not_covered": 0.0046,
1980
- "parts_covered_labour_charged": 0.9797,
1981
- "pro_rata_replacement_credit": 0.0031
1982
  },
1983
- "latency_s": 0.082
1984
  },
1985
  {
1986
  "id": "hard-opus-a-long_policy-09",
@@ -1991,12 +1991,12 @@
1991
  "valid": true,
1992
  "error": null,
1993
  "probs": {
1994
- "license_exception_lvs": 0.0002,
1995
- "license_required": 0.9993,
1996
- "no_license_required": 0.0003,
1997
- "shipment_prohibited": 0.0002
1998
  },
1999
- "latency_s": 0.084
2000
  },
2001
  {
2002
  "id": "hard-opus-a-long_policy-11",
@@ -2007,24 +2007,24 @@
2007
  "valid": true,
2008
  "error": null,
2009
  "probs": {
2010
- "yes": 0.0049,
2011
- "no": 0.9951
2012
  },
2013
- "latency_s": 0.081
2014
  },
2015
  {
2016
  "id": "hard-opus-a-long_policy-13",
2017
  "tier": "hard",
2018
  "expected": "yes",
2019
- "predicted": "no",
2020
- "correct": false,
2021
  "valid": true,
2022
  "error": null,
2023
  "probs": {
2024
- "yes": 0.2127,
2025
- "no": 0.7873
2026
  },
2027
- "latency_s": 0.082
2028
  },
2029
  {
2030
  "id": "hard-opus-a-long_policy-17",
@@ -2035,13 +2035,13 @@
2035
  "valid": true,
2036
  "error": null,
2037
  "probs": {
2038
- "database_oncall": 0.9962,
2039
- "no_page_ticket_only": 0.0012,
2040
- "platform_oncall": 0.0013,
2041
- "security_incident_response": 0.0005,
2042
  "vendor_support": 0.0008
2043
  },
2044
- "latency_s": 0.081
2045
  },
2046
  {
2047
  "id": "hard-opus-a-long_policy-19",
@@ -2052,10 +2052,10 @@
2052
  "valid": true,
2053
  "error": null,
2054
  "probs": {
2055
- "yes": 0.0081,
2056
- "no": 0.9919
2057
  },
2058
- "latency_s": 0.081
2059
  },
2060
  {
2061
  "id": "hard-opus-a-probability-03",
@@ -2066,10 +2066,10 @@
2066
  "valid": true,
2067
  "error": null,
2068
  "probs": {
2069
- "yes": 0.4425,
2070
- "no": 0.5575
2071
  },
2072
- "latency_s": 0.069
2073
  },
2074
  {
2075
  "id": "hard-opus-a-probability-04",
@@ -2080,11 +2080,11 @@
2080
  "valid": true,
2081
  "error": null,
2082
  "probs": {
2083
- "early": 0.0011,
2084
- "late": 0.0364,
2085
- "on_time": 0.9625
2086
  },
2087
- "latency_s": 0.071
2088
  },
2089
  {
2090
  "id": "hard-opus-a-probability-07",
@@ -2095,11 +2095,11 @@
2095
  "valid": true,
2096
  "error": null,
2097
  "probs": {
2098
- "escalated": 0.9957,
2099
- "reopened": 0.001,
2100
- "resolved_first_contact": 0.0034
2101
  },
2102
- "latency_s": 0.071
2103
  },
2104
  {
2105
  "id": "hard-opus-a-probability-08",
@@ -2110,10 +2110,10 @@
2110
  "valid": true,
2111
  "error": null,
2112
  "probs": {
2113
- "yes": 0.4616,
2114
- "no": 0.5384
2115
  },
2116
- "latency_s": 0.068
2117
  },
2118
  {
2119
  "id": "hard-opus-a-temporal_numeric-03",
@@ -2124,10 +2124,10 @@
2124
  "valid": true,
2125
  "error": null,
2126
  "probs": {
2127
- "yes": 0.7312,
2128
- "no": 0.2688
2129
  },
2130
- "latency_s": 0.068
2131
  },
2132
  {
2133
  "id": "hard-opus-a-temporal_numeric-06",
@@ -2138,10 +2138,10 @@
2138
  "valid": true,
2139
  "error": null,
2140
  "probs": {
2141
- "yes": 0.038,
2142
- "no": 0.962
2143
  },
2144
- "latency_s": 0.068
2145
  },
2146
  {
2147
  "id": "hard-opus-a-temporal_numeric-07",
@@ -2152,13 +2152,13 @@
2152
  "valid": true,
2153
  "error": null,
2154
  "probs": {
2155
- "sep_24": 0.3247,
2156
- "sep_25": 0.0116,
2157
- "sep_26": 0.003,
2158
- "sep_27": 0.008,
2159
- "sep_28": 0.6526
2160
  },
2161
- "latency_s": 0.07
2162
  },
2163
  {
2164
  "id": "hard-opus-a-temporal_numeric-09",
@@ -2169,10 +2169,10 @@
2169
  "valid": true,
2170
  "error": null,
2171
  "probs": {
2172
- "yes": 0.1605,
2173
- "no": 0.8395
2174
  },
2175
- "latency_s": 0.068
2176
  },
2177
  {
2178
  "id": "hard-opus-a-temporal_numeric-12",
@@ -2183,12 +2183,12 @@
2183
  "valid": true,
2184
  "error": null,
2185
  "probs": {
2186
- "0": 0.0918,
2187
- "1": 0.519,
2188
- "2": 0.3855,
2189
- "3": 0.0036
2190
  },
2191
- "latency_s": 0.074
2192
  },
2193
  {
2194
  "id": "hard-opus-b-ambiguous-02",
@@ -2199,11 +2199,11 @@
2199
  "valid": true,
2200
  "error": null,
2201
  "probs": {
2202
- "cannot_determine": 0.0096,
2203
- "false_positive": 0.0183,
2204
- "real_incident": 0.9721
2205
  },
2206
- "latency_s": 0.07
2207
  },
2208
  {
2209
  "id": "hard-opus-b-ambiguous-03",
@@ -2214,11 +2214,11 @@
2214
  "valid": true,
2215
  "error": null,
2216
  "probs": {
2217
- "insufficient_information": 0.0015,
2218
- "time_barred": 0.3559,
2219
- "within_limitation": 0.6425
2220
  },
2221
- "latency_s": 0.068
2222
  },
2223
  {
2224
  "id": "hard-opus-b-ambiguous-07",
@@ -2229,11 +2229,11 @@
2229
  "valid": true,
2230
  "error": null,
2231
  "probs": {
2232
- "insufficient_information": 0.096,
2233
- "over_limit": 0.0817,
2234
- "within_policy": 0.8223
2235
  },
2236
- "latency_s": 0.068
2237
  },
2238
  {
2239
  "id": "hard-opus-b-ambiguous-09",
@@ -2244,11 +2244,11 @@
2244
  "valid": true,
2245
  "error": null,
2246
  "probs": {
2247
- "compromised": 0.2069,
2248
- "inconclusive": 0.001,
2249
- "legitimate": 0.7921
2250
  },
2251
- "latency_s": 0.07
2252
  },
2253
  {
2254
  "id": "hard-opus-b-ambiguous-10",
@@ -2259,11 +2259,11 @@
2259
  "valid": true,
2260
  "error": null,
2261
  "probs": {
2262
- "ask_user_which_alex": 0.0985,
2263
- "send_to_alex_moreno": 0.8906,
2264
  "send_to_alexandra_lind": 0.0109
2265
  },
2266
- "latency_s": 0.069
2267
  },
2268
  {
2269
  "id": "hard-opus-b-ambiguous-11",
@@ -2274,11 +2274,11 @@
2274
  "valid": true,
2275
  "error": null,
2276
  "probs": {
2277
- "cannot_determine": 0.9923,
2278
- "grade_correct": 0.0067,
2279
- "grade_should_be_higher": 0.001
2280
  },
2281
- "latency_s": 0.069
2282
  },
2283
  {
2284
  "id": "hard-opus-b-ambiguous-13",
@@ -2289,10 +2289,10 @@
2289
  "valid": true,
2290
  "error": null,
2291
  "probs": {
2292
- "cannot_determine": 0.9787,
2293
- "charge_data_team": 0.0078,
2294
- "charge_platform_team": 0.0092,
2295
- "split_50_50": 0.0043
2296
  },
2297
  "latency_s": 0.07
2298
  },
@@ -2305,10 +2305,10 @@
2305
  "valid": true,
2306
  "error": null,
2307
  "probs": {
2308
- "tier1_team_lead": 0.0675,
2309
- "tier2_department_head": 0.0394,
2310
- "tier3_cfo": 0.888,
2311
- "tier4_board": 0.0051
2312
  },
2313
  "latency_s": 0.09
2314
  },
@@ -2321,14 +2321,14 @@
2321
  "valid": true,
2322
  "error": null,
2323
  "probs": {
2324
- "ana": 0.0552,
2325
- "bjorn": 0.1235,
2326
- "chen": 0.6185,
2327
- "dmitri": 0.0099,
2328
- "esra": 0.013,
2329
- "no_page_ticket_only": 0.1799
2330
  },
2331
- "latency_s": 0.08
2332
  },
2333
  {
2334
  "id": "hard-opus-b-multi_hop-05",
@@ -2339,13 +2339,13 @@
2339
  "valid": true,
2340
  "error": null,
2341
  "probs": {
2342
- "0": 0.0009,
2343
- "1": 0.0067,
2344
- "2": 0.9879,
2345
- "3": 0.0022,
2346
- "4": 0.0022
2347
  },
2348
- "latency_s": 0.369
2349
  },
2350
  {
2351
  "id": "hard-opus-b-multi_hop-07",
@@ -2356,12 +2356,12 @@
2356
  "valid": true,
2357
  "error": null,
2358
  "probs": {
2359
- "credit_10_percent": 0.0045,
2360
- "credit_20_percent": 0.0072,
2361
- "credit_25_percent": 0.1625,
2362
- "credit_50_percent": 0.6562,
2363
- "credit_full_month": 0.1625,
2364
- "no_credit": 0.0072
2365
  },
2366
  "latency_s": 0.088
2367
  },
@@ -2374,10 +2374,10 @@
2374
  "valid": true,
2375
  "error": null,
2376
  "probs": {
2377
- "yes": 0.0366,
2378
- "no": 0.9634
2379
  },
2380
- "latency_s": 0.077
2381
  },
2382
  {
2383
  "id": "hard-opus-b-probability-01",
@@ -2388,11 +2388,11 @@
2388
  "valid": true,
2389
  "error": null,
2390
  "probs": {
2391
- "customer_wins": 0.0006,
2392
- "customer_withdraws": 0.0004,
2393
- "merchant_wins": 0.9991
2394
  },
2395
- "latency_s": 0.074
2396
  },
2397
  {
2398
  "id": "hard-opus-b-probability-02",
@@ -2403,11 +2403,11 @@
2403
  "valid": true,
2404
  "error": null,
2405
  "probs": {
2406
- "bad_push": 0.0149,
2407
- "database_hardware": 0.0021,
2408
- "upstream_provider": 0.983
2409
  },
2410
- "latency_s": 0.07
2411
  },
2412
  {
2413
  "id": "hard-opus-b-probability-03",
@@ -2418,10 +2418,10 @@
2418
  "valid": true,
2419
  "error": null,
2420
  "probs": {
2421
- "yes": 0.9269,
2422
- "no": 0.0731
2423
  },
2424
- "latency_s": 0.071
2425
  },
2426
  {
2427
  "id": "hard-opus-b-probability-04",
@@ -2432,11 +2432,11 @@
2432
  "valid": true,
2433
  "error": null,
2434
  "probs": {
2435
- "attended": 0.9984,
2436
- "late_cancellation": 0.001,
2437
- "no_show": 0.0006
2438
  },
2439
- "latency_s": 0.07
2440
  },
2441
  {
2442
  "id": "hard-opus-b-probability-06",
@@ -2448,10 +2448,10 @@
2448
  "error": null,
2449
  "probs": {
2450
  "modified": 0.0001,
2451
- "overturned": 0.0006,
2452
- "upheld": 0.9993
2453
  },
2454
- "latency_s": 0.071
2455
  },
2456
  {
2457
  "id": "hard-opus-b-tradeoff-01",
@@ -2462,13 +2462,13 @@
2462
  "valid": true,
2463
  "error": null,
2464
  "probs": {
2465
- "escalate_to_engineering": 0.8595,
2466
- "escalate_to_security": 0.1385,
2467
- "page_oncall": 0.0009,
2468
- "process_refund": 0.0007,
2469
- "send_kb_article": 0.0004
2470
  },
2471
- "latency_s": 0.071
2472
  },
2473
  {
2474
  "id": "hard-opus-b-tradeoff-03",
@@ -2479,12 +2479,12 @@
2479
  "valid": true,
2480
  "error": null,
2481
  "probs": {
2482
- "order_a": 0.003,
2483
- "order_b": 0.9929,
2484
- "order_c": 0.0018,
2485
- "order_d": 0.0023
2486
  },
2487
- "latency_s": 0.071
2488
  },
2489
  {
2490
  "id": "hard-opus-b-tradeoff-06",
@@ -2495,10 +2495,10 @@
2495
  "valid": true,
2496
  "error": null,
2497
  "probs": {
2498
- "yes": 0.966,
2499
- "no": 0.034
2500
  },
2501
- "latency_s": 0.07
2502
  },
2503
  {
2504
  "id": "hard-opus-b-tradeoff-07",
@@ -2511,11 +2511,11 @@
2511
  "probs": {
2512
  "out_of_scope": 0.0001,
2513
  "p0_fix_24h": 0.9994,
2514
- "p1_fix_7d": 0.0003,
2515
  "p2_fix_30d": 0.0001,
2516
  "p3_backlog": 0.0001
2517
  },
2518
- "latency_s": 0.071
2519
  },
2520
  {
2521
  "id": "hard-opus-b-tradeoff-08",
@@ -2526,10 +2526,10 @@
2526
  "valid": true,
2527
  "error": null,
2528
  "probs": {
2529
- "yes": 0.4904,
2530
- "no": 0.5096
2531
  },
2532
- "latency_s": 0.07
2533
  },
2534
  {
2535
  "id": "hard-opus-b-tradeoff-12",
@@ -2540,13 +2540,13 @@
2540
  "valid": true,
2541
  "error": null,
2542
  "probs": {
2543
- "deny_refund": 0.0001,
2544
- "escalate_to_fraud_team": 0.0001,
2545
- "offer_store_credit": 0.0002,
2546
- "refund_now": 0.9996,
2547
- "wait_for_inspection": 0.0001
2548
  },
2549
- "latency_s": 0.07
2550
  },
2551
  {
2552
  "id": "hard-opus-c-long_policy-02",
@@ -2557,11 +2557,11 @@
2557
  "valid": true,
2558
  "error": null,
2559
  "probs": {
2560
- "erase_all": 0.0004,
2561
- "erase_but_retain_transaction_records": 0.0216,
2562
- "erase_marketing_only": 0.0004,
2563
- "reject_unverified_request": 0.0002,
2564
- "restrict_all_pending_legal_hold": 0.9773
2565
  },
2566
  "latency_s": 0.118
2567
  },
@@ -2569,17 +2569,17 @@
2569
  "id": "hard-opus-c-long_policy-03",
2570
  "tier": "hard",
2571
  "expected": "eur_400",
2572
- "predicted": "eur_150",
2573
- "correct": false,
2574
  "valid": true,
2575
  "error": null,
2576
  "probs": {
2577
- "eur_0": 0.0407,
2578
- "eur_150": 0.4322,
2579
- "eur_200": 0.215,
2580
- "eur_300": 0.0593,
2581
- "eur_400": 0.1399,
2582
- "eur_600": 0.1129
2583
  },
2584
  "latency_s": 0.118
2585
  },
@@ -2587,13 +2587,13 @@
2587
  "id": "hard-opus-c-long_policy-04",
2588
  "tier": "hard",
2589
  "expected": "yes",
2590
- "predicted": "yes",
2591
- "correct": true,
2592
  "valid": true,
2593
  "error": null,
2594
  "probs": {
2595
- "yes": 0.6134,
2596
- "no": 0.3866
2597
  },
2598
  "latency_s": 0.105
2599
  },
@@ -2606,12 +2606,12 @@
2606
  "valid": true,
2607
  "error": null,
2608
  "probs": {
2609
- "0": 0.0088,
2610
- "1": 0.3272,
2611
- "2": 0.6427,
2612
- "3": 0.0212
2613
  },
2614
- "latency_s": 0.374
2615
  },
2616
  {
2617
  "id": "hard-opus-c-long_policy-08",
@@ -2622,10 +2622,10 @@
2622
  "valid": true,
2623
  "error": null,
2624
  "probs": {
2625
- "0": 0.0013,
2626
- "1": 0.9959,
2627
- "2": 0.0012,
2628
- "3": 0.0017
2629
  },
2630
  "latency_s": 0.38
2631
  },
@@ -2638,12 +2638,12 @@
2638
  "valid": true,
2639
  "error": null,
2640
  "probs": {
2641
- "credit_minus_restocking_fee": 0.9975,
2642
- "full_credit": 0.001,
2643
- "reject_return": 0.0011,
2644
- "repair_or_replace_only": 0.0004
2645
  },
2646
- "latency_s": 0.121
2647
  },
2648
  {
2649
  "id": "hard-opus-c-long_policy-11",
@@ -2654,10 +2654,10 @@
2654
  "valid": true,
2655
  "error": null,
2656
  "probs": {
2657
- "yes": 0.7235,
2658
- "no": 0.2765
2659
  },
2660
- "latency_s": 0.105
2661
  },
2662
  {
2663
  "id": "hard-opus-c-probability-03",
@@ -2668,11 +2668,11 @@
2668
  "valid": true,
2669
  "error": null,
2670
  "probs": {
2671
- "halvorsen_polymer": 0.9962,
2672
- "kuznets_technik": 0.0018,
2673
- "ortega_seal_works": 0.0021
2674
  },
2675
- "latency_s": 0.071
2676
  },
2677
  {
2678
  "id": "hard-opus-c-temporal_numeric-02",
@@ -2683,10 +2683,10 @@
2683
  "valid": true,
2684
  "error": null,
2685
  "probs": {
2686
- "yes": 0.1766,
2687
- "no": 0.8234
2688
  },
2689
- "latency_s": 0.071
2690
  },
2691
  {
2692
  "id": "hard-opus-c-temporal_numeric-03",
@@ -2697,48 +2697,48 @@
2697
  "valid": true,
2698
  "error": null,
2699
  "probs": {
2700
- "eur_617_00": 0.0165,
2701
- "eur_688_00": 0.0538,
2702
- "eur_692_00": 0.2553,
2703
- "eur_693_90": 0.6712,
2704
- "eur_812_00": 0.0031
2705
  },
2706
- "latency_s": 0.071
2707
  },
2708
  {
2709
  "id": "hard-opus-c-temporal_numeric-04",
2710
  "tier": "hard",
2711
  "expected": "usd_2335_00",
2712
- "predicted": "usd_2303_01",
2713
  "correct": false,
2714
  "valid": true,
2715
  "error": null,
2716
  "probs": {
2717
- "usd_2222_50": 0.0251,
2718
- "usd_2303_01": 0.597,
2719
- "usd_2303_33": 0.3134,
2720
- "usd_2335_00": 0.0533,
2721
- "usd_2340_00": 0.0112
2722
  },
2723
- "latency_s": 0.07
2724
  },
2725
  {
2726
  "id": "hard-opus-c-temporal_numeric-06",
2727
  "tier": "hard",
2728
  "expected": "sep_18_book_set",
2729
- "predicted": "none_declined",
2730
  "correct": false,
2731
  "valid": true,
2732
  "error": null,
2733
  "probs": {
2734
- "none_declined": 0.6977,
2735
- "sep_15_office_chair": 0.101,
2736
- "sep_16_taxi": 0.0202,
2737
- "sep_17_parking": 0.1252,
2738
- "sep_18_book_set": 0.0405,
2739
- "sep_21_client_dinner": 0.0154
2740
  },
2741
- "latency_s": 0.07
2742
  },
2743
  {
2744
  "id": "hard-opus-c-temporal_numeric-08",
@@ -2749,13 +2749,13 @@
2749
  "valid": true,
2750
  "error": null,
2751
  "probs": {
2752
- "eur_497_40": 0.0582,
2753
- "eur_498_81": 0.2482,
2754
- "eur_506_06": 0.1451,
2755
- "eur_513_69": 0.4989,
2756
- "eur_532_01": 0.0496
2757
  },
2758
- "latency_s": 0.07
2759
  },
2760
  {
2761
  "id": "hard-opus-c-temporal_numeric-12",
@@ -2766,11 +2766,11 @@
2766
  "valid": true,
2767
  "error": null,
2768
  "probs": {
2769
- "covered": 0.0038,
2770
- "expired_24_month_term": 0.2964,
2771
- "expired_30_month_cap": 0.6998
2772
  },
2773
- "latency_s": 0.07
2774
  },
2775
  {
2776
  "id": "hard-sol-a-adversarial-01",
@@ -2781,12 +2781,12 @@
2781
  "valid": true,
2782
  "error": null,
2783
  "probs": {
2784
- "deny_inactive_user": 0.0001,
2785
- "deny_missing_approval": 0.9991,
2786
- "grant_access": 0.0005,
2787
  "grant_owner_exception": 0.0003
2788
  },
2789
- "latency_s": 0.068
2790
  },
2791
  {
2792
  "id": "hard-sol-a-adversarial-06",
@@ -2797,10 +2797,10 @@
2797
  "valid": true,
2798
  "error": null,
2799
  "probs": {
2800
- "yes": 0.0039,
2801
- "no": 0.9961
2802
  },
2803
- "latency_s": 0.068
2804
  },
2805
  {
2806
  "id": "hard-sol-a-adversarial-07",
@@ -2811,12 +2811,12 @@
2811
  "valid": true,
2812
  "error": null,
2813
  "probs": {
2814
- "delete_now": 0.0004,
2815
  "needs_signature_review": 0.0001,
2816
- "retain_indefinitely": 0.0002,
2817
- "retain_until_2031": 0.9993
2818
  },
2819
- "latency_s": 0.068
2820
  },
2821
  {
2822
  "id": "hard-sol-a-adversarial-08",
@@ -2827,10 +2827,10 @@
2827
  "valid": true,
2828
  "error": null,
2829
  "probs": {
2830
- "yes": 0.9959,
2831
- "no": 0.0041
2832
  },
2833
- "latency_s": 0.067
2834
  },
2835
  {
2836
  "id": "hard-sol-a-adversarial-09",
@@ -2841,44 +2841,44 @@
2841
  "valid": true,
2842
  "error": null,
2843
  "probs": {
2844
- "0": 0.0189,
2845
- "1": 0.0945,
2846
- "2": 0.8729,
2847
- "3": 0.0136
2848
  },
2849
- "latency_s": 0.07
2850
  },
2851
  {
2852
  "id": "hard-sol-a-adversarial-11",
2853
  "tier": "hard",
2854
  "expected": "standard_research",
2855
- "predicted": "ethics_review",
2856
- "correct": false,
2857
  "valid": true,
2858
  "error": null,
2859
  "probs": {
2860
- "ethics_review": 0.6565,
2861
- "major_grants": 0.0137,
2862
- "manual_hold": 0.0031,
2863
- "standard_research": 0.3266
2864
  },
2865
- "latency_s": 0.068
2866
  },
2867
  {
2868
  "id": "hard-sol-a-multi_hop-01",
2869
  "tier": "hard",
2870
  "expected": "accept_s2_dusk",
2871
- "predicted": "accept_s2_dusk",
2872
- "correct": true,
2873
  "valid": true,
2874
  "error": null,
2875
  "probs": {
2876
- "accept_s2_dusk": 0.6402,
2877
- "reject_cutoff": 0.3546,
2878
- "reject_tier": 0.003,
2879
  "reroute_s3": 0.0022
2880
  },
2881
- "latency_s": 0.083
2882
  },
2883
  {
2884
  "id": "hard-sol-a-multi_hop-02",
@@ -2889,12 +2889,12 @@
2889
  "valid": true,
2890
  "error": null,
2891
  "probs": {
2892
- "executive_bridge": 0.9964,
2893
- "heron_q2": 0.0013,
2894
- "insufficient_information": 0.0005,
2895
- "security_primary": 0.0018
2896
  },
2897
- "latency_s": 0.082
2898
  },
2899
  {
2900
  "id": "hard-sol-a-multi_hop-05",
@@ -2905,12 +2905,12 @@
2905
  "valid": true,
2906
  "error": null,
2907
  "probs": {
2908
- "pay_manager_approved": 0.0133,
2909
- "reject_category": 0.0074,
2910
- "require_director": 0.9778,
2911
- "require_finance_board": 0.0015
2912
  },
2913
- "latency_s": 0.081
2914
  },
2915
  {
2916
  "id": "hard-sol-a-multi_hop-07",
@@ -2921,12 +2921,12 @@
2921
  "valid": true,
2922
  "error": null,
2923
  "probs": {
2924
- "admit": 0.2409,
2925
- "deny_prerequisite": 0.7438,
2926
- "deny_score": 0.0091,
2927
- "manual_equivalency": 0.0063
2928
  },
2929
- "latency_s": 0.082
2930
  },
2931
  {
2932
  "id": "hard-sol-a-multi_hop-08",
@@ -2937,12 +2937,12 @@
2937
  "valid": true,
2938
  "error": null,
2939
  "probs": {
2940
- "central_engineering": 0.9135,
2941
- "insufficient_log": 0.0073,
2942
- "local_maintenance": 0.0773,
2943
- "safety_shutdown": 0.002
2944
  },
2945
- "latency_s": 0.082
2946
  },
2947
  {
2948
  "id": "hard-sol-a-multi_hop-09",
@@ -2953,12 +2953,12 @@
2953
  "valid": true,
2954
  "error": null,
2955
  "probs": {
2956
- "confirm_l3": 0.0035,
2957
- "reject_capacity": 0.0204,
2958
- "require_steward": 0.9748,
2959
- "switch_l4": 0.0014
2960
  },
2961
- "latency_s": 0.082
2962
  },
2963
  {
2964
  "id": "hard-sol-a-multi_hop-10",
@@ -2969,12 +2969,12 @@
2969
  "valid": true,
2970
  "error": null,
2971
  "probs": {
2972
- "approve_45_days": 0.9661,
2973
- "deny_sensitive": 0.0106,
2974
- "needs_field_review": 0.0069,
2975
- "reduce_to_30_days": 0.0163
2976
  },
2977
- "latency_s": 0.082
2978
  },
2979
  {
2980
  "id": "hard-sol-a-multi_hop-12",
@@ -2985,12 +2985,12 @@
2985
  "valid": true,
2986
  "error": null,
2987
  "probs": {
2988
- "covered_recurrence": 0.7149,
2989
- "expired": 0.177,
2990
- "needs_authentication": 0.0981,
2991
- "wrong_issue_class": 0.01
2992
  },
2993
- "latency_s": 0.082
2994
  },
2995
  {
2996
  "id": "hard-sol-a-trap-02",
@@ -3001,10 +3001,10 @@
3001
  "valid": true,
3002
  "error": null,
3003
  "probs": {
3004
- "yes": 0.0018,
3005
- "no": 0.9982
3006
  },
3007
- "latency_s": 0.068
3008
  },
3009
  {
3010
  "id": "hard-sol-a-trap-04",
@@ -3015,10 +3015,10 @@
3015
  "valid": true,
3016
  "error": null,
3017
  "probs": {
3018
- "yes": 0.0133,
3019
- "no": 0.9867
3020
  },
3021
- "latency_s": 0.068
3022
  },
3023
  {
3024
  "id": "hard-sol-a-trap-05",
@@ -3029,12 +3029,12 @@
3029
  "valid": true,
3030
  "error": null,
3031
  "probs": {
3032
- "0": 0.9908,
3033
- "1": 0.002,
3034
- "2": 0.0067,
3035
- "3": 0.0005
3036
  },
3037
- "latency_s": 0.072
3038
  },
3039
  {
3040
  "id": "hard-sol-a-trap-06",
@@ -3045,12 +3045,12 @@
3045
  "valid": true,
3046
  "error": null,
3047
  "probs": {
3048
- "await_internal_metrics": 0.0002,
3049
  "close_agent_verified": 0.0002,
3050
- "close_confirmed": 0.0005,
3051
- "keep_open_unresolved": 0.9992
3052
  },
3053
- "latency_s": 0.068
3054
  },
3055
  {
3056
  "id": "hard-sol-a-trap-08",
@@ -3062,11 +3062,11 @@
3062
  "error": null,
3063
  "probs": {
3064
  "canceled": 0.0,
3065
- "indeterminate": 0.0001,
3066
- "late": 0.9998,
3067
- "on_time": 0.0001
3068
  },
3069
- "latency_s": 0.068
3070
  },
3071
  {
3072
  "id": "hard-sol-a-trap-10",
@@ -3077,12 +3077,12 @@
3077
  "valid": true,
3078
  "error": null,
3079
  "probs": {
3080
- "blocked_failed_test": 0.0007,
3081
- "blocked_second_review": 0.994,
3082
- "ready_one_review": 0.0016,
3083
- "ready_two_reviews": 0.0037
3084
  },
3085
- "latency_s": 0.067
3086
  },
3087
  {
3088
  "id": "hard-sol-a-trap-13",
@@ -3093,10 +3093,10 @@
3093
  "valid": true,
3094
  "error": null,
3095
  "probs": {
3096
- "yes": 0.0053,
3097
- "no": 0.9947
3098
  },
3099
- "latency_s": 0.067
3100
  },
3101
  {
3102
  "id": "hard-sol-a-trap-15",
@@ -3107,10 +3107,10 @@
3107
  "valid": true,
3108
  "error": null,
3109
  "probs": {
3110
- "yes": 0.1823,
3111
- "no": 0.8177
3112
  },
3113
- "latency_s": 0.068
3114
  },
3115
  {
3116
  "id": "hard-sol-b-judge_hard-01",
@@ -3121,10 +3121,10 @@
3121
  "valid": true,
3122
  "error": null,
3123
  "probs": {
3124
- "yes": 0.7079,
3125
- "no": 0.2921
3126
  },
3127
- "latency_s": 0.068
3128
  },
3129
  {
3130
  "id": "hard-sol-b-judge_hard-02",
@@ -3135,10 +3135,10 @@
3135
  "valid": true,
3136
  "error": null,
3137
  "probs": {
3138
- "yes": 0.8925,
3139
- "no": 0.1075
3140
  },
3141
- "latency_s": 0.068
3142
  },
3143
  {
3144
  "id": "hard-sol-b-judge_hard-05",
@@ -3149,10 +3149,10 @@
3149
  "valid": true,
3150
  "error": null,
3151
  "probs": {
3152
- "yes": 0.7387,
3153
- "no": 0.2613
3154
  },
3155
- "latency_s": 0.067
3156
  },
3157
  {
3158
  "id": "hard-sol-b-judge_hard-08",
@@ -3163,10 +3163,10 @@
3163
  "valid": true,
3164
  "error": null,
3165
  "probs": {
3166
- "yes": 0.9452,
3167
- "no": 0.0548
3168
  },
3169
- "latency_s": 0.067
3170
  },
3171
  {
3172
  "id": "hard-sol-b-judge_hard-10",
@@ -3177,10 +3177,10 @@
3177
  "valid": true,
3178
  "error": null,
3179
  "probs": {
3180
- "yes": 0.3249,
3181
- "no": 0.6751
3182
  },
3183
- "latency_s": 0.067
3184
  },
3185
  {
3186
  "id": "hard-sol-b-judge_hard-14",
@@ -3191,10 +3191,10 @@
3191
  "valid": true,
3192
  "error": null,
3193
  "probs": {
3194
- "yes": 0.0138,
3195
- "no": 0.9862
3196
  },
3197
- "latency_s": 0.067
3198
  },
3199
  {
3200
  "id": "hard-sol-b-judge_hard-15",
@@ -3205,10 +3205,10 @@
3205
  "valid": true,
3206
  "error": null,
3207
  "probs": {
3208
- "yes": 0.8849,
3209
- "no": 0.1151
3210
  },
3211
- "latency_s": 0.067
3212
  },
3213
  {
3214
  "id": "hard-sol-b-judge_hard-18",
@@ -3219,10 +3219,10 @@
3219
  "valid": true,
3220
  "error": null,
3221
  "probs": {
3222
- "yes": 0.9799,
3223
- "no": 0.0201
3224
  },
3225
- "latency_s": 0.067
3226
  },
3227
  {
3228
  "id": "hard-sol-b-long_policy-01",
@@ -3233,28 +3233,28 @@
3233
  "valid": true,
3234
  "error": null,
3235
  "probs": {
3236
- "approve_all": 0.004,
3237
- "approve_lodging_only": 0.0248,
3238
- "approve_meals_only": 0.0145,
3239
- "reject_entire_claim": 0.9566
3240
  },
3241
- "latency_s": 0.09
3242
  },
3243
  {
3244
  "id": "hard-sol-b-long_policy-02",
3245
  "tier": "hard",
3246
  "expected": "deny_current_export",
3247
- "predicted": "privacy_only",
3248
- "correct": false,
3249
  "valid": true,
3250
  "error": null,
3251
  "probs": {
3252
- "analytics_owner_only": 0.006,
3253
- "deny_current_export": 0.4371,
3254
- "privacy_and_security": 0.0434,
3255
- "privacy_only": 0.5135
3256
  },
3257
- "latency_s": 0.087
3258
  },
3259
  {
3260
  "id": "hard-sol-b-long_policy-05",
@@ -3265,12 +3265,12 @@
3265
  "valid": true,
3266
  "error": null,
3267
  "probs": {
3268
- "delete_everything": 0.0006,
3269
- "delete_nothing": 0.9697,
3270
- "delete_only_profile": 0.028,
3271
- "delete_profile_omar_april_and_table": 0.0016
3272
  },
3273
- "latency_s": 0.084
3274
  },
3275
  {
3276
  "id": "hard-sol-b-long_policy-06",
@@ -3281,12 +3281,12 @@
3281
  "valid": true,
3282
  "error": null,
3283
  "probs": {
3284
- "east_ward": 0.0067,
3285
- "harbor_neighbors": 0.0012,
3286
- "no_award": 0.0024,
3287
- "river_arts": 0.9897
3288
  },
3289
- "latency_s": 0.087
3290
  },
3291
  {
3292
  "id": "hard-sol-b-routing_hard-01",
@@ -3300,10 +3300,10 @@
3300
  "data_pipeline": 0.0,
3301
  "desktop_support": 0.0001,
3302
  "finance_review": 0.0001,
3303
- "office_file_repair": 0.9995,
3304
  "spreadsheet_formula": 0.0003
3305
  },
3306
- "latency_s": 0.068
3307
  },
3308
  {
3309
  "id": "hard-sol-b-routing_hard-02",
@@ -3315,12 +3315,12 @@
3315
  "error": null,
3316
  "probs": {
3317
  "customer_support": 0.0001,
3318
- "fraud_review": 0.0001,
3319
  "ledger_reconciliation": 0.9996,
3320
  "mobile_app": 0.0001,
3321
  "payment_disputes": 0.0002
3322
  },
3323
- "latency_s": 0.068
3324
  },
3325
  {
3326
  "id": "hard-sol-b-routing_hard-03",
@@ -3331,13 +3331,13 @@
3331
  "valid": true,
3332
  "error": null,
3333
  "probs": {
3334
- "api_platform": 0.0003,
3335
- "authoritative_dns": 0.0006,
3336
- "endpoint_devices": 0.0032,
3337
- "network_dns": 0.9956,
3338
- "tls_certificates": 0.0003
3339
  },
3340
- "latency_s": 0.069
3341
  },
3342
  {
3343
  "id": "hard-sol-b-routing_hard-07",
@@ -3349,12 +3349,12 @@
3349
  "error": null,
3350
  "probs": {
3351
  "ci_infrastructure": 0.0001,
3352
- "code_signing": 0.0003,
3353
- "endpoint_security": 0.0002,
3354
- "macos_runtime": 0.0002,
3355
- "release_notarization": 0.9993
3356
  },
3357
- "latency_s": 0.068
3358
  },
3359
  {
3360
  "id": "hard-sol-b-routing_hard-09",
@@ -3371,7 +3371,7 @@
3371
  "source_access": 0.9996,
3372
  "vpn_network": 0.0002
3373
  },
3374
- "latency_s": 0.068
3375
  },
3376
  {
3377
  "id": "hard-sol-b-temporal_numeric-01",
@@ -3382,12 +3382,12 @@
3382
  "valid": true,
3383
  "error": null,
3384
  "probs": {
3385
- "before_open": 0.0166,
3386
- "late_by_2h_or_more": 0.0877,
3387
- "late_by_under_2h": 0.2566,
3388
- "within_window": 0.6392
3389
  },
3390
- "latency_s": 0.068
3391
  },
3392
  {
3393
  "id": "hard-sol-b-temporal_numeric-02",
@@ -3398,12 +3398,12 @@
3398
  "valid": true,
3399
  "error": null,
3400
  "probs": {
3401
- "at_least_100gb_below": 0.2542,
3402
- "exactly_equal": 0.041,
3403
- "over_limit": 0.0368,
3404
- "under_100gb_below": 0.6681
3405
  },
3406
- "latency_s": 0.068
3407
  },
3408
  {
3409
  "id": "hard-sol-b-temporal_numeric-03",
@@ -3414,12 +3414,12 @@
3414
  "valid": true,
3415
  "error": null,
3416
  "probs": {
3417
- "12_credits": 0.1823,
3418
- "15_credits": 0.059,
3419
- "6_credits": 0.1321,
3420
- "9_credits": 0.6267
3421
  },
3422
- "latency_s": 0.067
3423
  },
3424
  {
3425
  "id": "hard-sol-b-temporal_numeric-04",
@@ -3430,11 +3430,11 @@
3430
  "valid": true,
3431
  "error": null,
3432
  "probs": {
3433
- "clear_fail": 0.1427,
3434
- "narrow_fail": 0.7539,
3435
- "pass": 0.1034
3436
  },
3437
- "latency_s": 0.067
3438
  },
3439
  {
3440
  "id": "hard-sol-c-judge_hard-03",
@@ -3445,10 +3445,10 @@
3445
  "valid": true,
3446
  "error": null,
3447
  "probs": {
3448
- "yes": 0.4616,
3449
- "no": 0.5384
3450
  },
3451
- "latency_s": 0.068
3452
  },
3453
  {
3454
  "id": "hard-sol-c-judge_hard-05",
@@ -3459,10 +3459,10 @@
3459
  "valid": true,
3460
  "error": null,
3461
  "probs": {
3462
- "yes": 0.5764,
3463
- "no": 0.4236
3464
  },
3465
- "latency_s": 0.068
3466
  },
3467
  {
3468
  "id": "hard-sol-c-judge_hard-07",
@@ -3473,10 +3473,10 @@
3473
  "valid": true,
3474
  "error": null,
3475
  "probs": {
3476
- "yes": 0.2688,
3477
- "no": 0.7312
3478
  },
3479
- "latency_s": 0.068
3480
  },
3481
  {
3482
  "id": "hard-sol-c-judge_hard-08",
@@ -3487,10 +3487,10 @@
3487
  "valid": true,
3488
  "error": null,
3489
  "probs": {
3490
- "yes": 0.9389,
3491
- "no": 0.0611
3492
  },
3493
- "latency_s": 0.068
3494
  },
3495
  {
3496
  "id": "hard-sol-c-judge_hard-09",
@@ -3501,10 +3501,10 @@
3501
  "valid": true,
3502
  "error": null,
3503
  "probs": {
3504
- "yes": 0.9127,
3505
- "no": 0.0873
3506
  },
3507
- "latency_s": 0.068
3508
  },
3509
  {
3510
  "id": "hard-sol-c-judge_hard-10",
@@ -3515,10 +3515,10 @@
3515
  "valid": true,
3516
  "error": null,
3517
  "probs": {
3518
- "yes": 0.7603,
3519
- "no": 0.2397
3520
  },
3521
- "latency_s": 0.068
3522
  },
3523
  {
3524
  "id": "hard-sol-c-judge_hard-11",
@@ -3529,24 +3529,24 @@
3529
  "valid": true,
3530
  "error": null,
3531
  "probs": {
3532
- "yes": 0.0194,
3533
- "no": 0.9806
3534
  },
3535
- "latency_s": 0.068
3536
  },
3537
  {
3538
  "id": "hard-sol-c-judge_hard-13",
3539
  "tier": "hard",
3540
  "expected": "no",
3541
- "predicted": "yes",
3542
- "correct": false,
3543
  "valid": true,
3544
  "error": null,
3545
  "probs": {
3546
- "yes": 0.5192,
3547
- "no": 0.4808
3548
  },
3549
- "latency_s": 0.068
3550
  },
3551
  {
3552
  "id": "hard-sol-c-judge_hard-15",
@@ -3557,10 +3557,10 @@
3557
  "valid": true,
3558
  "error": null,
3559
  "probs": {
3560
- "yes": 0.9606,
3561
- "no": 0.0394
3562
  },
3563
- "latency_s": 0.068
3564
  },
3565
  {
3566
  "id": "hard-sol-c-multi_hop-07",
@@ -3571,10 +3571,10 @@
3571
  "valid": true,
3572
  "error": null,
3573
  "probs": {
3574
- "partial_release": 0.0027,
3575
- "programme_review": 0.0108,
3576
- "reject_draw": 0.002,
3577
- "release_draw": 0.9844
3578
  },
3579
  "latency_s": 0.084
3580
  },
@@ -3587,12 +3587,12 @@
3587
  "valid": true,
3588
  "error": null,
3589
  "probs": {
3590
- "bus_substitution": 0.0289,
3591
- "call_fen_cross": 0.0444,
3592
- "pass_fen_cross": 0.8508,
3593
- "terminate_oakmere": 0.0759
3594
  },
3595
- "latency_s": 0.083
3596
  },
3597
  {
3598
  "id": "hard-sol-c-multi_hop-11",
@@ -3603,12 +3603,12 @@
3603
  "valid": true,
3604
  "error": null,
3605
  "probs": {
3606
- "adjuster_referral": 0.0013,
3607
- "coverage_confirmed": 0.1088,
3608
- "coverage_excluded": 0.8832,
3609
- "documentation_needed": 0.0067
3610
  },
3611
- "latency_s": 0.086
3612
  },
3613
  {
3614
  "id": "hard-sol-c-multi_hop-12",
@@ -3619,12 +3619,12 @@
3619
  "valid": true,
3620
  "error": null,
3621
  "probs": {
3622
- "conservation_hold": 0.9577,
3623
- "deny_request": 0.0138,
3624
- "digital_copy_only": 0.0223,
3625
- "supervised_delivery": 0.0062
3626
  },
3627
- "latency_s": 0.069
3628
  },
3629
  {
3630
  "id": "hard-sol-c-multi_hop-13",
@@ -3637,10 +3637,10 @@
3637
  "probs": {
3638
  "engineering_review": 0.0,
3639
  "normal_release": 0.9999,
3640
- "rework": 0.0001,
3641
- "scrap_lot": 0.0001
3642
  },
3643
- "latency_s": 0.086
3644
  }
3645
  ]
3646
  }
 
5
  "n": 48,
6
  "accuracy": 1.0,
7
  "invalid": 0,
8
+ "p50_s": 0.069
9
  },
10
  "standard": {
11
  "n": 72,
12
+ "accuracy": 0.8611,
13
  "invalid": 0,
14
+ "p50_s": 0.069
15
  },
16
  "hard": {
17
  "n": 111,
18
+ "accuracy": 0.6937,
19
  "invalid": 0,
20
+ "p50_s": 0.073
21
  }
22
  },
23
  "rows": [
 
34
  "cancel_order": 0.0,
35
  "change_address": 0.0,
36
  "report_damage": 0.0,
37
+ "track_order": 1.0
38
  },
39
+ "latency_s": 1.987
40
  },
41
  {
42
  "id": "easy-intent-01",
 
53
  "report_damage": 0.0,
54
  "track_order": 0.0
55
  },
56
+ "latency_s": 0.071
57
  },
58
  {
59
  "id": "easy-intent-02",
 
65
  "error": null,
66
  "probs": {
67
  "billing_question": 0.0,
68
+ "cancel_order": 0.0,
69
  "change_address": 0.9998,
70
+ "report_damage": 0.0,
71
  "track_order": 0.0
72
  },
73
+ "latency_s": 0.067
74
  },
75
  {
76
  "id": "easy-intent-03",
 
82
  "error": null,
83
  "probs": {
84
  "billing_question": 0.0,
85
+ "cancel_order": 0.0005,
86
  "change_address": 0.0001,
87
+ "report_damage": 0.9993,
88
  "track_order": 0.0001
89
  },
90
+ "latency_s": 0.067
91
  },
92
  {
93
  "id": "easy-intent-04",
 
98
  "valid": true,
99
  "error": null,
100
  "probs": {
101
+ "billing_question": 0.9991,
102
+ "cancel_order": 0.0003,
103
+ "change_address": 0.0002,
104
+ "report_damage": 0.0001,
105
  "track_order": 0.0002
106
  },
107
+ "latency_s": 0.079
108
  },
109
  {
110
  "id": "easy-intent-05",
 
118
  "billing_question": 0.0,
119
  "cancel_order": 0.0001,
120
  "change_address": 0.0001,
121
+ "report_damage": 0.0,
122
+ "track_order": 0.9998
123
  },
124
+ "latency_s": 0.068
125
  },
126
  {
127
  "id": "easy-intent-06",
 
134
  "probs": {
135
  "play_music": 0.0,
136
  "send_message": 0.0001,
137
+ "set_alarm": 0.9998,
138
  "turn_off_lights": 0.0001,
139
  "weather": 0.0001
140
  },
141
+ "latency_s": 0.068
142
  },
143
  {
144
  "id": "easy-intent-07",
 
149
  "valid": true,
150
  "error": null,
151
  "probs": {
152
+ "play_music": 0.9996,
153
+ "send_message": 0.0001,
154
+ "set_alarm": 0.0001,
155
  "turn_off_lights": 0.0001,
156
  "weather": 0.0001
157
  },
158
+ "latency_s": 0.068
159
  },
160
  {
161
  "id": "easy-intent-08",
 
172
  "turn_off_lights": 0.0,
173
  "weather": 1.0
174
  },
175
+ "latency_s": 0.069
176
  },
177
  {
178
  "id": "easy-intent-09",
 
185
  "probs": {
186
  "play_music": 0.0001,
187
  "send_message": 0.9993,
188
+ "set_alarm": 0.0001,
189
+ "turn_off_lights": 0.0001,
190
+ "weather": 0.0003
191
  },
192
+ "latency_s": 0.068
193
  },
194
  {
195
  "id": "easy-intent-10",
 
204
  "send_message": 0.0001,
205
  "set_alarm": 0.0002,
206
  "turn_off_lights": 0.9994,
207
+ "weather": 0.0002
208
  },
209
+ "latency_s": 0.068
210
  },
211
  {
212
  "id": "easy-intent-11",
 
218
  "error": null,
219
  "probs": {
220
  "play_music": 0.0,
221
+ "send_message": 0.0,
222
+ "set_alarm": 0.9999,
223
+ "turn_off_lights": 0.0,
224
+ "weather": 0.0
225
  },
226
+ "latency_s": 0.069
227
  },
228
  {
229
  "id": "easy-fact-00",
 
234
  "valid": true,
235
  "error": null,
236
  "probs": {
237
+ "yes": 0.9834,
238
+ "no": 0.0166
239
  },
240
+ "latency_s": 0.068
241
  },
242
  {
243
  "id": "easy-fact-01",
 
248
  "valid": true,
249
  "error": null,
250
  "probs": {
251
+ "yes": 0.0033,
252
+ "no": 0.9967
253
  },
254
+ "latency_s": 0.069
255
  },
256
  {
257
  "id": "easy-fact-02",
 
262
  "valid": true,
263
  "error": null,
264
  "probs": {
265
+ "yes": 0.9857,
266
+ "no": 0.0143
267
  },
268
+ "latency_s": 0.068
269
  },
270
  {
271
  "id": "easy-fact-03",
 
276
  "valid": true,
277
  "error": null,
278
  "probs": {
279
+ "yes": 0.0008,
280
+ "no": 0.9992
281
  },
282
+ "latency_s": 0.068
283
  },
284
  {
285
  "id": "easy-fact-04",
 
290
  "valid": true,
291
  "error": null,
292
  "probs": {
293
+ "yes": 0.9707,
294
+ "no": 0.0293
295
  },
296
+ "latency_s": 0.067
297
  },
298
  {
299
  "id": "easy-fact-05",
 
304
  "valid": true,
305
  "error": null,
306
  "probs": {
307
+ "yes": 0.0067,
308
+ "no": 0.9933
309
  },
310
+ "latency_s": 0.067
311
  },
312
  {
313
  "id": "easy-fact-06",
 
318
  "valid": true,
319
  "error": null,
320
  "probs": {
321
+ "yes": 0.956,
322
+ "no": 0.044
323
  },
324
+ "latency_s": 0.068
325
  },
326
  {
327
  "id": "easy-fact-07",
 
332
  "valid": true,
333
  "error": null,
334
  "probs": {
335
+ "yes": 0.0028,
336
+ "no": 0.9972
337
  },
338
+ "latency_s": 0.069
339
  },
340
  {
341
  "id": "easy-fact-08",
 
346
  "valid": true,
347
  "error": null,
348
  "probs": {
349
+ "yes": 0.9852,
350
+ "no": 0.0148
351
  },
352
+ "latency_s": 0.075
353
  },
354
  {
355
  "id": "easy-fact-09",
 
360
  "valid": true,
361
  "error": null,
362
  "probs": {
363
+ "yes": 0.0022,
364
+ "no": 0.9978
365
  },
366
+ "latency_s": 0.072
367
  },
368
  {
369
  "id": "easy-fact-10",
 
374
  "valid": true,
375
  "error": null,
376
  "probs": {
377
+ "yes": 0.9158,
378
+ "no": 0.0842
379
  },
380
+ "latency_s": 0.073
381
  },
382
  {
383
  "id": "easy-fact-11",
 
388
  "valid": true,
389
  "error": null,
390
  "probs": {
391
+ "yes": 0.0028,
392
+ "no": 0.9972
393
  },
394
+ "latency_s": 0.07
395
  },
396
  {
397
  "id": "easy-extraction-00",
 
402
  "valid": true,
403
  "error": null,
404
  "probs": {
405
+ "bank_transfer": 0.0001,
406
  "cash": 0.0,
407
  "credit_card": 0.0001,
408
  "paypal": 0.9998
409
  },
410
+ "latency_s": 0.069
411
  },
412
  {
413
  "id": "easy-extraction-01",
 
420
  "probs": {
421
  "bank_transfer": 0.0001,
422
  "cash": 0.9996,
423
+ "credit_card": 0.0002,
424
  "paypal": 0.0001
425
  },
426
+ "latency_s": 0.069
427
  },
428
  {
429
  "id": "easy-extraction-02",
 
434
  "valid": true,
435
  "error": null,
436
  "probs": {
437
+ "bank_transfer": 0.9998,
438
+ "cash": 0.0,
439
  "credit_card": 0.0001,
440
+ "paypal": 0.0
441
  },
442
+ "latency_s": 0.067
443
  },
444
  {
445
  "id": "easy-extraction-03",
 
450
  "valid": true,
451
  "error": null,
452
  "probs": {
453
+ "high": 0.9749,
454
+ "low": 0.0009,
455
+ "medium": 0.0027,
456
+ "urgent": 0.0215
457
  },
458
+ "latency_s": 0.071
459
  },
460
  {
461
  "id": "easy-extraction-04",
 
466
  "valid": true,
467
  "error": null,
468
  "probs": {
469
+ "high": 0.0018,
470
+ "low": 0.9962,
471
+ "medium": 0.0017,
472
+ "urgent": 0.0004
473
  },
474
+ "latency_s": 0.068
475
  },
476
  {
477
  "id": "easy-extraction-05",
 
482
  "valid": true,
483
  "error": null,
484
  "probs": {
485
+ "high": 0.0014,
486
  "low": 0.0002,
487
  "medium": 0.0006,
488
+ "urgent": 0.9978
489
  },
490
+ "latency_s": 0.069
491
  },
492
  {
493
  "id": "easy-extraction-06",
 
498
  "valid": true,
499
  "error": null,
500
  "probs": {
501
+ "L": 0.0009,
502
+ "M": 0.9987,
503
  "S": 0.0002,
504
+ "XL": 0.0002
505
  },
506
+ "latency_s": 0.07
507
  },
508
  {
509
  "id": "easy-extraction-07",
 
514
  "valid": true,
515
  "error": null,
516
  "probs": {
517
+ "L": 0.0003,
518
  "M": 0.0001,
519
  "S": 0.0001,
520
+ "XL": 0.9995
521
  },
522
+ "latency_s": 0.071
523
  },
524
  {
525
  "id": "easy-extraction-08",
 
530
  "valid": true,
531
  "error": null,
532
  "probs": {
533
+ "friday": 0.0013,
534
+ "monday": 0.001,
535
+ "thursday": 0.9964,
536
+ "tuesday": 0.0005,
537
+ "wednesday": 0.0007
538
  },
539
+ "latency_s": 0.07
540
  },
541
  {
542
  "id": "easy-extraction-09",
 
547
  "valid": true,
548
  "error": null,
549
  "probs": {
550
+ "friday": 0.0155,
551
+ "monday": 0.965,
552
+ "thursday": 0.0062,
553
+ "tuesday": 0.0053,
554
+ "wednesday": 0.0081
555
  },
556
+ "latency_s": 0.071
557
  },
558
  {
559
  "id": "easy-extraction-10",
 
564
  "valid": true,
565
  "error": null,
566
  "probs": {
567
+ "friday": 0.996,
568
+ "monday": 0.0007,
569
+ "thursday": 0.0011,
570
+ "tuesday": 0.0006,
571
+ "wednesday": 0.0015
572
  },
573
+ "latency_s": 0.074
574
  },
575
  {
576
  "id": "easy-extraction-11",
 
583
  "probs": {
584
  "bank_transfer": 0.0001,
585
  "cash": 0.0001,
586
+ "credit_card": 0.9998,
587
  "paypal": 0.0001
588
  },
589
+ "latency_s": 0.07
590
  },
591
  {
592
  "id": "easy-tool_selection-00",
 
599
  "probs": {
600
  "convert_currency": 0.0,
601
  "create_calendar_event": 0.0,
602
+ "get_weather": 0.9999,
603
+ "send_email": 0.0,
604
  "translate_text": 0.0
605
  },
606
+ "latency_s": 0.069
607
  },
608
  {
609
  "id": "easy-tool_selection-01",
 
620
  "send_email": 0.9998,
621
  "translate_text": 0.0001
622
  },
623
+ "latency_s": 0.07
624
  },
625
  {
626
  "id": "easy-tool_selection-02",
 
637
  "send_email": 0.0,
638
  "translate_text": 0.0
639
  },
640
+ "latency_s": 0.068
641
  },
642
  {
643
  "id": "easy-tool_selection-03",
 
648
  "valid": true,
649
  "error": null,
650
  "probs": {
651
+ "convert_currency": 0.9998,
652
+ "create_calendar_event": 0.0,
653
+ "get_weather": 0.0,
654
+ "send_email": 0.0,
655
  "translate_text": 0.0001
656
  },
657
+ "latency_s": 0.068
658
  },
659
  {
660
  "id": "easy-tool_selection-04",
 
671
  "send_email": 0.0,
672
  "translate_text": 0.9999
673
  },
674
+ "latency_s": 0.068
675
  },
676
  {
677
  "id": "easy-tool_selection-05",
 
684
  "probs": {
685
  "convert_currency": 0.0,
686
  "create_calendar_event": 0.9998,
687
+ "get_weather": 0.0,
688
  "send_email": 0.0,
689
+ "translate_text": 0.0001
690
  },
691
+ "latency_s": 0.068
692
  },
693
  {
694
  "id": "easy-tool_selection-06",
 
705
  "search_flights": 0.9999,
706
  "set_reminder": 0.0001
707
  },
708
+ "latency_s": 0.072
709
  },
710
  {
711
  "id": "easy-tool_selection-07",
 
716
  "valid": true,
717
  "error": null,
718
  "probs": {
719
+ "book_restaurant": 0.9995,
720
+ "call_taxi": 0.0,
721
  "get_stock_price": 0.0001,
722
+ "search_flights": 0.0,
723
  "set_reminder": 0.0003
724
  },
725
+ "latency_s": 0.069
726
  },
727
  {
728
  "id": "easy-tool_selection-08",
 
737
  "call_taxi": 0.0,
738
  "get_stock_price": 0.0001,
739
  "search_flights": 0.0,
740
+ "set_reminder": 0.9999
741
  },
742
+ "latency_s": 0.069
743
  },
744
  {
745
  "id": "easy-tool_selection-09",
 
751
  "error": null,
752
  "probs": {
753
  "book_restaurant": 0.0,
754
+ "call_taxi": 0.0,
755
+ "get_stock_price": 0.9999,
756
  "search_flights": 0.0,
757
  "set_reminder": 0.0001
758
  },
759
+ "latency_s": 0.069
760
  },
761
  {
762
  "id": "easy-tool_selection-10",
 
771
  "call_taxi": 0.9995,
772
  "get_stock_price": 0.0001,
773
  "search_flights": 0.0001,
774
+ "set_reminder": 0.0003
775
  },
776
+ "latency_s": 0.067
777
  },
778
  {
779
  "id": "easy-tool_selection-11",
 
790
  "search_flights": 0.0,
791
  "set_reminder": 0.9999
792
  },
793
+ "latency_s": 0.067
794
  },
795
  {
796
  "id": "original-policy-01-0",
 
801
  "valid": true,
802
  "error": null,
803
  "probs": {
804
+ "yes": 0.011,
805
+ "no": 0.989
806
  },
807
+ "latency_s": 0.068
808
  },
809
  {
810
  "id": "original-policy-01-1",
 
815
  "valid": true,
816
  "error": null,
817
  "probs": {
818
+ "yes": 0.0138,
819
+ "no": 0.9862
820
  },
821
+ "latency_s": 0.068
822
  },
823
  {
824
  "id": "original-policy-02-0",
 
829
  "valid": true,
830
  "error": null,
831
  "probs": {
832
+ "yes": 0.9432,
833
+ "no": 0.0568
834
  },
835
+ "latency_s": 0.068
836
  },
837
  {
838
  "id": "original-policy-02-1",
 
843
  "valid": true,
844
  "error": null,
845
  "probs": {
846
+ "yes": 0.194,
847
+ "no": 0.806
848
  },
849
+ "latency_s": 0.068
850
  },
851
  {
852
  "id": "original-policy-03-0",
853
  "tier": "standard",
854
  "expected": "yes",
855
+ "predicted": "no",
856
+ "correct": false,
857
  "valid": true,
858
  "error": null,
859
  "probs": {
860
+ "yes": 0.4616,
861
+ "no": 0.5384
862
  },
863
+ "latency_s": 0.068
864
  },
865
  {
866
  "id": "original-policy-03-1",
867
  "tier": "standard",
868
  "expected": "yes",
869
+ "predicted": "no",
870
+ "correct": false,
871
  "valid": true,
872
  "error": null,
873
  "probs": {
874
+ "yes": 0.2467,
875
+ "no": 0.7533
876
  },
877
+ "latency_s": 0.068
878
  },
879
  {
880
  "id": "original-policy-04-0",
 
885
  "valid": true,
886
  "error": null,
887
  "probs": {
888
+ "yes": 0.0304,
889
+ "no": 0.9696
890
  },
891
+ "latency_s": 0.068
892
  },
893
  {
894
  "id": "original-policy-04-1",
 
899
  "valid": true,
900
  "error": null,
901
  "probs": {
902
+ "yes": 0.0057,
903
+ "no": 0.9943
904
  },
905
+ "latency_s": 0.068
906
  },
907
  {
908
  "id": "original-policy-05-0",
 
913
  "valid": true,
914
  "error": null,
915
  "probs": {
916
+ "yes": 0.038,
917
+ "no": 0.962
918
  },
919
+ "latency_s": 0.071
920
  },
921
  {
922
  "id": "original-policy-05-1",
 
927
  "valid": true,
928
  "error": null,
929
  "probs": {
930
+ "yes": 0.038,
931
+ "no": 0.962
932
  },
933
+ "latency_s": 0.068
934
  },
935
  {
936
  "id": "original-policy-06-0",
 
941
  "valid": true,
942
  "error": null,
943
  "probs": {
944
+ "yes": 0.9031,
945
+ "no": 0.0969
946
  },
947
+ "latency_s": 0.068
948
  },
949
  {
950
  "id": "original-policy-06-1",
 
955
  "valid": true,
956
  "error": null,
957
  "probs": {
958
+ "yes": 0.6917,
959
+ "no": 0.3083
960
  },
961
+ "latency_s": 0.068
962
  },
963
  {
964
  "id": "original-intent-01-0",
 
969
  "valid": true,
970
  "error": null,
971
  "probs": {
972
+ "cancel": 0.0024,
973
+ "change_address": 0.0005,
974
+ "other": 0.0103,
975
+ "refund": 0.9864,
976
+ "status": 0.0005
977
  },
978
+ "latency_s": 0.07
979
  },
980
  {
981
  "id": "original-intent-01-1",
 
986
  "valid": true,
987
  "error": null,
988
  "probs": {
989
+ "cancel": 0.0393,
990
  "change_address": 0.0012,
991
+ "other": 0.0243,
992
+ "refund": 0.9342,
993
+ "status": 0.001
994
  },
995
+ "latency_s": 0.068
996
  },
997
  {
998
  "id": "original-intent-02-0",
 
1003
  "valid": true,
1004
  "error": null,
1005
  "probs": {
1006
+ "cancel": 0.001,
1007
+ "change_address": 0.0021,
1008
+ "other": 0.2534,
1009
+ "refund": 0.0018,
1010
+ "status": 0.7416
1011
  },
1012
+ "latency_s": 0.068
1013
  },
1014
  {
1015
  "id": "original-intent-02-1",
 
1022
  "probs": {
1023
  "cancel": 0.0001,
1024
  "change_address": 0.0001,
1025
+ "other": 0.0015,
1026
  "refund": 0.0001,
1027
+ "status": 0.9983
1028
  },
1029
+ "latency_s": 0.069
1030
  },
1031
  {
1032
  "id": "original-intent-03-0",
 
1037
  "valid": true,
1038
  "error": null,
1039
  "probs": {
1040
+ "cancel": 0.001,
1041
+ "change_address": 0.9593,
1042
+ "other": 0.0363,
1043
+ "refund": 0.0021,
1044
+ "status": 0.0014
1045
  },
1046
+ "latency_s": 0.068
1047
  },
1048
  {
1049
  "id": "original-intent-03-1",
 
1055
  "error": null,
1056
  "probs": {
1057
  "cancel": 0.0,
1058
+ "change_address": 0.9991,
1059
+ "other": 0.0007,
1060
  "refund": 0.0001,
1061
+ "status": 0.0001
1062
  },
1063
+ "latency_s": 0.068
1064
  },
1065
  {
1066
  "id": "original-intent-04-0",
 
1071
  "valid": true,
1072
  "error": null,
1073
  "probs": {
1074
+ "cancel": 0.992,
1075
+ "change_address": 0.0003,
1076
+ "other": 0.0067,
1077
+ "refund": 0.0008,
1078
+ "status": 0.0002
1079
  },
1080
+ "latency_s": 0.068
1081
  },
1082
  {
1083
  "id": "original-intent-04-1",
 
1088
  "valid": true,
1089
  "error": null,
1090
  "probs": {
1091
+ "cancel": 0.9948,
1092
+ "change_address": 0.0002,
1093
+ "other": 0.0046,
1094
+ "refund": 0.0002,
1095
  "status": 0.0002
1096
  },
1097
+ "latency_s": 0.067
1098
  },
1099
  {
1100
  "id": "original-intent-05-0",
 
1105
  "valid": true,
1106
  "error": null,
1107
  "probs": {
1108
+ "cancel": 0.0017,
1109
+ "change_address": 0.0019,
1110
+ "other": 0.8804,
1111
+ "refund": 0.1144,
1112
+ "status": 0.0016
1113
  },
1114
+ "latency_s": 0.068
1115
  },
1116
  {
1117
  "id": "original-intent-05-1",
 
1122
  "valid": true,
1123
  "error": null,
1124
  "probs": {
1125
+ "cancel": 0.0046,
1126
+ "change_address": 0.0024,
1127
+ "other": 0.7179,
1128
+ "refund": 0.2731,
1129
+ "status": 0.002
1130
  },
1131
+ "latency_s": 0.068
1132
  },
1133
  {
1134
  "id": "original-intent-06-0",
 
1141
  "probs": {
1142
  "cancel": 0.0001,
1143
  "change_address": 0.0001,
1144
+ "other": 0.0023,
1145
  "refund": 0.0001,
1146
+ "status": 0.9974
1147
  },
1148
+ "latency_s": 0.068
1149
  },
1150
  {
1151
  "id": "original-intent-06-1",
 
1156
  "valid": true,
1157
  "error": null,
1158
  "probs": {
1159
+ "cancel": 0.0,
1160
  "change_address": 0.0,
1161
+ "other": 0.001,
1162
+ "refund": 0.0,
1163
+ "status": 0.9989
1164
  },
1165
+ "latency_s": 0.071
1166
  },
1167
  {
1168
  "id": "original-ordinal-01-0",
 
1173
  "valid": true,
1174
  "error": null,
1175
  "probs": {
1176
+ "0": 0.9415,
1177
+ "1": 0.0569,
1178
+ "2": 0.001,
1179
+ "3": 0.0005
1180
  },
1181
+ "latency_s": 0.075
1182
  },
1183
  {
1184
  "id": "original-ordinal-01-1",
 
1189
  "valid": true,
1190
  "error": null,
1191
  "probs": {
1192
+ "0": 0.8823,
1193
+ "1": 0.1155,
1194
+ "2": 0.0017,
1195
+ "3": 0.0005
1196
  },
1197
+ "latency_s": 0.069
1198
  },
1199
  {
1200
  "id": "original-ordinal-02-0",
 
1205
  "valid": true,
1206
  "error": null,
1207
  "probs": {
1208
+ "0": 0.0105,
1209
+ "1": 0.9441,
1210
+ "2": 0.0438,
1211
+ "3": 0.0016
1212
  },
1213
+ "latency_s": 0.069
1214
  },
1215
  {
1216
  "id": "original-ordinal-02-1",
 
1221
  "valid": true,
1222
  "error": null,
1223
  "probs": {
1224
+ "0": 0.009,
1225
+ "1": 0.9873,
1226
+ "2": 0.0025,
1227
+ "3": 0.0012
1228
  },
1229
+ "latency_s": 0.069
1230
  },
1231
  {
1232
  "id": "original-ordinal-03-0",
 
1237
  "valid": true,
1238
  "error": null,
1239
  "probs": {
1240
+ "0": 0.0128,
1241
+ "1": 0.0342,
1242
+ "2": 0.9525,
1243
+ "3": 0.0005
1244
  },
1245
+ "latency_s": 0.069
1246
  },
1247
  {
1248
  "id": "original-ordinal-03-1",
 
1253
  "valid": true,
1254
  "error": null,
1255
  "probs": {
1256
+ "0": 0.0102,
1257
+ "1": 0.0555,
1258
+ "2": 0.9326,
1259
+ "3": 0.0017
1260
  },
1261
+ "latency_s": 0.069
1262
  },
1263
  {
1264
  "id": "original-ordinal-04-0",
 
1269
  "valid": true,
1270
  "error": null,
1271
  "probs": {
1272
+ "0": 0.0021,
1273
+ "1": 0.001,
1274
+ "2": 0.0028,
1275
+ "3": 0.994
1276
  },
1277
+ "latency_s": 0.069
1278
  },
1279
  {
1280
  "id": "original-ordinal-04-1",
 
1285
  "valid": true,
1286
  "error": null,
1287
  "probs": {
1288
+ "0": 0.0029,
1289
+ "1": 0.0022,
1290
+ "2": 0.0248,
1291
+ "3": 0.9702
1292
  },
1293
+ "latency_s": 0.069
1294
  },
1295
  {
1296
  "id": "original-ordinal-05-0",
 
1301
  "valid": true,
1302
  "error": null,
1303
  "probs": {
1304
+ "0": 0.9899,
1305
+ "1": 0.0085,
1306
+ "2": 0.0011,
1307
+ "3": 0.0004
1308
  },
1309
+ "latency_s": 0.069
1310
  },
1311
  {
1312
  "id": "original-ordinal-05-1",
 
1317
  "valid": true,
1318
  "error": null,
1319
  "probs": {
1320
+ "0": 0.7309,
1321
+ "1": 0.1318,
1322
+ "2": 0.1179,
1323
+ "3": 0.0194
1324
  },
1325
+ "latency_s": 0.069
1326
  },
1327
  {
1328
  "id": "original-ordinal-06-0",
 
1333
  "valid": true,
1334
  "error": null,
1335
  "probs": {
1336
+ "0": 0.0108,
1337
+ "1": 0.0575,
1338
+ "2": 0.9315,
1339
+ "3": 0.0002
1340
  },
1341
+ "latency_s": 0.069
1342
  },
1343
  {
1344
  "id": "original-ordinal-06-1",
 
1349
  "valid": true,
1350
  "error": null,
1351
  "probs": {
1352
+ "0": 0.0022,
1353
+ "1": 0.0409,
1354
+ "2": 0.9566,
1355
+ "3": 0.0003
1356
  },
1357
+ "latency_s": 0.072
1358
  },
1359
  {
1360
  "id": "original-extraction-01-0",
 
1365
  "valid": true,
1366
  "error": null,
1367
  "probs": {
1368
+ "courier": 0.0037,
1369
+ "pickup": 0.9829,
1370
+ "post": 0.0014,
1371
+ "unknown": 0.012
1372
  },
1373
+ "latency_s": 0.072
1374
  },
1375
  {
1376
  "id": "original-extraction-01-1",
 
1381
  "valid": true,
1382
  "error": null,
1383
  "probs": {
1384
+ "courier": 0.0009,
1385
+ "pickup": 0.989,
1386
+ "post": 0.0009,
1387
+ "unknown": 0.0093
1388
  },
1389
+ "latency_s": 0.074
1390
  },
1391
  {
1392
  "id": "original-extraction-02-0",
 
1397
  "valid": true,
1398
  "error": null,
1399
  "probs": {
1400
+ "courier": 0.0252,
1401
+ "pickup": 0.0069,
1402
+ "post": 0.3692,
1403
+ "unknown": 0.5986
1404
  },
1405
+ "latency_s": 0.071
1406
  },
1407
  {
1408
  "id": "original-extraction-02-1",
 
1413
  "valid": true,
1414
  "error": null,
1415
  "probs": {
1416
+ "courier": 0.0053,
1417
+ "pickup": 0.0016,
1418
+ "post": 0.0265,
1419
+ "unknown": 0.9666
1420
  },
1421
+ "latency_s": 0.071
1422
  },
1423
  {
1424
  "id": "original-extraction-03-0",
 
1429
  "valid": true,
1430
  "error": null,
1431
  "probs": {
1432
+ "courier": 0.9056,
1433
+ "pickup": 0.0161,
1434
+ "post": 0.013,
1435
+ "unknown": 0.0652
1436
  },
1437
+ "latency_s": 0.068
1438
  },
1439
  {
1440
  "id": "original-extraction-03-1",
 
1445
  "valid": true,
1446
  "error": null,
1447
  "probs": {
1448
+ "courier": 0.9996,
1449
+ "pickup": 0.0001,
1450
  "post": 0.0002,
1451
  "unknown": 0.0001
1452
  },
1453
+ "latency_s": 0.067
1454
  },
1455
  {
1456
  "id": "original-extraction-04-0",
 
1461
  "valid": true,
1462
  "error": null,
1463
  "probs": {
1464
+ "courier": 0.0003,
1465
  "pickup": 0.0004,
1466
+ "post": 0.9978,
1467
+ "unknown": 0.0015
1468
  },
1469
+ "latency_s": 0.068
1470
  },
1471
  {
1472
  "id": "original-extraction-04-1",
 
1477
  "valid": true,
1478
  "error": null,
1479
  "probs": {
1480
+ "courier": 0.0006,
1481
+ "pickup": 0.0004,
1482
+ "post": 0.9983,
1483
+ "unknown": 0.0007
1484
  },
1485
+ "latency_s": 0.068
1486
  },
1487
  {
1488
  "id": "original-extraction-05-0",
 
1493
  "valid": true,
1494
  "error": null,
1495
  "probs": {
1496
+ "courier": 0.001,
1497
+ "pickup": 0.0049,
1498
+ "post": 0.0019,
1499
+ "unknown": 0.9923
1500
  },
1501
+ "latency_s": 0.069
1502
  },
1503
  {
1504
  "id": "original-extraction-05-1",
 
1509
  "valid": true,
1510
  "error": null,
1511
  "probs": {
1512
+ "courier": 0.8517,
1513
+ "pickup": 0.004,
1514
+ "post": 0.0072,
1515
+ "unknown": 0.1372
1516
  },
1517
+ "latency_s": 0.068
1518
  },
1519
  {
1520
  "id": "original-extraction-06-0",
 
1525
  "valid": true,
1526
  "error": null,
1527
  "probs": {
1528
+ "courier": 0.9995,
1529
+ "pickup": 0.0001,
1530
+ "post": 0.0002,
1531
+ "unknown": 0.0002
1532
  },
1533
+ "latency_s": 0.068
1534
  },
1535
  {
1536
  "id": "original-extraction-06-1",
 
1541
  "valid": true,
1542
  "error": null,
1543
  "probs": {
1544
+ "courier": 0.9994,
1545
+ "pickup": 0.0001,
1546
+ "post": 0.0002,
1547
+ "unknown": 0.0003
1548
  },
1549
+ "latency_s": 0.07
1550
  },
1551
  {
1552
  "id": "original-adequacy-01-0",
 
1557
  "valid": true,
1558
  "error": null,
1559
  "probs": {
1560
+ "yes": 0.8289,
1561
+ "no": 0.1711
1562
  },
1563
+ "latency_s": 0.069
1564
  },
1565
  {
1566
  "id": "original-adequacy-01-1",
 
1571
  "valid": true,
1572
  "error": null,
1573
  "probs": {
1574
+ "yes": 0.7158,
1575
+ "no": 0.2842
1576
  },
1577
+ "latency_s": 0.069
1578
  },
1579
  {
1580
  "id": "original-adequacy-02-0",
 
1585
  "valid": true,
1586
  "error": null,
1587
  "probs": {
1588
+ "yes": 0.0327,
1589
+ "no": 0.9673
1590
  },
1591
+ "latency_s": 0.07
1592
  },
1593
  {
1594
  "id": "original-adequacy-02-1",
 
1599
  "valid": true,
1600
  "error": null,
1601
  "probs": {
1602
+ "yes": 0.0424,
1603
+ "no": 0.9576
1604
  },
1605
+ "latency_s": 0.07
1606
  },
1607
  {
1608
  "id": "original-adequacy-03-0",
 
1613
  "valid": true,
1614
  "error": null,
1615
  "probs": {
1616
+ "yes": 0.7603,
1617
+ "no": 0.2397
1618
  },
1619
+ "latency_s": 0.073
1620
  },
1621
  {
1622
  "id": "original-adequacy-03-1",
 
1627
  "valid": true,
1628
  "error": null,
1629
  "probs": {
1630
+ "yes": 0.6404,
1631
+ "no": 0.3596
1632
  },
1633
+ "latency_s": 0.07
1634
  },
1635
  {
1636
  "id": "original-adequacy-04-0",
 
1641
  "valid": true,
1642
  "error": null,
1643
  "probs": {
1644
+ "yes": 0.7807,
1645
+ "no": 0.2193
1646
  },
1647
+ "latency_s": 0.068
1648
  },
1649
  {
1650
  "id": "original-adequacy-04-1",
 
1655
  "valid": true,
1656
  "error": null,
1657
  "probs": {
1658
+ "yes": 0.5192,
1659
+ "no": 0.4808
1660
  },
1661
+ "latency_s": 0.069
1662
  },
1663
  {
1664
  "id": "original-adequacy-05-0",
1665
  "tier": "standard",
1666
  "expected": "no",
1667
+ "predicted": "no",
1668
+ "correct": true,
1669
  "valid": true,
1670
  "error": null,
1671
  "probs": {
1672
+ "yes": 0.405,
1673
+ "no": 0.595
1674
  },
1675
+ "latency_s": 0.069
1676
  },
1677
  {
1678
  "id": "original-adequacy-05-1",
 
1683
  "valid": true,
1684
  "error": null,
1685
  "probs": {
1686
+ "yes": 0.5669,
1687
+ "no": 0.4331
1688
  },
1689
+ "latency_s": 0.069
1690
  },
1691
  {
1692
  "id": "original-adequacy-06-0",
1693
  "tier": "standard",
1694
  "expected": "yes",
1695
+ "predicted": "no",
1696
+ "correct": false,
1697
  "valid": true,
1698
  "error": null,
1699
  "probs": {
1700
+ "yes": 0.4712,
1701
+ "no": 0.5288
1702
  },
1703
+ "latency_s": 0.072
1704
  },
1705
  {
1706
  "id": "original-adequacy-06-1",
1707
  "tier": "standard",
1708
  "expected": "yes",
1709
+ "predicted": "no",
1710
+ "correct": false,
1711
  "valid": true,
1712
  "error": null,
1713
  "probs": {
1714
+ "yes": 0.3866,
1715
+ "no": 0.6134
1716
  },
1717
+ "latency_s": 0.071
1718
  },
1719
  {
1720
  "id": "original-routing-01-0",
 
1725
  "valid": true,
1726
  "error": null,
1727
  "probs": {
1728
+ "coding": 0.0106,
1729
+ "coding_agent": 0.0005,
1730
+ "document": 0.0018,
1731
+ "general": 0.0224,
1732
+ "math": 0.9616,
1733
+ "tools": 0.0031
1734
  },
1735
+ "latency_s": 0.071
1736
  },
1737
  {
1738
  "id": "original-routing-01-1",
 
1743
  "valid": true,
1744
  "error": null,
1745
  "probs": {
1746
+ "coding": 0.0081,
1747
+ "coding_agent": 0.0004,
1748
+ "document": 0.002,
1749
+ "general": 0.0182,
1750
+ "math": 0.9693,
1751
+ "tools": 0.0019
1752
  },
1753
+ "latency_s": 0.072
1754
  },
1755
  {
1756
  "id": "original-routing-02-0",
 
1761
  "valid": true,
1762
  "error": null,
1763
  "probs": {
1764
+ "coding": 0.9827,
1765
+ "coding_agent": 0.0063,
1766
+ "document": 0.0013,
1767
+ "general": 0.0083,
1768
+ "math": 0.0007,
1769
+ "tools": 0.0007
1770
  },
1771
  "latency_s": 0.07
1772
  },
 
1779
  "valid": true,
1780
  "error": null,
1781
  "probs": {
1782
+ "coding": 0.9886,
1783
+ "coding_agent": 0.0027,
1784
+ "document": 0.0012,
1785
+ "general": 0.0064,
1786
+ "math": 0.0006,
1787
+ "tools": 0.0005
1788
  },
1789
  "latency_s": 0.069
1790
  },
 
1797
  "valid": true,
1798
  "error": null,
1799
  "probs": {
1800
+ "coding": 0.001,
1801
  "coding_agent": 0.9965,
1802
+ "document": 0.0004,
1803
  "general": 0.0011,
1804
  "math": 0.0002,
1805
+ "tools": 0.0008
1806
  },
1807
  "latency_s": 0.069
1808
  },
 
1817
  "probs": {
1818
  "coding": 0.0007,
1819
  "coding_agent": 0.9959,
1820
+ "document": 0.0005,
1821
+ "general": 0.0019,
1822
  "math": 0.0002,
1823
+ "tools": 0.0008
1824
  },
1825
+ "latency_s": 0.069
1826
  },
1827
  {
1828
  "id": "original-routing-04-0",
1829
  "tier": "standard",
1830
+ "expected": "document",
1831
+ "predicted": "document",
1832
+ "correct": true,
1833
  "valid": true,
1834
  "error": null,
1835
  "probs": {
1836
+ "coding": 0.004,
1837
  "coding_agent": 0.004,
1838
+ "document": 0.5233,
1839
+ "general": 0.4455,
1840
+ "math": 0.0024,
1841
+ "tools": 0.0209
1842
  },
1843
+ "latency_s": 0.073
1844
  },
1845
  {
1846
  "id": "original-routing-04-1",
 
1851
  "valid": true,
1852
  "error": null,
1853
  "probs": {
1854
+ "coding": 0.0022,
1855
+ "coding_agent": 0.0024,
1856
+ "document": 0.8222,
1857
+ "general": 0.1556,
1858
+ "math": 0.0013,
1859
+ "tools": 0.0163
1860
  },
1861
+ "latency_s": 0.068
1862
  },
1863
  {
1864
  "id": "original-routing-05-0",
 
1869
  "valid": true,
1870
  "error": null,
1871
  "probs": {
1872
+ "coding": 0.0029,
1873
  "coding_agent": 0.0014,
1874
+ "document": 0.0059,
1875
+ "general": 0.3284,
1876
+ "math": 0.0015,
1877
+ "tools": 0.6599
1878
  },
1879
+ "latency_s": 0.068
1880
  },
1881
  {
1882
  "id": "original-routing-05-1",
 
1887
  "valid": true,
1888
  "error": null,
1889
  "probs": {
1890
+ "coding": 0.0013,
1891
  "coding_agent": 0.0009,
1892
+ "document": 0.003,
1893
+ "general": 0.8796,
1894
+ "math": 0.0009,
1895
+ "tools": 0.1143
1896
  },
1897
+ "latency_s": 0.069
1898
  },
1899
  {
1900
  "id": "original-routing-06-0",
 
1905
  "valid": true,
1906
  "error": null,
1907
  "probs": {
1908
+ "coding": 0.0009,
1909
+ "coding_agent": 0.0007,
1910
+ "document": 0.006,
1911
+ "general": 0.9903,
1912
+ "math": 0.0008,
1913
  "tools": 0.0013
1914
  },
1915
+ "latency_s": 0.069
1916
  },
1917
  {
1918
  "id": "original-routing-06-1",
 
1923
  "valid": true,
1924
  "error": null,
1925
  "probs": {
1926
+ "coding": 0.0011,
1927
+ "coding_agent": 0.0007,
1928
+ "document": 0.0049,
1929
+ "general": 0.9907,
1930
+ "math": 0.0008,
1931
+ "tools": 0.0018
1932
  },
1933
+ "latency_s": 0.069
1934
  },
1935
  {
1936
  "id": "hard-opus-a-long_policy-01",
 
1941
  "valid": true,
1942
  "error": null,
1943
  "probs": {
1944
+ "deny_repeated_seepage": 0.0005,
1945
+ "deny_vacancy_exclusion": 0.0005,
1946
+ "pay_full_estimate_less_deductible": 0.0006,
1947
+ "pay_subject_to_10000_sublimit": 0.0345,
1948
+ "pay_subject_to_15000_sublimit": 0.9638
1949
  },
1950
+ "latency_s": 0.155
1951
  },
1952
  {
1953
  "id": "hard-opus-a-long_policy-04",
 
1958
  "valid": true,
1959
  "error": null,
1960
  "probs": {
1961
+ "budget_holder": 0.0014,
1962
+ "cfo": 0.0377,
1963
+ "department_head": 0.0136,
1964
+ "executive_committee": 0.001,
1965
+ "vp_and_finance_director": 0.9463
1966
  },
1967
+ "latency_s": 0.103
1968
  },
1969
  {
1970
  "id": "hard-opus-a-long_policy-08",
 
1975
  "valid": true,
1976
  "error": null,
1977
  "probs": {
1978
+ "full_repair_no_charge": 0.2848,
1979
+ "not_covered": 0.0037,
1980
+ "parts_covered_labour_charged": 0.7096,
1981
+ "pro_rata_replacement_credit": 0.0019
1982
  },
1983
+ "latency_s": 0.083
1984
  },
1985
  {
1986
  "id": "hard-opus-a-long_policy-09",
 
1991
  "valid": true,
1992
  "error": null,
1993
  "probs": {
1994
+ "license_exception_lvs": 0.0001,
1995
+ "license_required": 0.9997,
1996
+ "no_license_required": 0.0001,
1997
+ "shipment_prohibited": 0.0001
1998
  },
1999
+ "latency_s": 0.086
2000
  },
2001
  {
2002
  "id": "hard-opus-a-long_policy-11",
 
2007
  "valid": true,
2008
  "error": null,
2009
  "probs": {
2010
+ "yes": 0.0225,
2011
+ "no": 0.9775
2012
  },
2013
+ "latency_s": 0.082
2014
  },
2015
  {
2016
  "id": "hard-opus-a-long_policy-13",
2017
  "tier": "hard",
2018
  "expected": "yes",
2019
+ "predicted": "yes",
2020
+ "correct": true,
2021
  "valid": true,
2022
  "error": null,
2023
  "probs": {
2024
+ "yes": 0.746,
2025
+ "no": 0.254
2026
  },
2027
+ "latency_s": 0.084
2028
  },
2029
  {
2030
  "id": "hard-opus-a-long_policy-17",
 
2035
  "valid": true,
2036
  "error": null,
2037
  "probs": {
2038
+ "database_oncall": 0.9946,
2039
+ "no_page_ticket_only": 0.0004,
2040
+ "platform_oncall": 0.0039,
2041
+ "security_incident_response": 0.0002,
2042
  "vendor_support": 0.0008
2043
  },
2044
+ "latency_s": 0.085
2045
  },
2046
  {
2047
  "id": "hard-opus-a-long_policy-19",
 
2052
  "valid": true,
2053
  "error": null,
2054
  "probs": {
2055
+ "yes": 0.0016,
2056
+ "no": 0.9984
2057
  },
2058
+ "latency_s": 0.082
2059
  },
2060
  {
2061
  "id": "hard-opus-a-probability-03",
 
2066
  "valid": true,
2067
  "error": null,
2068
  "probs": {
2069
+ "yes": 0.2921,
2070
+ "no": 0.7079
2071
  },
2072
+ "latency_s": 0.072
2073
  },
2074
  {
2075
  "id": "hard-opus-a-probability-04",
 
2080
  "valid": true,
2081
  "error": null,
2082
  "probs": {
2083
+ "early": 0.0018,
2084
+ "late": 0.047,
2085
+ "on_time": 0.9512
2086
  },
2087
+ "latency_s": 0.072
2088
  },
2089
  {
2090
  "id": "hard-opus-a-probability-07",
 
2095
  "valid": true,
2096
  "error": null,
2097
  "probs": {
2098
+ "escalated": 0.9973,
2099
+ "reopened": 0.0004,
2100
+ "resolved_first_contact": 0.0023
2101
  },
2102
+ "latency_s": 0.073
2103
  },
2104
  {
2105
  "id": "hard-opus-a-probability-08",
 
2110
  "valid": true,
2111
  "error": null,
2112
  "probs": {
2113
+ "yes": 0.3775,
2114
+ "no": 0.6225
2115
  },
2116
+ "latency_s": 0.07
2117
  },
2118
  {
2119
  "id": "hard-opus-a-temporal_numeric-03",
 
2124
  "valid": true,
2125
  "error": null,
2126
  "probs": {
2127
+ "yes": 0.7079,
2128
+ "no": 0.2921
2129
  },
2130
+ "latency_s": 0.07
2131
  },
2132
  {
2133
  "id": "hard-opus-a-temporal_numeric-06",
 
2138
  "valid": true,
2139
  "error": null,
2140
  "probs": {
2141
+ "yes": 0.0225,
2142
+ "no": 0.9775
2143
  },
2144
+ "latency_s": 0.069
2145
  },
2146
  {
2147
  "id": "hard-opus-a-temporal_numeric-07",
 
2152
  "valid": true,
2153
  "error": null,
2154
  "probs": {
2155
+ "sep_24": 0.3883,
2156
+ "sep_25": 0.0066,
2157
+ "sep_26": 0.0031,
2158
+ "sep_27": 0.0053,
2159
+ "sep_28": 0.5967
2160
  },
2161
+ "latency_s": 0.072
2162
  },
2163
  {
2164
  "id": "hard-opus-a-temporal_numeric-09",
 
2169
  "valid": true,
2170
  "error": null,
2171
  "probs": {
2172
+ "yes": 0.0633,
2173
+ "no": 0.9367
2174
  },
2175
+ "latency_s": 0.07
2176
  },
2177
  {
2178
  "id": "hard-opus-a-temporal_numeric-12",
 
2183
  "valid": true,
2184
  "error": null,
2185
  "probs": {
2186
+ "0": 0.0634,
2187
+ "1": 0.6449,
2188
+ "2": 0.2902,
2189
+ "3": 0.0015
2190
  },
2191
+ "latency_s": 0.075
2192
  },
2193
  {
2194
  "id": "hard-opus-b-ambiguous-02",
 
2199
  "valid": true,
2200
  "error": null,
2201
  "probs": {
2202
+ "cannot_determine": 0.0031,
2203
+ "false_positive": 0.1587,
2204
+ "real_incident": 0.8383
2205
  },
2206
+ "latency_s": 0.071
2207
  },
2208
  {
2209
  "id": "hard-opus-b-ambiguous-03",
 
2214
  "valid": true,
2215
  "error": null,
2216
  "probs": {
2217
+ "insufficient_information": 0.0009,
2218
+ "time_barred": 0.1983,
2219
+ "within_limitation": 0.8008
2220
  },
2221
+ "latency_s": 0.07
2222
  },
2223
  {
2224
  "id": "hard-opus-b-ambiguous-07",
 
2229
  "valid": true,
2230
  "error": null,
2231
  "probs": {
2232
+ "insufficient_information": 0.1528,
2233
+ "over_limit": 0.0399,
2234
+ "within_policy": 0.8073
2235
  },
2236
+ "latency_s": 0.075
2237
  },
2238
  {
2239
  "id": "hard-opus-b-ambiguous-09",
 
2244
  "valid": true,
2245
  "error": null,
2246
  "probs": {
2247
+ "compromised": 0.0425,
2248
+ "inconclusive": 0.0003,
2249
+ "legitimate": 0.9572
2250
  },
2251
+ "latency_s": 0.073
2252
  },
2253
  {
2254
  "id": "hard-opus-b-ambiguous-10",
 
2259
  "valid": true,
2260
  "error": null,
2261
  "probs": {
2262
+ "ask_user_which_alex": 0.2726,
2263
+ "send_to_alex_moreno": 0.7165,
2264
  "send_to_alexandra_lind": 0.0109
2265
  },
2266
+ "latency_s": 0.072
2267
  },
2268
  {
2269
  "id": "hard-opus-b-ambiguous-11",
 
2274
  "valid": true,
2275
  "error": null,
2276
  "probs": {
2277
+ "cannot_determine": 0.9952,
2278
+ "grade_correct": 0.0035,
2279
+ "grade_should_be_higher": 0.0013
2280
  },
2281
+ "latency_s": 0.072
2282
  },
2283
  {
2284
  "id": "hard-opus-b-ambiguous-13",
 
2289
  "valid": true,
2290
  "error": null,
2291
  "probs": {
2292
+ "cannot_determine": 0.9017,
2293
+ "charge_data_team": 0.0065,
2294
+ "charge_platform_team": 0.0896,
2295
+ "split_50_50": 0.0023
2296
  },
2297
  "latency_s": 0.07
2298
  },
 
2305
  "valid": true,
2306
  "error": null,
2307
  "probs": {
2308
+ "tier1_team_lead": 0.0134,
2309
+ "tier2_department_head": 0.0017,
2310
+ "tier3_cfo": 0.9829,
2311
+ "tier4_board": 0.002
2312
  },
2313
  "latency_s": 0.09
2314
  },
 
2321
  "valid": true,
2322
  "error": null,
2323
  "probs": {
2324
+ "ana": 0.0996,
2325
+ "bjorn": 0.2618,
2326
+ "chen": 0.3245,
2327
+ "dmitri": 0.0056,
2328
+ "esra": 0.0169,
2329
+ "no_page_ticket_only": 0.2915
2330
  },
2331
+ "latency_s": 0.082
2332
  },
2333
  {
2334
  "id": "hard-opus-b-multi_hop-05",
 
2339
  "valid": true,
2340
  "error": null,
2341
  "probs": {
2342
+ "0": 0.0002,
2343
+ "1": 0.0122,
2344
+ "2": 0.9856,
2345
+ "3": 0.0008,
2346
+ "4": 0.0011
2347
  },
2348
+ "latency_s": 0.37
2349
  },
2350
  {
2351
  "id": "hard-opus-b-multi_hop-07",
 
2356
  "valid": true,
2357
  "error": null,
2358
  "probs": {
2359
+ "credit_10_percent": 0.002,
2360
+ "credit_20_percent": 0.0059,
2361
+ "credit_25_percent": 0.114,
2362
+ "credit_50_percent": 0.7468,
2363
+ "credit_full_month": 0.127,
2364
+ "no_credit": 0.0043
2365
  },
2366
  "latency_s": 0.088
2367
  },
 
2374
  "valid": true,
2375
  "error": null,
2376
  "probs": {
2377
+ "yes": 0.0118,
2378
+ "no": 0.9882
2379
  },
2380
+ "latency_s": 0.081
2381
  },
2382
  {
2383
  "id": "hard-opus-b-probability-01",
 
2388
  "valid": true,
2389
  "error": null,
2390
  "probs": {
2391
+ "customer_wins": 0.0004,
2392
+ "customer_withdraws": 0.0001,
2393
+ "merchant_wins": 0.9995
2394
  },
2395
+ "latency_s": 0.076
2396
  },
2397
  {
2398
  "id": "hard-opus-b-probability-02",
 
2403
  "valid": true,
2404
  "error": null,
2405
  "probs": {
2406
+ "bad_push": 0.0011,
2407
+ "database_hardware": 0.0006,
2408
+ "upstream_provider": 0.9983
2409
  },
2410
+ "latency_s": 0.072
2411
  },
2412
  {
2413
  "id": "hard-opus-b-probability-03",
 
2418
  "valid": true,
2419
  "error": null,
2420
  "probs": {
2421
+ "yes": 0.9491,
2422
+ "no": 0.0509
2423
  },
2424
+ "latency_s": 0.075
2425
  },
2426
  {
2427
  "id": "hard-opus-b-probability-04",
 
2432
  "valid": true,
2433
  "error": null,
2434
  "probs": {
2435
+ "attended": 0.9994,
2436
+ "late_cancellation": 0.0003,
2437
+ "no_show": 0.0003
2438
  },
2439
+ "latency_s": 0.074
2440
  },
2441
  {
2442
  "id": "hard-opus-b-probability-06",
 
2448
  "error": null,
2449
  "probs": {
2450
  "modified": 0.0001,
2451
+ "overturned": 0.0002,
2452
+ "upheld": 0.9997
2453
  },
2454
+ "latency_s": 0.08
2455
  },
2456
  {
2457
  "id": "hard-opus-b-tradeoff-01",
 
2462
  "valid": true,
2463
  "error": null,
2464
  "probs": {
2465
+ "escalate_to_engineering": 0.9635,
2466
+ "escalate_to_security": 0.0345,
2467
+ "page_oncall": 0.0004,
2468
+ "process_refund": 0.0011,
2469
+ "send_kb_article": 0.0005
2470
  },
2471
+ "latency_s": 0.079
2472
  },
2473
  {
2474
  "id": "hard-opus-b-tradeoff-03",
 
2479
  "valid": true,
2480
  "error": null,
2481
  "probs": {
2482
+ "order_a": 0.0008,
2483
+ "order_b": 0.9985,
2484
+ "order_c": 0.0004,
2485
+ "order_d": 0.0003
2486
  },
2487
+ "latency_s": 0.075
2488
  },
2489
  {
2490
  "id": "hard-opus-b-tradeoff-06",
 
2495
  "valid": true,
2496
  "error": null,
2497
  "probs": {
2498
+ "yes": 0.9591,
2499
+ "no": 0.0409
2500
  },
2501
+ "latency_s": 0.077
2502
  },
2503
  {
2504
  "id": "hard-opus-b-tradeoff-07",
 
2511
  "probs": {
2512
  "out_of_scope": 0.0001,
2513
  "p0_fix_24h": 0.9994,
2514
+ "p1_fix_7d": 0.0002,
2515
  "p2_fix_30d": 0.0001,
2516
  "p3_backlog": 0.0001
2517
  },
2518
+ "latency_s": 0.074
2519
  },
2520
  {
2521
  "id": "hard-opus-b-tradeoff-08",
 
2526
  "valid": true,
2527
  "error": null,
2528
  "probs": {
2529
+ "yes": 0.0904,
2530
+ "no": 0.9096
2531
  },
2532
+ "latency_s": 0.072
2533
  },
2534
  {
2535
  "id": "hard-opus-b-tradeoff-12",
 
2540
  "valid": true,
2541
  "error": null,
2542
  "probs": {
2543
+ "deny_refund": 0.0,
2544
+ "escalate_to_fraud_team": 0.0,
2545
+ "offer_store_credit": 0.0001,
2546
+ "refund_now": 0.9998,
2547
+ "wait_for_inspection": 0.0
2548
  },
2549
+ "latency_s": 0.071
2550
  },
2551
  {
2552
  "id": "hard-opus-c-long_policy-02",
 
2557
  "valid": true,
2558
  "error": null,
2559
  "probs": {
2560
+ "erase_all": 0.0001,
2561
+ "erase_but_retain_transaction_records": 0.0083,
2562
+ "erase_marketing_only": 0.0001,
2563
+ "reject_unverified_request": 0.0,
2564
+ "restrict_all_pending_legal_hold": 0.9915
2565
  },
2566
  "latency_s": 0.118
2567
  },
 
2569
  "id": "hard-opus-c-long_policy-03",
2570
  "tier": "hard",
2571
  "expected": "eur_400",
2572
+ "predicted": "eur_400",
2573
+ "correct": true,
2574
  "valid": true,
2575
  "error": null,
2576
  "probs": {
2577
+ "eur_0": 0.0194,
2578
+ "eur_150": 0.1575,
2579
+ "eur_200": 0.039,
2580
+ "eur_300": 0.0704,
2581
+ "eur_400": 0.4139,
2582
+ "eur_600": 0.2999
2583
  },
2584
  "latency_s": 0.118
2585
  },
 
2587
  "id": "hard-opus-c-long_policy-04",
2588
  "tier": "hard",
2589
  "expected": "yes",
2590
+ "predicted": "no",
2591
+ "correct": false,
2592
  "valid": true,
2593
  "error": null,
2594
  "probs": {
2595
+ "yes": 0.0936,
2596
+ "no": 0.9064
2597
  },
2598
  "latency_s": 0.105
2599
  },
 
2606
  "valid": true,
2607
  "error": null,
2608
  "probs": {
2609
+ "0": 0.0057,
2610
+ "1": 0.2939,
2611
+ "2": 0.6751,
2612
+ "3": 0.0253
2613
  },
2614
+ "latency_s": 0.372
2615
  },
2616
  {
2617
  "id": "hard-opus-c-long_policy-08",
 
2622
  "valid": true,
2623
  "error": null,
2624
  "probs": {
2625
+ "0": 0.0002,
2626
+ "1": 0.9992,
2627
+ "2": 0.0003,
2628
+ "3": 0.0002
2629
  },
2630
  "latency_s": 0.38
2631
  },
 
2638
  "valid": true,
2639
  "error": null,
2640
  "probs": {
2641
+ "credit_minus_restocking_fee": 0.975,
2642
+ "full_credit": 0.0204,
2643
+ "reject_return": 0.0037,
2644
+ "repair_or_replace_only": 0.0009
2645
  },
2646
+ "latency_s": 0.122
2647
  },
2648
  {
2649
  "id": "hard-opus-c-long_policy-11",
 
2654
  "valid": true,
2655
  "error": null,
2656
  "probs": {
2657
+ "yes": 0.9295,
2658
+ "no": 0.0705
2659
  },
2660
+ "latency_s": 0.109
2661
  },
2662
  {
2663
  "id": "hard-opus-c-probability-03",
 
2668
  "valid": true,
2669
  "error": null,
2670
  "probs": {
2671
+ "halvorsen_polymer": 0.9951,
2672
+ "kuznets_technik": 0.0011,
2673
+ "ortega_seal_works": 0.0037
2674
  },
2675
+ "latency_s": 0.074
2676
  },
2677
  {
2678
  "id": "hard-opus-c-temporal_numeric-02",
 
2683
  "valid": true,
2684
  "error": null,
2685
  "probs": {
2686
+ "yes": 0.1003,
2687
+ "no": 0.8997
2688
  },
2689
+ "latency_s": 0.087
2690
  },
2691
  {
2692
  "id": "hard-opus-c-temporal_numeric-03",
 
2697
  "valid": true,
2698
  "error": null,
2699
  "probs": {
2700
+ "eur_617_00": 0.0204,
2701
+ "eur_688_00": 0.0631,
2702
+ "eur_692_00": 0.3713,
2703
+ "eur_693_90": 0.5408,
2704
+ "eur_812_00": 0.0043
2705
  },
2706
+ "latency_s": 0.084
2707
  },
2708
  {
2709
  "id": "hard-opus-c-temporal_numeric-04",
2710
  "tier": "hard",
2711
  "expected": "usd_2335_00",
2712
+ "predicted": "usd_2303_33",
2713
  "correct": false,
2714
  "valid": true,
2715
  "error": null,
2716
  "probs": {
2717
+ "usd_2222_50": 0.0142,
2718
+ "usd_2303_01": 0.4418,
2719
+ "usd_2303_33": 0.4919,
2720
+ "usd_2335_00": 0.0336,
2721
+ "usd_2340_00": 0.0186
2722
  },
2723
+ "latency_s": 0.08
2724
  },
2725
  {
2726
  "id": "hard-opus-c-temporal_numeric-06",
2727
  "tier": "hard",
2728
  "expected": "sep_18_book_set",
2729
+ "predicted": "sep_17_parking",
2730
  "correct": false,
2731
  "valid": true,
2732
  "error": null,
2733
  "probs": {
2734
+ "none_declined": 0.3353,
2735
+ "sep_15_office_chair": 0.1146,
2736
+ "sep_16_taxi": 0.0241,
2737
+ "sep_17_parking": 0.4628,
2738
+ "sep_18_book_set": 0.0512,
2739
+ "sep_21_client_dinner": 0.012
2740
  },
2741
+ "latency_s": 0.074
2742
  },
2743
  {
2744
  "id": "hard-opus-c-temporal_numeric-08",
 
2749
  "valid": true,
2750
  "error": null,
2751
  "probs": {
2752
+ "eur_497_40": 0.0649,
2753
+ "eur_498_81": 0.3079,
2754
+ "eur_506_06": 0.1899,
2755
+ "eur_513_69": 0.3428,
2756
+ "eur_532_01": 0.0945
2757
  },
2758
+ "latency_s": 0.073
2759
  },
2760
  {
2761
  "id": "hard-opus-c-temporal_numeric-12",
 
2766
  "valid": true,
2767
  "error": null,
2768
  "probs": {
2769
+ "covered": 0.0012,
2770
+ "expired_24_month_term": 0.0403,
2771
+ "expired_30_month_cap": 0.9585
2772
  },
2773
+ "latency_s": 0.073
2774
  },
2775
  {
2776
  "id": "hard-sol-a-adversarial-01",
 
2781
  "valid": true,
2782
  "error": null,
2783
  "probs": {
2784
+ "deny_inactive_user": 0.0002,
2785
+ "deny_missing_approval": 0.9989,
2786
+ "grant_access": 0.0006,
2787
  "grant_owner_exception": 0.0003
2788
  },
2789
+ "latency_s": 0.072
2790
  },
2791
  {
2792
  "id": "hard-sol-a-adversarial-06",
 
2797
  "valid": true,
2798
  "error": null,
2799
  "probs": {
2800
+ "yes": 0.0018,
2801
+ "no": 0.9982
2802
  },
2803
+ "latency_s": 0.071
2804
  },
2805
  {
2806
  "id": "hard-sol-a-adversarial-07",
 
2811
  "valid": true,
2812
  "error": null,
2813
  "probs": {
2814
+ "delete_now": 0.0006,
2815
  "needs_signature_review": 0.0001,
2816
+ "retain_indefinitely": 0.0001,
2817
+ "retain_until_2031": 0.9992
2818
  },
2819
+ "latency_s": 0.071
2820
  },
2821
  {
2822
  "id": "hard-sol-a-adversarial-08",
 
2827
  "valid": true,
2828
  "error": null,
2829
  "probs": {
2830
+ "yes": 0.9954,
2831
+ "no": 0.0046
2832
  },
2833
+ "latency_s": 0.07
2834
  },
2835
  {
2836
  "id": "hard-sol-a-adversarial-09",
 
2841
  "valid": true,
2842
  "error": null,
2843
  "probs": {
2844
+ "0": 0.0112,
2845
+ "1": 0.1149,
2846
+ "2": 0.8693,
2847
+ "3": 0.0046
2848
  },
2849
+ "latency_s": 0.077
2850
  },
2851
  {
2852
  "id": "hard-sol-a-adversarial-11",
2853
  "tier": "hard",
2854
  "expected": "standard_research",
2855
+ "predicted": "standard_research",
2856
+ "correct": true,
2857
  "valid": true,
2858
  "error": null,
2859
  "probs": {
2860
+ "ethics_review": 0.1951,
2861
+ "major_grants": 0.0538,
2862
+ "manual_hold": 0.0041,
2863
+ "standard_research": 0.747
2864
  },
2865
+ "latency_s": 0.071
2866
  },
2867
  {
2868
  "id": "hard-sol-a-multi_hop-01",
2869
  "tier": "hard",
2870
  "expected": "accept_s2_dusk",
2871
+ "predicted": "reject_cutoff",
2872
+ "correct": false,
2873
  "valid": true,
2874
  "error": null,
2875
  "probs": {
2876
+ "accept_s2_dusk": 0.3302,
2877
+ "reject_cutoff": 0.6637,
2878
+ "reject_tier": 0.0038,
2879
  "reroute_s3": 0.0022
2880
  },
2881
+ "latency_s": 0.084
2882
  },
2883
  {
2884
  "id": "hard-sol-a-multi_hop-02",
 
2889
  "valid": true,
2890
  "error": null,
2891
  "probs": {
2892
+ "executive_bridge": 0.9983,
2893
+ "heron_q2": 0.0004,
2894
+ "insufficient_information": 0.0003,
2895
+ "security_primary": 0.001
2896
  },
2897
+ "latency_s": 0.084
2898
  },
2899
  {
2900
  "id": "hard-sol-a-multi_hop-05",
 
2905
  "valid": true,
2906
  "error": null,
2907
  "probs": {
2908
+ "pay_manager_approved": 0.0011,
2909
+ "reject_category": 0.0175,
2910
+ "require_director": 0.9808,
2911
+ "require_finance_board": 0.0006
2912
  },
2913
+ "latency_s": 0.083
2914
  },
2915
  {
2916
  "id": "hard-sol-a-multi_hop-07",
 
2921
  "valid": true,
2922
  "error": null,
2923
  "probs": {
2924
+ "admit": 0.0984,
2925
+ "deny_prerequisite": 0.8894,
2926
+ "deny_score": 0.0051,
2927
+ "manual_equivalency": 0.0071
2928
  },
2929
+ "latency_s": 0.083
2930
  },
2931
  {
2932
  "id": "hard-sol-a-multi_hop-08",
 
2937
  "valid": true,
2938
  "error": null,
2939
  "probs": {
2940
+ "central_engineering": 0.9292,
2941
+ "insufficient_log": 0.0028,
2942
+ "local_maintenance": 0.0669,
2943
+ "safety_shutdown": 0.0011
2944
  },
2945
+ "latency_s": 0.083
2946
  },
2947
  {
2948
  "id": "hard-sol-a-multi_hop-09",
 
2953
  "valid": true,
2954
  "error": null,
2955
  "probs": {
2956
+ "confirm_l3": 0.0023,
2957
+ "reject_capacity": 0.0166,
2958
+ "require_steward": 0.9806,
2959
+ "switch_l4": 0.0006
2960
  },
2961
+ "latency_s": 0.084
2962
  },
2963
  {
2964
  "id": "hard-sol-a-multi_hop-10",
 
2969
  "valid": true,
2970
  "error": null,
2971
  "probs": {
2972
+ "approve_45_days": 0.9896,
2973
+ "deny_sensitive": 0.003,
2974
+ "needs_field_review": 0.0026,
2975
+ "reduce_to_30_days": 0.0049
2976
  },
2977
+ "latency_s": 0.084
2978
  },
2979
  {
2980
  "id": "hard-sol-a-multi_hop-12",
 
2985
  "valid": true,
2986
  "error": null,
2987
  "probs": {
2988
+ "covered_recurrence": 0.949,
2989
+ "expired": 0.0305,
2990
+ "needs_authentication": 0.0179,
2991
+ "wrong_issue_class": 0.0026
2992
  },
2993
+ "latency_s": 0.083
2994
  },
2995
  {
2996
  "id": "hard-sol-a-trap-02",
 
3001
  "valid": true,
3002
  "error": null,
3003
  "probs": {
3004
+ "yes": 0.0004,
3005
+ "no": 0.9996
3006
  },
3007
+ "latency_s": 0.072
3008
  },
3009
  {
3010
  "id": "hard-sol-a-trap-04",
 
3015
  "valid": true,
3016
  "error": null,
3017
  "probs": {
3018
+ "yes": 0.0033,
3019
+ "no": 0.9967
3020
  },
3021
+ "latency_s": 0.07
3022
  },
3023
  {
3024
  "id": "hard-sol-a-trap-05",
 
3029
  "valid": true,
3030
  "error": null,
3031
  "probs": {
3032
+ "0": 0.9849,
3033
+ "1": 0.0012,
3034
+ "2": 0.0137,
3035
+ "3": 0.0003
3036
  },
3037
+ "latency_s": 0.08
3038
  },
3039
  {
3040
  "id": "hard-sol-a-trap-06",
 
3045
  "valid": true,
3046
  "error": null,
3047
  "probs": {
3048
+ "await_internal_metrics": 0.0004,
3049
  "close_agent_verified": 0.0002,
3050
+ "close_confirmed": 0.0007,
3051
+ "keep_open_unresolved": 0.9987
3052
  },
3053
+ "latency_s": 0.072
3054
  },
3055
  {
3056
  "id": "hard-sol-a-trap-08",
 
3062
  "error": null,
3063
  "probs": {
3064
  "canceled": 0.0,
3065
+ "indeterminate": 0.0,
3066
+ "late": 0.9999,
3067
+ "on_time": 0.0
3068
  },
3069
+ "latency_s": 0.072
3070
  },
3071
  {
3072
  "id": "hard-sol-a-trap-10",
 
3077
  "valid": true,
3078
  "error": null,
3079
  "probs": {
3080
+ "blocked_failed_test": 0.0015,
3081
+ "blocked_second_review": 0.9946,
3082
+ "ready_one_review": 0.0014,
3083
+ "ready_two_reviews": 0.0024
3084
  },
3085
+ "latency_s": 0.071
3086
  },
3087
  {
3088
  "id": "hard-sol-a-trap-13",
 
3093
  "valid": true,
3094
  "error": null,
3095
  "probs": {
3096
+ "yes": 0.0014,
3097
+ "no": 0.9986
3098
  },
3099
+ "latency_s": 0.07
3100
  },
3101
  {
3102
  "id": "hard-sol-a-trap-15",
 
3107
  "valid": true,
3108
  "error": null,
3109
  "probs": {
3110
+ "yes": 0.0731,
3111
+ "no": 0.9269
3112
  },
3113
+ "latency_s": 0.07
3114
  },
3115
  {
3116
  "id": "hard-sol-b-judge_hard-01",
 
3121
  "valid": true,
3122
  "error": null,
3123
  "probs": {
3124
+ "yes": 0.7999,
3125
+ "no": 0.2001
3126
  },
3127
+ "latency_s": 0.07
3128
  },
3129
  {
3130
  "id": "hard-sol-b-judge_hard-02",
 
3135
  "valid": true,
3136
  "error": null,
3137
  "probs": {
3138
+ "yes": 0.9031,
3139
+ "no": 0.0969
3140
  },
3141
+ "latency_s": 0.07
3142
  },
3143
  {
3144
  "id": "hard-sol-b-judge_hard-05",
 
3149
  "valid": true,
3150
  "error": null,
3151
  "probs": {
3152
+ "yes": 0.7807,
3153
+ "no": 0.2193
3154
  },
3155
+ "latency_s": 0.069
3156
  },
3157
  {
3158
  "id": "hard-sol-b-judge_hard-08",
 
3163
  "valid": true,
3164
  "error": null,
3165
  "probs": {
3166
+ "yes": 0.9491,
3167
+ "no": 0.0509
3168
  },
3169
+ "latency_s": 0.07
3170
  },
3171
  {
3172
  "id": "hard-sol-b-judge_hard-10",
 
3177
  "valid": true,
3178
  "error": null,
3179
  "probs": {
3180
+ "yes": 0.2921,
3181
+ "no": 0.7079
3182
  },
3183
+ "latency_s": 0.069
3184
  },
3185
  {
3186
  "id": "hard-sol-b-judge_hard-14",
 
3191
  "valid": true,
3192
  "error": null,
3193
  "probs": {
3194
+ "yes": 0.0051,
3195
+ "no": 0.9949
3196
  },
3197
+ "latency_s": 0.069
3198
  },
3199
  {
3200
  "id": "hard-sol-b-judge_hard-15",
 
3205
  "valid": true,
3206
  "error": null,
3207
  "probs": {
3208
+ "yes": 0.8768,
3209
+ "no": 0.1232
3210
  },
3211
+ "latency_s": 0.069
3212
  },
3213
  {
3214
  "id": "hard-sol-b-judge_hard-18",
 
3219
  "valid": true,
3220
  "error": null,
3221
  "probs": {
3222
+ "yes": 0.9783,
3223
+ "no": 0.0217
3224
  },
3225
+ "latency_s": 0.07
3226
  },
3227
  {
3228
  "id": "hard-sol-b-long_policy-01",
 
3233
  "valid": true,
3234
  "error": null,
3235
  "probs": {
3236
+ "approve_all": 0.0028,
3237
+ "approve_lodging_only": 0.0983,
3238
+ "approve_meals_only": 0.0109,
3239
+ "reject_entire_claim": 0.8881
3240
  },
3241
+ "latency_s": 0.091
3242
  },
3243
  {
3244
  "id": "hard-sol-b-long_policy-02",
3245
  "tier": "hard",
3246
  "expected": "deny_current_export",
3247
+ "predicted": "deny_current_export",
3248
+ "correct": true,
3249
  "valid": true,
3250
  "error": null,
3251
  "probs": {
3252
+ "analytics_owner_only": 0.0099,
3253
+ "deny_current_export": 0.6492,
3254
+ "privacy_and_security": 0.094,
3255
+ "privacy_only": 0.247
3256
  },
3257
+ "latency_s": 0.088
3258
  },
3259
  {
3260
  "id": "hard-sol-b-long_policy-05",
 
3265
  "valid": true,
3266
  "error": null,
3267
  "probs": {
3268
+ "delete_everything": 0.0005,
3269
+ "delete_nothing": 0.9919,
3270
+ "delete_only_profile": 0.0064,
3271
+ "delete_profile_omar_april_and_table": 0.0012
3272
  },
3273
+ "latency_s": 0.086
3274
  },
3275
  {
3276
  "id": "hard-sol-b-long_policy-06",
 
3281
  "valid": true,
3282
  "error": null,
3283
  "probs": {
3284
+ "east_ward": 0.0103,
3285
+ "harbor_neighbors": 0.0017,
3286
+ "no_award": 0.0035,
3287
+ "river_arts": 0.9845
3288
  },
3289
+ "latency_s": 0.088
3290
  },
3291
  {
3292
  "id": "hard-sol-b-routing_hard-01",
 
3300
  "data_pipeline": 0.0,
3301
  "desktop_support": 0.0001,
3302
  "finance_review": 0.0001,
3303
+ "office_file_repair": 0.9996,
3304
  "spreadsheet_formula": 0.0003
3305
  },
3306
+ "latency_s": 0.071
3307
  },
3308
  {
3309
  "id": "hard-sol-b-routing_hard-02",
 
3315
  "error": null,
3316
  "probs": {
3317
  "customer_support": 0.0001,
3318
+ "fraud_review": 0.0,
3319
  "ledger_reconciliation": 0.9996,
3320
  "mobile_app": 0.0001,
3321
  "payment_disputes": 0.0002
3322
  },
3323
+ "latency_s": 0.07
3324
  },
3325
  {
3326
  "id": "hard-sol-b-routing_hard-03",
 
3331
  "valid": true,
3332
  "error": null,
3333
  "probs": {
3334
+ "api_platform": 0.0004,
3335
+ "authoritative_dns": 0.0008,
3336
+ "endpoint_devices": 0.0024,
3337
+ "network_dns": 0.9959,
3338
+ "tls_certificates": 0.0004
3339
  },
3340
+ "latency_s": 0.07
3341
  },
3342
  {
3343
  "id": "hard-sol-b-routing_hard-07",
 
3349
  "error": null,
3350
  "probs": {
3351
  "ci_infrastructure": 0.0001,
3352
+ "code_signing": 0.0002,
3353
+ "endpoint_security": 0.0001,
3354
+ "macos_runtime": 0.0001,
3355
+ "release_notarization": 0.9994
3356
  },
3357
+ "latency_s": 0.069
3358
  },
3359
  {
3360
  "id": "hard-sol-b-routing_hard-09",
 
3371
  "source_access": 0.9996,
3372
  "vpn_network": 0.0002
3373
  },
3374
+ "latency_s": 0.07
3375
  },
3376
  {
3377
  "id": "hard-sol-b-temporal_numeric-01",
 
3382
  "valid": true,
3383
  "error": null,
3384
  "probs": {
3385
+ "before_open": 0.0292,
3386
+ "late_by_2h_or_more": 0.09,
3387
+ "late_by_under_2h": 0.2243,
3388
+ "within_window": 0.6565
3389
  },
3390
+ "latency_s": 0.071
3391
  },
3392
  {
3393
  "id": "hard-sol-b-temporal_numeric-02",
 
3398
  "valid": true,
3399
  "error": null,
3400
  "probs": {
3401
+ "at_least_100gb_below": 0.2445,
3402
+ "exactly_equal": 0.0711,
3403
+ "over_limit": 0.0416,
3404
+ "under_100gb_below": 0.6428
3405
  },
3406
+ "latency_s": 0.071
3407
  },
3408
  {
3409
  "id": "hard-sol-b-temporal_numeric-03",
 
3414
  "valid": true,
3415
  "error": null,
3416
  "probs": {
3417
+ "12_credits": 0.2833,
3418
+ "15_credits": 0.0566,
3419
+ "6_credits": 0.1487,
3420
+ "9_credits": 0.5114
3421
  },
3422
+ "latency_s": 0.07
3423
  },
3424
  {
3425
  "id": "hard-sol-b-temporal_numeric-04",
 
3430
  "valid": true,
3431
  "error": null,
3432
  "probs": {
3433
+ "clear_fail": 0.1675,
3434
+ "narrow_fail": 0.7535,
3435
+ "pass": 0.079
3436
  },
3437
+ "latency_s": 0.07
3438
  },
3439
  {
3440
  "id": "hard-sol-c-judge_hard-03",
 
3445
  "valid": true,
3446
  "error": null,
3447
  "probs": {
3448
+ "yes": 0.3866,
3449
+ "no": 0.6134
3450
  },
3451
+ "latency_s": 0.073
3452
  },
3453
  {
3454
  "id": "hard-sol-c-judge_hard-05",
 
3459
  "valid": true,
3460
  "error": null,
3461
  "probs": {
3462
+ "yes": 0.595,
3463
+ "no": 0.405
3464
  },
3465
+ "latency_s": 0.078
3466
  },
3467
  {
3468
  "id": "hard-sol-c-judge_hard-07",
 
3473
  "valid": true,
3474
  "error": null,
3475
  "probs": {
3476
+ "yes": 0.2259,
3477
+ "no": 0.7741
3478
  },
3479
+ "latency_s": 0.073
3480
  },
3481
  {
3482
  "id": "hard-sol-c-judge_hard-08",
 
3487
  "valid": true,
3488
  "error": null,
3489
  "probs": {
3490
+ "yes": 0.9509,
3491
+ "no": 0.0491
3492
  },
3493
+ "latency_s": 0.073
3494
  },
3495
  {
3496
  "id": "hard-sol-c-judge_hard-09",
 
3501
  "valid": true,
3502
  "error": null,
3503
  "probs": {
3504
+ "yes": 0.9215,
3505
+ "no": 0.0785
3506
  },
3507
+ "latency_s": 0.072
3508
  },
3509
  {
3510
  "id": "hard-sol-c-judge_hard-10",
 
3515
  "valid": true,
3516
  "error": null,
3517
  "probs": {
3518
+ "yes": 0.7873,
3519
+ "no": 0.2127
3520
  },
3521
+ "latency_s": 0.072
3522
  },
3523
  {
3524
  "id": "hard-sol-c-judge_hard-11",
 
3529
  "valid": true,
3530
  "error": null,
3531
  "probs": {
3532
+ "yes": 0.0078,
3533
+ "no": 0.9922
3534
  },
3535
+ "latency_s": 0.073
3536
  },
3537
  {
3538
  "id": "hard-sol-c-judge_hard-13",
3539
  "tier": "hard",
3540
  "expected": "no",
3541
+ "predicted": "no",
3542
+ "correct": true,
3543
  "valid": true,
3544
  "error": null,
3545
  "probs": {
3546
+ "yes": 0.4808,
3547
+ "no": 0.5192
3548
  },
3549
+ "latency_s": 0.073
3550
  },
3551
  {
3552
  "id": "hard-sol-c-judge_hard-15",
 
3557
  "valid": true,
3558
  "error": null,
3559
  "probs": {
3560
+ "yes": 0.9748,
3561
+ "no": 0.0252
3562
  },
3563
+ "latency_s": 0.071
3564
  },
3565
  {
3566
  "id": "hard-sol-c-multi_hop-07",
 
3571
  "valid": true,
3572
  "error": null,
3573
  "probs": {
3574
+ "partial_release": 0.0061,
3575
+ "programme_review": 0.0895,
3576
+ "reject_draw": 0.0041,
3577
+ "release_draw": 0.9003
3578
  },
3579
  "latency_s": 0.084
3580
  },
 
3587
  "valid": true,
3588
  "error": null,
3589
  "probs": {
3590
+ "bus_substitution": 0.0321,
3591
+ "call_fen_cross": 0.0991,
3592
+ "pass_fen_cross": 0.849,
3593
+ "terminate_oakmere": 0.0198
3594
  },
3595
+ "latency_s": 0.085
3596
  },
3597
  {
3598
  "id": "hard-sol-c-multi_hop-11",
 
3603
  "valid": true,
3604
  "error": null,
3605
  "probs": {
3606
+ "adjuster_referral": 0.0008,
3607
+ "coverage_confirmed": 0.0051,
3608
+ "coverage_excluded": 0.9917,
3609
+ "documentation_needed": 0.0024
3610
  },
3611
+ "latency_s": 0.087
3612
  },
3613
  {
3614
  "id": "hard-sol-c-multi_hop-12",
 
3619
  "valid": true,
3620
  "error": null,
3621
  "probs": {
3622
+ "conservation_hold": 0.953,
3623
+ "deny_request": 0.0341,
3624
+ "digital_copy_only": 0.0094,
3625
+ "supervised_delivery": 0.0034
3626
  },
3627
+ "latency_s": 0.072
3628
  },
3629
  {
3630
  "id": "hard-sol-c-multi_hop-13",
 
3637
  "probs": {
3638
  "engineering_review": 0.0,
3639
  "normal_release": 0.9999,
3640
+ "rework": 0.0,
3641
+ "scrap_lot": 0.0
3642
  },
3643
+ "latency_s": 0.087
3644
  }
3645
  ]
3646
  }
eval/train.log CHANGED
@@ -1,68 +1,85 @@
1
  [args] {"model": "Mapika/decider-2b", "data": "/content/data/coreai.pkl", "out": "/content/runs/delta", "epochs": 3.0, "lr": 8e-06, "wd": 0.0, "warmup": 20, "max_tokens": 12288, "accum": 2, "max_ctx": 16384, "brier_w": 0.0, "label_smooth": 0.0, "eval_every": 100000, "eval_limit": 100, "train_cap": 0, "seed": 0, "resample_every_epoch": true, "none_prob": 0.0, "schema_first_prob": 0.0, "max_options": 255}
2
- [data] train examples 5819; tokenizing...
3
- [data] 5819 items, 5819 questions, 8.8M tokens, tokenized in 21s
4
- [sched] 406 optimizer steps/epoch, 1218 total
5
- [train] step 20/1218 ep 0 ce 0.8302 gn 49.25 lr 8.00e-06 3.6min eta 217min 2061tok/s mem 20GB
6
- [train] step 40/1218 ep 0 ce 0.6959 gn 49.75 lr 7.99e-06 4.5min eta 51min 8572tok/s mem 20GB
7
- [train] step 60/1218 ep 0 ce 0.6086 gn 70.00 lr 7.98e-06 5.4min eta 50min 8558tok/s mem 20GB
8
- [train] step 80/1218 ep 0 ce 0.5988 gn 109.00 lr 7.95e-06 6.2min eta 48min 8809tok/s mem 20GB
9
- [train] step 100/1218 ep 0 ce 0.5918 gn 30.62 lr 7.91e-06 7.0min eta 46min 8740tok/s mem 20GB
10
- [train] step 120/1218 ep 0 ce 0.6117 gn 57.00 lr 7.86e-06 7.9min eta 47min 8607tok/s mem 20GB
11
- [train] step 140/1218 ep 0 ce 0.6376 gn 94.50 lr 7.80e-06 8.7min eta 46min 8773tok/s mem 20GB
12
- [train] step 160/1218 ep 0 ce 0.6409 gn 71.00 lr 7.73e-06 9.6min eta 44min 8878tok/s mem 20GB
13
- [train] step 180/1218 ep 0 ce 0.5535 gn 58.75 lr 7.65e-06 10.4min eta 44min 9017tok/s mem 20GB
14
- [train] step 200/1218 ep 0 ce 0.6176 gn 64.50 lr 7.56e-06 11.3min eta 43min 8735tok/s mem 20GB
15
- [train] step 220/1218 ep 0 ce 0.5510 gn 44.00 lr 7.46e-06 12.1min eta 42min 8932tok/s mem 20GB
16
- [train] step 240/1218 ep 0 ce 0.5052 gn 110.50 lr 7.35e-06 12.9min eta 41min 8974tok/s mem 20GB
17
- [train] step 260/1218 ep 0 ce 0.6494 gn 56.50 lr 7.23e-06 13.8min eta 41min 8816tok/s mem 20GB
18
- [train] step 280/1218 ep 0 ce 0.5310 gn 59.25 lr 7.11e-06 14.6min eta 39min 8953tok/s mem 20GB
19
- [train] step 300/1218 ep 0 ce 0.4870 gn 53.00 lr 6.97e-06 15.5min eta 39min 8890tok/s mem 20GB
20
- [train] step 320/1218 ep 0 ce 0.5785 gn 54.00 lr 6.82e-06 16.3min eta 37min 8781tok/s mem 20GB
21
- [train] step 340/1218 ep 0 ce 0.5031 gn 116.50 lr 6.67e-06 17.2min eta 38min 8821tok/s mem 20GB
22
- [train] step 360/1218 ep 0 ce 0.4754 gn 138.00 lr 6.51e-06 18.0min eta 37min 8783tok/s mem 20GB
23
- [train] step 380/1218 ep 0 ce 0.5028 gn 57.25 lr 6.35e-06 18.9min eta 37min 8916tok/s mem 20GB
24
- [train] step 400/1218 ep 0 ce 0.5541 gn 66.00 lr 6.17e-06 19.8min eta 37min 8586tok/s mem 20GB
25
- [train] step 420/1218 ep 1 ce 0.3341 gn 73.00 lr 5.99e-06 21.1min eta 51min 5894tok/s mem 20GB
26
- [train] step 440/1218 ep 1 ce 0.1610 gn 117.50 lr 5.81e-06 21.9min eta 33min 8812tok/s mem 20GB
27
- [train] step 460/1218 ep 1 ce 0.3019 gn 123.00 lr 5.62e-06 22.7min eta 31min 8813tok/s mem 20GB
28
- [train] step 480/1218 ep 1 ce 0.1749 gn 63.75 lr 5.43e-06 23.6min eta 30min 8908tok/s mem 20GB
29
- [train] step 500/1218 ep 1 ce 0.1769 gn 38.50 lr 5.23e-06 24.4min eta 29min 8855tok/s mem 20GB
30
- [train] step 520/1218 ep 1 ce 0.2283 gn 102.00 lr 5.03e-06 25.2min eta 28min 8890tok/s mem 20GB
31
- [train] step 540/1218 ep 1 ce 0.2253 gn 184.00 lr 4.82e-06 26.1min eta 29min 8862tok/s mem 20GB
32
- [train] step 560/1218 ep 1 ce 0.1854 gn 12.06 lr 4.62e-06 26.9min eta 28min 8789tok/s mem 20GB
33
- [train] step 580/1218 ep 1 ce 0.1464 gn 62.75 lr 4.41e-06 27.8min eta 27min 8884tok/s mem 20GB
34
- [train] step 600/1218 ep 1 ce 0.2939 gn 76.00 lr 4.20e-06 28.6min eta 26min 8997tok/s mem 20GB
35
- [train] step 620/1218 ep 1 ce 0.1688 gn 4.09 lr 3.99e-06 29.4min eta 25min 8672tok/s mem 20GB
36
- [train] step 640/1218 ep 1 ce 0.2019 gn 104.50 lr 3.78e-06 30.3min eta 24min 9022tok/s mem 20GB
37
- [train] step 660/1218 ep 1 ce 0.1530 gn 7.69 lr 3.57e-06 31.1min eta 24min 8881tok/s mem 20GB
38
- [train] step 680/1218 ep 1 ce 0.2115 gn 204.00 lr 3.36e-06 32.0min eta 23min 8829tok/s mem 20GB
39
- [train] step 700/1218 ep 1 ce 0.2408 gn 171.00 lr 3.16e-06 32.8min eta 22min 8923tok/s mem 20GB
40
- [train] step 720/1218 ep 1 ce 0.2177 gn 103.00 lr 2.95e-06 33.7min eta 21min 8826tok/s mem 20GB
41
- [train] step 740/1218 ep 1 ce 0.2206 gn 75.00 lr 2.75e-06 34.5min eta 20min 9010tok/s mem 20GB
42
- [train] step 760/1218 ep 1 ce 0.2344 gn 181.00 lr 2.55e-06 35.4min eta 19min 8918tok/s mem 20GB
43
- [train] step 780/1218 ep 1 ce 0.2430 gn 222.00 lr 2.36e-06 36.2min eta 19min 8740tok/s mem 20GB
44
- [train] step 800/1218 ep 1 ce 0.1455 gn 1.77 lr 2.17e-06 37.1min eta 18min 8824tok/s mem 20GB
45
- [train] step 820/1218 ep 2 ce 0.1816 gn 17.12 lr 1.99e-06 38.4min eta 26min 5805tok/s mem 20GB
46
- [train] step 840/1218 ep 2 ce 0.1665 gn 31.75 lr 1.81e-06 39.3min eta 16min 8815tok/s mem 20GB
47
- [train] step 860/1218 ep 2 ce 0.1074 gn 73.00 lr 1.64e-06 40.1min eta 15min 8901tok/s mem 20GB
48
- [train] step 880/1218 ep 2 ce 0.0816 gn 10.62 lr 1.47e-06 40.9min eta 14min 8936tok/s mem 20GB
49
- [train] step 900/1218 ep 2 ce 0.1037 gn 156.00 lr 1.31e-06 41.7min eta 13min 8797tok/s mem 20GB
50
- [train] step 920/1218 ep 2 ce 0.0824 gn 160.00 lr 1.16e-06 42.6min eta 13min 8919tok/s mem 20GB
51
- [train] step 940/1218 ep 2 ce 0.1162 gn 75.50 lr 1.02e-06 43.4min eta 12min 8878tok/s mem 20GB
52
- [train] step 960/1218 ep 2 ce 0.0624 gn 91.00 lr 8.81e-07 44.3min eta 11min 8992tok/s mem 20GB
53
- [train] step 980/1218 ep 2 ce 0.0627 gn 39.75 lr 7.54e-07 45.1min eta 10min 8834tok/s mem 20GB
54
- [train] step 1000/1218 ep 2 ce 0.0791 gn 20.12 lr 6.36e-07 46.0min eta 9min 8884tok/s mem 20GB
55
- [train] step 1020/1218 ep 2 ce 0.0345 gn 0.20 lr 5.27e-07 46.8min eta 8min 8808tok/s mem 20GB
56
- [train] step 1040/1218 ep 2 ce 0.0773 gn 117.00 lr 4.28e-07 47.7min eta 8min 8749tok/s mem 20GB
57
- [train] step 1060/1218 ep 2 ce 0.0836 gn 71.00 lr 3.38e-07 48.5min eta 7min 8958tok/s mem 20GB
58
- [train] step 1080/1218 ep 2 ce 0.0712 gn 64.00 lr 2.59e-07 49.4min eta 6min 8718tok/s mem 20GB
59
- [train] step 1100/1218 ep 2 ce 0.0736 gn 46.00 lr 1.90e-07 50.2min eta 5min 8868tok/s mem 20GB
60
- [train] step 1120/1218 ep 2 ce 0.2134 gn 16.75 lr 1.31e-07 51.1min eta 4min 8989tok/s mem 20GB
61
- [train] step 1140/1218 ep 2 ce 0.1140 gn 1.55 lr 8.34e-08 51.9min eta 3min 8915tok/s mem 20GB
62
- [train] step 1160/1218 ep 2 ce 0.0907 gn 147.00 lr 4.62e-08 52.7min eta 2min 8805tok/s mem 20GB
63
- [train] step 1180/1218 ep 2 ce 0.1033 gn 22.50 lr 1.98e-08 53.6min eta 2min 8901tok/s mem 20GB
64
- [train] step 1200/1218 ep 2 ce 0.0813 gn 177.00 lr 4.46e-09 54.4min eta 1min 8899tok/s mem 20GB
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
  [save] /content/runs/delta/model
66
- [eval] coreai_v2_heldout n= 100 acc=0.830 (chance 0.30) nll=0.806 brier=0.314 ece=0.149 aurc=0.056 acc@80=0.875
67
- [eval-agg] step 1218 {"in_task": {"acc": 0.83, "nll": 0.806352436542511, "brier": 0.3144715130329132, "ece": 0.14947944998741147, "aurc": 0.05629271727862271, "acc_at_80": 0.875, "chance": 0.29995238095238097}, "heldout": {}, "n_in": 1, "n_heldout": 0}
68
  [done] saved /content/runs/delta/model
 
1
  [args] {"model": "Mapika/decider-2b", "data": "/content/data/coreai.pkl", "out": "/content/runs/delta", "epochs": 3.0, "lr": 8e-06, "wd": 0.0, "warmup": 20, "max_tokens": 12288, "accum": 2, "max_ctx": 16384, "brier_w": 0.0, "label_smooth": 0.0, "eval_every": 100000, "eval_limit": 100, "train_cap": 0, "seed": 0, "resample_every_epoch": true, "none_prob": 0.0, "schema_first_prob": 0.0, "max_options": 255}
2
+ [data] train examples 6362; tokenizing...
3
+ [data] 6362 items, 6362 questions, 10.9M tokens, tokenized in 28s
4
+ [sched] 515 optimizer steps/epoch, 1545 total
5
+ [train] step 20/1545 ep 0 ce 0.8879 gn 74.00 lr 8.00e-06 3.7min eta 284min 1940tok/s mem 20GB
6
+ [train] step 40/1545 ep 0 ce 0.7011 gn 136.00 lr 8.00e-06 4.6min eta 69min 8256tok/s mem 20GB
7
+ [train] step 60/1545 ep 0 ce 0.6629 gn 100.00 lr 7.99e-06 5.5min eta 68min 8023tok/s mem 20GB
8
+ [train] step 80/1545 ep 0 ce 0.5311 gn 127.00 lr 7.97e-06 6.4min eta 63min 8794tok/s mem 20GB
9
+ [train] step 100/1545 ep 0 ce 0.7026 gn 113.00 lr 7.95e-06 7.3min eta 62min 8640tok/s mem 20GB
10
+ [train] step 120/1545 ep 0 ce 0.5571 gn 69.00 lr 7.92e-06 8.1min eta 61min 8762tok/s mem 20GB
11
+ [train] step 140/1545 ep 0 ce 0.7724 gn 84.00 lr 7.88e-06 9.0min eta 61min 8448tok/s mem 20GB
12
+ [train] step 160/1545 ep 0 ce 0.6581 gn 154.00 lr 7.83e-06 9.8min eta 59min 8911tok/s mem 20GB
13
+ [train] step 180/1545 ep 0 ce 0.5204 gn 45.25 lr 7.78e-06 10.7min eta 61min 8208tok/s mem 20GB
14
+ [train] step 200/1545 ep 0 ce 0.5297 gn 54.00 lr 7.73e-06 11.6min eta 59min 8508tok/s mem 20GB
15
+ [train] step 220/1545 ep 0 ce 0.5509 gn 42.00 lr 7.67e-06 12.5min eta 55min 8601tok/s mem 20GB
16
+ [train] step 240/1545 ep 0 ce 0.7435 gn 50.50 lr 7.60e-06 13.3min eta 56min 8717tok/s mem 20GB
17
+ [train] step 260/1545 ep 0 ce 0.5157 gn 107.00 lr 7.52e-06 14.2min eta 55min 8709tok/s mem 20GB
18
+ [train] step 280/1545 ep 0 ce 0.6843 gn 147.00 lr 7.44e-06 15.0min eta 54min 8624tok/s mem 20GB
19
+ [train] step 300/1545 ep 0 ce 0.5498 gn 76.50 lr 7.35e-06 15.9min eta 53min 8609tok/s mem 20GB
20
+ [train] step 320/1545 ep 0 ce 0.6019 gn 294.00 lr 7.26e-06 16.7min eta 51min 8647tok/s mem 20GB
21
+ [train] step 340/1545 ep 0 ce 0.5030 gn 81.00 lr 7.16e-06 17.5min eta 50min 8658tok/s mem 20GB
22
+ [train] step 360/1545 ep 0 ce 0.5059 gn 68.50 lr 7.06e-06 18.4min eta 50min 8506tok/s mem 20GB
23
+ [train] step 380/1545 ep 0 ce 0.4691 gn 59.75 lr 6.95e-06 19.2min eta 48min 8813tok/s mem 20GB
24
+ [train] step 400/1545 ep 0 ce 0.5794 gn 37.00 lr 6.84e-06 20.0min eta 48min 8746tok/s mem 20GB
25
+ [train] step 420/1545 ep 0 ce 0.4682 gn 135.00 lr 6.72e-06 20.9min eta 48min 8628tok/s mem 20GB
26
+ [train] step 440/1545 ep 0 ce 0.5367 gn 27.75 lr 6.59e-06 21.7min eta 47min 8686tok/s mem 20GB
27
+ [train] step 460/1545 ep 0 ce 0.5448 gn 122.00 lr 6.47e-06 22.6min eta 45min 8779tok/s mem 20GB
28
+ [train] step 480/1545 ep 0 ce 0.5098 gn 98.00 lr 6.33e-06 23.4min eta 45min 8563tok/s mem 20GB
29
+ [train] step 500/1545 ep 0 ce 0.3887 gn 110.50 lr 6.20e-06 24.3min eta 44min 8392tok/s mem 20GB
30
+ [train] step 520/1545 ep 1 ce 0.4085 gn 22.50 lr 6.06e-06 25.8min eta 77min 4948tok/s mem 20GB
31
+ [train] step 540/1545 ep 1 ce 0.2176 gn 85.50 lr 5.92e-06 26.6min eta 43min 8637tok/s mem 20GB
32
+ [train] step 560/1545 ep 1 ce 0.3074 gn 99.00 lr 5.77e-06 27.5min eta 42min 8626tok/s mem 20GB
33
+ [train] step 580/1545 ep 1 ce 0.1576 gn 35.00 lr 5.62e-06 28.3min eta 42min 8643tok/s mem 20GB
34
+ [train] step 600/1545 ep 1 ce 0.2292 gn 73.50 lr 5.47e-06 29.2min eta 40min 8502tok/s mem 20GB
35
+ [train] step 620/1545 ep 1 ce 0.1425 gn 73.00 lr 5.31e-06 30.0min eta 40min 8695tok/s mem 20GB
36
+ [train] step 640/1545 ep 1 ce 0.1507 gn 22.38 lr 5.16e-06 30.9min eta 39min 8543tok/s mem 20GB
37
+ [train] step 660/1545 ep 1 ce 0.2277 gn 44.25 lr 5.00e-06 31.7min eta 37min 8738tok/s mem 20GB
38
+ [train] step 680/1545 ep 1 ce 0.3366 gn 3.41 lr 4.84e-06 32.5min eta 36min 8654tok/s mem 20GB
39
+ [train] step 700/1545 ep 1 ce 0.2382 gn 59.25 lr 4.68e-06 33.4min eta 36min 8660tok/s mem 20GB
40
+ [train] step 720/1545 ep 1 ce 0.2296 gn 11.00 lr 4.51e-06 34.3min eta 36min 8664tok/s mem 20GB
41
+ [train] step 740/1545 ep 1 ce 0.1894 gn 93.50 lr 4.35e-06 35.1min eta 35min 8703tok/s mem 20GB
42
+ [train] step 760/1545 ep 1 ce 0.2151 gn 310.00 lr 4.19e-06 36.0min eta 34min 8598tok/s mem 20GB
43
+ [train] step 780/1545 ep 1 ce 0.0980 gn 0.61 lr 4.02e-06 36.8min eta 32min 8575tok/s mem 20GB
44
+ [train] step 800/1545 ep 1 ce 0.2232 gn 70.00 lr 3.86e-06 37.6min eta 31min 8746tok/s mem 20GB
45
+ [train] step 820/1545 ep 1 ce 0.2448 gn 32.75 lr 3.69e-06 38.5min eta 30min 8608tok/s mem 20GB
46
+ [train] step 840/1545 ep 1 ce 0.1860 gn 3.02 lr 3.53e-06 39.3min eta 30min 8670tok/s mem 20GB
47
+ [train] step 860/1545 ep 1 ce 0.1953 gn 208.00 lr 3.36e-06 40.2min eta 29min 8810tok/s mem 20GB
48
+ [train] step 880/1545 ep 1 ce 0.3090 gn 5.84 lr 3.20e-06 41.0min eta 28min 8754tok/s mem 20GB
49
+ [train] step 900/1545 ep 1 ce 0.1677 gn 148.00 lr 3.04e-06 41.8min eta 27min 8746tok/s mem 20GB
50
+ [train] step 920/1545 ep 1 ce 0.1698 gn 32.00 lr 2.88e-06 42.7min eta 27min 8412tok/s mem 20GB
51
+ [train] step 940/1545 ep 1 ce 0.2119 gn 225.00 lr 2.72e-06 43.6min eta 26min 8684tok/s mem 20GB
52
+ [train] step 960/1545 ep 1 ce 0.1439 gn 193.00 lr 2.57e-06 44.4min eta 25min 8686tok/s mem 20GB
53
+ [train] step 980/1545 ep 1 ce 0.1796 gn 18.50 lr 2.42e-06 45.3min eta 24min 8560tok/s mem 20GB
54
+ [train] step 1000/1545 ep 1 ce 0.2553 gn 68.50 lr 2.27e-06 46.1min eta 23min 8685tok/s mem 20GB
55
+ [train] step 1020/1545 ep 1 ce 0.2300 gn 110.50 lr 2.12e-06 47.0min eta 22min 8932tok/s mem 20GB
56
+ [train] step 1040/1545 ep 2 ce 0.0862 gn 33.00 lr 1.98e-06 48.5min eta 39min 4722tok/s mem 20GB
57
+ [train] step 1060/1545 ep 2 ce 0.1130 gn 106.50 lr 1.84e-06 49.3min eta 20min 8767tok/s mem 20GB
58
+ [train] step 1080/1545 ep 2 ce 0.0386 gn 0.01 lr 1.70e-06 50.2min eta 20min 8638tok/s mem 20GB
59
+ [train] step 1100/1545 ep 2 ce 0.0717 gn 33.75 lr 1.57e-06 51.0min eta 19min 8790tok/s mem 20GB
60
+ [train] step 1120/1545 ep 2 ce 0.0447 gn 13.38 lr 1.44e-06 51.9min eta 18min 8591tok/s mem 20GB
61
+ [train] step 1140/1545 ep 2 ce 0.1981 gn 0.03 lr 1.31e-06 52.7min eta 17min 8626tok/s mem 20GB
62
+ [train] step 1160/1545 ep 2 ce 0.1301 gn 107.50 lr 1.19e-06 53.6min eta 16min 8833tok/s mem 20GB
63
+ [train] step 1180/1545 ep 2 ce 0.0541 gn 0.02 lr 1.08e-06 54.4min eta 15min 8582tok/s mem 20GB
64
+ [train] step 1200/1545 ep 2 ce 0.0595 gn 0.04 lr 9.68e-07 55.3min eta 15min 8781tok/s mem 20GB
65
+ [train] step 1220/1545 ep 2 ce 0.0695 gn 54.75 lr 8.64e-07 56.1min eta 14min 8783tok/s mem 20GB
66
+ [train] step 1240/1545 ep 2 ce 0.0858 gn 27.00 lr 7.64e-07 57.0min eta 13min 8472tok/s mem 20GB
67
+ [train] step 1260/1545 ep 2 ce 0.0751 gn 0.00 lr 6.70e-07 57.8min eta 12min 8529tok/s mem 20GB
68
+ [train] step 1280/1545 ep 2 ce 0.0745 gn 8.31 lr 5.81e-07 58.7min eta 11min 8842tok/s mem 20GB
69
+ [train] step 1300/1545 ep 2 ce 0.1348 gn 0.00 lr 4.99e-07 59.5min eta 10min 8645tok/s mem 20GB
70
+ [train] step 1320/1545 ep 2 ce 0.0701 gn 12.94 lr 4.22e-07 60.4min eta 9min 8519tok/s mem 20GB
71
+ [train] step 1340/1545 ep 2 ce 0.0502 gn 2.12 lr 3.51e-07 61.2min eta 9min 8520tok/s mem 20GB
72
+ [train] step 1360/1545 ep 2 ce 0.1445 gn 139.00 lr 2.87e-07 62.1min eta 8min 8803tok/s mem 20GB
73
+ [train] step 1380/1545 ep 2 ce 0.1415 gn 0.04 lr 2.29e-07 62.9min eta 7min 8597tok/s mem 20GB
74
+ [train] step 1400/1545 ep 2 ce 0.1068 gn 0.04 lr 1.77e-07 63.8min eta 6min 8598tok/s mem 20GB
75
+ [train] step 1420/1545 ep 2 ce 0.0716 gn 3.88 lr 1.32e-07 64.6min eta 5min 8702tok/s mem 20GB
76
+ [train] step 1440/1545 ep 2 ce 0.0790 gn 0.01 lr 9.32e-08 65.5min eta 4min 8575tok/s mem 20GB
77
+ [train] step 1460/1545 ep 2 ce 0.1613 gn 62.25 lr 6.12e-08 66.3min eta 4min 8680tok/s mem 20GB
78
+ [train] step 1480/1545 ep 2 ce 0.0795 gn 147.00 lr 3.58e-08 67.2min eta 3min 8674tok/s mem 20GB
79
+ [train] step 1500/1545 ep 2 ce 0.1021 gn 0.10 lr 1.72e-08 68.0min eta 2min 8719tok/s mem 20GB
80
+ [train] step 1520/1545 ep 2 ce 0.0393 gn 0.01 lr 5.30e-09 68.9min eta 1min 8546tok/s mem 20GB
81
+ [train] step 1540/1545 ep 2 ce 0.1235 gn 98.50 lr 2.12e-10 69.7min eta 0min 8705tok/s mem 20GB
82
  [save] /content/runs/delta/model
83
+ [eval] coreai_v2_heldout n= 100 acc=0.810 (chance 0.30) nll=0.928 brier=0.325 ece=0.170 aurc=0.066 acc@80=0.863
84
+ [eval-agg] step 1545 {"in_task": {"acc": 0.81, "nll": 0.9282295107841492, "brier": 0.3245173692703247, "ece": 0.1702162915468216, "aurc": 0.06609745285128044, "acc_at_80": 0.8625, "chance": 0.29995238095238097}, "heldout": {}, "n_in": 1, "n_heldout": 0}
85
  [done] saved /content/runs/delta/model
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4e41928a2baea2d2ec419f72b0c0574a996c079804861e35746f97eba32eec09
3
  size 3763692048
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e450087a70068fa605028b538bf90f38be0b667abaeae2b805717089288e2fa
3
  size 3763692048