File size: 1,608 Bytes
b65d05b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | # RL Agent evaluation
Metrics after calibration. Zero-shot = task families held out of training entirely.
## In-task
| task family | questions | accuracy | ECE | NLL |
|---|---|---|---|---|
| conversation outcomes | 3600 | 0.482 | 0.019 | 0.693 |
| email triage | 2691 | 0.732 | 0.017 | 0.595 |
| emotion and tone | 1825 | 0.906 | 0.018 | 0.238 |
| inference and fact checking | 3022 | 0.883 | 0.054 | 0.340 |
| instruction-following tasks | 600 | 0.878 | 0.046 | 0.302 |
| intent and routing | 1475 | 0.991 | 0.009 | 0.181 |
| moderation and safety | 2708 | 0.967 | 0.061 | 0.153 |
| reading comprehension | 770 | 0.847 | 0.083 | 0.409 |
| response quality scoring | 3146 | 0.581 | 0.023 | 1.009 |
| robustness checks | 744 | 0.851 | 0.108 | 1.058 |
| search relevance | 733 | 0.628 | 0.066 | 0.728 |
| sentiment and rating | 961 | 0.442 | 0.438 | 3.545 |
| topic classification | 749 | 0.939 | 0.029 | 0.196 |
Overall: accuracy 0.753, ECE 0.030, Brier 0.308, accuracy at 50% coverage 0.947
## Zero-shot
| task family | questions | accuracy | ECE | NLL |
|---|---|---|---|---|
| emotion and tone | 600 | 0.583 | 0.318 | 1.976 |
| instruction-following tasks | 600 | 0.863 | 0.045 | 0.319 |
| moderation and safety | 600 | 0.797 | 0.171 | 1.415 |
| sentiment and rating | 600 | 0.362 | 0.291 | 1.798 |
Overall: accuracy 0.651, ECE 0.204, Brier 0.532, accuracy at 50% coverage 0.818
## Latency
```
{
"1_questions": {
"p50_ms": 38.4,
"p95_ms": 42.1
},
"10_questions": {
"p50_ms": 156.0,
"p95_ms": 158.4
},
"50_questions": {
"p50_ms": 721.4,
"p95_ms": 733.0
}
}
```
|