# rei-v1: evaluation

Accuracy means agreeing with the teacher ensemble's answer, on research questions never seen in training.
Temperatures: choice 1.55, noul 1.8. Trained on 74855 items, 3.0 epochs, 10478s.

**Test: 76.7% accuracy** (majority baseline 42.8%), calibration error (ECE) 0.031.
Validation: 79.0%.

## Test by task
| | items | accuracy | majority baseline | soft CE |
|---|---|---|---|---|
| claim_check | 1085 | 81.4% | 33.9% | 0.537 |
| custom | 421 | 72.0% | 15.0% | 0.674 |
| needs_fresh | 181 | 82.9% | 60.2% | 0.375 |
| next_step | 177 | 76.3% | 56.5% | 0.686 |
| relevance | 1041 | 68.5% | 48.2% | 0.781 |
| same_info | 180 | 88.3% | 86.1% | 0.323 |
| search_type | 173 | 76.9% | 31.8% | 0.767 |
| source_type | 347 | 80.1% | 18.2% | 0.647 |
| worth_opening | 363 | 79.6% | 78.5% | 0.467 |

Score tasks, mean absolute error in scale points: custom 0.50, relevance 0.33

Noul tasks, AUC (how well P(true) ranks true above false; 0.5 is chance): custom 0.83, needs_fresh 0.91, same_info 0.83, worth_opening 0.76

## Test by language
| | items | accuracy | majority baseline | soft CE |
|---|---|---|---|---|
| ar | 166 | 79.5% | 43.4% | 0.602 |
| de | 254 | 80.3% | 50.4% | 0.563 |
| en | 374 | 77.0% | 39.0% | 0.627 |
| es | 322 | 79.8% | 44.4% | 0.555 |
| fr | 275 | 80.4% | 44.7% | 0.550 |
| hi | 140 | 75.7% | 43.6% | 0.654 |
| id | 402 | 74.4% | 42.8% | 0.661 |
| is | 474 | 69.6% | 42.0% | 0.764 |
| ja | 468 | 78.4% | 41.0% | 0.575 |
| ko | 301 | 75.7% | 42.9% | 0.608 |
| pt | 306 | 75.5% | 43.8% | 0.635 |
| ru | 212 | 75.0% | 42.9% | 0.631 |
| zh | 274 | 80.7% | 40.1% | 0.536 |

## Test by profile
| | items | accuracy | majority baseline | soft CE |
|---|---|---|---|---|
| academic | 742 | 77.8% | 41.6% | 0.591 |
| anime | 976 | 78.4% | 38.7% | 0.570 |
| general | 1486 | 74.4% | 44.5% | 0.684 |
| programming | 764 | 78.0% | 45.9% | 0.580 |