eval/injection_eval.md

# injection evaluation

**Threshold 0.02 (calibrated on the validation split, objective recall_at_fpr).** The config default of 0.5 scored 0.986 against 0.991 for the calibrated value, on validation. Every table below is on test, at the calibrated threshold.

> the chosen threshold 0.02 is the lowest value in the sweep, so the scores are compressed toward zero and this is underfitting rather than a calibration success. Check average precision before trusting the F1.

## Per language

| Language | Support | P | R | F1 | Notes |
|---|---|---|---|---|---|
| `bg` Bulgarian | 41 | 1.000 | 1.000 | 1.000 |  |
| `cs` Czech | 42 | 0.955 | 1.000 | 0.977 |  |
| `da` Danish | 41 | 0.976 | 1.000 | 0.988 |  |
| `de` German | 42 | 1.000 | 1.000 | 1.000 |  |
| `el` Greek | 42 | 1.000 | 1.000 | 1.000 |  |
| `en` English | 41 | 1.000 | 1.000 | 1.000 |  |
| `es` Spanish | 40 | 0.976 | 1.000 | 0.988 |  |
| `et` Estonian | 41 | 1.000 | 1.000 | 1.000 |  |
| `fi` Finnish | 42 | 1.000 | 1.000 | 1.000 |  |
| `fr` French | 41 | 1.000 | 1.000 | 1.000 |  |
| `ga` Irish | 42 | 0.976 | 0.976 | 0.976 |  |
| `hr` Croatian | 42 | 0.977 | 1.000 | 0.988 |  |
| `hu` Hungarian | 42 | 0.955 | 1.000 | 0.977 |  |
| `it` Italian | 42 | 1.000 | 1.000 | 1.000 |  |
| `lt` Lithuanian | 42 | 1.000 | 1.000 | 1.000 |  |
| `lv` Latvian | 41 | 0.976 | 1.000 | 0.988 |  |
| `mt` Maltese | 42 | 0.804 | 0.976 | 0.882 | not in base model pretraining |
| `nl` Dutch | 41 | 1.000 | 0.976 | 0.988 |  |
| `pl` Polish | 41 | 1.000 | 1.000 | 1.000 |  |
| `pt` Portuguese | 42 | 0.977 | 1.000 | 0.988 |  |
| `ro` Romanian | 42 | 0.977 | 1.000 | 0.988 |  |
| `sk` Slovak | 41 | 1.000 | 1.000 | 1.000 |  |
| `sl` Slovenian | 42 | 0.977 | 1.000 | 0.988 |  |
| `sv` Swedish | 41 | 1.000 | 1.000 | 1.000 |  |
| `tr` Turkish | 42 | 1.000 | 1.000 | 1.000 |  |
| `az` Azerbaijani | 42 | 1.000 | 1.000 | 1.000 |  |

The base-model note is a fact about pretraining, not a cause of the score beside it. `nsfw` Maltese carried the same note at 0.000 and reached 1.000 on corpus size alone, with nothing about the base model changed. Check how many examples a weak score rests on before reaching for this.

## Per register

| Register | Support | P | R | F1 | FPR |
|---|---|---|---|---|---|
| `encoded_payload` | 155 | 1.000 | 1.000 | 1.000 | 0.000 |
| `hypothetical_framing` | 156 | 1.000 | 0.994 | 0.997 | 0.000 |
| `ignore_instructions` | 155 | 1.000 | 1.000 | 1.000 | 0.000 |
| `instruction_in_document` | 155 | 1.000 | 0.994 | 0.997 | 0.000 |
| `instruction_in_tool_output` | 148 | 1.000 | 0.993 | 0.997 | 0.000 |
| `meta_question` | 0 | 0.000 | 0.000 | 0.000 | 0.012 |
| `mundane_account_access` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `mundane_informational` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `mundane_operational` | 0 | 0.000 | 0.000 | 0.000 | 0.005 |
| `mundane_transactional` | 0 | 0.000 | 0.000 | 0.000 | 0.005 |
| `ordinary_instruction` | 0 | 0.000 | 0.000 | 0.000 | 0.000 |
| `ordinary_question` | 0 | 0.000 | 0.000 | 0.000 | 0.006 |
| `persona_override` | 155 | 1.000 | 1.000 | 1.000 | 0.000 |
| `quoted_attack` | 0 | 0.000 | 0.000 | 0.000 | 0.009 |
| `roleplay_benign` | 0 | 0.000 | 0.000 | 0.000 | 0.012 |
| `security_discussion` | 0 | 0.000 | 0.000 | 0.000 | 0.009 |
| `system_prompt_extraction` | 156 | 1.000 | 1.000 | 1.000 | 0.000 |
| `technical_identifiers` | 0 | 0.000 | 0.000 | 0.000 | 0.006 |
| `technical_payload` | 0 | 0.000 | 0.000 | 0.000 | 0.006 |

## Known weaknesses

The three weakest languages by F1: `mt` at 0.882, `ga` at 0.976, `cs` at 0.977.

These are published rather than dropped. A coverage table with the bad rows removed is not a coverage table.