File size: 2,221 Bytes
36a1890 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | ---
license: apache-2.0
base_model: allenai/OLMo-2-1124-7B-Instruct
tags: [tokenization, robustness, safety, adversarial-tokenization]
---
# inv_mse_ref_multi
Research checkpoint from a study of **tokenization (reader) invariance** and its
effect on robustness to adversarial re-tokenization. Fine-tuned from
`allenai/OLMo-2-1124-7B-Instruct`.
**These are research artifacts, not products.** Numbers below are measured on a
200-prompt AdvBench holdout that was excluded from training by construction.
## Measured
| metric | value |
| --- | --- |
| AdvTok ASR (greedy, 200-prompt AdvBench holdout) | 0.055 |
| AdvTok ASR (t=1) | 0.141 |
| Canonical ASR (greedy, no attack) | 0.045 |
| Canonical refusal (greedy) | 0.91 |
| XSTest over-refusal (safe) | 0.07 |
| XSTest refusal (unsafe) | 0.882 |
| Alpaca over-refusal | 0.018 |
| Alpaca token F1 | 0.424 |
| Alpaca NLL/token | 1.353 |
| Training cost (GPU-hours, 2xA100) | 20.4 |
| Reference (zeroshot) AdvTok ASR greedy / t=1 | 0.565 / 0.584 |
`AdvTok ASR` is attack success rate under adversarial tokenization
(Geh et al., arXiv:2503.02174), Llama-Guard-3-8B judged, greedy decoding.
`XSTest over-refusal` is the refusal rate on *safe* prompts — the cost side.
## Training configuration
| field | value |
| --- | --- |
| `mode` | `reference` |
| `objective` | `mse_lastpos` |
| `kl_direction` | `forward` |
| `ce_weighting` | `uniform` |
| `ema_beta` | `None` |
| `harmful_mix` | `mixed` |
| `harmful_fraction` | `0.286` |
| `num_encodings` | `8` |
| `cvar_quantile` | `0.25` |
| `max_steps` | `700` |
| `learning_rate` | `1e-05` |
| `grad_accum` | `8` |
| `seed` | `42` |
| `reference_model` | `allenai/OLMo-2-1124-7B-Instruct` |
| `max_new_tokens` | `128` |
| `prefix_tokens` | `8` |
| `stochastok_p` | `0.3` |
## Parameter drift from base (training-happened guard)
| group | relative L2 |
| --- | --- |
| attn | 0.01047 |
| embed_tokens | 0.00091 |
| lm_head | 0.00000 |
| mlp | 0.01137 |
| norm | 0.00421 |
## Caveats
- Single seed. No claim of significance across seeds.
- Evaluated on English AdvBench/XSTest/Alpaca only.
- The safety numbers are for the *specific* attack studied; they do not imply
robustness to other jailbreaks.
|