# OVERFLOWGUARD Checkpoints

Router checkpoints for **OVERFLOWGUARD**, a routing framework for detecting when compressed-context representations are likely to fail and selectively falling back to the full context.

This repository contains trained routing classifiers for multiple compressors, language models, and training datasets.

## Repository Structure

Each checkpoint contains two files:

```text
{compressor}/{model}/{dataset}/
├── router_config.json
└── routing_clf.pt
```

Available compressors:

- `xrag`
- `pisco`
- `oscar`

Available models:

- `mistral-7b`
- `mixtral-8x7b`
- `mistral-24b`
- `qwen-7b`
- `solar-7b`
- `llama-8b`

Available training datasets:

- `squad`
- `hotpotqa`
- `triviaqa`
- `combined`

Not every compressor/model/dataset combination is available.

## Loading a Checkpoint

The implementation of the `PiscoRouter` class, along with two other router classes, is available in our [Git repository](https://github.com/s-nlp/overflowguard/tree/main/examples).
The routers can be loaded directly from the Hugging Face repository using `from_pretrained`:

```python
from train_pisco import PiscoRouter

router = PiscoRouter.from_pretrained(
    "s-nlp/OVERFLOWGUARD-checkpoints",
    hf_local_path="pisco/mistral-7b/squad",
)
```

For other compressors, use the corresponding router class:

```python
router = XragRouter.from_pretrained(
    "s-nlp/OVERFLOWGUARD-checkpoints",
    hf_local_path="xrag/mistral-7b/squad",
)
```

Only the requested `router_config.json` and `routing_clf.pt` are downloaded from the repository.

The underlying base language model specified in `router_config.json` is loaded separately.

## Results

The table below reports the performance of the released routers.

### Metrics

- **AUC** — ROC-AUC of the routing classifier.
- **Acc_c → Acc_r** — accuracy before routing (`Acc_c`, compressed context) and after routing (`Acc_r`, routed between compressed and full context).
- **ΔAcc** — absolute accuracy improvement from routing.
- **R_c** — percentage of examples initially handled using compressed context.
- **S_tok** — percentage of input tokens saved relative to always using the full context.

### SQuADv2

| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
|---|---:|---:|---:|---:|---:|
| xRAG · mistral-7b | 72.3 | 38.5 → **79.0** | +40.6 | 44.3 | 42.6 |
| xRAG · mixtral-8x7b | 72.0 | 45.5 → **80.4** | +35.0 | 48.1 | 48.0 |
| OSCAR · mistral-7b | 77.1 | 85.6 → **91.6** | +6.0 | 66.5 | 60.4 |
| OSCAR · mistral-24b | 80.7 | 87.5 → **94.9** | +7.4 | 66.0 | 59.2 |
| OSCAR · qwen-7b | 80.1 | 81.7 → **89.8** | +8.1 | 75.0 | 67.2 |
| PISCO · mistral-7b | 69.2 | 76.5 → **88.8** | +12.3 | 62.1 | 56.1 |
| PISCO · solar-7b | 64.4 | 80.0 → **88.4** | +8.4 | 68.7 | 60.8 |
| PISCO · llama-8b | 75.1 | 77.6 → **91.2** | +13.7 | 58.0 | 51.3 |

### HotpotQA

| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
|---|---:|---:|---:|---:|---:|
| xRAG · mistral-7b | 73.6 | 50.1 → **79.5** | +29.4 | 49.6 | 48.3 |
| xRAG · mixtral-8x7b | 74.9 | 56.4 → **79.8** | +23.4 | 56.3 | 55.1 |
| OSCAR · mistral-7b | 81.2 | 77.0 → **86.7** | +9.7 | 61.4 | 53.5 |
| OSCAR · mistral-24b | 82.9 | 77.9 → **87.8** | +9.9 | 73.2 | 64.3 |
| OSCAR · qwen-7b | 80.4 | 72.1 → **84.7** | +12.6 | 60.8 | 52.5 |
| PISCO · mistral-7b | 74.0 | 76.7 → **87.0** | +10.3 | 59.5 | 52.2 |
| PISCO · solar-7b | 67.8 | 82.6 → **89.3** | +6.8 | 60.8 | 53.5 |
| PISCO · llama-8b | 74.0 | 78.1 → **88.0** | +10.0 | 56.9 | 50.8 |

### TriviaQA

| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
|---|---:|---:|---:|---:|---:|
| xRAG · mistral-7b | 71.5 | 71.7 → **75.1** | +3.4 | 54.8 | 54.5 |
| xRAG · mixtral-8x7b | 69.0 | 79.7 → **79.5** | -0.2 | 55.2 | 56.7 |
| OSCAR · mistral-7b | 90.1 | 68.5 → **69.7** | +1.1 | 66.4 | 61.5 |
| OSCAR · mistral-24b | 92.9 | 72.3 → **73.3** | +1.0 | 66.6 | 61.6 |
| OSCAR · qwen-7b | 89.7 | 67.2 → **69.7** | +2.5 | 60.9 | 55.9 |
| PISCO · mistral-7b | 70.5 | 71.1 → **72.8** | +1.7 | 54.0 | 48.6 |
| PISCO · solar-7b | 71.8 | 71.4 → **73.4** | +2.0 | 41.5 | 37.5 |
| PISCO · llama-8b | 68.2 | 72.3 → **71.9** | -0.4 | 54.9 | 49.9 |

### Combined

The combined dataset pools examples from SQuADv2, HotpotQA, and TriviaQA.

| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
|---|---:|---:|---:|---:|---:|
| xRAG · mistral-7b | 80.7 | 53.7 → **80.1** | +26.4 | 54.9 | 53.4 |
| xRAG · mixtral-8x7b | 82.9 | 60.7 → **85.4** | +24.6 | 51.5 | 50.5 |
| OSCAR · mistral-7b | 76.9 | 76.2 → **81.9** | +5.7 | 71.8 | 64.3 |
| OSCAR · mistral-24b | 86.1 | 78.6 → **84.6** | +6.0 | 68.3 | 61.7 |
| OSCAR · qwen-7b | 83.5 | 73.4 → **81.6** | +8.2 | 63.3 | 56.9 |
| PISCO · mistral-7b | 74.1 | 74.2 → **83.1** | +8.9 | 62.8 | 55.6 |
| PISCO · solar-7b | 72.0 | 78.0 → **85.3** | +7.4 | 56.3 | 49.3 |
| PISCO · llama-8b | 76.1 | 76.7 → **85.1** | +8.4 | 54.0 | 47.6 |

## Notes
* Except for PISCO-Llama-8B, routers **transfer** across models, generally with only a modest few-percentage-point drop in AUC. **However**, the decision threshold does not transfer, rendering routing degenerate.

* The routers are trained independently for each compressor/model/dataset configuration. The `combined` routers are trained on the combined SQuADv2, HotpotQA, and TriviaQA data rather than being transferred from a single-dataset router.

* The repository contains only the router configuration and classifier checkpoint; the underlying language models are not included.