Create README.md

README.md

@@ -0,0 +1,139 @@
+# OVERFLOWGUARD Checkpoints
+
+Router checkpoints for **OVERFLOWGUARD**, a routing framework for detecting when compressed-context representations are likely to fail and selectively falling back to the full context.
+
+This repository contains trained routing classifiers for multiple compressors, language models, and training datasets.
+
+## Repository Structure
+
+Each checkpoint contains two files:
+
+```text
+{compressor}/{model}/{dataset}/
+├── router_config.json
+└── routing_clf.pt
+```
+
+Available compressors:
+
+- `xrag`
+- `pisco`
+- `oscar`
+
+Available models:
+
+- `mistral-7b`
+- `mixtral-8x7b`
+- `mistral-24b`
+- `qwen-7b`
+- `solar-7b`
+- `llama-8b`
+
+Available training datasets:
+
+- `squad`
+- `hotpotqa`
+- `triviaqa`
+- `combined`
+
+Not every compressor/model/dataset combination is available.
+
+## Loading a Checkpoint
+
+The routers can be loaded directly from the Hugging Face repository using `from_pretrained`:
+
+```python
+from your_router_module import PiscoRouter
+
+router = PiscoRouter.from_pretrained(
+    "wexumin/OVERFLOWGUARD-checkpoints",
+    hf_local_path="pisco/mistral-7b/squad",
+)
+```
+
+For other compressors, use the corresponding router class:
+
+```python
+router = XragRouter.from_pretrained(
+    "wexumin/OVERFLOWGUARD-checkpoints",
+    hf_local_path="xrag/mistral-7b/squad",
+)
+```
+
+Only the requested `router_config.json` and `routing_clf.pt` are downloaded from the repository.
+
+The underlying base language model specified in `router_config.json` is loaded separately.
+
+## Results
+
+The table below reports the performance of the released routers.
+
+### Metrics
+
+- **AUC** — ROC-AUC of the routing classifier.
+- **Acc_c → Acc_r** — accuracy before routing (`Acc_c`, compressed context) and after routing (`Acc_r`, routed between compressed and full context).
+- **ΔAcc** — absolute accuracy improvement from routing.
+- **R_c** — percentage of examples initially handled using compressed context.
+- **S_tok** — percentage of input tokens saved relative to always using the full context.
+
+### SQuADv2
+
+| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
+|---|---:|---:|---:|---:|---:|
+| xRAG · mistral-7b | 72.3 | 38.5 → **79.0** | +40.6 | 44.3 | 42.6 |
+| xRAG · mixtral-8x7b | 72.0 | 45.5 → **80.4** | +35.0 | 48.1 | 48.0 |
+| OSCAR · mistral-7b | 77.1 | 85.6 → **91.6** | +6.0 | 66.5 | 60.4 |
+| OSCAR · mistral-24b | 80.7 | 87.5 → **94.9** | +7.4 | 66.0 | 59.2 |
+| OSCAR · qwen-7b | 80.1 | 81.7 → **89.8** | +8.1 | 75.0 | 67.2 |
+| PISCO · mistral-7b | 69.2 | 76.5 → **88.8** | +12.3 | 62.1 | 56.1 |
+| PISCO · solar-7b | 64.4 | 80.0 → **88.4** | +8.4 | 68.7 | 60.8 |
+| PISCO · llama-8b | 75.1 | 77.6 → **91.2** | +13.7 | 58.0 | 51.3 |
+
+### HotpotQA
+
+| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
+|---|---:|---:|---:|---:|---:|
+| xRAG · mistral-7b | 73.6 | 50.1 → **79.5** | +29.4 | 49.6 | 48.3 |
+| xRAG · mixtral-8x7b | 74.9 | 56.4 → **79.8** | +23.4 | 56.3 | 55.1 |
+| OSCAR · mistral-7b | 81.2 | 77.0 → **86.7** | +9.7 | 61.4 | 53.5 |
+| OSCAR · mistral-24b | 82.9 | 77.9 → **87.8** | +9.9 | 73.2 | 64.3 |
+| OSCAR · qwen-7b | 80.4 | 72.1 → **84.7** | +12.6 | 60.8 | 52.5 |
+| PISCO · mistral-7b | 74.0 | 76.7 → **87.0** | +10.3 | 59.5 | 52.2 |
+| PISCO · solar-7b | 67.8 | 82.6 → **89.3** | +6.8 | 60.8 | 53.5 |
+| PISCO · llama-8b | 74.0 | 78.1 → **88.0** | +10.0 | 56.9 | 50.8 |
+
+### TriviaQA
+
+| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
+|---|---:|---:|---:|---:|---:|
+| xRAG · mistral-7b | 71.5 | 71.7 → **75.1** | +3.4 | 54.8 | 54.5 |
+| xRAG · mixtral-8x7b | 69.0 | 79.7 → **79.5** | -0.2 | 55.2 | 56.7 |
+| OSCAR · mistral-7b | 90.1 | 68.5 → **69.7** | +1.1 | 66.4 | 61.5 |
+| OSCAR · mistral-24b | 92.9 | 72.3 → **73.3** | +1.0 | 66.6 | 61.6 |
+| OSCAR · qwen-7b | 89.7 | 67.2 → **69.7** | +2.5 | 60.9 | 55.9 |
+| PISCO · mistral-7b | 70.5 | 71.1 → **72.8** | +1.7 | 54.0 | 48.6 |
+| PISCO · solar-7b | 71.8 | 71.4 → **73.4** | +2.0 | 41.5 | 37.5 |
+| PISCO · llama-8b | 68.2 | 72.3 → **71.9** | -0.4 | 54.9 | 49.9 |
+
+### Combined
+
+The combined dataset pools examples from SQuADv2, HotpotQA, and TriviaQA.
+
+| Compressor · Model | AUC | Acc_c → Acc_r | ΔAcc | R_c | S_tok |
+|---|---:|---:|---:|---:|---:|
+| xRAG · mistral-7b | 80.7 | 53.7 → **80.1** | +26.4 | 54.9 | 53.4 |
+| xRAG · mixtral-8x7b | 82.9 | 60.7 → **85.4** | +24.6 | 51.5 | 50.5 |
+| OSCAR · mistral-7b | 76.9 | 76.2 → **81.9** | +5.7 | 71.8 | 64.3 |
+| OSCAR · mistral-24b | 86.1 | 78.6 → **84.6** | +6.0 | 68.3 | 61.7 |
+| OSCAR · qwen-7b | 83.5 | 73.4 → **81.6** | +8.2 | 63.3 | 56.9 |
+| PISCO · mistral-7b | 74.1 | 74.2 → **83.1** | +8.9 | 62.8 | 55.6 |
+| PISCO · solar-7b | 72.0 | 78.0 → **85.3** | +7.4 | 56.3 | 49.3 |
+| PISCO · llama-8b | 76.1 | 76.7 → **85.1** | +8.4 | 54.0 | 47.6 |
+
+## Notes
+* Except for PISCO-Llama-8B, routers **transfer** across models, generally with only a modest few-percentage-point drop in AUC. **However**, the decision threshold does not transfer, rendering routing degenerate.
+
+* The routers are trained independently for each compressor/model/dataset configuration. The `combined` routers are trained on the combined SQuADv2, HotpotQA, and TriviaQA data rather than being transferred from a single-dataset router.
+
+* The repository contains only the router configuration and classifier checkpoint; the underlying language models are not included.
+