| # Galvanize 2 public benchmark |
|
|
| This folder reproduces the public benchmark numbers of `usezn/Galvanize-2` against open prompt-injection detectors on |
| [`Horizon-Labs/prompt-injection-eval-suite`](https://huggingface.co/datasets/Horizon-Labs/prompt-injection-eval-suite). |
|
|
| Numbers on our private held-out sets are reported in the model card and cannot be reproduced from this folder. |
|
|
| ## Files |
|
|
| | File | What it is | |
| |---|---| |
| | `run_benchmark.py` | Standalone script: downloads the suite and the models from Hugging Face, scores, writes `results.json` and `results.md`. | |
| | `results.md`, `results.json` | Our published numbers from the full reference run (both protocols, every model). | |
| | `decontam_excluded_ids.json` | Eval-suite rows removed for the decontaminated subset (set name + 0-based row index). | |
| | `horizon_published_metrics.json` | Horizon's own published per-set numbers, used only for the reproduction check of Horizon's protocol. | |
| | `requirements.txt` | Package versions. | |
|
|
| ## What is measured |
|
|
| - **Data.** `Horizon-Labs/prompt-injection-eval-suite` at revision `27c7bcd6f46e842f4c4d8b114b30eb6b1d892965`; the script |
| checks the SHA-256 of every parquet file and the row and label counts. Thirteen sets are used: the ten external sets of |
| Horizon's model-card table (NotInject, XSTest, OR-Bench-hard-1k, deepset test, jackhhao test, Simsonsun jailbreaks, |
| agentic boundary pairs test, BIPIA, PIArena, LLMail-Inject phase 2) plus three supplementary public test splits |
| (neuralchemy core, S-Labs, agentic-prompt-injection-5k). Labels are used as published: 1 = injection or jailbreak, |
| 0 = benign. Under this suite's definition a plainly harmful request without an injection is benign. |
| Horizon's `synthetic_*` sets are not used: they come from the same generator as Horizon's training data, so they are |
| in-distribution for Horizon and, because Galvanize 2 is fine-tuned from Horizon small, for Galvanize 2 too. |
| - **Models** (revision and SHA-256 of `model.safetensors` pinned in the script and checked after download): |
| Galvanize 2 (`usezn/Galvanize-2`), Horizon small, base and large (`Horizon-Labs/prompt-injection-guard-*`), |
| Galvanize-60M (`usezn/Galvanize-60M`), ProtectAI v2 (`protectai/deberta-v3-base-prompt-injection-v2`), |
| deepset (`deepset/deberta-v3-base-injection`), Wolf Defender (`patronus-studio/wolf-defender-prompt-injection`) and |
| NeuralTrust small (`NeuralTrust/prompt-guard-oss-small`). |
| - **Score.** Every model is reduced to a margin `z` with `P(injection) = sigmoid(z)`. For the two-class heads, |
| `z = logit(positive) - logit(negative)`, where the positive class is the label matching |
| `inject|jailbreak|malicious|unsafe|attack` (the script also checks it is the label used in the reference run). |
| Galvanize-60M has a single-logit head and is scored with its published calibration, |
| `z = 0.7 * (raw_logit + 17.6063)`, using the FP32 weights and the loader from its model card. |
| - **Serving protocol (primary).** The text is whitespace-flattened exactly as Galvanize 2 is served, |
| `re.sub(r"\s+", " ", text).strip()`, tokenised with each model's own tokenizer, and only the **first 512 tokens** are |
| scored, in one window. fp32, TF32 off. |
| - **Horizon's protocol (secondary, `--protocol hz` or `both`).** Horizon's `evaluate.py` windowing: raw text, windows of |
| `min(2048, max positions)` tokens (512 for the DeBERTa models and Galvanize-60M), stride of a quarter window, document |
| score = max over windows. Run in fp32. |
| - **Thresholds.** Galvanize 2 at **0.662** (its development-set threshold; exact value 0.661988) and also at **0.5**; |
| Galvanize-60M at 0.95 (its published calibrated threshold); every other model at 0.5, its default. |
| - **Metrics.** As in Horizon's card: accuracy for the all-benign sets, recall for the all-attack sets, F1 (shown with the |
| false-positive rate) otherwise. ROC AUC and normalised partial AUC up to 10% FPR are reported as threshold-free |
| metrics. The macro average is the mean over the ten card sets. |
| - **Decontaminated subset (primary).** Rows whose text matches a row of the Galvanize 2 training data are removed (see |
| below). The full sets are reported as a secondary view. |
|
|
| ## Decontamination |
|
|
| The reference run hashed every eval row and every Galvanize 2 training row with three keys: SHA-256 of the exact text, |
| SHA-256 of the whitespace-flattened lowercased text, and SHA-256 of the first 200 characters of that normalised text |
| (texts of 200 characters or more). An eval row is excluded if any key matches. The training data is private, so this |
| match cannot be recomputed here. `decontam_excluded_ids.json` records its outcome as **row ids**: for each set, the |
| 0-based row positions in `data/<set>.parquet` at the pinned revision (the suite has no id column; the pinned SHA-256 keeps |
| the positions stable). It contains no text and no hashes. |
|
|
| 275 rows are excluded in total: jackhhao test 126 (of 262), Simsonsun jailbreaks 131 (of 2,426), XSTest 9, OR-Bench-hard-1k |
| 7, neuralchemy core test 2; no rows in the other eight sets. Per-set counts are in the file and in `results.md`. |
|
|
| `results.md` also lists, for information only, the overlap with the public *train* splits of the sources Horizon lists |
| as training data (Horizon's own training set is not published). That overlap is not used to filter rows. |
|
|
| ## Why our numbers for other models differ from their own model cards |
|
|
| - **Different subset.** The primary numbers drop the rows above; the cards use full sets (or different sets). |
| - **Different input.** The serving protocol scores only the first 512 tokens of the flattened text. Horizon's card scores |
| sliding windows of up to 2,048 tokens and keeps the maximum, which changes the long indirect sets (BIPIA, PIArena, |
| LLMail-Inject) the most. Both protocols are in `results.md`. |
| - **Different thresholds.** Every third-party model is scored at 0.5; a vendor's recommended operating point may differ. |
| - **Different precision.** We run fp32; Horizon's published numbers use bf16 autocast. Our run of Horizon's protocol |
| matches Horizon's published numbers within 0.009 accuracy and 0.002 ROC AUC per set (reproduction check in |
| `results.md`). |
| - **Different macro average.** Our macro covers ten sets. Horizon's card also averages Qualifire and two Mindgard sets, |
| which are gated and non-commercial and are not in the public suite. |
| - **Different label definition.** This suite labels plain harmful requests as benign. Detectors that also target |
| jailbreaks or harmful content can score lower on XSTest and OR-Bench by design. |
| - **Galvanize-60M** is served as an INT8 ONNX graph; here it is scored with its FP32 weights. In the reference run the two |
| agreed on 29 of 32 decisions in a spot check, so small differences from its own card are expected. |
|
|
| ## How to run |
|
|
| ```bash |
| python -m venv .venv && . .venv/bin/activate |
| pip install -r requirements.txt |
| python run_benchmark.py --limit 24 --models galvanize-2,horizon-small # smoke test, a few minutes |
| python run_benchmark.py # every model, serving protocol |
| python run_benchmark.py --protocol both # also Horizon's protocol |
| ``` |
|
|
| Options: `--models` (comma-separated keys: `galvanize-2`, `horizon-small`, `horizon-base`, `horizon-large`, |
| `galvanize-60m`, `protectai-v2`, `deepset`, `wolf-defender`, `neuraltrust-small`), `--device cpu|cuda` (default: CUDA if |
| available), `--limit N` (smoke test: up to N/2 attack and N/2 benign rows per set, fixed seed), `--protocol |
| serve|hz|both`, `--out-dir` (default `run_output/`, so the published files are not overwritten), |
| `--galvanize2-revision`. |
| |
| The script writes `run_output/results.json` and `run_output/results.md` in the same format as the published files. On a |
| full run it also compares every metric with the published `results.json` and reports the largest difference per model. |
| It refuses to score a model whose `model.safetensors` differs from the benchmarked file, and it never prints benchmark |
| text. No Hugging Face token is needed: every model and the dataset are public and ungated. |
| |
| ### Hardware and expected runtime |
| |
| - **GPU (recommended).** The reference run used one NVIDIA L4 (24 GB). Scoring all nine models took about 17 minutes |
| for the serving protocol and about 26 minutes for Horizon's protocol, plus downloads. Expect roughly 25 to 30 minutes |
| for the default run and 45 to 55 minutes for `--protocol both`. Any CUDA GPU with 16 GB or more should work. |
| - **CPU.** Works, but slowly: the suite is about 15,400 rows and 2.7 million tokens per model. As a rough estimate, |
| several hours for all models on a recent 16-core x86 machine (Horizon large is the largest share); 16 GB of RAM is |
| enough. CPU and GPU fp32 kernels differ slightly, so a few borderline rows can flip; expect differences of about 0.001. |
| - **Disk.** About 8 GB of model downloads, cached by `huggingface_hub`. |
|
|
| ## What this folder does not reproduce |
|
|
| - The decontamination match itself (it needs our private training data); the excluded row ids are provided instead. |
| - Numbers on our private held-out sets (see the line at the top). |
| - Models not evaluated: `leolee99/PIGuard` (needs `trust_remote_code`), `meta-llama/Llama-Prompt-Guard-2-86M` and `-22M` |
| (gated), and commercial APIs. The Qualifire and Mindgard datasets are gated and not in the public suite. |
|
|
| ## License |
|
|
| `run_benchmark.py` and the other files in this folder are released under the Apache License 2.0. The eval suite is not |
| redistributed here (only row ids); it is governed by its own licenses, which differ per subset: see the |
| [dataset card](https://huggingface.co/datasets/Horizon-Labs/prompt-injection-eval-suite). Each model is under the |
| license on its own model card. |
|
|
