# Galvanize 2 public benchmark This folder reproduces the public benchmark numbers of `usezn/Galvanize-2` against open prompt-injection detectors on [`Horizon-Labs/prompt-injection-eval-suite`](https://huggingface.co/datasets/Horizon-Labs/prompt-injection-eval-suite). Numbers on our private held-out sets are reported in the model card and cannot be reproduced from this folder. ## Files | File | What it is | |---|---| | `run_benchmark.py` | Standalone script: downloads the suite and the models from Hugging Face, scores, writes `results.json` and `results.md`. | | `results.md`, `results.json` | Our published numbers from the full reference run (both protocols, every model). | | `decontam_excluded_ids.json` | Eval-suite rows removed for the decontaminated subset (set name + 0-based row index). | | `horizon_published_metrics.json` | Horizon's own published per-set numbers, used only for the reproduction check of Horizon's protocol. | | `requirements.txt` | Package versions. | ## What is measured - **Data.** `Horizon-Labs/prompt-injection-eval-suite` at revision `27c7bcd6f46e842f4c4d8b114b30eb6b1d892965`; the script checks the SHA-256 of every parquet file and the row and label counts. Thirteen sets are used: the ten external sets of Horizon's model-card table (NotInject, XSTest, OR-Bench-hard-1k, deepset test, jackhhao test, Simsonsun jailbreaks, agentic boundary pairs test, BIPIA, PIArena, LLMail-Inject phase 2) plus three supplementary public test splits (neuralchemy core, S-Labs, agentic-prompt-injection-5k). Labels are used as published: 1 = injection or jailbreak, 0 = benign. Under this suite's definition a plainly harmful request without an injection is benign. Horizon's `synthetic_*` sets are not used: they come from the same generator as Horizon's training data, so they are in-distribution for Horizon and, because Galvanize 2 is fine-tuned from Horizon small, for Galvanize 2 too. - **Models** (revision and SHA-256 of `model.safetensors` pinned in the script and checked after download): Galvanize 2 (`usezn/Galvanize-2`), Horizon small, base and large (`Horizon-Labs/prompt-injection-guard-*`), Galvanize-60M (`usezn/Galvanize-60M`), ProtectAI v2 (`protectai/deberta-v3-base-prompt-injection-v2`), deepset (`deepset/deberta-v3-base-injection`), Wolf Defender (`patronus-studio/wolf-defender-prompt-injection`) and NeuralTrust small (`NeuralTrust/prompt-guard-oss-small`). - **Score.** Every model is reduced to a margin `z` with `P(injection) = sigmoid(z)`. For the two-class heads, `z = logit(positive) - logit(negative)`, where the positive class is the label matching `inject|jailbreak|malicious|unsafe|attack` (the script also checks it is the label used in the reference run). Galvanize-60M has a single-logit head and is scored with its published calibration, `z = 0.7 * (raw_logit + 17.6063)`, using the FP32 weights and the loader from its model card. - **Serving protocol (primary).** The text is whitespace-flattened exactly as Galvanize 2 is served, `re.sub(r"\s+", " ", text).strip()`, tokenised with each model's own tokenizer, and only the **first 512 tokens** are scored, in one window. fp32, TF32 off. - **Horizon's protocol (secondary, `--protocol hz` or `both`).** Horizon's `evaluate.py` windowing: raw text, windows of `min(2048, max positions)` tokens (512 for the DeBERTa models and Galvanize-60M), stride of a quarter window, document score = max over windows. Run in fp32. - **Thresholds.** Galvanize 2 at **0.662** (its development-set threshold; exact value 0.661988) and also at **0.5**; Galvanize-60M at 0.95 (its published calibrated threshold); every other model at 0.5, its default. - **Metrics.** As in Horizon's card: accuracy for the all-benign sets, recall for the all-attack sets, F1 (shown with the false-positive rate) otherwise. ROC AUC and normalised partial AUC up to 10% FPR are reported as threshold-free metrics. The macro average is the mean over the ten card sets. - **Decontaminated subset (primary).** Rows whose text matches a row of the Galvanize 2 training data are removed (see below). The full sets are reported as a secondary view. ## Decontamination The reference run hashed every eval row and every Galvanize 2 training row with three keys: SHA-256 of the exact text, SHA-256 of the whitespace-flattened lowercased text, and SHA-256 of the first 200 characters of that normalised text (texts of 200 characters or more). An eval row is excluded if any key matches. The training data is private, so this match cannot be recomputed here. `decontam_excluded_ids.json` records its outcome as **row ids**: for each set, the 0-based row positions in `data/.parquet` at the pinned revision (the suite has no id column; the pinned SHA-256 keeps the positions stable). It contains no text and no hashes. 275 rows are excluded in total: jackhhao test 126 (of 262), Simsonsun jailbreaks 131 (of 2,426), XSTest 9, OR-Bench-hard-1k 7, neuralchemy core test 2; no rows in the other eight sets. Per-set counts are in the file and in `results.md`. `results.md` also lists, for information only, the overlap with the public *train* splits of the sources Horizon lists as training data (Horizon's own training set is not published). That overlap is not used to filter rows. ## Why our numbers for other models differ from their own model cards - **Different subset.** The primary numbers drop the rows above; the cards use full sets (or different sets). - **Different input.** The serving protocol scores only the first 512 tokens of the flattened text. Horizon's card scores sliding windows of up to 2,048 tokens and keeps the maximum, which changes the long indirect sets (BIPIA, PIArena, LLMail-Inject) the most. Both protocols are in `results.md`. - **Different thresholds.** Every third-party model is scored at 0.5; a vendor's recommended operating point may differ. - **Different precision.** We run fp32; Horizon's published numbers use bf16 autocast. Our run of Horizon's protocol matches Horizon's published numbers within 0.009 accuracy and 0.002 ROC AUC per set (reproduction check in `results.md`). - **Different macro average.** Our macro covers ten sets. Horizon's card also averages Qualifire and two Mindgard sets, which are gated and non-commercial and are not in the public suite. - **Different label definition.** This suite labels plain harmful requests as benign. Detectors that also target jailbreaks or harmful content can score lower on XSTest and OR-Bench by design. - **Galvanize-60M** is served as an INT8 ONNX graph; here it is scored with its FP32 weights. In the reference run the two agreed on 29 of 32 decisions in a spot check, so small differences from its own card are expected. ## How to run ```bash python -m venv .venv && . .venv/bin/activate pip install -r requirements.txt python run_benchmark.py --limit 24 --models galvanize-2,horizon-small # smoke test, a few minutes python run_benchmark.py # every model, serving protocol python run_benchmark.py --protocol both # also Horizon's protocol ``` Options: `--models` (comma-separated keys: `galvanize-2`, `horizon-small`, `horizon-base`, `horizon-large`, `galvanize-60m`, `protectai-v2`, `deepset`, `wolf-defender`, `neuraltrust-small`), `--device cpu|cuda` (default: CUDA if available), `--limit N` (smoke test: up to N/2 attack and N/2 benign rows per set, fixed seed), `--protocol serve|hz|both`, `--out-dir` (default `run_output/`, so the published files are not overwritten), `--galvanize2-revision`. The script writes `run_output/results.json` and `run_output/results.md` in the same format as the published files. On a full run it also compares every metric with the published `results.json` and reports the largest difference per model. It refuses to score a model whose `model.safetensors` differs from the benchmarked file, and it never prints benchmark text. No Hugging Face token is needed: every model and the dataset are public and ungated. ### Hardware and expected runtime - **GPU (recommended).** The reference run used one NVIDIA L4 (24 GB). Scoring all nine models took about 17 minutes for the serving protocol and about 26 minutes for Horizon's protocol, plus downloads. Expect roughly 25 to 30 minutes for the default run and 45 to 55 minutes for `--protocol both`. Any CUDA GPU with 16 GB or more should work. - **CPU.** Works, but slowly: the suite is about 15,400 rows and 2.7 million tokens per model. As a rough estimate, several hours for all models on a recent 16-core x86 machine (Horizon large is the largest share); 16 GB of RAM is enough. CPU and GPU fp32 kernels differ slightly, so a few borderline rows can flip; expect differences of about 0.001. - **Disk.** About 8 GB of model downloads, cached by `huggingface_hub`. ## What this folder does not reproduce - The decontamination match itself (it needs our private training data); the excluded row ids are provided instead. - Numbers on our private held-out sets (see the line at the top). - Models not evaluated: `leolee99/PIGuard` (needs `trust_remote_code`), `meta-llama/Llama-Prompt-Guard-2-86M` and `-22M` (gated), and commercial APIs. The Qualifire and Mindgard datasets are gated and not in the public suite. ## License `run_benchmark.py` and the other files in this folder are released under the Apache License 2.0. The eval suite is not redistributed here (only row ids); it is governed by its own licenses, which differ per subset: see the [dataset card](https://huggingface.co/datasets/Horizon-Labs/prompt-injection-eval-suite). Each model is under the license on its own model card.