# Benchmark Protocol

The benchmark scores [`gliner2.5-PII-code-en`](../README.md) against Fastino baselines and TypeSafe’s Jev on 8,688 SPY documents and 603 code documents. Every protocol pin, revision and SHA-256 hash is in [`release_integrity.json`](https://huggingface.co/datasets/nyuuzyou/gliner2.5-PII-code-en-data/blob/main/evidence/benchmark/release_integrity.json), and [the results page](results.md) holds the full score tables.

## Compared Systems

| System | Source | Parameters | Architecture |
| --- | --- | --- | --- |
| Fastino base | `fastino/gliner2.5-base-v1` | 193M | `BoundaryExtractor` |
| Fastino PII | `fastino/gliner2-privacy-filter-PII-multi` | 307M | `SpanExtractor` |
| `gliner2.5-PII-code-en` | `nyuuzyou/gliner2.5-PII-code-en` | 193M | `BoundaryExtractor` |
| Jev | `typesafe-ai/jev` | not published | hosted API |

Fastino PII uses a span head capped at 8 tokens, while the boundary models have no width cap.

## Frozen Inputs

SPY is an English PII benchmark from [Savkin et al. (2025)](https://aclanthology.org/2025.naacl-srw.23/) with 4,197 legal and 4,491 medical documents (58,794 gold spans). Its upstream loader fills placeholders with Faker after an unseeded shuffle. This release rebuilt the text from [`mks-logic/SPY`](https://huggingface.co/datasets/mks-logic/SPY/tree/093be5a0235c9e15bc1c364f10ac48b9563de34c) with Faker seed 42 and froze `benchmarks/spy.jsonl`.

The code suite (`benchmarks/code.jsonl`) has 603 documents and 1,006 gold spans across source files, configs, logs and coding-agent transcripts:

| Stratum | Documents | Role |
| --- | --- | --- |
| Mixed development | 199 | Live secrets beside placeholders; reused for checkpoint selection |
| Explicit placeholder | 160 | Placeholders like `<substitute-vendor-credential>` that must stay silent |
| Live context | 80 | Live values marked as active |
| Example context | 80 | Matching values framed as examples |
| Clean code | 40 | Code without sensitive values |
| Ambiguous | 44 | Unclear cases, reported without error scoring |

## Inference Settings

Each GLiNER2 checkpoint ran every document in a single window (batch size 8, float32 weights) with this model’s label descriptions. Extraction kept spans from confidence 0.05 upward and applied thresholds during scoring.

Prose (`author7`) label mapping:

| Label | Scored type |
| --- | --- |
| `author email` | `EMAIL` |
| `author name` | `NAME` |
| `author url` | `URL` |
| `author phone number` | `PHONE_NUM` |
| `author id number` | `ID_NUM` |
| `author address` | `ADDRESS` |
| `author username` | `USERNAME` |

Code (`code`) secret and identity mapping:

| Label | Scored type |
| --- | --- |
| `api key` | `API_KEY` |
| `auth token` | `TOKEN` |
| `password` | `PASSWORD` |
| `connection string` | `CONN_STRING` |
| `private key` | `PRIVATE_KEY` |
| `personal file path` | `USER_PATH` |
| `internal host` | `HOST` |
| `developer email` | `DEV_EMAIL` |
| `developer name` | `DEV_NAME` |
| `developer username` | `DEV_USERNAME` |

## Thresholds and Scoring

The primary comparison uses the common threshold 0.5, with secondary thresholds 0.9, 0.99 and 0.995 reported alongside it. SPY feedback set the prose threshold of 0.995.

SPY scores exact typed spans after trimming trailing punctuation from valid predictions, while invalid offsets stay false positives. Because the GLiNER2 word splitter keeps trailing punctuation inside URL tokens, only 1,628 of 8,565 URL gold spans end on a word boundary. The SPY score is the unweighted mean of the legal and medical micro-F1 values, and [the results](results.md#spy-normalization-sensitivity) also report untrimmed spans.

Code scores exact typed spans without trimming across 559 documents (44 ambiguous documents excluded). Placeholder hits count any overlap with a marked placeholder. A live/example pair succeeds only when the live document matches its gold spans and the example document stays empty.

## Jev

Jev is TypeSafe’s hosted judgment model (`systemone`). It returns a yes/no probability per question and proposes no spans.

It scores a gold-blind candidate pool from the GLiNER2 outputs at the 0.05 floor plus regex matches. SPY uses `email`, `url` and `phone`, and code adds `pem`, `uri`, `path`, `ipv4`, `quoted` and `assignment`. SPY candidates include a punctuation-trimmed copy.

Each candidate boundary pairs with every task label (1,173,844 SPY and 130,750 code questions). Each question sends the document text, label definitions, offsets and surface text, and Jev accepts at probability 0.5 or higher.

The pool covers 57,727 of 58,794 SPY gold spans and 996 of 1,006 on code. That caps recall at 0.982 and 0.990. Probabilities step by 0.01 up to 0.94 on SPY and 0.91 on code. F1 is zero at 0.99 and 0.995.

## OpenAI Privacy Filter Protocol

OpenAI Privacy Filter (`openai/privacy-filter`) ran natively on the same frozen inputs as an external reference. It uses a fixed taxonomy and calibrated Viterbi decoder with no confidence threshold. Without custom label descriptions, third-party mentions count as false positives on SPY.

Every document fit a single window (up to 1,956 tokens on SPY and 1,412 on code). Native classes map to SPY types as follows:

| Native class | SPY type |
| --- | --- |
| `account_number` | `ID_NUM` |
| `private_address` | `ADDRESS` |
| `private_email` | `EMAIL` |
| `private_person` | `NAME` |
| `private_phone` | `PHONE_NUM` |
| `private_url` | `URL` |

The typed SPY view drops native `private_date` and `secret` predictions and leaves username gold spans unmatched. The label-blind view matches any native prediction against any gold boundary.

On code, the primary view is label-blind across all 10 gold types. A coarse view merges `API_KEY`, `TOKEN`, `PASSWORD`, `CONN_STRING` and `PRIVATE_KEY` into `SECRET`, leaves `USER_PATH`, `HOST` and `DEV_USERNAME` gold spans unmatched and maps the remaining classes:

| Native class | Coarse code type |
| --- | --- |
| `secret` | `SECRET` |
| `private_email` | `DEV_EMAIL` |
| `private_person` | `DEV_NAME` |

[The results](results.md#openai-privacy-filter) report both views and their controls.

## Bootstrap Intervals

Confidence intervals use a 95% paired bootstrap (2,000 replicates, seed 20260923), resampling SPY within domain and keeping code live/example pairs together. They measure document sampling only.

## Overlap and Reuse Limits

No benchmark document matches a training file. Masked 13-gram matching finds shared phrasing in 61 SPY and 59 code documents, listed in [`input_audit.json`](https://huggingface.co/datasets/nyuuzyou/gliner2.5-PII-code-en-data/blob/main/evidence/benchmark/input_audit.json).

Checkpoint selection reused the 199 mixed-development code documents, so [the results](results.md#code-strata-and-controls-at-05) score the other 404 separately.