Benchmark Protocol

The benchmark scores gliner2.5-PII-code-en against Fastino baselines and TypeSafe’s Jev on 8,688 SPY documents and 603 code documents. Every protocol pin, revision and SHA-256 hash is in release_integrity.json, and the results page holds the full score tables.

Compared Systems

System	Source	Parameters	Architecture
Fastino base	`fastino/gliner2.5-base-v1`	193M	`BoundaryExtractor`
Fastino PII	`fastino/gliner2-privacy-filter-PII-multi`	307M	`SpanExtractor`
`gliner2.5-PII-code-en`	`nyuuzyou/gliner2.5-PII-code-en`	193M	`BoundaryExtractor`
Jev	`typesafe-ai/jev`	not published	hosted API

Fastino PII uses a span head capped at 8 tokens, while the boundary models have no width cap.

Frozen Inputs

SPY is an English PII benchmark from Savkin et al. (2025) with 4,197 legal and 4,491 medical documents (58,794 gold spans). Its upstream loader fills placeholders with Faker after an unseeded shuffle. This release rebuilt the text from mks-logic/SPY with Faker seed 42 and froze benchmarks/spy.jsonl.

The code suite (benchmarks/code.jsonl) has 603 documents and 1,006 gold spans across source files, configs, logs and coding-agent transcripts:

Stratum	Documents	Role
Mixed development	199	Live secrets beside placeholders; reused for checkpoint selection
Explicit placeholder	160	Placeholders like `<substitute-vendor-credential>` that must stay silent
Live context	80	Live values marked as active
Example context	80	Matching values framed as examples
Clean code	40	Code without sensitive values
Ambiguous	44	Unclear cases, reported without error scoring

Inference Settings

Each GLiNER2 checkpoint ran every document in a single window (batch size 8, float32 weights) with this model’s label descriptions. Extraction kept spans from confidence 0.05 upward and applied thresholds during scoring.

Prose (author7) label mapping:

Label	Scored type
`author email`	`EMAIL`
`author name`	`NAME`
`author url`	`URL`
`author phone number`	`PHONE_NUM`
`author id number`	`ID_NUM`
`author address`	`ADDRESS`
`author username`	`USERNAME`

Code (code) secret and identity mapping:

Label	Scored type
`api key`	`API_KEY`
`auth token`	`TOKEN`
`password`	`PASSWORD`
`connection string`	`CONN_STRING`
`private key`	`PRIVATE_KEY`
`personal file path`	`USER_PATH`
`internal host`	`HOST`
`developer email`	`DEV_EMAIL`
`developer name`	`DEV_NAME`
`developer username`	`DEV_USERNAME`

Thresholds and Scoring

The primary comparison uses the common threshold 0.5, with secondary thresholds 0.9, 0.99 and 0.995 reported alongside it. SPY feedback set the prose threshold of 0.995.

SPY scores exact typed spans after trimming trailing punctuation from valid predictions, while invalid offsets stay false positives. Because the GLiNER2 word splitter keeps trailing punctuation inside URL tokens, only 1,628 of 8,565 URL gold spans end on a word boundary. The SPY score is the unweighted mean of the legal and medical micro-F1 values, and the results also report untrimmed spans.

Code scores exact typed spans without trimming across 559 documents (44 ambiguous documents excluded). Placeholder hits count any overlap with a marked placeholder. A live/example pair succeeds only when the live document matches its gold spans and the example document stays empty.

Jev

Jev is TypeSafe’s hosted judgment model (systemone). It returns a yes/no probability per question and proposes no spans.

It scores a gold-blind candidate pool from the GLiNER2 outputs at the 0.05 floor plus regex matches. SPY uses email, url and phone, and code adds pem, uri, path, ipv4, quoted and assignment. SPY candidates include a punctuation-trimmed copy.

Each candidate boundary pairs with every task label (1,173,844 SPY and 130,750 code questions). Each question sends the document text, label definitions, offsets and surface text, and Jev accepts at probability 0.5 or higher.

The pool covers 57,727 of 58,794 SPY gold spans and 996 of 1,006 on code. That caps recall at 0.982 and 0.990. Probabilities step by 0.01 up to 0.94 on SPY and 0.91 on code. F1 is zero at 0.99 and 0.995.

OpenAI Privacy Filter Protocol

OpenAI Privacy Filter (openai/privacy-filter) ran natively on the same frozen inputs as an external reference. It uses a fixed taxonomy and calibrated Viterbi decoder with no confidence threshold. Without custom label descriptions, third-party mentions count as false positives on SPY.

Every document fit a single window (up to 1,956 tokens on SPY and 1,412 on code). Native classes map to SPY types as follows:

Native class	SPY type
`account_number`	`ID_NUM`
`private_address`	`ADDRESS`
`private_email`	`EMAIL`
`private_person`	`NAME`
`private_phone`	`PHONE_NUM`
`private_url`	`URL`

The typed SPY view drops native private_date and secret predictions and leaves username gold spans unmatched. The label-blind view matches any native prediction against any gold boundary.

On code, the primary view is label-blind across all 10 gold types. A coarse view merges API_KEY, TOKEN, PASSWORD, CONN_STRING and PRIVATE_KEY into SECRET, leaves USER_PATH, HOST and DEV_USERNAME gold spans unmatched and maps the remaining classes:

Native class	Coarse code type
`secret`	`SECRET`
`private_email`	`DEV_EMAIL`
`private_person`	`DEV_NAME`

The results report both views and their controls.

Bootstrap Intervals

Confidence intervals use a 95% paired bootstrap (2,000 replicates, seed 20260923), resampling SPY within domain and keeping code live/example pairs together. They measure document sampling only.

Overlap and Reuse Limits

No benchmark document matches a training file. Masked 13-gram matching finds shared phrasing in 61 SPY and 59 code documents, listed in input_audit.json.

Checkpoint selection reused the 199 mixed-development code documents, so the results score the other 404 separately.