README.md

---
license: other
license_name: proprietary-sciencesoft
license_link: https://huggingface.co/ScienceSoft/scnsoft-pii-encoder/blob/main/README.md#licence
language: [en, de, fr, it, es, ru, ar]
pipeline_tag: token-classification
tags: [pii, ner, privacy, dlp, onnx, xlm-roberta]
library_name: onnxruntime
base_model: FacebookAI/xlm-roberta-base
base_model_relation: finetune
model-index:
- name: scnsoft-pii-encoder
  results:
  - task:
      type: token-classification
      name: PII detection (format-less entities)
    dataset:
      type: ScienceSoft/piibench
      name: PII Bench (piibench)
    metrics:
    - type: f1
      value: 0.7766
      name: F1, character-level, label-agnostic, averaged over 7 datasets
    - type: precision
      value: 0.7104
      name: Precision
    - type: recall
      value: 0.8780
      name: Recall
---

# ScienceSoft PII encoder

A multilingual token-classification model that detects personal data **without a fixed
format** — the kinds that regular expressions cannot find: names of people and
organisations, locations, addresses and dates. It covers seven languages (English, German,
French, Italian, Spanish, Russian, Arabic), four Arabic registers and Arabizi.

It is the named-entity component of a data-loss-prevention agent that inspects outbound
requests to AI assistants on a user's device and redacts personal data before the request
leaves it.

Architecture: `xlm-roberta-base`, fine-tuned on a fully synthetic corpus, exported to ONNX and
quantised to int8 — **278 MB**, CPU inference.

## Labels

`PERSON` `ORGANIZATION` `LOCATION` `ADDRESS` `DATE_TIME` (BIO tagging, 11 classes).

**Identifiers with a fixed format — email addresses, telephone, payment-card, IBAN and
passport numbers — are deliberately not labels.** They have a defined structure and usually a
checksum, so a regular expression recognises them exactly; a classifier would add a second,
less reliable decision about something already decided. The model is trained to leave them
unmarked, and the *touch rate* below measures how well it does so.

## Evaluation on PII Bench

[PII Bench](https://huggingface.co/datasets/ScienceSoft/piibench) (piibench) scores detection
of the five entity types above in text serialised the way outbound traffic carries it — JSON,
XML, SQL, CSV, key=value, log lines, Markdown tables, OCR output — as well as prose. It
comprises 3,500 rows, 500 from each of seven third-party datasets. Precision, recall and F1 are
computed over characters and are label-agnostic, micro-averaged within each dataset and then
averaged across the seven. Live results for every entrant:
[leaderboard](https://huggingface.co/spaces/ScienceSoft/piibench-leaderboard).

**Conflict of interest.** PII Bench is built and published by ScienceSoft, the developer of
this model. Four of its seven datasets (TAB, FactRuEval, MEDDOCAN, IAHLT) informed changes to
this model's training corpus; the other three did not. Both facts should be weighed when
reading the comparison below.

Comparison with the strongest independent entrant, `nvidia/gliner-PII`, on the same rows and
the same host. Intervals are 95 % percentile bootstrap intervals over rows, resampled within
each dataset (1,000 resamples for each model's F1; 2,000 for the paired difference below):

| | this model | `nvidia/gliner-PII` |
|---|---|---|
| F1 | **0.777** [0.769, 0.784] | 0.753 [0.746, 0.761] |
| precision | 0.710 | 0.744 |
| recall | **0.878** | 0.783 |
| touch rate (not ranked; lower is better) | **0.021** | 0.288 |
| latency, median ms per row, batch size 1 | **20.3** † | 140.6 |
| latency, 95th percentile ms per row | **33.5** † | 199.7 |

† Measured on the previous release (v15, revision `e98225e07225`), which has the same
architecture and is exported to int8 ONNX the same way; latency has not been re-measured for
this build.

The paired difference in average F1 is **+0.0233, 95 % CI [+0.0147, +0.0322]** (2,000
resamples, seed 0); the interval excludes zero.

Three qualifications belong with that result:

- **Recall is higher and precision is lower.** For a data-loss-prevention agent a missed
  entity leaves the device unredacted while a false positive masks a word unnecessarily, so
  the trade is in the intended direction — but it is a trade, and the F1 average conceals it.
- **Latency was measured on a shared workstation** under a higher load average for this model
  (1-minute load 20.7) than for the comparison model (11.1), and for this model on the previous
  release (†). Absolute milliseconds depend on the host; the ordering is the informative part.
- **Touch rate** is the share of characters belonging to format-fixed identifiers that a model
  marks anyway. It is reported, not ranked.

### Variability across training seeds

The result above describes **this artefact** — one set of weights, trained with one random
seed — not the training recipe. The recipe was trained with three seeds, and each run was
scored through the same production inference engine (the `-engine-raw` path on the
leaderboard):

| training seed | F1 | Δ F1 vs `nvidia/gliner-PII` | 95 % CI of Δ | CI excludes 0 |
|---|---|---|---|---|
| 1234 — **this model** (`main`) | 0.7725 | +0.0192 | [+0.0105, +0.0278] | yes |
| 4321 (branch `seed-4321`) | 0.7590 | +0.0057 | [−0.0028, +0.0145] | no |
| 7777 (branch `seed-7777`) | 0.7543 | +0.0011 | [−0.0080, +0.0103] | no |

Only the released run is separably better than the reference. This model is therefore the
upper end of the recipe's observed spread, and another training run of the same recipe would
not be expected to reproduce its lead. The weights of the other two runs are published on the
branches named above so that this can be checked.

## Release checks for this build (v23)

Run on 2026-09-09. Accuracy lines are scored through the production Rust inference engine,
not through a Python decode of the same ONNX file. All eight checks passed for both training
seeds.

| check | seed 1234 (released) | seed 4321 | threshold |
|---|---|---|---|
| known defect cases | 21/21 | 21/21 | all pass |
| format-fixed identifiers wrongly marked | 0.1 % | 0.0 % | ≤ 15 % |
| recall on Russian PII (hivetrace) | 0.9429 | 0.9357 | ≥ 0.90 |
| F1 cost of int8 vs fp32 | −0.0003 | +0.0043 | ≤ 0.02 |
| latency, 95th percentile at 512 tokens, CPU | 192.6 ms | 188.9 ms | ≤ 200 ms |
| name-origin bias (max F1 gap between origins) | 0.0000 | 0.0000 | ≤ 0.05 |
| vs previous model, worst language (ru), Δ F1 [95 % CI] | +0.1538 [+0.1012, +0.2082] | +0.1308 [+0.0809, +0.1870] | CI lower bound ≥ −0.02 |
| held-out register frames, engine, filtered F1 | 0.8870 | 0.8860 | ≥ 0.8786 |

**Change from the previous release (v15).** One corpus change: training frames in which a
definite article or head noun immediately precedes an organisation name and lies **outside**
its span. It targets the most frequent ORGANIZATION error measured on two real-text datasets
that were not used to shape training: the previous build started 9–10 % of GermEval's gold
organisations and 16 % of TAB's one word too early.

## Other evaluations

**External datasets** (reported, not gated): hivetrace ru 0.8713 · TAB en, including
quasi-identifiers, 0.7111 · IAHLT ar 0.5490 · Wojood ar 0.5384. hivetrace is itself synthetic,
though expert-annotated.

**In-distribution synthetic evaluation** (n = 3,300; drawn from the same generator as the
training data, so it overstates real-world accuracy): filtered F1 0.9353 (precision 0.9899,
recall 0.8864); for the entities the product masks (PERSON and ADDRESS) 0.9795. On held-out
register frames: raw F1 0.9438, filtered 0.8870.

## Limitations

- **Arabic is the weakest language, and the gap is recall** — about 0.54–0.55 F1 on the Arabic
  datasets against 0.87 on Russian. The aggregate score is not uniform across languages.
- **Trained only on synthetic data.** Entity pools are finite, so scores on generated text
  overstate real-world ability; the external and PII Bench results are the better guide.
- **Arabizi is a reconstruction.** No corpus contains it; its conventions (3 = ʿayn,
  7 = ḥāʾ, 2 = hamza) are a model of how people type, not observed data.
- **DATE_TIME scores 0.000 on the in-distribution file by design.** The product keeps a date
  only when an explicit birth-date cue (`born on`, `DOB:` …) is adjacent, and that file
  contains none. On held-out register frames, where such cues occur, filtered DATE_TIME F1 is
  0.4933 at precision 0.9418.

## Use

Required files: `config.json`, `tokenizer.json`, `onnx/model_quantized.onnx`.

```python
import json, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ScienceSoft/scnsoft-pii-encoder")
cfg = json.load(open("config.json"))
id2label = {int(k): v for k, v in cfg["id2label"].items()}
sess = ort.InferenceSession("onnx/model_quantized.onnx",
                            providers=["CPUExecutionProvider"])

enc = tok(text, return_offsets_mapping=True, return_tensors="np",
          truncation=True, max_length=256)
offsets = enc.pop("offset_mapping")[0]
logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
# argmax -> id2label -> merge BIO spans using `offsets` for character positions
```

**Encode with the special tokens `<s>` and `</s>`.** The model was trained with them;
`AutoTokenizer.__call__` adds them by default, whereas `encode(..., add_special_tokens=False)`
does not, and omitting them measurably lowers accuracy.

## Provenance

The training corpus is synthetic text produced by this project's own generator; no production
traffic and no scraped text is used for training. Entity values inserted into that text come
from project-written lists and from three third-party sources: Faker (person, company and city
names), libpostal (address vocabulary) and GeoNames (Arabic-script city names) — see
*Licence*. The generator configuration, seeds and library versions
are recorded with each run. The organisation-name pool (`REAL_ORGS`) is **off** in this build:
it was evaluated in an earlier run and cost more accuracy than it added.

Training stack: `transformers==4.57.1`, `tokenizers==0.22.1`, `torch==2.9.0a0`,
`faker==37.12.0`; two epochs.

## Licence

Proprietary to ScienceSoft. The weights are published so that the results on this page can be
evaluated and reproduced; no licence to use them in production, to modify them or to
redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).

**Basis for these terms.** The model is derived from
[`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base) (MIT
licence), and its training corpus includes entity values from Faker (MIT), libpostal (MIT) and
GeoNames (CC BY 4.0). All four licences permit a derivative work to be distributed under
proprietary terms, provided their notices are retained and GeoNames is attributed; none is
share-alike and none restricts commercial use. The notices and the attribution are in
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).