README.md

---
license: other
license_name: proprietary-sciencesoft
license_link: https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-small/blob/main/README.md#licence
language: [en, de, fr, it, es, ru, ar]
pipeline_tag: token-classification
tags: [pii, ner, privacy, dlp, onnx, minilm, distillation]
library_name: onnxruntime
base_model: microsoft/Multilingual-MiniLM-L12-H384
base_model_relation: finetune
model-index:
- name: scnsoft-pii-encoder-small
  results:
  - task:
      type: token-classification
      name: PII detection (format-less entities)
    dataset:
      type: ScienceSoft/piibench
      name: PII Bench (piibench)
    metrics:
    - type: f1
      value: 0.5965
      name: F1, character-level, label-agnostic, averaged over 7 datasets (production engine, before post-filters)
    - type: precision
      value: 0.4580
      name: Precision
    - type: recall
      value: 0.9024
      name: Recall
---

# ScienceSoft PII encoder — small

A fast, multilingual token-classification model for personal data **without a fixed format**:
names of people and organisations, locations, addresses and dates. It is a 6-layer, 384-wide
student of [`ScienceSoft/scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder),
and it runs **3.9× faster** than that model on CPU.

**It is considerably less accurate than the model it was distilled from.** Its recall matches the
full model's, but its precision does not: on real text it marks far more words that are not
personal data. Before training, a bar was set for releasing a small model: an average PII Bench
F1 of at least 0.72, with no language more than 0.08 below the full model. **This model does not
meet that bar.** It is published with that result stated, so the trade between speed and accuracy
can be judged on the numbers.

Architecture: [`microsoft/Multilingual-MiniLM-L12-H384`](https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384),
reduced to 6 of its 12 layers (layers 1, 3, 5, 7, 9 and 11), with XLM-R's tokenizer. It is trained
on the full model's synthetic corpus with the full model as teacher, then exported to ONNX and
quantised to int8. The file is **107 MB**, and inference runs on CPU.

## Labels

`PERSON` `ORGANIZATION` `LOCATION` `ADDRESS` `DATE_TIME` (BIO tagging, 11 classes), the same set
and order as the full model.

Identifiers with a fixed format (email addresses, telephone, payment-card, IBAN and passport
numbers) are not labels; a regular expression recognises them exactly. The full model is trained
to leave them unmarked. This model does so much less reliably; see *touch rate* below.

## Speed

Measured on 300 PII Bench rows:

- int8 ONNX Runtime 1.19.2 on CPU, batch size 1;
- all three models interleaved row by row;
- an Apple M4 Max with 16 cores, a shared workstation with a 1-minute load average of 18–19.

Absolute milliseconds depend on the host; the ratio is the informative part.

| model | layers × width | int8 size | median ms per row | 95th percentile | speed-up |
|---|---|---|---|---|---|
| `scnsoft-pii-encoder` | 12 × 768 | 278 MB | 22.2 | 36.0 | 1.0× |
| [`scnsoft-pii-encoder-pruned-6l`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-pruned-6l) | 6 × 768 | 235 MB | 12.2 | 20.7 | 1.8× |
| **this model** | 6 × 384 | 107 MB | **5.7** | **8.5** | **3.9×** |

## Evaluation on PII Bench

[PII Bench](https://huggingface.co/datasets/ScienceSoft/piibench) (piibench) scores detection
of the five entity types above in text serialised the way outbound traffic carries it (JSON,
XML, SQL, CSV, key=value, log lines, Markdown tables, OCR output) as well as prose.

- It comprises 3,500 rows, 500 from each of seven third-party datasets.
- Precision, recall and F1 are computed over characters and are label-agnostic.
- Scores are micro-averaged within each dataset and then averaged across the seven.
- Leaderboard: [ScienceSoft/piibench-leaderboard](https://huggingface.co/spaces/ScienceSoft/piibench-leaderboard).

**How these numbers were produced.** All models in the tables below are scored the same way: by
the production Rust inference engine, before its post-filters. That is the same path as the
leaderboard's `scnsoft-pii-encoder-engine-raw` row, which this scoring reproduces exactly
(0.7725). The intervals are 95 % percentile bootstrap intervals over rows, resampled within each
dataset (1,000 resamples).

**Conflict of interest.** PII Bench is built and published by ScienceSoft, the developer of
this model. Four of its seven datasets (TAB, FactRuEval, MEDDOCAN, IAHLT) informed changes to the
training corpus this model shares with the full model.

| | this model | `scnsoft-pii-encoder` | bar set before training |
|---|---|---|---|
| F1 | **0.597** [0.587, 0.605] | 0.773 [0.765, 0.780] | ≥ 0.72 |
| precision | 0.458 | 0.685 | |
| recall | 0.902 | 0.908 | |
| touch rate (lower is better) | 0.237 | 0.020 | |

| language | this model | `scnsoft-pii-encoder` | bar (full model − 0.08) |
|---|---|---|---|
| ar | 0.525 | 0.681 | 0.601 |
| de | 0.435 | 0.657 | 0.577 |
| en | 0.640 | 0.804 | 0.724 |
| es | 0.678 | 0.844 | 0.764 |
| ru | 0.599 | 0.754 | 0.674 |

**Where the accuracy is lost.**

- Recall is the full model's (0.902 against 0.908). The loss is in **precision** (0.458 against
  0.685), and it appears in every dataset and in every free-text format.
- On an in-distribution synthetic evaluation, drawn from the training generator, the model scores
  F1 0.986, close to the full model's 0.993. It learned the task on the kind of text it was trained
  on. It fires too often on real text, which its training data (synthetic, like the full model's)
  never showed it.
- Distillation did not close the gap, because the teacher's guidance was also only given on
  synthetic text.
- **Touch rate** is the share of characters belonging to format-fixed identifiers that a model
  marks anyway: 0.237 here, against 0.020 for the full model. In a product that already masks
  those identifiers by regular expression, this costs little. As a stand-alone detector, it
  over-marks them.

### Variability across training seeds

Two training runs of the same recipe, differing only in the random seed:

| training seed | F1 | 95 % CI | precision | recall |
|---|---|---|---|---|
| 1234 — **this model** (`main`) | 0.5965 | [0.5871, 0.6052] | 0.458 | 0.902 |
| 4321 (branch `seed-4321`) | 0.6035 | [0.5944, 0.6123] | 0.465 | 0.901 |

The two runs agree to within 0.007 F1. Seed 1234 is published on `main` by convention, not
because it scored better; it did not.

## Training

- **Corpus:** the same synthetic corpus as `scnsoft-pii-encoder` (v23), rebuilt from that run's
  own generator snapshot and configuration and verified row for row by count: 155,141 training and
  5,300 evaluation rows.
- **Distillation from `scnsoft-pii-encoder`:** the loss is 0.5 × cross-entropy on the gold labels
  plus 0.5 × T² × KL(teacher ‖ student) on the teacher's token distributions at temperature
  T = 2, over non-padding tokens.
- **Schedule:** two epochs, 19 minutes on one GPU.
- **Stack:** `transformers==4.57.1`, `tokenizers==0.22.1`, `torch==2.9.0a0`, `faker==37.12.0`.
- **int8 against fp32:** −0.0037 F1 on the synthetic evaluation, 95 % CI [−0.0055, −0.0022].

## Limitations

- **Below the release bar in every language** (see above). Where accuracy matters more than
  latency, use [`scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder).
- **Low precision on real text.** Expect far more false positives than from the full model, and
  many more format-fixed identifiers marked.
- **Trained only on synthetic data**, like the full model. Scores on generated text overstate
  real-world ability.
- The limitations of the full model apply too. Arabic is its weakest language; Arabizi is a
  reconstruction; `DATE_TIME` is kept by the product only next to a birth-date cue.

## Use

Required files: `config.json`, `tokenizer.json`, `onnx/model_quantized.onnx`.

```python
import json, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("ScienceSoft/scnsoft-pii-encoder-small")
cfg = json.load(open("config.json"))
id2label = {int(k): v for k, v in cfg["id2label"].items()}
sess = ort.InferenceSession("onnx/model_quantized.onnx",
                            providers=["CPUExecutionProvider"])

enc = tok(text, return_offsets_mapping=True, return_tensors="np",
          truncation=True, max_length=256)
offsets = enc.pop("offset_mapping")[0]
logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
# argmax -> id2label -> merge BIO spans using `offsets` for character positions
```

The ONNX graph takes `input_ids` and `attention_mask` only. Encode with the special tokens
`<s>` and `</s>`, as the model was trained; `AutoTokenizer.__call__` adds them by default.

## Provenance

The training corpus is synthetic text produced by ScienceSoft's own generator; no production
traffic and no scraped text is used for training. Entity values come from project-written lists
and from Faker (person, company and city names), libpostal (address vocabulary) and GeoNames
(Arabic-script city names).

## Licence

Proprietary to ScienceSoft. The weights are published so that the results on this page can be
evaluated and reproduced; no licence to use them in production, to modify them or to redistribute
them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).

**Basis for these terms.**

- **Initialised from** `microsoft/Multilingual-MiniLM-L12-H384` (MIT licence).
- **Uses the tokenizer of** `FacebookAI/xlm-roberta-base` (MIT licence).
- **Distilled from** ScienceSoft's own `scnsoft-pii-encoder`.
- **Training corpus** includes entity values from Faker (MIT), libpostal (MIT) and GeoNames
  (CC BY 4.0).

All of these licences permit a derivative work to be distributed under proprietary terms,
provided their notices are retained and GeoNames is attributed. None is share-alike, and none
restricts commercial use. The notices and the attribution are in
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).