File size: 16,457 Bytes
84eeb99 0eb6443 84eeb99 0eb6443 84eeb99 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 | ---
license: other
license_name: proprietary-sciencesoft
license_link: https://huggingface.co/ScienceSoft/scnsoft-pii-encoder/blob/main/README.md#licence
language: [en, de, fr, it, es, ru, ar]
pipeline_tag: token-classification
tags: [pii, ner, privacy, dlp, onnx, xlm-roberta, document-classification]
library_name: onnxruntime
base_model: FacebookAI/xlm-roberta-base
base_model_relation: finetune
---
**Release candidate for supportive monitoring. Not an enforcement tool.** This is a document-type head, V3a, seed 7001. Its
scores support a person looking at documents; nothing here claims that a document should be blocked, redacted or refused
because of a score. The weights are published for evaluation and reproduction only (see Licence). The training data is not
published.
## Document-type head V3a (seed 7001)
This model adds a second output, `doc_logits`, to the same int8 graph as
[`ScienceSoft/scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder), so that a window of text is
also scored for 17 document types (non-disclosure and master agreements, statements of work, other contracts, requests
for proposals, proposals, price lists, source code, design and security documents, financial and personal-finance
documents, personnel files, strategy documents, published documents and patient records). The PII output (`logits`) is the
encoder's own.
### What this release says, and does not say
- **Four types have been evaluated, on one measure only.** `legal.nda`, `tech.design`, `tech.security` and
`finance.personal` met a false-record line on held-out benign documents on one machine (table below). That is the only
evaluation of them. Their recall on documents of the type, their behaviour on other machines and their cost on customer
hardware are not evaluated. Meeting the line does not make a type ready for any decision.
- **`legal.contract_other` is not yet monitoring-grade.** It missed the same line (8 false records on 18,075 benign
documents). In `doc_labels.json` it is set to `rules`: the model adds nothing for it.
- **Every other type with a calibration value is fused at that value as an unvalidated suggestion, never enforcement.**
That is eleven labels: `legal.master_agreement`, `legal.sow`, `sales.rfp`, `sales.proposal`, `sales.pricing`,
`tech.source_code`, `finance.report`, `finance.billing`, `hr.personnel`, `corporate.strategy` and
`health.patient_record`. They are not validated: no recall, no per-language result and no line was evaluated for them,
and no accuracy is claimed. What was measured is false records on benign documents, in numbers: `health.patient_record`
**54 of 18,075**, `corporate.strategy` **12 of 18,075**, `legal.master_agreement` **5 of 18,075** (all German),
`finance.report` **5 of 18,075**, `sales.proposal` **4 of 18,075**, and 1 or 0 for the others (table below). A finding of
any of them is a suggestion for a person to look at; the counts say how often a benign document reached the value, and
nothing about how often a document of the type does.
`public.published` has no calibration value and stays `rules`.
- **One seed.** The head is seed 7001, the highest of three seeds on the development split. No three-seed repeat was done.
### Files
| file | what it is |
|---|---|
| `README.md` | This model card. |
| `onnx/model_quantized.onnx` | Two outputs: `logits` (PII, unchanged) and `doc_logits` `[batch, 17]`. 279,041,365 bytes, sha256 `0c9c1780673d719310afd1bce2538d8e078ea2adf0bc3a6e397c564ac6486b6e`. |
| `doc_labels.json` | Version 2: the 17 labels in output order and, per label, a mode, `t_add`, `t_confirm`, `delta` and a `status` that says whether the type is evaluated or an unvalidated suggestion. |
| `tokenizer.json` | The encoder's tokenizer (vocabulary and rules), byte-identical to `ScienceSoft/scnsoft-pii-encoder` at the revision below. sha256 `a514807cffabd8abaf028cfaffe7ff0c4f60b97ea2db80c41f14172ae6b018ca`. |
| `config.json` | The encoder's model configuration, byte-identical to the encoder's. sha256 `1e840bedc5fac622c796f19c2021c4040641326fcacb9d18ab87c5d2e050e0db`. |
| `tokenizer_config.json` | The encoder's tokenizer settings, byte-identical to the encoder's. sha256 `ccf223ba3d5b3cc7fa6c3bf451f3bb40557a5c92b0aa33f63d17802ff1a96fd9`. |
| `special_tokens_map.json` | The encoder's special-token map (start, end, padding, unknown and mask tokens), byte-identical to the encoder's. sha256 `06e405a36dfe4b9604f484f6a1e619af1a7f7d09e34a8555eb0b77b66318067f`. |
| `THIRD_PARTY_NOTICES.md` | Attributions for the training sources that ask for them. |
| `MANIFEST.sha256` | The sha256 of every file above. |
The encoder is the one at revision `b1a36210f99ef293a4d42697ef6b2ba548dd9208` (blob sha256
`c996fc5ad1118e0571263ee637f541b3d0981bec648e57211ad4676a3dc1ab47`). The tokenizer and `config.json` are the encoder's own.
No separate PII-only graph is included, so `pii_only_graph` in `doc_labels.json` is `null`.
### The head
Two layers over the mean-pooled encoder states of a 450-character window: 768 to 256, exact GELU (written with `Erf`, as the
released graph already does), 256 to 17. The encoder is frozen. The graph is the released graph with nodes added; none of
its nodes is changed.
**The PII output is unchanged.** On this graph `logits` is byte-identical to the released encoder's on all 6,825 rows of the
PII release check, at batch size 1:
| host | onnxruntime | threads |
|---|---|---|
| aarch64 Linux (NVIDIA GB10 host) | 1.24.2 | 1 and 4 |
| Apple Mac, arm64 | 1.24.2 | 1 and 4 |
| Apple Mac, arm64 | 1.19.2 | 1 and 4 |
Not checked: the fp32 graph, Windows, and any batch size above 1.
### Modes and thresholds
| mode | meaning |
|---|---|
| `rules` | A rule-based classifier decides. The head's score is information only. |
| `add` | The rule-based classifier's decision stands, unchanged. When the rules find nothing for the type and the document's score reaches `t_add`, the head's finding is reported as a model-added type: a suggestion, softer than a rule decision. |
False records are counted at `t_add` on 18,075 benign documents (see "What was measured").
| label | mode | `t_add` | false records | status |
|---|---|---|---|---|
| `legal.nda` | `add` | 0.84 | 1 of 18,075 | false-record line met; nothing else evaluated |
| `legal.master_agreement` | `add` | 0.82 | 5 of 18,075 | unvalidated suggestion |
| `legal.sow` | `add` | 0.87 | 1 of 18,075 | unvalidated suggestion |
| `legal.contract_other` | `rules` | | 8 of 18,075 | **not yet monitoring-grade**: line missed |
| `sales.rfp` | `add` | 0.70 | 1 of 18,075 | unvalidated suggestion |
| `sales.proposal` | `add` | 0.72 | 4 of 18,075 | unvalidated suggestion |
| `sales.pricing` | `add` | 0.90 | 1 of 18,075 | unvalidated suggestion |
| `tech.source_code` | `add` | 0.99 | 0 of 18,075 | unvalidated suggestion |
| `tech.design` | `add` | 0.97 | 0 of 18,075 | false-record line met; nothing else evaluated |
| `tech.security` | `add` | 0.97 | 2 of 18,075 | false-record line met; nothing else evaluated |
| `finance.report` | `add` | 0.98 | 5 of 18,075 | unvalidated suggestion |
| `finance.billing` | `add` | 0.69 | 1 of 18,075 | unvalidated suggestion |
| `finance.personal` | `add` | 0.81 | 0 of 18,075 | false-record line met; nothing else evaluated |
| `hr.personnel` | `add` | 0.93 | 0 of 18,075 | unvalidated suggestion |
| `corporate.strategy` | `add` | 0.66 | 12 of 18,075 | unvalidated suggestion |
| `public.published` | `rules` | | no reporting value | the rules decide |
| `health.patient_record` | `add` | 0.95 | 54 of 18,075 | unvalidated suggestion |
**How `t_add` was set.** Each is the *reporting value* of the type: the smallest value from 0.05 to 0.99 at which a
document's score (rule `lead2`, below) stays at or under a false-positive rate of 2.78 x 10^-4 in every one of seven
languages, with the upper 95% Wilson bound over all languages under it too, read on 80,739 ordinary documents of a
calibration set. **Setting `t_add` equal to the reporting value is a choice made for this release; it is not a registered
rule.** The reporting value says nothing about recall.
### How a document is scored
1. **Windows.** The text is cut into 450-character windows overlapping by 50 characters.
2. **Scores.** Each window is scored on its own, at **batch size 1**, with `sigmoid(doc_logits)`.
3. **Evidence rule, `lead2`.** A document reaches a type when both of its first two windows reach the threshold (the only
window, for a one-window text), or when at least `max(3, ceil(n / 5))` of its `n` windows do.
**The thresholds are valid at batch size 1 only.** The features and thresholds were produced one window per call on one
thread. On the same 200 windows, scores computed in padded batches of 64 differ from batch-1 scores by up to 0.53 (per
type the largest difference is between 0.14 and 0.53; 197 of the 200 windows moved by more than 0.001 on some type).
Thread count made no difference. Do not score in batches.
A file name or title is not used.
### What was measured
**False records (held-out benign documents).** A false record is a benign document whose gold label lacks the type and whose
score is at or above the type's calibration value. Population: 18,075 benign documents from two held-out sets (English
3,139; German 2,509; French 2,509; Italian 2,510; Spanish 2,509; Russian 2,450; Arabic 2,449), read once for this
head. Machine: aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, batch 1, 1 thread. The line, fixed before the read: met when the
pooled count is at most 5 and no language has more than 2.
| type | value | pooled | by language | line |
|---|---|---|---|---|
| `legal.nda` | 0.84 | 1 | es 1 | met |
| `legal.contract_other` | 0.86 | 8 | en 3, de 3, es 1, ru 1 | **missed** |
| `tech.design` | 0.97 | 0 | none | met |
| `tech.security` | 0.97 | 2 | ru 2 | met |
| `finance.personal` | 0.81 | 0 | none | met |
Reported only, with no line and no verdict (these eleven types are fused as unvalidated suggestions; the numbers describe
the head's score on benign documents, not an evaluation of the type):
| type | value | pooled | by language |
|---|---|---|---|
| `health.patient_record` | 0.95 | 54 | en 33, es 6, it 6, fr 4, de 3, ru 2 |
| `corporate.strategy` | 0.66 | 12 | ru 3, en 2, de 2, it 2, es 2, fr 1 |
| `legal.master_agreement` | 0.82 | 5 | de 5 |
| `finance.report` | 0.98 | 5 | en 3, de 1, it 1 |
| `sales.proposal` | 0.72 | 4 | de 3, ar 1 |
| `legal.sow` | 0.87 | 1 | es 1 |
| `sales.rfp` | 0.70 | 1 | de 1 |
| `sales.pricing` | 0.90 | 1 | fr 1 |
| `finance.billing` | 0.69 | 1 | en 1 |
| `tech.source_code` | 0.99 | 0 | none |
| `hr.personnel` | 0.93 | 0 | none |
`public.published` has no calibration value. Meeting the false-record line does not make a type ready: recall,
agreement between machines and cost on customer hardware are further conditions, and none of them is met.
**`legal.contract_other` against the earlier preview (`doc-head-v2`), recall.** 484 held-out contract documents from 430
filers, each model at its own calibration value (V3a 0.86, v2 0.85), batch 1 for both. V3a found 448 (recall 0.9256, 95%
interval 0.9017 to 0.9481), v2 found 442 (0.9132, 0.8866 to 0.9380). Paired difference +0.0124 (interval -0.0063 to
+0.0327). The difference is inside the interval around zero: V3a is **not shown to be better** than v2 on this type. Absolute
recall is optimistic, because the set keeps only documents on which the rules and a model agreed; only the difference is a
comparison, and it is inside noise.
**Cost.** p50 time per 450-character window at batch size 1 (aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, 200 windows, one
pass, no repeat): 41.6 ms on 1 thread and 14.1 ms on 4 threads for this graph; 41.5 and 13.7 ms for the released encoder
alone. The head's cost is **not distinguishable from zero** at this resolution (about plus or minus 0.5 ms); it is not
claimed to be zero. Peak memory 502 MB, against 501 MB for the encoder alone. Cost on a Mac, on Windows and on customer
machines was not evaluated, nor was engine time per file.
**PII output.** Byte-identical to the encoder's on 6,825 rows at batch 1 (table above).
### Limitations
- **Machines disagree.** Two files of `doc_logits` for the same 6,825 rows, computed on the aarch64 Linux host and on an Apple
Mac, differ in bytes on every row. Against the Mac with onnxruntime 1.24.2, 611 rows fall on opposite sides of a
threshold for some evaluated type (`legal.nda` 466, `legal.contract_other` 167, `tech.security` 7); against the Mac with
1.19.2, 138 (103, 35 and 2). Those files were produced without the engine's graph-arithmetic setting, so they are not
a measurement of an engine and not a test of agreement between machines; the engine-level cells on a Mac and on Windows are
pending, and the agreement bar between machines is **not met**. A document near a threshold may reach it on one machine
and not on another. Thresholds were read on the aarch64 Linux host only.
- **Batch composition changes scores** (above). Only batch size 1 is described here.
- **Not evaluated.** Recall of the four types on documents of the type; the cross-machine bar; cost on a Mac, on Windows and
on customer machines; engine time per file; a three-seed repeat; the fp32 graph; the PII output at batch sizes above 1;
`hr.personnel`; and, beyond the false records above, every type other than the four.
- **Training data.** Six types have no real training document, only generated text: `corporate.strategy`,
`finance.billing`, `health.patient_record`, `sales.pricing`, `sales.proposal` and `sales.rfp`. For the ten other types
that have real training documents, all the real documents are in English and their other-language training documents
are machine translations and generated text; only `public.published`, the examples of no type, has real documents in
other languages.
### Training data
47,824 training documents and 6,185 development documents. The training documents are 20,561 real, 6,680 machine
translations and 20,583 generated. The data is **not published**. Provenance, as it is:
- **Real public documents**, under open licences or in the public domain: agreements from CUAD, ContractNLI, the KL3M
collection of EDGAR agreements, Common Paper and Bonterms (CC BY 4.0), SEC EDGAR filings, official legislation (EU, French
and Italian texts), United Nations documents, European Union and UK government pages (the GOV.UK pages are used as
examples of documents that are none of the types, under the Open Government Licence v3.0), software and
security documentation and advisories, Python Enhancement Proposals, US drug labels, payslip examples, and Wikinews.
- **Machine translations** of registered training documents, made with OPUS-MT (CC BY 4.0).
- **Generated text** written by an open-weight language model released under a permissive open licence, from prompts
written for each type, and documents based on templates written by ScienceSoft.
Attributions that the sources ask for are in `THIRD_PARTY_NOTICES.md`. Text was cut into 450-character windows, pooled by the
frozen encoder and used only to fit the head; no source text is redistributed with the model.
## Licence
Proprietary to ScienceSoft. The weights are published so that the results on this page can be
evaluated and reproduced; no licence to use them in production, to modify them or to
redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).
**Basis for these terms.** The model is derived from
[`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base) (MIT
licence). Its head was fitted on documents from the sources listed in
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) and described above. The licences those sources carry that this release has
checked (CC BY 4.0, the Open Government Licence v3.0, the Licence Ouverte 2.0, the European Commission's reuse policy for
EUR-Lex, the United Nations corpus terms, and public-domain and permissive software and documentation licences) permit a
derivative work to be distributed under proprietary terms, provided notices are retained and attribution is given; none
is share-alike and none restricts commercial use. The SEC EDGAR filings used for training come through a dataset whose card
says only that EDGAR data is publicly available; this release has not confirmed a licence statement for them beyond that.
|