--- license: other license_name: proprietary-sciencesoft license_link: https://huggingface.co/ScienceSoft/scnsoft-pii-encoder/blob/main/README.md#licence language: [en, de, fr, it, es, ru, ar] pipeline_tag: token-classification tags: [pii, ner, privacy, dlp, onnx, xlm-roberta, document-classification] library_name: onnxruntime base_model: FacebookAI/xlm-roberta-base base_model_relation: finetune --- **Release candidate for supportive monitoring. Not an enforcement tool.** This is a document-type head, V3a, seed 7001. Its scores support a person looking at documents; nothing here claims that a document should be blocked, redacted or refused because of a score. The weights are published for evaluation and reproduction only (see Licence). The training data is not published. ## Document-type head V3a (seed 7001) This model adds a second output, `doc_logits`, to the same int8 graph as [`ScienceSoft/scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder), so that a window of text is also scored for 17 document types (non-disclosure and master agreements, statements of work, other contracts, requests for proposals, proposals, price lists, source code, design and security documents, financial and personal-finance documents, personnel files, strategy documents, published documents and patient records). The PII output (`logits`) is the encoder's own. ### What this release says, and does not say - **Four types have been evaluated, on one measure only.** `legal.nda`, `tech.design`, `tech.security` and `finance.personal` met a false-record line on held-out benign documents on one machine (table below). That is the only evaluation of them. Their recall on documents of the type, their behaviour on other machines and their cost on customer hardware are not evaluated. Meeting the line does not make a type ready for any decision. - **`legal.contract_other` is not yet monitoring-grade.** It missed the same line (8 false records on 18,075 benign documents). In `doc_labels.json` it is set to `rules`: the model adds nothing for it. - **Every other type with a calibration value is fused at that value as an unvalidated suggestion, never enforcement.** That is eleven labels: `legal.master_agreement`, `legal.sow`, `sales.rfp`, `sales.proposal`, `sales.pricing`, `tech.source_code`, `finance.report`, `finance.billing`, `hr.personnel`, `corporate.strategy` and `health.patient_record`. They are not validated: no recall, no per-language result and no line was evaluated for them, and no accuracy is claimed. What was measured is false records on benign documents, in numbers: `health.patient_record` **54 of 18,075**, `corporate.strategy` **12 of 18,075**, `legal.master_agreement` **5 of 18,075** (all German), `finance.report` **5 of 18,075**, `sales.proposal` **4 of 18,075**, and 1 or 0 for the others (table below). A finding of any of them is a suggestion for a person to look at; the counts say how often a benign document reached the value, and nothing about how often a document of the type does. `public.published` has no calibration value and stays `rules`. - **One seed.** The head is seed 7001, the highest of three seeds on the development split. No three-seed repeat was done. ### Files | file | what it is | |---|---| | `README.md` | This model card. | | `onnx/model_quantized.onnx` | Two outputs: `logits` (PII, unchanged) and `doc_logits` `[batch, 17]`. 279,041,365 bytes, sha256 `0c9c1780673d719310afd1bce2538d8e078ea2adf0bc3a6e397c564ac6486b6e`. | | `doc_labels.json` | Version 2: the 17 labels in output order and, per label, a mode, `t_add`, `t_confirm`, `delta` and a `status` that says whether the type is evaluated or an unvalidated suggestion. | | `tokenizer.json` | The encoder's tokenizer (vocabulary and rules), byte-identical to `ScienceSoft/scnsoft-pii-encoder` at the revision below. sha256 `a514807cffabd8abaf028cfaffe7ff0c4f60b97ea2db80c41f14172ae6b018ca`. | | `config.json` | The encoder's model configuration, byte-identical to the encoder's. sha256 `1e840bedc5fac622c796f19c2021c4040641326fcacb9d18ab87c5d2e050e0db`. | | `tokenizer_config.json` | The encoder's tokenizer settings, byte-identical to the encoder's. sha256 `ccf223ba3d5b3cc7fa6c3bf451f3bb40557a5c92b0aa33f63d17802ff1a96fd9`. | | `special_tokens_map.json` | The encoder's special-token map (start, end, padding, unknown and mask tokens), byte-identical to the encoder's. sha256 `06e405a36dfe4b9604f484f6a1e619af1a7f7d09e34a8555eb0b77b66318067f`. | | `THIRD_PARTY_NOTICES.md` | Attributions for the training sources that ask for them. | | `MANIFEST.sha256` | The sha256 of every file above. | The encoder is the one at revision `b1a36210f99ef293a4d42697ef6b2ba548dd9208` (blob sha256 `c996fc5ad1118e0571263ee637f541b3d0981bec648e57211ad4676a3dc1ab47`). The tokenizer and `config.json` are the encoder's own. No separate PII-only graph is included, so `pii_only_graph` in `doc_labels.json` is `null`. ### The head Two layers over the mean-pooled encoder states of a 450-character window: 768 to 256, exact GELU (written with `Erf`, as the released graph already does), 256 to 17. The encoder is frozen. The graph is the released graph with nodes added; none of its nodes is changed. **The PII output is unchanged.** On this graph `logits` is byte-identical to the released encoder's on all 6,825 rows of the PII release check, at batch size 1: | host | onnxruntime | threads | |---|---|---| | aarch64 Linux (NVIDIA GB10 host) | 1.24.2 | 1 and 4 | | Apple Mac, arm64 | 1.24.2 | 1 and 4 | | Apple Mac, arm64 | 1.19.2 | 1 and 4 | Not checked: the fp32 graph, Windows, and any batch size above 1. ### Modes and thresholds | mode | meaning | |---|---| | `rules` | A rule-based classifier decides. The head's score is information only. | | `add` | The rule-based classifier's decision stands, unchanged. When the rules find nothing for the type and the document's score reaches `t_add`, the head's finding is reported as a model-added type: a suggestion, softer than a rule decision. | False records are counted at `t_add` on 18,075 benign documents (see "What was measured"). | label | mode | `t_add` | false records | status | |---|---|---|---|---| | `legal.nda` | `add` | 0.84 | 1 of 18,075 | false-record line met; nothing else evaluated | | `legal.master_agreement` | `add` | 0.82 | 5 of 18,075 | unvalidated suggestion | | `legal.sow` | `add` | 0.87 | 1 of 18,075 | unvalidated suggestion | | `legal.contract_other` | `rules` | | 8 of 18,075 | **not yet monitoring-grade**: line missed | | `sales.rfp` | `add` | 0.70 | 1 of 18,075 | unvalidated suggestion | | `sales.proposal` | `add` | 0.72 | 4 of 18,075 | unvalidated suggestion | | `sales.pricing` | `add` | 0.90 | 1 of 18,075 | unvalidated suggestion | | `tech.source_code` | `add` | 0.99 | 0 of 18,075 | unvalidated suggestion | | `tech.design` | `add` | 0.97 | 0 of 18,075 | false-record line met; nothing else evaluated | | `tech.security` | `add` | 0.97 | 2 of 18,075 | false-record line met; nothing else evaluated | | `finance.report` | `add` | 0.98 | 5 of 18,075 | unvalidated suggestion | | `finance.billing` | `add` | 0.69 | 1 of 18,075 | unvalidated suggestion | | `finance.personal` | `add` | 0.81 | 0 of 18,075 | false-record line met; nothing else evaluated | | `hr.personnel` | `add` | 0.93 | 0 of 18,075 | unvalidated suggestion | | `corporate.strategy` | `add` | 0.66 | 12 of 18,075 | unvalidated suggestion | | `public.published` | `rules` | | no reporting value | the rules decide | | `health.patient_record` | `add` | 0.95 | 54 of 18,075 | unvalidated suggestion | **How `t_add` was set.** Each is the *reporting value* of the type: the smallest value from 0.05 to 0.99 at which a document's score (rule `lead2`, below) stays at or under a false-positive rate of 2.78 x 10^-4 in every one of seven languages, with the upper 95% Wilson bound over all languages under it too, read on 80,739 ordinary documents of a calibration set. **Setting `t_add` equal to the reporting value is a choice made for this release; it is not a registered rule.** The reporting value says nothing about recall. ### How a document is scored 1. **Windows.** The text is cut into 450-character windows overlapping by 50 characters. 2. **Scores.** Each window is scored on its own, at **batch size 1**, with `sigmoid(doc_logits)`. 3. **Evidence rule, `lead2`.** A document reaches a type when both of its first two windows reach the threshold (the only window, for a one-window text), or when at least `max(3, ceil(n / 5))` of its `n` windows do. **The thresholds are valid at batch size 1 only.** The features and thresholds were produced one window per call on one thread. On the same 200 windows, scores computed in padded batches of 64 differ from batch-1 scores by up to 0.53 (per type the largest difference is between 0.14 and 0.53; 197 of the 200 windows moved by more than 0.001 on some type). Thread count made no difference. Do not score in batches. A file name or title is not used. ### What was measured **False records (held-out benign documents).** A false record is a benign document whose gold label lacks the type and whose score is at or above the type's calibration value. Population: 18,075 benign documents from two held-out sets (English 3,139; German 2,509; French 2,509; Italian 2,510; Spanish 2,509; Russian 2,450; Arabic 2,449), read once for this head. Machine: aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, batch 1, 1 thread. The line, fixed before the read: met when the pooled count is at most 5 and no language has more than 2. | type | value | pooled | by language | line | |---|---|---|---|---| | `legal.nda` | 0.84 | 1 | es 1 | met | | `legal.contract_other` | 0.86 | 8 | en 3, de 3, es 1, ru 1 | **missed** | | `tech.design` | 0.97 | 0 | none | met | | `tech.security` | 0.97 | 2 | ru 2 | met | | `finance.personal` | 0.81 | 0 | none | met | Reported only, with no line and no verdict (these eleven types are fused as unvalidated suggestions; the numbers describe the head's score on benign documents, not an evaluation of the type): | type | value | pooled | by language | |---|---|---|---| | `health.patient_record` | 0.95 | 54 | en 33, es 6, it 6, fr 4, de 3, ru 2 | | `corporate.strategy` | 0.66 | 12 | ru 3, en 2, de 2, it 2, es 2, fr 1 | | `legal.master_agreement` | 0.82 | 5 | de 5 | | `finance.report` | 0.98 | 5 | en 3, de 1, it 1 | | `sales.proposal` | 0.72 | 4 | de 3, ar 1 | | `legal.sow` | 0.87 | 1 | es 1 | | `sales.rfp` | 0.70 | 1 | de 1 | | `sales.pricing` | 0.90 | 1 | fr 1 | | `finance.billing` | 0.69 | 1 | en 1 | | `tech.source_code` | 0.99 | 0 | none | | `hr.personnel` | 0.93 | 0 | none | `public.published` has no calibration value. Meeting the false-record line does not make a type ready: recall, agreement between machines and cost on customer hardware are further conditions, and none of them is met. **`legal.contract_other` against the earlier preview (`doc-head-v2`), recall.** 484 held-out contract documents from 430 filers, each model at its own calibration value (V3a 0.86, v2 0.85), batch 1 for both. V3a found 448 (recall 0.9256, 95% interval 0.9017 to 0.9481), v2 found 442 (0.9132, 0.8866 to 0.9380). Paired difference +0.0124 (interval -0.0063 to +0.0327). The difference is inside the interval around zero: V3a is **not shown to be better** than v2 on this type. Absolute recall is optimistic, because the set keeps only documents on which the rules and a model agreed; only the difference is a comparison, and it is inside noise. **Cost.** p50 time per 450-character window at batch size 1 (aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, 200 windows, one pass, no repeat): 41.6 ms on 1 thread and 14.1 ms on 4 threads for this graph; 41.5 and 13.7 ms for the released encoder alone. The head's cost is **not distinguishable from zero** at this resolution (about plus or minus 0.5 ms); it is not claimed to be zero. Peak memory 502 MB, against 501 MB for the encoder alone. Cost on a Mac, on Windows and on customer machines was not evaluated, nor was engine time per file. **PII output.** Byte-identical to the encoder's on 6,825 rows at batch 1 (table above). ### Limitations - **Machines disagree.** Two files of `doc_logits` for the same 6,825 rows, computed on the aarch64 Linux host and on an Apple Mac, differ in bytes on every row. Against the Mac with onnxruntime 1.24.2, 611 rows fall on opposite sides of a threshold for some evaluated type (`legal.nda` 466, `legal.contract_other` 167, `tech.security` 7); against the Mac with 1.19.2, 138 (103, 35 and 2). Those files were produced without the engine's graph-arithmetic setting, so they are not a measurement of an engine and not a test of agreement between machines; the engine-level cells on a Mac and on Windows are pending, and the agreement bar between machines is **not met**. A document near a threshold may reach it on one machine and not on another. Thresholds were read on the aarch64 Linux host only. - **Batch composition changes scores** (above). Only batch size 1 is described here. - **Not evaluated.** Recall of the four types on documents of the type; the cross-machine bar; cost on a Mac, on Windows and on customer machines; engine time per file; a three-seed repeat; the fp32 graph; the PII output at batch sizes above 1; `hr.personnel`; and, beyond the false records above, every type other than the four. - **Training data.** Six types have no real training document, only generated text: `corporate.strategy`, `finance.billing`, `health.patient_record`, `sales.pricing`, `sales.proposal` and `sales.rfp`. For the ten other types that have real training documents, all the real documents are in English and their other-language training documents are machine translations and generated text; only `public.published`, the examples of no type, has real documents in other languages. ### Training data 47,824 training documents and 6,185 development documents. The training documents are 20,561 real, 6,680 machine translations and 20,583 generated. The data is **not published**. Provenance, as it is: - **Real public documents**, under open licences or in the public domain: agreements from CUAD, ContractNLI, the KL3M collection of EDGAR agreements, Common Paper and Bonterms (CC BY 4.0), SEC EDGAR filings, official legislation (EU, French and Italian texts), United Nations documents, European Union and UK government pages (the GOV.UK pages are used as examples of documents that are none of the types, under the Open Government Licence v3.0), software and security documentation and advisories, Python Enhancement Proposals, US drug labels, payslip examples, and Wikinews. - **Machine translations** of registered training documents, made with OPUS-MT (CC BY 4.0). - **Generated text** written by an open-weight language model released under a permissive open licence, from prompts written for each type, and documents based on templates written by ScienceSoft. Attributions that the sources ask for are in `THIRD_PARTY_NOTICES.md`. Text was cut into 450-character windows, pooled by the frozen encoder and used only to fit the head; no source text is redistributed with the model. ## Licence Proprietary to ScienceSoft. The weights are published so that the results on this page can be evaluated and reproduced; no licence to use them in production, to modify them or to redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com). **Basis for these terms.** The model is derived from [`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base) (MIT licence). Its head was fitted on documents from the sources listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) and described above. The licences those sources carry that this release has checked (CC BY 4.0, the Open Government Licence v3.0, the Licence Ouverte 2.0, the European Commission's reuse policy for EUR-Lex, the United Nations corpus terms, and public-domain and permissive software and documentation licences) permit a derivative work to be distributed under proprietary terms, provided notices are retained and attribution is given; none is share-alike and none restricts commercial use. The SEC EDGAR filings used for training come through a dataset whose card says only that EDGAR data is publicly available; this release has not confirmed a licence statement for them beyond that.