|
Download README.md from ScienceSoft/scnsoft-pii-encoder-doc-type-head: direct link, hf CLI and curl.
- Browser
- Download file 16.5 kB
-
https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-doc-type-head/resolve/main/README.md
- Command line
-
hf download hf://ScienceSoft/scnsoft-pii-encoder-doc-type-head/README.md
-
curl -L -o README.md https://huggingface.co/ScienceSoft/scnsoft-pii-encoder-doc-type-head/resolve/main/README.md
16.5 kB
| license: other | |
| license_name: proprietary-sciencesoft | |
| license_link: https://huggingface.co/ScienceSoft/scnsoft-pii-encoder/blob/main/README.md#licence | |
| language: [en, de, fr, it, es, ru, ar] | |
| pipeline_tag: token-classification | |
| tags: [pii, ner, privacy, dlp, onnx, xlm-roberta, document-classification] | |
| library_name: onnxruntime | |
| base_model: FacebookAI/xlm-roberta-base | |
| base_model_relation: finetune | |
| **Release candidate for supportive monitoring. Not an enforcement tool.** This is a document-type head, V3a, seed 7001. Its | |
| scores support a person looking at documents; nothing here claims that a document should be blocked, redacted or refused | |
| because of a score. The weights are published for evaluation and reproduction only (see Licence). The training data is not | |
| published. | |
| ## Document-type head V3a (seed 7001) | |
| This model adds a second output, `doc_logits`, to the same int8 graph as | |
| [`ScienceSoft/scnsoft-pii-encoder`](https://huggingface.co/ScienceSoft/scnsoft-pii-encoder), so that a window of text is | |
| also scored for 17 document types (non-disclosure and master agreements, statements of work, other contracts, requests | |
| for proposals, proposals, price lists, source code, design and security documents, financial and personal-finance | |
| documents, personnel files, strategy documents, published documents and patient records). The PII output (`logits`) is the | |
| encoder's own. | |
| ### What this release says, and does not say | |
| - **Four types have been evaluated, on one measure only.** `legal.nda`, `tech.design`, `tech.security` and | |
| `finance.personal` met a false-record line on held-out benign documents on one machine (table below). That is the only | |
| evaluation of them. Their recall on documents of the type, their behaviour on other machines and their cost on customer | |
| hardware are not evaluated. Meeting the line does not make a type ready for any decision. | |
| - **`legal.contract_other` is not yet monitoring-grade.** It missed the same line (8 false records on 18,075 benign | |
| documents). In `doc_labels.json` it is set to `rules`: the model adds nothing for it. | |
| - **Every other type with a calibration value is fused at that value as an unvalidated suggestion, never enforcement.** | |
| That is eleven labels: `legal.master_agreement`, `legal.sow`, `sales.rfp`, `sales.proposal`, `sales.pricing`, | |
| `tech.source_code`, `finance.report`, `finance.billing`, `hr.personnel`, `corporate.strategy` and | |
| `health.patient_record`. They are not validated: no recall, no per-language result and no line was evaluated for them, | |
| and no accuracy is claimed. What was measured is false records on benign documents, in numbers: `health.patient_record` | |
| **54 of 18,075**, `corporate.strategy` **12 of 18,075**, `legal.master_agreement` **5 of 18,075** (all German), | |
| `finance.report` **5 of 18,075**, `sales.proposal` **4 of 18,075**, and 1 or 0 for the others (table below). A finding of | |
| any of them is a suggestion for a person to look at; the counts say how often a benign document reached the value, and | |
| nothing about how often a document of the type does. | |
| `public.published` has no calibration value and stays `rules`. | |
| - **One seed.** The head is seed 7001, the highest of three seeds on the development split. No three-seed repeat was done. | |
| ### Files | |
| | file | what it is | | |
| |---|---| | |
| | `README.md` | This model card. | | |
| | `onnx/model_quantized.onnx` | Two outputs: `logits` (PII, unchanged) and `doc_logits` `[batch, 17]`. 279,041,365 bytes, sha256 `0c9c1780673d719310afd1bce2538d8e078ea2adf0bc3a6e397c564ac6486b6e`. | | |
| | `doc_labels.json` | Version 2: the 17 labels in output order and, per label, a mode, `t_add`, `t_confirm`, `delta` and a `status` that says whether the type is evaluated or an unvalidated suggestion. | | |
| | `tokenizer.json` | The encoder's tokenizer (vocabulary and rules), byte-identical to `ScienceSoft/scnsoft-pii-encoder` at the revision below. sha256 `a514807cffabd8abaf028cfaffe7ff0c4f60b97ea2db80c41f14172ae6b018ca`. | | |
| | `config.json` | The encoder's model configuration, byte-identical to the encoder's. sha256 `1e840bedc5fac622c796f19c2021c4040641326fcacb9d18ab87c5d2e050e0db`. | | |
| | `tokenizer_config.json` | The encoder's tokenizer settings, byte-identical to the encoder's. sha256 `ccf223ba3d5b3cc7fa6c3bf451f3bb40557a5c92b0aa33f63d17802ff1a96fd9`. | | |
| | `special_tokens_map.json` | The encoder's special-token map (start, end, padding, unknown and mask tokens), byte-identical to the encoder's. sha256 `06e405a36dfe4b9604f484f6a1e619af1a7f7d09e34a8555eb0b77b66318067f`. | | |
| | `THIRD_PARTY_NOTICES.md` | Attributions for the training sources that ask for them. | | |
| | `MANIFEST.sha256` | The sha256 of every file above. | | |
| The encoder is the one at revision `b1a36210f99ef293a4d42697ef6b2ba548dd9208` (blob sha256 | |
| `c996fc5ad1118e0571263ee637f541b3d0981bec648e57211ad4676a3dc1ab47`). The tokenizer and `config.json` are the encoder's own. | |
| No separate PII-only graph is included, so `pii_only_graph` in `doc_labels.json` is `null`. | |
| ### The head | |
| Two layers over the mean-pooled encoder states of a 450-character window: 768 to 256, exact GELU (written with `Erf`, as the | |
| released graph already does), 256 to 17. The encoder is frozen. The graph is the released graph with nodes added; none of | |
| its nodes is changed. | |
| **The PII output is unchanged.** On this graph `logits` is byte-identical to the released encoder's on all 6,825 rows of the | |
| PII release check, at batch size 1: | |
| | host | onnxruntime | threads | | |
| |---|---|---| | |
| | aarch64 Linux (NVIDIA GB10 host) | 1.24.2 | 1 and 4 | | |
| | Apple Mac, arm64 | 1.24.2 | 1 and 4 | | |
| | Apple Mac, arm64 | 1.19.2 | 1 and 4 | | |
| Not checked: the fp32 graph, Windows, and any batch size above 1. | |
| ### Modes and thresholds | |
| | mode | meaning | | |
| |---|---| | |
| | `rules` | A rule-based classifier decides. The head's score is information only. | | |
| | `add` | The rule-based classifier's decision stands, unchanged. When the rules find nothing for the type and the document's score reaches `t_add`, the head's finding is reported as a model-added type: a suggestion, softer than a rule decision. | | |
| False records are counted at `t_add` on 18,075 benign documents (see "What was measured"). | |
| | label | mode | `t_add` | false records | status | | |
| |---|---|---|---|---| | |
| | `legal.nda` | `add` | 0.84 | 1 of 18,075 | false-record line met; nothing else evaluated | | |
| | `legal.master_agreement` | `add` | 0.82 | 5 of 18,075 | unvalidated suggestion | | |
| | `legal.sow` | `add` | 0.87 | 1 of 18,075 | unvalidated suggestion | | |
| | `legal.contract_other` | `rules` | | 8 of 18,075 | **not yet monitoring-grade**: line missed | | |
| | `sales.rfp` | `add` | 0.70 | 1 of 18,075 | unvalidated suggestion | | |
| | `sales.proposal` | `add` | 0.72 | 4 of 18,075 | unvalidated suggestion | | |
| | `sales.pricing` | `add` | 0.90 | 1 of 18,075 | unvalidated suggestion | | |
| | `tech.source_code` | `add` | 0.99 | 0 of 18,075 | unvalidated suggestion | | |
| | `tech.design` | `add` | 0.97 | 0 of 18,075 | false-record line met; nothing else evaluated | | |
| | `tech.security` | `add` | 0.97 | 2 of 18,075 | false-record line met; nothing else evaluated | | |
| | `finance.report` | `add` | 0.98 | 5 of 18,075 | unvalidated suggestion | | |
| | `finance.billing` | `add` | 0.69 | 1 of 18,075 | unvalidated suggestion | | |
| | `finance.personal` | `add` | 0.81 | 0 of 18,075 | false-record line met; nothing else evaluated | | |
| | `hr.personnel` | `add` | 0.93 | 0 of 18,075 | unvalidated suggestion | | |
| | `corporate.strategy` | `add` | 0.66 | 12 of 18,075 | unvalidated suggestion | | |
| | `public.published` | `rules` | | no reporting value | the rules decide | | |
| | `health.patient_record` | `add` | 0.95 | 54 of 18,075 | unvalidated suggestion | | |
| **How `t_add` was set.** Each is the *reporting value* of the type: the smallest value from 0.05 to 0.99 at which a | |
| document's score (rule `lead2`, below) stays at or under a false-positive rate of 2.78 x 10^-4 in every one of seven | |
| languages, with the upper 95% Wilson bound over all languages under it too, read on 80,739 ordinary documents of a | |
| calibration set. **Setting `t_add` equal to the reporting value is a choice made for this release; it is not a registered | |
| rule.** The reporting value says nothing about recall. | |
| ### How a document is scored | |
| 1. **Windows.** The text is cut into 450-character windows overlapping by 50 characters. | |
| 2. **Scores.** Each window is scored on its own, at **batch size 1**, with `sigmoid(doc_logits)`. | |
| 3. **Evidence rule, `lead2`.** A document reaches a type when both of its first two windows reach the threshold (the only | |
| window, for a one-window text), or when at least `max(3, ceil(n / 5))` of its `n` windows do. | |
| **The thresholds are valid at batch size 1 only.** The features and thresholds were produced one window per call on one | |
| thread. On the same 200 windows, scores computed in padded batches of 64 differ from batch-1 scores by up to 0.53 (per | |
| type the largest difference is between 0.14 and 0.53; 197 of the 200 windows moved by more than 0.001 on some type). | |
| Thread count made no difference. Do not score in batches. | |
| A file name or title is not used. | |
| ### What was measured | |
| **False records (held-out benign documents).** A false record is a benign document whose gold label lacks the type and whose | |
| score is at or above the type's calibration value. Population: 18,075 benign documents from two held-out sets (English | |
| 3,139; German 2,509; French 2,509; Italian 2,510; Spanish 2,509; Russian 2,450; Arabic 2,449), read once for this | |
| head. Machine: aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, batch 1, 1 thread. The line, fixed before the read: met when the | |
| pooled count is at most 5 and no language has more than 2. | |
| | type | value | pooled | by language | line | | |
| |---|---|---|---|---| | |
| | `legal.nda` | 0.84 | 1 | es 1 | met | | |
| | `legal.contract_other` | 0.86 | 8 | en 3, de 3, es 1, ru 1 | **missed** | | |
| | `tech.design` | 0.97 | 0 | none | met | | |
| | `tech.security` | 0.97 | 2 | ru 2 | met | | |
| | `finance.personal` | 0.81 | 0 | none | met | | |
| Reported only, with no line and no verdict (these eleven types are fused as unvalidated suggestions; the numbers describe | |
| the head's score on benign documents, not an evaluation of the type): | |
| | type | value | pooled | by language | | |
| |---|---|---|---| | |
| | `health.patient_record` | 0.95 | 54 | en 33, es 6, it 6, fr 4, de 3, ru 2 | | |
| | `corporate.strategy` | 0.66 | 12 | ru 3, en 2, de 2, it 2, es 2, fr 1 | | |
| | `legal.master_agreement` | 0.82 | 5 | de 5 | | |
| | `finance.report` | 0.98 | 5 | en 3, de 1, it 1 | | |
| | `sales.proposal` | 0.72 | 4 | de 3, ar 1 | | |
| | `legal.sow` | 0.87 | 1 | es 1 | | |
| | `sales.rfp` | 0.70 | 1 | de 1 | | |
| | `sales.pricing` | 0.90 | 1 | fr 1 | | |
| | `finance.billing` | 0.69 | 1 | en 1 | | |
| | `tech.source_code` | 0.99 | 0 | none | | |
| | `hr.personnel` | 0.93 | 0 | none | | |
| `public.published` has no calibration value. Meeting the false-record line does not make a type ready: recall, | |
| agreement between machines and cost on customer hardware are further conditions, and none of them is met. | |
| **`legal.contract_other` against the earlier preview (`doc-head-v2`), recall.** 484 held-out contract documents from 430 | |
| filers, each model at its own calibration value (V3a 0.86, v2 0.85), batch 1 for both. V3a found 448 (recall 0.9256, 95% | |
| interval 0.9017 to 0.9481), v2 found 442 (0.9132, 0.8866 to 0.9380). Paired difference +0.0124 (interval -0.0063 to | |
| +0.0327). The difference is inside the interval around zero: V3a is **not shown to be better** than v2 on this type. Absolute | |
| recall is optimistic, because the set keeps only documents on which the rules and a model agreed; only the difference is a | |
| comparison, and it is inside noise. | |
| **Cost.** p50 time per 450-character window at batch size 1 (aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, 200 windows, one | |
| pass, no repeat): 41.6 ms on 1 thread and 14.1 ms on 4 threads for this graph; 41.5 and 13.7 ms for the released encoder | |
| alone. The head's cost is **not distinguishable from zero** at this resolution (about plus or minus 0.5 ms); it is not | |
| claimed to be zero. Peak memory 502 MB, against 501 MB for the encoder alone. Cost on a Mac, on Windows and on customer | |
| machines was not evaluated, nor was engine time per file. | |
| **PII output.** Byte-identical to the encoder's on 6,825 rows at batch 1 (table above). | |
| ### Limitations | |
| - **Machines disagree.** Two files of `doc_logits` for the same 6,825 rows, computed on the aarch64 Linux host and on an Apple | |
| Mac, differ in bytes on every row. Against the Mac with onnxruntime 1.24.2, 611 rows fall on opposite sides of a | |
| threshold for some evaluated type (`legal.nda` 466, `legal.contract_other` 167, `tech.security` 7); against the Mac with | |
| 1.19.2, 138 (103, 35 and 2). Those files were produced without the engine's graph-arithmetic setting, so they are not | |
| a measurement of an engine and not a test of agreement between machines; the engine-level cells on a Mac and on Windows are | |
| pending, and the agreement bar between machines is **not met**. A document near a threshold may reach it on one machine | |
| and not on another. Thresholds were read on the aarch64 Linux host only. | |
| - **Batch composition changes scores** (above). Only batch size 1 is described here. | |
| - **Not evaluated.** Recall of the four types on documents of the type; the cross-machine bar; cost on a Mac, on Windows and | |
| on customer machines; engine time per file; a three-seed repeat; the fp32 graph; the PII output at batch sizes above 1; | |
| `hr.personnel`; and, beyond the false records above, every type other than the four. | |
| - **Training data.** Six types have no real training document, only generated text: `corporate.strategy`, | |
| `finance.billing`, `health.patient_record`, `sales.pricing`, `sales.proposal` and `sales.rfp`. For the ten other types | |
| that have real training documents, all the real documents are in English and their other-language training documents | |
| are machine translations and generated text; only `public.published`, the examples of no type, has real documents in | |
| other languages. | |
| ### Training data | |
| 47,824 training documents and 6,185 development documents. The training documents are 20,561 real, 6,680 machine | |
| translations and 20,583 generated. The data is **not published**. Provenance, as it is: | |
| - **Real public documents**, under open licences or in the public domain: agreements from CUAD, ContractNLI, the KL3M | |
| collection of EDGAR agreements, Common Paper and Bonterms (CC BY 4.0), SEC EDGAR filings, official legislation (EU, French | |
| and Italian texts), United Nations documents, European Union and UK government pages (the GOV.UK pages are used as | |
| examples of documents that are none of the types, under the Open Government Licence v3.0), software and | |
| security documentation and advisories, Python Enhancement Proposals, US drug labels, payslip examples, and Wikinews. | |
| - **Machine translations** of registered training documents, made with OPUS-MT (CC BY 4.0). | |
| - **Generated text** written by an open-weight language model released under a permissive open licence, from prompts | |
| written for each type, and documents based on templates written by ScienceSoft. | |
| Attributions that the sources ask for are in `THIRD_PARTY_NOTICES.md`. Text was cut into 450-character windows, pooled by the | |
| frozen encoder and used only to fit the head; no source text is redistributed with the model. | |
| ## Licence | |
| Proprietary to ScienceSoft. The weights are published so that the results on this page can be | |
| evaluated and reproduced; no licence to use them in production, to modify them or to | |
| redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com). | |
| **Basis for these terms.** The model is derived from | |
| [`FacebookAI/xlm-roberta-base`](https://huggingface.co/FacebookAI/xlm-roberta-base) (MIT | |
| licence). Its head was fitted on documents from the sources listed in | |
| [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) and described above. The licences those sources carry that this release has | |
| checked (CC BY 4.0, the Open Government Licence v3.0, the Licence Ouverte 2.0, the European Commission's reuse policy for | |
| EUR-Lex, the United Nations corpus terms, and public-domain and permissive software and documentation licences) permit a | |
| derivative work to be distributed under proprietary terms, provided notices are retained and attribution is given; none | |
| is share-alike and none restricts commercial use. The SEC EDGAR filings used for training come through a dataset whose card | |
| says only that EDGAR data is publicly available; this release has not confirmed a licence statement for them beyond that. | |