kubre commited on
Commit
86b2247
·
verified ·
1 Parent(s): 14cd859

Card: metadata, evaluation results, load from Hugging Face

Browse files
Files changed (1) hide show
  1. README.md +91 -20
README.md CHANGED
@@ -13,10 +13,80 @@ tags:
13
  - pii
14
  - privacy
15
  - ner
 
16
  - anonymization
 
17
  - on-device
18
  - browser
19
  - typescript
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  ---
21
 
22
  # Veil
@@ -35,7 +105,7 @@ import { loadModel } from "@tars-technologies/veil/model"
35
  const veil = createVeil({ model: loadModel() })
36
 
37
  await veil.mask({
38
- text: "I'm Laura Bianchi, call me on +33 6 12 34 56 78 or write to laura.bianchi@example.com",
39
  types: ["PERSON", "PHONE", "EMAIL"]
40
  })
41
  // "I'm [NAME], call me on [PHONE] or write to [EMAIL]"
@@ -44,11 +114,20 @@ await veil.mask({
44
  ## Install
45
 
46
  ```sh
47
- npm install github:Tars-Technologies/veil
48
  ```
49
 
50
  The package ships TypeScript source. Use it with a bundler or runtime that compiles TypeScript, such as Vite,
51
- esbuild, tsx or Bun.
 
 
 
 
 
 
 
 
 
52
 
53
  ## Usage
54
 
@@ -117,19 +196,21 @@ const { data, spans, isPending, error } = useMask({ text: draft, types: ["PERSON
117
  |---|---|
118
  | Author | Vaibhav Kubre, Tars Technologies |
119
  | Labels | `PERSON`, `ADDRESS`, `DOB`, `ID`, `FINANCIAL`, `PHONE`, `EMAIL` |
120
- | Architecture | Hashed character n-gram embeddings (32,768 buckets × 64, 2-bit codes), word-shape features and three word-list flags, into a 2-layer bidirectional LSTM with 96 units. Weights are int8. |
121
  | Size | 1,177,096 bytes (`veil-0.3.1.bin`) |
122
  | Languages | Latin script. Most training text is English, German, French, Spanish, Dutch, Portuguese or Italian. |
123
  | Input | Windows of 256 tokens with a 64-token overlap. Up to 300 characters of `context`. |
124
- | Licence | Apache-2.0 for code and model. `NOTICE` credits the data sources. |
125
 
126
  The model file carries its name, version and format in a header and ends with a checksum. The library reads format 3
127
  and rejects a file whose checksum does not match. Each new model gets a new file name.
128
 
129
  ## Evaluation
130
 
131
- We measured `PERSON` on five public test sets. None of them is in the training data. All models use their default
132
- settings. Exact-match F1:
 
 
133
 
134
  | Test set | Names | Veil 0.3.1 (1.2 MB) | spaCy sm (15 MB) | newmean nano (2.6 MB) | newmean small (7.1 MB) | newmean large (20.7 MB) |
135
  |---|---|---|---|---|---|---|
@@ -164,7 +245,7 @@ The training data includes these public datasets:
164
 
165
  | Source | Rows | Licence |
166
  |---|---|---|
167
- | Wikipedia articles (DE, FR, EN), labelled by a larger model | 181,689 | CC BY-SA |
168
  | Wikipedia talk pages (FR, NL, ES, PT, DE, IT) | 40,371 | CC BY-SA 4.0 |
169
  | ABCD customer-service dialogues | 74,792 | MIT |
170
  | Taskmaster dialogues | 39,723 | CC BY 4.0 |
@@ -174,8 +255,8 @@ The training data includes these public datasets:
174
  | French public-service texts | 7,206 | Licence Ouverte 2.0 |
175
  | NVIDIA Nemotron-PII | 96,050 | CC BY 4.0 |
176
 
177
- The word-list flags come from three lists in `training/assets/gazetteer.json.gz`: given names and surnames
178
- (including Wikidata), places from GeoNames, and common words from the training text. The training set itself is not
179
  published. [`training/`](https://github.com/Tars-Technologies/veil/blob/main/training/README.md) has the trainer, the checkpoint and the export script, which rebuilds
180
  `veil-0.3.1.bin` byte for byte.
181
 
@@ -188,16 +269,6 @@ published. [`training/`](https://github.com/Tars-Technologies/veil/blob/main/tra
188
  - Titles such as Mr or Dr are not part of the span.
189
  - Only `PERSON` has been measured on public data. The other six labels have no published evaluation yet.
190
 
191
- ## Development
192
-
193
- ```sh
194
- pnpm install
195
- pnpm test:ci
196
- pnpm type-check
197
- pnpm demo # demo page on http://localhost:5173
198
- pnpm embed # after a model change, regenerates src/model.ts from the .bin file
199
- ```
200
-
201
  ## Citation
202
 
203
  ```bibtex
 
13
  - pii
14
  - privacy
15
  - ner
16
+ - redaction
17
  - anonymization
18
+ - multilingual
19
  - on-device
20
  - browser
21
  - typescript
22
+ datasets:
23
+ - wikimedia/wikipedia
24
+ - nvidia/Nemotron-PII
25
+ - OpenAssistant/oasst2
26
+ - AmazonScience/massive
27
+ - google-research-datasets/taskmaster1
28
+ metrics:
29
+ - f1
30
+ model-index:
31
+ - name: veil-0.3.1
32
+ results:
33
+ - task:
34
+ type: token-classification
35
+ name: Person name detection
36
+ dataset:
37
+ type: eriktks/conll2003
38
+ name: CoNLL-2003 English
39
+ split: test
40
+ metrics:
41
+ - type: f1
42
+ value: 73.1
43
+ name: F1, PERSON, exact span
44
+ - task:
45
+ type: token-classification
46
+ name: Person name detection
47
+ dataset:
48
+ type: leondz/wnut_17
49
+ name: WNUT-17
50
+ split: test
51
+ metrics:
52
+ - type: f1
53
+ value: 51.5
54
+ name: F1, PERSON, exact span
55
+ - task:
56
+ type: token-classification
57
+ name: Person name detection
58
+ dataset:
59
+ type: eriktks/conll2002
60
+ name: CoNLL-2002 Spanish
61
+ config: es
62
+ split: test
63
+ metrics:
64
+ - type: f1
65
+ value: 75.0
66
+ name: F1, PERSON, exact span
67
+ - task:
68
+ type: token-classification
69
+ name: Person name detection
70
+ dataset:
71
+ type: eriktks/conll2002
72
+ name: CoNLL-2002 Dutch
73
+ config: nl
74
+ split: test
75
+ metrics:
76
+ - type: f1
77
+ value: 70.5
78
+ name: F1, PERSON, exact span
79
+ - task:
80
+ type: token-classification
81
+ name: Person name detection
82
+ dataset:
83
+ type: ildpil/text-anonymization-benchmark
84
+ name: Text Anonymization Benchmark
85
+ split: test
86
+ metrics:
87
+ - type: f1
88
+ value: 88.1
89
+ name: F1, PERSON, leading title ignored
90
  ---
91
 
92
  # Veil
 
105
  const veil = createVeil({ model: loadModel() })
106
 
107
  await veil.mask({
108
+ text: "I'm Laura Bianchi, call me on +33 6 39 98 12 34 or write to laura.bianchi@example.com",
109
  types: ["PERSON", "PHONE", "EMAIL"]
110
  })
111
  // "I'm [NAME], call me on [PHONE] or write to [EMAIL]"
 
114
  ## Install
115
 
116
  ```sh
117
+ npm install github:Tars-Technologies/veil#v0.3.1
118
  ```
119
 
120
  The package ships TypeScript source. Use it with a bundler or runtime that compiles TypeScript, such as Vite,
121
+ esbuild, tsx or Bun. Node's built-in TypeScript support does not compile packages in `node_modules`, so in Node run
122
+ your code with tsx, as an ES module (`"type": "module"` or a `.mts` file).
123
+
124
+ To load the model straight from this page:
125
+
126
+ ```ts
127
+ const veil = createVeil({
128
+ model: "https://huggingface.co/tars-technologies/veil/resolve/main/veil-0.3.1.bin"
129
+ })
130
+ ```
131
 
132
  ## Usage
133
 
 
196
  |---|---|
197
  | Author | Vaibhav Kubre, Tars Technologies |
198
  | Labels | `PERSON`, `ADDRESS`, `DOB`, `ID`, `FINANCIAL`, `PHONE`, `EMAIL` |
199
+ | Architecture | Hashed character n-gram embeddings (32,768 buckets × 64, 2-bit codes), word-shape features and three word-list flags, into a 2-layer bidirectional LSTM with 96 units. LSTM and output weights are int8 with one scale per row. |
200
  | Size | 1,177,096 bytes (`veil-0.3.1.bin`) |
201
  | Languages | Latin script. Most training text is English, German, French, Spanish, Dutch, Portuguese or Italian. |
202
  | Input | Windows of 256 tokens with a 64-token overlap. Up to 300 characters of `context`. |
203
+ | Licence | Apache-2.0 for code and model. `NOTICE` credits the data sources. The word lists keep their source licences (Wikidata CC0, GeoNames CC BY 4.0). |
204
 
205
  The model file carries its name, version and format in a header and ends with a checksum. The library reads format 3
206
  and rejects a file whose checksum does not match. Each new model gets a new file name.
207
 
208
  ## Evaluation
209
 
210
+ We measured `PERSON` on five public test sets. None of them is in the training data. The model that labelled the
211
+ Wikipedia rows was trained on the CoNLL-2003 English and CoNLL-2002 Spanish and Dutch training splits; their test
212
+ splits were not used. All models use their default settings. F1 on exact span match, except that the TAB row ignores
213
+ a leading title (see below):
214
 
215
  | Test set | Names | Veil 0.3.1 (1.2 MB) | spaCy sm (15 MB) | newmean nano (2.6 MB) | newmean small (7.1 MB) | newmean large (20.7 MB) |
216
  |---|---|---|---|---|---|---|
 
245
 
246
  | Source | Rows | Licence |
247
  |---|---|---|
248
+ | Wikipedia articles (DE, FR, EN), labelled by a larger model | 181,689 | CC BY-SA 4.0 |
249
  | Wikipedia talk pages (FR, NL, ES, PT, DE, IT) | 40,371 | CC BY-SA 4.0 |
250
  | ABCD customer-service dialogues | 74,792 | MIT |
251
  | Taskmaster dialogues | 39,723 | CC BY 4.0 |
 
255
  | French public-service texts | 7,206 | Licence Ouverte 2.0 |
256
  | NVIDIA Nemotron-PII | 96,050 | CC BY 4.0 |
257
 
258
+ The word-list flags come from three lists in `training/assets/gazetteer.json.gz`: given names and surnames,
259
+ places from GeoNames, and common words from the training text. The training set itself is not
260
  published. [`training/`](https://github.com/Tars-Technologies/veil/blob/main/training/README.md) has the trainer, the checkpoint and the export script, which rebuilds
261
  `veil-0.3.1.bin` byte for byte.
262
 
 
269
  - Titles such as Mr or Dr are not part of the span.
270
  - Only `PERSON` has been measured on public data. The other six labels have no published evaluation yet.
271
 
 
 
 
 
 
 
 
 
 
 
272
  ## Citation
273
 
274
  ```bibtex