Hugging Face's logo

UMCU
/
PII_RobBERT

Token Classification
Transformers
Safetensors
Dutch
roberta
pii
deidentification
custom_code
Model card Files
xet
 Community
UMCU commited on
Commit
dbda169
·
verified ·
1 Parent(s): b2df892

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +352 -0
README.md CHANGED
@@ -1,3 +1,355 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ datasets:
4
+ - ai4privacy/pii-masking-openpii-1.5m
5
+ language:
6
+ - nl
7
+ base_model:
8
+ - DTAI-KULeuven/robbert-2023-dutch-large
9
+ pipeline_tag: token-classification
10
+ library_name: transformers
11
+ tags:
12
+ - pii
13
+ - deidentification
14
  ---
15
+
16
+
17
+ Finetuning was done using [MedNER](https://github.com/UPOD-datascience/MedNER.nl) with the following script:
18
+
19
+ We limited the training to Dutch.
20
+
21
+ The model was trained in a multilabel-sense, using a binary cross-entropy loss per label, which followed the standard IOB-schema
22
+ (that is **O**utside the span, **B**eginning of the span, **I**nside the span)
23
+ We replaced the standard 768-weight linear layer by 3x768 dense layers with 10\% dropout and ReLu activations.
24
+
25
+ Token classification scores (answering the question: ***given** the span, to which class does it belong?*):
26
+
27
+ ```json
28
+ {
29
+ "eval_AGE": {
30
+ "f1": 0.958,
31
+ "precision": 0.957,
32
+ "recall": 0.959
33
+ },
34
+ "eval_BUILDINGNUM": {
35
+ "f1": 0.968,
36
+ "precision": 0.968,
37
+ "recall": 0.969
38
+ },
39
+ "eval_CITY": {
40
+ "f1": 0.974,
41
+ "precision": 0.968,
42
+ "recall": 0.980
43
+ },
44
+ "eval_CREDITCARDNUMBER": {
45
+ "f1": 0.954,
46
+ "precision": 0.951,
47
+ "recall": 0.957
48
+ },
49
+ "eval_DATE": {
50
+ "f1": 0.998,
51
+ "precision": 0.996,
52
+ "recall": 0.999
53
+ },
54
+ "eval_DRIVERLICENSENUM": {
55
+ "f1": 0.961,
56
+ "precision": 0.957,
57
+ "recall": 0.966
58
+ },
59
+ "eval_EMAIL": {
60
+ "f1": 0.970,
61
+ "precision": 0.958,
62
+ "recall": 0.984
63
+ },
64
+ "eval_GENDER": {
65
+ "f1": 0.936,
66
+ "precision": 0.928,
67
+ "recall": 0.945
68
+ },
69
+ "eval_GIVENNAME": {
70
+ "f1": 0.903,
71
+ "precision": 0.898,
72
+ "recall": 0.908
73
+ },
74
+ "eval_IDCARDNUM": {
75
+ "f1": 0.815,
76
+ "precision": 0.792,
77
+ "recall": 0.841
78
+ },
79
+ "eval_PASSPORTNUM": {
80
+ "f1": 0.976,
81
+ "precision": 0.972,
82
+ "recall": 0.981
83
+ },
84
+ "eval_SEX": {
85
+ "f1": 0.950,
86
+ "precision": 0.952,
87
+ "recall": 0.948
88
+ },
89
+ "eval_SOCIALNUM": {
90
+ "f1": 0.640,
91
+ "precision": 0.650,
92
+ "recall": 0.632
93
+ },
94
+ "eval_STREET": {
95
+ "f1": 0.980,
96
+ "precision": 0.979,
97
+ "recall": 0.981
98
+ },
99
+ "eval_SURNAME": {
100
+ "f1": 0.898,
101
+ "precision": 0.894,
102
+ "recall": 0.901
103
+ },
104
+ "eval_TAXNUM": {
105
+ "f1": 0.746,
106
+ "precision": 0.740,
107
+ "recall": 0.751
108
+ },
109
+ "eval_TELEPHONENUM": {
110
+ "f1": 0.984,
111
+ "precision": 0.979,
112
+ "recall": 0.989
113
+ },
114
+ "eval_TITLE": {
115
+ "f1": 0.998,
116
+ "precision": 0.997,
117
+ "recall": 0.999
118
+ },
119
+ "eval_ZIPCODE": {
120
+ "f1": 0.966,
121
+ "precision": 0.968,
122
+ "recall": 0.964
123
+ },
124
+ "eval_overall": {
125
+ "accuracy": 0.985,
126
+ "f1": 0.939,
127
+ "precision": 0.935,
128
+ "recall": 0.943
129
+ }
130
+ }
131
+ ```
132
+
133
+ End2End scores (answering the question: *given a text, what are the spans **and** to which class to they belong?*):
134
+ ```json
135
+ {
136
+ "strict": {
137
+ "per_category": {
138
+ "AGE": {
139
+ "Precision": 0.85,
140
+ "Recall": 0.82,
141
+ "F1": 0.83
142
+ },
143
+ "BUILDINGNUM": {
144
+ "Precision": 0.91,
145
+ "Recall": 0.84,
146
+ "F1": 0.88
147
+ },
148
+ "CITY": {
149
+ "Precision": 0.91,
150
+ "Recall": 0.9,
151
+ "F1": 0.9
152
+ },
153
+ "CREDITCARDNUMBER": {
154
+ "Precision": 0.97,
155
+ "Recall": 0.96,
156
+ "F1": 0.96
157
+ },
158
+ "DATE": {
159
+ "Precision": 0.89,
160
+ "Recall": 0.87,
161
+ "F1": 0.88
162
+ },
163
+ "DRIVERLICENSENUM": {
164
+ "Precision": 0.81,
165
+ "Recall": 0.75,
166
+ "F1": 0.78
167
+ },
168
+ "EMAIL": {
169
+ "Precision": 0.8,
170
+ "Recall": 0.8,
171
+ "F1": 0.8
172
+ },
173
+ "GENDER": {
174
+ "Precision": 0.91,
175
+ "Recall": 0.89,
176
+ "F1": 0.9
177
+ },
178
+ "GIVENNAME": {
179
+ "Precision": 0.85,
180
+ "Recall": 0.88,
181
+ "F1": 0.87
182
+ },
183
+ "IDCARDNUM": {
184
+ "Precision": 0.78,
185
+ "Recall": 0.78,
186
+ "F1": 0.78
187
+ },
188
+ "PASSPORTNUM": {
189
+ "Precision": 0.82,
190
+ "Recall": 0.77,
191
+ "F1": 0.8
192
+ },
193
+ "SEX": {
194
+ "Precision": 0.93,
195
+ "Recall": 0.88,
196
+ "F1": 0.9
197
+ },
198
+ "SOCIALNUM": {
199
+ "Precision": 0.65,
200
+ "Recall": 0.7,
201
+ "F1": 0.67
202
+ },
203
+ "STREET": {
204
+ "Precision": 0.95,
205
+ "Recall": 0.94,
206
+ "F1": 0.94
207
+ },
208
+ "SURNAME": {
209
+ "Precision": 0.87,
210
+ "Recall": 0.86,
211
+ "F1": 0.86
212
+ },
213
+ "TAXNUM": {
214
+ "Precision": 0.79,
215
+ "Recall": 0.69,
216
+ "F1": 0.74
217
+ },
218
+ "TELEPHONENUM": {
219
+ "Precision": 0.72,
220
+ "Recall": 0.7,
221
+ "F1": 0.71
222
+ },
223
+ "TITLE": {
224
+ "Precision": 0.99,
225
+ "Recall": 0.99,
226
+ "F1": 0.99
227
+ },
228
+ "ZIPCODE": {
229
+ "Precision": 0.86,
230
+ "Recall": 0.94,
231
+ "F1": 0.89
232
+ }
233
+ },
234
+ "micro": {
235
+ "Precision": 0.86,
236
+ "Recall": 0.85,
237
+ "F1": 0.86
238
+ },
239
+ "macro": {
240
+ "Precision": 0.86,
241
+ "Recall": 0.84,
242
+ "F1": 0.85
243
+ }
244
+ },
245
+ "relaxed": {
246
+ "per_category": {
247
+ "AGE": {
248
+ "Precision": 0.94,
249
+ "Recall": 0.91,
250
+ "F1": 0.92
251
+ },
252
+ "BUILDINGNUM": {
253
+ "Precision": 0.93,
254
+ "Recall": 0.86,
255
+ "F1": 0.89
256
+ },
257
+ "CITY": {
258
+ "Precision": 0.96,
259
+ "Recall": 0.96,
260
+ "F1": 0.96
261
+ },
262
+ "CREDITCARDNUMBER": {
263
+ "Precision": 0.98,
264
+ "Recall": 0.97,
265
+ "F1": 0.98
266
+ },
267
+ "DATE": {
268
+ "Precision": 0.99,
269
+ "Recall": 0.96,
270
+ "F1": 0.98
271
+ },
272
+ "DRIVERLICENSENUM": {
273
+ "Precision": 0.98,
274
+ "Recall": 0.91,
275
+ "F1": 0.94
276
+ },
277
+ "EMAIL": {
278
+ "Precision": 0.99,
279
+ "Recall": 0.99,
280
+ "F1": 0.99
281
+ },
282
+ "GENDER": {
283
+ "Precision": 0.94,
284
+ "Recall": 0.92,
285
+ "F1": 0.93
286
+ },
287
+ "GIVENNAME": {
288
+ "Precision": 0.94,
289
+ "Recall": 0.97,
290
+ "F1": 0.95
291
+ },
292
+ "IDCARDNUM": {
293
+ "Precision": 0.78,
294
+ "Recall": 0.78,
295
+ "F1": 0.78
296
+ },
297
+ "PASSPORTNUM": {
298
+ "Precision": 0.98,
299
+ "Recall": 0.93,
300
+ "F1": 0.95
301
+ },
302
+ "SEX": {
303
+ "Precision": 0.95,
304
+ "Recall": 0.89,
305
+ "F1": 0.92
306
+ },
307
+ "SOCIALNUM": {
308
+ "Precision": 0.65,
309
+ "Recall": 0.7,
310
+ "F1": 0.67
311
+ },
312
+ "STREET": {
313
+ "Precision": 0.98,
314
+ "Recall": 0.96,
315
+ "F1": 0.97
316
+ },
317
+ "SURNAME": {
318
+ "Precision": 0.96,
319
+ "Recall": 0.95,
320
+ "F1": 0.96
321
+ },
322
+ "TAXNUM": {
323
+ "Precision": 0.79,
324
+ "Recall": 0.7,
325
+ "F1": 0.74
326
+ },
327
+ "TELEPHONENUM": {
328
+ "Precision": 0.97,
329
+ "Recall": 0.95,
330
+ "F1": 0.96
331
+ },
332
+ "TITLE": {
333
+ "Precision": 0.99,
334
+ "Recall": 0.99,
335
+ "F1": 0.99
336
+ },
337
+ "ZIPCODE": {
338
+ "Precision": 0.88,
339
+ "Recall": 0.96,
340
+ "F1": 0.92
341
+ }
342
+ },
343
+ "micro": {
344
+ "Precision": 0.95,
345
+ "Recall": 0.93,
346
+ "F1": 0.94
347
+ },
348
+ "macro": {
349
+ "Precision": 0.93,
350
+ "Recall": 0.91,
351
+ "F1": 0.92
352
+ }
353
+ }
354
+ }
355
+ ```