ProCreations commited on
Commit
e4805d0
·
verified ·
1 Parent(s): b4caad1

Zap v1: 20.7M-param web form & field classifier for autofill (bf16, MIT)

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +196 -0
  2. __pycache__/zap_extract.cpython-312.pyc +0 -0
  3. __pycache__/zap_infer.cpython-312.pyc +0 -0
  4. config.json +107 -0
  5. eval/data_stats.json +90 -0
  6. eval/eval_test_bfloat16.json +1353 -0
  7. eval/eval_test_float32.json +1353 -0
  8. eval/eval_val_bfloat16.json +1338 -0
  9. eval/export_parity.json +7 -0
  10. eval/val_metrics.json +367 -0
  11. model.safetensors +3 -0
  12. onnx/model.onnx +3 -0
  13. onnx/model_quantized.onnx +3 -0
  14. tokenizer.json +0 -0
  15. tokenizer_config.json +9 -0
  16. training/card/infer_parity_py.py +13 -0
  17. training/card/zap_extract.py +580 -0
  18. training/card/zap_infer.py +99 -0
  19. training/harvest/cc_candidates.py +124 -0
  20. training/harvest/fetch_extract.py +144 -0
  21. training/harvest/parity_py.py +10 -0
  22. training/harvest/polite_harvest.py +254 -0
  23. training/harvest/sample_pages.py +30 -0
  24. training/harvest/warc_stream.py +181 -0
  25. training/js/bench.mjs +18 -0
  26. training/js/infer-parity-test.mjs +29 -0
  27. training/js/parity-test.js +30 -0
  28. training/js/zap-extract.js +326 -0
  29. training/js/zap-infer.js +89 -0
  30. training/label/common.py +42 -0
  31. training/label/eval_calib.py +48 -0
  32. training/label/prep_calib.py +15 -0
  33. training/label/select.py +126 -0
  34. training/label/select_rare.py +64 -0
  35. training/label/stats.py +19 -0
  36. training/label/synth.py +162 -0
  37. training/label/teacher.py +191 -0
  38. training/runner/run.sh +27 -0
  39. training/runner/t00_setup.sh +15 -0
  40. training/runner/t01_pylib.sh +8 -0
  41. training/runner/t02_pilot.sh +6 -0
  42. training/runner/t03_harvest.sh +8 -0
  43. training/runner/t04_sglang.sh +18 -0
  44. training/runner/t05_grep.sh +3 -0
  45. training/runner/t06_harvest.sh +3 -0
  46. training/runner/t07_calib_prep.sh +3 -0
  47. training/runner/t08_calib.sh +5 -0
  48. training/runner/t09_warc_test.sh +3 -0
  49. training/runner/t10_warc.sh +3 -0
  50. training/runner/t11_label.sh +6 -0
README.md ADDED
@@ -0,0 +1,196 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: transformers
4
+ pipeline_tag: token-classification
5
+ language:
6
+ - multilingual
7
+ tags:
8
+ - autofill
9
+ - password-manager
10
+ - form-classification
11
+ - field-classification
12
+ - browser-extension
13
+ - bert
14
+ - onnx
15
+ ---
16
+
17
+ # Zap ⚡ — a tiny web-form classifier for autofill
18
+
19
+ **Zap** looks at a web form and tells a password manager what every input wants and what the form is for.
20
+ It runs locally, has **20.7M parameters**, and ships **bf16 weights (41 MB)**, plus ONNX files (fp32 83 MB, int8 21 MB).
21
+
22
+ * **Field types (26):** `username`, `email`, `current-password`, `new-password`, `one-time-code`, `tel`, names, address parts, payment-card parts, `bday`, `search`, `captcha`, `other`
23
+ * **Form purposes (11):** `login`, `signup`, `password-change`, `password-recovery`, `otp`, `payment`, `address`, `newsletter`, `contact`, `search`, `other`
24
+
25
+ So your autofill can answer questions like *"is this a login form?"*, *"which box gets the saved password, and which one gets a newly generated password?"*, *"is this the email field, or is it a honeypot?"*
26
+
27
+ **Held-out websites:** 6,838 real forms and 19,811 fields from Common Crawl pages on hosts never seen in training:
28
+
29
+ | | **Zap** | keyword heuristic¹ |
30
+ |---|---|---|
31
+ | field accuracy (26 classes) | **97.2%** | 82.7% |
32
+ | field macro-F1 | **0.945** | 0.723 |
33
+ | form-purpose accuracy (11 classes) | **95.7%** | 71.6% |
34
+ | password-field detection F1 | **0.996** | 0.968 |
35
+ | current vs new password (on password fields) | **99.2%** | 85.5% |
36
+ | login-identifier (`username`/`email`) F1 | **0.994** | 0.945 |
37
+ | `autocomplete` removed from the input: recovers the developer's own token² | **97.8%** | 78.7% |
38
+
39
+ ¹ A baseline of the kind password managers ship: it trusts a valid `autocomplete`, then the input `type`, then multilingual keyword regexes over name/id/placeholder/label/nearby text.
40
+ ² On 1,622 test fields where the site's `autocomplete` token agrees with the reference label, the token is deleted from the model input. The score is how often Zap still predicts it. This check doesn't depend on the LLM teacher.
41
+
42
+ ## Quick start (Python)
43
+
44
+ ```python
45
+ # pip install transformers torch lxml selectolax huggingface_hub
46
+ import sys
47
+ from huggingface_hub import snapshot_download
48
+ sys.path.insert(0, snapshot_download("ProCreations/Zap"))
49
+ from zap_infer import Zap
50
+
51
+ zap = Zap.from_pretrained("ProCreations/Zap") # CPU, fp32 compute from the bf16 weights
52
+ for form in zap.classify_html(html, url="https://northwind.example/login"):
53
+ print(form["form"], round(form["form_score"], 3))
54
+ for f in form["fields"]:
55
+ print(" ", f["type"], f["name"], "->", f["label"], round(f["score"], 3))
56
+ ```
57
+
58
+ Output for a page that has a sign-in form and a separate "New here? Create account" block:
59
+
60
+ ```
61
+ login 0.966
62
+ input text login -> username 0.96
63
+ input password pw -> current-password 0.962
64
+ signup 0.961
65
+ input text email -> email 0.957
66
+ input password pass 1 -> new-password 0.956
67
+ input password pass 2 -> new-password 0.955
68
+ ```
69
+
70
+ Neither block sets a useful `autocomplete` attribute: the login box has `autocomplete="off"`, and the signup inputs are typeless `<input>`s with placeholders. Zap tells them apart from context.
71
+
72
+ ## In a browser extension (JavaScript)
73
+
74
+ Zap doesn't read raw HTML. It reads a compact text rendering of each form: page title, URL words, form attributes, heading, buttons and links, and for every field its type, name, id, autocomplete, placeholder, label, nearby text and so on. `zap-extract.js` builds that rendering from the **live DOM**, so it sees exactly what the model was trained on. `zap-infer.js` handles windowing and decoding with transformers.js.
75
+
76
+ ```js
77
+ import * as transformers from "@huggingface/transformers";
78
+ // load zap-extract.js and zap-infer.js first (content scripts), or require() them in Node
79
+ const zap = await ZapModel.load(transformers, "ProCreations/Zap"); // onnx/model.onnx; pass { dtype: "q8" } for the int8 file
80
+ const results = await zap.classify(Zap.zapExtract(document, location.href));
81
+ for (const r of results) {
82
+ console.log(r.form, r.formScore);
83
+ for (const f of r.fields) console.log(f.element, f.label, f.score); // f.element is the <input>/<select>/<textarea>
84
+ }
85
+ ```
86
+
87
+ **Parity checks:** these run on 400 random real pages.
88
+ * **Extraction:** the JS extractor (DOM via jsdom) and the Python extractor produce identical model input for **all 821 forms** the JS side finds. The Python side finds 4 extra forms on 2 malformed spam pages. The Python side re-parses pages with an HTML5-spec parser so its tree matches a browser's.
89
+ * **End to end:** JS (transformers.js + ONNX) and Python (PyTorch) predict **identical labels for 818/818 forms and 1,413/1,413 fields**.
90
+
91
+ **Speed:** measured on an Apple M4 Max CPU with onnxruntime via transformers.js, model only.
92
+ * **5.3 ms per form** with fp32 ONNX, and **3.1 ms** with int8 ONNX.
93
+ * The int8 file agrees with fp32 on 99.91% of predictions, and bf16 PyTorch on 99.99%.
94
+
95
+ How the output is decoded: each field is read at its `[FLD]` token, restricted to the `field:*` labels. The form purpose is read at `[CLS]`, restricted to the `form:*` labels. A form longer than 512 tokens is split into windows that each repeat the form context, and the `[CLS]` logits are averaged across windows.
96
+
97
+ ## Labels
98
+
99
+ | field label | meaning |
100
+ |---|---|
101
+ | `username` | account identifier that is not strictly an email: username, user ID, "email or username", "email or phone" |
102
+ | `email` | an email address (login, signup, newsletter, checkout, …) |
103
+ | `current-password` | existing password / sign-in PIN (login, "current password") |
104
+ | `new-password` | password being created or set, including "confirm password" |
105
+ | `one-time-code` | 2FA / OTP / SMS / email verification code, including each box of split code inputs |
106
+ | `tel` | phone number or part of one |
107
+ | `name`, `given-name`, `family-name`, `organization` | person / company names |
108
+ | `street-address`, `address-line2`, `address-level2` (city), `address-level1` (state/region), `postal-code`, `country` | address parts |
109
+ | `cc-name`, `cc-number`, `cc-exp`, `cc-exp-month`, `cc-exp-year`, `cc-csc` | payment card |
110
+ | `bday` | date of birth (or part of it) |
111
+ | `search`, `captcha` | search box, CAPTCHA / anti-spam answer |
112
+ | `other` | everything else: messages, coupons, quantities, honeypots… |
113
+
114
+ Label names follow the HTML `autocomplete` tokens where one exists. `username` vs `email` is decided by what the field accepts: an email-only login box is `email`. On a `login` form, fill the saved login identifier into either one.
115
+
116
+ ## Evaluation
117
+
118
+ Splits are made **by website host**: 4% test, 2% val. Every deduplicated form signature belongs to exactly one split. Headline numbers use only real web forms. Synthetic forms (see below) are scored separately.
119
+
120
+ **Per class, real held-out test** (bf16 weights; F1, with n in parentheses):
121
+
122
+ | field | F1 (n) | field | F1 (n) |
123
+ |---|---|---|---|
124
+ | email | 0.990 (3193) | postal-code | 0.970 (322) |
125
+ | current-password | 0.991 (1838) | captcha | 0.953 (301) |
126
+ | new-password | 0.987 (712) | address-level2 | 0.897 (292) |
127
+ | username | 0.976 (1284) | organization | 0.947 (256) |
128
+ | tel | 0.988 (1337) | street-address | 0.958 (208) |
129
+ | name | 0.962 (1461) | address-level1 | 0.891 (175) |
130
+ | given-name | 0.943 (603) | country | 0.913 (158) |
131
+ | family-name | 0.982 (506) | one-time-code | 0.929 (135) |
132
+ | search | 0.963 (948) | bday | 0.932 (53) |
133
+ | other | 0.968 (5955) | address-line2 | 0.847 (48) |
134
+
135
+ | form | F1 (n) | form | F1 (n) |
136
+ |---|---|---|---|
137
+ | login | 0.992 (1836) | password-change | 0.943 (50) |
138
+ | contact | 0.963 (1640) | otp | 0.897 (40) |
139
+ | search | 0.954 (1043) | password-recovery | 0.933 (38) |
140
+ | newsletter | 0.965 (667) | address | 0.783 (35) |
141
+ | signup | 0.962 (417) | payment | 0.667 (8) |
142
+ | other | 0.893 (1064) | | |
143
+
144
+ Real payment-card forms are very rare in a static crawl: the real test has 8 payment forms and 6 card fields. On the **synthetic** held-out test (84 forms), the card fields `cc-number`, `cc-name`, `cc-exp`, `cc-exp-month`, `cc-exp-year` and `cc-csc` score F1 = 1.00, `payment` forms 0.98 and `otp` forms 1.00. Synthetic forms are easier than real ones, so treat these as a sanity check, not a real-world number.
145
+
146
+ **Most common confusions:** `given-name`↔`name` (a lone "Name" field sitting next to "Last name"), `search`↔`other` (site filters), `username`↔`email` ("Login" boxes that probably take an email), and `address-level1/2`↔`other`.
147
+
148
+ **Label quality:** a second, independent reviewer model re-labeled a stratified sample of 80 teacher-labeled test forms (217 fields) straight from the HTML. It found **1 clear field error out of 210 checkable fields (0.5%)** and 1 clear form-purpose error in 80 (1.3%), plus 6 ambiguous cases of each kind. The one systematic teacher weakness it found is honeypot fields that carry realistic labels.
149
+
150
+ The full reports are in `eval/`, including per-class precision/recall and confusions for bf16 and fp32 weights and the heuristic baseline.
151
+
152
+ ## How it was made
153
+
154
+ **Data.** 62 random WARC files from Common Crawl `CC-MAIN-2026-39` (September 2026) were streamed in full, one request per file. That's 998,946 pages with at least one fillable field, 2.30M forms, and **948,742 unique forms** after deduplicating by form signature. Pages are re-serialized through an HTML5-spec parser (lexbor) before extraction so the tree matches what a browser builds. Fields outside any `<form>` are grouped the way a single-page app lays them out (nearest ancestor with a button).
155
+
156
+ **Labels.** A private, high-batch **Qwen3.8-Flash-Next** (NVFP4, SGLang, thinking off, greedy decoding) ran inside the training job. It labeled every field and the form purpose for **170,163 unique forms** in three rounds:
157
+ 1. A priority-weighted pass: password forms first, then multi-field forms, then single-field ones.
158
+ 2. A targeted pass over not-yet-labeled real forms whose fields look like address, OTP or payment fields.
159
+ 3. The synthetic top-up described below.
160
+
161
+ The teacher saw the cleaned form HTML with numbered fields, the page title and the URL, plus a label spec with rules for the hard cases: current vs new password, `email` vs `username`, honeypots, and `autocomplete` values set only to block autofill.
162
+
163
+ * *Calibration:* on 222 forms where site developers set standard `autocomplete` tokens, the teacher (with `autocomplete` hidden) agreed with the developer token on 84% of fields. Manual inspection showed almost every disagreement was the site being wrong, e.g. Gravity Forms honeypots tagged `autocomplete="new-password"`, or "First Name" fields tagged `name`.
164
+ * *Synthetic top-up:* static crawls rarely contain card-payment, OTP and password-change pages. The teacher therefore also wrote **2,578 synthetic forms** in 23 languages and 12 markup styles, from React div-soup with no `<form>` to split card fields, with deliberately wrong or missing `autocomplete`. They go through the same extractor and the same labeling pass, and make up about 1% of training forms.
165
+
166
+ **Model.** `BertForTokenClassification`: 8 layers, hidden size 384, 6 heads, FFN 1536, 16k WordPiece vocabulary trained on serialized forms, 512 positions. A single token-classification head predicts `form:*` at `[CLS]` and `field:*` at every `[FLD]` marker, so the whole form is classified in one pass and every field sees its neighbours. That is how Zap tells `new-password` from `current-password`, and a signup `email` from a login `username`.
167
+
168
+ **Training.** Done on one RTX PRO 6000 (Blackwell) with bf16 autocast:
169
+ 1. MLM pretraining on 892k unlabeled train-split forms (108M tokens, 3 epochs, about 8 minutes).
170
+ 2. Fine-tuning on 217,782 labeled rows, with rare-class forms oversampled ×3 and label smoothing 0.05. Hint dropout randomly removes `autocomplete` (25%), labels, names and placeholders, and flips 8% of password inputs to `type=text` to mimic "show password" toggles.
171
+ 3. The best of 6 epochs was picked on validation.
172
+
173
+ The published weights are bf16. The whole project took about 6 hours of workstation time: harvesting, three rounds of teacher labeling, synthesis, two training runs, evaluation and export.
174
+
175
+ ## Limitations
176
+
177
+ * Zap reads the DOM text and attributes, not pixels or CSS. In a browser, drop invisible inputs (honeypots) before classifying, e.g. with `offsetParent === null` or `getComputedStyle`. The training crawl only saw static HTML, where some CSS-hidden honeypots remained, so Zap usually labels those `other`, but not always.
178
+ * The training pages come from a static crawl, so heavily scripted flows (identifier-first sign-in, checkout iframes) are covered mainly by synthetic examples. Real-world accuracy on payment forms is not well measured (8 test forms).
179
+ * The labels come from an LLM teacher. It is audited, but it isn't human annotation, so expect a small amount of label noise, especially on ambiguous `name`/`given-name` and `search`/`other` cases.
180
+ * Fields inside cross-origin iframes (e.g. hosted card fields) are invisible to any content script, including `zap-extract.js`.
181
+
182
+ ## Files
183
+
184
+ | file | what |
185
+ |---|---|
186
+ | `model.safetensors` | bf16 weights (BertForTokenClassification, 20.7M params) |
187
+ | `config.json`, `tokenizer.json`, `tokenizer_config.json` | config with the 37-label map; 16k WordPiece tokenizer |
188
+ | `onnx/model.onnx`, `onnx/model_quantized.onnx` | fp32 and int8 ONNX for onnxruntime / transformers.js |
189
+ | `zap_extract.py` / `zap-extract.js` | form extractor (Python canonical / DOM port) |
190
+ | `zap_infer.py` / `zap-infer.js` | windowing + inference helpers |
191
+ | `eval/` | evaluation reports, export parity, dataset statistics |
192
+ | `training/` | the code used to harvest, label, synthesize, train and export |
193
+
194
+ ## License
195
+
196
+ MIT
__pycache__/zap_extract.cpython-312.pyc ADDED
Binary file (31.2 kB). View file
 
__pycache__/zap_infer.cpython-312.pyc ADDED
Binary file (8.5 kB). View file
 
config.json ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_cross_attention": false,
3
+ "architectures": [
4
+ "BertForTokenClassification"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": null,
8
+ "classifier_dropout": null,
9
+ "dtype": "bfloat16",
10
+ "eos_token_id": null,
11
+ "hidden_act": "gelu",
12
+ "hidden_dropout_prob": 0.1,
13
+ "hidden_size": 384,
14
+ "id2label": {
15
+ "0": "form:login",
16
+ "1": "form:signup",
17
+ "2": "form:password-change",
18
+ "3": "form:password-recovery",
19
+ "4": "form:otp",
20
+ "5": "form:payment",
21
+ "6": "form:address",
22
+ "7": "form:newsletter",
23
+ "8": "form:contact",
24
+ "9": "form:search",
25
+ "10": "form:other",
26
+ "11": "field:username",
27
+ "12": "field:email",
28
+ "13": "field:current-password",
29
+ "14": "field:new-password",
30
+ "15": "field:one-time-code",
31
+ "16": "field:tel",
32
+ "17": "field:name",
33
+ "18": "field:given-name",
34
+ "19": "field:family-name",
35
+ "20": "field:organization",
36
+ "21": "field:street-address",
37
+ "22": "field:address-line2",
38
+ "23": "field:address-level2",
39
+ "24": "field:address-level1",
40
+ "25": "field:postal-code",
41
+ "26": "field:country",
42
+ "27": "field:cc-name",
43
+ "28": "field:cc-number",
44
+ "29": "field:cc-exp",
45
+ "30": "field:cc-exp-month",
46
+ "31": "field:cc-exp-year",
47
+ "32": "field:cc-csc",
48
+ "33": "field:bday",
49
+ "34": "field:search",
50
+ "35": "field:captcha",
51
+ "36": "field:other"
52
+ },
53
+ "initializer_range": 0.02,
54
+ "intermediate_size": 1536,
55
+ "is_decoder": false,
56
+ "label2id": {
57
+ "field:address-level1": 24,
58
+ "field:address-level2": 23,
59
+ "field:address-line2": 22,
60
+ "field:bday": 33,
61
+ "field:captcha": 35,
62
+ "field:cc-csc": 32,
63
+ "field:cc-exp": 29,
64
+ "field:cc-exp-month": 30,
65
+ "field:cc-exp-year": 31,
66
+ "field:cc-name": 27,
67
+ "field:cc-number": 28,
68
+ "field:country": 26,
69
+ "field:current-password": 13,
70
+ "field:email": 12,
71
+ "field:family-name": 19,
72
+ "field:given-name": 18,
73
+ "field:name": 17,
74
+ "field:new-password": 14,
75
+ "field:one-time-code": 15,
76
+ "field:organization": 20,
77
+ "field:other": 36,
78
+ "field:postal-code": 25,
79
+ "field:search": 34,
80
+ "field:street-address": 21,
81
+ "field:tel": 16,
82
+ "field:username": 11,
83
+ "form:address": 6,
84
+ "form:contact": 8,
85
+ "form:login": 0,
86
+ "form:newsletter": 7,
87
+ "form:other": 10,
88
+ "form:otp": 4,
89
+ "form:password-change": 2,
90
+ "form:password-recovery": 3,
91
+ "form:payment": 5,
92
+ "form:search": 9,
93
+ "form:signup": 1
94
+ },
95
+ "layer_norm_eps": 1e-12,
96
+ "max_position_embeddings": 512,
97
+ "model_type": "bert",
98
+ "num_attention_heads": 6,
99
+ "num_hidden_layers": 8,
100
+ "pad_token_id": 0,
101
+ "tie_word_embeddings": true,
102
+ "transformers_version": "5.12.1",
103
+ "type_vocab_size": 1,
104
+ "use_cache": true,
105
+ "vocab_size": 16384,
106
+ "torch_dtype": "bfloat16"
107
+ }
eval/data_stats.json ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "stats": {
3
+ "mlm": 893251,
4
+ "train": 217782,
5
+ "val": 3408,
6
+ "test": 6922
7
+ },
8
+ "train_form_labels": {
9
+ "search": 36885,
10
+ "other": 33139,
11
+ "newsletter": 22424,
12
+ "contact": 44859,
13
+ "login": 59124,
14
+ "signup": 13290,
15
+ "otp": 1992,
16
+ "address": 1694,
17
+ "payment": 1170,
18
+ "password-change": 1605,
19
+ "password-recovery": 1600
20
+ },
21
+ "train_field_labels": {
22
+ "search": 33272,
23
+ "other": 171342,
24
+ "email": 98108,
25
+ "name": 40121,
26
+ "tel": 37210,
27
+ "street-address": 6196,
28
+ "postal-code": 10203,
29
+ "captcha": 8692,
30
+ "username": 41964,
31
+ "current-password": 59298,
32
+ "organization": 7200,
33
+ "given-name": 18361,
34
+ "family-name": 15434,
35
+ "new-password": 22226,
36
+ "one-time-code": 5861,
37
+ "address-level1": 4746,
38
+ "cc-name": 780,
39
+ "address-level2": 8701,
40
+ "country": 4902,
41
+ "cc-number": 961,
42
+ "cc-exp-month": 297,
43
+ "cc-exp-year": 290,
44
+ "cc-csc": 887,
45
+ "bday": 2171,
46
+ "cc-exp": 625,
47
+ "address-line2": 1254
48
+ },
49
+ "field_labels": [
50
+ "username",
51
+ "email",
52
+ "current-password",
53
+ "new-password",
54
+ "one-time-code",
55
+ "tel",
56
+ "name",
57
+ "given-name",
58
+ "family-name",
59
+ "organization",
60
+ "street-address",
61
+ "address-line2",
62
+ "address-level2",
63
+ "address-level1",
64
+ "postal-code",
65
+ "country",
66
+ "cc-name",
67
+ "cc-number",
68
+ "cc-exp",
69
+ "cc-exp-month",
70
+ "cc-exp-year",
71
+ "cc-csc",
72
+ "bday",
73
+ "search",
74
+ "captcha",
75
+ "other"
76
+ ],
77
+ "form_labels": [
78
+ "login",
79
+ "signup",
80
+ "password-change",
81
+ "password-recovery",
82
+ "otp",
83
+ "payment",
84
+ "address",
85
+ "newsletter",
86
+ "contact",
87
+ "search",
88
+ "other"
89
+ ]
90
+ }
eval/eval_test_bfloat16.json ADDED
@@ -0,0 +1,1353 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "split": "test",
3
+ "dtype": "bfloat16",
4
+ "forms_per_s_gpu": 2022.5467036804282,
5
+ "model": {
6
+ "n_forms": 6922,
7
+ "n_fields": 20130,
8
+ "field_acc": 0.9719821162444113,
9
+ "field_macro_f1": 0.9597690495523868,
10
+ "form_acc": 0.9572377925455071,
11
+ "form_macro_f1": 0.9327853059996515,
12
+ "password_precision": 0.9942263279445728,
13
+ "password_recall": 0.9976825028968713,
14
+ "password_f1": 0.9959514170040485,
15
+ "cur_vs_new_acc": 0.9915025106218617,
16
+ "identifier_f1": 0.9939004103360319,
17
+ "per_field": {
18
+ "username": {
19
+ "p": 0.9782101167315175,
20
+ "r": 0.9736638264910922,
21
+ "f1": 0.9759316770186335,
22
+ "n": 1291
23
+ },
24
+ "email": {
25
+ "p": 0.9870049504950495,
26
+ "r": 0.9940791523839202,
27
+ "f1": 0.9905294208973761,
28
+ "n": 3209
29
+ },
30
+ "current-password": {
31
+ "p": 0.9866238630283574,
32
+ "r": 0.9956803455723542,
33
+ "f1": 0.9911314162859447,
34
+ "n": 1852
35
+ },
36
+ "new-password": {
37
+ "p": 0.9917695473251029,
38
+ "r": 0.9810040705563093,
39
+ "f1": 0.9863574351978172,
40
+ "n": 737
41
+ },
42
+ "one-time-code": {
43
+ "p": 0.9318181818181818,
44
+ "r": 0.9425287356321839,
45
+ "f1": 0.9371428571428572,
46
+ "n": 174
47
+ },
48
+ "tel": {
49
+ "p": 0.9809802487198245,
50
+ "r": 0.9933333333333333,
51
+ "f1": 0.9871181450128819,
52
+ "n": 1350
53
+ },
54
+ "name": {
55
+ "p": 0.9501661129568106,
56
+ "r": 0.9747784594410361,
57
+ "f1": 0.9623149394347241,
58
+ "n": 1467
59
+ },
60
+ "given-name": {
61
+ "p": 0.9754385964912281,
62
+ "r": 0.9129720853858785,
63
+ "f1": 0.9431721798134012,
64
+ "n": 609
65
+ },
66
+ "family-name": {
67
+ "p": 0.9881422924901185,
68
+ "r": 0.9765625,
69
+ "f1": 0.9823182711198427,
70
+ "n": 512
71
+ },
72
+ "organization": {
73
+ "p": 0.953307392996109,
74
+ "r": 0.9423076923076923,
75
+ "f1": 0.9477756286266925,
76
+ "n": 260
77
+ },
78
+ "street-address": {
79
+ "p": 0.9396551724137931,
80
+ "r": 0.9775784753363229,
81
+ "f1": 0.9582417582417583,
82
+ "n": 223
83
+ },
84
+ "address-line2": {
85
+ "p": 0.975609756097561,
86
+ "r": 0.7547169811320755,
87
+ "f1": 0.851063829787234,
88
+ "n": 53
89
+ },
90
+ "address-level2": {
91
+ "p": 0.9149659863945578,
92
+ "r": 0.8848684210526315,
93
+ "f1": 0.8996655518394647,
94
+ "n": 304
95
+ },
96
+ "address-level1": {
97
+ "p": 0.9005524861878453,
98
+ "r": 0.8858695652173914,
99
+ "f1": 0.8931506849315068,
100
+ "n": 184
101
+ },
102
+ "postal-code": {
103
+ "p": 0.9731343283582089,
104
+ "r": 0.9673590504451038,
105
+ "f1": 0.9702380952380952,
106
+ "n": 337
107
+ },
108
+ "country": {
109
+ "p": 0.8743169398907104,
110
+ "r": 0.9580838323353293,
111
+ "f1": 0.9142857142857143,
112
+ "n": 167
113
+ },
114
+ "cc-name": {
115
+ "p": 0.9583333333333334,
116
+ "r": 1.0,
117
+ "f1": 0.9787234042553191,
118
+ "n": 23
119
+ },
120
+ "cc-number": {
121
+ "p": 0.9696969696969697,
122
+ "r": 1.0,
123
+ "f1": 0.9846153846153847,
124
+ "n": 32
125
+ },
126
+ "cc-exp": {
127
+ "p": 1.0,
128
+ "r": 1.0,
129
+ "f1": 1.0,
130
+ "n": 18
131
+ },
132
+ "cc-exp-month": {
133
+ "p": 1.0,
134
+ "r": 1.0,
135
+ "f1": 1.0,
136
+ "n": 14
137
+ },
138
+ "cc-exp-year": {
139
+ "p": 1.0,
140
+ "r": 1.0,
141
+ "f1": 1.0,
142
+ "n": 14
143
+ },
144
+ "cc-csc": {
145
+ "p": 1.0,
146
+ "r": 0.967741935483871,
147
+ "f1": 0.9836065573770492,
148
+ "n": 31
149
+ },
150
+ "bday": {
151
+ "p": 0.96,
152
+ "r": 0.9056603773584906,
153
+ "f1": 0.9320388349514563,
154
+ "n": 53
155
+ },
156
+ "search": {
157
+ "p": 0.9670212765957447,
158
+ "r": 0.9588607594936709,
159
+ "f1": 0.9629237288135594,
160
+ "n": 948
161
+ },
162
+ "captcha": {
163
+ "p": 0.9632107023411371,
164
+ "r": 0.9442622950819672,
165
+ "f1": 0.9536423841059603,
166
+ "n": 305
167
+ },
168
+ "other": {
169
+ "p": 0.9693911873528422,
170
+ "r": 0.9666275364749287,
171
+ "f1": 0.9680073893693845,
172
+ "n": 5963
173
+ }
174
+ },
175
+ "per_form": {
176
+ "login": {
177
+ "p": 0.9886914378029079,
178
+ "r": 0.9956616052060737,
179
+ "f1": 0.9921642799243447,
180
+ "n": 1844
181
+ },
182
+ "signup": {
183
+ "p": 0.9777227722772277,
184
+ "r": 0.947242206235012,
185
+ "f1": 0.9622411693057246,
186
+ "n": 417
187
+ },
188
+ "password-change": {
189
+ "p": 0.9154929577464789,
190
+ "r": 0.9848484848484849,
191
+ "f1": 0.948905109489051,
192
+ "n": 66
193
+ },
194
+ "password-recovery": {
195
+ "p": 0.9555555555555556,
196
+ "r": 0.9347826086956522,
197
+ "f1": 0.945054945054945,
198
+ "n": 46
199
+ },
200
+ "otp": {
201
+ "p": 0.9464285714285714,
202
+ "r": 0.9137931034482759,
203
+ "f1": 0.9298245614035087,
204
+ "n": 58
205
+ },
206
+ "payment": {
207
+ "p": 0.9393939393939394,
208
+ "r": 0.8857142857142857,
209
+ "f1": 0.9117647058823529,
210
+ "n": 35
211
+ },
212
+ "address": {
213
+ "p": 0.8048780487804879,
214
+ "r": 0.7857142857142857,
215
+ "f1": 0.7951807228915663,
216
+ "n": 42
217
+ },
218
+ "newsletter": {
219
+ "p": 0.9740458015267176,
220
+ "r": 0.9565217391304348,
221
+ "f1": 0.9652042360060515,
222
+ "n": 667
223
+ },
224
+ "contact": {
225
+ "p": 0.9459141681363904,
226
+ "r": 0.9810975609756097,
227
+ "f1": 0.9631846752469321,
228
+ "n": 1640
229
+ },
230
+ "search": {
231
+ "p": 0.967487684729064,
232
+ "r": 0.9415148609779482,
233
+ "f1": 0.9543245869776482,
234
+ "n": 1043
235
+ },
236
+ "other": {
237
+ "p": 0.9013409961685823,
238
+ "r": 0.8843984962406015,
239
+ "f1": 0.8927893738140418,
240
+ "n": 1064
241
+ }
242
+ },
243
+ "top_confusions": [
244
+ [
245
+ "given-name",
246
+ "name",
247
+ 48
248
+ ],
249
+ [
250
+ "search",
251
+ "other",
252
+ 39
253
+ ],
254
+ [
255
+ "address-level2",
256
+ "other",
257
+ 28
258
+ ],
259
+ [
260
+ "username",
261
+ "email",
262
+ 26
263
+ ],
264
+ [
265
+ "other",
266
+ "search",
267
+ 25
268
+ ],
269
+ [
270
+ "other",
271
+ "address-level2",
272
+ 22
273
+ ],
274
+ [
275
+ "other",
276
+ "country",
277
+ 19
278
+ ],
279
+ [
280
+ "address-level1",
281
+ "other",
282
+ 18
283
+ ],
284
+ [
285
+ "name",
286
+ "other",
287
+ 15
288
+ ],
289
+ [
290
+ "other",
291
+ "address-level1",
292
+ 15
293
+ ],
294
+ [
295
+ "other",
296
+ "tel",
297
+ 15
298
+ ],
299
+ [
300
+ "new-password",
301
+ "current-password",
302
+ 14
303
+ ],
304
+ [
305
+ "captcha",
306
+ "other",
307
+ 14
308
+ ],
309
+ [
310
+ "other",
311
+ "username",
312
+ 13
313
+ ],
314
+ [
315
+ "other",
316
+ "name",
317
+ 12
318
+ ],
319
+ [
320
+ "other",
321
+ "email",
322
+ 11
323
+ ],
324
+ [
325
+ "address-line2",
326
+ "other",
327
+ 11
328
+ ],
329
+ [
330
+ "other",
331
+ "street-address",
332
+ 11
333
+ ],
334
+ [
335
+ "other",
336
+ "current-password",
337
+ 10
338
+ ],
339
+ [
340
+ "name",
341
+ "given-name",
342
+ 10
343
+ ],
344
+ [
345
+ "other",
346
+ "captcha",
347
+ 9
348
+ ],
349
+ [
350
+ "other",
351
+ "postal-code",
352
+ 9
353
+ ],
354
+ [
355
+ "email",
356
+ "username",
357
+ 8
358
+ ],
359
+ [
360
+ "email",
361
+ "other",
362
+ 8
363
+ ],
364
+ [
365
+ "organization",
366
+ "other",
367
+ 8
368
+ ]
369
+ ]
370
+ },
371
+ "model_real": {
372
+ "n_forms": 6838,
373
+ "n_fields": 19811,
374
+ "field_acc": 0.971884306698299,
375
+ "field_macro_f1": 0.9447870267362889,
376
+ "form_acc": 0.9571512138052062,
377
+ "form_macro_f1": 0.9049098319929233,
378
+ "password_precision": 0.9941383352872216,
379
+ "password_recall": 0.9976470588235294,
380
+ "password_f1": 0.9958896065766295,
381
+ "cur_vs_new_acc": 0.991764705882353,
382
+ "identifier_f1": 0.993869133875822,
383
+ "per_field": {
384
+ "username": {
385
+ "p": 0.9780907668231612,
386
+ "r": 0.9735202492211839,
387
+ "f1": 0.975800156128025,
388
+ "n": 1284
389
+ },
390
+ "email": {
391
+ "p": 0.9869402985074627,
392
+ "r": 0.9940494832445975,
393
+ "f1": 0.9904821344983618,
394
+ "n": 3193
395
+ },
396
+ "current-password": {
397
+ "p": 0.9870550161812298,
398
+ "r": 0.9956474428726877,
399
+ "f1": 0.991332611050921,
400
+ "n": 1838
401
+ },
402
+ "new-password": {
403
+ "p": 0.9914893617021276,
404
+ "r": 0.9817415730337079,
405
+ "f1": 0.9865913902611151,
406
+ "n": 712
407
+ },
408
+ "one-time-code": {
409
+ "p": 0.9328358208955224,
410
+ "r": 0.9259259259259259,
411
+ "f1": 0.929368029739777,
412
+ "n": 135
413
+ },
414
+ "tel": {
415
+ "p": 0.9822485207100592,
416
+ "r": 0.9932685115931189,
417
+ "f1": 0.9877277798438081,
418
+ "n": 1337
419
+ },
420
+ "name": {
421
+ "p": 0.9499666444296198,
422
+ "r": 0.974674880219028,
423
+ "f1": 0.9621621621621621,
424
+ "n": 1461
425
+ },
426
+ "given-name": {
427
+ "p": 0.975177304964539,
428
+ "r": 0.912106135986733,
429
+ "f1": 0.9425878320479862,
430
+ "n": 603
431
+ },
432
+ "family-name": {
433
+ "p": 0.988,
434
+ "r": 0.9762845849802372,
435
+ "f1": 0.9821073558648111,
436
+ "n": 506
437
+ },
438
+ "organization": {
439
+ "p": 0.9525691699604744,
440
+ "r": 0.94140625,
441
+ "f1": 0.9469548133595285,
442
+ "n": 256
443
+ },
444
+ "street-address": {
445
+ "p": 0.9357798165137615,
446
+ "r": 0.9807692307692307,
447
+ "f1": 0.9577464788732395,
448
+ "n": 208
449
+ },
450
+ "address-line2": {
451
+ "p": 0.972972972972973,
452
+ "r": 0.75,
453
+ "f1": 0.8470588235294119,
454
+ "n": 48
455
+ },
456
+ "address-level2": {
457
+ "p": 0.911660777385159,
458
+ "r": 0.8835616438356164,
459
+ "f1": 0.8973913043478261,
460
+ "n": 292
461
+ },
462
+ "address-level1": {
463
+ "p": 0.8959537572254336,
464
+ "r": 0.8857142857142857,
465
+ "f1": 0.8908045977011495,
466
+ "n": 175
467
+ },
468
+ "postal-code": {
469
+ "p": 0.9719626168224299,
470
+ "r": 0.968944099378882,
471
+ "f1": 0.9704510108864696,
472
+ "n": 322
473
+ },
474
+ "country": {
475
+ "p": 0.8685714285714285,
476
+ "r": 0.9620253164556962,
477
+ "f1": 0.9129129129129129,
478
+ "n": 158
479
+ },
480
+ "cc-name": {
481
+ "p": 0.6666666666666666,
482
+ "r": 1.0,
483
+ "f1": 0.8,
484
+ "n": 2
485
+ },
486
+ "cc-number": {
487
+ "p": 0.8571428571428571,
488
+ "r": 1.0,
489
+ "f1": 0.923076923076923,
490
+ "n": 6
491
+ },
492
+ "cc-exp-month": {
493
+ "p": 1.0,
494
+ "r": 1.0,
495
+ "f1": 1.0,
496
+ "n": 6
497
+ },
498
+ "cc-exp-year": {
499
+ "p": 1.0,
500
+ "r": 1.0,
501
+ "f1": 1.0,
502
+ "n": 6
503
+ },
504
+ "cc-csc": {
505
+ "p": 1.0,
506
+ "r": 0.8333333333333334,
507
+ "f1": 0.9090909090909091,
508
+ "n": 6
509
+ },
510
+ "bday": {
511
+ "p": 0.96,
512
+ "r": 0.9056603773584906,
513
+ "f1": 0.9320388349514563,
514
+ "n": 53
515
+ },
516
+ "search": {
517
+ "p": 0.9670212765957447,
518
+ "r": 0.9588607594936709,
519
+ "f1": 0.9629237288135594,
520
+ "n": 948
521
+ },
522
+ "captcha": {
523
+ "p": 0.9627118644067797,
524
+ "r": 0.9435215946843853,
525
+ "f1": 0.953020134228188,
526
+ "n": 301
527
+ },
528
+ "other": {
529
+ "p": 0.9695132221660772,
530
+ "r": 0.9665827036104114,
531
+ "f1": 0.9680457450386813,
532
+ "n": 5955
533
+ }
534
+ },
535
+ "per_form": {
536
+ "login": {
537
+ "p": 0.9891774891774892,
538
+ "r": 0.9956427015250545,
539
+ "f1": 0.992399565689468,
540
+ "n": 1836
541
+ },
542
+ "signup": {
543
+ "p": 0.9777227722772277,
544
+ "r": 0.947242206235012,
545
+ "f1": 0.9622411693057246,
546
+ "n": 417
547
+ },
548
+ "password-change": {
549
+ "p": 0.8928571428571429,
550
+ "r": 1.0,
551
+ "f1": 0.9433962264150945,
552
+ "n": 50
553
+ },
554
+ "password-recovery": {
555
+ "p": 0.9459459459459459,
556
+ "r": 0.9210526315789473,
557
+ "f1": 0.9333333333333332,
558
+ "n": 38
559
+ },
560
+ "otp": {
561
+ "p": 0.9210526315789473,
562
+ "r": 0.875,
563
+ "f1": 0.8974358974358975,
564
+ "n": 40
565
+ },
566
+ "payment": {
567
+ "p": 0.7142857142857143,
568
+ "r": 0.625,
569
+ "f1": 0.6666666666666666,
570
+ "n": 8
571
+ },
572
+ "address": {
573
+ "p": 0.7941176470588235,
574
+ "r": 0.7714285714285715,
575
+ "f1": 0.782608695652174,
576
+ "n": 35
577
+ },
578
+ "newsletter": {
579
+ "p": 0.9740458015267176,
580
+ "r": 0.9565217391304348,
581
+ "f1": 0.9652042360060515,
582
+ "n": 667
583
+ },
584
+ "contact": {
585
+ "p": 0.9459141681363904,
586
+ "r": 0.9810975609756097,
587
+ "f1": 0.9631846752469321,
588
+ "n": 1640
589
+ },
590
+ "search": {
591
+ "p": 0.967487684729064,
592
+ "r": 0.9415148609779482,
593
+ "f1": 0.9543245869776482,
594
+ "n": 1043
595
+ },
596
+ "other": {
597
+ "p": 0.9022051773729626,
598
+ "r": 0.8843984962406015,
599
+ "f1": 0.8932130991931656,
600
+ "n": 1064
601
+ }
602
+ },
603
+ "top_confusions": [
604
+ [
605
+ "given-name",
606
+ "name",
607
+ 48
608
+ ],
609
+ [
610
+ "search",
611
+ "other",
612
+ 39
613
+ ],
614
+ [
615
+ "address-level2",
616
+ "other",
617
+ 27
618
+ ],
619
+ [
620
+ "username",
621
+ "email",
622
+ 26
623
+ ],
624
+ [
625
+ "other",
626
+ "search",
627
+ 25
628
+ ],
629
+ [
630
+ "other",
631
+ "address-level2",
632
+ 22
633
+ ],
634
+ [
635
+ "other",
636
+ "country",
637
+ 19
638
+ ],
639
+ [
640
+ "address-level1",
641
+ "other",
642
+ 18
643
+ ],
644
+ [
645
+ "name",
646
+ "other",
647
+ 15
648
+ ],
649
+ [
650
+ "other",
651
+ "address-level1",
652
+ 15
653
+ ],
654
+ [
655
+ "other",
656
+ "tel",
657
+ 15
658
+ ],
659
+ [
660
+ "captcha",
661
+ "other",
662
+ 14
663
+ ],
664
+ [
665
+ "new-password",
666
+ "current-password",
667
+ 13
668
+ ],
669
+ [
670
+ "other",
671
+ "username",
672
+ 13
673
+ ],
674
+ [
675
+ "other",
676
+ "name",
677
+ 12
678
+ ],
679
+ [
680
+ "other",
681
+ "email",
682
+ 11
683
+ ],
684
+ [
685
+ "address-line2",
686
+ "other",
687
+ 11
688
+ ],
689
+ [
690
+ "other",
691
+ "street-address",
692
+ 11
693
+ ],
694
+ [
695
+ "other",
696
+ "current-password",
697
+ 10
698
+ ],
699
+ [
700
+ "name",
701
+ "given-name",
702
+ 10
703
+ ],
704
+ [
705
+ "other",
706
+ "captcha",
707
+ 9
708
+ ],
709
+ [
710
+ "other",
711
+ "postal-code",
712
+ 9
713
+ ],
714
+ [
715
+ "email",
716
+ "username",
717
+ 8
718
+ ],
719
+ [
720
+ "email",
721
+ "other",
722
+ 8
723
+ ],
724
+ [
725
+ "organization",
726
+ "other",
727
+ 8
728
+ ]
729
+ ]
730
+ },
731
+ "model_synthetic": {
732
+ "n_forms": 84,
733
+ "n_fields": 319,
734
+ "field_acc": 0.9780564263322884,
735
+ "field_macro_f1": 0.9765257496746269,
736
+ "form_acc": 0.9642857142857143,
737
+ "form_macro_f1": 0.9578655564477768,
738
+ "password_precision": 1.0,
739
+ "password_recall": 1.0,
740
+ "password_f1": 1.0,
741
+ "cur_vs_new_acc": 0.9743589743589743,
742
+ "identifier_f1": 1.0,
743
+ "per_field": {
744
+ "username": {
745
+ "p": 1.0,
746
+ "r": 1.0,
747
+ "f1": 1.0,
748
+ "n": 7
749
+ },
750
+ "email": {
751
+ "p": 1.0,
752
+ "r": 1.0,
753
+ "f1": 1.0,
754
+ "n": 16
755
+ },
756
+ "current-password": {
757
+ "p": 0.9333333333333333,
758
+ "r": 1.0,
759
+ "f1": 0.9655172413793104,
760
+ "n": 14
761
+ },
762
+ "new-password": {
763
+ "p": 1.0,
764
+ "r": 0.96,
765
+ "f1": 0.9795918367346939,
766
+ "n": 25
767
+ },
768
+ "one-time-code": {
769
+ "p": 0.9285714285714286,
770
+ "r": 1.0,
771
+ "f1": 0.962962962962963,
772
+ "n": 39
773
+ },
774
+ "tel": {
775
+ "p": 0.8666666666666667,
776
+ "r": 1.0,
777
+ "f1": 0.9285714285714286,
778
+ "n": 13
779
+ },
780
+ "name": {
781
+ "p": 1.0,
782
+ "r": 1.0,
783
+ "f1": 1.0,
784
+ "n": 6
785
+ },
786
+ "given-name": {
787
+ "p": 1.0,
788
+ "r": 1.0,
789
+ "f1": 1.0,
790
+ "n": 6
791
+ },
792
+ "family-name": {
793
+ "p": 1.0,
794
+ "r": 1.0,
795
+ "f1": 1.0,
796
+ "n": 6
797
+ },
798
+ "organization": {
799
+ "p": 1.0,
800
+ "r": 1.0,
801
+ "f1": 1.0,
802
+ "n": 4
803
+ },
804
+ "street-address": {
805
+ "p": 1.0,
806
+ "r": 0.9333333333333333,
807
+ "f1": 0.9655172413793104,
808
+ "n": 15
809
+ },
810
+ "address-line2": {
811
+ "p": 1.0,
812
+ "r": 0.8,
813
+ "f1": 0.888888888888889,
814
+ "n": 5
815
+ },
816
+ "address-level2": {
817
+ "p": 1.0,
818
+ "r": 0.9166666666666666,
819
+ "f1": 0.9565217391304348,
820
+ "n": 12
821
+ },
822
+ "address-level1": {
823
+ "p": 1.0,
824
+ "r": 0.8888888888888888,
825
+ "f1": 0.9411764705882353,
826
+ "n": 9
827
+ },
828
+ "postal-code": {
829
+ "p": 1.0,
830
+ "r": 0.9333333333333333,
831
+ "f1": 0.9655172413793104,
832
+ "n": 15
833
+ },
834
+ "country": {
835
+ "p": 1.0,
836
+ "r": 0.8888888888888888,
837
+ "f1": 0.9411764705882353,
838
+ "n": 9
839
+ },
840
+ "cc-name": {
841
+ "p": 1.0,
842
+ "r": 1.0,
843
+ "f1": 1.0,
844
+ "n": 21
845
+ },
846
+ "cc-number": {
847
+ "p": 1.0,
848
+ "r": 1.0,
849
+ "f1": 1.0,
850
+ "n": 26
851
+ },
852
+ "cc-exp": {
853
+ "p": 1.0,
854
+ "r": 1.0,
855
+ "f1": 1.0,
856
+ "n": 18
857
+ },
858
+ "cc-exp-month": {
859
+ "p": 1.0,
860
+ "r": 1.0,
861
+ "f1": 1.0,
862
+ "n": 8
863
+ },
864
+ "cc-exp-year": {
865
+ "p": 1.0,
866
+ "r": 1.0,
867
+ "f1": 1.0,
868
+ "n": 8
869
+ },
870
+ "cc-csc": {
871
+ "p": 1.0,
872
+ "r": 1.0,
873
+ "f1": 1.0,
874
+ "n": 25
875
+ },
876
+ "captcha": {
877
+ "p": 1.0,
878
+ "r": 1.0,
879
+ "f1": 1.0,
880
+ "n": 4
881
+ },
882
+ "other": {
883
+ "p": 0.8888888888888888,
884
+ "r": 1.0,
885
+ "f1": 0.9411764705882353,
886
+ "n": 8
887
+ }
888
+ },
889
+ "per_form": {
890
+ "login": {
891
+ "p": 0.8888888888888888,
892
+ "r": 1.0,
893
+ "f1": 0.9411764705882353,
894
+ "n": 8
895
+ },
896
+ "password-change": {
897
+ "p": 1.0,
898
+ "r": 0.9375,
899
+ "f1": 0.967741935483871,
900
+ "n": 16
901
+ },
902
+ "password-recovery": {
903
+ "p": 1.0,
904
+ "r": 1.0,
905
+ "f1": 1.0,
906
+ "n": 8
907
+ },
908
+ "otp": {
909
+ "p": 1.0,
910
+ "r": 1.0,
911
+ "f1": 1.0,
912
+ "n": 18
913
+ },
914
+ "payment": {
915
+ "p": 1.0,
916
+ "r": 0.9629629629629629,
917
+ "f1": 0.9811320754716981,
918
+ "n": 27
919
+ },
920
+ "address": {
921
+ "p": 0.8571428571428571,
922
+ "r": 0.8571428571428571,
923
+ "f1": 0.8571428571428571,
924
+ "n": 7
925
+ }
926
+ },
927
+ "top_confusions": [
928
+ [
929
+ "new-password",
930
+ "current-password",
931
+ 1
932
+ ],
933
+ [
934
+ "street-address",
935
+ "tel",
936
+ 1
937
+ ],
938
+ [
939
+ "address-line2",
940
+ "one-time-code",
941
+ 1
942
+ ],
943
+ [
944
+ "address-level2",
945
+ "other",
946
+ 1
947
+ ],
948
+ [
949
+ "address-level1",
950
+ "one-time-code",
951
+ 1
952
+ ],
953
+ [
954
+ "postal-code",
955
+ "one-time-code",
956
+ 1
957
+ ],
958
+ [
959
+ "country",
960
+ "tel",
961
+ 1
962
+ ]
963
+ ]
964
+ },
965
+ "heuristic_baseline_real": {
966
+ "n_forms": 6838,
967
+ "n_fields": 19811,
968
+ "field_acc": 0.827368633587401,
969
+ "field_macro_f1": 0.7233335250918939,
970
+ "form_acc": 0.7157063468850541,
971
+ "form_macro_f1": 0.48529320010648663,
972
+ "password_precision": 0.9391143911439115,
973
+ "password_recall": 0.9980392156862745,
974
+ "password_f1": 0.967680608365019,
975
+ "cur_vs_new_acc": 0.8545098039215686,
976
+ "identifier_f1": 0.9453604914322663,
977
+ "per_field": {
978
+ "username": {
979
+ "p": 0.7456213511259383,
980
+ "r": 0.6962616822429907,
981
+ "f1": 0.7200966572694322,
982
+ "n": 1284
983
+ },
984
+ "email": {
985
+ "p": 0.8662225922842076,
986
+ "r": 0.9774506733479487,
987
+ "f1": 0.918481459682166,
988
+ "n": 3193
989
+ },
990
+ "current-password": {
991
+ "p": 0.8279176201372997,
992
+ "r": 0.984221980413493,
993
+ "f1": 0.8993288590604027,
994
+ "n": 1838
995
+ },
996
+ "new-password": {
997
+ "p": 0.7047619047619048,
998
+ "r": 0.5196629213483146,
999
+ "f1": 0.5982215036378336,
1000
+ "n": 712
1001
+ },
1002
+ "one-time-code": {
1003
+ "p": 0.6486486486486487,
1004
+ "r": 0.35555555555555557,
1005
+ "f1": 0.45933014354066987,
1006
+ "n": 135
1007
+ },
1008
+ "tel": {
1009
+ "p": 0.8867791842475387,
1010
+ "r": 0.9431563201196709,
1011
+ "f1": 0.91409931134469,
1012
+ "n": 1337
1013
+ },
1014
+ "name": {
1015
+ "p": 0.8157695223654283,
1016
+ "r": 0.7364818617385352,
1017
+ "f1": 0.7741007194244605,
1018
+ "n": 1461
1019
+ },
1020
+ "given-name": {
1021
+ "p": 0.6842105263157895,
1022
+ "r": 0.7114427860696517,
1023
+ "f1": 0.697560975609756,
1024
+ "n": 603
1025
+ },
1026
+ "family-name": {
1027
+ "p": 0.9160671462829736,
1028
+ "r": 0.7549407114624506,
1029
+ "f1": 0.8277356446370531,
1030
+ "n": 506
1031
+ },
1032
+ "organization": {
1033
+ "p": 0.8041666666666667,
1034
+ "r": 0.75390625,
1035
+ "f1": 0.778225806451613,
1036
+ "n": 256
1037
+ },
1038
+ "street-address": {
1039
+ "p": 0.6370370370370371,
1040
+ "r": 0.8269230769230769,
1041
+ "f1": 0.7196652719665272,
1042
+ "n": 208
1043
+ },
1044
+ "address-line2": {
1045
+ "p": 0.453125,
1046
+ "r": 0.6041666666666666,
1047
+ "f1": 0.5178571428571429,
1048
+ "n": 48
1049
+ },
1050
+ "address-level2": {
1051
+ "p": 0.6318537859007833,
1052
+ "r": 0.8287671232876712,
1053
+ "f1": 0.717037037037037,
1054
+ "n": 292
1055
+ },
1056
+ "address-level1": {
1057
+ "p": 0.5258964143426295,
1058
+ "r": 0.7542857142857143,
1059
+ "f1": 0.6197183098591549,
1060
+ "n": 175
1061
+ },
1062
+ "postal-code": {
1063
+ "p": 0.925,
1064
+ "r": 0.8043478260869565,
1065
+ "f1": 0.8604651162790697,
1066
+ "n": 322
1067
+ },
1068
+ "country": {
1069
+ "p": 0.8226950354609929,
1070
+ "r": 0.7341772151898734,
1071
+ "f1": 0.7759197324414716,
1072
+ "n": 158
1073
+ },
1074
+ "cc-name": {
1075
+ "p": 0.5,
1076
+ "r": 1.0,
1077
+ "f1": 0.6666666666666666,
1078
+ "n": 2
1079
+ },
1080
+ "cc-number": {
1081
+ "p": 0.5,
1082
+ "r": 1.0,
1083
+ "f1": 0.6666666666666666,
1084
+ "n": 6
1085
+ },
1086
+ "cc-exp-month": {
1087
+ "p": 1.0,
1088
+ "r": 0.5,
1089
+ "f1": 0.6666666666666666,
1090
+ "n": 6
1091
+ },
1092
+ "cc-exp-year": {
1093
+ "p": 0.6,
1094
+ "r": 0.5,
1095
+ "f1": 0.5454545454545454,
1096
+ "n": 6
1097
+ },
1098
+ "cc-csc": {
1099
+ "p": 0.4166666666666667,
1100
+ "r": 0.8333333333333334,
1101
+ "f1": 0.5555555555555556,
1102
+ "n": 6
1103
+ },
1104
+ "bday": {
1105
+ "p": 0.9318181818181818,
1106
+ "r": 0.7735849056603774,
1107
+ "f1": 0.8453608247422681,
1108
+ "n": 53
1109
+ },
1110
+ "search": {
1111
+ "p": 0.8284714119019837,
1112
+ "r": 0.7489451476793249,
1113
+ "f1": 0.7867036011080333,
1114
+ "n": 948
1115
+ },
1116
+ "captcha": {
1117
+ "p": 0.9297297297297298,
1118
+ "r": 0.5714285714285714,
1119
+ "f1": 0.7078189300411523,
1120
+ "n": 301
1121
+ },
1122
+ "other": {
1123
+ "p": 0.8645796693633486,
1124
+ "r": 0.8255247691015953,
1125
+ "f1": 0.8446009792973114,
1126
+ "n": 5955
1127
+ }
1128
+ },
1129
+ "per_form": {
1130
+ "login": {
1131
+ "p": 0.94069640914037,
1132
+ "r": 0.9417211328976035,
1133
+ "f1": 0.9412084921066958,
1134
+ "n": 1836
1135
+ },
1136
+ "signup": {
1137
+ "p": 0.5643224699828473,
1138
+ "r": 0.7889688249400479,
1139
+ "f1": 0.6579999999999999,
1140
+ "n": 417
1141
+ },
1142
+ "password-change": {
1143
+ "p": 0.0,
1144
+ "r": 0.0,
1145
+ "f1": 0.0,
1146
+ "n": 50
1147
+ },
1148
+ "password-recovery": {
1149
+ "p": 0.26666666666666666,
1150
+ "r": 0.21052631578947367,
1151
+ "f1": 0.23529411764705882,
1152
+ "n": 38
1153
+ },
1154
+ "otp": {
1155
+ "p": 0.3,
1156
+ "r": 0.375,
1157
+ "f1": 0.33333333333333326,
1158
+ "n": 40
1159
+ },
1160
+ "payment": {
1161
+ "p": 0.21052631578947367,
1162
+ "r": 0.5,
1163
+ "f1": 0.2962962962962963,
1164
+ "n": 8
1165
+ },
1166
+ "address": {
1167
+ "p": 0.03722504230118443,
1168
+ "r": 0.6285714285714286,
1169
+ "f1": 0.07028753993610225,
1170
+ "n": 35
1171
+ },
1172
+ "newsletter": {
1173
+ "p": 0.8864774624373957,
1174
+ "r": 0.7961019490254873,
1175
+ "f1": 0.8388625592417062,
1176
+ "n": 667
1177
+ },
1178
+ "contact": {
1179
+ "p": 0.9479371316306483,
1180
+ "r": 0.5884146341463414,
1181
+ "f1": 0.7261098570353649,
1182
+ "n": 1640
1183
+ },
1184
+ "search": {
1185
+ "p": 0.984313725490196,
1186
+ "r": 0.4813039309683605,
1187
+ "f1": 0.6464906632324533,
1188
+ "n": 1043
1189
+ },
1190
+ "other": {
1191
+ "p": 0.493125,
1192
+ "r": 0.7415413533834586,
1193
+ "f1": 0.5923423423423424,
1194
+ "n": 1064
1195
+ }
1196
+ },
1197
+ "top_confusions": [
1198
+ [
1199
+ "new-password",
1200
+ "current-password",
1201
+ 339
1202
+ ],
1203
+ [
1204
+ "username",
1205
+ "email",
1206
+ 309
1207
+ ],
1208
+ [
1209
+ "name",
1210
+ "given-name",
1211
+ 148
1212
+ ],
1213
+ [
1214
+ "name",
1215
+ "other",
1216
+ 135
1217
+ ],
1218
+ [
1219
+ "other",
1220
+ "search",
1221
+ 134
1222
+ ],
1223
+ [
1224
+ "search",
1225
+ "other",
1226
+ 128
1227
+ ],
1228
+ [
1229
+ "other",
1230
+ "email",
1231
+ 118
1232
+ ],
1233
+ [
1234
+ "other",
1235
+ "new-password",
1236
+ 117
1237
+ ],
1238
+ [
1239
+ "other",
1240
+ "address-level2",
1241
+ 108
1242
+ ],
1243
+ [
1244
+ "captcha",
1245
+ "other",
1246
+ 100
1247
+ ],
1248
+ [
1249
+ "given-name",
1250
+ "name",
1251
+ 91
1252
+ ],
1253
+ [
1254
+ "other",
1255
+ "name",
1256
+ 89
1257
+ ],
1258
+ [
1259
+ "other",
1260
+ "username",
1261
+ 84
1262
+ ],
1263
+ [
1264
+ "other",
1265
+ "tel",
1266
+ 84
1267
+ ],
1268
+ [
1269
+ "other",
1270
+ "address-level1",
1271
+ 82
1272
+ ],
1273
+ [
1274
+ "family-name",
1275
+ "other",
1276
+ 63
1277
+ ],
1278
+ [
1279
+ "email",
1280
+ "username",
1281
+ 62
1282
+ ],
1283
+ [
1284
+ "tel",
1285
+ "other",
1286
+ 54
1287
+ ],
1288
+ [
1289
+ "name",
1290
+ "username",
1291
+ 52
1292
+ ],
1293
+ [
1294
+ "username",
1295
+ "other",
1296
+ 49
1297
+ ],
1298
+ [
1299
+ "given-name",
1300
+ "other",
1301
+ 39
1302
+ ],
1303
+ [
1304
+ "other",
1305
+ "street-address",
1306
+ 39
1307
+ ],
1308
+ [
1309
+ "one-time-code",
1310
+ "other",
1311
+ 39
1312
+ ],
1313
+ [
1314
+ "other",
1315
+ "current-password",
1316
+ 32
1317
+ ],
1318
+ [
1319
+ "other",
1320
+ "address-line2",
1321
+ 31
1322
+ ]
1323
+ ]
1324
+ },
1325
+ "autocomplete_masked": {
1326
+ "n_fields": 1622,
1327
+ "acc": 0.9784217016029593,
1328
+ "per_class": {
1329
+ "address-level1": "4/4",
1330
+ "address-level2": "12/14",
1331
+ "address-line2": "0/3",
1332
+ "bday": "5/5",
1333
+ "cc-csc": "1/1",
1334
+ "cc-exp-month": "1/1",
1335
+ "cc-exp-year": "1/1",
1336
+ "cc-number": "1/1",
1337
+ "country": "6/6",
1338
+ "current-password": "323/324",
1339
+ "email": "417/418",
1340
+ "family-name": "34/37",
1341
+ "given-name": "39/42",
1342
+ "name": "182/187",
1343
+ "new-password": "180/180",
1344
+ "one-time-code": "7/7",
1345
+ "organization": "16/16",
1346
+ "postal-code": "26/30",
1347
+ "street-address": "11/13",
1348
+ "tel": "111/111",
1349
+ "username": "210/221"
1350
+ },
1351
+ "heuristic_acc": 0.7866831072749692
1352
+ }
1353
+ }
eval/eval_test_float32.json ADDED
@@ -0,0 +1,1353 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "split": "test",
3
+ "dtype": "float32",
4
+ "forms_per_s_gpu": 3416.2872663427547,
5
+ "model": {
6
+ "n_forms": 6922,
7
+ "n_fields": 20130,
8
+ "field_acc": 0.9719324391455539,
9
+ "field_macro_f1": 0.9597248737921183,
10
+ "form_acc": 0.9569488587113552,
11
+ "form_macro_f1": 0.932616790369714,
12
+ "password_precision": 0.9942263279445728,
13
+ "password_recall": 0.9976825028968713,
14
+ "password_f1": 0.9959514170040485,
15
+ "cur_vs_new_acc": 0.9915025106218617,
16
+ "identifier_f1": 0.9939004103360319,
17
+ "per_field": {
18
+ "username": {
19
+ "p": 0.9782101167315175,
20
+ "r": 0.9736638264910922,
21
+ "f1": 0.9759316770186335,
22
+ "n": 1291
23
+ },
24
+ "email": {
25
+ "p": 0.9870049504950495,
26
+ "r": 0.9940791523839202,
27
+ "f1": 0.9905294208973761,
28
+ "n": 3209
29
+ },
30
+ "current-password": {
31
+ "p": 0.9866238630283574,
32
+ "r": 0.9956803455723542,
33
+ "f1": 0.9911314162859447,
34
+ "n": 1852
35
+ },
36
+ "new-password": {
37
+ "p": 0.9917695473251029,
38
+ "r": 0.9810040705563093,
39
+ "f1": 0.9863574351978172,
40
+ "n": 737
41
+ },
42
+ "one-time-code": {
43
+ "p": 0.9318181818181818,
44
+ "r": 0.9425287356321839,
45
+ "f1": 0.9371428571428572,
46
+ "n": 174
47
+ },
48
+ "tel": {
49
+ "p": 0.9809802487198245,
50
+ "r": 0.9933333333333333,
51
+ "f1": 0.9871181450128819,
52
+ "n": 1350
53
+ },
54
+ "name": {
55
+ "p": 0.9501329787234043,
56
+ "r": 0.9740967961826857,
57
+ "f1": 0.9619656681252104,
58
+ "n": 1467
59
+ },
60
+ "given-name": {
61
+ "p": 0.9737302977232924,
62
+ "r": 0.9129720853858785,
63
+ "f1": 0.9423728813559322,
64
+ "n": 609
65
+ },
66
+ "family-name": {
67
+ "p": 0.9881422924901185,
68
+ "r": 0.9765625,
69
+ "f1": 0.9823182711198427,
70
+ "n": 512
71
+ },
72
+ "organization": {
73
+ "p": 0.953307392996109,
74
+ "r": 0.9423076923076923,
75
+ "f1": 0.9477756286266925,
76
+ "n": 260
77
+ },
78
+ "street-address": {
79
+ "p": 0.9396551724137931,
80
+ "r": 0.9775784753363229,
81
+ "f1": 0.9582417582417583,
82
+ "n": 223
83
+ },
84
+ "address-line2": {
85
+ "p": 0.975609756097561,
86
+ "r": 0.7547169811320755,
87
+ "f1": 0.851063829787234,
88
+ "n": 53
89
+ },
90
+ "address-level2": {
91
+ "p": 0.9149659863945578,
92
+ "r": 0.8848684210526315,
93
+ "f1": 0.8996655518394647,
94
+ "n": 304
95
+ },
96
+ "address-level1": {
97
+ "p": 0.9005524861878453,
98
+ "r": 0.8858695652173914,
99
+ "f1": 0.8931506849315068,
100
+ "n": 184
101
+ },
102
+ "postal-code": {
103
+ "p": 0.9731343283582089,
104
+ "r": 0.9673590504451038,
105
+ "f1": 0.9702380952380952,
106
+ "n": 337
107
+ },
108
+ "country": {
109
+ "p": 0.8743169398907104,
110
+ "r": 0.9580838323353293,
111
+ "f1": 0.9142857142857143,
112
+ "n": 167
113
+ },
114
+ "cc-name": {
115
+ "p": 0.9583333333333334,
116
+ "r": 1.0,
117
+ "f1": 0.9787234042553191,
118
+ "n": 23
119
+ },
120
+ "cc-number": {
121
+ "p": 0.9696969696969697,
122
+ "r": 1.0,
123
+ "f1": 0.9846153846153847,
124
+ "n": 32
125
+ },
126
+ "cc-exp": {
127
+ "p": 1.0,
128
+ "r": 1.0,
129
+ "f1": 1.0,
130
+ "n": 18
131
+ },
132
+ "cc-exp-month": {
133
+ "p": 1.0,
134
+ "r": 1.0,
135
+ "f1": 1.0,
136
+ "n": 14
137
+ },
138
+ "cc-exp-year": {
139
+ "p": 1.0,
140
+ "r": 1.0,
141
+ "f1": 1.0,
142
+ "n": 14
143
+ },
144
+ "cc-csc": {
145
+ "p": 1.0,
146
+ "r": 0.967741935483871,
147
+ "f1": 0.9836065573770492,
148
+ "n": 31
149
+ },
150
+ "bday": {
151
+ "p": 0.96,
152
+ "r": 0.9056603773584906,
153
+ "f1": 0.9320388349514563,
154
+ "n": 53
155
+ },
156
+ "search": {
157
+ "p": 0.9670212765957447,
158
+ "r": 0.9588607594936709,
159
+ "f1": 0.9629237288135594,
160
+ "n": 948
161
+ },
162
+ "captcha": {
163
+ "p": 0.9632107023411371,
164
+ "r": 0.9442622950819672,
165
+ "f1": 0.9536423841059603,
166
+ "n": 305
167
+ },
168
+ "other": {
169
+ "p": 0.9693911873528422,
170
+ "r": 0.9666275364749287,
171
+ "f1": 0.9680073893693845,
172
+ "n": 5963
173
+ }
174
+ },
175
+ "per_form": {
176
+ "login": {
177
+ "p": 0.9886914378029079,
178
+ "r": 0.9956616052060737,
179
+ "f1": 0.9921642799243447,
180
+ "n": 1844
181
+ },
182
+ "signup": {
183
+ "p": 0.9777227722772277,
184
+ "r": 0.947242206235012,
185
+ "f1": 0.9622411693057246,
186
+ "n": 417
187
+ },
188
+ "password-change": {
189
+ "p": 0.9154929577464789,
190
+ "r": 0.9848484848484849,
191
+ "f1": 0.948905109489051,
192
+ "n": 66
193
+ },
194
+ "password-recovery": {
195
+ "p": 0.9555555555555556,
196
+ "r": 0.9347826086956522,
197
+ "f1": 0.945054945054945,
198
+ "n": 46
199
+ },
200
+ "otp": {
201
+ "p": 0.9464285714285714,
202
+ "r": 0.9137931034482759,
203
+ "f1": 0.9298245614035087,
204
+ "n": 58
205
+ },
206
+ "payment": {
207
+ "p": 0.9393939393939394,
208
+ "r": 0.8857142857142857,
209
+ "f1": 0.9117647058823529,
210
+ "n": 35
211
+ },
212
+ "address": {
213
+ "p": 0.8048780487804879,
214
+ "r": 0.7857142857142857,
215
+ "f1": 0.7951807228915663,
216
+ "n": 42
217
+ },
218
+ "newsletter": {
219
+ "p": 0.9725609756097561,
220
+ "r": 0.9565217391304348,
221
+ "f1": 0.9644746787603932,
222
+ "n": 667
223
+ },
224
+ "contact": {
225
+ "p": 0.9453262786596119,
226
+ "r": 0.9804878048780488,
227
+ "f1": 0.9625860520802155,
228
+ "n": 1640
229
+ },
230
+ "search": {
231
+ "p": 0.967487684729064,
232
+ "r": 0.9415148609779482,
233
+ "f1": 0.9543245869776482,
234
+ "n": 1043
235
+ },
236
+ "other": {
237
+ "p": 0.9012464046021093,
238
+ "r": 0.8834586466165414,
239
+ "f1": 0.8922638822971048,
240
+ "n": 1064
241
+ }
242
+ },
243
+ "top_confusions": [
244
+ [
245
+ "given-name",
246
+ "name",
247
+ 48
248
+ ],
249
+ [
250
+ "search",
251
+ "other",
252
+ 39
253
+ ],
254
+ [
255
+ "address-level2",
256
+ "other",
257
+ 28
258
+ ],
259
+ [
260
+ "username",
261
+ "email",
262
+ 26
263
+ ],
264
+ [
265
+ "other",
266
+ "search",
267
+ 25
268
+ ],
269
+ [
270
+ "other",
271
+ "address-level2",
272
+ 22
273
+ ],
274
+ [
275
+ "other",
276
+ "country",
277
+ 19
278
+ ],
279
+ [
280
+ "address-level1",
281
+ "other",
282
+ 18
283
+ ],
284
+ [
285
+ "name",
286
+ "other",
287
+ 15
288
+ ],
289
+ [
290
+ "other",
291
+ "address-level1",
292
+ 15
293
+ ],
294
+ [
295
+ "other",
296
+ "tel",
297
+ 15
298
+ ],
299
+ [
300
+ "new-password",
301
+ "current-password",
302
+ 14
303
+ ],
304
+ [
305
+ "captcha",
306
+ "other",
307
+ 14
308
+ ],
309
+ [
310
+ "other",
311
+ "username",
312
+ 13
313
+ ],
314
+ [
315
+ "other",
316
+ "name",
317
+ 12
318
+ ],
319
+ [
320
+ "other",
321
+ "email",
322
+ 11
323
+ ],
324
+ [
325
+ "name",
326
+ "given-name",
327
+ 11
328
+ ],
329
+ [
330
+ "address-line2",
331
+ "other",
332
+ 11
333
+ ],
334
+ [
335
+ "other",
336
+ "street-address",
337
+ 11
338
+ ],
339
+ [
340
+ "other",
341
+ "current-password",
342
+ 10
343
+ ],
344
+ [
345
+ "other",
346
+ "captcha",
347
+ 9
348
+ ],
349
+ [
350
+ "other",
351
+ "postal-code",
352
+ 9
353
+ ],
354
+ [
355
+ "email",
356
+ "username",
357
+ 8
358
+ ],
359
+ [
360
+ "email",
361
+ "other",
362
+ 8
363
+ ],
364
+ [
365
+ "organization",
366
+ "other",
367
+ 8
368
+ ]
369
+ ]
370
+ },
371
+ "model_real": {
372
+ "n_forms": 6838,
373
+ "n_fields": 19811,
374
+ "field_acc": 0.971833829690576,
375
+ "field_macro_f1": 0.9447407167557139,
376
+ "form_acc": 0.9568587306229892,
377
+ "form_macro_f1": 0.9047413119700765,
378
+ "password_precision": 0.9941383352872216,
379
+ "password_recall": 0.9976470588235294,
380
+ "password_f1": 0.9958896065766295,
381
+ "cur_vs_new_acc": 0.991764705882353,
382
+ "identifier_f1": 0.993869133875822,
383
+ "per_field": {
384
+ "username": {
385
+ "p": 0.9780907668231612,
386
+ "r": 0.9735202492211839,
387
+ "f1": 0.975800156128025,
388
+ "n": 1284
389
+ },
390
+ "email": {
391
+ "p": 0.9869402985074627,
392
+ "r": 0.9940494832445975,
393
+ "f1": 0.9904821344983618,
394
+ "n": 3193
395
+ },
396
+ "current-password": {
397
+ "p": 0.9870550161812298,
398
+ "r": 0.9956474428726877,
399
+ "f1": 0.991332611050921,
400
+ "n": 1838
401
+ },
402
+ "new-password": {
403
+ "p": 0.9914893617021276,
404
+ "r": 0.9817415730337079,
405
+ "f1": 0.9865913902611151,
406
+ "n": 712
407
+ },
408
+ "one-time-code": {
409
+ "p": 0.9328358208955224,
410
+ "r": 0.9259259259259259,
411
+ "f1": 0.929368029739777,
412
+ "n": 135
413
+ },
414
+ "tel": {
415
+ "p": 0.9822485207100592,
416
+ "r": 0.9932685115931189,
417
+ "f1": 0.9877277798438081,
418
+ "n": 1337
419
+ },
420
+ "name": {
421
+ "p": 0.9499332443257676,
422
+ "r": 0.973990417522245,
423
+ "f1": 0.9618114227779655,
424
+ "n": 1461
425
+ },
426
+ "given-name": {
427
+ "p": 0.9734513274336283,
428
+ "r": 0.912106135986733,
429
+ "f1": 0.9417808219178082,
430
+ "n": 603
431
+ },
432
+ "family-name": {
433
+ "p": 0.988,
434
+ "r": 0.9762845849802372,
435
+ "f1": 0.9821073558648111,
436
+ "n": 506
437
+ },
438
+ "organization": {
439
+ "p": 0.9525691699604744,
440
+ "r": 0.94140625,
441
+ "f1": 0.9469548133595285,
442
+ "n": 256
443
+ },
444
+ "street-address": {
445
+ "p": 0.9357798165137615,
446
+ "r": 0.9807692307692307,
447
+ "f1": 0.9577464788732395,
448
+ "n": 208
449
+ },
450
+ "address-line2": {
451
+ "p": 0.972972972972973,
452
+ "r": 0.75,
453
+ "f1": 0.8470588235294119,
454
+ "n": 48
455
+ },
456
+ "address-level2": {
457
+ "p": 0.911660777385159,
458
+ "r": 0.8835616438356164,
459
+ "f1": 0.8973913043478261,
460
+ "n": 292
461
+ },
462
+ "address-level1": {
463
+ "p": 0.8959537572254336,
464
+ "r": 0.8857142857142857,
465
+ "f1": 0.8908045977011495,
466
+ "n": 175
467
+ },
468
+ "postal-code": {
469
+ "p": 0.9719626168224299,
470
+ "r": 0.968944099378882,
471
+ "f1": 0.9704510108864696,
472
+ "n": 322
473
+ },
474
+ "country": {
475
+ "p": 0.8685714285714285,
476
+ "r": 0.9620253164556962,
477
+ "f1": 0.9129129129129129,
478
+ "n": 158
479
+ },
480
+ "cc-name": {
481
+ "p": 0.6666666666666666,
482
+ "r": 1.0,
483
+ "f1": 0.8,
484
+ "n": 2
485
+ },
486
+ "cc-number": {
487
+ "p": 0.8571428571428571,
488
+ "r": 1.0,
489
+ "f1": 0.923076923076923,
490
+ "n": 6
491
+ },
492
+ "cc-exp-month": {
493
+ "p": 1.0,
494
+ "r": 1.0,
495
+ "f1": 1.0,
496
+ "n": 6
497
+ },
498
+ "cc-exp-year": {
499
+ "p": 1.0,
500
+ "r": 1.0,
501
+ "f1": 1.0,
502
+ "n": 6
503
+ },
504
+ "cc-csc": {
505
+ "p": 1.0,
506
+ "r": 0.8333333333333334,
507
+ "f1": 0.9090909090909091,
508
+ "n": 6
509
+ },
510
+ "bday": {
511
+ "p": 0.96,
512
+ "r": 0.9056603773584906,
513
+ "f1": 0.9320388349514563,
514
+ "n": 53
515
+ },
516
+ "search": {
517
+ "p": 0.9670212765957447,
518
+ "r": 0.9588607594936709,
519
+ "f1": 0.9629237288135594,
520
+ "n": 948
521
+ },
522
+ "captcha": {
523
+ "p": 0.9627118644067797,
524
+ "r": 0.9435215946843853,
525
+ "f1": 0.953020134228188,
526
+ "n": 301
527
+ },
528
+ "other": {
529
+ "p": 0.9695132221660772,
530
+ "r": 0.9665827036104114,
531
+ "f1": 0.9680457450386813,
532
+ "n": 5955
533
+ }
534
+ },
535
+ "per_form": {
536
+ "login": {
537
+ "p": 0.9891774891774892,
538
+ "r": 0.9956427015250545,
539
+ "f1": 0.992399565689468,
540
+ "n": 1836
541
+ },
542
+ "signup": {
543
+ "p": 0.9777227722772277,
544
+ "r": 0.947242206235012,
545
+ "f1": 0.9622411693057246,
546
+ "n": 417
547
+ },
548
+ "password-change": {
549
+ "p": 0.8928571428571429,
550
+ "r": 1.0,
551
+ "f1": 0.9433962264150945,
552
+ "n": 50
553
+ },
554
+ "password-recovery": {
555
+ "p": 0.9459459459459459,
556
+ "r": 0.9210526315789473,
557
+ "f1": 0.9333333333333332,
558
+ "n": 38
559
+ },
560
+ "otp": {
561
+ "p": 0.9210526315789473,
562
+ "r": 0.875,
563
+ "f1": 0.8974358974358975,
564
+ "n": 40
565
+ },
566
+ "payment": {
567
+ "p": 0.7142857142857143,
568
+ "r": 0.625,
569
+ "f1": 0.6666666666666666,
570
+ "n": 8
571
+ },
572
+ "address": {
573
+ "p": 0.7941176470588235,
574
+ "r": 0.7714285714285715,
575
+ "f1": 0.782608695652174,
576
+ "n": 35
577
+ },
578
+ "newsletter": {
579
+ "p": 0.9725609756097561,
580
+ "r": 0.9565217391304348,
581
+ "f1": 0.9644746787603932,
582
+ "n": 667
583
+ },
584
+ "contact": {
585
+ "p": 0.9453262786596119,
586
+ "r": 0.9804878048780488,
587
+ "f1": 0.9625860520802155,
588
+ "n": 1640
589
+ },
590
+ "search": {
591
+ "p": 0.967487684729064,
592
+ "r": 0.9415148609779482,
593
+ "f1": 0.9543245869776482,
594
+ "n": 1043
595
+ },
596
+ "other": {
597
+ "p": 0.9021113243761996,
598
+ "r": 0.8834586466165414,
599
+ "f1": 0.892687559354226,
600
+ "n": 1064
601
+ }
602
+ },
603
+ "top_confusions": [
604
+ [
605
+ "given-name",
606
+ "name",
607
+ 48
608
+ ],
609
+ [
610
+ "search",
611
+ "other",
612
+ 39
613
+ ],
614
+ [
615
+ "address-level2",
616
+ "other",
617
+ 27
618
+ ],
619
+ [
620
+ "username",
621
+ "email",
622
+ 26
623
+ ],
624
+ [
625
+ "other",
626
+ "search",
627
+ 25
628
+ ],
629
+ [
630
+ "other",
631
+ "address-level2",
632
+ 22
633
+ ],
634
+ [
635
+ "other",
636
+ "country",
637
+ 19
638
+ ],
639
+ [
640
+ "address-level1",
641
+ "other",
642
+ 18
643
+ ],
644
+ [
645
+ "name",
646
+ "other",
647
+ 15
648
+ ],
649
+ [
650
+ "other",
651
+ "address-level1",
652
+ 15
653
+ ],
654
+ [
655
+ "other",
656
+ "tel",
657
+ 15
658
+ ],
659
+ [
660
+ "captcha",
661
+ "other",
662
+ 14
663
+ ],
664
+ [
665
+ "new-password",
666
+ "current-password",
667
+ 13
668
+ ],
669
+ [
670
+ "other",
671
+ "username",
672
+ 13
673
+ ],
674
+ [
675
+ "other",
676
+ "name",
677
+ 12
678
+ ],
679
+ [
680
+ "other",
681
+ "email",
682
+ 11
683
+ ],
684
+ [
685
+ "name",
686
+ "given-name",
687
+ 11
688
+ ],
689
+ [
690
+ "address-line2",
691
+ "other",
692
+ 11
693
+ ],
694
+ [
695
+ "other",
696
+ "street-address",
697
+ 11
698
+ ],
699
+ [
700
+ "other",
701
+ "current-password",
702
+ 10
703
+ ],
704
+ [
705
+ "other",
706
+ "captcha",
707
+ 9
708
+ ],
709
+ [
710
+ "other",
711
+ "postal-code",
712
+ 9
713
+ ],
714
+ [
715
+ "email",
716
+ "username",
717
+ 8
718
+ ],
719
+ [
720
+ "email",
721
+ "other",
722
+ 8
723
+ ],
724
+ [
725
+ "organization",
726
+ "other",
727
+ 8
728
+ ]
729
+ ]
730
+ },
731
+ "model_synthetic": {
732
+ "n_forms": 84,
733
+ "n_fields": 319,
734
+ "field_acc": 0.9780564263322884,
735
+ "field_macro_f1": 0.9765257496746269,
736
+ "form_acc": 0.9642857142857143,
737
+ "form_macro_f1": 0.9578655564477768,
738
+ "password_precision": 1.0,
739
+ "password_recall": 1.0,
740
+ "password_f1": 1.0,
741
+ "cur_vs_new_acc": 0.9743589743589743,
742
+ "identifier_f1": 1.0,
743
+ "per_field": {
744
+ "username": {
745
+ "p": 1.0,
746
+ "r": 1.0,
747
+ "f1": 1.0,
748
+ "n": 7
749
+ },
750
+ "email": {
751
+ "p": 1.0,
752
+ "r": 1.0,
753
+ "f1": 1.0,
754
+ "n": 16
755
+ },
756
+ "current-password": {
757
+ "p": 0.9333333333333333,
758
+ "r": 1.0,
759
+ "f1": 0.9655172413793104,
760
+ "n": 14
761
+ },
762
+ "new-password": {
763
+ "p": 1.0,
764
+ "r": 0.96,
765
+ "f1": 0.9795918367346939,
766
+ "n": 25
767
+ },
768
+ "one-time-code": {
769
+ "p": 0.9285714285714286,
770
+ "r": 1.0,
771
+ "f1": 0.962962962962963,
772
+ "n": 39
773
+ },
774
+ "tel": {
775
+ "p": 0.8666666666666667,
776
+ "r": 1.0,
777
+ "f1": 0.9285714285714286,
778
+ "n": 13
779
+ },
780
+ "name": {
781
+ "p": 1.0,
782
+ "r": 1.0,
783
+ "f1": 1.0,
784
+ "n": 6
785
+ },
786
+ "given-name": {
787
+ "p": 1.0,
788
+ "r": 1.0,
789
+ "f1": 1.0,
790
+ "n": 6
791
+ },
792
+ "family-name": {
793
+ "p": 1.0,
794
+ "r": 1.0,
795
+ "f1": 1.0,
796
+ "n": 6
797
+ },
798
+ "organization": {
799
+ "p": 1.0,
800
+ "r": 1.0,
801
+ "f1": 1.0,
802
+ "n": 4
803
+ },
804
+ "street-address": {
805
+ "p": 1.0,
806
+ "r": 0.9333333333333333,
807
+ "f1": 0.9655172413793104,
808
+ "n": 15
809
+ },
810
+ "address-line2": {
811
+ "p": 1.0,
812
+ "r": 0.8,
813
+ "f1": 0.888888888888889,
814
+ "n": 5
815
+ },
816
+ "address-level2": {
817
+ "p": 1.0,
818
+ "r": 0.9166666666666666,
819
+ "f1": 0.9565217391304348,
820
+ "n": 12
821
+ },
822
+ "address-level1": {
823
+ "p": 1.0,
824
+ "r": 0.8888888888888888,
825
+ "f1": 0.9411764705882353,
826
+ "n": 9
827
+ },
828
+ "postal-code": {
829
+ "p": 1.0,
830
+ "r": 0.9333333333333333,
831
+ "f1": 0.9655172413793104,
832
+ "n": 15
833
+ },
834
+ "country": {
835
+ "p": 1.0,
836
+ "r": 0.8888888888888888,
837
+ "f1": 0.9411764705882353,
838
+ "n": 9
839
+ },
840
+ "cc-name": {
841
+ "p": 1.0,
842
+ "r": 1.0,
843
+ "f1": 1.0,
844
+ "n": 21
845
+ },
846
+ "cc-number": {
847
+ "p": 1.0,
848
+ "r": 1.0,
849
+ "f1": 1.0,
850
+ "n": 26
851
+ },
852
+ "cc-exp": {
853
+ "p": 1.0,
854
+ "r": 1.0,
855
+ "f1": 1.0,
856
+ "n": 18
857
+ },
858
+ "cc-exp-month": {
859
+ "p": 1.0,
860
+ "r": 1.0,
861
+ "f1": 1.0,
862
+ "n": 8
863
+ },
864
+ "cc-exp-year": {
865
+ "p": 1.0,
866
+ "r": 1.0,
867
+ "f1": 1.0,
868
+ "n": 8
869
+ },
870
+ "cc-csc": {
871
+ "p": 1.0,
872
+ "r": 1.0,
873
+ "f1": 1.0,
874
+ "n": 25
875
+ },
876
+ "captcha": {
877
+ "p": 1.0,
878
+ "r": 1.0,
879
+ "f1": 1.0,
880
+ "n": 4
881
+ },
882
+ "other": {
883
+ "p": 0.8888888888888888,
884
+ "r": 1.0,
885
+ "f1": 0.9411764705882353,
886
+ "n": 8
887
+ }
888
+ },
889
+ "per_form": {
890
+ "login": {
891
+ "p": 0.8888888888888888,
892
+ "r": 1.0,
893
+ "f1": 0.9411764705882353,
894
+ "n": 8
895
+ },
896
+ "password-change": {
897
+ "p": 1.0,
898
+ "r": 0.9375,
899
+ "f1": 0.967741935483871,
900
+ "n": 16
901
+ },
902
+ "password-recovery": {
903
+ "p": 1.0,
904
+ "r": 1.0,
905
+ "f1": 1.0,
906
+ "n": 8
907
+ },
908
+ "otp": {
909
+ "p": 1.0,
910
+ "r": 1.0,
911
+ "f1": 1.0,
912
+ "n": 18
913
+ },
914
+ "payment": {
915
+ "p": 1.0,
916
+ "r": 0.9629629629629629,
917
+ "f1": 0.9811320754716981,
918
+ "n": 27
919
+ },
920
+ "address": {
921
+ "p": 0.8571428571428571,
922
+ "r": 0.8571428571428571,
923
+ "f1": 0.8571428571428571,
924
+ "n": 7
925
+ }
926
+ },
927
+ "top_confusions": [
928
+ [
929
+ "new-password",
930
+ "current-password",
931
+ 1
932
+ ],
933
+ [
934
+ "street-address",
935
+ "tel",
936
+ 1
937
+ ],
938
+ [
939
+ "address-line2",
940
+ "one-time-code",
941
+ 1
942
+ ],
943
+ [
944
+ "address-level2",
945
+ "other",
946
+ 1
947
+ ],
948
+ [
949
+ "address-level1",
950
+ "one-time-code",
951
+ 1
952
+ ],
953
+ [
954
+ "postal-code",
955
+ "one-time-code",
956
+ 1
957
+ ],
958
+ [
959
+ "country",
960
+ "tel",
961
+ 1
962
+ ]
963
+ ]
964
+ },
965
+ "heuristic_baseline_real": {
966
+ "n_forms": 6838,
967
+ "n_fields": 19811,
968
+ "field_acc": 0.827368633587401,
969
+ "field_macro_f1": 0.7233335250918939,
970
+ "form_acc": 0.7157063468850541,
971
+ "form_macro_f1": 0.48529320010648663,
972
+ "password_precision": 0.9391143911439115,
973
+ "password_recall": 0.9980392156862745,
974
+ "password_f1": 0.967680608365019,
975
+ "cur_vs_new_acc": 0.8545098039215686,
976
+ "identifier_f1": 0.9453604914322663,
977
+ "per_field": {
978
+ "username": {
979
+ "p": 0.7456213511259383,
980
+ "r": 0.6962616822429907,
981
+ "f1": 0.7200966572694322,
982
+ "n": 1284
983
+ },
984
+ "email": {
985
+ "p": 0.8662225922842076,
986
+ "r": 0.9774506733479487,
987
+ "f1": 0.918481459682166,
988
+ "n": 3193
989
+ },
990
+ "current-password": {
991
+ "p": 0.8279176201372997,
992
+ "r": 0.984221980413493,
993
+ "f1": 0.8993288590604027,
994
+ "n": 1838
995
+ },
996
+ "new-password": {
997
+ "p": 0.7047619047619048,
998
+ "r": 0.5196629213483146,
999
+ "f1": 0.5982215036378336,
1000
+ "n": 712
1001
+ },
1002
+ "one-time-code": {
1003
+ "p": 0.6486486486486487,
1004
+ "r": 0.35555555555555557,
1005
+ "f1": 0.45933014354066987,
1006
+ "n": 135
1007
+ },
1008
+ "tel": {
1009
+ "p": 0.8867791842475387,
1010
+ "r": 0.9431563201196709,
1011
+ "f1": 0.91409931134469,
1012
+ "n": 1337
1013
+ },
1014
+ "name": {
1015
+ "p": 0.8157695223654283,
1016
+ "r": 0.7364818617385352,
1017
+ "f1": 0.7741007194244605,
1018
+ "n": 1461
1019
+ },
1020
+ "given-name": {
1021
+ "p": 0.6842105263157895,
1022
+ "r": 0.7114427860696517,
1023
+ "f1": 0.697560975609756,
1024
+ "n": 603
1025
+ },
1026
+ "family-name": {
1027
+ "p": 0.9160671462829736,
1028
+ "r": 0.7549407114624506,
1029
+ "f1": 0.8277356446370531,
1030
+ "n": 506
1031
+ },
1032
+ "organization": {
1033
+ "p": 0.8041666666666667,
1034
+ "r": 0.75390625,
1035
+ "f1": 0.778225806451613,
1036
+ "n": 256
1037
+ },
1038
+ "street-address": {
1039
+ "p": 0.6370370370370371,
1040
+ "r": 0.8269230769230769,
1041
+ "f1": 0.7196652719665272,
1042
+ "n": 208
1043
+ },
1044
+ "address-line2": {
1045
+ "p": 0.453125,
1046
+ "r": 0.6041666666666666,
1047
+ "f1": 0.5178571428571429,
1048
+ "n": 48
1049
+ },
1050
+ "address-level2": {
1051
+ "p": 0.6318537859007833,
1052
+ "r": 0.8287671232876712,
1053
+ "f1": 0.717037037037037,
1054
+ "n": 292
1055
+ },
1056
+ "address-level1": {
1057
+ "p": 0.5258964143426295,
1058
+ "r": 0.7542857142857143,
1059
+ "f1": 0.6197183098591549,
1060
+ "n": 175
1061
+ },
1062
+ "postal-code": {
1063
+ "p": 0.925,
1064
+ "r": 0.8043478260869565,
1065
+ "f1": 0.8604651162790697,
1066
+ "n": 322
1067
+ },
1068
+ "country": {
1069
+ "p": 0.8226950354609929,
1070
+ "r": 0.7341772151898734,
1071
+ "f1": 0.7759197324414716,
1072
+ "n": 158
1073
+ },
1074
+ "cc-name": {
1075
+ "p": 0.5,
1076
+ "r": 1.0,
1077
+ "f1": 0.6666666666666666,
1078
+ "n": 2
1079
+ },
1080
+ "cc-number": {
1081
+ "p": 0.5,
1082
+ "r": 1.0,
1083
+ "f1": 0.6666666666666666,
1084
+ "n": 6
1085
+ },
1086
+ "cc-exp-month": {
1087
+ "p": 1.0,
1088
+ "r": 0.5,
1089
+ "f1": 0.6666666666666666,
1090
+ "n": 6
1091
+ },
1092
+ "cc-exp-year": {
1093
+ "p": 0.6,
1094
+ "r": 0.5,
1095
+ "f1": 0.5454545454545454,
1096
+ "n": 6
1097
+ },
1098
+ "cc-csc": {
1099
+ "p": 0.4166666666666667,
1100
+ "r": 0.8333333333333334,
1101
+ "f1": 0.5555555555555556,
1102
+ "n": 6
1103
+ },
1104
+ "bday": {
1105
+ "p": 0.9318181818181818,
1106
+ "r": 0.7735849056603774,
1107
+ "f1": 0.8453608247422681,
1108
+ "n": 53
1109
+ },
1110
+ "search": {
1111
+ "p": 0.8284714119019837,
1112
+ "r": 0.7489451476793249,
1113
+ "f1": 0.7867036011080333,
1114
+ "n": 948
1115
+ },
1116
+ "captcha": {
1117
+ "p": 0.9297297297297298,
1118
+ "r": 0.5714285714285714,
1119
+ "f1": 0.7078189300411523,
1120
+ "n": 301
1121
+ },
1122
+ "other": {
1123
+ "p": 0.8645796693633486,
1124
+ "r": 0.8255247691015953,
1125
+ "f1": 0.8446009792973114,
1126
+ "n": 5955
1127
+ }
1128
+ },
1129
+ "per_form": {
1130
+ "login": {
1131
+ "p": 0.94069640914037,
1132
+ "r": 0.9417211328976035,
1133
+ "f1": 0.9412084921066958,
1134
+ "n": 1836
1135
+ },
1136
+ "signup": {
1137
+ "p": 0.5643224699828473,
1138
+ "r": 0.7889688249400479,
1139
+ "f1": 0.6579999999999999,
1140
+ "n": 417
1141
+ },
1142
+ "password-change": {
1143
+ "p": 0.0,
1144
+ "r": 0.0,
1145
+ "f1": 0.0,
1146
+ "n": 50
1147
+ },
1148
+ "password-recovery": {
1149
+ "p": 0.26666666666666666,
1150
+ "r": 0.21052631578947367,
1151
+ "f1": 0.23529411764705882,
1152
+ "n": 38
1153
+ },
1154
+ "otp": {
1155
+ "p": 0.3,
1156
+ "r": 0.375,
1157
+ "f1": 0.33333333333333326,
1158
+ "n": 40
1159
+ },
1160
+ "payment": {
1161
+ "p": 0.21052631578947367,
1162
+ "r": 0.5,
1163
+ "f1": 0.2962962962962963,
1164
+ "n": 8
1165
+ },
1166
+ "address": {
1167
+ "p": 0.03722504230118443,
1168
+ "r": 0.6285714285714286,
1169
+ "f1": 0.07028753993610225,
1170
+ "n": 35
1171
+ },
1172
+ "newsletter": {
1173
+ "p": 0.8864774624373957,
1174
+ "r": 0.7961019490254873,
1175
+ "f1": 0.8388625592417062,
1176
+ "n": 667
1177
+ },
1178
+ "contact": {
1179
+ "p": 0.9479371316306483,
1180
+ "r": 0.5884146341463414,
1181
+ "f1": 0.7261098570353649,
1182
+ "n": 1640
1183
+ },
1184
+ "search": {
1185
+ "p": 0.984313725490196,
1186
+ "r": 0.4813039309683605,
1187
+ "f1": 0.6464906632324533,
1188
+ "n": 1043
1189
+ },
1190
+ "other": {
1191
+ "p": 0.493125,
1192
+ "r": 0.7415413533834586,
1193
+ "f1": 0.5923423423423424,
1194
+ "n": 1064
1195
+ }
1196
+ },
1197
+ "top_confusions": [
1198
+ [
1199
+ "new-password",
1200
+ "current-password",
1201
+ 339
1202
+ ],
1203
+ [
1204
+ "username",
1205
+ "email",
1206
+ 309
1207
+ ],
1208
+ [
1209
+ "name",
1210
+ "given-name",
1211
+ 148
1212
+ ],
1213
+ [
1214
+ "name",
1215
+ "other",
1216
+ 135
1217
+ ],
1218
+ [
1219
+ "other",
1220
+ "search",
1221
+ 134
1222
+ ],
1223
+ [
1224
+ "search",
1225
+ "other",
1226
+ 128
1227
+ ],
1228
+ [
1229
+ "other",
1230
+ "email",
1231
+ 118
1232
+ ],
1233
+ [
1234
+ "other",
1235
+ "new-password",
1236
+ 117
1237
+ ],
1238
+ [
1239
+ "other",
1240
+ "address-level2",
1241
+ 108
1242
+ ],
1243
+ [
1244
+ "captcha",
1245
+ "other",
1246
+ 100
1247
+ ],
1248
+ [
1249
+ "given-name",
1250
+ "name",
1251
+ 91
1252
+ ],
1253
+ [
1254
+ "other",
1255
+ "name",
1256
+ 89
1257
+ ],
1258
+ [
1259
+ "other",
1260
+ "username",
1261
+ 84
1262
+ ],
1263
+ [
1264
+ "other",
1265
+ "tel",
1266
+ 84
1267
+ ],
1268
+ [
1269
+ "other",
1270
+ "address-level1",
1271
+ 82
1272
+ ],
1273
+ [
1274
+ "family-name",
1275
+ "other",
1276
+ 63
1277
+ ],
1278
+ [
1279
+ "email",
1280
+ "username",
1281
+ 62
1282
+ ],
1283
+ [
1284
+ "tel",
1285
+ "other",
1286
+ 54
1287
+ ],
1288
+ [
1289
+ "name",
1290
+ "username",
1291
+ 52
1292
+ ],
1293
+ [
1294
+ "username",
1295
+ "other",
1296
+ 49
1297
+ ],
1298
+ [
1299
+ "given-name",
1300
+ "other",
1301
+ 39
1302
+ ],
1303
+ [
1304
+ "other",
1305
+ "street-address",
1306
+ 39
1307
+ ],
1308
+ [
1309
+ "one-time-code",
1310
+ "other",
1311
+ 39
1312
+ ],
1313
+ [
1314
+ "other",
1315
+ "current-password",
1316
+ 32
1317
+ ],
1318
+ [
1319
+ "other",
1320
+ "address-line2",
1321
+ 31
1322
+ ]
1323
+ ]
1324
+ },
1325
+ "autocomplete_masked": {
1326
+ "n_fields": 1622,
1327
+ "acc": 0.9784217016029593,
1328
+ "per_class": {
1329
+ "address-level1": "4/4",
1330
+ "address-level2": "12/14",
1331
+ "address-line2": "0/3",
1332
+ "bday": "5/5",
1333
+ "cc-csc": "1/1",
1334
+ "cc-exp-month": "1/1",
1335
+ "cc-exp-year": "1/1",
1336
+ "cc-number": "1/1",
1337
+ "country": "6/6",
1338
+ "current-password": "323/324",
1339
+ "email": "417/418",
1340
+ "family-name": "34/37",
1341
+ "given-name": "39/42",
1342
+ "name": "182/187",
1343
+ "new-password": "180/180",
1344
+ "one-time-code": "7/7",
1345
+ "organization": "16/16",
1346
+ "postal-code": "26/30",
1347
+ "street-address": "11/13",
1348
+ "tel": "111/111",
1349
+ "username": "210/221"
1350
+ },
1351
+ "heuristic_acc": 0.7866831072749692
1352
+ }
1353
+ }
eval/eval_val_bfloat16.json ADDED
@@ -0,0 +1,1338 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "split": "val",
3
+ "dtype": "bfloat16",
4
+ "forms_per_s_gpu": 2872.840982485025,
5
+ "model": {
6
+ "n_forms": 3408,
7
+ "n_fields": 9819,
8
+ "field_acc": 0.9747428455036155,
9
+ "field_macro_f1": 0.9662059517142381,
10
+ "form_acc": 0.9568661971830986,
11
+ "form_macro_f1": 0.9375185465621217,
12
+ "password_precision": 0.9898358092259578,
13
+ "password_recall": 0.998422712933754,
14
+ "password_f1": 0.994110718492344,
15
+ "cur_vs_new_acc": 0.9944794952681388,
16
+ "identifier_f1": 0.9945380063723259,
17
+ "per_field": {
18
+ "username": {
19
+ "p": 0.9921752738654147,
20
+ "r": 0.9768875192604006,
21
+ "f1": 0.984472049689441,
22
+ "n": 649
23
+ },
24
+ "email": {
25
+ "p": 0.9884541372674791,
26
+ "r": 0.9961215255332903,
27
+ "f1": 0.9922730199613651,
28
+ "n": 1547
29
+ },
30
+ "current-password": {
31
+ "p": 0.9814814814814815,
32
+ "r": 0.9977851605758582,
33
+ "f1": 0.9895661724327293,
34
+ "n": 903
35
+ },
36
+ "new-password": {
37
+ "p": 0.997229916897507,
38
+ "r": 0.9863013698630136,
39
+ "f1": 0.9917355371900827,
40
+ "n": 365
41
+ },
42
+ "one-time-code": {
43
+ "p": 0.9901960784313726,
44
+ "r": 0.9805825242718447,
45
+ "f1": 0.9853658536585367,
46
+ "n": 103
47
+ },
48
+ "tel": {
49
+ "p": 0.9815950920245399,
50
+ "r": 0.9937888198757764,
51
+ "f1": 0.9876543209876544,
52
+ "n": 644
53
+ },
54
+ "name": {
55
+ "p": 0.9592123769338959,
56
+ "r": 0.9855491329479769,
57
+ "f1": 0.9722024233784746,
58
+ "n": 692
59
+ },
60
+ "given-name": {
61
+ "p": 0.978494623655914,
62
+ "r": 0.9161073825503355,
63
+ "f1": 0.9462738301559791,
64
+ "n": 298
65
+ },
66
+ "family-name": {
67
+ "p": 0.9961089494163424,
68
+ "r": 0.9808429118773946,
69
+ "f1": 0.9884169884169886,
70
+ "n": 261
71
+ },
72
+ "organization": {
73
+ "p": 0.9761904761904762,
74
+ "r": 0.9318181818181818,
75
+ "f1": 0.9534883720930233,
76
+ "n": 132
77
+ },
78
+ "street-address": {
79
+ "p": 0.9369369369369369,
80
+ "r": 0.9122807017543859,
81
+ "f1": 0.9244444444444444,
82
+ "n": 114
83
+ },
84
+ "address-line2": {
85
+ "p": 1.0,
86
+ "r": 0.9090909090909091,
87
+ "f1": 0.9523809523809523,
88
+ "n": 22
89
+ },
90
+ "address-level2": {
91
+ "p": 0.8843537414965986,
92
+ "r": 0.8904109589041096,
93
+ "f1": 0.887372013651877,
94
+ "n": 146
95
+ },
96
+ "address-level1": {
97
+ "p": 0.898989898989899,
98
+ "r": 0.9368421052631579,
99
+ "f1": 0.9175257731958762,
100
+ "n": 95
101
+ },
102
+ "postal-code": {
103
+ "p": 0.9724137931034482,
104
+ "r": 0.9724137931034482,
105
+ "f1": 0.9724137931034482,
106
+ "n": 145
107
+ },
108
+ "country": {
109
+ "p": 0.8863636363636364,
110
+ "r": 0.9873417721518988,
111
+ "f1": 0.9341317365269461,
112
+ "n": 79
113
+ },
114
+ "cc-name": {
115
+ "p": 1.0,
116
+ "r": 0.95,
117
+ "f1": 0.9743589743589743,
118
+ "n": 20
119
+ },
120
+ "cc-number": {
121
+ "p": 0.9565217391304348,
122
+ "r": 1.0,
123
+ "f1": 0.9777777777777777,
124
+ "n": 22
125
+ },
126
+ "cc-exp": {
127
+ "p": 1.0,
128
+ "r": 1.0,
129
+ "f1": 1.0,
130
+ "n": 13
131
+ },
132
+ "cc-exp-month": {
133
+ "p": 1.0,
134
+ "r": 1.0,
135
+ "f1": 1.0,
136
+ "n": 9
137
+ },
138
+ "cc-exp-year": {
139
+ "p": 1.0,
140
+ "r": 1.0,
141
+ "f1": 1.0,
142
+ "n": 8
143
+ },
144
+ "cc-csc": {
145
+ "p": 0.9523809523809523,
146
+ "r": 1.0,
147
+ "f1": 0.975609756097561,
148
+ "n": 20
149
+ },
150
+ "bday": {
151
+ "p": 1.0,
152
+ "r": 0.8461538461538461,
153
+ "f1": 0.9166666666666666,
154
+ "n": 39
155
+ },
156
+ "search": {
157
+ "p": 0.9624217118997912,
158
+ "r": 0.9808510638297873,
159
+ "f1": 0.9715489989462592,
160
+ "n": 470
161
+ },
162
+ "captcha": {
163
+ "p": 0.9814814814814815,
164
+ "r": 0.9352941176470588,
165
+ "f1": 0.9578313253012049,
166
+ "n": 170
167
+ },
168
+ "other": {
169
+ "p": 0.9704016913319239,
170
+ "r": 0.9652996845425867,
171
+ "f1": 0.9678439641539274,
172
+ "n": 2853
173
+ }
174
+ },
175
+ "per_form": {
176
+ "login": {
177
+ "p": 0.9900552486187846,
178
+ "r": 0.9933481152993349,
179
+ "f1": 0.991698948533481,
180
+ "n": 902
181
+ },
182
+ "signup": {
183
+ "p": 0.9951456310679612,
184
+ "r": 0.9855769230769231,
185
+ "f1": 0.9903381642512078,
186
+ "n": 208
187
+ },
188
+ "password-change": {
189
+ "p": 1.0,
190
+ "r": 1.0,
191
+ "f1": 1.0,
192
+ "n": 22
193
+ },
194
+ "password-recovery": {
195
+ "p": 0.8461538461538461,
196
+ "r": 0.9565217391304348,
197
+ "f1": 0.8979591836734695,
198
+ "n": 23
199
+ },
200
+ "otp": {
201
+ "p": 0.8888888888888888,
202
+ "r": 0.8571428571428571,
203
+ "f1": 0.8727272727272727,
204
+ "n": 28
205
+ },
206
+ "payment": {
207
+ "p": 1.0,
208
+ "r": 1.0,
209
+ "f1": 1.0,
210
+ "n": 22
211
+ },
212
+ "address": {
213
+ "p": 0.8666666666666667,
214
+ "r": 0.7222222222222222,
215
+ "f1": 0.7878787878787877,
216
+ "n": 36
217
+ },
218
+ "newsletter": {
219
+ "p": 0.9774193548387097,
220
+ "r": 0.9439252336448598,
221
+ "f1": 0.9603803486529319,
222
+ "n": 321
223
+ },
224
+ "contact": {
225
+ "p": 0.9369482976040353,
226
+ "r": 0.9750656167979003,
227
+ "f1": 0.9556270096463023,
228
+ "n": 762
229
+ },
230
+ "search": {
231
+ "p": 0.9554730983302412,
232
+ "r": 0.9662288930581614,
233
+ "f1": 0.9608208955223881,
234
+ "n": 533
235
+ },
236
+ "other": {
237
+ "p": 0.9147727272727273,
238
+ "r": 0.8765880217785844,
239
+ "f1": 0.8952734012974978,
240
+ "n": 551
241
+ }
242
+ },
243
+ "top_confusions": [
244
+ [
245
+ "given-name",
246
+ "name",
247
+ 22
248
+ ],
249
+ [
250
+ "other",
251
+ "address-level2",
252
+ 16
253
+ ],
254
+ [
255
+ "other",
256
+ "search",
257
+ 16
258
+ ],
259
+ [
260
+ "address-level2",
261
+ "other",
262
+ 15
263
+ ],
264
+ [
265
+ "other",
266
+ "current-password",
267
+ 11
268
+ ],
269
+ [
270
+ "other",
271
+ "address-level1",
272
+ 10
273
+ ],
274
+ [
275
+ "captcha",
276
+ "other",
277
+ 9
278
+ ],
279
+ [
280
+ "other",
281
+ "country",
282
+ 9
283
+ ],
284
+ [
285
+ "search",
286
+ "other",
287
+ 9
288
+ ],
289
+ [
290
+ "other",
291
+ "email",
292
+ 9
293
+ ],
294
+ [
295
+ "username",
296
+ "email",
297
+ 8
298
+ ],
299
+ [
300
+ "street-address",
301
+ "other",
302
+ 7
303
+ ],
304
+ [
305
+ "organization",
306
+ "other",
307
+ 7
308
+ ],
309
+ [
310
+ "address-level1",
311
+ "other",
312
+ 6
313
+ ],
314
+ [
315
+ "name",
316
+ "given-name",
317
+ 5
318
+ ],
319
+ [
320
+ "username",
321
+ "other",
322
+ 5
323
+ ],
324
+ [
325
+ "other",
326
+ "tel",
327
+ 5
328
+ ],
329
+ [
330
+ "other",
331
+ "street-address",
332
+ 5
333
+ ],
334
+ [
335
+ "bday",
336
+ "other",
337
+ 5
338
+ ],
339
+ [
340
+ "new-password",
341
+ "current-password",
342
+ 4
343
+ ],
344
+ [
345
+ "name",
346
+ "other",
347
+ 4
348
+ ],
349
+ [
350
+ "other",
351
+ "postal-code",
352
+ 4
353
+ ],
354
+ [
355
+ "postal-code",
356
+ "other",
357
+ 4
358
+ ],
359
+ [
360
+ "other",
361
+ "name",
362
+ 3
363
+ ],
364
+ [
365
+ "family-name",
366
+ "name",
367
+ 3
368
+ ]
369
+ ]
370
+ },
371
+ "model_real": {
372
+ "n_forms": 3362,
373
+ "n_fields": 9619,
374
+ "field_acc": 0.9747374987004886,
375
+ "field_macro_f1": 0.9605143456394969,
376
+ "form_acc": 0.9562760261748959,
377
+ "form_macro_f1": 0.9290284576847291,
378
+ "password_precision": 0.9912490055688147,
379
+ "password_recall": 0.9983974358974359,
380
+ "password_f1": 0.9948103792415169,
381
+ "cur_vs_new_acc": 0.9951923076923077,
382
+ "identifier_f1": 0.9945054945054944,
383
+ "per_field": {
384
+ "username": {
385
+ "p": 0.9921135646687698,
386
+ "r": 0.9767080745341615,
387
+ "f1": 0.9843505477308295,
388
+ "n": 644
389
+ },
390
+ "email": {
391
+ "p": 0.988394584139265,
392
+ "r": 0.9961013645224172,
393
+ "f1": 0.992233009708738,
394
+ "n": 1539
395
+ },
396
+ "current-password": {
397
+ "p": 0.9835706462212487,
398
+ "r": 0.9988876529477196,
399
+ "f1": 0.9911699779249449,
400
+ "n": 899
401
+ },
402
+ "new-password": {
403
+ "p": 1.0,
404
+ "r": 0.9856733524355301,
405
+ "f1": 0.9927849927849928,
406
+ "n": 349
407
+ },
408
+ "one-time-code": {
409
+ "p": 0.9868421052631579,
410
+ "r": 0.974025974025974,
411
+ "f1": 0.9803921568627451,
412
+ "n": 77
413
+ },
414
+ "tel": {
415
+ "p": 0.9813953488372092,
416
+ "r": 0.9937205651491365,
417
+ "f1": 0.987519500780031,
418
+ "n": 637
419
+ },
420
+ "name": {
421
+ "p": 0.9588652482269504,
422
+ "r": 0.9854227405247813,
423
+ "f1": 0.9719626168224299,
424
+ "n": 686
425
+ },
426
+ "given-name": {
427
+ "p": 0.9783393501805054,
428
+ "r": 0.9155405405405406,
429
+ "f1": 0.9458987783595114,
430
+ "n": 296
431
+ },
432
+ "family-name": {
433
+ "p": 0.996078431372549,
434
+ "r": 0.9806949806949807,
435
+ "f1": 0.9883268482490273,
436
+ "n": 259
437
+ },
438
+ "organization": {
439
+ "p": 0.9758064516129032,
440
+ "r": 0.9307692307692308,
441
+ "f1": 0.952755905511811,
442
+ "n": 130
443
+ },
444
+ "street-address": {
445
+ "p": 0.9326923076923077,
446
+ "r": 0.9065420560747663,
447
+ "f1": 0.919431279620853,
448
+ "n": 107
449
+ },
450
+ "address-line2": {
451
+ "p": 1.0,
452
+ "r": 0.8823529411764706,
453
+ "f1": 0.9375,
454
+ "n": 17
455
+ },
456
+ "address-level2": {
457
+ "p": 0.8794326241134752,
458
+ "r": 0.8857142857142857,
459
+ "f1": 0.8825622775800712,
460
+ "n": 140
461
+ },
462
+ "address-level1": {
463
+ "p": 0.8947368421052632,
464
+ "r": 0.9340659340659341,
465
+ "f1": 0.913978494623656,
466
+ "n": 91
467
+ },
468
+ "postal-code": {
469
+ "p": 0.9708029197080292,
470
+ "r": 0.9708029197080292,
471
+ "f1": 0.9708029197080292,
472
+ "n": 137
473
+ },
474
+ "country": {
475
+ "p": 0.8809523809523809,
476
+ "r": 0.9866666666666667,
477
+ "f1": 0.930817610062893,
478
+ "n": 75
479
+ },
480
+ "cc-name": {
481
+ "p": 1.0,
482
+ "r": 1.0,
483
+ "f1": 1.0,
484
+ "n": 3
485
+ },
486
+ "cc-number": {
487
+ "p": 0.75,
488
+ "r": 1.0,
489
+ "f1": 0.8571428571428571,
490
+ "n": 3
491
+ },
492
+ "cc-exp-month": {
493
+ "p": 1.0,
494
+ "r": 1.0,
495
+ "f1": 1.0,
496
+ "n": 3
497
+ },
498
+ "cc-exp-year": {
499
+ "p": 1.0,
500
+ "r": 1.0,
501
+ "f1": 1.0,
502
+ "n": 3
503
+ },
504
+ "cc-csc": {
505
+ "p": 1.0,
506
+ "r": 1.0,
507
+ "f1": 1.0,
508
+ "n": 3
509
+ },
510
+ "bday": {
511
+ "p": 1.0,
512
+ "r": 0.8461538461538461,
513
+ "f1": 0.9166666666666666,
514
+ "n": 39
515
+ },
516
+ "search": {
517
+ "p": 0.9624217118997912,
518
+ "r": 0.9808510638297873,
519
+ "f1": 0.9715489989462592,
520
+ "n": 470
521
+ },
522
+ "captcha": {
523
+ "p": 0.9808917197452229,
524
+ "r": 0.9333333333333333,
525
+ "f1": 0.9565217391304348,
526
+ "n": 165
527
+ },
528
+ "other": {
529
+ "p": 0.970712773465067,
530
+ "r": 0.9662802950474183,
531
+ "f1": 0.968491462770639,
532
+ "n": 2847
533
+ }
534
+ },
535
+ "per_form": {
536
+ "login": {
537
+ "p": 0.9900110987791343,
538
+ "r": 0.9933184855233853,
539
+ "f1": 0.9916620344635908,
540
+ "n": 898
541
+ },
542
+ "signup": {
543
+ "p": 0.9951219512195122,
544
+ "r": 0.9855072463768116,
545
+ "f1": 0.9902912621359223,
546
+ "n": 207
547
+ },
548
+ "password-change": {
549
+ "p": 1.0,
550
+ "r": 1.0,
551
+ "f1": 1.0,
552
+ "n": 15
553
+ },
554
+ "password-recovery": {
555
+ "p": 0.8181818181818182,
556
+ "r": 0.9473684210526315,
557
+ "f1": 0.8780487804878049,
558
+ "n": 19
559
+ },
560
+ "otp": {
561
+ "p": 0.8571428571428571,
562
+ "r": 0.8181818181818182,
563
+ "f1": 0.8372093023255814,
564
+ "n": 22
565
+ },
566
+ "payment": {
567
+ "p": 1.0,
568
+ "r": 1.0,
569
+ "f1": 1.0,
570
+ "n": 3
571
+ },
572
+ "address": {
573
+ "p": 0.84,
574
+ "r": 0.6774193548387096,
575
+ "f1": 0.75,
576
+ "n": 31
577
+ },
578
+ "newsletter": {
579
+ "p": 0.9774193548387097,
580
+ "r": 0.9439252336448598,
581
+ "f1": 0.9603803486529319,
582
+ "n": 321
583
+ },
584
+ "contact": {
585
+ "p": 0.9369482976040353,
586
+ "r": 0.9750656167979003,
587
+ "f1": 0.9556270096463023,
588
+ "n": 762
589
+ },
590
+ "search": {
591
+ "p": 0.9554730983302412,
592
+ "r": 0.9662288930581614,
593
+ "f1": 0.9608208955223881,
594
+ "n": 533
595
+ },
596
+ "other": {
597
+ "p": 0.9147727272727273,
598
+ "r": 0.8765880217785844,
599
+ "f1": 0.8952734012974978,
600
+ "n": 551
601
+ }
602
+ },
603
+ "top_confusions": [
604
+ [
605
+ "given-name",
606
+ "name",
607
+ 22
608
+ ],
609
+ [
610
+ "other",
611
+ "address-level2",
612
+ 16
613
+ ],
614
+ [
615
+ "other",
616
+ "search",
617
+ 16
618
+ ],
619
+ [
620
+ "address-level2",
621
+ "other",
622
+ 15
623
+ ],
624
+ [
625
+ "other",
626
+ "address-level1",
627
+ 10
628
+ ],
629
+ [
630
+ "captcha",
631
+ "other",
632
+ 9
633
+ ],
634
+ [
635
+ "other",
636
+ "country",
637
+ 9
638
+ ],
639
+ [
640
+ "search",
641
+ "other",
642
+ 9
643
+ ],
644
+ [
645
+ "other",
646
+ "email",
647
+ 9
648
+ ],
649
+ [
650
+ "other",
651
+ "current-password",
652
+ 9
653
+ ],
654
+ [
655
+ "username",
656
+ "email",
657
+ 8
658
+ ],
659
+ [
660
+ "street-address",
661
+ "other",
662
+ 7
663
+ ],
664
+ [
665
+ "organization",
666
+ "other",
667
+ 7
668
+ ],
669
+ [
670
+ "address-level1",
671
+ "other",
672
+ 6
673
+ ],
674
+ [
675
+ "name",
676
+ "given-name",
677
+ 5
678
+ ],
679
+ [
680
+ "username",
681
+ "other",
682
+ 5
683
+ ],
684
+ [
685
+ "other",
686
+ "tel",
687
+ 5
688
+ ],
689
+ [
690
+ "other",
691
+ "street-address",
692
+ 5
693
+ ],
694
+ [
695
+ "bday",
696
+ "other",
697
+ 5
698
+ ],
699
+ [
700
+ "new-password",
701
+ "current-password",
702
+ 4
703
+ ],
704
+ [
705
+ "name",
706
+ "other",
707
+ 4
708
+ ],
709
+ [
710
+ "other",
711
+ "postal-code",
712
+ 4
713
+ ],
714
+ [
715
+ "postal-code",
716
+ "other",
717
+ 4
718
+ ],
719
+ [
720
+ "other",
721
+ "name",
722
+ 3
723
+ ],
724
+ [
725
+ "family-name",
726
+ "name",
727
+ 3
728
+ ]
729
+ ]
730
+ },
731
+ "model_synthetic": {
732
+ "n_forms": 46,
733
+ "n_fields": 200,
734
+ "field_acc": 0.975,
735
+ "field_macro_f1": 0.9657287157287158,
736
+ "form_acc": 1.0,
737
+ "form_macro_f1": 1.0,
738
+ "password_precision": 0.9090909090909091,
739
+ "password_recall": 1.0,
740
+ "password_f1": 0.9523809523809523,
741
+ "cur_vs_new_acc": 0.95,
742
+ "identifier_f1": 1.0,
743
+ "per_field": {
744
+ "username": {
745
+ "p": 1.0,
746
+ "r": 1.0,
747
+ "f1": 1.0,
748
+ "n": 5
749
+ },
750
+ "email": {
751
+ "p": 1.0,
752
+ "r": 1.0,
753
+ "f1": 1.0,
754
+ "n": 8
755
+ },
756
+ "current-password": {
757
+ "p": 0.6,
758
+ "r": 0.75,
759
+ "f1": 0.6666666666666665,
760
+ "n": 4
761
+ },
762
+ "new-password": {
763
+ "p": 0.9411764705882353,
764
+ "r": 1.0,
765
+ "f1": 0.9696969696969697,
766
+ "n": 16
767
+ },
768
+ "one-time-code": {
769
+ "p": 1.0,
770
+ "r": 1.0,
771
+ "f1": 1.0,
772
+ "n": 26
773
+ },
774
+ "tel": {
775
+ "p": 1.0,
776
+ "r": 1.0,
777
+ "f1": 1.0,
778
+ "n": 7
779
+ },
780
+ "name": {
781
+ "p": 1.0,
782
+ "r": 1.0,
783
+ "f1": 1.0,
784
+ "n": 6
785
+ },
786
+ "given-name": {
787
+ "p": 1.0,
788
+ "r": 1.0,
789
+ "f1": 1.0,
790
+ "n": 2
791
+ },
792
+ "family-name": {
793
+ "p": 1.0,
794
+ "r": 1.0,
795
+ "f1": 1.0,
796
+ "n": 2
797
+ },
798
+ "organization": {
799
+ "p": 1.0,
800
+ "r": 1.0,
801
+ "f1": 1.0,
802
+ "n": 2
803
+ },
804
+ "street-address": {
805
+ "p": 1.0,
806
+ "r": 1.0,
807
+ "f1": 1.0,
808
+ "n": 7
809
+ },
810
+ "address-line2": {
811
+ "p": 1.0,
812
+ "r": 1.0,
813
+ "f1": 1.0,
814
+ "n": 5
815
+ },
816
+ "address-level2": {
817
+ "p": 1.0,
818
+ "r": 1.0,
819
+ "f1": 1.0,
820
+ "n": 6
821
+ },
822
+ "address-level1": {
823
+ "p": 1.0,
824
+ "r": 1.0,
825
+ "f1": 1.0,
826
+ "n": 4
827
+ },
828
+ "postal-code": {
829
+ "p": 1.0,
830
+ "r": 1.0,
831
+ "f1": 1.0,
832
+ "n": 8
833
+ },
834
+ "country": {
835
+ "p": 1.0,
836
+ "r": 1.0,
837
+ "f1": 1.0,
838
+ "n": 4
839
+ },
840
+ "cc-name": {
841
+ "p": 1.0,
842
+ "r": 0.9411764705882353,
843
+ "f1": 0.9696969696969697,
844
+ "n": 17
845
+ },
846
+ "cc-number": {
847
+ "p": 1.0,
848
+ "r": 1.0,
849
+ "f1": 1.0,
850
+ "n": 19
851
+ },
852
+ "cc-exp": {
853
+ "p": 1.0,
854
+ "r": 1.0,
855
+ "f1": 1.0,
856
+ "n": 13
857
+ },
858
+ "cc-exp-month": {
859
+ "p": 1.0,
860
+ "r": 1.0,
861
+ "f1": 1.0,
862
+ "n": 6
863
+ },
864
+ "cc-exp-year": {
865
+ "p": 1.0,
866
+ "r": 1.0,
867
+ "f1": 1.0,
868
+ "n": 5
869
+ },
870
+ "cc-csc": {
871
+ "p": 0.9444444444444444,
872
+ "r": 1.0,
873
+ "f1": 0.9714285714285714,
874
+ "n": 17
875
+ },
876
+ "captcha": {
877
+ "p": 1.0,
878
+ "r": 1.0,
879
+ "f1": 1.0,
880
+ "n": 5
881
+ },
882
+ "other": {
883
+ "p": 0.75,
884
+ "r": 0.5,
885
+ "f1": 0.6,
886
+ "n": 6
887
+ }
888
+ },
889
+ "per_form": {
890
+ "login": {
891
+ "p": 1.0,
892
+ "r": 1.0,
893
+ "f1": 1.0,
894
+ "n": 4
895
+ },
896
+ "signup": {
897
+ "p": 1.0,
898
+ "r": 1.0,
899
+ "f1": 1.0,
900
+ "n": 1
901
+ },
902
+ "password-change": {
903
+ "p": 1.0,
904
+ "r": 1.0,
905
+ "f1": 1.0,
906
+ "n": 7
907
+ },
908
+ "password-recovery": {
909
+ "p": 1.0,
910
+ "r": 1.0,
911
+ "f1": 1.0,
912
+ "n": 4
913
+ },
914
+ "otp": {
915
+ "p": 1.0,
916
+ "r": 1.0,
917
+ "f1": 1.0,
918
+ "n": 6
919
+ },
920
+ "payment": {
921
+ "p": 1.0,
922
+ "r": 1.0,
923
+ "f1": 1.0,
924
+ "n": 19
925
+ },
926
+ "address": {
927
+ "p": 1.0,
928
+ "r": 1.0,
929
+ "f1": 1.0,
930
+ "n": 5
931
+ }
932
+ },
933
+ "top_confusions": [
934
+ [
935
+ "other",
936
+ "current-password",
937
+ 2
938
+ ],
939
+ [
940
+ "cc-name",
941
+ "other",
942
+ 1
943
+ ],
944
+ [
945
+ "current-password",
946
+ "new-password",
947
+ 1
948
+ ],
949
+ [
950
+ "other",
951
+ "cc-csc",
952
+ 1
953
+ ]
954
+ ]
955
+ },
956
+ "heuristic_baseline_real": {
957
+ "n_forms": 3362,
958
+ "n_fields": 9619,
959
+ "field_acc": 0.819523859029005,
960
+ "field_macro_f1": 0.7496204195913461,
961
+ "form_acc": 0.7245687091017252,
962
+ "form_macro_f1": 0.5030916170134985,
963
+ "password_precision": 0.9431818181818182,
964
+ "password_recall": 0.9975961538461539,
965
+ "password_f1": 0.9696261682242991,
966
+ "cur_vs_new_acc": 0.8333333333333334,
967
+ "identifier_f1": 0.9389296024872307,
968
+ "per_field": {
969
+ "username": {
970
+ "p": 0.7634782608695653,
971
+ "r": 0.6816770186335404,
972
+ "f1": 0.7202625102543068,
973
+ "n": 644
974
+ },
975
+ "email": {
976
+ "p": 0.8578796561604585,
977
+ "r": 0.9727095516569201,
978
+ "f1": 0.9116930572472595,
979
+ "n": 1539
980
+ },
981
+ "current-password": {
982
+ "p": 0.8096980786825252,
983
+ "r": 0.9844271412680756,
984
+ "f1": 0.8885542168674699,
985
+ "n": 899
986
+ },
987
+ "new-password": {
988
+ "p": 0.6828193832599119,
989
+ "r": 0.44412607449856734,
990
+ "f1": 0.5381944444444444,
991
+ "n": 349
992
+ },
993
+ "one-time-code": {
994
+ "p": 0.7538461538461538,
995
+ "r": 0.6363636363636364,
996
+ "f1": 0.6901408450704226,
997
+ "n": 77
998
+ },
999
+ "tel": {
1000
+ "p": 0.8990963855421686,
1001
+ "r": 0.9372056514913658,
1002
+ "f1": 0.9177555726364336,
1003
+ "n": 637
1004
+ },
1005
+ "name": {
1006
+ "p": 0.7942583732057417,
1007
+ "r": 0.7259475218658892,
1008
+ "f1": 0.7585681645087586,
1009
+ "n": 686
1010
+ },
1011
+ "given-name": {
1012
+ "p": 0.6656050955414012,
1013
+ "r": 0.706081081081081,
1014
+ "f1": 0.6852459016393442,
1015
+ "n": 296
1016
+ },
1017
+ "family-name": {
1018
+ "p": 0.9074074074074074,
1019
+ "r": 0.7567567567567568,
1020
+ "f1": 0.8252631578947369,
1021
+ "n": 259
1022
+ },
1023
+ "organization": {
1024
+ "p": 0.744,
1025
+ "r": 0.7153846153846154,
1026
+ "f1": 0.7294117647058823,
1027
+ "n": 130
1028
+ },
1029
+ "street-address": {
1030
+ "p": 0.7105263157894737,
1031
+ "r": 0.7570093457943925,
1032
+ "f1": 0.7330316742081447,
1033
+ "n": 107
1034
+ },
1035
+ "address-line2": {
1036
+ "p": 0.4782608695652174,
1037
+ "r": 0.6470588235294118,
1038
+ "f1": 0.55,
1039
+ "n": 17
1040
+ },
1041
+ "address-level2": {
1042
+ "p": 0.5699481865284974,
1043
+ "r": 0.7857142857142857,
1044
+ "f1": 0.6606606606606606,
1045
+ "n": 140
1046
+ },
1047
+ "address-level1": {
1048
+ "p": 0.5511811023622047,
1049
+ "r": 0.7692307692307693,
1050
+ "f1": 0.6422018348623854,
1051
+ "n": 91
1052
+ },
1053
+ "postal-code": {
1054
+ "p": 0.957983193277311,
1055
+ "r": 0.8321167883211679,
1056
+ "f1": 0.890625,
1057
+ "n": 137
1058
+ },
1059
+ "country": {
1060
+ "p": 0.8028169014084507,
1061
+ "r": 0.76,
1062
+ "f1": 0.7808219178082192,
1063
+ "n": 75
1064
+ },
1065
+ "cc-name": {
1066
+ "p": 1.0,
1067
+ "r": 1.0,
1068
+ "f1": 1.0,
1069
+ "n": 3
1070
+ },
1071
+ "cc-number": {
1072
+ "p": 0.6,
1073
+ "r": 1.0,
1074
+ "f1": 0.7499999999999999,
1075
+ "n": 3
1076
+ },
1077
+ "cc-exp-month": {
1078
+ "p": 1.0,
1079
+ "r": 1.0,
1080
+ "f1": 1.0,
1081
+ "n": 3
1082
+ },
1083
+ "cc-exp-year": {
1084
+ "p": 1.0,
1085
+ "r": 1.0,
1086
+ "f1": 1.0,
1087
+ "n": 3
1088
+ },
1089
+ "cc-csc": {
1090
+ "p": 0.0,
1091
+ "r": 0.0,
1092
+ "f1": 0.0,
1093
+ "n": 3
1094
+ },
1095
+ "bday": {
1096
+ "p": 0.8125,
1097
+ "r": 0.6666666666666666,
1098
+ "f1": 0.7323943661971831,
1099
+ "n": 39
1100
+ },
1101
+ "search": {
1102
+ "p": 0.8271889400921659,
1103
+ "r": 0.7638297872340426,
1104
+ "f1": 0.7942477876106194,
1105
+ "n": 470
1106
+ },
1107
+ "captcha": {
1108
+ "p": 0.9393939393939394,
1109
+ "r": 0.5636363636363636,
1110
+ "f1": 0.7045454545454546,
1111
+ "n": 165
1112
+ },
1113
+ "other": {
1114
+ "p": 0.855465884079237,
1115
+ "r": 0.8191078328064629,
1116
+ "f1": 0.8368921586219272,
1117
+ "n": 2847
1118
+ }
1119
+ },
1120
+ "per_form": {
1121
+ "login": {
1122
+ "p": 0.9395973154362416,
1123
+ "r": 0.9354120267260579,
1124
+ "f1": 0.9375,
1125
+ "n": 898
1126
+ },
1127
+ "signup": {
1128
+ "p": 0.5955882352941176,
1129
+ "r": 0.782608695652174,
1130
+ "f1": 0.6764091858037579,
1131
+ "n": 207
1132
+ },
1133
+ "password-change": {
1134
+ "p": 0.0,
1135
+ "r": 0.0,
1136
+ "f1": 0.0,
1137
+ "n": 15
1138
+ },
1139
+ "password-recovery": {
1140
+ "p": 0.13333333333333333,
1141
+ "r": 0.10526315789473684,
1142
+ "f1": 0.11764705882352941,
1143
+ "n": 19
1144
+ },
1145
+ "otp": {
1146
+ "p": 0.4482758620689655,
1147
+ "r": 0.5909090909090909,
1148
+ "f1": 0.5098039215686274,
1149
+ "n": 22
1150
+ },
1151
+ "payment": {
1152
+ "p": 0.1875,
1153
+ "r": 1.0,
1154
+ "f1": 0.3157894736842105,
1155
+ "n": 3
1156
+ },
1157
+ "address": {
1158
+ "p": 0.06713780918727916,
1159
+ "r": 0.6129032258064516,
1160
+ "f1": 0.12101910828025478,
1161
+ "n": 31
1162
+ },
1163
+ "newsletter": {
1164
+ "p": 0.8866666666666667,
1165
+ "r": 0.8286604361370716,
1166
+ "f1": 0.856682769726248,
1167
+ "n": 321
1168
+ },
1169
+ "contact": {
1170
+ "p": 0.9390756302521008,
1171
+ "r": 0.5866141732283464,
1172
+ "f1": 0.7221324717285944,
1173
+ "n": 762
1174
+ },
1175
+ "search": {
1176
+ "p": 0.9743589743589743,
1177
+ "r": 0.49906191369606,
1178
+ "f1": 0.6600496277915633,
1179
+ "n": 533
1180
+ },
1181
+ "other": {
1182
+ "p": 0.5199004975124378,
1183
+ "r": 0.7586206896551724,
1184
+ "f1": 0.6169741697416974,
1185
+ "n": 551
1186
+ }
1187
+ },
1188
+ "top_confusions": [
1189
+ [
1190
+ "new-password",
1191
+ "current-password",
1192
+ 192
1193
+ ],
1194
+ [
1195
+ "username",
1196
+ "email",
1197
+ 155
1198
+ ],
1199
+ [
1200
+ "name",
1201
+ "given-name",
1202
+ 78
1203
+ ],
1204
+ [
1205
+ "other",
1206
+ "email",
1207
+ 71
1208
+ ],
1209
+ [
1210
+ "other",
1211
+ "search",
1212
+ 71
1213
+ ],
1214
+ [
1215
+ "search",
1216
+ "other",
1217
+ 69
1218
+ ],
1219
+ [
1220
+ "other",
1221
+ "address-level2",
1222
+ 62
1223
+ ],
1224
+ [
1225
+ "name",
1226
+ "other",
1227
+ 62
1228
+ ],
1229
+ [
1230
+ "given-name",
1231
+ "name",
1232
+ 51
1233
+ ],
1234
+ [
1235
+ "captcha",
1236
+ "other",
1237
+ 50
1238
+ ],
1239
+ [
1240
+ "other",
1241
+ "username",
1242
+ 48
1243
+ ],
1244
+ [
1245
+ "other",
1246
+ "address-level1",
1247
+ 47
1248
+ ],
1249
+ [
1250
+ "other",
1251
+ "new-password",
1252
+ 45
1253
+ ],
1254
+ [
1255
+ "other",
1256
+ "name",
1257
+ 28
1258
+ ],
1259
+ [
1260
+ "other",
1261
+ "tel",
1262
+ 28
1263
+ ],
1264
+ [
1265
+ "tel",
1266
+ "other",
1267
+ 28
1268
+ ],
1269
+ [
1270
+ "name",
1271
+ "username",
1272
+ 25
1273
+ ],
1274
+ [
1275
+ "one-time-code",
1276
+ "other",
1277
+ 24
1278
+ ],
1279
+ [
1280
+ "email",
1281
+ "username",
1282
+ 23
1283
+ ],
1284
+ [
1285
+ "other",
1286
+ "organization",
1287
+ 23
1288
+ ],
1289
+ [
1290
+ "username",
1291
+ "other",
1292
+ 23
1293
+ ],
1294
+ [
1295
+ "family-name",
1296
+ "other",
1297
+ 22
1298
+ ],
1299
+ [
1300
+ "family-name",
1301
+ "name",
1302
+ 19
1303
+ ],
1304
+ [
1305
+ "organization",
1306
+ "other",
1307
+ 17
1308
+ ],
1309
+ [
1310
+ "street-address",
1311
+ "other",
1312
+ 16
1313
+ ]
1314
+ ]
1315
+ },
1316
+ "autocomplete_masked": {
1317
+ "n_fields": 641,
1318
+ "acc": 0.9859594383775351,
1319
+ "per_class": {
1320
+ "address-level2": "3/4",
1321
+ "address-line2": "2/2",
1322
+ "country": "1/1",
1323
+ "current-password": "157/157",
1324
+ "email": "161/161",
1325
+ "family-name": "11/11",
1326
+ "given-name": "14/15",
1327
+ "name": "74/75",
1328
+ "new-password": "55/58",
1329
+ "one-time-code": "13/13",
1330
+ "organization": "5/5",
1331
+ "postal-code": "6/6",
1332
+ "street-address": "4/4",
1333
+ "tel": "26/26",
1334
+ "username": "100/103"
1335
+ },
1336
+ "heuristic_acc": 0.7550702028081123
1337
+ }
1338
+ }
eval/export_parity.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "positions": 11605,
3
+ "fp32_onnx_max_abs_logit_diff": 0.00010442733764648438,
4
+ "bf16_torch_argmax_agreement": 0.9999138302455838,
5
+ "fp32_onnx_argmax_agreement": 1.0,
6
+ "int8_onnx_argmax_agreement": 0.999138302455838
7
+ }
eval/val_metrics.json ADDED
@@ -0,0 +1,367 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 2,
3
+ "n_forms": 3408,
4
+ "n_fields": 9819,
5
+ "field_acc": 0.9746410021387106,
6
+ "field_macro_f1": 0.9661696867284804,
7
+ "form_acc": 0.9571596244131455,
8
+ "form_macro_f1": 0.9376820897318026,
9
+ "password_precision": 0.9898358092259578,
10
+ "password_recall": 0.998422712933754,
11
+ "password_f1": 0.994110718492344,
12
+ "cur_vs_new_acc": 0.9944794952681388,
13
+ "identifier_f1": 0.9945380063723259,
14
+ "per_field": {
15
+ "username": {
16
+ "p": 0.9921752738654147,
17
+ "r": 0.9768875192604006,
18
+ "f1": 0.984472049689441,
19
+ "n": 649
20
+ },
21
+ "email": {
22
+ "p": 0.9884541372674791,
23
+ "r": 0.9961215255332903,
24
+ "f1": 0.9922730199613651,
25
+ "n": 1547
26
+ },
27
+ "current-password": {
28
+ "p": 0.9814814814814815,
29
+ "r": 0.9977851605758582,
30
+ "f1": 0.9895661724327293,
31
+ "n": 903
32
+ },
33
+ "new-password": {
34
+ "p": 0.997229916897507,
35
+ "r": 0.9863013698630136,
36
+ "f1": 0.9917355371900827,
37
+ "n": 365
38
+ },
39
+ "one-time-code": {
40
+ "p": 0.9901960784313726,
41
+ "r": 0.9805825242718447,
42
+ "f1": 0.9853658536585367,
43
+ "n": 103
44
+ },
45
+ "tel": {
46
+ "p": 0.9800918836140888,
47
+ "r": 0.9937888198757764,
48
+ "f1": 0.9868928296067848,
49
+ "n": 644
50
+ },
51
+ "name": {
52
+ "p": 0.9592123769338959,
53
+ "r": 0.9855491329479769,
54
+ "f1": 0.9722024233784746,
55
+ "n": 692
56
+ },
57
+ "given-name": {
58
+ "p": 0.978494623655914,
59
+ "r": 0.9161073825503355,
60
+ "f1": 0.9462738301559791,
61
+ "n": 298
62
+ },
63
+ "family-name": {
64
+ "p": 0.9961089494163424,
65
+ "r": 0.9808429118773946,
66
+ "f1": 0.9884169884169886,
67
+ "n": 261
68
+ },
69
+ "organization": {
70
+ "p": 0.9761904761904762,
71
+ "r": 0.9318181818181818,
72
+ "f1": 0.9534883720930233,
73
+ "n": 132
74
+ },
75
+ "street-address": {
76
+ "p": 0.9369369369369369,
77
+ "r": 0.9122807017543859,
78
+ "f1": 0.9244444444444444,
79
+ "n": 114
80
+ },
81
+ "address-line2": {
82
+ "p": 1.0,
83
+ "r": 0.9090909090909091,
84
+ "f1": 0.9523809523809523,
85
+ "n": 22
86
+ },
87
+ "address-level2": {
88
+ "p": 0.8843537414965986,
89
+ "r": 0.8904109589041096,
90
+ "f1": 0.887372013651877,
91
+ "n": 146
92
+ },
93
+ "address-level1": {
94
+ "p": 0.898989898989899,
95
+ "r": 0.9368421052631579,
96
+ "f1": 0.9175257731958762,
97
+ "n": 95
98
+ },
99
+ "postal-code": {
100
+ "p": 0.9724137931034482,
101
+ "r": 0.9724137931034482,
102
+ "f1": 0.9724137931034482,
103
+ "n": 145
104
+ },
105
+ "country": {
106
+ "p": 0.8863636363636364,
107
+ "r": 0.9873417721518988,
108
+ "f1": 0.9341317365269461,
109
+ "n": 79
110
+ },
111
+ "cc-name": {
112
+ "p": 1.0,
113
+ "r": 0.95,
114
+ "f1": 0.9743589743589743,
115
+ "n": 20
116
+ },
117
+ "cc-number": {
118
+ "p": 0.9565217391304348,
119
+ "r": 1.0,
120
+ "f1": 0.9777777777777777,
121
+ "n": 22
122
+ },
123
+ "cc-exp": {
124
+ "p": 1.0,
125
+ "r": 1.0,
126
+ "f1": 1.0,
127
+ "n": 13
128
+ },
129
+ "cc-exp-month": {
130
+ "p": 1.0,
131
+ "r": 1.0,
132
+ "f1": 1.0,
133
+ "n": 9
134
+ },
135
+ "cc-exp-year": {
136
+ "p": 1.0,
137
+ "r": 1.0,
138
+ "f1": 1.0,
139
+ "n": 8
140
+ },
141
+ "cc-csc": {
142
+ "p": 0.9523809523809523,
143
+ "r": 1.0,
144
+ "f1": 0.975609756097561,
145
+ "n": 20
146
+ },
147
+ "bday": {
148
+ "p": 1.0,
149
+ "r": 0.8461538461538461,
150
+ "f1": 0.9166666666666666,
151
+ "n": 39
152
+ },
153
+ "search": {
154
+ "p": 0.9624217118997912,
155
+ "r": 0.9808510638297873,
156
+ "f1": 0.9715489989462592,
157
+ "n": 470
158
+ },
159
+ "captcha": {
160
+ "p": 0.9814814814814815,
161
+ "r": 0.9352941176470588,
162
+ "f1": 0.9578313253012049,
163
+ "n": 170
164
+ },
165
+ "other": {
166
+ "p": 0.9703912583715192,
167
+ "r": 0.9649491763056431,
168
+ "f1": 0.9676625659050966,
169
+ "n": 2853
170
+ }
171
+ },
172
+ "per_form": {
173
+ "login": {
174
+ "p": 0.9900552486187846,
175
+ "r": 0.9933481152993349,
176
+ "f1": 0.991698948533481,
177
+ "n": 902
178
+ },
179
+ "signup": {
180
+ "p": 0.9951456310679612,
181
+ "r": 0.9855769230769231,
182
+ "f1": 0.9903381642512078,
183
+ "n": 208
184
+ },
185
+ "password-change": {
186
+ "p": 1.0,
187
+ "r": 1.0,
188
+ "f1": 1.0,
189
+ "n": 22
190
+ },
191
+ "password-recovery": {
192
+ "p": 0.8461538461538461,
193
+ "r": 0.9565217391304348,
194
+ "f1": 0.8979591836734695,
195
+ "n": 23
196
+ },
197
+ "otp": {
198
+ "p": 0.8888888888888888,
199
+ "r": 0.8571428571428571,
200
+ "f1": 0.8727272727272727,
201
+ "n": 28
202
+ },
203
+ "payment": {
204
+ "p": 1.0,
205
+ "r": 1.0,
206
+ "f1": 1.0,
207
+ "n": 22
208
+ },
209
+ "address": {
210
+ "p": 0.8666666666666667,
211
+ "r": 0.7222222222222222,
212
+ "f1": 0.7878787878787877,
213
+ "n": 36
214
+ },
215
+ "newsletter": {
216
+ "p": 0.9774193548387097,
217
+ "r": 0.9439252336448598,
218
+ "f1": 0.9603803486529319,
219
+ "n": 321
220
+ },
221
+ "contact": {
222
+ "p": 0.9369482976040353,
223
+ "r": 0.9750656167979003,
224
+ "f1": 0.9556270096463023,
225
+ "n": 762
226
+ },
227
+ "search": {
228
+ "p": 0.9555555555555556,
229
+ "r": 0.9681050656660413,
230
+ "f1": 0.961789375582479,
231
+ "n": 533
232
+ },
233
+ "other": {
234
+ "p": 0.9165085388994307,
235
+ "r": 0.8765880217785844,
236
+ "f1": 0.8961038961038961,
237
+ "n": 551
238
+ }
239
+ },
240
+ "top_confusions": [
241
+ [
242
+ "given-name",
243
+ "name",
244
+ 22
245
+ ],
246
+ [
247
+ "other",
248
+ "address-level2",
249
+ 16
250
+ ],
251
+ [
252
+ "other",
253
+ "search",
254
+ 16
255
+ ],
256
+ [
257
+ "address-level2",
258
+ "other",
259
+ 15
260
+ ],
261
+ [
262
+ "other",
263
+ "current-password",
264
+ 11
265
+ ],
266
+ [
267
+ "other",
268
+ "address-level1",
269
+ 10
270
+ ],
271
+ [
272
+ "captcha",
273
+ "other",
274
+ 9
275
+ ],
276
+ [
277
+ "other",
278
+ "country",
279
+ 9
280
+ ],
281
+ [
282
+ "search",
283
+ "other",
284
+ 9
285
+ ],
286
+ [
287
+ "other",
288
+ "email",
289
+ 9
290
+ ],
291
+ [
292
+ "username",
293
+ "email",
294
+ 8
295
+ ],
296
+ [
297
+ "street-address",
298
+ "other",
299
+ 7
300
+ ],
301
+ [
302
+ "organization",
303
+ "other",
304
+ 7
305
+ ],
306
+ [
307
+ "address-level1",
308
+ "other",
309
+ 6
310
+ ],
311
+ [
312
+ "other",
313
+ "tel",
314
+ 6
315
+ ],
316
+ [
317
+ "name",
318
+ "given-name",
319
+ 5
320
+ ],
321
+ [
322
+ "username",
323
+ "other",
324
+ 5
325
+ ],
326
+ [
327
+ "other",
328
+ "street-address",
329
+ 5
330
+ ],
331
+ [
332
+ "bday",
333
+ "other",
334
+ 5
335
+ ],
336
+ [
337
+ "new-password",
338
+ "current-password",
339
+ 4
340
+ ],
341
+ [
342
+ "name",
343
+ "other",
344
+ 4
345
+ ],
346
+ [
347
+ "other",
348
+ "postal-code",
349
+ 4
350
+ ],
351
+ [
352
+ "postal-code",
353
+ "other",
354
+ 4
355
+ ],
356
+ [
357
+ "other",
358
+ "name",
359
+ 3
360
+ ],
361
+ [
362
+ "family-name",
363
+ "name",
364
+ 3
365
+ ]
366
+ ]
367
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7c4424bdc370732e57af4c22907420b23f2cb0ea649db4a4b4ff67aa85b22695
3
+ size 41414010
onnx/model.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8c24a379a5d484cab9ab82ff1a3a420987c9fd751cf10a2143c33aacef4a2645
3
+ size 82955341
onnx/model_quantized.onnx ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fd39c40e43437a726f4e82691e36c3cbfdc36b8269dd528e1ceb2d8ee5bbcbf4
3
+ size 21107484
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "mask_token": "[MASK]",
4
+ "model_max_length": 512,
5
+ "pad_token": "[PAD]",
6
+ "sep_token": "[SEP]",
7
+ "tokenizer_class": "PreTrainedTokenizerFast",
8
+ "unk_token": "[UNK]"
9
+ }
training/card/infer_parity_py.py ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Python side of the JS/Python inference parity test: classify saved pages with zap_infer.Zap."""
2
+ import json, sys, os
3
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
4
+ from zap_infer import Zap
5
+ model_dir, pdir, out, n = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
6
+ zap = Zap.from_pretrained(model_dir)
7
+ pages = json.load(open(os.path.join(pdir, "py.json")))[:n]
8
+ res = []
9
+ for p in pages:
10
+ r = zap.classify_html(open(os.path.join(pdir, p["file"])).read(), p["url"])
11
+ res.append({"file": p["file"], "forms": [{"form": f["form"], "fields": [x["label"] for x in f["fields"]]} for f in r]})
12
+ json.dump(res, open(out, "w"))
13
+ print("ok", sum(len(r["forms"]) for r in res))
training/card/zap_extract.py ADDED
@@ -0,0 +1,580 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Zap form extractor (canonical Python implementation).
2
+
3
+ Finds fillable fields in an HTML page, groups them into forms, and produces
4
+ * the serialized text the Zap model reads (``serialize``), and
5
+ * a cleaned, index-annotated HTML snippet for the teacher LLM (``teacher_html``).
6
+
7
+ The JavaScript port in ``js/zap-extract.js`` implements the same rules on a live DOM.
8
+ Every rule here is expressible with standard DOM APIs, so keep it that way.
9
+ """
10
+ from __future__ import annotations
11
+
12
+ import copy
13
+ import hashlib
14
+ import re
15
+ import unicodedata
16
+ from urllib.parse import unquote, urlsplit
17
+
18
+ import lxml.html
19
+ from lxml import etree
20
+
21
+ SKIP_INPUT_TYPES = {"hidden", "submit", "button", "reset", "image", "file", "checkbox", "radio", "range", "color"}
22
+ FIELD_TAGS = {"input", "select", "textarea"}
23
+ NO_TEXT_TAGS = {"script", "style", "noscript", "template", "svg", "select", "textarea", "option", "head"}
24
+ HEADING_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6", "legend"}
25
+ DATA_ATTRS = ["data-testid", "data-test", "data-qa", "data-cy", "formcontrolname", "data-field", "data-name", "data-type"]
26
+ MAX_FIELDS = 40
27
+ MAX_FORMS = 10
28
+
29
+ # Key tokens (special tokens of the Zap tokenizer).
30
+ CTX_KEYS = ["[TITLE]", "[URL]", "[FORM]", "[HEAD]", "[BTN]", "[LINK]", "[TXT]", "[HID]"]
31
+ FIELD_KEYS = ["[FLD]", "[TYPE]", "[NAME]", "[ID]", "[CLASS]", "[AC]", "[PH]", "[LBL]", "[ARIA]", "[TIP]",
32
+ "[NEAR]", "[AFTER]", "[MODE]", "[MAX]", "[REQ]", "[OPT]", "[DATA]"]
33
+ SPECIAL_TOKENS = CTX_KEYS + FIELD_KEYS
34
+
35
+ # Character budgets (Unicode code points).
36
+ CTX_LIMITS = {"title": 60, "url": 60, "form": 60, "head": 60, "btn": 60, "link": 80, "txt": 120, "hid": 40}
37
+ FIELD_LIMITS = {"name": 40, "id": 40, "class": 30, "ac": 30, "ph": 50, "lbl": 60, "aria": 50, "tip": 40,
38
+ "near": 50, "after": 30, "mode": 12, "max": 6, "opt": 60, "data": 30}
39
+ CTX_ORDER = [("title", "[TITLE]"), ("url", "[URL]"), ("form", "[FORM]"), ("head", "[HEAD]"), ("btn", "[BTN]"),
40
+ ("link", "[LINK]"), ("txt", "[TXT]"), ("hid", "[HID]")]
41
+ FIELD_ORDER = [("type", "[TYPE]"), ("name", "[NAME]"), ("id", "[ID]"), ("class", "[CLASS]"), ("ac", "[AC]"),
42
+ ("ph", "[PH]"), ("lbl", "[LBL]"), ("aria", "[ARIA]"), ("tip", "[TIP]"), ("near", "[NEAR]"),
43
+ ("after", "[AFTER]"), ("mode", "[MODE]"), ("max", "[MAX]"), ("req", "[REQ]"), ("opt", "[OPT]"),
44
+ ("data", "[DATA]")]
45
+
46
+ _WS = re.compile(r"\s+")
47
+
48
+
49
+ def norm(s: str | None) -> str:
50
+ """NFKC + collapse whitespace + strip."""
51
+ if not s:
52
+ return ""
53
+ return _WS.sub(" ", unicodedata.normalize("NFKC", s)).strip()
54
+
55
+
56
+ def cut(s: str, n: int) -> str:
57
+ return s[:n].rstrip()
58
+
59
+
60
+ def words(s: str | None) -> str:
61
+ """Identifier -> lowercase words: 'loginForm_Email2' -> 'login form email 2'."""
62
+ if not s:
63
+ return ""
64
+ s = unicodedata.normalize("NFKC", s)
65
+ s = re.sub(r"([a-z])([A-Z])", r"\1 \2", s)
66
+ s = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1 \2", s)
67
+ s = re.sub(r"([A-Za-z])([0-9])", r"\1 \2", s)
68
+ s = re.sub(r"([0-9])([A-Za-z])", r"\1 \2", s)
69
+ out = "".join(c if unicodedata.category(c)[0] in "LNM" else " " for c in s)
70
+ return _WS.sub(" ", out).strip().lower()
71
+
72
+
73
+ def tag(el) -> str:
74
+ t = el.tag
75
+ return t.lower() if isinstance(t, str) else ""
76
+
77
+
78
+ def attr(el, name: str) -> str:
79
+ v = el.get(name)
80
+ return v if v is not None else ""
81
+
82
+
83
+ def is_element(node) -> bool:
84
+ return isinstance(node.tag, str)
85
+
86
+
87
+ def text(el) -> str:
88
+ """textContent without script/style/select/textarea/option/svg/template/noscript, normalized."""
89
+ parts: list[str] = []
90
+
91
+ def walk(e):
92
+ if tag(e) in NO_TEXT_TAGS:
93
+ return
94
+ if e.text and is_element(e):
95
+ parts.append(e.text)
96
+ for c in e:
97
+ if is_element(c):
98
+ walk(c)
99
+ if c.tail:
100
+ parts.append(c.tail)
101
+
102
+ if is_element(el):
103
+ walk(el)
104
+ return norm(" ".join(parts))
105
+
106
+
107
+ def input_type(el) -> str:
108
+ t = tag(el)
109
+ if t == "input":
110
+ return (attr(el, "type").strip().lower() or "text")
111
+ return t
112
+
113
+
114
+ def hidden_attr(el) -> bool:
115
+ if el.get("hidden") is not None or attr(el, "aria-hidden").strip().lower() == "true":
116
+ return True
117
+ style = attr(el, "style").replace(" ", "").lower()
118
+ return "display:none" in style or "visibility:hidden" in style
119
+
120
+
121
+ def is_field(el) -> bool:
122
+ t = tag(el)
123
+ if t not in FIELD_TAGS:
124
+ return False
125
+ if t == "input" and input_type(el) in SKIP_INPUT_TYPES:
126
+ return False
127
+ return el.get("disabled") is None
128
+
129
+
130
+ def visible_within(el, root) -> bool:
131
+ node = el
132
+ while node is not None:
133
+ if hidden_attr(node):
134
+ return False
135
+ if node is root:
136
+ return True
137
+ node = node.getparent()
138
+ return True
139
+
140
+
141
+ def contains_field(el) -> bool:
142
+ if tag(el) in FIELD_TAGS:
143
+ return True
144
+ return any(tag(d) in FIELD_TAGS for d in el.iter() if d is not el)
145
+
146
+
147
+ # ---------------------------------------------------------------- sibling walks (DOM node order)
148
+
149
+ def prev_nodes(el):
150
+ """Previous siblings of el in DOM order, nearest first: ('t', str) or ('e', element)."""
151
+ parent = el.getparent()
152
+ if parent is None:
153
+ return
154
+ kids = list(parent)
155
+ i = kids.index(el)
156
+ for j in range(i - 1, -1, -1):
157
+ k = kids[j]
158
+ if k.tail:
159
+ yield ("t", k.tail)
160
+ if is_element(k):
161
+ yield ("e", k)
162
+ if parent.text:
163
+ yield ("t", parent.text)
164
+
165
+
166
+ def next_nodes(el):
167
+ parent = el.getparent()
168
+ if parent is None:
169
+ return
170
+ if el.tail:
171
+ yield ("t", el.tail)
172
+ kids = list(parent)
173
+ i = kids.index(el)
174
+ for k in kids[i + 1:]:
175
+ if is_element(k):
176
+ yield ("e", k)
177
+ if k.tail:
178
+ yield ("t", k.tail)
179
+
180
+
181
+ def near_text(el, root) -> str:
182
+ node = el
183
+ for _level in range(3):
184
+ seen = 0
185
+ for kind, v in prev_nodes(node):
186
+ if kind == "t":
187
+ t = norm(v)
188
+ else:
189
+ if contains_field(v):
190
+ return ""
191
+ t = text(v)
192
+ if t:
193
+ return t
194
+ if kind == "e":
195
+ seen += 1
196
+ if seen >= 3:
197
+ break
198
+ if node is root:
199
+ break
200
+ node = node.getparent()
201
+ if node is None or node is root:
202
+ break
203
+ return ""
204
+
205
+
206
+ def after_text(el) -> str:
207
+ seen = 0
208
+ for kind, v in next_nodes(el):
209
+ if kind == "t":
210
+ t = norm(v)
211
+ else:
212
+ if contains_field(v):
213
+ return ""
214
+ t = text(v)
215
+ if t:
216
+ return t
217
+ if kind == "e":
218
+ seen += 1
219
+ if seen >= 2:
220
+ break
221
+ return ""
222
+
223
+
224
+ # ---------------------------------------------------------------- labels
225
+
226
+ class Page:
227
+ def __init__(self, doc, url: str):
228
+ self.doc = doc
229
+ self.url = url
230
+ self.by_id: dict[str, object] = {}
231
+ self.label_for: dict[str, list] = {}
232
+ for e in doc.iter():
233
+ if not is_element(e):
234
+ continue
235
+ i = e.get("id")
236
+ if i and i not in self.by_id:
237
+ self.by_id[i] = e
238
+ if tag(e) == "label" and e.get("for"):
239
+ self.label_for.setdefault(e.get("for"), []).append(e)
240
+ t = doc.find(".//title")
241
+ self.title = norm(t.text_content()) if t is not None else ""
242
+
243
+
244
+ def label_text(el, page: Page) -> str:
245
+ out: list[str] = []
246
+ for ref in attr(el, "aria-labelledby").split():
247
+ e = page.by_id.get(ref)
248
+ if e is not None:
249
+ out.append(text(e))
250
+ i = el.get("id")
251
+ if i:
252
+ for lab in page.label_for.get(i, []):
253
+ out.append(text(lab))
254
+ node = el.getparent()
255
+ while node is not None:
256
+ if tag(node) == "label":
257
+ out.append(text(node))
258
+ break
259
+ node = node.getparent()
260
+ seen, uniq = set(), []
261
+ for t in out:
262
+ if t and t not in seen:
263
+ seen.add(t)
264
+ uniq.append(t)
265
+ return " | ".join(uniq)
266
+
267
+
268
+ # ---------------------------------------------------------------- form context
269
+
270
+ def is_buttonish(e) -> bool:
271
+ t = tag(e)
272
+ if t == "button":
273
+ return True
274
+ if t == "input" and input_type(e) in ("submit", "button", "image"):
275
+ return True
276
+ return attr(e, "role").lower() == "button"
277
+
278
+
279
+ def heading(root) -> str:
280
+ for e in root.iter():
281
+ if is_element(e) and (tag(e) in HEADING_TAGS or attr(e, "role").lower() == "heading"):
282
+ t = text(e)
283
+ if t:
284
+ return t
285
+ node = root
286
+ for _level in range(4):
287
+ n = 0
288
+ for kind, sib in prev_nodes(node):
289
+ if kind != "e":
290
+ continue
291
+ n += 1
292
+ if n > 5:
293
+ break
294
+ if tag(sib) in HEADING_TAGS or attr(sib, "role").lower() == "heading":
295
+ t = text(sib)
296
+ if t:
297
+ return t
298
+ hs = [d for d in sib.iter() if is_element(d) and d is not sib and
299
+ (tag(d) in HEADING_TAGS or attr(d, "role").lower() == "heading")]
300
+ for d in reversed(hs):
301
+ t = text(d)
302
+ if t:
303
+ return t
304
+ node = node.getparent()
305
+ if node is None or tag(node) in ("body", "html"):
306
+ break
307
+ return ""
308
+
309
+
310
+ def button_texts(root) -> str:
311
+ out: list[str] = []
312
+ for e in root.iter():
313
+ if not is_element(e) or not is_buttonish(e):
314
+ continue
315
+ t = text(e) or norm(attr(e, "value")) or norm(attr(e, "aria-label")) or norm(attr(e, "alt")) or norm(attr(e, "title"))
316
+ if t and t not in out:
317
+ out.append(cut(t, 30))
318
+ if len(out) >= 3:
319
+ break
320
+ return " | ".join(out)
321
+
322
+
323
+ def link_texts(root) -> str:
324
+ out: list[str] = []
325
+ for e in root.iter():
326
+ if not is_element(e) or tag(e) != "a":
327
+ continue
328
+ t = text(e) or norm(attr(e, "aria-label")) or norm(attr(e, "title"))
329
+ if t and t not in out:
330
+ out.append(cut(t, 30))
331
+ if len(out) >= 4:
332
+ break
333
+ return " | ".join(out)
334
+
335
+
336
+ def hidden_names(root) -> str:
337
+ out: list[str] = []
338
+ for e in root.iter():
339
+ if is_element(e) and tag(e) == "input" and input_type(e) == "hidden":
340
+ w = words(attr(e, "name"))
341
+ if w and w not in out:
342
+ out.append(w)
343
+ if len(out) >= 4:
344
+ break
345
+ return " ".join(out)
346
+
347
+
348
+ def url_words(url: str) -> str:
349
+ try:
350
+ u = urlsplit(url)
351
+ except ValueError:
352
+ return ""
353
+ host = (u.hostname or "").split(".")
354
+ lead = host[0] if len(host) >= 3 and host[0] != "www" else ""
355
+ return words(lead + " " + unquote(u.path))
356
+
357
+
358
+ def form_words(root) -> str:
359
+ parts = [attr(root, "id"), attr(root, "name"), attr(root, "class")]
360
+ if tag(root) == "form":
361
+ try:
362
+ parts.append(unquote(urlsplit(attr(root, "action")).path))
363
+ except ValueError:
364
+ pass
365
+ else:
366
+ parts.append(attr(root, "role"))
367
+ return (words(" ".join(parts)) + " " + norm(attr(root, "aria-label")).lower()).strip()
368
+
369
+
370
+ # ---------------------------------------------------------------- fields
371
+
372
+ def field_features(el, root, page: Page) -> dict:
373
+ t = tag(el)
374
+ f: dict = {"type": t if t != "input" else "input " + input_type(el)}
375
+ f["name"] = words(attr(el, "name"))
376
+ f["id"] = words(attr(el, "id"))
377
+ f["class"] = words(attr(el, "class"))
378
+ f["ac"] = norm(attr(el, "autocomplete")).lower()
379
+ f["ph"] = norm(attr(el, "placeholder"))
380
+ f["lbl"] = label_text(el, page)
381
+ f["aria"] = norm(attr(el, "aria-label"))
382
+ f["tip"] = norm(attr(el, "title"))
383
+ near = near_text(el, root)
384
+ f["near"] = "" if near in (f["lbl"], f["ph"], f["aria"]) else near
385
+ f["after"] = after_text(el)
386
+ f["mode"] = norm(attr(el, "inputmode")).lower()
387
+ f["max"] = norm(attr(el, "maxlength"))
388
+ f["req"] = el.get("required") is not None or attr(el, "aria-required").lower() == "true"
389
+ if t == "select":
390
+ opts = [text(o) or norm(o.text_content()) for o in el.iter() if is_element(o) and tag(o) == "option"]
391
+ opts = [o for o in opts if o]
392
+ f["opt"] = (str(len(opts)) + ": " + " | ".join(opts[:6])) if opts else ""
393
+ else:
394
+ f["opt"] = ""
395
+ data = [norm(attr(el, a)) for a in DATA_ATTRS if attr(el, a).strip()]
396
+ f["data"] = words(" ".join(data[:2]))
397
+ for k, n in FIELD_LIMITS.items():
398
+ if isinstance(f.get(k), str):
399
+ f[k] = cut(f[k], n)
400
+ return f
401
+
402
+
403
+ def form_context(root, page: Page) -> dict:
404
+ c = {"title": page.title, "url": url_words(page.url), "form": form_words(root), "head": heading(root),
405
+ "btn": button_texts(root), "link": link_texts(root), "txt": text(root), "hid": hidden_names(root)}
406
+ return {k: cut(v, CTX_LIMITS[k]) for k, v in c.items()}
407
+
408
+
409
+ def group_root(el, body, in_form):
410
+ node = el.getparent()
411
+ for _level in range(6):
412
+ if node is None or node is body or tag(node) in ("body", "html"):
413
+ break
414
+ if any(is_element(d) and is_buttonish(d) and id(d) not in in_form for d in node.iter()):
415
+ return node
416
+ node = node.getparent()
417
+ node = el
418
+ for _level in range(3):
419
+ p = node.getparent()
420
+ if p is None or p is body or tag(p) in ("body", "html"):
421
+ break
422
+ node = p
423
+ return node
424
+
425
+
426
+ def find_forms(doc) -> list[tuple[object, list]]:
427
+ """[(root, [field elements])], document order, <form>s first-seen order."""
428
+ body = doc.find(".//body")
429
+ if body is None:
430
+ body = doc
431
+ groups: dict[int, tuple[object, list]] = {}
432
+ order: list[int] = []
433
+ # keep the proxies alive: lxml recycles element proxies, so id() is only stable while referenced
434
+ form_nodes = [d for f in body.iter("form") for d in f.iter()]
435
+ in_form = {id(d) for d in form_nodes}
436
+ for el in body.iter():
437
+ if not is_element(el) or not is_field(el):
438
+ continue
439
+ root = None
440
+ node = el.getparent()
441
+ while node is not None:
442
+ if tag(node) == "form":
443
+ root = node
444
+ break
445
+ node = node.getparent()
446
+ if root is None:
447
+ root = group_root(el, body, in_form)
448
+ if not visible_within(el, root):
449
+ continue
450
+ key = id(root)
451
+ if key not in groups:
452
+ groups[key] = (root, [])
453
+ order.append(key)
454
+ if len(groups[key][1]) < MAX_FIELDS:
455
+ groups[key][1].append(el)
456
+ return [groups[k] for k in order][:MAX_FORMS]
457
+
458
+
459
+ # ---------------------------------------------------------------- serialization
460
+
461
+ def serialize_ctx(ctx: dict) -> str:
462
+ return " ".join(f"{key} {ctx[k]}" for k, key in CTX_ORDER if ctx.get(k))
463
+
464
+
465
+ def serialize_field(f: dict) -> str:
466
+ out = ["[FLD]"]
467
+ for k, key in FIELD_ORDER:
468
+ v = f.get(k)
469
+ if k == "req":
470
+ if v:
471
+ out.append("[REQ]")
472
+ elif v:
473
+ out.append(f"{key} {v}")
474
+ return " ".join(out)
475
+
476
+
477
+ def serialize(ctx: dict, fields: list[dict]) -> str:
478
+ return " ".join([serialize_ctx(ctx)] + [serialize_field(f) for f in fields])
479
+
480
+
481
+ # ---------------------------------------------------------------- teacher view
482
+
483
+ KEEP_ATTRS = {"id", "name", "type", "class", "placeholder", "autocomplete", "title", "for", "role", "action", "method",
484
+ "inputmode", "maxlength", "minlength", "required", "value", "href", "data-zap", "pattern", "alt", "label",
485
+ "multiple", "readonly"} | set(DATA_ATTRS)
486
+ DROP_TAGS = {"script", "style", "noscript", "template", "svg", "iframe", "img", "picture", "video", "audio", "canvas",
487
+ "link", "meta", "object", "embed", "path", "source", "use", "head"}
488
+
489
+
490
+ def teacher_html(root, field_els, max_chars=9000) -> str | None:
491
+ for i, el in enumerate(field_els, 1):
492
+ el.set("data-zap", str(i))
493
+ try:
494
+ r = copy.deepcopy(root)
495
+ finally:
496
+ for el in field_els:
497
+ el.attrib.pop("data-zap", None)
498
+ r.tail = None
499
+ for e in list(r.iter()):
500
+ if not is_element(e): # comments / PIs
501
+ p = e.getparent()
502
+ if p is not None:
503
+ e.drop_tree() if hasattr(e, "drop_tree") else p.remove(e)
504
+ continue
505
+ if tag(e) in DROP_TAGS and e is not r:
506
+ e.drop_tree()
507
+ for e in r.iter():
508
+ if not is_element(e):
509
+ continue
510
+ for a in list(e.attrib):
511
+ if a.startswith("aria-") or a in KEEP_ATTRS:
512
+ v = e.attrib[a]
513
+ if tag(e) == "input" and a == "value" and input_type(e) not in ("submit", "button", "hidden"):
514
+ del e.attrib[a]
515
+ continue
516
+ lim = 60 if a == "class" else 80
517
+ if a == "href":
518
+ try:
519
+ v = urlsplit(v).path
520
+ except ValueError:
521
+ v = ""
522
+ if len(v) > lim:
523
+ e.attrib[a] = v[:lim]
524
+ elif a == "href":
525
+ e.attrib[a] = v
526
+ else:
527
+ del e.attrib[a]
528
+ if tag(e) == "select":
529
+ opts = [o for o in e if is_element(o) and tag(o) in ("option", "optgroup")]
530
+ if len(opts) > 6:
531
+ for o in opts[6:]:
532
+ e.remove(o)
533
+ e.append(etree.Comment(f" +{len(opts) - 6} more options "))
534
+ if e.text:
535
+ e.text = cut(norm(e.text), 200) + (" " if e.text[-1:].isspace() else "")
536
+ if e.tail:
537
+ e.tail = " " + cut(norm(e.tail), 200) + " "
538
+ html = lxml.html.tostring(r, encoding="unicode", method="html")
539
+ html = re.sub(r">\s+<", "> <", html)
540
+ html = _WS.sub(" ", html).strip()
541
+ return html if len(html) <= max_chars else None
542
+
543
+
544
+ def signature(ctx: dict, fields: list[dict]) -> str:
545
+ key = "|".join([ctx.get("form", ""), ctx.get("btn", "")] +
546
+ [f"{f['type']}/{f['name']}/{f['id']}/{f['ac']}/{f['ph']}/{f['lbl']}" for f in fields])
547
+ return hashlib.sha1(key.encode()).hexdigest()[:16]
548
+
549
+
550
+ def html5_normalize(html: str) -> str:
551
+ """Re-serialize through an HTML5-spec parser (lexbor) so the tree matches what browsers build
552
+ (e.g. <form> inside <table>); lxml's libxml2 parser nests malformed markup differently."""
553
+ try:
554
+ from selectolax.lexbor import LexborHTMLParser
555
+ except ImportError:
556
+ return html
557
+ try:
558
+ return LexborHTMLParser(html).html or html
559
+ except Exception: # noqa: BLE001
560
+ return html
561
+
562
+
563
+ def extract_page(html: str | bytes, url: str, want_teacher: bool = True, html5: bool = True) -> list[dict]:
564
+ if html5 and isinstance(html, str):
565
+ html = html5_normalize(html)
566
+ try:
567
+ doc = lxml.html.document_fromstring(html)
568
+ except (etree.ParserError, ValueError):
569
+ return []
570
+ page = Page(doc, url)
571
+ out = []
572
+ for root, els in find_forms(doc):
573
+ fields = [field_features(el, root, page) for el in els]
574
+ ctx = form_context(root, page)
575
+ rec = {"kind": "form" if tag(root) == "form" else "group", "ctx": ctx, "fields": fields,
576
+ "sig": signature(ctx, fields)}
577
+ if want_teacher:
578
+ rec["html"] = teacher_html(root, els)
579
+ out.append(rec)
580
+ return out
training/card/zap_infer.py ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Zap inference helper: classify every fillable field of a web page (and each form's purpose).
2
+
3
+ from zap_infer import Zap
4
+ zap = Zap.from_pretrained("ProCreations/Zap") # or a local directory
5
+ for form in zap.classify_html(html, url):
6
+ print(form["form"], [(f["label"], round(f["score"], 3)) for f in form["fields"]])
7
+
8
+ In a browser extension, run zap-extract.js on the live DOM and feed each form's `ctx`/`fields` to the model
9
+ through the same windowing (see README); `Zap.classify_forms` accepts the same dicts.
10
+ """
11
+ from __future__ import annotations
12
+
13
+ import torch
14
+
15
+ from zap_extract import extract_page, serialize_ctx, serialize_field
16
+
17
+ MAX_LEN = 512
18
+
19
+
20
+ def windows(tok, ctx, fields, max_len=MAX_LEN):
21
+ """[CLS] ctx + as many whole fields as fit + [SEP]; returns [(input_ids, field_positions, field_indices)]."""
22
+ ctx_ids = tok(serialize_ctx(ctx), add_special_tokens=False)["input_ids"][:160]
23
+ f_ids = [tok(serialize_field(f), add_special_tokens=False)["input_ids"][:120] for f in fields]
24
+ out, cur, cur_idx, used = [], [], [], 0
25
+ budget = max_len - 2 - len(ctx_ids)
26
+ for i, ids in enumerate(f_ids):
27
+ if cur and used + len(ids) > budget:
28
+ out.append((cur, cur_idx))
29
+ cur, cur_idx, used = [], [], 0
30
+ cur.append(ids)
31
+ cur_idx.append(i)
32
+ used += len(ids)
33
+ if cur or not out:
34
+ out.append((cur, cur_idx))
35
+ res = []
36
+ for chunk, idx in out:
37
+ ids, pos = [tok.cls_token_id] + ctx_ids, []
38
+ for f in chunk:
39
+ pos.append(len(ids))
40
+ ids += f
41
+ ids.append(tok.sep_token_id)
42
+ res.append((ids, pos, idx))
43
+ return res
44
+
45
+
46
+ class Zap:
47
+ def __init__(self, model, tok, device="cpu"):
48
+ self.model, self.tok, self.device = model.eval().to(device), tok, device
49
+ labels = [model.config.id2label[i] for i in range(model.config.num_labels)]
50
+ self.form_ids = [i for i, l in enumerate(labels) if l.startswith("form:")]
51
+ self.field_ids = [i for i, l in enumerate(labels) if l.startswith("field:")]
52
+ self.form_names = [labels[i][5:] for i in self.form_ids]
53
+ self.field_names = [labels[i][6:] for i in self.field_ids]
54
+
55
+ @classmethod
56
+ def from_pretrained(cls, path="ProCreations/Zap", device="cpu", dtype=torch.float32):
57
+ from transformers import AutoModelForTokenClassification, AutoTokenizer
58
+ model = AutoModelForTokenClassification.from_pretrained(path, dtype=dtype)
59
+ return cls(model, AutoTokenizer.from_pretrained(path), device)
60
+
61
+ @torch.no_grad()
62
+ def classify_forms(self, forms):
63
+ """forms: [{"ctx": {...}, "fields": [{...}, ...]}] as produced by zap_extract / zap-extract.js."""
64
+ results = []
65
+ for form in forms:
66
+ wins = windows(self.tok, form["ctx"], form["fields"])
67
+ n = max(len(w[0]) for w in wins)
68
+ ids = torch.tensor([w[0] + [self.tok.pad_token_id] * (n - len(w[0])) for w in wins], device=self.device)
69
+ logits = self.model(input_ids=ids, attention_mask=(ids != self.tok.pad_token_id).long()).logits.float()
70
+ form_p = torch.softmax(logits[:, 0, self.form_ids].mean(0), -1)
71
+ fields = [None] * len(form["fields"])
72
+ for b, (_, pos, idx) in enumerate(wins):
73
+ for p, k in zip(pos, idx):
74
+ pr = torch.softmax(logits[b, p, self.field_ids], -1)
75
+ top = int(pr.argmax())
76
+ fields[k] = {"label": self.field_names[top], "score": float(pr[top]),
77
+ "probs": {self.field_names[j]: float(pr[j]) for j in pr.argsort(descending=True)[:3].tolist()}}
78
+ top = int(form_p.argmax())
79
+ results.append({"form": self.form_names[top], "form_score": float(form_p[top]), "fields": fields})
80
+ return results
81
+
82
+ def classify_html(self, html, url):
83
+ forms = extract_page(html, url, want_teacher=False)
84
+ out = self.classify_forms(forms)
85
+ for f, r in zip(forms, out):
86
+ r["kind"] = f["kind"]
87
+ for fld, res in zip(f["fields"], r["fields"]):
88
+ res["name"], res["id"], res["type"] = fld["name"], fld["id"], fld["type"]
89
+ return out
90
+
91
+
92
+ if __name__ == "__main__":
93
+ import json
94
+ import sys
95
+ import urllib.request
96
+ zap = Zap.from_pretrained(sys.argv[2] if len(sys.argv) > 2 else "ProCreations/Zap")
97
+ url = sys.argv[1]
98
+ html = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})).read().decode("utf-8", "replace")
99
+ print(json.dumps(zap.classify_html(html, url), indent=1, ensure_ascii=False))
training/harvest/cc_candidates.py ADDED
@@ -0,0 +1,124 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Stage A: sample CDX index blocks of a Common Crawl crawl and keep form-likely URLs.
2
+
3
+ Usage: python cc_candidates.py OUT.jsonl NUM_BLOCKS [SEED]
4
+ """
5
+ import asyncio
6
+ import gzip
7
+ import json
8
+ import os
9
+ import random
10
+ import re
11
+ import sys
12
+ import time
13
+ from collections import Counter, defaultdict
14
+
15
+ import aiohttp
16
+
17
+ CRAWL = "CC-MAIN-2026-39"
18
+ BASE = "https://data.commoncrawl.org/"
19
+ IDX = f"{BASE}cc-index/collections/{CRAWL}/indexes/"
20
+
21
+ KW = re.compile(
22
+ r"log-?_?in|sign-?_?in|signon|logon|auth|sso|account|register|registration|sign-?_?up|join|create-?_?account|"
23
+ r"passw|passwd|forgot|reset|recover|checkout|cart|basket|billing|payment|subscribe|newsletter|contact|profile|"
24
+ r"member|wp-login|/user|customer|my-?account|anmelden|registrier|konto|passwort|connexion|inscription|compte|"
25
+ r"mot-de-passe|iniciar|sesion|registro|cuenta|contrase|entrar|cadastro|conta|senha|accedi|registrati|accesso|"
26
+ r"inloggen|registreren|wachtwoord|logowanie|rejestracja|zaloguj|haslo|giris|kayit|uye|sifre|dang-nhap|dang-ky|"
27
+ r"masuk|daftar|vhod|voiti|registraci|parol|2fa|verify|verification|otp|mfa|shipping|address|booking|reserv|"
28
+ r"quote|apply|enquiry|inquiry|order|signin|login|kirjaudu|logga|tilmeld|registr|prihlas|belepes|autentic|"
29
+ r"%e7%99%bb|%e6%b3%a8%e5%86%8c|%e3%83%ad%e3%82%b0|%d0%b2%d1%85%d0%be%d0%b4|%eb%a1%9c%ea%b7%b8")
30
+ HTML_MIMES = ("text/html", "application/xhtml+xml")
31
+
32
+
33
+ async def fetch(session, sem, shard, off, length, tries=6):
34
+ url = IDX + shard
35
+ for t in range(tries):
36
+ try:
37
+ async with sem:
38
+ async with session.get(url, headers={"Range": f"bytes={off}-{off + length - 1}"},
39
+ timeout=aiohttp.ClientTimeout(total=120)) as r:
40
+ if r.status in (200, 206):
41
+ return await r.read()
42
+ if r.status in (429, 503, 500, 502, 504):
43
+ await asyncio.sleep(2 ** t + random.random())
44
+ continue
45
+ return None
46
+ except (aiohttp.ClientError, asyncio.TimeoutError):
47
+ await asyncio.sleep(2 ** t + random.random())
48
+ return None
49
+
50
+
51
+ async def main(out_path, n_blocks, seed):
52
+ rng = random.Random(seed)
53
+ async with aiohttp.ClientSession() as s:
54
+ cache = os.path.join(os.path.dirname(os.path.abspath(out_path)), "cluster.idx")
55
+ for attempt in range(8):
56
+ if os.path.exists(cache):
57
+ break
58
+ async with s.get(IDX + "cluster.idx") as r:
59
+ if r.status == 200:
60
+ body = await r.read()
61
+ if len(body) > 10_000_000:
62
+ open(cache, "wb").write(body)
63
+ break
64
+ print(f"cluster.idx status {r.status}, retrying", flush=True)
65
+ await asyncio.sleep(5 * (attempt + 1))
66
+ idx = open(cache).read().splitlines()
67
+ blocks = []
68
+ for line in idx:
69
+ parts = line.split("\t")
70
+ if len(parts) >= 4:
71
+ blocks.append((parts[1], int(parts[2]), int(parts[3])))
72
+ print(f"{len(blocks)} blocks in index", flush=True)
73
+ pick = rng.sample(blocks, n_blocks)
74
+ sem = asyncio.Semaphore(48)
75
+ per_host_kw, per_host_rand = defaultdict(int), defaultdict(int)
76
+ stats = Counter()
77
+ t0 = time.time()
78
+ with open(out_path, "w") as out:
79
+ for i in range(0, len(pick), 480):
80
+ chunk = pick[i:i + 480]
81
+ datas = await asyncio.gather(*[fetch(s, sem, *b) for b in chunk])
82
+ for data in datas:
83
+ if not data:
84
+ stats["block_fail"] += 1
85
+ continue
86
+ try:
87
+ text = gzip.decompress(data).decode("utf-8", "replace")
88
+ except OSError:
89
+ stats["gz_fail"] += 1
90
+ continue
91
+ for line in text.splitlines():
92
+ sp = line.find(" {")
93
+ if sp < 0:
94
+ continue
95
+ surt = line[:line.find(" ")]
96
+ try:
97
+ j = json.loads(line[sp + 1:])
98
+ except json.JSONDecodeError:
99
+ continue
100
+ stats["records"] += 1
101
+ if j.get("status") != "200" or j.get("mime-detected", j.get("mime")) not in HTML_MIMES:
102
+ continue
103
+ stats["html"] += 1
104
+ path = surt[surt.find(")") + 1:].lower()
105
+ host = surt[:surt.find(")")]
106
+ rec = {"url": j["url"], "filename": j["filename"], "offset": int(j["offset"]),
107
+ "length": int(j["length"]), "lang": j.get("languages", ""), "host": host}
108
+ if KW.search(path):
109
+ if per_host_kw[host] < 6:
110
+ per_host_kw[host] += 1
111
+ rec["src"] = "kw"
112
+ out.write(json.dumps(rec) + "\n")
113
+ stats["kw"] += 1
114
+ elif rng.random() < 1 / 60 and per_host_rand[host] < 2:
115
+ per_host_rand[host] += 1
116
+ rec["src"] = "rand"
117
+ out.write(json.dumps(rec) + "\n")
118
+ stats["rand"] += 1
119
+ el = time.time() - t0
120
+ print(f"{i + len(chunk)}/{len(pick)} blocks {el:.0f}s {dict(stats)}", flush=True)
121
+
122
+
123
+ if __name__ == "__main__":
124
+ asyncio.run(main(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]) if len(sys.argv) > 3 else 1234))
training/harvest/fetch_extract.py ADDED
@@ -0,0 +1,144 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Stage B: fetch WARC records by byte range and extract forms.
2
+
3
+ Usage: python fetch_extract.py CANDIDATES.jsonl OUT_DIR N_PROCS [CONC_PER_PROC]
4
+ Writes OUT_DIR/forms-XX.jsonl.gz (one form per line, with page metadata).
5
+ """
6
+ import asyncio
7
+ import gzip
8
+ import json
9
+ import multiprocessing as mp
10
+ import os
11
+ import random
12
+ import re
13
+ import sys
14
+ import time
15
+
16
+ import aiohttp
17
+
18
+ sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
19
+ from zap.extract import extract_page # noqa: E402
20
+
21
+ BASE = "https://data.commoncrawl.org/"
22
+ CHARSET_RE = re.compile(rb"""<meta[^>]+charset=["']?([A-Za-z0-9_\-]+)""", re.I)
23
+ XMLDECL_RE = re.compile(r"^\s*<\?xml[^>]*\?>", re.I)
24
+ FORMISH = re.compile(rb"<(input|select|textarea)\b", re.I)
25
+
26
+
27
+ def decode_record(raw: bytes):
28
+ raw = gzip.decompress(raw)
29
+ i = raw.find(b"\r\n\r\n")
30
+ rest = raw[i + 4:]
31
+ j = rest.find(b"\r\n\r\n")
32
+ if j < 0:
33
+ return None
34
+ http_h, body = rest[:j].decode("latin-1").lower(), rest[j + 4:]
35
+ if "\ncontent-encoding: gzip" in http_h and body[:2] == b"\x1f\x8b":
36
+ try:
37
+ body = gzip.decompress(body)
38
+ except OSError:
39
+ return None
40
+ if len(body) > 3_000_000 or not FORMISH.search(body):
41
+ return None
42
+ cs = None
43
+ m = re.search(r"content-type:[^\n]*charset=([a-z0-9_\-]+)", http_h)
44
+ if m:
45
+ cs = m.group(1)
46
+ else:
47
+ m = CHARSET_RE.search(body[:4096])
48
+ if m:
49
+ cs = m.group(1).decode("ascii", "ignore").lower()
50
+ for enc in [cs, "utf-8", "cp1252"]:
51
+ if not enc:
52
+ continue
53
+ try:
54
+ html = body.decode(enc)
55
+ break
56
+ except (LookupError, UnicodeDecodeError):
57
+ continue
58
+ else:
59
+ html = body.decode("utf-8", "replace")
60
+ return XMLDECL_RE.sub("", html)
61
+
62
+
63
+ async def fetch(session, rec, tries=6):
64
+ for t in range(tries):
65
+ try:
66
+ async with session.get(BASE + rec["filename"],
67
+ headers={"Range": f"bytes={rec['offset']}-{rec['offset'] + rec['length'] - 1}"},
68
+ timeout=aiohttp.ClientTimeout(total=60)) as r:
69
+ if r.status in (200, 206):
70
+ return await r.read()
71
+ if r.status in (429, 503, 500, 502, 504):
72
+ await asyncio.sleep(2 ** t + random.random())
73
+ continue
74
+ return None
75
+ except (aiohttp.ClientError, asyncio.TimeoutError):
76
+ await asyncio.sleep(2 ** t + random.random())
77
+ return None
78
+
79
+
80
+ async def run_worker(wid, recs, out_dir, conc):
81
+ out = gzip.open(os.path.join(out_dir, f"forms-{wid:02d}.jsonl.gz"), "wt")
82
+ stats = {"pages": 0, "fetched": 0, "with_forms": 0, "forms": 0, "fail": 0}
83
+ q: asyncio.Queue = asyncio.Queue()
84
+ for r in recs:
85
+ q.put_nowait(r)
86
+ t0 = time.time()
87
+
88
+ async def consumer(session):
89
+ while True:
90
+ try:
91
+ rec = q.get_nowait()
92
+ except asyncio.QueueEmpty:
93
+ return
94
+ data = await fetch(session, rec)
95
+ stats["pages"] += 1
96
+ if data is None:
97
+ stats["fail"] += 1
98
+ continue
99
+ stats["fetched"] += 1
100
+ try:
101
+ html = decode_record(data)
102
+ forms = extract_page(html, rec["url"]) if html else []
103
+ except Exception as e: # noqa: BLE001 - malformed pages must not kill the worker
104
+ stats.setdefault("err", 0)
105
+ stats["err"] += 1
106
+ forms = []
107
+ if forms:
108
+ stats["with_forms"] += 1
109
+ for k, f in enumerate(forms):
110
+ f.update(url=rec["url"], host=rec["host"], lang=rec["lang"], src=rec["src"], fidx=k)
111
+ out.write(json.dumps(f, ensure_ascii=False) + "\n")
112
+ stats["forms"] += 1
113
+ if wid == 0 and stats["pages"] % 500 == 0:
114
+ el = time.time() - t0
115
+ print(f"w0 {stats} {stats['pages'] / el:.1f} pages/s", flush=True)
116
+ await asyncio.sleep(0)
117
+
118
+ conn = aiohttp.TCPConnector(limit=conc)
119
+ async with aiohttp.ClientSession(connector=conn) as session:
120
+ await asyncio.gather(*[consumer(session) for _ in range(conc)])
121
+ out.close()
122
+ return stats
123
+
124
+
125
+ def worker(args):
126
+ wid, recs, out_dir, conc = args
127
+ return asyncio.run(run_worker(wid, recs, out_dir, conc))
128
+
129
+
130
+ if __name__ == "__main__":
131
+ cand, out_dir, nproc = sys.argv[1], sys.argv[2], int(sys.argv[3])
132
+ conc = int(sys.argv[4]) if len(sys.argv) > 4 else 4
133
+ os.makedirs(out_dir, exist_ok=True)
134
+ recs = [json.loads(l) for l in open(cand)]
135
+ random.Random(7).shuffle(recs)
136
+ chunks = [(i, recs[i::nproc], out_dir, conc) for i in range(nproc)]
137
+ t0 = time.time()
138
+ with mp.Pool(nproc) as pool:
139
+ results = pool.map(worker, chunks)
140
+ tot = {}
141
+ for r in results:
142
+ for k, v in r.items():
143
+ tot[k] = tot.get(k, 0) + v
144
+ print(f"DONE {time.time() - t0:.0f}s {tot}", flush=True)
training/harvest/parity_py.py ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ import json, sys
2
+ sys.path.insert(0, "/job/work")
3
+ from zap.extract import extract_page, serialize
4
+ d = sys.argv[1]
5
+ py = json.load(open(f"{d}/py.json"))
6
+ for p in py:
7
+ html = open(f"{d}/{p['file']}").read()
8
+ p["texts"] = [serialize(f["ctx"], f["fields"]) for f in extract_page(html, p["url"], want_teacher=False)]
9
+ json.dump(py, open(f"{d}/py5.json", "w"), ensure_ascii=False)
10
+ print("ok", sum(len(p["texts"]) for p in py))
training/harvest/polite_harvest.py ADDED
@@ -0,0 +1,254 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Polite streaming Common Crawl harvester (replaces the burst-y two-stage pilot).
2
+
3
+ One asyncio fetcher with an adaptive global request rate (backs off hard on 403/429/503),
4
+ CDX index blocks fetched on demand, form-likely URLs prioritized, parsing in a process pool.
5
+ Forms are written to rotating shards OUT/part-NNNN.jsonl.gz (a shard is complete once renamed from .tmp).
6
+
7
+ Usage: python polite_harvest.py OUT_DIR MAX_PAGES [--rate 15] [--max-rate 40] [--procs 12]
8
+ """
9
+ import argparse
10
+ import asyncio
11
+ import gzip
12
+ import json
13
+ import os
14
+ import random
15
+ import re
16
+ import sys
17
+ import time
18
+ from collections import defaultdict
19
+ from concurrent.futures import ProcessPoolExecutor
20
+
21
+ import aiohttp
22
+
23
+ sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
24
+ from harvest.fetch_extract import decode_record # noqa: E402
25
+ from zap.extract import extract_page # noqa: E402
26
+
27
+ CRAWL = "CC-MAIN-2026-39"
28
+ BASE = "https://data.commoncrawl.org/"
29
+ IDX = f"{BASE}cc-index/collections/{CRAWL}/indexes/"
30
+ HTML_MIMES = ("text/html", "application/xhtml+xml")
31
+ TIER1 = re.compile(
32
+ r"log-?_?in|sign-?_?in|signon|logon|/auth|sso|register|registration|sign-?_?up|create-?_?account|passw|forgot|"
33
+ r"reset|recover|checkout|wp-login|my-?account|/account|anmelden|registrier|passwort|connexion|inscription|"
34
+ r"mot-de-passe|iniciar-sesion|registro|contrase|entrar|cadastro|senha|accedi|registrati|inloggen|registreren|"
35
+ r"wachtwoord|logowanie|rejestracja|zaloguj|giris|kayit|sifre|dang-nhap|dang-ky|masuk|daftar|vhod|parol|2fa|"
36
+ r"verify|otp|mfa|billing|payment|%e7%99%bb|%e6%b3%a8%e5%86%8c|%e3%83%ad%e3%82%b0|%eb%a1%9c%ea%b7%b8")
37
+ TIER2 = re.compile(
38
+ r"account|join|subscribe|newsletter|contact|profile|member|/user|customer|konto|compte|cuenta|conta|accesso|"
39
+ r"uye|verification|shipping|address|booking|reserv|quote|apply|enquiry|inquiry|order|cart|basket|kirjaudu|"
40
+ r"logga|tilmeld|registr|prihlas|belepes|autentic|%d0%b2%d1%85%d0%be%d0%b4")
41
+
42
+
43
+ class Limiter:
44
+ def __init__(self, rate, max_rate):
45
+ self.rate, self.max_rate = rate, max_rate
46
+ self.next_t = time.monotonic()
47
+ self.pause_until = 0.0
48
+ self.ok_streak = 0
49
+ self.blocks = 0
50
+
51
+ async def wait(self):
52
+ while True:
53
+ now = time.monotonic()
54
+ if now < self.pause_until:
55
+ await asyncio.sleep(self.pause_until - now)
56
+ continue
57
+ if now >= self.next_t:
58
+ self.next_t = max(self.next_t, now - 0.5) + 1.0 / self.rate
59
+ return
60
+ await asyncio.sleep(self.next_t - now)
61
+
62
+ def ok(self):
63
+ self.ok_streak += 1
64
+ if self.ok_streak >= 400 and self.rate < self.max_rate:
65
+ self.rate = min(self.max_rate, self.rate * 1.1)
66
+ self.ok_streak = 0
67
+
68
+ def throttled(self, status):
69
+ self.blocks += 1
70
+ self.ok_streak = 0
71
+ self.rate = max(2.0, self.rate / 2)
72
+ pause = 90 if status == 403 else 20
73
+ self.pause_until = max(self.pause_until, time.monotonic() + pause)
74
+ print(f"throttled {status}: pausing {pause}s, rate -> {self.rate:.1f}/s", flush=True)
75
+
76
+
77
+ async def get(session, lim, url, rng=None, tries=8):
78
+ for _ in range(tries):
79
+ await lim.wait()
80
+ try:
81
+ headers = {"Range": f"bytes={rng[0]}-{rng[1]}"} if rng else {}
82
+ async with session.get(url, headers=headers, timeout=aiohttp.ClientTimeout(total=90)) as r:
83
+ if r.status in (200, 206):
84
+ data = await r.read()
85
+ lim.ok()
86
+ return data
87
+ if r.status in (403, 429, 503):
88
+ lim.throttled(r.status)
89
+ continue
90
+ if r.status >= 500:
91
+ await asyncio.sleep(3)
92
+ continue
93
+ return None
94
+ except (aiohttp.ClientError, asyncio.TimeoutError):
95
+ await asyncio.sleep(3)
96
+ return None
97
+
98
+
99
+ def parse_job(args):
100
+ rec, data = args
101
+ try:
102
+ html = decode_record(data)
103
+ return rec, (extract_page(html, rec["url"]) if html else [])
104
+ except Exception: # noqa: BLE001
105
+ return rec, []
106
+
107
+
108
+ class Writer:
109
+ def __init__(self, out_dir, per_shard=5000):
110
+ self.out_dir, self.per_shard = out_dir, per_shard
111
+ self.n_shard = len([f for f in os.listdir(out_dir) if f.startswith("part-")])
112
+ self.f, self.count, self.total = None, 0, 0
113
+
114
+ def write(self, row):
115
+ if self.f is None:
116
+ self.path = os.path.join(self.out_dir, f"part-{self.n_shard:04d}.jsonl.gz")
117
+ self.f = gzip.open(self.path + ".tmp", "wt")
118
+ self.f.write(json.dumps(row, ensure_ascii=False) + "\n")
119
+ self.count += 1
120
+ self.total += 1
121
+ if self.count >= self.per_shard:
122
+ self.close()
123
+
124
+ def close(self):
125
+ if self.f:
126
+ self.f.close()
127
+ os.rename(self.path + ".tmp", self.path)
128
+ self.f, self.count = None, 0
129
+ self.n_shard += 1
130
+
131
+
132
+ async def main(a):
133
+ os.makedirs(a.out, exist_ok=True)
134
+ rnd = random.Random(a.seed)
135
+ lim = Limiter(a.rate, a.max_rate)
136
+ writer = Writer(a.out)
137
+ pool = ProcessPoolExecutor(a.procs)
138
+ loop = asyncio.get_running_loop()
139
+ queues = {"t1": [], "t2": [], "rand": []}
140
+ per_host = defaultdict(int)
141
+ stats = defaultdict(int)
142
+ t0 = time.time()
143
+ cache = os.path.join(a.out, "..", "cluster.idx")
144
+ async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=a.conc)) as s:
145
+ if not os.path.exists(cache):
146
+ data = await get(s, lim, IDX + "cluster.idx")
147
+ if not data:
148
+ raise SystemExit("cannot fetch cluster.idx")
149
+ open(cache, "wb").write(data)
150
+ blocks = []
151
+ for line in open(cache):
152
+ p = line.rstrip("\n").split("\t")
153
+ if len(p) >= 4:
154
+ blocks.append((p[1], int(p[2]), int(p[3])))
155
+ rnd.shuffle(blocks)
156
+ print(f"{len(blocks)} index blocks", flush=True)
157
+ block_i = 0
158
+ refill_lock = asyncio.Lock()
159
+
160
+ async def refill():
161
+ nonlocal block_i
162
+ async with refill_lock:
163
+ if len(queues["t1"]) > 300 or block_i >= len(blocks):
164
+ return
165
+ shard, off, ln = blocks[block_i]
166
+ block_i += 1
167
+ data = await get(s, lim, IDX + shard, (off, off + ln - 1))
168
+ stats["blocks"] += 1
169
+ if not data:
170
+ return
171
+ for line in gzip.decompress(data).decode("utf-8", "replace").splitlines():
172
+ sp = line.find(" {")
173
+ if sp < 0:
174
+ continue
175
+ surt = line[:line.find(" ")]
176
+ try:
177
+ j = json.loads(line[sp + 1:])
178
+ except json.JSONDecodeError:
179
+ continue
180
+ if j.get("status") != "200" or j.get("mime-detected", j.get("mime")) not in HTML_MIMES:
181
+ continue
182
+ host = surt[:surt.find(")")]
183
+ path = surt[surt.find(")") + 1:].lower()
184
+ rec = {"url": j["url"], "filename": j["filename"], "offset": int(j["offset"]),
185
+ "length": int(j["length"]), "lang": j.get("languages", ""), "host": host}
186
+ if TIER1.search(path):
187
+ tier = "t1"
188
+ elif TIER2.search(path):
189
+ tier = "t2"
190
+ elif rnd.random() < 1 / 150:
191
+ tier = "rand"
192
+ else:
193
+ continue
194
+ cap = 2 if tier == "rand" else 5
195
+ if per_host[(host, tier)] >= cap:
196
+ continue
197
+ per_host[(host, tier)] += 1
198
+ rec["src"] = tier
199
+ queues[tier].append(rec)
200
+ for q in queues.values():
201
+ rnd.shuffle(q)
202
+
203
+ async def next_rec():
204
+ while True:
205
+ if len(queues["t1"]) < 200:
206
+ await refill()
207
+ x = rnd.random()
208
+ order = ["t1", "t2", "rand"] if x < 0.6 else (["t2", "t1", "rand"] if x < 0.85 else ["rand", "t1", "t2"])
209
+ for t in order:
210
+ if queues[t]:
211
+ return queues[t].pop()
212
+ if block_i >= len(blocks):
213
+ return None
214
+
215
+ pages = 0
216
+
217
+ async def fetcher():
218
+ nonlocal pages
219
+ while pages < a.max_pages:
220
+ rec = await next_rec()
221
+ if rec is None:
222
+ return
223
+ pages += 1
224
+ data = await get(s, lim, BASE + rec["filename"], (rec["offset"], rec["offset"] + rec["length"] - 1))
225
+ if not data:
226
+ stats["fail"] += 1
227
+ continue
228
+ rec2, forms = await loop.run_in_executor(pool, parse_job, (rec, data))
229
+ stats["pages_" + rec["src"]] += 1
230
+ for k, f in enumerate(forms):
231
+ f.update(url=rec["url"], host=rec["host"], lang=rec["lang"], src=rec["src"], fidx=k)
232
+ writer.write(f)
233
+ stats["forms"] += 1
234
+ if pages % 1000 == 0:
235
+ el = time.time() - t0
236
+ print(f"{pages} pages {el:.0f}s {pages / el:.1f}/s rate {lim.rate:.1f} throttles {lim.blocks} "
237
+ f"{dict(stats)} queued {[len(q) for q in queues.values()]}", flush=True)
238
+
239
+ await asyncio.gather(*[fetcher() for _ in range(a.conc)])
240
+ writer.close()
241
+ pool.shutdown()
242
+ print(f"DONE {time.time() - t0:.0f}s pages {pages} {dict(stats)}", flush=True)
243
+
244
+
245
+ if __name__ == "__main__":
246
+ ap = argparse.ArgumentParser()
247
+ ap.add_argument("out")
248
+ ap.add_argument("max_pages", type=int)
249
+ ap.add_argument("--rate", type=float, default=15)
250
+ ap.add_argument("--max-rate", type=float, default=40)
251
+ ap.add_argument("--conc", type=int, default=24)
252
+ ap.add_argument("--procs", type=int, default=12)
253
+ ap.add_argument("--seed", type=int, default=2026)
254
+ asyncio.run(main(ap.parse_args()))
training/harvest/sample_pages.py ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Save N decoded HTML pages (with form fields) from one WARC + the Python extractor output, for JS parity tests."""
2
+ import json, os, sys, urllib.request
3
+ sys.path.insert(0, "/job/work")
4
+ from harvest.warc_stream import BASE, members, parse_record
5
+ from zap.extract import extract_page, serialize
6
+ out, n, path = sys.argv[1], int(sys.argv[2]), sys.argv[3]
7
+ os.makedirs(out, exist_ok=True)
8
+ res, k = [], 0
9
+ with urllib.request.urlopen(BASE + path, timeout=300) as r:
10
+ for raw in members(r):
11
+ rec = parse_record(raw)
12
+ if not rec:
13
+ continue
14
+ url, lang, http_h, body = rec
15
+ import re
16
+ m = re.search(r"charset=([a-z0-9_\-]+)", http_h)
17
+ try:
18
+ html = body.decode(m.group(1) if m else "utf-8")
19
+ except (LookupError, UnicodeDecodeError):
20
+ continue
21
+ forms = extract_page(html, url, want_teacher=False)
22
+ if not forms:
23
+ continue
24
+ open(f"{out}/{k:04d}.html", "w").write(html)
25
+ res.append({"file": f"{k:04d}.html", "url": url, "texts": [serialize(f["ctx"], f["fields"]) for f in forms]})
26
+ k += 1
27
+ if k >= n:
28
+ break
29
+ json.dump(res, open(f"{out}/py.json", "w"), ensure_ascii=False)
30
+ print("saved", k)
training/harvest/warc_stream.py ADDED
@@ -0,0 +1,181 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Stream whole Common Crawl WARC files (one HTTP request per file) and extract forms.
2
+
3
+ A WARC.gz is a concatenation of per-record gzip members; we split members on the fly and hand
4
+ HTML response records to a process pool. Far fewer requests than per-page range fetches.
5
+
6
+ Usage: python warc_stream.py OUT_DIR N_FILES [--procs 24] [--parallel 3] [--seed 7]
7
+ """
8
+ import argparse
9
+ import gzip
10
+ import json
11
+ import os
12
+ import random
13
+ import sys
14
+ import time
15
+ import urllib.request
16
+ import zlib
17
+ from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
18
+ from threading import Lock
19
+
20
+ sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
21
+ from harvest.fetch_extract import FORMISH # noqa: E402
22
+ from harvest.polite_harvest import Writer # noqa: E402
23
+ from zap.extract import extract_page # noqa: E402
24
+
25
+ BASE = "https://data.commoncrawl.org/"
26
+ CRAWL = "CC-MAIN-2026-39"
27
+
28
+
29
+ def members(stream):
30
+ """Yield decompressed gzip members from a byte stream."""
31
+ buf = b""
32
+ d = zlib.decompressobj(31)
33
+ out = []
34
+ while True:
35
+ chunk = stream.read(1 << 20)
36
+ if not chunk:
37
+ break
38
+ buf += chunk
39
+ while buf:
40
+ data = d.decompress(buf)
41
+ out.append(data)
42
+ if d.eof:
43
+ yield b"".join(out)
44
+ out = []
45
+ buf = d.unused_data
46
+ d = zlib.decompressobj(31)
47
+ else:
48
+ buf = b""
49
+
50
+
51
+ def parse_record(raw):
52
+ i = raw.find(b"\r\n\r\n")
53
+ if i < 0:
54
+ return None
55
+ wh = raw[:i].decode("latin-1")
56
+ if "WARC-Type: response" not in wh:
57
+ return None
58
+ url = ""
59
+ for line in wh.split("\r\n"):
60
+ if line.startswith("WARC-Target-URI:"):
61
+ url = line.split(":", 1)[1].strip()
62
+ lang = ""
63
+ for line in wh.split("\r\n"):
64
+ if line.startswith("WARC-Identified-Content-Language:"):
65
+ lang = line.split(":", 1)[1].strip()
66
+ rest = raw[i + 4:]
67
+ j = rest.find(b"\r\n\r\n")
68
+ if j < 0:
69
+ return None
70
+ http_h = rest[:j].decode("latin-1").lower()
71
+ if "content-type: text/html" not in http_h and "application/xhtml" not in http_h:
72
+ return None
73
+ body = rest[j + 4:]
74
+ if len(body) > 3_000_000 or not FORMISH.search(body):
75
+ return None
76
+ return url, lang, http_h, body
77
+
78
+
79
+ def extract_job(item):
80
+ url, lang, http_h, body = item
81
+ import re
82
+ cs = None
83
+ m = re.search(r"content-type:[^\n]*charset=([a-z0-9_\-]+)", http_h)
84
+ if m:
85
+ cs = m.group(1)
86
+ else:
87
+ m = re.search(rb"""<meta[^>]+charset=["']?([A-Za-z0-9_\-]+)""", body[:4096], re.I)
88
+ if m:
89
+ cs = m.group(1).decode("ascii", "ignore").lower()
90
+ html = None
91
+ for enc in [cs, "utf-8", "cp1252"]:
92
+ if not enc:
93
+ continue
94
+ try:
95
+ html = body.decode(enc)
96
+ break
97
+ except (LookupError, UnicodeDecodeError):
98
+ continue
99
+ if html is None:
100
+ html = body.decode("utf-8", "replace")
101
+ html = re.sub(r"^\s*<\?xml[^>]*\?>", "", html, flags=re.I)
102
+ try:
103
+ return url, lang, extract_page(html, url)
104
+ except Exception: # noqa: BLE001
105
+ return url, lang, []
106
+
107
+
108
+ def host_of(url):
109
+ try:
110
+ h = url.split("/")[2].lower()
111
+ except IndexError:
112
+ return ""
113
+ return ",".join(reversed(h.split(":")[0].split(".")))
114
+
115
+
116
+ def process_file(path, pool, writer, lock, stats):
117
+ t0 = time.time()
118
+ req = urllib.request.Request(BASE + path)
119
+ batch, futures = [], []
120
+ with urllib.request.urlopen(req, timeout=300) as r:
121
+ for raw in members(r):
122
+ rec = parse_record(raw)
123
+ if rec:
124
+ batch.append(rec)
125
+ if len(batch) >= 64:
126
+ futures.append(pool.submit(_batch, batch))
127
+ batch = []
128
+ if batch:
129
+ futures.append(pool.submit(_batch, batch))
130
+ n_pages = n_forms = 0
131
+ for fu in futures:
132
+ for url, lang, forms in fu.result():
133
+ n_pages += 1
134
+ host = host_of(url)
135
+ with lock:
136
+ for k, f in enumerate(forms):
137
+ f.update(url=url, host=host, lang=lang, src="warc", fidx=k)
138
+ writer.write(f)
139
+ n_forms += 1
140
+ with lock:
141
+ stats["files"] += 1
142
+ stats["pages"] += n_pages
143
+ stats["forms"] += n_forms
144
+ print(f"{path.split('/')[-1]} {n_pages} form-pages {n_forms} forms {time.time() - t0:.0f}s total {stats}", flush=True)
145
+
146
+
147
+ def _batch(items):
148
+ return [extract_job(it) for it in items]
149
+
150
+
151
+ def main(a):
152
+ os.makedirs(a.out, exist_ok=True)
153
+ paths_file = os.path.join(a.out, "..", "warc.paths")
154
+ if not os.path.exists(paths_file):
155
+ with urllib.request.urlopen(f"{BASE}crawl-data/{CRAWL}/warc.paths.gz") as r:
156
+ open(paths_file, "wb").write(gzip.decompress(r.read()))
157
+ paths = open(paths_file).read().split()
158
+ random.Random(a.seed).shuffle(paths)
159
+ paths = paths[:a.n_files]
160
+ writer = Writer(a.out, per_shard=10000)
161
+ lock = Lock()
162
+ stats = {"files": 0, "pages": 0, "forms": 0}
163
+ with ProcessPoolExecutor(a.procs) as pool, ThreadPoolExecutor(a.parallel) as tp:
164
+ futs = [tp.submit(process_file, p, pool, writer, lock, stats) for p in paths]
165
+ for f in futs:
166
+ try:
167
+ f.result()
168
+ except Exception as e: # noqa: BLE001
169
+ print("file failed:", e, flush=True)
170
+ writer.close()
171
+ print("DONE", stats, flush=True)
172
+
173
+
174
+ if __name__ == "__main__":
175
+ ap = argparse.ArgumentParser()
176
+ ap.add_argument("out")
177
+ ap.add_argument("n_files", type=int)
178
+ ap.add_argument("--procs", type=int, default=24)
179
+ ap.add_argument("--parallel", type=int, default=3)
180
+ ap.add_argument("--seed", type=int, default=7)
181
+ main(ap.parse_args())
training/js/bench.mjs ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // CPU latency benchmark: model-only time per form (extraction excluded), onnxruntime-node via transformers.js
2
+ import fs from "fs"; import path from "path"; import { createRequire } from "module";
3
+ const require = createRequire(import.meta.url);
4
+ const tf = require("@huggingface/transformers");
5
+ const { JSDOM, VirtualConsole } = require("jsdom");
6
+ const { zapExtract } = require("./zap-extract.js");
7
+ const { ZapModel } = require("./zap-infer.js");
8
+ const [modelDir, dir, dtype] = process.argv.slice(2);
9
+ tf.env.allowRemoteModels = false; tf.env.localModelPath = path.dirname(path.resolve(modelDir)) + "/";
10
+ const zap = await ZapModel.load(tf, path.basename(modelDir), { dtype: dtype || "fp32" });
11
+ const pages = JSON.parse(fs.readFileSync(path.join(dir, "py.json"), "utf8")).slice(0, 200);
12
+ const vc = new VirtualConsole(); const forms = []; let tx = 0;
13
+ for (const p of pages) { const d = new JSDOM(fs.readFileSync(path.join(dir, p.file), "utf8"), { url: p.url, virtualConsole: vc });
14
+ const t = performance.now(); forms.push(...zapExtract(d.window.document, p.url)); tx += performance.now() - t; }
15
+ await zap.classify(forms.slice(0, 20)); // warmup
16
+ const t0 = performance.now(); const res = await zap.classify(forms); const dt = performance.now() - t0;
17
+ const nf = forms.reduce((a, f) => a + f.fields.length, 0);
18
+ console.log(JSON.stringify({ dtype: dtype || "fp32", forms: forms.length, fields: nf, ms_per_form: +(dt / forms.length).toFixed(2), extract_ms_per_form: +(tx / forms.length).toFixed(2) }));
training/js/infer-parity-test.mjs ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // End-to-end JS inference (jsdom + zap-extract.js + zap-infer.js + transformers.js/onnxruntime-node)
2
+ // usage: node infer-parity-test.mjs MODEL_DIR PARITY_DIR OUT.json
3
+ import fs from "fs";
4
+ import path from "path";
5
+ import { createRequire } from "module";
6
+ const require = createRequire(import.meta.url);
7
+ const tf = require("@huggingface/transformers");
8
+ const { JSDOM, VirtualConsole } = require("jsdom");
9
+ const { zapExtract } = require("./zap-extract.js");
10
+ const { ZapModel } = require("./zap-infer.js");
11
+ const [modelDir, dir, outFile] = process.argv.slice(2);
12
+ tf.env.allowRemoteModels = false;
13
+ tf.env.localModelPath = path.dirname(path.resolve(modelDir)) + "/";
14
+ const zap = await ZapModel.load(tf, path.basename(modelDir));
15
+ const pages = JSON.parse(fs.readFileSync(path.join(dir, "py.json"), "utf8")).slice(0, Number(process.env.N || 150));
16
+ const vc = new VirtualConsole();
17
+ const out = [];
18
+ const t0 = Date.now();
19
+ let nf = 0;
20
+ for (const p of pages) {
21
+ const dom = new JSDOM(fs.readFileSync(path.join(dir, p.file), "utf8"), { url: p.url, virtualConsole: vc });
22
+ const forms = zapExtract(dom.window.document, p.url);
23
+ const res = await zap.classify(forms);
24
+ nf += forms.length;
25
+ out.push({ file: p.file, forms: res.map((r) => ({ form: r.form, fields: r.fields.map((f) => f.label) })) });
26
+ dom.window.close();
27
+ }
28
+ console.log(`classified ${nf} forms on ${pages.length} pages in ${Date.now() - t0} ms`);
29
+ fs.writeFileSync(outFile, JSON.stringify(out));
training/js/parity-test.js ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Parity test: JS extractor (jsdom) vs Python extractor on saved real pages.
2
+ // usage: NODE_PATH=<dir with jsdom> node parity-test.js PARITY_DIR
3
+ const fs = require("fs");
4
+ const path = require("path");
5
+ const { JSDOM, VirtualConsole } = require("jsdom");
6
+ const vc = new VirtualConsole();
7
+ const { zapExtract } = require("./zap-extract.js");
8
+ const dir = process.argv[2];
9
+ const py = JSON.parse(fs.readFileSync(path.join(dir, process.argv[3] || "py.json"), "utf8"));
10
+ let pages = 0, samePages = 0, forms = 0, sameForms = 0, pyForms = 0;
11
+ const diffs = [];
12
+ for (const p of py) {
13
+ const html = fs.readFileSync(path.join(dir, p.file), "utf8");
14
+ const dom = new JSDOM(html, { url: p.url, virtualConsole: vc });
15
+ const js = zapExtract(dom.window.document, p.url).map((f) => f.text);
16
+ pages++; pyForms += p.texts.length; forms += js.length;
17
+ const pySet = new Set(p.texts);
18
+ let same = 0;
19
+ for (const t of js) if (pySet.has(t)) same++;
20
+ sameForms += same;
21
+ if (same === p.texts.length && js.length === p.texts.length) samePages++;
22
+ else if (diffs.length < 12) {
23
+ const a = p.texts.find((t) => !js.includes(t)) || "", b = js.find((t) => !pySet.has(t)) || "";
24
+ let i = 0; while (i < Math.min(a.length, b.length) && a[i] === b[i]) i++;
25
+ diffs.push({ file: p.file, py: p.texts.length, js: js.length, at: i, pyText: a.slice(Math.max(0, i - 60), i + 100), jsText: b.slice(Math.max(0, i - 60), i + 100) });
26
+ }
27
+ dom.window.close();
28
+ }
29
+ console.log(JSON.stringify({ pages, samePages, pyForms, jsForms: forms, identicalForms: sameForms }, null, 1));
30
+ for (const d of diffs) console.log(JSON.stringify(d));
training/js/zap-extract.js ADDED
@@ -0,0 +1,326 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Zap form extractor — JavaScript port of zap/extract.py (keep the two in lockstep).
2
+ // Runs on a live DOM (browser extension content script) or jsdom.
3
+ // const forms = zapExtract(document, location.href);
4
+ // forms[i] = { root, elements, ctx, fields, text } // `text` is what the Zap model reads
5
+ (function (global) {
6
+ "use strict";
7
+
8
+ const SKIP_INPUT_TYPES = new Set(["hidden", "submit", "button", "reset", "image", "file", "checkbox", "radio", "range", "color"]);
9
+ const FIELD_TAGS = new Set(["input", "select", "textarea"]);
10
+ const NO_TEXT_TAGS = new Set(["script", "style", "noscript", "template", "svg", "select", "textarea", "option", "head"]);
11
+ const HEADING_TAGS = new Set(["h1", "h2", "h3", "h4", "h5", "h6", "legend"]);
12
+ const DATA_ATTRS = ["data-testid", "data-test", "data-qa", "data-cy", "formcontrolname", "data-field", "data-name", "data-type"];
13
+ const MAX_FIELDS = 40;
14
+ const MAX_FORMS = 10;
15
+ const CTX_LIMITS = { title: 60, url: 60, form: 60, head: 60, btn: 60, link: 80, txt: 120, hid: 40 };
16
+ const FIELD_LIMITS = { name: 40, id: 40, class: 30, ac: 30, ph: 50, lbl: 60, aria: 50, tip: 40, near: 50, after: 30, mode: 12, max: 6, opt: 60, data: 30 };
17
+ const CTX_ORDER = [["title", "[TITLE]"], ["url", "[URL]"], ["form", "[FORM]"], ["head", "[HEAD]"], ["btn", "[BTN]"], ["link", "[LINK]"], ["txt", "[TXT]"], ["hid", "[HID]"]];
18
+ const FIELD_ORDER = [["type", "[TYPE]"], ["name", "[NAME]"], ["id", "[ID]"], ["class", "[CLASS]"], ["ac", "[AC]"], ["ph", "[PH]"], ["lbl", "[LBL]"], ["aria", "[ARIA]"], ["tip", "[TIP]"], ["near", "[NEAR]"], ["after", "[AFTER]"], ["mode", "[MODE]"], ["max", "[MAX]"], ["req", "[REQ]"], ["opt", "[OPT]"], ["data", "[DATA]"]];
19
+
20
+ const ELEMENT = 1, TEXT = 3, CDATA = 4;
21
+
22
+ function norm(s) {
23
+ if (!s) return "";
24
+ return s.normalize("NFKC").replace(/\s+/g, " ").trim();
25
+ }
26
+ function cut(s, n) {
27
+ const cps = Array.from(s);
28
+ return cps.length <= n ? s.trimEnd() : cps.slice(0, n).join("").trimEnd();
29
+ }
30
+ function words(s) {
31
+ if (!s) return "";
32
+ s = s.normalize("NFKC");
33
+ s = s.replace(/([a-z])([A-Z])/g, "$1 $2");
34
+ s = s.replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2");
35
+ s = s.replace(/([A-Za-z])([0-9])/g, "$1 $2");
36
+ s = s.replace(/([0-9])([A-Za-z])/g, "$1 $2");
37
+ s = s.replace(/[^\p{L}\p{N}\p{M}]/gu, " ");
38
+ return s.replace(/\s+/g, " ").trim().toLowerCase();
39
+ }
40
+ function tag(el) { return el && el.nodeType === ELEMENT ? el.localName.toLowerCase() : ""; }
41
+ function attr(el, name) { const v = el.getAttribute(name); return v === null ? "" : v; }
42
+ function has(el, name) { return el.hasAttribute(name); }
43
+ function unquote(p) { try { return decodeURIComponent(p); } catch (e) { return p; } }
44
+
45
+ function text(el) {
46
+ const parts = [];
47
+ (function walk(e) {
48
+ if (NO_TEXT_TAGS.has(tag(e))) return;
49
+ for (let c = e.firstChild; c; c = c.nextSibling) {
50
+ if (c.nodeType === ELEMENT) walk(c);
51
+ else if (c.nodeType === TEXT || c.nodeType === CDATA) parts.push(c.data);
52
+ }
53
+ })(el);
54
+ return norm(parts.join(" "));
55
+ }
56
+ function inputType(el) {
57
+ const t = tag(el);
58
+ if (t === "input") return attr(el, "type").trim().toLowerCase() || "text";
59
+ return t;
60
+ }
61
+ function hiddenAttr(el) {
62
+ if (has(el, "hidden") || attr(el, "aria-hidden").trim().toLowerCase() === "true") return true;
63
+ const style = attr(el, "style").replace(/ /g, "").toLowerCase();
64
+ return style.includes("display:none") || style.includes("visibility:hidden");
65
+ }
66
+ function isField(el) {
67
+ const t = tag(el);
68
+ if (!FIELD_TAGS.has(t)) return false;
69
+ if (t === "input" && SKIP_INPUT_TYPES.has(inputType(el))) return false;
70
+ return !has(el, "disabled");
71
+ }
72
+ function visibleWithin(el, root) {
73
+ for (let n = el; n; n = n.parentElement) {
74
+ if (hiddenAttr(n)) return false;
75
+ if (n === root) return true;
76
+ }
77
+ return true;
78
+ }
79
+ function containsField(el) {
80
+ return FIELD_TAGS.has(tag(el)) || !!el.querySelector("input,select,textarea");
81
+ }
82
+
83
+ // Previous / next siblings in DOM order, nearest first: ["t", string] or ["e", element]
84
+ function* prevNodes(el) {
85
+ for (let n = el.previousSibling; n; n = n.previousSibling) {
86
+ if (n.nodeType === TEXT || n.nodeType === CDATA) yield ["t", n.data];
87
+ else if (n.nodeType === ELEMENT) yield ["e", n];
88
+ }
89
+ }
90
+ function* nextNodes(el) {
91
+ for (let n = el.nextSibling; n; n = n.nextSibling) {
92
+ if (n.nodeType === TEXT || n.nodeType === CDATA) yield ["t", n.data];
93
+ else if (n.nodeType === ELEMENT) yield ["e", n];
94
+ }
95
+ }
96
+ function nearText(el, root) {
97
+ let node = el;
98
+ for (let level = 0; level < 3; level++) {
99
+ let seen = 0;
100
+ for (const [kind, v] of prevNodes(node)) {
101
+ let t;
102
+ if (kind === "t") t = norm(v);
103
+ else { if (containsField(v)) return ""; t = text(v); }
104
+ if (t) return t;
105
+ if (kind === "e") { seen++; if (seen >= 3) break; }
106
+ }
107
+ if (node === root) break;
108
+ node = node.parentElement;
109
+ if (!node || node === root) break;
110
+ }
111
+ return "";
112
+ }
113
+ function afterText(el) {
114
+ let seen = 0;
115
+ for (const [kind, v] of nextNodes(el)) {
116
+ let t;
117
+ if (kind === "t") t = norm(v);
118
+ else { if (containsField(v)) return ""; t = text(v); }
119
+ if (t) return t;
120
+ if (kind === "e") { seen++; if (seen >= 2) break; }
121
+ }
122
+ return "";
123
+ }
124
+
125
+ function makePage(doc, url) {
126
+ const byId = new Map(), labelFor = new Map();
127
+ for (const e of doc.getElementsByTagName("*")) {
128
+ const i = e.getAttribute("id");
129
+ if (i && !byId.has(i)) byId.set(i, e);
130
+ if (tag(e) === "label" && e.getAttribute("for")) {
131
+ const f = e.getAttribute("for");
132
+ if (!labelFor.has(f)) labelFor.set(f, []);
133
+ labelFor.get(f).push(e);
134
+ }
135
+ }
136
+ const t = doc.querySelector("title");
137
+ return { doc, url, byId, labelFor, title: t ? norm(t.textContent) : "" };
138
+ }
139
+ function labelText(el, page) {
140
+ const out = [];
141
+ for (const ref of attr(el, "aria-labelledby").split(/\s+/).filter(Boolean)) {
142
+ const e = page.byId.get(ref);
143
+ if (e) out.push(text(e));
144
+ }
145
+ const i = el.getAttribute("id");
146
+ if (i) for (const lab of page.labelFor.get(i) || []) out.push(text(lab));
147
+ for (let n = el.parentElement; n; n = n.parentElement) {
148
+ if (tag(n) === "label") { out.push(text(n)); break; }
149
+ }
150
+ const uniq = [];
151
+ for (const t of out) if (t && !uniq.includes(t)) uniq.push(t);
152
+ return uniq.join(" | ");
153
+ }
154
+
155
+ function isButtonish(e) {
156
+ const t = tag(e);
157
+ if (t === "button") return true;
158
+ if (t === "input" && ["submit", "button", "image"].includes(inputType(e))) return true;
159
+ return attr(e, "role").toLowerCase() === "button";
160
+ }
161
+ function isHeading(e) { return HEADING_TAGS.has(tag(e)) || attr(e, "role").toLowerCase() === "heading"; }
162
+ function* iterSelf(root) { yield root; yield* root.getElementsByTagName("*"); }
163
+
164
+ function heading(root) {
165
+ for (const e of iterSelf(root)) {
166
+ if (isHeading(e)) { const t = text(e); if (t) return t; }
167
+ }
168
+ let node = root;
169
+ for (let level = 0; level < 4; level++) {
170
+ let n = 0;
171
+ for (const [kind, sib] of prevNodes(node)) {
172
+ if (kind !== "e") continue;
173
+ n++;
174
+ if (n > 5) break;
175
+ if (isHeading(sib)) { const t = text(sib); if (t) return t; }
176
+ const hs = Array.from(sib.getElementsByTagName("*")).filter(isHeading);
177
+ for (let k = hs.length - 1; k >= 0; k--) { const t = text(hs[k]); if (t) return t; }
178
+ }
179
+ node = node.parentElement;
180
+ if (!node || tag(node) === "body" || tag(node) === "html") break;
181
+ }
182
+ return "";
183
+ }
184
+ function buttonTexts(root) {
185
+ const out = [];
186
+ for (const e of iterSelf(root)) {
187
+ if (!isButtonish(e)) continue;
188
+ const t = text(e) || norm(attr(e, "value")) || norm(attr(e, "aria-label")) || norm(attr(e, "alt")) || norm(attr(e, "title"));
189
+ if (t && !out.includes(t)) out.push(cut(t, 30));
190
+ if (out.length >= 3) break;
191
+ }
192
+ return out.join(" | ");
193
+ }
194
+ function linkTexts(root) {
195
+ const out = [];
196
+ for (const e of iterSelf(root)) {
197
+ if (tag(e) !== "a") continue;
198
+ const t = text(e) || norm(attr(e, "aria-label")) || norm(attr(e, "title"));
199
+ if (t && !out.includes(t)) out.push(cut(t, 30));
200
+ if (out.length >= 4) break;
201
+ }
202
+ return out.join(" | ");
203
+ }
204
+ function hiddenNames(root) {
205
+ const out = [];
206
+ for (const e of iterSelf(root)) {
207
+ if (tag(e) === "input" && inputType(e) === "hidden") {
208
+ const w = words(attr(e, "name"));
209
+ if (w && !out.includes(w)) out.push(w);
210
+ if (out.length >= 4) break;
211
+ }
212
+ }
213
+ return out.join(" ");
214
+ }
215
+ function urlPath(u, base) {
216
+ try { const x = base ? new URL(u, base) : new URL(u); return { host: x.hostname, path: x.pathname }; } catch (e) { return null; }
217
+ }
218
+ function urlWords(url) {
219
+ const u = urlPath(url);
220
+ if (!u) return "";
221
+ const host = u.host.split(".");
222
+ const lead = host.length >= 3 && host[0] !== "www" ? host[0] : "";
223
+ return words(lead + " " + unquote(u.path));
224
+ }
225
+ function formWords(root) {
226
+ const parts = [attr(root, "id"), attr(root, "name"), attr(root, "class")];
227
+ if (tag(root) === "form") {
228
+ const a = attr(root, "action");
229
+ const u = a ? urlPath(a, "http://x/") : null;
230
+ parts.push(u ? unquote(u.path) : "");
231
+ } else parts.push(attr(root, "role"));
232
+ return (words(parts.join(" ")) + " " + norm(attr(root, "aria-label")).toLowerCase()).trim();
233
+ }
234
+
235
+ function fieldFeatures(el, root, page) {
236
+ const t = tag(el);
237
+ const f = { type: t !== "input" ? t : "input " + inputType(el) };
238
+ f.name = words(attr(el, "name"));
239
+ f.id = words(attr(el, "id"));
240
+ f.class = words(attr(el, "class"));
241
+ f.ac = norm(attr(el, "autocomplete")).toLowerCase();
242
+ f.ph = norm(attr(el, "placeholder"));
243
+ f.lbl = labelText(el, page);
244
+ f.aria = norm(attr(el, "aria-label"));
245
+ f.tip = norm(attr(el, "title"));
246
+ const near = nearText(el, root);
247
+ f.near = near === f.lbl || near === f.ph || near === f.aria ? "" : near;
248
+ f.after = afterText(el);
249
+ f.mode = norm(attr(el, "inputmode")).toLowerCase();
250
+ f.max = norm(attr(el, "maxlength"));
251
+ f.req = has(el, "required") || attr(el, "aria-required").toLowerCase() === "true";
252
+ if (t === "select") {
253
+ const opts = Array.from(el.getElementsByTagName("option")).map((o) => norm(o.textContent)).filter(Boolean);
254
+ f.opt = opts.length ? opts.length + ": " + opts.slice(0, 6).join(" | ") : "";
255
+ } else f.opt = "";
256
+ const data = DATA_ATTRS.filter((a) => attr(el, a).trim()).map((a) => norm(attr(el, a)));
257
+ f.data = words(data.slice(0, 2).join(" "));
258
+ for (const [k, n] of Object.entries(FIELD_LIMITS)) if (typeof f[k] === "string") f[k] = cut(f[k], n);
259
+ return f;
260
+ }
261
+ function formContext(root, page) {
262
+ const c = { title: page.title, url: urlWords(page.url), form: formWords(root), head: heading(root),
263
+ btn: buttonTexts(root), link: linkTexts(root), txt: text(root), hid: hiddenNames(root) };
264
+ for (const k of Object.keys(c)) c[k] = cut(c[k], CTX_LIMITS[k]);
265
+ return c;
266
+ }
267
+ function groupRoot(el, body) {
268
+ let node = el.parentElement;
269
+ for (let level = 0; level < 6; level++) {
270
+ if (!node || node === body || tag(node) === "body" || tag(node) === "html") break;
271
+ for (const d of iterSelf(node)) {
272
+ if (isButtonish(d) && !d.closest("form")) return node;
273
+ }
274
+ node = node.parentElement;
275
+ }
276
+ node = el;
277
+ for (let level = 0; level < 3; level++) {
278
+ const p = node.parentElement;
279
+ if (!p || p === body || tag(p) === "body" || tag(p) === "html") break;
280
+ node = p;
281
+ }
282
+ return node;
283
+ }
284
+ function findForms(doc) {
285
+ const body = doc.body || doc.documentElement;
286
+ const groups = new Map();
287
+ for (const el of body.getElementsByTagName("*")) {
288
+ if (!isField(el)) continue;
289
+ let root = el.parentElement ? el.parentElement.closest("form") : null;
290
+ if (!root) root = groupRoot(el, body);
291
+ if (!visibleWithin(el, root)) continue;
292
+ if (!groups.has(root)) groups.set(root, []);
293
+ const g = groups.get(root);
294
+ if (g.length < MAX_FIELDS) g.push(el);
295
+ }
296
+ return Array.from(groups.entries()).slice(0, MAX_FORMS);
297
+ }
298
+
299
+ function serializeCtx(ctx) {
300
+ return CTX_ORDER.filter(([k]) => ctx[k]).map(([k, key]) => key + " " + ctx[k]).join(" ");
301
+ }
302
+ function serializeField(f) {
303
+ const out = ["[FLD]"];
304
+ for (const [k, key] of FIELD_ORDER) {
305
+ const v = f[k];
306
+ if (k === "req") { if (v) out.push("[REQ]"); } else if (v) out.push(key + " " + v);
307
+ }
308
+ return out.join(" ");
309
+ }
310
+ function serialize(ctx, fields) {
311
+ return [serializeCtx(ctx)].concat(fields.map(serializeField)).join(" ");
312
+ }
313
+
314
+ function zapExtract(doc, url) {
315
+ const page = makePage(doc, url);
316
+ return findForms(doc).map(([root, elements]) => {
317
+ const fields = elements.map((el) => fieldFeatures(el, root, page));
318
+ const ctx = formContext(root, page);
319
+ return { kind: tag(root) === "form" ? "form" : "group", root, elements, ctx, fields, text: serialize(ctx, fields) };
320
+ });
321
+ }
322
+
323
+ const api = { zapExtract, serialize, serializeCtx, serializeField, words, norm };
324
+ if (typeof module !== "undefined" && module.exports) module.exports = api;
325
+ else global.Zap = Object.assign(global.Zap || {}, api);
326
+ })(typeof globalThis !== "undefined" ? globalThis : this);
training/js/zap-infer.js ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Zap inference in JavaScript (browser extension / Node) with transformers.js.
2
+ // import * as transformers from "@huggingface/transformers";
3
+ // const zap = await ZapModel.load(transformers, "ProCreations/Zap");
4
+ // const results = await zap.classify(Zap.zapExtract(document, location.href));
5
+ // Mirrors zap_infer.py: [CLS] + form context + as many whole fields as fit in 512 tokens + [SEP] per window;
6
+ // form = mean of [CLS] form logits over windows, each field = its [FLD] token restricted to field labels.
7
+ (function (global) {
8
+ "use strict";
9
+ const MAX_LEN = 512;
10
+ const extract = () => (global.Zap || require("./zap-extract.js"));
11
+
12
+ class ZapModel {
13
+ constructor(tf, tok, model) {
14
+ this.Tensor = tf.Tensor;
15
+ this.tok = tok;
16
+ this.model = model;
17
+ const id2label = model.config.id2label;
18
+ this.labels = Object.keys(id2label).sort((a, b) => a - b).map((k) => id2label[k]);
19
+ this.formIdx = this.labels.map((l, i) => (l.startsWith("form:") ? i : -1)).filter((i) => i >= 0);
20
+ this.fieldIdx = this.labels.map((l, i) => (l.startsWith("field:") ? i : -1)).filter((i) => i >= 0);
21
+ }
22
+ static async load(tf, path = "ProCreations/Zap", opts = {}) {
23
+ return new ZapModel(tf, await tf.AutoTokenizer.from_pretrained(path),
24
+ await tf.AutoModelForTokenClassification.from_pretrained(path, { dtype: "fp32", ...opts }));
25
+ }
26
+ ids(text) {
27
+ return Array.from(this.tok.encode(text, { add_special_tokens: false }));
28
+ }
29
+ windows(form) {
30
+ const { serializeCtx, serializeField } = extract();
31
+ const ctx = this.ids(serializeCtx(form.ctx)).slice(0, 160);
32
+ const fields = form.fields.map((f) => this.ids(serializeField(f)).slice(0, 120));
33
+ const cls = this.tok.cls_token_id ?? this.tok.model.tokens_to_ids.get("[CLS]");
34
+ const sep = this.tok.sep_token_id ?? this.tok.model.tokens_to_ids.get("[SEP]");
35
+ const budget = MAX_LEN - 2 - ctx.length;
36
+ const groups = [];
37
+ let cur = [], used = 0;
38
+ fields.forEach((f, i) => {
39
+ if (cur.length && used + f.length > budget) { groups.push(cur); cur = []; used = 0; }
40
+ cur.push(i); used += f.length;
41
+ });
42
+ if (cur.length || !groups.length) groups.push(cur);
43
+ return groups.map((idx) => {
44
+ const ids = [cls, ...ctx], pos = [];
45
+ for (const i of idx) { pos.push(ids.length); ids.push(...fields[i]); }
46
+ ids.push(sep);
47
+ return { ids, pos, idx };
48
+ });
49
+ }
50
+ softmax(row, idx) {
51
+ const v = idx.map((i) => row[i]);
52
+ const m = Math.max(...v);
53
+ const e = v.map((x) => Math.exp(x - m));
54
+ const s = e.reduce((a, b) => a + b, 0);
55
+ return e.map((x) => x / s);
56
+ }
57
+ async classify(forms) {
58
+ const Tensor = this.Tensor;
59
+ const out = [];
60
+ for (const form of forms) {
61
+ const wins = this.windows(form);
62
+ const n = Math.max(...wins.map((w) => w.ids.length));
63
+ const B = wins.length, L = this.labels.length;
64
+ const ids = new BigInt64Array(B * n), mask = new BigInt64Array(B * n);
65
+ wins.forEach((w, b) => w.ids.forEach((t, j) => { ids[b * n + j] = BigInt(t); mask[b * n + j] = 1n; }));
66
+ const { logits } = await this.model({ input_ids: new Tensor("int64", ids, [B, n]), attention_mask: new Tensor("int64", mask, [B, n]) });
67
+ const data = logits.data;
68
+ const row = (b, p) => data.subarray((b * n + p) * L, (b * n + p + 1) * L);
69
+ const formLogits = new Array(this.formIdx.length).fill(0);
70
+ const fields = new Array(form.fields.length);
71
+ wins.forEach((w, b) => {
72
+ const r0 = row(b, 0);
73
+ this.formIdx.forEach((li, k) => { formLogits[k] += r0[li] / B; });
74
+ w.pos.forEach((p, k) => {
75
+ const pr = this.softmax(row(b, p), this.fieldIdx);
76
+ const top = pr.indexOf(Math.max(...pr));
77
+ fields[w.idx[k]] = { label: this.labels[this.fieldIdx[top]].slice(6), score: pr[top], element: form.elements && form.elements[w.idx[k]] };
78
+ });
79
+ });
80
+ const fp = this.softmax(formLogits, formLogits.map((_, i) => i));
81
+ const ft = fp.indexOf(Math.max(...fp));
82
+ out.push({ form: this.labels[this.formIdx[ft]].slice(5), formScore: fp[ft], fields });
83
+ }
84
+ return out;
85
+ }
86
+ }
87
+ if (typeof module !== "undefined" && module.exports) module.exports = { ZapModel };
88
+ else global.ZapModel = ZapModel;
89
+ })(typeof globalThis !== "undefined" ? globalThis : this);
training/label/common.py ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Shared helpers: autocomplete-token mapping, uids, loading harvested forms."""
2
+ import glob
3
+ import gzip
4
+ import hashlib
5
+ import json
6
+
7
+ AC_MAP = {
8
+ "username": "username", "email": "email", "current-password": "current-password",
9
+ "new-password": "new-password", "one-time-code": "one-time-code",
10
+ "tel": "tel", "tel-national": "tel", "tel-local": "tel", "tel-country-code": "tel", "tel-area-code": "tel",
11
+ "tel-extension": "tel", "tel-local-prefix": "tel", "tel-local-suffix": "tel",
12
+ "name": "name", "given-name": "given-name", "additional-name": "given-name", "family-name": "family-name",
13
+ "organization": "organization", "street-address": "street-address", "address-line1": "street-address",
14
+ "address-line2": "address-line2", "address-line3": "address-line2", "address-level2": "address-level2",
15
+ "address-level1": "address-level1", "postal-code": "postal-code", "country": "country", "country-name": "country",
16
+ "cc-name": "cc-name", "cc-given-name": "cc-name", "cc-family-name": "cc-name", "cc-number": "cc-number",
17
+ "cc-exp": "cc-exp", "cc-exp-month": "cc-exp-month", "cc-exp-year": "cc-exp-year", "cc-csc": "cc-csc",
18
+ "bday": "bday", "bday-day": "bday", "bday-month": "bday", "bday-year": "bday",
19
+ }
20
+ AC_SKIP = {"section-", "shipping", "billing", "home", "work", "mobile", "fax", "pager", "webauthn"}
21
+
22
+
23
+ def ac_label(ac: str):
24
+ """Map an autocomplete attribute to a Zap field label, or None when it carries no field meaning."""
25
+ toks = [t for t in (ac or "").lower().replace('"', "").split() if t]
26
+ toks = [t for t in toks if not t.startswith("section-") and t not in AC_SKIP]
27
+ if len(toks) != 1:
28
+ return None
29
+ return AC_MAP.get(toks[0])
30
+
31
+
32
+ def uid(row) -> str:
33
+ return hashlib.sha1(f"{row['url']}#{row['fidx']}".encode()).hexdigest()[:16]
34
+
35
+
36
+ def load_forms(pattern):
37
+ for path in sorted(glob.glob(pattern)):
38
+ with gzip.open(path, "rt") as f:
39
+ for line in f:
40
+ row = json.loads(line)
41
+ row["uid"] = uid(row)
42
+ yield row
training/label/eval_calib.py ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Score teacher labels against developer autocomplete tokens (the teacher may or may not have seen them).
2
+
3
+ Usage: python eval_calib.py CALIB.jsonl TEACHER.jsonl [N_EXAMPLES]
4
+ """
5
+ import json
6
+ import sys
7
+ from collections import Counter, defaultdict
8
+
9
+ sys.path.insert(0, "/job/work")
10
+ from label.common import ac_label # noqa: E402
11
+ from zap.extract import serialize_field # noqa: E402
12
+
13
+ # Format-based semantics: an identifier field with autocomplete=username may legitimately be email/tel.
14
+ ALSO_OK = {"username": {"email", "tel"}}
15
+
16
+ forms = {json.loads(l)["uid"]: json.loads(l) for l in open(sys.argv[1])}
17
+ teach = {}
18
+ for l in open(sys.argv[2]):
19
+ j = json.loads(l)
20
+ teach[j["uid"]] = j
21
+ n_ex = int(sys.argv[3]) if len(sys.argv) > 3 else 25
22
+ per = defaultdict(Counter)
23
+ conf = Counter()
24
+ bad = []
25
+ fail = 0
26
+ for uid, f in forms.items():
27
+ t = teach.get(uid)
28
+ if not t or not t["fields"]:
29
+ fail += 1
30
+ continue
31
+ for fld, tl in zip(f["fields"], t["fields"]):
32
+ g = ac_label(fld["ac"])
33
+ if not g:
34
+ continue
35
+ ok = tl == g or tl in ALSO_OK.get(g, ())
36
+ per[g]["ok" if ok else "bad"] += 1
37
+ if not ok:
38
+ conf[(g, tl)] += 1
39
+ bad.append((g, tl, t["form"], f["url"], serialize_field(fld)))
40
+ tot_ok = sum(c["ok"] for c in per.values())
41
+ tot = sum(c["ok"] + c["bad"] for c in per.values())
42
+ print(f"teacher failures {fail}/{len(forms)}; autocomplete agreement {tot_ok}/{tot} = {tot_ok / max(tot, 1):.3f}")
43
+ for g, c in sorted(per.items(), key=lambda x: -(x[1]["ok"] + x[1]["bad"])):
44
+ print(f" {g:18s} {c['ok']:4d}/{c['ok'] + c['bad']:4d}")
45
+ print("confusions:", conf.most_common(15))
46
+ for b in bad[:n_ex]:
47
+ print(" AC", b[0], "-> T", b[1], "| form", b[2], "|", b[3][:80], "|", b[4][:200])
48
+ print("form labels:", Counter(t["form"] for t in teach.values()).most_common())
training/label/prep_calib.py ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Calibration set: deduped pilot forms with >=1 mappable autocomplete token."""
2
+ import json, sys
3
+ sys.path.insert(0, "/job/work")
4
+ from label.common import load_forms, ac_label
5
+ seen, out = set(), []
6
+ for r in load_forms(sys.argv[1]):
7
+ if not r.get("html") or r["sig"] in seen:
8
+ continue
9
+ seen.add(r["sig"])
10
+ if any(ac_label(f["ac"]) for f in r["fields"]):
11
+ out.append(r)
12
+ with open(sys.argv[2], "w") as f:
13
+ for r in out:
14
+ f.write(json.dumps(r, ensure_ascii=False) + "\n")
15
+ print(len(out), "calibration forms")
training/label/select.py ADDED
@@ -0,0 +1,126 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Dedup harvested forms, assign host-level splits, and build prioritized teacher-labeling chunks.
2
+
3
+ Usage: python select.py OUT_DIR PATTERN [PATTERN ...] [--labeled DIR ...]
4
+ Signatures already labeled in a previous round (DIR/queue-*.jsonl + DIR/labels-*.jsonl) are not queued again.
5
+ Writes OUT_DIR/queue-NNN.jsonl (teacher input, 10k rows each, interleaved priorities),
6
+ OUT_DIR/reps.jsonl.gz (one representative per signature, with split), OUT_DIR/select_stats.json.
7
+ """
8
+ import gzip
9
+ import hashlib
10
+ import json
11
+ import os
12
+ import random
13
+ import sys
14
+ from collections import Counter
15
+
16
+ sys.path.insert(0, "/job/work")
17
+ from label.common import load_forms # noqa: E402
18
+
19
+ SEARCHY = {"q", "s", "search", "query", "keyword", "keywords", "search query", "k", "term", "search term"}
20
+ MIX = [(0, 0.40), (1, 0.25), (2, 0.17), (3, 0.12), (4, 0.06)]
21
+ CAPS = {3: 60000, 4: 20000}
22
+
23
+
24
+ def split_of(host):
25
+ h = int(hashlib.md5(host.encode()).hexdigest()[:8], 16) % 100
26
+ return "test" if h < 4 else ("val" if h < 6 else "train")
27
+
28
+
29
+ def prio(r):
30
+ fs = r["fields"]
31
+ types = [f["type"] for f in fs]
32
+ if "input password" in types:
33
+ return 0
34
+ if len(fs) >= 3:
35
+ return 1
36
+ if len(fs) == 2 or any(t in ("input email", "input tel") for t in types):
37
+ return 2
38
+ f = fs[0]
39
+ if f["type"] == "input search" or f["name"] in SEARCHY or f["id"] in SEARCHY:
40
+ return 4
41
+ return 3
42
+
43
+
44
+ def labeled_sigs(dirs):
45
+ import glob
46
+ out = {}
47
+ for d in dirs:
48
+ uid2sig = {}
49
+ for q in glob.glob(os.path.join(d, "queue-*.jsonl")):
50
+ for line in open(q):
51
+ j = json.loads(line)
52
+ uid2sig[j["uid"]] = j["sig"]
53
+ for lf in glob.glob(os.path.join(d, "labels-*.jsonl")):
54
+ for line in open(lf):
55
+ try:
56
+ j = json.loads(line)
57
+ except json.JSONDecodeError: # partial last line of an interrupted run
58
+ continue
59
+ if j.get("form") and j.get("fields") and j["uid"] in uid2sig:
60
+ out[uid2sig[j["uid"]]] = j
61
+ return out
62
+
63
+
64
+ def main(out_dir, patterns, labeled_dirs=()):
65
+ os.makedirs(out_dir, exist_ok=True)
66
+ done = labeled_sigs(labeled_dirs)
67
+ print(f"{len(done)} signatures already labeled", flush=True)
68
+ reps, copies, seen = {}, Counter(), set()
69
+ n = 0
70
+ for pat in patterns:
71
+ for r in load_forms(pat):
72
+ n += 1
73
+ if r["uid"] in seen:
74
+ continue
75
+ seen.add(r["uid"])
76
+ copies[r["sig"]] += 1
77
+ cur = reps.get(r["sig"])
78
+ if cur is None or (not cur.get("html") and r.get("html")):
79
+ reps[r["sig"]] = r
80
+ print(f"{n} rows, {len(seen)} unique uids, {len(reps)} unique signatures", flush=True)
81
+ rng = random.Random(13)
82
+ pools = {p: [] for p, _ in MIX}
83
+ with gzip.open(os.path.join(out_dir, "reps.jsonl.gz"), "wt") as g:
84
+ for sig, r in reps.items():
85
+ r["split"] = split_of(r["host"])
86
+ r["copies"] = copies[sig]
87
+ r["prio"] = prio(r)
88
+ g.write(json.dumps(r, ensure_ascii=False) + "\n")
89
+ if r.get("html") and sig not in done:
90
+ pools[r["prio"]].append(r)
91
+ for p in pools:
92
+ rng.shuffle(pools[p])
93
+ if p in CAPS:
94
+ pools[p] = pools[p][:CAPS[p]]
95
+ stats = {"rows": n, "uids": len(seen), "sigs": len(reps), "pools": {p: len(v) for p, v in pools.items()},
96
+ "splits": Counter(split_of(r["host"]) for r in reps.values())}
97
+ print(stats, flush=True)
98
+ order, idx = [], {p: 0 for p in pools}
99
+ while any(idx[p] < len(pools[p]) for p in pools):
100
+ x, acc = rng.random(), 0.0
101
+ live = [(p, w) for p, w in MIX if idx[p] < len(pools[p])]
102
+ tot = sum(w for _, w in live)
103
+ for p, w in live:
104
+ acc += w / tot
105
+ if x <= acc:
106
+ order.append(pools[p][idx[p]])
107
+ idx[p] += 1
108
+ break
109
+ keep = ("uid", "url", "host", "ctx", "fields", "html", "sig", "split", "prio")
110
+ for c in range(0, len(order), 10000):
111
+ with open(os.path.join(out_dir, f"queue-{c // 10000:03d}.jsonl"), "w") as f:
112
+ for r in order[c:c + 10000]:
113
+ f.write(json.dumps({k: r[k] for k in keep}, ensure_ascii=False) + "\n")
114
+ stats["queued"] = len(order)
115
+ json.dump(stats, open(os.path.join(out_dir, "select_stats.json"), "w"), indent=1)
116
+ print("queued", len(order), flush=True)
117
+
118
+
119
+ if __name__ == "__main__":
120
+ import argparse
121
+ ap = argparse.ArgumentParser()
122
+ ap.add_argument("out")
123
+ ap.add_argument("patterns", nargs="+")
124
+ ap.add_argument("--labeled", action="append", default=[])
125
+ a = ap.parse_args()
126
+ main(a.out, a.patterns, a.labeled)
training/label/select_rare.py ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Round 3 selection: unlabeled real forms whose fields look like rare classes (card payment, OTP, address parts).
2
+
3
+ Usage: python select_rare.py REPS_DIR OUT_DIR --labeled DIR [--labeled DIR ...]
4
+ """
5
+ import argparse
6
+ import gzip
7
+ import json
8
+ import os
9
+ import random
10
+ import re
11
+ import sys
12
+
13
+ sys.path.insert(0, "/job/work")
14
+ from label.select import labeled_sigs # noqa: E402
15
+
16
+ POOLS = {
17
+ "payment": (re.compile(r"card.?(number|no\b|num)|\bcc.?(num|number|exp|cvv|csc|name)|\bcvv|\bcvc|\bcsc\b|expir|exp.?(date|month|year)|"
18
+ r"kartennummer|tarjeta|carte|cartão|cardholder|name.?on.?card|security.?code|card.?code|kreditkart"), 20000),
19
+ "otp": (re.compile(r"\botp\b|one.?time|verification.?code|verify.?code|sms.?code|\b2fa\b|\bmfa\b|totp|authenticat\w*.?code|"
20
+ r"passcode|auth.?code|confirmation.?code|code.?(sent|sms|e.?mail)|pin.?code|digit|認証コード|验证码|код"), 20000),
21
+ "address": (re.compile(r"address.?(line)?.?[23]|\bapt\b|apartment|suite|\bstate\b|province|region|county|\bcity\b|town|\bzip|postal|"
22
+ r"postcode|\bplz\b|street|house.?(number|no)|hausnummer|ciudad|\bcep\b|district|stadt|ville|provincia"), 16000),
23
+ }
24
+
25
+
26
+ def field_text(f):
27
+ return " ".join(str(f.get(k, "")) for k in ("name", "id", "ac", "ph", "lbl", "aria", "near", "data")).lower()
28
+
29
+
30
+ def main(a):
31
+ os.makedirs(a.out, exist_ok=True)
32
+ done = labeled_sigs(a.labeled)
33
+ rng = random.Random(21)
34
+ picked = {p: [] for p in POOLS}
35
+ with gzip.open(os.path.join(a.reps, "reps.jsonl.gz"), "rt") as f:
36
+ for line in f:
37
+ r = json.loads(line)
38
+ if r["sig"] in done or not r.get("html"):
39
+ continue
40
+ txt = " ".join(field_text(x) for x in r["fields"])
41
+ for p, (rx, _) in POOLS.items():
42
+ if rx.search(txt):
43
+ picked[p].append(r)
44
+ break
45
+ keep = ("uid", "url", "host", "ctx", "fields", "html", "sig", "split", "prio")
46
+ rows = []
47
+ for p, (_, cap) in POOLS.items():
48
+ rng.shuffle(picked[p])
49
+ rows += picked[p][:cap]
50
+ print(p, len(picked[p]), "->", min(cap, len(picked[p])), flush=True)
51
+ rng.shuffle(rows)
52
+ for c in range(0, len(rows), 10000):
53
+ with open(os.path.join(a.out, f"queue-{c // 10000:03d}.jsonl"), "w") as f:
54
+ for r in rows[c:c + 10000]:
55
+ f.write(json.dumps({k: r[k] for k in keep}, ensure_ascii=False) + "\n")
56
+ print("queued", len(rows), flush=True)
57
+
58
+
59
+ if __name__ == "__main__":
60
+ ap = argparse.ArgumentParser()
61
+ ap.add_argument("reps")
62
+ ap.add_argument("out")
63
+ ap.add_argument("--labeled", action="append", default=[])
64
+ main(ap.parse_args())
training/label/stats.py ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import sys, collections
2
+ sys.path.insert(0, "/job/work")
3
+ from label.common import load_forms
4
+ C = collections.Counter
5
+ rows = list(load_forms(sys.argv[1]))
6
+ sigs = C(r["sig"] for r in rows)
7
+ hosts = C(r["host"] for r in rows)
8
+ print("forms", len(rows), "unique sigs", len(sigs), "hosts", len(hosts))
9
+ uniq = {}
10
+ for r in rows:
11
+ uniq.setdefault(r["sig"], r)
12
+ u = list(uniq.values())
13
+ pw = [r for r in u if any(f["type"] == "input password" for f in r["fields"])]
14
+ print("unique with password", len(pw), "html ok", sum(1 for r in u if r.get("html")))
15
+ print("nfields", sorted(C(min(len(r["fields"]), 12) for r in u).items()))
16
+ print("types", C(f["type"] for r in u for f in r["fields"]).most_common(12))
17
+ print("top hosts", hosts.most_common(5))
18
+ single_search = sum(1 for r in u if len(r["fields"]) == 1 and (r["fields"][0]["type"] == "input search" or r["fields"][0]["name"] in ("q", "s", "search", "query", "keyword", "keywords")))
19
+ print("single-field search-like", single_search)
training/label/synth.py ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Generate synthetic forms for classes the static web crawl under-represents (card payment, OTP, password
2
+ change/reset, checkout address, identifier-first logins, SPA div-soup markup), then extract them with the
3
+ standard extractor. They are labeled later by the same teacher pass as real forms.
4
+
5
+ Usage: python synth.py OUT_DIR N [--conc 32] [--seed 5]
6
+ """
7
+ import argparse
8
+ import asyncio
9
+ import gzip
10
+ import json
11
+ import os
12
+ import random
13
+ import re
14
+ import sys
15
+
16
+ import aiohttp
17
+
18
+ sys.path.insert(0, "/job/work")
19
+ from zap.extract import extract_page # noqa: E402
20
+
21
+ SCENARIOS = [
22
+ (8, "checkout payment step with credit/debit card fields (card number, name on card, expiry, CVC)"),
23
+ (4, "checkout page with shipping address AND card payment fields in one form"),
24
+ (4, "card payment form where expiry month and expiry year are two separate <select> dropdowns"),
25
+ (2, "donation form with amount, name, email and card details"),
26
+ (3, "subscription billing form with card number, expiry (MM/YY single field), CVC and billing ZIP"),
27
+ (2, "add a new payment card form in account settings (cardholder name, number, expiry, security code)"),
28
+ (4, "two-factor authentication page asking for a 6-digit code from an authenticator app (one input)"),
29
+ (4, "SMS verification page where the 6-digit code is split into six single-character inputs"),
30
+ (3, "email verification page asking for a code sent by email, with a resend link"),
31
+ (1, "2FA page asking for a backup/recovery code"),
32
+ (4, "change password page for a logged-in user: current password, new password, confirm new password"),
33
+ (3, "reset password page reached from an email link: new password and confirm password only"),
34
+ (2, "forgot password page asking for the account email address"),
35
+ (1, "forgot password page asking for username or email"),
36
+ (3, "identifier-first sign-in step 1: only an email or phone field and a Next button"),
37
+ (3, "sign-in step 2: the username is shown as text and only the password field is present"),
38
+ (1, "login form with phone number and password"),
39
+ (4, "shipping address form: full name, address line 1, address line 2, city, state/province, postal code, country, phone"),
40
+ (2, "billing address form with first name, last name, company, street, house number, city, ZIP, country"),
41
+ (2, "account profile / billing details edit form with company name and VAT field"),
42
+ (1, "login form inside a modal dialog on a shop homepage"),
43
+ (1, "registration form with username, email, password and repeat password"),
44
+ ]
45
+ STYLES = [
46
+ "plain semantic HTML with <form>, <label for> and ids",
47
+ "Bootstrap 5 markup",
48
+ "Tailwind CSS utility classes and no <form> element (a React SPA, fields inside divs, button type=button)",
49
+ "Material UI style: floating labels placed after the input, generated class names like css-1x2y3z",
50
+ "Angular reactive form with formcontrolname attributes and mat-form-field wrappers",
51
+ "Vue app with data-v-xxxx attributes and v-model style names",
52
+ "WooCommerce / WordPress markup",
53
+ "Shopify-like checkout markup with data-testid attributes",
54
+ "legacy table-based layout with <td> labels",
55
+ "Stripe Elements-like structure where each card field is its own div with an input",
56
+ "minimal markup where inputs have only placeholders and no labels",
57
+ "inputs with aria-label only and obfuscated names like field_7f3a2 or input_12",
58
+ ]
59
+ QUIRKS = [
60
+ "no autocomplete attributes at all",
61
+ "autocomplete=\"off\" on every input",
62
+ "correct HTML autocomplete tokens on every input",
63
+ "some inputs use type=\"text\" even for email/phone/number",
64
+ "the password inputs have a show/hide toggle button next to them",
65
+ "field names are in the page language (not English)",
66
+ "a hidden honeypot text input styled off-screen",
67
+ "a CAPTCHA text input near the submit button",
68
+ "inputmode and maxlength attributes where appropriate",
69
+ "",
70
+ ]
71
+ LANGS = ["English"] * 8 + ["Spanish", "French", "German", "Portuguese (Brazil)", "Italian", "Dutch", "Polish",
72
+ "Russian", "Japanese", "Chinese (Simplified)", "Korean", "Turkish", "Arabic", "Hindi",
73
+ "Indonesian", "Vietnamese", "Swedish", "Czech", "Ukrainian", "Greek", "Hebrew", "Thai"]
74
+ SITES = ["online store", "bank", "airline", "streaming service", "SaaS dashboard", "forum", "news site", "university portal",
75
+ "government service", "hotel booking site", "food delivery app", "crypto exchange", "telecom operator",
76
+ "insurance company", "game platform", "email provider", "charity", "gym", "car rental", "marketplace",
77
+ "developer platform", "social network", "utility company", "ticketing site", "healthcare portal"]
78
+
79
+ PROMPT = """Write the HTML of a realistic web page section for this form.
80
+
81
+ Form: {scenario}
82
+ Website: a {site} (invent a plausible brand name)
83
+ Language of all visible text: {lang}
84
+ Markup style: {style}
85
+ Detail: {quirk}
86
+
87
+ Output exactly:
88
+ TITLE: <the page's <title> text>
89
+ URL: <a plausible https URL of the page>
90
+ <the HTML of the page section containing the form: include a heading, the fields, buttons and links; no <html>, <head>, <script> or <style>>"""
91
+
92
+
93
+ async def gen(session, url, prompt, sem):
94
+ body = {"model": "qwen", "messages": [{"role": "user", "content": prompt}], "temperature": 1.0, "top_p": 0.95,
95
+ "max_tokens": 1800, "chat_template_kwargs": {"enable_thinking": False}}
96
+ async with sem:
97
+ for _ in range(3):
98
+ try:
99
+ async with session.post(url, json=body, timeout=aiohttp.ClientTimeout(total=900)) as r:
100
+ j = await r.json()
101
+ return j["choices"][0]["message"]["content"] or ""
102
+ except Exception: # noqa: BLE001
103
+ await asyncio.sleep(3)
104
+ return ""
105
+
106
+
107
+ def parse(text):
108
+ t = re.search(r"TITLE:\s*(.+)", text)
109
+ u = re.search(r"URL:\s*(\S+)", text)
110
+ m = re.search(r"<[a-zA-Z]", text[u.end():] if u else text)
111
+ if not (t and u and m):
112
+ return None
113
+ html = (text[u.end():] if u else text)[m.start():]
114
+ html = re.sub(r"```\s*$", "", html.strip()).strip()
115
+ return t.group(1).strip()[:200], u.group(1).strip(), html
116
+
117
+
118
+ async def main(a):
119
+ os.makedirs(a.out, exist_ok=True)
120
+ rng = random.Random(a.seed)
121
+ weights = [w for w, _ in SCENARIOS]
122
+ jobs = []
123
+ for i in range(a.n):
124
+ sc = rng.choices(SCENARIOS, weights)[0][1]
125
+ jobs.append({"i": i, "scenario": sc, "site": rng.choice(SITES), "lang": rng.choice(LANGS),
126
+ "style": rng.choice(STYLES), "quirk": rng.choice(QUIRKS)})
127
+ url = f"http://127.0.0.1:{a.port}/v1/chat/completions"
128
+ sem = asyncio.Semaphore(a.conc)
129
+ out = gzip.open(os.path.join(a.out, f"part-synth-{a.seed}.jsonl.gz.tmp"), "wt")
130
+ stats = {"ok": 0, "bad": 0, "forms": 0}
131
+ async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=a.conc)) as s:
132
+ async def one(job):
133
+ text = await gen(s, url, PROMPT.format(**job), sem)
134
+ p = parse(text)
135
+ if not p:
136
+ stats["bad"] += 1
137
+ return
138
+ title, page_url, html = p
139
+ doc = f"<html><head><title>{title}</title></head><body>{html}</body></html>"
140
+ forms = extract_page(doc, page_url)
141
+ stats["ok"] += 1
142
+ for k, f in enumerate(forms):
143
+ f.update(url=page_url + f"#synth{job['i']}", host=f"synth,{a.seed},{job['i']}", lang=job["lang"],
144
+ src="synth", fidx=k, synth=job)
145
+ out.write(json.dumps(f, ensure_ascii=False) + "\n")
146
+ stats["forms"] += 1
147
+ if (stats["ok"] + stats["bad"]) % 200 == 0:
148
+ print(stats, flush=True)
149
+ await asyncio.gather(*[one(j) for j in jobs])
150
+ out.close()
151
+ os.rename(os.path.join(a.out, f"part-synth-{a.seed}.jsonl.gz.tmp"), os.path.join(a.out, f"part-synth-{a.seed}.jsonl.gz"))
152
+ print("DONE", stats, flush=True)
153
+
154
+
155
+ if __name__ == "__main__":
156
+ ap = argparse.ArgumentParser()
157
+ ap.add_argument("out")
158
+ ap.add_argument("n", type=int)
159
+ ap.add_argument("--conc", type=int, default=32)
160
+ ap.add_argument("--seed", type=int, default=5)
161
+ ap.add_argument("--port", type=int, default=30020)
162
+ asyncio.run(main(ap.parse_args()))
training/label/teacher.py ADDED
@@ -0,0 +1,191 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Teacher labeling: Qwen3.8-Flash-Next labels every field and the form purpose.
2
+
3
+ Usage: python teacher.py IN.jsonl OUT.jsonl [--conc 128] [--mask-ac] [--port 30020] [--limit N]
4
+ IN rows need: uid, url, ctx.title, html (with data-zap indices), fields.
5
+ """
6
+ import argparse
7
+ import asyncio
8
+ import json
9
+ import re
10
+ import time
11
+
12
+ import aiohttp
13
+
14
+ FIELD_LABELS = ["username", "email", "current-password", "new-password", "one-time-code", "tel", "name",
15
+ "given-name", "family-name", "organization", "street-address", "address-line2", "address-level2",
16
+ "address-level1", "postal-code", "country", "cc-name", "cc-number", "cc-exp", "cc-exp-month",
17
+ "cc-exp-year", "cc-csc", "bday", "search", "captcha", "other"]
18
+ FORM_LABELS = ["login", "signup", "password-change", "password-recovery", "otp", "payment", "address", "newsletter",
19
+ "contact", "search", "other"]
20
+
21
+ SYSTEM = """You label HTML form fields for a password manager's autofill engine.
22
+
23
+ You get one web form: the page URL, the page title, and the cleaned HTML of the form (or of the page region that holds the fields). Every fillable field carries data-zap="N". Give every numbered field exactly one FIELD label, and the whole form one FORM label.
24
+
25
+ FIELD labels (what data goes INTO the field):
26
+ username - account identifier that is not strictly an email address: user name, login name, user ID, member/customer/account number used to sign in, "email or username", "email or phone", "login".
27
+ email - an email address, in any context: login "Email", signup email, newsletter, contact, checkout, "confirm email".
28
+ current-password - the user's EXISTING password or sign-in PIN/passcode: login password, "current/old password" in a change-password form, password re-entry to confirm identity.
29
+ new-password - a password being CREATED or SET: signup password, "new password", "confirm/repeat/retype password" on signup, change or reset forms.
30
+ one-time-code - one-time code: 2FA/OTP, SMS or email verification code, authenticator code, backup/recovery code, activation code (a single box or one of several digit boxes).
31
+ tel - phone/mobile number or part of one (country dialing code, area code, extension).
32
+ name - full name in a single field.
33
+ given-name - first name (also middle name).
34
+ family-name - last name / surname.
35
+ organization - company / organization / business name.
36
+ street-address - street address, address line 1, street name, house number.
37
+ address-line2 - apartment, suite, unit, floor, building, address line 2 or 3.
38
+ address-level2 - city / town / locality.
39
+ address-level1 - state / province / region / county.
40
+ postal-code - ZIP / postal code / postcode.
41
+ country - country or country/region.
42
+ cc-name - name on the payment card.
43
+ cc-number - payment card number.
44
+ cc-exp - card expiry date in one field (MM/YY).
45
+ cc-exp-month - card expiry month only.
46
+ cc-exp-year - card expiry year only.
47
+ cc-csc - card security code (CVV/CVC/CID).
48
+ bday - date of birth or one part of it (birth day/month/year).
49
+ search - site search / search query box.
50
+ captcha - CAPTCHA or anti-spam challenge answer (e.g. "what is 3+4?").
51
+ other - anything else: message/comment, subject, quantity, amount, coupon/promo code, booking dates, gender, website URL, security-question answers, IBAN/tax/ID numbers, nickname/display name on a non-login form, honeypot fields meant to stay empty, filters, sorting, etc.
52
+
53
+ FORM labels (what the whole form is for):
54
+ login - signing in to an existing account, including the "enter your email/username" first step of a multi-step sign-in.
55
+ signup - creating a new account / registration (may also ask for name, address, phone...).
56
+ password-change - setting a new password for an existing account (change-password page, or the reset-link page that asks for the new password).
57
+ password-recovery - requesting a password reset / "forgot password" / "forgot username" (asks only for an identifier).
58
+ otp - entering a one-time / verification / 2FA code.
59
+ payment - entering payment card details (checkout payment, donation by card, billing with card number).
60
+ address - shipping/billing/contact details without a card: checkout address, profile/account details, delivery info.
61
+ newsletter - subscribing an email (or phone) to a newsletter or updates, possibly with a name.
62
+ contact - sending a message: contact form, comment form, feedback, quote/enquiry, support ticket, review.
63
+ search - searching the site (search box, optionally with filters).
64
+ other - everything else: booking/availability search, calculators, filter-only forms, polls, job applications, settings, age gates, language/currency pickers, etc.
65
+
66
+ Rules:
67
+ - Decide from ALL evidence: type, name, id, autocomplete, placeholder, labels, nearby text, buttons, headings, page title and URL. Text can be in any language.
68
+ - autocomplete attributes are strong hints but can be wrong (e.g. "new-password" or "off" set only to block autofill). Follow what the field really asks for.
69
+ - A type="password" field is never username or email. Decide current-password vs new-password from the form's purpose. A masked field for a code sent by SMS/email is one-time-code.
70
+ - On a login form: a field accepting "email or username" (or "email or phone") is username; a field asking only for an email is email; a phone-only field is tel.
71
+ - If a form mixes login and signup, pick the FORM label of the action its submit button performs.
72
+ - If the numbered fields are unrelated bits of a page, use FORM "other" unless one clear purpose exists.
73
+
74
+ Output ONLY this format, no explanations:
75
+ FORM: <form label>
76
+ 1: <field label>
77
+ 2: <field label>
78
+ ..."""
79
+
80
+ AC_RE = re.compile(r'\sautocomplete="[^"]*"')
81
+ LINE_RE = re.compile(r"^\s*(\d+)\s*[:.)-]\s*([a-z0-9\-]+)\s*$")
82
+ FORM_RE = re.compile(r"^\s*FORM\s*:\s*([a-z\-]+)\s*$", re.I)
83
+
84
+
85
+ def build_user(row, mask_ac=False):
86
+ html = row["html"]
87
+ if mask_ac:
88
+ html = AC_RE.sub("", html)
89
+ return f"URL: {row['url']}\nTitle: {row['ctx'].get('title', '')}\nHTML:\n{html}"
90
+
91
+
92
+ def parse(text, n_fields):
93
+ form, fields = None, {}
94
+ for line in (text or "").splitlines():
95
+ m = FORM_RE.match(line)
96
+ if m:
97
+ form = m.group(1).lower()
98
+ continue
99
+ m = LINE_RE.match(line)
100
+ if m:
101
+ fields[int(m.group(1))] = m.group(2).lower()
102
+ if form not in FORM_LABELS:
103
+ return None
104
+ labels = [fields.get(i) for i in range(1, n_fields + 1)]
105
+ if any(l not in FIELD_LABELS for l in labels):
106
+ return None
107
+ return form, labels
108
+
109
+
110
+ async def label_one(session, url, row, args, stats):
111
+ n = len(row["fields"])
112
+ body = {"model": "qwen", "messages": [{"role": "system", "content": SYSTEM},
113
+ {"role": "user", "content": build_user(row, args.mask_ac)}],
114
+ "temperature": 0.0, "max_tokens": 24 + 10 * n,
115
+ "chat_template_kwargs": {"enable_thinking": False}}
116
+ for attempt in range(3):
117
+ try:
118
+ async with session.post(url, json=body, timeout=aiohttp.ClientTimeout(total=600)) as r:
119
+ j = await r.json()
120
+ msg = j["choices"][0]["message"]
121
+ text = msg.get("content") or ""
122
+ usage = j.get("usage", {})
123
+ stats["prompt"] += usage.get("prompt_tokens", 0)
124
+ stats["completion"] += usage.get("completion_tokens", 0)
125
+ p = parse(text, n)
126
+ if p:
127
+ return {"uid": row["uid"], "form": p[0], "fields": p[1], "raw": text}
128
+ stats["parse_fail"] += 1
129
+ body["temperature"] = 0.3
130
+ except Exception as e: # noqa: BLE001
131
+ stats["http_fail"] += 1
132
+ await asyncio.sleep(2 + attempt * 3)
133
+ return {"uid": row["uid"], "form": None, "fields": None, "raw": None}
134
+
135
+
136
+ async def main(args):
137
+ rows = [json.loads(l) for l in open(args.inp)]
138
+ if args.limit:
139
+ rows = rows[:args.limit]
140
+ done = set()
141
+ try:
142
+ for l in open(args.out):
143
+ try:
144
+ done.add(json.loads(l)["uid"])
145
+ except json.JSONDecodeError:
146
+ pass
147
+ except FileNotFoundError:
148
+ pass
149
+ rows = [r for r in rows if r["uid"] not in done]
150
+ print(f"{len(rows)} to label ({len(done)} already done)", flush=True)
151
+ url = f"http://127.0.0.1:{args.port}/v1/chat/completions"
152
+ stats = {"prompt": 0, "completion": 0, "parse_fail": 0, "http_fail": 0, "ok": 0, "bad": 0}
153
+ q: asyncio.Queue = asyncio.Queue()
154
+ for r in rows:
155
+ q.put_nowait(r)
156
+ t0 = time.time()
157
+ out = open(args.out, "a")
158
+
159
+ async def worker(session):
160
+ while True:
161
+ try:
162
+ row = q.get_nowait()
163
+ except asyncio.QueueEmpty:
164
+ return
165
+ res = await label_one(session, url, row, args, stats)
166
+ stats["ok" if res["form"] else "bad"] += 1
167
+ out.write(json.dumps(res, ensure_ascii=False) + "\n")
168
+ n = stats["ok"] + stats["bad"]
169
+ if n % 500 == 0:
170
+ out.flush()
171
+ el = time.time() - t0
172
+ print(f"{n}/{len(rows)} {el:.0f}s {n / el:.1f} forms/s prompt {stats['prompt'] / el:.0f} tok/s "
173
+ f"gen {stats['completion'] / el:.0f} tok/s {stats}", flush=True)
174
+
175
+ conn = aiohttp.TCPConnector(limit=args.conc)
176
+ async with aiohttp.ClientSession(connector=conn) as session:
177
+ await asyncio.gather(*[worker(session) for _ in range(args.conc)])
178
+ out.close()
179
+ el = time.time() - t0
180
+ print(f"DONE {el:.0f}s {stats}", flush=True)
181
+
182
+
183
+ if __name__ == "__main__":
184
+ ap = argparse.ArgumentParser()
185
+ ap.add_argument("inp")
186
+ ap.add_argument("out")
187
+ ap.add_argument("--conc", type=int, default=128)
188
+ ap.add_argument("--mask-ac", action="store_true")
189
+ ap.add_argument("--port", type=int, default=30020)
190
+ ap.add_argument("--limit", type=int, default=0)
191
+ asyncio.run(main(ap.parse_args()))
training/runner/run.sh ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ # Queue runner for the Zap temporary job.
3
+ # Every *.sh dropped into /job/work/queue/ is launched in the background (tasks may run in parallel;
4
+ # a task that needs ordering waits on another task's /job/work/state/NAME.exit file itself).
5
+ # /job/work/queue/STOP -> stop launching, wait for running tasks, exit.
6
+ set -u
7
+ Q=/job/work/queue; S=/job/work/state; L=/job/logs/tasks
8
+ mkdir -p "$Q" "$S" "$L"
9
+ echo "$(date -Is) runner up"
10
+ declare -A PIDS
11
+ while true; do
12
+ for f in $(ls "$Q"/*.sh 2>/dev/null | sort); do
13
+ name=$(basename "$f" .sh)
14
+ mv "$f" "$S/$name.sh"
15
+ ( bash "$S/$name.sh" > "$L/$name.log" 2>&1; echo $? > "$S/$name.exit" ) &
16
+ PIDS[$name]=$!
17
+ echo "$(date -Is) launched $name pid ${PIDS[$name]}"
18
+ done
19
+ for name in "${!PIDS[@]}"; do
20
+ if ! kill -0 "${PIDS[$name]}" 2>/dev/null; then
21
+ echo "$(date -Is) finished $name exit $(cat "$S/$name.exit" 2>/dev/null)"
22
+ unset "PIDS[$name]"
23
+ fi
24
+ done
25
+ if [ -f "$Q/STOP" ] && [ ${#PIDS[@]} -eq 0 ]; then echo "$(date -Is) STOP"; break; fi
26
+ sleep 5
27
+ done
training/runner/t00_setup.sh ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ set -x
2
+ cd /job/work
3
+ python3 -m venv --system-site-packages /job/work/.venv
4
+ . /job/work/.venv/bin/activate
5
+ pip install -q selectolax warcio onnx onnxruntime onnxscript tldextract aiohttp 2>&1 | tail -3
6
+ python3 -c "import selectolax, onnx, onnxruntime, aiohttp, torch; print('ok', torch.cuda.is_available())"
7
+ # compare baked overlays in image vs production overlay files
8
+ P=/home/user/.local/share/rtx-pro-apps/qwen3.8-flash-next/runtime-speed-20260924
9
+ for pair in "qwen_sparse_attn_backend.py:layers/attention/qwen_sparse_attn_backend.py" "qsa_sparse_attn.py:layers/attention/qsa/sparse_attn.py" "qwen4_exp_mtp_hotmap.py:models/qwen4_exp_mtp.py" "eagle_worker_v2.py:speculative/eagle_worker_v2.py" "metadata.py:layers/attention/qsa/metadata.py" "graph_metadata.py:layers/attention/qsa/graph_metadata.py" "qsa_indexer.py:layers/attention/qsa/qsa_indexer.py" "qsa_kv_pool.py:mem_cache/qsa_kv_pool.py" "eagle_utils.py:speculative/eagle_utils.py" "qwenfast.py:qwenfast.py" "qwen4_exp_startup.py:models/qwen4_exp.py" "loader_startup.py:model_loader/loader.py"; do
10
+ a=${pair%%:*}; b=${pair#*:}
11
+ echo "$a $(sha256sum $P/$a | cut -c1-12) $(sha256sum /sgl-workspace/sglang/python/sglang/srt/$b | cut -c1-12)"
12
+ done
13
+ ls /opt/ | head; ls /opt/qwen* 2>/dev/null
14
+ curl -sI https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-39/warc.paths.gz | head -3
15
+ df -h /job | tail -1
training/runner/t01_pylib.sh ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ cd /job/work
2
+ rm -rf /job/work/.venv
3
+ pip install -q --no-deps --target /job/work/pylib selectolax onnx onnxruntime onnxscript onnx_ir coloredlogs humanfriendly flatbuffers 2>&1 | tail -3
4
+ export PYTHONPATH=/job/work/pylib
5
+ python3 -c "
6
+ import selectolax, onnx, onnxruntime, onnxscript, aiohttp, torch, numpy, lxml, tokenizers, transformers
7
+ print('ok', torch.__version__, numpy.__version__, onnx.__version__, onnxruntime.__version__, torch.cuda.is_available())
8
+ "
training/runner/t02_pilot.sh ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/pilot
3
+ python3 harvest/cc_candidates.py data/pilot/cand.jsonl 200 99 2>&1 | tail -3
4
+ wc -l data/pilot/cand.jsonl
5
+ head -3000 data/pilot/cand.jsonl > data/pilot/cand3k.jsonl
6
+ python3 harvest/fetch_extract.py data/pilot/cand3k.jsonl data/pilot/forms 28 4 2>&1 | tail -5
training/runner/t03_harvest.sh ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/cc
3
+ python3 harvest/cc_candidates.py data/cc/cand.jsonl 16000 2026 > /job/logs/tasks/cand.log 2>&1
4
+ tail -3 /job/logs/tasks/cand.log
5
+ wc -l data/cc/cand.jsonl || exit 1
6
+ python3 harvest/fetch_extract.py data/cc/cand.jsonl data/cc/forms 28 6 > /job/logs/tasks/fetch.log 2>&1
7
+ tail -3 /job/logs/tasks/fetch.log
8
+ du -sh data/cc
training/runner/t04_sglang.sh ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # private high-batch teacher server (loopback inside the job container only)
2
+ mkdir -p /job/cache /job/home
3
+ [ -e /job/home/.cache ] || ln -s /job/cache /job/home/.cache
4
+ RC=/home/user/ai/maintenance/20260919-upgrades/runtime-cache/qwen38-flash-next-sglang
5
+ rm -rf /job/cache/sglang; mkdir -p /job/cache/sglang/inductor
6
+ export SGLANG_CACHE_DIR=/job/cache/sglang TORCHINDUCTOR_CACHE_DIR=/job/cache/sglang/inductor
7
+ [ -d /job/cache/triton/cache ] || cp -r $RC/triton/. /job/cache/triton/ 2>/dev/null
8
+ [ -d /job/home/.tilelang ] || cp -r $RC/.tilelang /job/home/.tilelang 2>/dev/null
9
+ export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
10
+ M=/home/user/models/huggingface/hub/models--RadixArk--Qwen3.8-Flash-Next-NVFP4/snapshots/7b719225242aacd3dbd3f9407468c2ee9a9d2594
11
+ exec python3 -m sglang.launch_server --model-path $M --served-model-name qwen --tp 1 --trust-remote-code \
12
+ --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass --context-length 16384 \
13
+ --max-total-tokens 131072 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.97 --page-size 64 \
14
+ --chunked-prefill-size 8192 --ple-offload-embedding --linear-attn-prefill-backend triton \
15
+ --linear-attn-decode-backend flashinfer --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer \
16
+ --mamba-track-interval 64 --max-mamba-cache-size 128 --max-running-requests 64 --cuda-graph-max-bs-decode 64 \
17
+ --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":16}' \
18
+ --reasoning-parser auto --sampling-defaults model --host 127.0.0.1 --port 30020
training/runner/t05_grep.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ env | grep -i "cache\|sglang" | head -20
2
+ grep -rn "\.cache/sglang\|inductor" /sgl-workspace/sglang/python/sglang/srt --include=*.py | grep -i "cache_dir\|environ\|getenv\|expanduser\|/root" | head -20
3
+ grep -rn "root/.cache" /sgl-workspace/sglang/python/sglang/srt --include=*.py | head
training/runner/t06_harvest.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/cc/parts
3
+ python3 harvest/polite_harvest.py data/cc/parts 250000 --rate 10 --max-rate 25 --conc 24 --procs 12
training/runner/t07_calib_prep.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/calib
3
+ python3 label/prep_calib.py 'data/pilot/forms/*.jsonl.gz' data/calib/calib.jsonl
training/runner/t08_calib.sh ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work
3
+ curl -s http://127.0.0.1:30020/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5,"chat_template_kwargs":{"enable_thinking":false}}'; echo
4
+ python3 label/teacher.py data/calib/calib.jsonl data/calib/teacher_maskac.jsonl --conc 64 --mask-ac
5
+ python3 label/teacher.py data/calib/calib.jsonl data/calib/teacher_full.jsonl --conc 64
training/runner/t09_warc_test.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/warc/parts
3
+ python3 harvest/warc_stream.py data/warc/parts 2 --procs 16 --parallel 2 --seed 11
training/runner/t10_warc.sh ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/warc/parts
3
+ python3 harvest/warc_stream.py data/warc/parts 50 --procs 26 --parallel 4 --seed 11 2>&1 | grep -v "^CC-MAIN" ; ls data/warc/parts | wc -l
training/runner/t11_label.sh ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ export PYTHONPATH=/job/work/pylib:/job/work
2
+ cd /job/work && mkdir -p data/label
3
+ python3 label/select.py data/label 'data/pilot/forms/*.jsonl.gz' 'data/warc/parts/*.jsonl.gz' 'data/cc/parts/*.jsonl.gz'
4
+ for q in data/label/queue-*.jsonl; do
5
+ python3 label/teacher.py $q data/label/labels-all.jsonl --conc 64 2>&1 | grep -v "^\s*$"
6
+ done