Token Classification
Transformers
ONNX
Safetensors
multilingual
bert
autofill
password-manager
form-classification
field-classification
browser-extension
Instructions to use ProCreations/Zap with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ProCreations/Zap with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="ProCreations/Zap")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("ProCreations/Zap") model = AutoModelForTokenClassification.from_pretrained("ProCreations/Zap", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Zap v1: 20.7M-param web form & field classifier for autofill (bf16, MIT)
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- README.md +196 -0
- __pycache__/zap_extract.cpython-312.pyc +0 -0
- __pycache__/zap_infer.cpython-312.pyc +0 -0
- config.json +107 -0
- eval/data_stats.json +90 -0
- eval/eval_test_bfloat16.json +1353 -0
- eval/eval_test_float32.json +1353 -0
- eval/eval_val_bfloat16.json +1338 -0
- eval/export_parity.json +7 -0
- eval/val_metrics.json +367 -0
- model.safetensors +3 -0
- onnx/model.onnx +3 -0
- onnx/model_quantized.onnx +3 -0
- tokenizer.json +0 -0
- tokenizer_config.json +9 -0
- training/card/infer_parity_py.py +13 -0
- training/card/zap_extract.py +580 -0
- training/card/zap_infer.py +99 -0
- training/harvest/cc_candidates.py +124 -0
- training/harvest/fetch_extract.py +144 -0
- training/harvest/parity_py.py +10 -0
- training/harvest/polite_harvest.py +254 -0
- training/harvest/sample_pages.py +30 -0
- training/harvest/warc_stream.py +181 -0
- training/js/bench.mjs +18 -0
- training/js/infer-parity-test.mjs +29 -0
- training/js/parity-test.js +30 -0
- training/js/zap-extract.js +326 -0
- training/js/zap-infer.js +89 -0
- training/label/common.py +42 -0
- training/label/eval_calib.py +48 -0
- training/label/prep_calib.py +15 -0
- training/label/select.py +126 -0
- training/label/select_rare.py +64 -0
- training/label/stats.py +19 -0
- training/label/synth.py +162 -0
- training/label/teacher.py +191 -0
- training/runner/run.sh +27 -0
- training/runner/t00_setup.sh +15 -0
- training/runner/t01_pylib.sh +8 -0
- training/runner/t02_pilot.sh +6 -0
- training/runner/t03_harvest.sh +8 -0
- training/runner/t04_sglang.sh +18 -0
- training/runner/t05_grep.sh +3 -0
- training/runner/t06_harvest.sh +3 -0
- training/runner/t07_calib_prep.sh +3 -0
- training/runner/t08_calib.sh +5 -0
- training/runner/t09_warc_test.sh +3 -0
- training/runner/t10_warc.sh +3 -0
- training/runner/t11_label.sh +6 -0
README.md
ADDED
|
@@ -0,0 +1,196 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: transformers
|
| 4 |
+
pipeline_tag: token-classification
|
| 5 |
+
language:
|
| 6 |
+
- multilingual
|
| 7 |
+
tags:
|
| 8 |
+
- autofill
|
| 9 |
+
- password-manager
|
| 10 |
+
- form-classification
|
| 11 |
+
- field-classification
|
| 12 |
+
- browser-extension
|
| 13 |
+
- bert
|
| 14 |
+
- onnx
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# Zap ⚡ — a tiny web-form classifier for autofill
|
| 18 |
+
|
| 19 |
+
**Zap** looks at a web form and tells a password manager what every input wants and what the form is for.
|
| 20 |
+
It runs locally, has **20.7M parameters**, and ships **bf16 weights (41 MB)**, plus ONNX files (fp32 83 MB, int8 21 MB).
|
| 21 |
+
|
| 22 |
+
* **Field types (26):** `username`, `email`, `current-password`, `new-password`, `one-time-code`, `tel`, names, address parts, payment-card parts, `bday`, `search`, `captcha`, `other`
|
| 23 |
+
* **Form purposes (11):** `login`, `signup`, `password-change`, `password-recovery`, `otp`, `payment`, `address`, `newsletter`, `contact`, `search`, `other`
|
| 24 |
+
|
| 25 |
+
So your autofill can answer questions like *"is this a login form?"*, *"which box gets the saved password, and which one gets a newly generated password?"*, *"is this the email field, or is it a honeypot?"*
|
| 26 |
+
|
| 27 |
+
**Held-out websites:** 6,838 real forms and 19,811 fields from Common Crawl pages on hosts never seen in training:
|
| 28 |
+
|
| 29 |
+
| | **Zap** | keyword heuristic¹ |
|
| 30 |
+
|---|---|---|
|
| 31 |
+
| field accuracy (26 classes) | **97.2%** | 82.7% |
|
| 32 |
+
| field macro-F1 | **0.945** | 0.723 |
|
| 33 |
+
| form-purpose accuracy (11 classes) | **95.7%** | 71.6% |
|
| 34 |
+
| password-field detection F1 | **0.996** | 0.968 |
|
| 35 |
+
| current vs new password (on password fields) | **99.2%** | 85.5% |
|
| 36 |
+
| login-identifier (`username`/`email`) F1 | **0.994** | 0.945 |
|
| 37 |
+
| `autocomplete` removed from the input: recovers the developer's own token² | **97.8%** | 78.7% |
|
| 38 |
+
|
| 39 |
+
¹ A baseline of the kind password managers ship: it trusts a valid `autocomplete`, then the input `type`, then multilingual keyword regexes over name/id/placeholder/label/nearby text.
|
| 40 |
+
² On 1,622 test fields where the site's `autocomplete` token agrees with the reference label, the token is deleted from the model input. The score is how often Zap still predicts it. This check doesn't depend on the LLM teacher.
|
| 41 |
+
|
| 42 |
+
## Quick start (Python)
|
| 43 |
+
|
| 44 |
+
```python
|
| 45 |
+
# pip install transformers torch lxml selectolax huggingface_hub
|
| 46 |
+
import sys
|
| 47 |
+
from huggingface_hub import snapshot_download
|
| 48 |
+
sys.path.insert(0, snapshot_download("ProCreations/Zap"))
|
| 49 |
+
from zap_infer import Zap
|
| 50 |
+
|
| 51 |
+
zap = Zap.from_pretrained("ProCreations/Zap") # CPU, fp32 compute from the bf16 weights
|
| 52 |
+
for form in zap.classify_html(html, url="https://northwind.example/login"):
|
| 53 |
+
print(form["form"], round(form["form_score"], 3))
|
| 54 |
+
for f in form["fields"]:
|
| 55 |
+
print(" ", f["type"], f["name"], "->", f["label"], round(f["score"], 3))
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
Output for a page that has a sign-in form and a separate "New here? Create account" block:
|
| 59 |
+
|
| 60 |
+
```
|
| 61 |
+
login 0.966
|
| 62 |
+
input text login -> username 0.96
|
| 63 |
+
input password pw -> current-password 0.962
|
| 64 |
+
signup 0.961
|
| 65 |
+
input text email -> email 0.957
|
| 66 |
+
input password pass 1 -> new-password 0.956
|
| 67 |
+
input password pass 2 -> new-password 0.955
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
Neither block sets a useful `autocomplete` attribute: the login box has `autocomplete="off"`, and the signup inputs are typeless `<input>`s with placeholders. Zap tells them apart from context.
|
| 71 |
+
|
| 72 |
+
## In a browser extension (JavaScript)
|
| 73 |
+
|
| 74 |
+
Zap doesn't read raw HTML. It reads a compact text rendering of each form: page title, URL words, form attributes, heading, buttons and links, and for every field its type, name, id, autocomplete, placeholder, label, nearby text and so on. `zap-extract.js` builds that rendering from the **live DOM**, so it sees exactly what the model was trained on. `zap-infer.js` handles windowing and decoding with transformers.js.
|
| 75 |
+
|
| 76 |
+
```js
|
| 77 |
+
import * as transformers from "@huggingface/transformers";
|
| 78 |
+
// load zap-extract.js and zap-infer.js first (content scripts), or require() them in Node
|
| 79 |
+
const zap = await ZapModel.load(transformers, "ProCreations/Zap"); // onnx/model.onnx; pass { dtype: "q8" } for the int8 file
|
| 80 |
+
const results = await zap.classify(Zap.zapExtract(document, location.href));
|
| 81 |
+
for (const r of results) {
|
| 82 |
+
console.log(r.form, r.formScore);
|
| 83 |
+
for (const f of r.fields) console.log(f.element, f.label, f.score); // f.element is the <input>/<select>/<textarea>
|
| 84 |
+
}
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
**Parity checks:** these run on 400 random real pages.
|
| 88 |
+
* **Extraction:** the JS extractor (DOM via jsdom) and the Python extractor produce identical model input for **all 821 forms** the JS side finds. The Python side finds 4 extra forms on 2 malformed spam pages. The Python side re-parses pages with an HTML5-spec parser so its tree matches a browser's.
|
| 89 |
+
* **End to end:** JS (transformers.js + ONNX) and Python (PyTorch) predict **identical labels for 818/818 forms and 1,413/1,413 fields**.
|
| 90 |
+
|
| 91 |
+
**Speed:** measured on an Apple M4 Max CPU with onnxruntime via transformers.js, model only.
|
| 92 |
+
* **5.3 ms per form** with fp32 ONNX, and **3.1 ms** with int8 ONNX.
|
| 93 |
+
* The int8 file agrees with fp32 on 99.91% of predictions, and bf16 PyTorch on 99.99%.
|
| 94 |
+
|
| 95 |
+
How the output is decoded: each field is read at its `[FLD]` token, restricted to the `field:*` labels. The form purpose is read at `[CLS]`, restricted to the `form:*` labels. A form longer than 512 tokens is split into windows that each repeat the form context, and the `[CLS]` logits are averaged across windows.
|
| 96 |
+
|
| 97 |
+
## Labels
|
| 98 |
+
|
| 99 |
+
| field label | meaning |
|
| 100 |
+
|---|---|
|
| 101 |
+
| `username` | account identifier that is not strictly an email: username, user ID, "email or username", "email or phone" |
|
| 102 |
+
| `email` | an email address (login, signup, newsletter, checkout, …) |
|
| 103 |
+
| `current-password` | existing password / sign-in PIN (login, "current password") |
|
| 104 |
+
| `new-password` | password being created or set, including "confirm password" |
|
| 105 |
+
| `one-time-code` | 2FA / OTP / SMS / email verification code, including each box of split code inputs |
|
| 106 |
+
| `tel` | phone number or part of one |
|
| 107 |
+
| `name`, `given-name`, `family-name`, `organization` | person / company names |
|
| 108 |
+
| `street-address`, `address-line2`, `address-level2` (city), `address-level1` (state/region), `postal-code`, `country` | address parts |
|
| 109 |
+
| `cc-name`, `cc-number`, `cc-exp`, `cc-exp-month`, `cc-exp-year`, `cc-csc` | payment card |
|
| 110 |
+
| `bday` | date of birth (or part of it) |
|
| 111 |
+
| `search`, `captcha` | search box, CAPTCHA / anti-spam answer |
|
| 112 |
+
| `other` | everything else: messages, coupons, quantities, honeypots… |
|
| 113 |
+
|
| 114 |
+
Label names follow the HTML `autocomplete` tokens where one exists. `username` vs `email` is decided by what the field accepts: an email-only login box is `email`. On a `login` form, fill the saved login identifier into either one.
|
| 115 |
+
|
| 116 |
+
## Evaluation
|
| 117 |
+
|
| 118 |
+
Splits are made **by website host**: 4% test, 2% val. Every deduplicated form signature belongs to exactly one split. Headline numbers use only real web forms. Synthetic forms (see below) are scored separately.
|
| 119 |
+
|
| 120 |
+
**Per class, real held-out test** (bf16 weights; F1, with n in parentheses):
|
| 121 |
+
|
| 122 |
+
| field | F1 (n) | field | F1 (n) |
|
| 123 |
+
|---|---|---|---|
|
| 124 |
+
| email | 0.990 (3193) | postal-code | 0.970 (322) |
|
| 125 |
+
| current-password | 0.991 (1838) | captcha | 0.953 (301) |
|
| 126 |
+
| new-password | 0.987 (712) | address-level2 | 0.897 (292) |
|
| 127 |
+
| username | 0.976 (1284) | organization | 0.947 (256) |
|
| 128 |
+
| tel | 0.988 (1337) | street-address | 0.958 (208) |
|
| 129 |
+
| name | 0.962 (1461) | address-level1 | 0.891 (175) |
|
| 130 |
+
| given-name | 0.943 (603) | country | 0.913 (158) |
|
| 131 |
+
| family-name | 0.982 (506) | one-time-code | 0.929 (135) |
|
| 132 |
+
| search | 0.963 (948) | bday | 0.932 (53) |
|
| 133 |
+
| other | 0.968 (5955) | address-line2 | 0.847 (48) |
|
| 134 |
+
|
| 135 |
+
| form | F1 (n) | form | F1 (n) |
|
| 136 |
+
|---|---|---|---|
|
| 137 |
+
| login | 0.992 (1836) | password-change | 0.943 (50) |
|
| 138 |
+
| contact | 0.963 (1640) | otp | 0.897 (40) |
|
| 139 |
+
| search | 0.954 (1043) | password-recovery | 0.933 (38) |
|
| 140 |
+
| newsletter | 0.965 (667) | address | 0.783 (35) |
|
| 141 |
+
| signup | 0.962 (417) | payment | 0.667 (8) |
|
| 142 |
+
| other | 0.893 (1064) | | |
|
| 143 |
+
|
| 144 |
+
Real payment-card forms are very rare in a static crawl: the real test has 8 payment forms and 6 card fields. On the **synthetic** held-out test (84 forms), the card fields `cc-number`, `cc-name`, `cc-exp`, `cc-exp-month`, `cc-exp-year` and `cc-csc` score F1 = 1.00, `payment` forms 0.98 and `otp` forms 1.00. Synthetic forms are easier than real ones, so treat these as a sanity check, not a real-world number.
|
| 145 |
+
|
| 146 |
+
**Most common confusions:** `given-name`↔`name` (a lone "Name" field sitting next to "Last name"), `search`↔`other` (site filters), `username`↔`email` ("Login" boxes that probably take an email), and `address-level1/2`↔`other`.
|
| 147 |
+
|
| 148 |
+
**Label quality:** a second, independent reviewer model re-labeled a stratified sample of 80 teacher-labeled test forms (217 fields) straight from the HTML. It found **1 clear field error out of 210 checkable fields (0.5%)** and 1 clear form-purpose error in 80 (1.3%), plus 6 ambiguous cases of each kind. The one systematic teacher weakness it found is honeypot fields that carry realistic labels.
|
| 149 |
+
|
| 150 |
+
The full reports are in `eval/`, including per-class precision/recall and confusions for bf16 and fp32 weights and the heuristic baseline.
|
| 151 |
+
|
| 152 |
+
## How it was made
|
| 153 |
+
|
| 154 |
+
**Data.** 62 random WARC files from Common Crawl `CC-MAIN-2026-39` (September 2026) were streamed in full, one request per file. That's 998,946 pages with at least one fillable field, 2.30M forms, and **948,742 unique forms** after deduplicating by form signature. Pages are re-serialized through an HTML5-spec parser (lexbor) before extraction so the tree matches what a browser builds. Fields outside any `<form>` are grouped the way a single-page app lays them out (nearest ancestor with a button).
|
| 155 |
+
|
| 156 |
+
**Labels.** A private, high-batch **Qwen3.8-Flash-Next** (NVFP4, SGLang, thinking off, greedy decoding) ran inside the training job. It labeled every field and the form purpose for **170,163 unique forms** in three rounds:
|
| 157 |
+
1. A priority-weighted pass: password forms first, then multi-field forms, then single-field ones.
|
| 158 |
+
2. A targeted pass over not-yet-labeled real forms whose fields look like address, OTP or payment fields.
|
| 159 |
+
3. The synthetic top-up described below.
|
| 160 |
+
|
| 161 |
+
The teacher saw the cleaned form HTML with numbered fields, the page title and the URL, plus a label spec with rules for the hard cases: current vs new password, `email` vs `username`, honeypots, and `autocomplete` values set only to block autofill.
|
| 162 |
+
|
| 163 |
+
* *Calibration:* on 222 forms where site developers set standard `autocomplete` tokens, the teacher (with `autocomplete` hidden) agreed with the developer token on 84% of fields. Manual inspection showed almost every disagreement was the site being wrong, e.g. Gravity Forms honeypots tagged `autocomplete="new-password"`, or "First Name" fields tagged `name`.
|
| 164 |
+
* *Synthetic top-up:* static crawls rarely contain card-payment, OTP and password-change pages. The teacher therefore also wrote **2,578 synthetic forms** in 23 languages and 12 markup styles, from React div-soup with no `<form>` to split card fields, with deliberately wrong or missing `autocomplete`. They go through the same extractor and the same labeling pass, and make up about 1% of training forms.
|
| 165 |
+
|
| 166 |
+
**Model.** `BertForTokenClassification`: 8 layers, hidden size 384, 6 heads, FFN 1536, 16k WordPiece vocabulary trained on serialized forms, 512 positions. A single token-classification head predicts `form:*` at `[CLS]` and `field:*` at every `[FLD]` marker, so the whole form is classified in one pass and every field sees its neighbours. That is how Zap tells `new-password` from `current-password`, and a signup `email` from a login `username`.
|
| 167 |
+
|
| 168 |
+
**Training.** Done on one RTX PRO 6000 (Blackwell) with bf16 autocast:
|
| 169 |
+
1. MLM pretraining on 892k unlabeled train-split forms (108M tokens, 3 epochs, about 8 minutes).
|
| 170 |
+
2. Fine-tuning on 217,782 labeled rows, with rare-class forms oversampled ×3 and label smoothing 0.05. Hint dropout randomly removes `autocomplete` (25%), labels, names and placeholders, and flips 8% of password inputs to `type=text` to mimic "show password" toggles.
|
| 171 |
+
3. The best of 6 epochs was picked on validation.
|
| 172 |
+
|
| 173 |
+
The published weights are bf16. The whole project took about 6 hours of workstation time: harvesting, three rounds of teacher labeling, synthesis, two training runs, evaluation and export.
|
| 174 |
+
|
| 175 |
+
## Limitations
|
| 176 |
+
|
| 177 |
+
* Zap reads the DOM text and attributes, not pixels or CSS. In a browser, drop invisible inputs (honeypots) before classifying, e.g. with `offsetParent === null` or `getComputedStyle`. The training crawl only saw static HTML, where some CSS-hidden honeypots remained, so Zap usually labels those `other`, but not always.
|
| 178 |
+
* The training pages come from a static crawl, so heavily scripted flows (identifier-first sign-in, checkout iframes) are covered mainly by synthetic examples. Real-world accuracy on payment forms is not well measured (8 test forms).
|
| 179 |
+
* The labels come from an LLM teacher. It is audited, but it isn't human annotation, so expect a small amount of label noise, especially on ambiguous `name`/`given-name` and `search`/`other` cases.
|
| 180 |
+
* Fields inside cross-origin iframes (e.g. hosted card fields) are invisible to any content script, including `zap-extract.js`.
|
| 181 |
+
|
| 182 |
+
## Files
|
| 183 |
+
|
| 184 |
+
| file | what |
|
| 185 |
+
|---|---|
|
| 186 |
+
| `model.safetensors` | bf16 weights (BertForTokenClassification, 20.7M params) |
|
| 187 |
+
| `config.json`, `tokenizer.json`, `tokenizer_config.json` | config with the 37-label map; 16k WordPiece tokenizer |
|
| 188 |
+
| `onnx/model.onnx`, `onnx/model_quantized.onnx` | fp32 and int8 ONNX for onnxruntime / transformers.js |
|
| 189 |
+
| `zap_extract.py` / `zap-extract.js` | form extractor (Python canonical / DOM port) |
|
| 190 |
+
| `zap_infer.py` / `zap-infer.js` | windowing + inference helpers |
|
| 191 |
+
| `eval/` | evaluation reports, export parity, dataset statistics |
|
| 192 |
+
| `training/` | the code used to harvest, label, synthesize, train and export |
|
| 193 |
+
|
| 194 |
+
## License
|
| 195 |
+
|
| 196 |
+
MIT
|
__pycache__/zap_extract.cpython-312.pyc
ADDED
|
Binary file (31.2 kB). View file
|
|
|
__pycache__/zap_infer.cpython-312.pyc
ADDED
|
Binary file (8.5 kB). View file
|
|
|
config.json
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_cross_attention": false,
|
| 3 |
+
"architectures": [
|
| 4 |
+
"BertForTokenClassification"
|
| 5 |
+
],
|
| 6 |
+
"attention_probs_dropout_prob": 0.1,
|
| 7 |
+
"bos_token_id": null,
|
| 8 |
+
"classifier_dropout": null,
|
| 9 |
+
"dtype": "bfloat16",
|
| 10 |
+
"eos_token_id": null,
|
| 11 |
+
"hidden_act": "gelu",
|
| 12 |
+
"hidden_dropout_prob": 0.1,
|
| 13 |
+
"hidden_size": 384,
|
| 14 |
+
"id2label": {
|
| 15 |
+
"0": "form:login",
|
| 16 |
+
"1": "form:signup",
|
| 17 |
+
"2": "form:password-change",
|
| 18 |
+
"3": "form:password-recovery",
|
| 19 |
+
"4": "form:otp",
|
| 20 |
+
"5": "form:payment",
|
| 21 |
+
"6": "form:address",
|
| 22 |
+
"7": "form:newsletter",
|
| 23 |
+
"8": "form:contact",
|
| 24 |
+
"9": "form:search",
|
| 25 |
+
"10": "form:other",
|
| 26 |
+
"11": "field:username",
|
| 27 |
+
"12": "field:email",
|
| 28 |
+
"13": "field:current-password",
|
| 29 |
+
"14": "field:new-password",
|
| 30 |
+
"15": "field:one-time-code",
|
| 31 |
+
"16": "field:tel",
|
| 32 |
+
"17": "field:name",
|
| 33 |
+
"18": "field:given-name",
|
| 34 |
+
"19": "field:family-name",
|
| 35 |
+
"20": "field:organization",
|
| 36 |
+
"21": "field:street-address",
|
| 37 |
+
"22": "field:address-line2",
|
| 38 |
+
"23": "field:address-level2",
|
| 39 |
+
"24": "field:address-level1",
|
| 40 |
+
"25": "field:postal-code",
|
| 41 |
+
"26": "field:country",
|
| 42 |
+
"27": "field:cc-name",
|
| 43 |
+
"28": "field:cc-number",
|
| 44 |
+
"29": "field:cc-exp",
|
| 45 |
+
"30": "field:cc-exp-month",
|
| 46 |
+
"31": "field:cc-exp-year",
|
| 47 |
+
"32": "field:cc-csc",
|
| 48 |
+
"33": "field:bday",
|
| 49 |
+
"34": "field:search",
|
| 50 |
+
"35": "field:captcha",
|
| 51 |
+
"36": "field:other"
|
| 52 |
+
},
|
| 53 |
+
"initializer_range": 0.02,
|
| 54 |
+
"intermediate_size": 1536,
|
| 55 |
+
"is_decoder": false,
|
| 56 |
+
"label2id": {
|
| 57 |
+
"field:address-level1": 24,
|
| 58 |
+
"field:address-level2": 23,
|
| 59 |
+
"field:address-line2": 22,
|
| 60 |
+
"field:bday": 33,
|
| 61 |
+
"field:captcha": 35,
|
| 62 |
+
"field:cc-csc": 32,
|
| 63 |
+
"field:cc-exp": 29,
|
| 64 |
+
"field:cc-exp-month": 30,
|
| 65 |
+
"field:cc-exp-year": 31,
|
| 66 |
+
"field:cc-name": 27,
|
| 67 |
+
"field:cc-number": 28,
|
| 68 |
+
"field:country": 26,
|
| 69 |
+
"field:current-password": 13,
|
| 70 |
+
"field:email": 12,
|
| 71 |
+
"field:family-name": 19,
|
| 72 |
+
"field:given-name": 18,
|
| 73 |
+
"field:name": 17,
|
| 74 |
+
"field:new-password": 14,
|
| 75 |
+
"field:one-time-code": 15,
|
| 76 |
+
"field:organization": 20,
|
| 77 |
+
"field:other": 36,
|
| 78 |
+
"field:postal-code": 25,
|
| 79 |
+
"field:search": 34,
|
| 80 |
+
"field:street-address": 21,
|
| 81 |
+
"field:tel": 16,
|
| 82 |
+
"field:username": 11,
|
| 83 |
+
"form:address": 6,
|
| 84 |
+
"form:contact": 8,
|
| 85 |
+
"form:login": 0,
|
| 86 |
+
"form:newsletter": 7,
|
| 87 |
+
"form:other": 10,
|
| 88 |
+
"form:otp": 4,
|
| 89 |
+
"form:password-change": 2,
|
| 90 |
+
"form:password-recovery": 3,
|
| 91 |
+
"form:payment": 5,
|
| 92 |
+
"form:search": 9,
|
| 93 |
+
"form:signup": 1
|
| 94 |
+
},
|
| 95 |
+
"layer_norm_eps": 1e-12,
|
| 96 |
+
"max_position_embeddings": 512,
|
| 97 |
+
"model_type": "bert",
|
| 98 |
+
"num_attention_heads": 6,
|
| 99 |
+
"num_hidden_layers": 8,
|
| 100 |
+
"pad_token_id": 0,
|
| 101 |
+
"tie_word_embeddings": true,
|
| 102 |
+
"transformers_version": "5.12.1",
|
| 103 |
+
"type_vocab_size": 1,
|
| 104 |
+
"use_cache": true,
|
| 105 |
+
"vocab_size": 16384,
|
| 106 |
+
"torch_dtype": "bfloat16"
|
| 107 |
+
}
|
eval/data_stats.json
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"stats": {
|
| 3 |
+
"mlm": 893251,
|
| 4 |
+
"train": 217782,
|
| 5 |
+
"val": 3408,
|
| 6 |
+
"test": 6922
|
| 7 |
+
},
|
| 8 |
+
"train_form_labels": {
|
| 9 |
+
"search": 36885,
|
| 10 |
+
"other": 33139,
|
| 11 |
+
"newsletter": 22424,
|
| 12 |
+
"contact": 44859,
|
| 13 |
+
"login": 59124,
|
| 14 |
+
"signup": 13290,
|
| 15 |
+
"otp": 1992,
|
| 16 |
+
"address": 1694,
|
| 17 |
+
"payment": 1170,
|
| 18 |
+
"password-change": 1605,
|
| 19 |
+
"password-recovery": 1600
|
| 20 |
+
},
|
| 21 |
+
"train_field_labels": {
|
| 22 |
+
"search": 33272,
|
| 23 |
+
"other": 171342,
|
| 24 |
+
"email": 98108,
|
| 25 |
+
"name": 40121,
|
| 26 |
+
"tel": 37210,
|
| 27 |
+
"street-address": 6196,
|
| 28 |
+
"postal-code": 10203,
|
| 29 |
+
"captcha": 8692,
|
| 30 |
+
"username": 41964,
|
| 31 |
+
"current-password": 59298,
|
| 32 |
+
"organization": 7200,
|
| 33 |
+
"given-name": 18361,
|
| 34 |
+
"family-name": 15434,
|
| 35 |
+
"new-password": 22226,
|
| 36 |
+
"one-time-code": 5861,
|
| 37 |
+
"address-level1": 4746,
|
| 38 |
+
"cc-name": 780,
|
| 39 |
+
"address-level2": 8701,
|
| 40 |
+
"country": 4902,
|
| 41 |
+
"cc-number": 961,
|
| 42 |
+
"cc-exp-month": 297,
|
| 43 |
+
"cc-exp-year": 290,
|
| 44 |
+
"cc-csc": 887,
|
| 45 |
+
"bday": 2171,
|
| 46 |
+
"cc-exp": 625,
|
| 47 |
+
"address-line2": 1254
|
| 48 |
+
},
|
| 49 |
+
"field_labels": [
|
| 50 |
+
"username",
|
| 51 |
+
"email",
|
| 52 |
+
"current-password",
|
| 53 |
+
"new-password",
|
| 54 |
+
"one-time-code",
|
| 55 |
+
"tel",
|
| 56 |
+
"name",
|
| 57 |
+
"given-name",
|
| 58 |
+
"family-name",
|
| 59 |
+
"organization",
|
| 60 |
+
"street-address",
|
| 61 |
+
"address-line2",
|
| 62 |
+
"address-level2",
|
| 63 |
+
"address-level1",
|
| 64 |
+
"postal-code",
|
| 65 |
+
"country",
|
| 66 |
+
"cc-name",
|
| 67 |
+
"cc-number",
|
| 68 |
+
"cc-exp",
|
| 69 |
+
"cc-exp-month",
|
| 70 |
+
"cc-exp-year",
|
| 71 |
+
"cc-csc",
|
| 72 |
+
"bday",
|
| 73 |
+
"search",
|
| 74 |
+
"captcha",
|
| 75 |
+
"other"
|
| 76 |
+
],
|
| 77 |
+
"form_labels": [
|
| 78 |
+
"login",
|
| 79 |
+
"signup",
|
| 80 |
+
"password-change",
|
| 81 |
+
"password-recovery",
|
| 82 |
+
"otp",
|
| 83 |
+
"payment",
|
| 84 |
+
"address",
|
| 85 |
+
"newsletter",
|
| 86 |
+
"contact",
|
| 87 |
+
"search",
|
| 88 |
+
"other"
|
| 89 |
+
]
|
| 90 |
+
}
|
eval/eval_test_bfloat16.json
ADDED
|
@@ -0,0 +1,1353 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"split": "test",
|
| 3 |
+
"dtype": "bfloat16",
|
| 4 |
+
"forms_per_s_gpu": 2022.5467036804282,
|
| 5 |
+
"model": {
|
| 6 |
+
"n_forms": 6922,
|
| 7 |
+
"n_fields": 20130,
|
| 8 |
+
"field_acc": 0.9719821162444113,
|
| 9 |
+
"field_macro_f1": 0.9597690495523868,
|
| 10 |
+
"form_acc": 0.9572377925455071,
|
| 11 |
+
"form_macro_f1": 0.9327853059996515,
|
| 12 |
+
"password_precision": 0.9942263279445728,
|
| 13 |
+
"password_recall": 0.9976825028968713,
|
| 14 |
+
"password_f1": 0.9959514170040485,
|
| 15 |
+
"cur_vs_new_acc": 0.9915025106218617,
|
| 16 |
+
"identifier_f1": 0.9939004103360319,
|
| 17 |
+
"per_field": {
|
| 18 |
+
"username": {
|
| 19 |
+
"p": 0.9782101167315175,
|
| 20 |
+
"r": 0.9736638264910922,
|
| 21 |
+
"f1": 0.9759316770186335,
|
| 22 |
+
"n": 1291
|
| 23 |
+
},
|
| 24 |
+
"email": {
|
| 25 |
+
"p": 0.9870049504950495,
|
| 26 |
+
"r": 0.9940791523839202,
|
| 27 |
+
"f1": 0.9905294208973761,
|
| 28 |
+
"n": 3209
|
| 29 |
+
},
|
| 30 |
+
"current-password": {
|
| 31 |
+
"p": 0.9866238630283574,
|
| 32 |
+
"r": 0.9956803455723542,
|
| 33 |
+
"f1": 0.9911314162859447,
|
| 34 |
+
"n": 1852
|
| 35 |
+
},
|
| 36 |
+
"new-password": {
|
| 37 |
+
"p": 0.9917695473251029,
|
| 38 |
+
"r": 0.9810040705563093,
|
| 39 |
+
"f1": 0.9863574351978172,
|
| 40 |
+
"n": 737
|
| 41 |
+
},
|
| 42 |
+
"one-time-code": {
|
| 43 |
+
"p": 0.9318181818181818,
|
| 44 |
+
"r": 0.9425287356321839,
|
| 45 |
+
"f1": 0.9371428571428572,
|
| 46 |
+
"n": 174
|
| 47 |
+
},
|
| 48 |
+
"tel": {
|
| 49 |
+
"p": 0.9809802487198245,
|
| 50 |
+
"r": 0.9933333333333333,
|
| 51 |
+
"f1": 0.9871181450128819,
|
| 52 |
+
"n": 1350
|
| 53 |
+
},
|
| 54 |
+
"name": {
|
| 55 |
+
"p": 0.9501661129568106,
|
| 56 |
+
"r": 0.9747784594410361,
|
| 57 |
+
"f1": 0.9623149394347241,
|
| 58 |
+
"n": 1467
|
| 59 |
+
},
|
| 60 |
+
"given-name": {
|
| 61 |
+
"p": 0.9754385964912281,
|
| 62 |
+
"r": 0.9129720853858785,
|
| 63 |
+
"f1": 0.9431721798134012,
|
| 64 |
+
"n": 609
|
| 65 |
+
},
|
| 66 |
+
"family-name": {
|
| 67 |
+
"p": 0.9881422924901185,
|
| 68 |
+
"r": 0.9765625,
|
| 69 |
+
"f1": 0.9823182711198427,
|
| 70 |
+
"n": 512
|
| 71 |
+
},
|
| 72 |
+
"organization": {
|
| 73 |
+
"p": 0.953307392996109,
|
| 74 |
+
"r": 0.9423076923076923,
|
| 75 |
+
"f1": 0.9477756286266925,
|
| 76 |
+
"n": 260
|
| 77 |
+
},
|
| 78 |
+
"street-address": {
|
| 79 |
+
"p": 0.9396551724137931,
|
| 80 |
+
"r": 0.9775784753363229,
|
| 81 |
+
"f1": 0.9582417582417583,
|
| 82 |
+
"n": 223
|
| 83 |
+
},
|
| 84 |
+
"address-line2": {
|
| 85 |
+
"p": 0.975609756097561,
|
| 86 |
+
"r": 0.7547169811320755,
|
| 87 |
+
"f1": 0.851063829787234,
|
| 88 |
+
"n": 53
|
| 89 |
+
},
|
| 90 |
+
"address-level2": {
|
| 91 |
+
"p": 0.9149659863945578,
|
| 92 |
+
"r": 0.8848684210526315,
|
| 93 |
+
"f1": 0.8996655518394647,
|
| 94 |
+
"n": 304
|
| 95 |
+
},
|
| 96 |
+
"address-level1": {
|
| 97 |
+
"p": 0.9005524861878453,
|
| 98 |
+
"r": 0.8858695652173914,
|
| 99 |
+
"f1": 0.8931506849315068,
|
| 100 |
+
"n": 184
|
| 101 |
+
},
|
| 102 |
+
"postal-code": {
|
| 103 |
+
"p": 0.9731343283582089,
|
| 104 |
+
"r": 0.9673590504451038,
|
| 105 |
+
"f1": 0.9702380952380952,
|
| 106 |
+
"n": 337
|
| 107 |
+
},
|
| 108 |
+
"country": {
|
| 109 |
+
"p": 0.8743169398907104,
|
| 110 |
+
"r": 0.9580838323353293,
|
| 111 |
+
"f1": 0.9142857142857143,
|
| 112 |
+
"n": 167
|
| 113 |
+
},
|
| 114 |
+
"cc-name": {
|
| 115 |
+
"p": 0.9583333333333334,
|
| 116 |
+
"r": 1.0,
|
| 117 |
+
"f1": 0.9787234042553191,
|
| 118 |
+
"n": 23
|
| 119 |
+
},
|
| 120 |
+
"cc-number": {
|
| 121 |
+
"p": 0.9696969696969697,
|
| 122 |
+
"r": 1.0,
|
| 123 |
+
"f1": 0.9846153846153847,
|
| 124 |
+
"n": 32
|
| 125 |
+
},
|
| 126 |
+
"cc-exp": {
|
| 127 |
+
"p": 1.0,
|
| 128 |
+
"r": 1.0,
|
| 129 |
+
"f1": 1.0,
|
| 130 |
+
"n": 18
|
| 131 |
+
},
|
| 132 |
+
"cc-exp-month": {
|
| 133 |
+
"p": 1.0,
|
| 134 |
+
"r": 1.0,
|
| 135 |
+
"f1": 1.0,
|
| 136 |
+
"n": 14
|
| 137 |
+
},
|
| 138 |
+
"cc-exp-year": {
|
| 139 |
+
"p": 1.0,
|
| 140 |
+
"r": 1.0,
|
| 141 |
+
"f1": 1.0,
|
| 142 |
+
"n": 14
|
| 143 |
+
},
|
| 144 |
+
"cc-csc": {
|
| 145 |
+
"p": 1.0,
|
| 146 |
+
"r": 0.967741935483871,
|
| 147 |
+
"f1": 0.9836065573770492,
|
| 148 |
+
"n": 31
|
| 149 |
+
},
|
| 150 |
+
"bday": {
|
| 151 |
+
"p": 0.96,
|
| 152 |
+
"r": 0.9056603773584906,
|
| 153 |
+
"f1": 0.9320388349514563,
|
| 154 |
+
"n": 53
|
| 155 |
+
},
|
| 156 |
+
"search": {
|
| 157 |
+
"p": 0.9670212765957447,
|
| 158 |
+
"r": 0.9588607594936709,
|
| 159 |
+
"f1": 0.9629237288135594,
|
| 160 |
+
"n": 948
|
| 161 |
+
},
|
| 162 |
+
"captcha": {
|
| 163 |
+
"p": 0.9632107023411371,
|
| 164 |
+
"r": 0.9442622950819672,
|
| 165 |
+
"f1": 0.9536423841059603,
|
| 166 |
+
"n": 305
|
| 167 |
+
},
|
| 168 |
+
"other": {
|
| 169 |
+
"p": 0.9693911873528422,
|
| 170 |
+
"r": 0.9666275364749287,
|
| 171 |
+
"f1": 0.9680073893693845,
|
| 172 |
+
"n": 5963
|
| 173 |
+
}
|
| 174 |
+
},
|
| 175 |
+
"per_form": {
|
| 176 |
+
"login": {
|
| 177 |
+
"p": 0.9886914378029079,
|
| 178 |
+
"r": 0.9956616052060737,
|
| 179 |
+
"f1": 0.9921642799243447,
|
| 180 |
+
"n": 1844
|
| 181 |
+
},
|
| 182 |
+
"signup": {
|
| 183 |
+
"p": 0.9777227722772277,
|
| 184 |
+
"r": 0.947242206235012,
|
| 185 |
+
"f1": 0.9622411693057246,
|
| 186 |
+
"n": 417
|
| 187 |
+
},
|
| 188 |
+
"password-change": {
|
| 189 |
+
"p": 0.9154929577464789,
|
| 190 |
+
"r": 0.9848484848484849,
|
| 191 |
+
"f1": 0.948905109489051,
|
| 192 |
+
"n": 66
|
| 193 |
+
},
|
| 194 |
+
"password-recovery": {
|
| 195 |
+
"p": 0.9555555555555556,
|
| 196 |
+
"r": 0.9347826086956522,
|
| 197 |
+
"f1": 0.945054945054945,
|
| 198 |
+
"n": 46
|
| 199 |
+
},
|
| 200 |
+
"otp": {
|
| 201 |
+
"p": 0.9464285714285714,
|
| 202 |
+
"r": 0.9137931034482759,
|
| 203 |
+
"f1": 0.9298245614035087,
|
| 204 |
+
"n": 58
|
| 205 |
+
},
|
| 206 |
+
"payment": {
|
| 207 |
+
"p": 0.9393939393939394,
|
| 208 |
+
"r": 0.8857142857142857,
|
| 209 |
+
"f1": 0.9117647058823529,
|
| 210 |
+
"n": 35
|
| 211 |
+
},
|
| 212 |
+
"address": {
|
| 213 |
+
"p": 0.8048780487804879,
|
| 214 |
+
"r": 0.7857142857142857,
|
| 215 |
+
"f1": 0.7951807228915663,
|
| 216 |
+
"n": 42
|
| 217 |
+
},
|
| 218 |
+
"newsletter": {
|
| 219 |
+
"p": 0.9740458015267176,
|
| 220 |
+
"r": 0.9565217391304348,
|
| 221 |
+
"f1": 0.9652042360060515,
|
| 222 |
+
"n": 667
|
| 223 |
+
},
|
| 224 |
+
"contact": {
|
| 225 |
+
"p": 0.9459141681363904,
|
| 226 |
+
"r": 0.9810975609756097,
|
| 227 |
+
"f1": 0.9631846752469321,
|
| 228 |
+
"n": 1640
|
| 229 |
+
},
|
| 230 |
+
"search": {
|
| 231 |
+
"p": 0.967487684729064,
|
| 232 |
+
"r": 0.9415148609779482,
|
| 233 |
+
"f1": 0.9543245869776482,
|
| 234 |
+
"n": 1043
|
| 235 |
+
},
|
| 236 |
+
"other": {
|
| 237 |
+
"p": 0.9013409961685823,
|
| 238 |
+
"r": 0.8843984962406015,
|
| 239 |
+
"f1": 0.8927893738140418,
|
| 240 |
+
"n": 1064
|
| 241 |
+
}
|
| 242 |
+
},
|
| 243 |
+
"top_confusions": [
|
| 244 |
+
[
|
| 245 |
+
"given-name",
|
| 246 |
+
"name",
|
| 247 |
+
48
|
| 248 |
+
],
|
| 249 |
+
[
|
| 250 |
+
"search",
|
| 251 |
+
"other",
|
| 252 |
+
39
|
| 253 |
+
],
|
| 254 |
+
[
|
| 255 |
+
"address-level2",
|
| 256 |
+
"other",
|
| 257 |
+
28
|
| 258 |
+
],
|
| 259 |
+
[
|
| 260 |
+
"username",
|
| 261 |
+
"email",
|
| 262 |
+
26
|
| 263 |
+
],
|
| 264 |
+
[
|
| 265 |
+
"other",
|
| 266 |
+
"search",
|
| 267 |
+
25
|
| 268 |
+
],
|
| 269 |
+
[
|
| 270 |
+
"other",
|
| 271 |
+
"address-level2",
|
| 272 |
+
22
|
| 273 |
+
],
|
| 274 |
+
[
|
| 275 |
+
"other",
|
| 276 |
+
"country",
|
| 277 |
+
19
|
| 278 |
+
],
|
| 279 |
+
[
|
| 280 |
+
"address-level1",
|
| 281 |
+
"other",
|
| 282 |
+
18
|
| 283 |
+
],
|
| 284 |
+
[
|
| 285 |
+
"name",
|
| 286 |
+
"other",
|
| 287 |
+
15
|
| 288 |
+
],
|
| 289 |
+
[
|
| 290 |
+
"other",
|
| 291 |
+
"address-level1",
|
| 292 |
+
15
|
| 293 |
+
],
|
| 294 |
+
[
|
| 295 |
+
"other",
|
| 296 |
+
"tel",
|
| 297 |
+
15
|
| 298 |
+
],
|
| 299 |
+
[
|
| 300 |
+
"new-password",
|
| 301 |
+
"current-password",
|
| 302 |
+
14
|
| 303 |
+
],
|
| 304 |
+
[
|
| 305 |
+
"captcha",
|
| 306 |
+
"other",
|
| 307 |
+
14
|
| 308 |
+
],
|
| 309 |
+
[
|
| 310 |
+
"other",
|
| 311 |
+
"username",
|
| 312 |
+
13
|
| 313 |
+
],
|
| 314 |
+
[
|
| 315 |
+
"other",
|
| 316 |
+
"name",
|
| 317 |
+
12
|
| 318 |
+
],
|
| 319 |
+
[
|
| 320 |
+
"other",
|
| 321 |
+
"email",
|
| 322 |
+
11
|
| 323 |
+
],
|
| 324 |
+
[
|
| 325 |
+
"address-line2",
|
| 326 |
+
"other",
|
| 327 |
+
11
|
| 328 |
+
],
|
| 329 |
+
[
|
| 330 |
+
"other",
|
| 331 |
+
"street-address",
|
| 332 |
+
11
|
| 333 |
+
],
|
| 334 |
+
[
|
| 335 |
+
"other",
|
| 336 |
+
"current-password",
|
| 337 |
+
10
|
| 338 |
+
],
|
| 339 |
+
[
|
| 340 |
+
"name",
|
| 341 |
+
"given-name",
|
| 342 |
+
10
|
| 343 |
+
],
|
| 344 |
+
[
|
| 345 |
+
"other",
|
| 346 |
+
"captcha",
|
| 347 |
+
9
|
| 348 |
+
],
|
| 349 |
+
[
|
| 350 |
+
"other",
|
| 351 |
+
"postal-code",
|
| 352 |
+
9
|
| 353 |
+
],
|
| 354 |
+
[
|
| 355 |
+
"email",
|
| 356 |
+
"username",
|
| 357 |
+
8
|
| 358 |
+
],
|
| 359 |
+
[
|
| 360 |
+
"email",
|
| 361 |
+
"other",
|
| 362 |
+
8
|
| 363 |
+
],
|
| 364 |
+
[
|
| 365 |
+
"organization",
|
| 366 |
+
"other",
|
| 367 |
+
8
|
| 368 |
+
]
|
| 369 |
+
]
|
| 370 |
+
},
|
| 371 |
+
"model_real": {
|
| 372 |
+
"n_forms": 6838,
|
| 373 |
+
"n_fields": 19811,
|
| 374 |
+
"field_acc": 0.971884306698299,
|
| 375 |
+
"field_macro_f1": 0.9447870267362889,
|
| 376 |
+
"form_acc": 0.9571512138052062,
|
| 377 |
+
"form_macro_f1": 0.9049098319929233,
|
| 378 |
+
"password_precision": 0.9941383352872216,
|
| 379 |
+
"password_recall": 0.9976470588235294,
|
| 380 |
+
"password_f1": 0.9958896065766295,
|
| 381 |
+
"cur_vs_new_acc": 0.991764705882353,
|
| 382 |
+
"identifier_f1": 0.993869133875822,
|
| 383 |
+
"per_field": {
|
| 384 |
+
"username": {
|
| 385 |
+
"p": 0.9780907668231612,
|
| 386 |
+
"r": 0.9735202492211839,
|
| 387 |
+
"f1": 0.975800156128025,
|
| 388 |
+
"n": 1284
|
| 389 |
+
},
|
| 390 |
+
"email": {
|
| 391 |
+
"p": 0.9869402985074627,
|
| 392 |
+
"r": 0.9940494832445975,
|
| 393 |
+
"f1": 0.9904821344983618,
|
| 394 |
+
"n": 3193
|
| 395 |
+
},
|
| 396 |
+
"current-password": {
|
| 397 |
+
"p": 0.9870550161812298,
|
| 398 |
+
"r": 0.9956474428726877,
|
| 399 |
+
"f1": 0.991332611050921,
|
| 400 |
+
"n": 1838
|
| 401 |
+
},
|
| 402 |
+
"new-password": {
|
| 403 |
+
"p": 0.9914893617021276,
|
| 404 |
+
"r": 0.9817415730337079,
|
| 405 |
+
"f1": 0.9865913902611151,
|
| 406 |
+
"n": 712
|
| 407 |
+
},
|
| 408 |
+
"one-time-code": {
|
| 409 |
+
"p": 0.9328358208955224,
|
| 410 |
+
"r": 0.9259259259259259,
|
| 411 |
+
"f1": 0.929368029739777,
|
| 412 |
+
"n": 135
|
| 413 |
+
},
|
| 414 |
+
"tel": {
|
| 415 |
+
"p": 0.9822485207100592,
|
| 416 |
+
"r": 0.9932685115931189,
|
| 417 |
+
"f1": 0.9877277798438081,
|
| 418 |
+
"n": 1337
|
| 419 |
+
},
|
| 420 |
+
"name": {
|
| 421 |
+
"p": 0.9499666444296198,
|
| 422 |
+
"r": 0.974674880219028,
|
| 423 |
+
"f1": 0.9621621621621621,
|
| 424 |
+
"n": 1461
|
| 425 |
+
},
|
| 426 |
+
"given-name": {
|
| 427 |
+
"p": 0.975177304964539,
|
| 428 |
+
"r": 0.912106135986733,
|
| 429 |
+
"f1": 0.9425878320479862,
|
| 430 |
+
"n": 603
|
| 431 |
+
},
|
| 432 |
+
"family-name": {
|
| 433 |
+
"p": 0.988,
|
| 434 |
+
"r": 0.9762845849802372,
|
| 435 |
+
"f1": 0.9821073558648111,
|
| 436 |
+
"n": 506
|
| 437 |
+
},
|
| 438 |
+
"organization": {
|
| 439 |
+
"p": 0.9525691699604744,
|
| 440 |
+
"r": 0.94140625,
|
| 441 |
+
"f1": 0.9469548133595285,
|
| 442 |
+
"n": 256
|
| 443 |
+
},
|
| 444 |
+
"street-address": {
|
| 445 |
+
"p": 0.9357798165137615,
|
| 446 |
+
"r": 0.9807692307692307,
|
| 447 |
+
"f1": 0.9577464788732395,
|
| 448 |
+
"n": 208
|
| 449 |
+
},
|
| 450 |
+
"address-line2": {
|
| 451 |
+
"p": 0.972972972972973,
|
| 452 |
+
"r": 0.75,
|
| 453 |
+
"f1": 0.8470588235294119,
|
| 454 |
+
"n": 48
|
| 455 |
+
},
|
| 456 |
+
"address-level2": {
|
| 457 |
+
"p": 0.911660777385159,
|
| 458 |
+
"r": 0.8835616438356164,
|
| 459 |
+
"f1": 0.8973913043478261,
|
| 460 |
+
"n": 292
|
| 461 |
+
},
|
| 462 |
+
"address-level1": {
|
| 463 |
+
"p": 0.8959537572254336,
|
| 464 |
+
"r": 0.8857142857142857,
|
| 465 |
+
"f1": 0.8908045977011495,
|
| 466 |
+
"n": 175
|
| 467 |
+
},
|
| 468 |
+
"postal-code": {
|
| 469 |
+
"p": 0.9719626168224299,
|
| 470 |
+
"r": 0.968944099378882,
|
| 471 |
+
"f1": 0.9704510108864696,
|
| 472 |
+
"n": 322
|
| 473 |
+
},
|
| 474 |
+
"country": {
|
| 475 |
+
"p": 0.8685714285714285,
|
| 476 |
+
"r": 0.9620253164556962,
|
| 477 |
+
"f1": 0.9129129129129129,
|
| 478 |
+
"n": 158
|
| 479 |
+
},
|
| 480 |
+
"cc-name": {
|
| 481 |
+
"p": 0.6666666666666666,
|
| 482 |
+
"r": 1.0,
|
| 483 |
+
"f1": 0.8,
|
| 484 |
+
"n": 2
|
| 485 |
+
},
|
| 486 |
+
"cc-number": {
|
| 487 |
+
"p": 0.8571428571428571,
|
| 488 |
+
"r": 1.0,
|
| 489 |
+
"f1": 0.923076923076923,
|
| 490 |
+
"n": 6
|
| 491 |
+
},
|
| 492 |
+
"cc-exp-month": {
|
| 493 |
+
"p": 1.0,
|
| 494 |
+
"r": 1.0,
|
| 495 |
+
"f1": 1.0,
|
| 496 |
+
"n": 6
|
| 497 |
+
},
|
| 498 |
+
"cc-exp-year": {
|
| 499 |
+
"p": 1.0,
|
| 500 |
+
"r": 1.0,
|
| 501 |
+
"f1": 1.0,
|
| 502 |
+
"n": 6
|
| 503 |
+
},
|
| 504 |
+
"cc-csc": {
|
| 505 |
+
"p": 1.0,
|
| 506 |
+
"r": 0.8333333333333334,
|
| 507 |
+
"f1": 0.9090909090909091,
|
| 508 |
+
"n": 6
|
| 509 |
+
},
|
| 510 |
+
"bday": {
|
| 511 |
+
"p": 0.96,
|
| 512 |
+
"r": 0.9056603773584906,
|
| 513 |
+
"f1": 0.9320388349514563,
|
| 514 |
+
"n": 53
|
| 515 |
+
},
|
| 516 |
+
"search": {
|
| 517 |
+
"p": 0.9670212765957447,
|
| 518 |
+
"r": 0.9588607594936709,
|
| 519 |
+
"f1": 0.9629237288135594,
|
| 520 |
+
"n": 948
|
| 521 |
+
},
|
| 522 |
+
"captcha": {
|
| 523 |
+
"p": 0.9627118644067797,
|
| 524 |
+
"r": 0.9435215946843853,
|
| 525 |
+
"f1": 0.953020134228188,
|
| 526 |
+
"n": 301
|
| 527 |
+
},
|
| 528 |
+
"other": {
|
| 529 |
+
"p": 0.9695132221660772,
|
| 530 |
+
"r": 0.9665827036104114,
|
| 531 |
+
"f1": 0.9680457450386813,
|
| 532 |
+
"n": 5955
|
| 533 |
+
}
|
| 534 |
+
},
|
| 535 |
+
"per_form": {
|
| 536 |
+
"login": {
|
| 537 |
+
"p": 0.9891774891774892,
|
| 538 |
+
"r": 0.9956427015250545,
|
| 539 |
+
"f1": 0.992399565689468,
|
| 540 |
+
"n": 1836
|
| 541 |
+
},
|
| 542 |
+
"signup": {
|
| 543 |
+
"p": 0.9777227722772277,
|
| 544 |
+
"r": 0.947242206235012,
|
| 545 |
+
"f1": 0.9622411693057246,
|
| 546 |
+
"n": 417
|
| 547 |
+
},
|
| 548 |
+
"password-change": {
|
| 549 |
+
"p": 0.8928571428571429,
|
| 550 |
+
"r": 1.0,
|
| 551 |
+
"f1": 0.9433962264150945,
|
| 552 |
+
"n": 50
|
| 553 |
+
},
|
| 554 |
+
"password-recovery": {
|
| 555 |
+
"p": 0.9459459459459459,
|
| 556 |
+
"r": 0.9210526315789473,
|
| 557 |
+
"f1": 0.9333333333333332,
|
| 558 |
+
"n": 38
|
| 559 |
+
},
|
| 560 |
+
"otp": {
|
| 561 |
+
"p": 0.9210526315789473,
|
| 562 |
+
"r": 0.875,
|
| 563 |
+
"f1": 0.8974358974358975,
|
| 564 |
+
"n": 40
|
| 565 |
+
},
|
| 566 |
+
"payment": {
|
| 567 |
+
"p": 0.7142857142857143,
|
| 568 |
+
"r": 0.625,
|
| 569 |
+
"f1": 0.6666666666666666,
|
| 570 |
+
"n": 8
|
| 571 |
+
},
|
| 572 |
+
"address": {
|
| 573 |
+
"p": 0.7941176470588235,
|
| 574 |
+
"r": 0.7714285714285715,
|
| 575 |
+
"f1": 0.782608695652174,
|
| 576 |
+
"n": 35
|
| 577 |
+
},
|
| 578 |
+
"newsletter": {
|
| 579 |
+
"p": 0.9740458015267176,
|
| 580 |
+
"r": 0.9565217391304348,
|
| 581 |
+
"f1": 0.9652042360060515,
|
| 582 |
+
"n": 667
|
| 583 |
+
},
|
| 584 |
+
"contact": {
|
| 585 |
+
"p": 0.9459141681363904,
|
| 586 |
+
"r": 0.9810975609756097,
|
| 587 |
+
"f1": 0.9631846752469321,
|
| 588 |
+
"n": 1640
|
| 589 |
+
},
|
| 590 |
+
"search": {
|
| 591 |
+
"p": 0.967487684729064,
|
| 592 |
+
"r": 0.9415148609779482,
|
| 593 |
+
"f1": 0.9543245869776482,
|
| 594 |
+
"n": 1043
|
| 595 |
+
},
|
| 596 |
+
"other": {
|
| 597 |
+
"p": 0.9022051773729626,
|
| 598 |
+
"r": 0.8843984962406015,
|
| 599 |
+
"f1": 0.8932130991931656,
|
| 600 |
+
"n": 1064
|
| 601 |
+
}
|
| 602 |
+
},
|
| 603 |
+
"top_confusions": [
|
| 604 |
+
[
|
| 605 |
+
"given-name",
|
| 606 |
+
"name",
|
| 607 |
+
48
|
| 608 |
+
],
|
| 609 |
+
[
|
| 610 |
+
"search",
|
| 611 |
+
"other",
|
| 612 |
+
39
|
| 613 |
+
],
|
| 614 |
+
[
|
| 615 |
+
"address-level2",
|
| 616 |
+
"other",
|
| 617 |
+
27
|
| 618 |
+
],
|
| 619 |
+
[
|
| 620 |
+
"username",
|
| 621 |
+
"email",
|
| 622 |
+
26
|
| 623 |
+
],
|
| 624 |
+
[
|
| 625 |
+
"other",
|
| 626 |
+
"search",
|
| 627 |
+
25
|
| 628 |
+
],
|
| 629 |
+
[
|
| 630 |
+
"other",
|
| 631 |
+
"address-level2",
|
| 632 |
+
22
|
| 633 |
+
],
|
| 634 |
+
[
|
| 635 |
+
"other",
|
| 636 |
+
"country",
|
| 637 |
+
19
|
| 638 |
+
],
|
| 639 |
+
[
|
| 640 |
+
"address-level1",
|
| 641 |
+
"other",
|
| 642 |
+
18
|
| 643 |
+
],
|
| 644 |
+
[
|
| 645 |
+
"name",
|
| 646 |
+
"other",
|
| 647 |
+
15
|
| 648 |
+
],
|
| 649 |
+
[
|
| 650 |
+
"other",
|
| 651 |
+
"address-level1",
|
| 652 |
+
15
|
| 653 |
+
],
|
| 654 |
+
[
|
| 655 |
+
"other",
|
| 656 |
+
"tel",
|
| 657 |
+
15
|
| 658 |
+
],
|
| 659 |
+
[
|
| 660 |
+
"captcha",
|
| 661 |
+
"other",
|
| 662 |
+
14
|
| 663 |
+
],
|
| 664 |
+
[
|
| 665 |
+
"new-password",
|
| 666 |
+
"current-password",
|
| 667 |
+
13
|
| 668 |
+
],
|
| 669 |
+
[
|
| 670 |
+
"other",
|
| 671 |
+
"username",
|
| 672 |
+
13
|
| 673 |
+
],
|
| 674 |
+
[
|
| 675 |
+
"other",
|
| 676 |
+
"name",
|
| 677 |
+
12
|
| 678 |
+
],
|
| 679 |
+
[
|
| 680 |
+
"other",
|
| 681 |
+
"email",
|
| 682 |
+
11
|
| 683 |
+
],
|
| 684 |
+
[
|
| 685 |
+
"address-line2",
|
| 686 |
+
"other",
|
| 687 |
+
11
|
| 688 |
+
],
|
| 689 |
+
[
|
| 690 |
+
"other",
|
| 691 |
+
"street-address",
|
| 692 |
+
11
|
| 693 |
+
],
|
| 694 |
+
[
|
| 695 |
+
"other",
|
| 696 |
+
"current-password",
|
| 697 |
+
10
|
| 698 |
+
],
|
| 699 |
+
[
|
| 700 |
+
"name",
|
| 701 |
+
"given-name",
|
| 702 |
+
10
|
| 703 |
+
],
|
| 704 |
+
[
|
| 705 |
+
"other",
|
| 706 |
+
"captcha",
|
| 707 |
+
9
|
| 708 |
+
],
|
| 709 |
+
[
|
| 710 |
+
"other",
|
| 711 |
+
"postal-code",
|
| 712 |
+
9
|
| 713 |
+
],
|
| 714 |
+
[
|
| 715 |
+
"email",
|
| 716 |
+
"username",
|
| 717 |
+
8
|
| 718 |
+
],
|
| 719 |
+
[
|
| 720 |
+
"email",
|
| 721 |
+
"other",
|
| 722 |
+
8
|
| 723 |
+
],
|
| 724 |
+
[
|
| 725 |
+
"organization",
|
| 726 |
+
"other",
|
| 727 |
+
8
|
| 728 |
+
]
|
| 729 |
+
]
|
| 730 |
+
},
|
| 731 |
+
"model_synthetic": {
|
| 732 |
+
"n_forms": 84,
|
| 733 |
+
"n_fields": 319,
|
| 734 |
+
"field_acc": 0.9780564263322884,
|
| 735 |
+
"field_macro_f1": 0.9765257496746269,
|
| 736 |
+
"form_acc": 0.9642857142857143,
|
| 737 |
+
"form_macro_f1": 0.9578655564477768,
|
| 738 |
+
"password_precision": 1.0,
|
| 739 |
+
"password_recall": 1.0,
|
| 740 |
+
"password_f1": 1.0,
|
| 741 |
+
"cur_vs_new_acc": 0.9743589743589743,
|
| 742 |
+
"identifier_f1": 1.0,
|
| 743 |
+
"per_field": {
|
| 744 |
+
"username": {
|
| 745 |
+
"p": 1.0,
|
| 746 |
+
"r": 1.0,
|
| 747 |
+
"f1": 1.0,
|
| 748 |
+
"n": 7
|
| 749 |
+
},
|
| 750 |
+
"email": {
|
| 751 |
+
"p": 1.0,
|
| 752 |
+
"r": 1.0,
|
| 753 |
+
"f1": 1.0,
|
| 754 |
+
"n": 16
|
| 755 |
+
},
|
| 756 |
+
"current-password": {
|
| 757 |
+
"p": 0.9333333333333333,
|
| 758 |
+
"r": 1.0,
|
| 759 |
+
"f1": 0.9655172413793104,
|
| 760 |
+
"n": 14
|
| 761 |
+
},
|
| 762 |
+
"new-password": {
|
| 763 |
+
"p": 1.0,
|
| 764 |
+
"r": 0.96,
|
| 765 |
+
"f1": 0.9795918367346939,
|
| 766 |
+
"n": 25
|
| 767 |
+
},
|
| 768 |
+
"one-time-code": {
|
| 769 |
+
"p": 0.9285714285714286,
|
| 770 |
+
"r": 1.0,
|
| 771 |
+
"f1": 0.962962962962963,
|
| 772 |
+
"n": 39
|
| 773 |
+
},
|
| 774 |
+
"tel": {
|
| 775 |
+
"p": 0.8666666666666667,
|
| 776 |
+
"r": 1.0,
|
| 777 |
+
"f1": 0.9285714285714286,
|
| 778 |
+
"n": 13
|
| 779 |
+
},
|
| 780 |
+
"name": {
|
| 781 |
+
"p": 1.0,
|
| 782 |
+
"r": 1.0,
|
| 783 |
+
"f1": 1.0,
|
| 784 |
+
"n": 6
|
| 785 |
+
},
|
| 786 |
+
"given-name": {
|
| 787 |
+
"p": 1.0,
|
| 788 |
+
"r": 1.0,
|
| 789 |
+
"f1": 1.0,
|
| 790 |
+
"n": 6
|
| 791 |
+
},
|
| 792 |
+
"family-name": {
|
| 793 |
+
"p": 1.0,
|
| 794 |
+
"r": 1.0,
|
| 795 |
+
"f1": 1.0,
|
| 796 |
+
"n": 6
|
| 797 |
+
},
|
| 798 |
+
"organization": {
|
| 799 |
+
"p": 1.0,
|
| 800 |
+
"r": 1.0,
|
| 801 |
+
"f1": 1.0,
|
| 802 |
+
"n": 4
|
| 803 |
+
},
|
| 804 |
+
"street-address": {
|
| 805 |
+
"p": 1.0,
|
| 806 |
+
"r": 0.9333333333333333,
|
| 807 |
+
"f1": 0.9655172413793104,
|
| 808 |
+
"n": 15
|
| 809 |
+
},
|
| 810 |
+
"address-line2": {
|
| 811 |
+
"p": 1.0,
|
| 812 |
+
"r": 0.8,
|
| 813 |
+
"f1": 0.888888888888889,
|
| 814 |
+
"n": 5
|
| 815 |
+
},
|
| 816 |
+
"address-level2": {
|
| 817 |
+
"p": 1.0,
|
| 818 |
+
"r": 0.9166666666666666,
|
| 819 |
+
"f1": 0.9565217391304348,
|
| 820 |
+
"n": 12
|
| 821 |
+
},
|
| 822 |
+
"address-level1": {
|
| 823 |
+
"p": 1.0,
|
| 824 |
+
"r": 0.8888888888888888,
|
| 825 |
+
"f1": 0.9411764705882353,
|
| 826 |
+
"n": 9
|
| 827 |
+
},
|
| 828 |
+
"postal-code": {
|
| 829 |
+
"p": 1.0,
|
| 830 |
+
"r": 0.9333333333333333,
|
| 831 |
+
"f1": 0.9655172413793104,
|
| 832 |
+
"n": 15
|
| 833 |
+
},
|
| 834 |
+
"country": {
|
| 835 |
+
"p": 1.0,
|
| 836 |
+
"r": 0.8888888888888888,
|
| 837 |
+
"f1": 0.9411764705882353,
|
| 838 |
+
"n": 9
|
| 839 |
+
},
|
| 840 |
+
"cc-name": {
|
| 841 |
+
"p": 1.0,
|
| 842 |
+
"r": 1.0,
|
| 843 |
+
"f1": 1.0,
|
| 844 |
+
"n": 21
|
| 845 |
+
},
|
| 846 |
+
"cc-number": {
|
| 847 |
+
"p": 1.0,
|
| 848 |
+
"r": 1.0,
|
| 849 |
+
"f1": 1.0,
|
| 850 |
+
"n": 26
|
| 851 |
+
},
|
| 852 |
+
"cc-exp": {
|
| 853 |
+
"p": 1.0,
|
| 854 |
+
"r": 1.0,
|
| 855 |
+
"f1": 1.0,
|
| 856 |
+
"n": 18
|
| 857 |
+
},
|
| 858 |
+
"cc-exp-month": {
|
| 859 |
+
"p": 1.0,
|
| 860 |
+
"r": 1.0,
|
| 861 |
+
"f1": 1.0,
|
| 862 |
+
"n": 8
|
| 863 |
+
},
|
| 864 |
+
"cc-exp-year": {
|
| 865 |
+
"p": 1.0,
|
| 866 |
+
"r": 1.0,
|
| 867 |
+
"f1": 1.0,
|
| 868 |
+
"n": 8
|
| 869 |
+
},
|
| 870 |
+
"cc-csc": {
|
| 871 |
+
"p": 1.0,
|
| 872 |
+
"r": 1.0,
|
| 873 |
+
"f1": 1.0,
|
| 874 |
+
"n": 25
|
| 875 |
+
},
|
| 876 |
+
"captcha": {
|
| 877 |
+
"p": 1.0,
|
| 878 |
+
"r": 1.0,
|
| 879 |
+
"f1": 1.0,
|
| 880 |
+
"n": 4
|
| 881 |
+
},
|
| 882 |
+
"other": {
|
| 883 |
+
"p": 0.8888888888888888,
|
| 884 |
+
"r": 1.0,
|
| 885 |
+
"f1": 0.9411764705882353,
|
| 886 |
+
"n": 8
|
| 887 |
+
}
|
| 888 |
+
},
|
| 889 |
+
"per_form": {
|
| 890 |
+
"login": {
|
| 891 |
+
"p": 0.8888888888888888,
|
| 892 |
+
"r": 1.0,
|
| 893 |
+
"f1": 0.9411764705882353,
|
| 894 |
+
"n": 8
|
| 895 |
+
},
|
| 896 |
+
"password-change": {
|
| 897 |
+
"p": 1.0,
|
| 898 |
+
"r": 0.9375,
|
| 899 |
+
"f1": 0.967741935483871,
|
| 900 |
+
"n": 16
|
| 901 |
+
},
|
| 902 |
+
"password-recovery": {
|
| 903 |
+
"p": 1.0,
|
| 904 |
+
"r": 1.0,
|
| 905 |
+
"f1": 1.0,
|
| 906 |
+
"n": 8
|
| 907 |
+
},
|
| 908 |
+
"otp": {
|
| 909 |
+
"p": 1.0,
|
| 910 |
+
"r": 1.0,
|
| 911 |
+
"f1": 1.0,
|
| 912 |
+
"n": 18
|
| 913 |
+
},
|
| 914 |
+
"payment": {
|
| 915 |
+
"p": 1.0,
|
| 916 |
+
"r": 0.9629629629629629,
|
| 917 |
+
"f1": 0.9811320754716981,
|
| 918 |
+
"n": 27
|
| 919 |
+
},
|
| 920 |
+
"address": {
|
| 921 |
+
"p": 0.8571428571428571,
|
| 922 |
+
"r": 0.8571428571428571,
|
| 923 |
+
"f1": 0.8571428571428571,
|
| 924 |
+
"n": 7
|
| 925 |
+
}
|
| 926 |
+
},
|
| 927 |
+
"top_confusions": [
|
| 928 |
+
[
|
| 929 |
+
"new-password",
|
| 930 |
+
"current-password",
|
| 931 |
+
1
|
| 932 |
+
],
|
| 933 |
+
[
|
| 934 |
+
"street-address",
|
| 935 |
+
"tel",
|
| 936 |
+
1
|
| 937 |
+
],
|
| 938 |
+
[
|
| 939 |
+
"address-line2",
|
| 940 |
+
"one-time-code",
|
| 941 |
+
1
|
| 942 |
+
],
|
| 943 |
+
[
|
| 944 |
+
"address-level2",
|
| 945 |
+
"other",
|
| 946 |
+
1
|
| 947 |
+
],
|
| 948 |
+
[
|
| 949 |
+
"address-level1",
|
| 950 |
+
"one-time-code",
|
| 951 |
+
1
|
| 952 |
+
],
|
| 953 |
+
[
|
| 954 |
+
"postal-code",
|
| 955 |
+
"one-time-code",
|
| 956 |
+
1
|
| 957 |
+
],
|
| 958 |
+
[
|
| 959 |
+
"country",
|
| 960 |
+
"tel",
|
| 961 |
+
1
|
| 962 |
+
]
|
| 963 |
+
]
|
| 964 |
+
},
|
| 965 |
+
"heuristic_baseline_real": {
|
| 966 |
+
"n_forms": 6838,
|
| 967 |
+
"n_fields": 19811,
|
| 968 |
+
"field_acc": 0.827368633587401,
|
| 969 |
+
"field_macro_f1": 0.7233335250918939,
|
| 970 |
+
"form_acc": 0.7157063468850541,
|
| 971 |
+
"form_macro_f1": 0.48529320010648663,
|
| 972 |
+
"password_precision": 0.9391143911439115,
|
| 973 |
+
"password_recall": 0.9980392156862745,
|
| 974 |
+
"password_f1": 0.967680608365019,
|
| 975 |
+
"cur_vs_new_acc": 0.8545098039215686,
|
| 976 |
+
"identifier_f1": 0.9453604914322663,
|
| 977 |
+
"per_field": {
|
| 978 |
+
"username": {
|
| 979 |
+
"p": 0.7456213511259383,
|
| 980 |
+
"r": 0.6962616822429907,
|
| 981 |
+
"f1": 0.7200966572694322,
|
| 982 |
+
"n": 1284
|
| 983 |
+
},
|
| 984 |
+
"email": {
|
| 985 |
+
"p": 0.8662225922842076,
|
| 986 |
+
"r": 0.9774506733479487,
|
| 987 |
+
"f1": 0.918481459682166,
|
| 988 |
+
"n": 3193
|
| 989 |
+
},
|
| 990 |
+
"current-password": {
|
| 991 |
+
"p": 0.8279176201372997,
|
| 992 |
+
"r": 0.984221980413493,
|
| 993 |
+
"f1": 0.8993288590604027,
|
| 994 |
+
"n": 1838
|
| 995 |
+
},
|
| 996 |
+
"new-password": {
|
| 997 |
+
"p": 0.7047619047619048,
|
| 998 |
+
"r": 0.5196629213483146,
|
| 999 |
+
"f1": 0.5982215036378336,
|
| 1000 |
+
"n": 712
|
| 1001 |
+
},
|
| 1002 |
+
"one-time-code": {
|
| 1003 |
+
"p": 0.6486486486486487,
|
| 1004 |
+
"r": 0.35555555555555557,
|
| 1005 |
+
"f1": 0.45933014354066987,
|
| 1006 |
+
"n": 135
|
| 1007 |
+
},
|
| 1008 |
+
"tel": {
|
| 1009 |
+
"p": 0.8867791842475387,
|
| 1010 |
+
"r": 0.9431563201196709,
|
| 1011 |
+
"f1": 0.91409931134469,
|
| 1012 |
+
"n": 1337
|
| 1013 |
+
},
|
| 1014 |
+
"name": {
|
| 1015 |
+
"p": 0.8157695223654283,
|
| 1016 |
+
"r": 0.7364818617385352,
|
| 1017 |
+
"f1": 0.7741007194244605,
|
| 1018 |
+
"n": 1461
|
| 1019 |
+
},
|
| 1020 |
+
"given-name": {
|
| 1021 |
+
"p": 0.6842105263157895,
|
| 1022 |
+
"r": 0.7114427860696517,
|
| 1023 |
+
"f1": 0.697560975609756,
|
| 1024 |
+
"n": 603
|
| 1025 |
+
},
|
| 1026 |
+
"family-name": {
|
| 1027 |
+
"p": 0.9160671462829736,
|
| 1028 |
+
"r": 0.7549407114624506,
|
| 1029 |
+
"f1": 0.8277356446370531,
|
| 1030 |
+
"n": 506
|
| 1031 |
+
},
|
| 1032 |
+
"organization": {
|
| 1033 |
+
"p": 0.8041666666666667,
|
| 1034 |
+
"r": 0.75390625,
|
| 1035 |
+
"f1": 0.778225806451613,
|
| 1036 |
+
"n": 256
|
| 1037 |
+
},
|
| 1038 |
+
"street-address": {
|
| 1039 |
+
"p": 0.6370370370370371,
|
| 1040 |
+
"r": 0.8269230769230769,
|
| 1041 |
+
"f1": 0.7196652719665272,
|
| 1042 |
+
"n": 208
|
| 1043 |
+
},
|
| 1044 |
+
"address-line2": {
|
| 1045 |
+
"p": 0.453125,
|
| 1046 |
+
"r": 0.6041666666666666,
|
| 1047 |
+
"f1": 0.5178571428571429,
|
| 1048 |
+
"n": 48
|
| 1049 |
+
},
|
| 1050 |
+
"address-level2": {
|
| 1051 |
+
"p": 0.6318537859007833,
|
| 1052 |
+
"r": 0.8287671232876712,
|
| 1053 |
+
"f1": 0.717037037037037,
|
| 1054 |
+
"n": 292
|
| 1055 |
+
},
|
| 1056 |
+
"address-level1": {
|
| 1057 |
+
"p": 0.5258964143426295,
|
| 1058 |
+
"r": 0.7542857142857143,
|
| 1059 |
+
"f1": 0.6197183098591549,
|
| 1060 |
+
"n": 175
|
| 1061 |
+
},
|
| 1062 |
+
"postal-code": {
|
| 1063 |
+
"p": 0.925,
|
| 1064 |
+
"r": 0.8043478260869565,
|
| 1065 |
+
"f1": 0.8604651162790697,
|
| 1066 |
+
"n": 322
|
| 1067 |
+
},
|
| 1068 |
+
"country": {
|
| 1069 |
+
"p": 0.8226950354609929,
|
| 1070 |
+
"r": 0.7341772151898734,
|
| 1071 |
+
"f1": 0.7759197324414716,
|
| 1072 |
+
"n": 158
|
| 1073 |
+
},
|
| 1074 |
+
"cc-name": {
|
| 1075 |
+
"p": 0.5,
|
| 1076 |
+
"r": 1.0,
|
| 1077 |
+
"f1": 0.6666666666666666,
|
| 1078 |
+
"n": 2
|
| 1079 |
+
},
|
| 1080 |
+
"cc-number": {
|
| 1081 |
+
"p": 0.5,
|
| 1082 |
+
"r": 1.0,
|
| 1083 |
+
"f1": 0.6666666666666666,
|
| 1084 |
+
"n": 6
|
| 1085 |
+
},
|
| 1086 |
+
"cc-exp-month": {
|
| 1087 |
+
"p": 1.0,
|
| 1088 |
+
"r": 0.5,
|
| 1089 |
+
"f1": 0.6666666666666666,
|
| 1090 |
+
"n": 6
|
| 1091 |
+
},
|
| 1092 |
+
"cc-exp-year": {
|
| 1093 |
+
"p": 0.6,
|
| 1094 |
+
"r": 0.5,
|
| 1095 |
+
"f1": 0.5454545454545454,
|
| 1096 |
+
"n": 6
|
| 1097 |
+
},
|
| 1098 |
+
"cc-csc": {
|
| 1099 |
+
"p": 0.4166666666666667,
|
| 1100 |
+
"r": 0.8333333333333334,
|
| 1101 |
+
"f1": 0.5555555555555556,
|
| 1102 |
+
"n": 6
|
| 1103 |
+
},
|
| 1104 |
+
"bday": {
|
| 1105 |
+
"p": 0.9318181818181818,
|
| 1106 |
+
"r": 0.7735849056603774,
|
| 1107 |
+
"f1": 0.8453608247422681,
|
| 1108 |
+
"n": 53
|
| 1109 |
+
},
|
| 1110 |
+
"search": {
|
| 1111 |
+
"p": 0.8284714119019837,
|
| 1112 |
+
"r": 0.7489451476793249,
|
| 1113 |
+
"f1": 0.7867036011080333,
|
| 1114 |
+
"n": 948
|
| 1115 |
+
},
|
| 1116 |
+
"captcha": {
|
| 1117 |
+
"p": 0.9297297297297298,
|
| 1118 |
+
"r": 0.5714285714285714,
|
| 1119 |
+
"f1": 0.7078189300411523,
|
| 1120 |
+
"n": 301
|
| 1121 |
+
},
|
| 1122 |
+
"other": {
|
| 1123 |
+
"p": 0.8645796693633486,
|
| 1124 |
+
"r": 0.8255247691015953,
|
| 1125 |
+
"f1": 0.8446009792973114,
|
| 1126 |
+
"n": 5955
|
| 1127 |
+
}
|
| 1128 |
+
},
|
| 1129 |
+
"per_form": {
|
| 1130 |
+
"login": {
|
| 1131 |
+
"p": 0.94069640914037,
|
| 1132 |
+
"r": 0.9417211328976035,
|
| 1133 |
+
"f1": 0.9412084921066958,
|
| 1134 |
+
"n": 1836
|
| 1135 |
+
},
|
| 1136 |
+
"signup": {
|
| 1137 |
+
"p": 0.5643224699828473,
|
| 1138 |
+
"r": 0.7889688249400479,
|
| 1139 |
+
"f1": 0.6579999999999999,
|
| 1140 |
+
"n": 417
|
| 1141 |
+
},
|
| 1142 |
+
"password-change": {
|
| 1143 |
+
"p": 0.0,
|
| 1144 |
+
"r": 0.0,
|
| 1145 |
+
"f1": 0.0,
|
| 1146 |
+
"n": 50
|
| 1147 |
+
},
|
| 1148 |
+
"password-recovery": {
|
| 1149 |
+
"p": 0.26666666666666666,
|
| 1150 |
+
"r": 0.21052631578947367,
|
| 1151 |
+
"f1": 0.23529411764705882,
|
| 1152 |
+
"n": 38
|
| 1153 |
+
},
|
| 1154 |
+
"otp": {
|
| 1155 |
+
"p": 0.3,
|
| 1156 |
+
"r": 0.375,
|
| 1157 |
+
"f1": 0.33333333333333326,
|
| 1158 |
+
"n": 40
|
| 1159 |
+
},
|
| 1160 |
+
"payment": {
|
| 1161 |
+
"p": 0.21052631578947367,
|
| 1162 |
+
"r": 0.5,
|
| 1163 |
+
"f1": 0.2962962962962963,
|
| 1164 |
+
"n": 8
|
| 1165 |
+
},
|
| 1166 |
+
"address": {
|
| 1167 |
+
"p": 0.03722504230118443,
|
| 1168 |
+
"r": 0.6285714285714286,
|
| 1169 |
+
"f1": 0.07028753993610225,
|
| 1170 |
+
"n": 35
|
| 1171 |
+
},
|
| 1172 |
+
"newsletter": {
|
| 1173 |
+
"p": 0.8864774624373957,
|
| 1174 |
+
"r": 0.7961019490254873,
|
| 1175 |
+
"f1": 0.8388625592417062,
|
| 1176 |
+
"n": 667
|
| 1177 |
+
},
|
| 1178 |
+
"contact": {
|
| 1179 |
+
"p": 0.9479371316306483,
|
| 1180 |
+
"r": 0.5884146341463414,
|
| 1181 |
+
"f1": 0.7261098570353649,
|
| 1182 |
+
"n": 1640
|
| 1183 |
+
},
|
| 1184 |
+
"search": {
|
| 1185 |
+
"p": 0.984313725490196,
|
| 1186 |
+
"r": 0.4813039309683605,
|
| 1187 |
+
"f1": 0.6464906632324533,
|
| 1188 |
+
"n": 1043
|
| 1189 |
+
},
|
| 1190 |
+
"other": {
|
| 1191 |
+
"p": 0.493125,
|
| 1192 |
+
"r": 0.7415413533834586,
|
| 1193 |
+
"f1": 0.5923423423423424,
|
| 1194 |
+
"n": 1064
|
| 1195 |
+
}
|
| 1196 |
+
},
|
| 1197 |
+
"top_confusions": [
|
| 1198 |
+
[
|
| 1199 |
+
"new-password",
|
| 1200 |
+
"current-password",
|
| 1201 |
+
339
|
| 1202 |
+
],
|
| 1203 |
+
[
|
| 1204 |
+
"username",
|
| 1205 |
+
"email",
|
| 1206 |
+
309
|
| 1207 |
+
],
|
| 1208 |
+
[
|
| 1209 |
+
"name",
|
| 1210 |
+
"given-name",
|
| 1211 |
+
148
|
| 1212 |
+
],
|
| 1213 |
+
[
|
| 1214 |
+
"name",
|
| 1215 |
+
"other",
|
| 1216 |
+
135
|
| 1217 |
+
],
|
| 1218 |
+
[
|
| 1219 |
+
"other",
|
| 1220 |
+
"search",
|
| 1221 |
+
134
|
| 1222 |
+
],
|
| 1223 |
+
[
|
| 1224 |
+
"search",
|
| 1225 |
+
"other",
|
| 1226 |
+
128
|
| 1227 |
+
],
|
| 1228 |
+
[
|
| 1229 |
+
"other",
|
| 1230 |
+
"email",
|
| 1231 |
+
118
|
| 1232 |
+
],
|
| 1233 |
+
[
|
| 1234 |
+
"other",
|
| 1235 |
+
"new-password",
|
| 1236 |
+
117
|
| 1237 |
+
],
|
| 1238 |
+
[
|
| 1239 |
+
"other",
|
| 1240 |
+
"address-level2",
|
| 1241 |
+
108
|
| 1242 |
+
],
|
| 1243 |
+
[
|
| 1244 |
+
"captcha",
|
| 1245 |
+
"other",
|
| 1246 |
+
100
|
| 1247 |
+
],
|
| 1248 |
+
[
|
| 1249 |
+
"given-name",
|
| 1250 |
+
"name",
|
| 1251 |
+
91
|
| 1252 |
+
],
|
| 1253 |
+
[
|
| 1254 |
+
"other",
|
| 1255 |
+
"name",
|
| 1256 |
+
89
|
| 1257 |
+
],
|
| 1258 |
+
[
|
| 1259 |
+
"other",
|
| 1260 |
+
"username",
|
| 1261 |
+
84
|
| 1262 |
+
],
|
| 1263 |
+
[
|
| 1264 |
+
"other",
|
| 1265 |
+
"tel",
|
| 1266 |
+
84
|
| 1267 |
+
],
|
| 1268 |
+
[
|
| 1269 |
+
"other",
|
| 1270 |
+
"address-level1",
|
| 1271 |
+
82
|
| 1272 |
+
],
|
| 1273 |
+
[
|
| 1274 |
+
"family-name",
|
| 1275 |
+
"other",
|
| 1276 |
+
63
|
| 1277 |
+
],
|
| 1278 |
+
[
|
| 1279 |
+
"email",
|
| 1280 |
+
"username",
|
| 1281 |
+
62
|
| 1282 |
+
],
|
| 1283 |
+
[
|
| 1284 |
+
"tel",
|
| 1285 |
+
"other",
|
| 1286 |
+
54
|
| 1287 |
+
],
|
| 1288 |
+
[
|
| 1289 |
+
"name",
|
| 1290 |
+
"username",
|
| 1291 |
+
52
|
| 1292 |
+
],
|
| 1293 |
+
[
|
| 1294 |
+
"username",
|
| 1295 |
+
"other",
|
| 1296 |
+
49
|
| 1297 |
+
],
|
| 1298 |
+
[
|
| 1299 |
+
"given-name",
|
| 1300 |
+
"other",
|
| 1301 |
+
39
|
| 1302 |
+
],
|
| 1303 |
+
[
|
| 1304 |
+
"other",
|
| 1305 |
+
"street-address",
|
| 1306 |
+
39
|
| 1307 |
+
],
|
| 1308 |
+
[
|
| 1309 |
+
"one-time-code",
|
| 1310 |
+
"other",
|
| 1311 |
+
39
|
| 1312 |
+
],
|
| 1313 |
+
[
|
| 1314 |
+
"other",
|
| 1315 |
+
"current-password",
|
| 1316 |
+
32
|
| 1317 |
+
],
|
| 1318 |
+
[
|
| 1319 |
+
"other",
|
| 1320 |
+
"address-line2",
|
| 1321 |
+
31
|
| 1322 |
+
]
|
| 1323 |
+
]
|
| 1324 |
+
},
|
| 1325 |
+
"autocomplete_masked": {
|
| 1326 |
+
"n_fields": 1622,
|
| 1327 |
+
"acc": 0.9784217016029593,
|
| 1328 |
+
"per_class": {
|
| 1329 |
+
"address-level1": "4/4",
|
| 1330 |
+
"address-level2": "12/14",
|
| 1331 |
+
"address-line2": "0/3",
|
| 1332 |
+
"bday": "5/5",
|
| 1333 |
+
"cc-csc": "1/1",
|
| 1334 |
+
"cc-exp-month": "1/1",
|
| 1335 |
+
"cc-exp-year": "1/1",
|
| 1336 |
+
"cc-number": "1/1",
|
| 1337 |
+
"country": "6/6",
|
| 1338 |
+
"current-password": "323/324",
|
| 1339 |
+
"email": "417/418",
|
| 1340 |
+
"family-name": "34/37",
|
| 1341 |
+
"given-name": "39/42",
|
| 1342 |
+
"name": "182/187",
|
| 1343 |
+
"new-password": "180/180",
|
| 1344 |
+
"one-time-code": "7/7",
|
| 1345 |
+
"organization": "16/16",
|
| 1346 |
+
"postal-code": "26/30",
|
| 1347 |
+
"street-address": "11/13",
|
| 1348 |
+
"tel": "111/111",
|
| 1349 |
+
"username": "210/221"
|
| 1350 |
+
},
|
| 1351 |
+
"heuristic_acc": 0.7866831072749692
|
| 1352 |
+
}
|
| 1353 |
+
}
|
eval/eval_test_float32.json
ADDED
|
@@ -0,0 +1,1353 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"split": "test",
|
| 3 |
+
"dtype": "float32",
|
| 4 |
+
"forms_per_s_gpu": 3416.2872663427547,
|
| 5 |
+
"model": {
|
| 6 |
+
"n_forms": 6922,
|
| 7 |
+
"n_fields": 20130,
|
| 8 |
+
"field_acc": 0.9719324391455539,
|
| 9 |
+
"field_macro_f1": 0.9597248737921183,
|
| 10 |
+
"form_acc": 0.9569488587113552,
|
| 11 |
+
"form_macro_f1": 0.932616790369714,
|
| 12 |
+
"password_precision": 0.9942263279445728,
|
| 13 |
+
"password_recall": 0.9976825028968713,
|
| 14 |
+
"password_f1": 0.9959514170040485,
|
| 15 |
+
"cur_vs_new_acc": 0.9915025106218617,
|
| 16 |
+
"identifier_f1": 0.9939004103360319,
|
| 17 |
+
"per_field": {
|
| 18 |
+
"username": {
|
| 19 |
+
"p": 0.9782101167315175,
|
| 20 |
+
"r": 0.9736638264910922,
|
| 21 |
+
"f1": 0.9759316770186335,
|
| 22 |
+
"n": 1291
|
| 23 |
+
},
|
| 24 |
+
"email": {
|
| 25 |
+
"p": 0.9870049504950495,
|
| 26 |
+
"r": 0.9940791523839202,
|
| 27 |
+
"f1": 0.9905294208973761,
|
| 28 |
+
"n": 3209
|
| 29 |
+
},
|
| 30 |
+
"current-password": {
|
| 31 |
+
"p": 0.9866238630283574,
|
| 32 |
+
"r": 0.9956803455723542,
|
| 33 |
+
"f1": 0.9911314162859447,
|
| 34 |
+
"n": 1852
|
| 35 |
+
},
|
| 36 |
+
"new-password": {
|
| 37 |
+
"p": 0.9917695473251029,
|
| 38 |
+
"r": 0.9810040705563093,
|
| 39 |
+
"f1": 0.9863574351978172,
|
| 40 |
+
"n": 737
|
| 41 |
+
},
|
| 42 |
+
"one-time-code": {
|
| 43 |
+
"p": 0.9318181818181818,
|
| 44 |
+
"r": 0.9425287356321839,
|
| 45 |
+
"f1": 0.9371428571428572,
|
| 46 |
+
"n": 174
|
| 47 |
+
},
|
| 48 |
+
"tel": {
|
| 49 |
+
"p": 0.9809802487198245,
|
| 50 |
+
"r": 0.9933333333333333,
|
| 51 |
+
"f1": 0.9871181450128819,
|
| 52 |
+
"n": 1350
|
| 53 |
+
},
|
| 54 |
+
"name": {
|
| 55 |
+
"p": 0.9501329787234043,
|
| 56 |
+
"r": 0.9740967961826857,
|
| 57 |
+
"f1": 0.9619656681252104,
|
| 58 |
+
"n": 1467
|
| 59 |
+
},
|
| 60 |
+
"given-name": {
|
| 61 |
+
"p": 0.9737302977232924,
|
| 62 |
+
"r": 0.9129720853858785,
|
| 63 |
+
"f1": 0.9423728813559322,
|
| 64 |
+
"n": 609
|
| 65 |
+
},
|
| 66 |
+
"family-name": {
|
| 67 |
+
"p": 0.9881422924901185,
|
| 68 |
+
"r": 0.9765625,
|
| 69 |
+
"f1": 0.9823182711198427,
|
| 70 |
+
"n": 512
|
| 71 |
+
},
|
| 72 |
+
"organization": {
|
| 73 |
+
"p": 0.953307392996109,
|
| 74 |
+
"r": 0.9423076923076923,
|
| 75 |
+
"f1": 0.9477756286266925,
|
| 76 |
+
"n": 260
|
| 77 |
+
},
|
| 78 |
+
"street-address": {
|
| 79 |
+
"p": 0.9396551724137931,
|
| 80 |
+
"r": 0.9775784753363229,
|
| 81 |
+
"f1": 0.9582417582417583,
|
| 82 |
+
"n": 223
|
| 83 |
+
},
|
| 84 |
+
"address-line2": {
|
| 85 |
+
"p": 0.975609756097561,
|
| 86 |
+
"r": 0.7547169811320755,
|
| 87 |
+
"f1": 0.851063829787234,
|
| 88 |
+
"n": 53
|
| 89 |
+
},
|
| 90 |
+
"address-level2": {
|
| 91 |
+
"p": 0.9149659863945578,
|
| 92 |
+
"r": 0.8848684210526315,
|
| 93 |
+
"f1": 0.8996655518394647,
|
| 94 |
+
"n": 304
|
| 95 |
+
},
|
| 96 |
+
"address-level1": {
|
| 97 |
+
"p": 0.9005524861878453,
|
| 98 |
+
"r": 0.8858695652173914,
|
| 99 |
+
"f1": 0.8931506849315068,
|
| 100 |
+
"n": 184
|
| 101 |
+
},
|
| 102 |
+
"postal-code": {
|
| 103 |
+
"p": 0.9731343283582089,
|
| 104 |
+
"r": 0.9673590504451038,
|
| 105 |
+
"f1": 0.9702380952380952,
|
| 106 |
+
"n": 337
|
| 107 |
+
},
|
| 108 |
+
"country": {
|
| 109 |
+
"p": 0.8743169398907104,
|
| 110 |
+
"r": 0.9580838323353293,
|
| 111 |
+
"f1": 0.9142857142857143,
|
| 112 |
+
"n": 167
|
| 113 |
+
},
|
| 114 |
+
"cc-name": {
|
| 115 |
+
"p": 0.9583333333333334,
|
| 116 |
+
"r": 1.0,
|
| 117 |
+
"f1": 0.9787234042553191,
|
| 118 |
+
"n": 23
|
| 119 |
+
},
|
| 120 |
+
"cc-number": {
|
| 121 |
+
"p": 0.9696969696969697,
|
| 122 |
+
"r": 1.0,
|
| 123 |
+
"f1": 0.9846153846153847,
|
| 124 |
+
"n": 32
|
| 125 |
+
},
|
| 126 |
+
"cc-exp": {
|
| 127 |
+
"p": 1.0,
|
| 128 |
+
"r": 1.0,
|
| 129 |
+
"f1": 1.0,
|
| 130 |
+
"n": 18
|
| 131 |
+
},
|
| 132 |
+
"cc-exp-month": {
|
| 133 |
+
"p": 1.0,
|
| 134 |
+
"r": 1.0,
|
| 135 |
+
"f1": 1.0,
|
| 136 |
+
"n": 14
|
| 137 |
+
},
|
| 138 |
+
"cc-exp-year": {
|
| 139 |
+
"p": 1.0,
|
| 140 |
+
"r": 1.0,
|
| 141 |
+
"f1": 1.0,
|
| 142 |
+
"n": 14
|
| 143 |
+
},
|
| 144 |
+
"cc-csc": {
|
| 145 |
+
"p": 1.0,
|
| 146 |
+
"r": 0.967741935483871,
|
| 147 |
+
"f1": 0.9836065573770492,
|
| 148 |
+
"n": 31
|
| 149 |
+
},
|
| 150 |
+
"bday": {
|
| 151 |
+
"p": 0.96,
|
| 152 |
+
"r": 0.9056603773584906,
|
| 153 |
+
"f1": 0.9320388349514563,
|
| 154 |
+
"n": 53
|
| 155 |
+
},
|
| 156 |
+
"search": {
|
| 157 |
+
"p": 0.9670212765957447,
|
| 158 |
+
"r": 0.9588607594936709,
|
| 159 |
+
"f1": 0.9629237288135594,
|
| 160 |
+
"n": 948
|
| 161 |
+
},
|
| 162 |
+
"captcha": {
|
| 163 |
+
"p": 0.9632107023411371,
|
| 164 |
+
"r": 0.9442622950819672,
|
| 165 |
+
"f1": 0.9536423841059603,
|
| 166 |
+
"n": 305
|
| 167 |
+
},
|
| 168 |
+
"other": {
|
| 169 |
+
"p": 0.9693911873528422,
|
| 170 |
+
"r": 0.9666275364749287,
|
| 171 |
+
"f1": 0.9680073893693845,
|
| 172 |
+
"n": 5963
|
| 173 |
+
}
|
| 174 |
+
},
|
| 175 |
+
"per_form": {
|
| 176 |
+
"login": {
|
| 177 |
+
"p": 0.9886914378029079,
|
| 178 |
+
"r": 0.9956616052060737,
|
| 179 |
+
"f1": 0.9921642799243447,
|
| 180 |
+
"n": 1844
|
| 181 |
+
},
|
| 182 |
+
"signup": {
|
| 183 |
+
"p": 0.9777227722772277,
|
| 184 |
+
"r": 0.947242206235012,
|
| 185 |
+
"f1": 0.9622411693057246,
|
| 186 |
+
"n": 417
|
| 187 |
+
},
|
| 188 |
+
"password-change": {
|
| 189 |
+
"p": 0.9154929577464789,
|
| 190 |
+
"r": 0.9848484848484849,
|
| 191 |
+
"f1": 0.948905109489051,
|
| 192 |
+
"n": 66
|
| 193 |
+
},
|
| 194 |
+
"password-recovery": {
|
| 195 |
+
"p": 0.9555555555555556,
|
| 196 |
+
"r": 0.9347826086956522,
|
| 197 |
+
"f1": 0.945054945054945,
|
| 198 |
+
"n": 46
|
| 199 |
+
},
|
| 200 |
+
"otp": {
|
| 201 |
+
"p": 0.9464285714285714,
|
| 202 |
+
"r": 0.9137931034482759,
|
| 203 |
+
"f1": 0.9298245614035087,
|
| 204 |
+
"n": 58
|
| 205 |
+
},
|
| 206 |
+
"payment": {
|
| 207 |
+
"p": 0.9393939393939394,
|
| 208 |
+
"r": 0.8857142857142857,
|
| 209 |
+
"f1": 0.9117647058823529,
|
| 210 |
+
"n": 35
|
| 211 |
+
},
|
| 212 |
+
"address": {
|
| 213 |
+
"p": 0.8048780487804879,
|
| 214 |
+
"r": 0.7857142857142857,
|
| 215 |
+
"f1": 0.7951807228915663,
|
| 216 |
+
"n": 42
|
| 217 |
+
},
|
| 218 |
+
"newsletter": {
|
| 219 |
+
"p": 0.9725609756097561,
|
| 220 |
+
"r": 0.9565217391304348,
|
| 221 |
+
"f1": 0.9644746787603932,
|
| 222 |
+
"n": 667
|
| 223 |
+
},
|
| 224 |
+
"contact": {
|
| 225 |
+
"p": 0.9453262786596119,
|
| 226 |
+
"r": 0.9804878048780488,
|
| 227 |
+
"f1": 0.9625860520802155,
|
| 228 |
+
"n": 1640
|
| 229 |
+
},
|
| 230 |
+
"search": {
|
| 231 |
+
"p": 0.967487684729064,
|
| 232 |
+
"r": 0.9415148609779482,
|
| 233 |
+
"f1": 0.9543245869776482,
|
| 234 |
+
"n": 1043
|
| 235 |
+
},
|
| 236 |
+
"other": {
|
| 237 |
+
"p": 0.9012464046021093,
|
| 238 |
+
"r": 0.8834586466165414,
|
| 239 |
+
"f1": 0.8922638822971048,
|
| 240 |
+
"n": 1064
|
| 241 |
+
}
|
| 242 |
+
},
|
| 243 |
+
"top_confusions": [
|
| 244 |
+
[
|
| 245 |
+
"given-name",
|
| 246 |
+
"name",
|
| 247 |
+
48
|
| 248 |
+
],
|
| 249 |
+
[
|
| 250 |
+
"search",
|
| 251 |
+
"other",
|
| 252 |
+
39
|
| 253 |
+
],
|
| 254 |
+
[
|
| 255 |
+
"address-level2",
|
| 256 |
+
"other",
|
| 257 |
+
28
|
| 258 |
+
],
|
| 259 |
+
[
|
| 260 |
+
"username",
|
| 261 |
+
"email",
|
| 262 |
+
26
|
| 263 |
+
],
|
| 264 |
+
[
|
| 265 |
+
"other",
|
| 266 |
+
"search",
|
| 267 |
+
25
|
| 268 |
+
],
|
| 269 |
+
[
|
| 270 |
+
"other",
|
| 271 |
+
"address-level2",
|
| 272 |
+
22
|
| 273 |
+
],
|
| 274 |
+
[
|
| 275 |
+
"other",
|
| 276 |
+
"country",
|
| 277 |
+
19
|
| 278 |
+
],
|
| 279 |
+
[
|
| 280 |
+
"address-level1",
|
| 281 |
+
"other",
|
| 282 |
+
18
|
| 283 |
+
],
|
| 284 |
+
[
|
| 285 |
+
"name",
|
| 286 |
+
"other",
|
| 287 |
+
15
|
| 288 |
+
],
|
| 289 |
+
[
|
| 290 |
+
"other",
|
| 291 |
+
"address-level1",
|
| 292 |
+
15
|
| 293 |
+
],
|
| 294 |
+
[
|
| 295 |
+
"other",
|
| 296 |
+
"tel",
|
| 297 |
+
15
|
| 298 |
+
],
|
| 299 |
+
[
|
| 300 |
+
"new-password",
|
| 301 |
+
"current-password",
|
| 302 |
+
14
|
| 303 |
+
],
|
| 304 |
+
[
|
| 305 |
+
"captcha",
|
| 306 |
+
"other",
|
| 307 |
+
14
|
| 308 |
+
],
|
| 309 |
+
[
|
| 310 |
+
"other",
|
| 311 |
+
"username",
|
| 312 |
+
13
|
| 313 |
+
],
|
| 314 |
+
[
|
| 315 |
+
"other",
|
| 316 |
+
"name",
|
| 317 |
+
12
|
| 318 |
+
],
|
| 319 |
+
[
|
| 320 |
+
"other",
|
| 321 |
+
"email",
|
| 322 |
+
11
|
| 323 |
+
],
|
| 324 |
+
[
|
| 325 |
+
"name",
|
| 326 |
+
"given-name",
|
| 327 |
+
11
|
| 328 |
+
],
|
| 329 |
+
[
|
| 330 |
+
"address-line2",
|
| 331 |
+
"other",
|
| 332 |
+
11
|
| 333 |
+
],
|
| 334 |
+
[
|
| 335 |
+
"other",
|
| 336 |
+
"street-address",
|
| 337 |
+
11
|
| 338 |
+
],
|
| 339 |
+
[
|
| 340 |
+
"other",
|
| 341 |
+
"current-password",
|
| 342 |
+
10
|
| 343 |
+
],
|
| 344 |
+
[
|
| 345 |
+
"other",
|
| 346 |
+
"captcha",
|
| 347 |
+
9
|
| 348 |
+
],
|
| 349 |
+
[
|
| 350 |
+
"other",
|
| 351 |
+
"postal-code",
|
| 352 |
+
9
|
| 353 |
+
],
|
| 354 |
+
[
|
| 355 |
+
"email",
|
| 356 |
+
"username",
|
| 357 |
+
8
|
| 358 |
+
],
|
| 359 |
+
[
|
| 360 |
+
"email",
|
| 361 |
+
"other",
|
| 362 |
+
8
|
| 363 |
+
],
|
| 364 |
+
[
|
| 365 |
+
"organization",
|
| 366 |
+
"other",
|
| 367 |
+
8
|
| 368 |
+
]
|
| 369 |
+
]
|
| 370 |
+
},
|
| 371 |
+
"model_real": {
|
| 372 |
+
"n_forms": 6838,
|
| 373 |
+
"n_fields": 19811,
|
| 374 |
+
"field_acc": 0.971833829690576,
|
| 375 |
+
"field_macro_f1": 0.9447407167557139,
|
| 376 |
+
"form_acc": 0.9568587306229892,
|
| 377 |
+
"form_macro_f1": 0.9047413119700765,
|
| 378 |
+
"password_precision": 0.9941383352872216,
|
| 379 |
+
"password_recall": 0.9976470588235294,
|
| 380 |
+
"password_f1": 0.9958896065766295,
|
| 381 |
+
"cur_vs_new_acc": 0.991764705882353,
|
| 382 |
+
"identifier_f1": 0.993869133875822,
|
| 383 |
+
"per_field": {
|
| 384 |
+
"username": {
|
| 385 |
+
"p": 0.9780907668231612,
|
| 386 |
+
"r": 0.9735202492211839,
|
| 387 |
+
"f1": 0.975800156128025,
|
| 388 |
+
"n": 1284
|
| 389 |
+
},
|
| 390 |
+
"email": {
|
| 391 |
+
"p": 0.9869402985074627,
|
| 392 |
+
"r": 0.9940494832445975,
|
| 393 |
+
"f1": 0.9904821344983618,
|
| 394 |
+
"n": 3193
|
| 395 |
+
},
|
| 396 |
+
"current-password": {
|
| 397 |
+
"p": 0.9870550161812298,
|
| 398 |
+
"r": 0.9956474428726877,
|
| 399 |
+
"f1": 0.991332611050921,
|
| 400 |
+
"n": 1838
|
| 401 |
+
},
|
| 402 |
+
"new-password": {
|
| 403 |
+
"p": 0.9914893617021276,
|
| 404 |
+
"r": 0.9817415730337079,
|
| 405 |
+
"f1": 0.9865913902611151,
|
| 406 |
+
"n": 712
|
| 407 |
+
},
|
| 408 |
+
"one-time-code": {
|
| 409 |
+
"p": 0.9328358208955224,
|
| 410 |
+
"r": 0.9259259259259259,
|
| 411 |
+
"f1": 0.929368029739777,
|
| 412 |
+
"n": 135
|
| 413 |
+
},
|
| 414 |
+
"tel": {
|
| 415 |
+
"p": 0.9822485207100592,
|
| 416 |
+
"r": 0.9932685115931189,
|
| 417 |
+
"f1": 0.9877277798438081,
|
| 418 |
+
"n": 1337
|
| 419 |
+
},
|
| 420 |
+
"name": {
|
| 421 |
+
"p": 0.9499332443257676,
|
| 422 |
+
"r": 0.973990417522245,
|
| 423 |
+
"f1": 0.9618114227779655,
|
| 424 |
+
"n": 1461
|
| 425 |
+
},
|
| 426 |
+
"given-name": {
|
| 427 |
+
"p": 0.9734513274336283,
|
| 428 |
+
"r": 0.912106135986733,
|
| 429 |
+
"f1": 0.9417808219178082,
|
| 430 |
+
"n": 603
|
| 431 |
+
},
|
| 432 |
+
"family-name": {
|
| 433 |
+
"p": 0.988,
|
| 434 |
+
"r": 0.9762845849802372,
|
| 435 |
+
"f1": 0.9821073558648111,
|
| 436 |
+
"n": 506
|
| 437 |
+
},
|
| 438 |
+
"organization": {
|
| 439 |
+
"p": 0.9525691699604744,
|
| 440 |
+
"r": 0.94140625,
|
| 441 |
+
"f1": 0.9469548133595285,
|
| 442 |
+
"n": 256
|
| 443 |
+
},
|
| 444 |
+
"street-address": {
|
| 445 |
+
"p": 0.9357798165137615,
|
| 446 |
+
"r": 0.9807692307692307,
|
| 447 |
+
"f1": 0.9577464788732395,
|
| 448 |
+
"n": 208
|
| 449 |
+
},
|
| 450 |
+
"address-line2": {
|
| 451 |
+
"p": 0.972972972972973,
|
| 452 |
+
"r": 0.75,
|
| 453 |
+
"f1": 0.8470588235294119,
|
| 454 |
+
"n": 48
|
| 455 |
+
},
|
| 456 |
+
"address-level2": {
|
| 457 |
+
"p": 0.911660777385159,
|
| 458 |
+
"r": 0.8835616438356164,
|
| 459 |
+
"f1": 0.8973913043478261,
|
| 460 |
+
"n": 292
|
| 461 |
+
},
|
| 462 |
+
"address-level1": {
|
| 463 |
+
"p": 0.8959537572254336,
|
| 464 |
+
"r": 0.8857142857142857,
|
| 465 |
+
"f1": 0.8908045977011495,
|
| 466 |
+
"n": 175
|
| 467 |
+
},
|
| 468 |
+
"postal-code": {
|
| 469 |
+
"p": 0.9719626168224299,
|
| 470 |
+
"r": 0.968944099378882,
|
| 471 |
+
"f1": 0.9704510108864696,
|
| 472 |
+
"n": 322
|
| 473 |
+
},
|
| 474 |
+
"country": {
|
| 475 |
+
"p": 0.8685714285714285,
|
| 476 |
+
"r": 0.9620253164556962,
|
| 477 |
+
"f1": 0.9129129129129129,
|
| 478 |
+
"n": 158
|
| 479 |
+
},
|
| 480 |
+
"cc-name": {
|
| 481 |
+
"p": 0.6666666666666666,
|
| 482 |
+
"r": 1.0,
|
| 483 |
+
"f1": 0.8,
|
| 484 |
+
"n": 2
|
| 485 |
+
},
|
| 486 |
+
"cc-number": {
|
| 487 |
+
"p": 0.8571428571428571,
|
| 488 |
+
"r": 1.0,
|
| 489 |
+
"f1": 0.923076923076923,
|
| 490 |
+
"n": 6
|
| 491 |
+
},
|
| 492 |
+
"cc-exp-month": {
|
| 493 |
+
"p": 1.0,
|
| 494 |
+
"r": 1.0,
|
| 495 |
+
"f1": 1.0,
|
| 496 |
+
"n": 6
|
| 497 |
+
},
|
| 498 |
+
"cc-exp-year": {
|
| 499 |
+
"p": 1.0,
|
| 500 |
+
"r": 1.0,
|
| 501 |
+
"f1": 1.0,
|
| 502 |
+
"n": 6
|
| 503 |
+
},
|
| 504 |
+
"cc-csc": {
|
| 505 |
+
"p": 1.0,
|
| 506 |
+
"r": 0.8333333333333334,
|
| 507 |
+
"f1": 0.9090909090909091,
|
| 508 |
+
"n": 6
|
| 509 |
+
},
|
| 510 |
+
"bday": {
|
| 511 |
+
"p": 0.96,
|
| 512 |
+
"r": 0.9056603773584906,
|
| 513 |
+
"f1": 0.9320388349514563,
|
| 514 |
+
"n": 53
|
| 515 |
+
},
|
| 516 |
+
"search": {
|
| 517 |
+
"p": 0.9670212765957447,
|
| 518 |
+
"r": 0.9588607594936709,
|
| 519 |
+
"f1": 0.9629237288135594,
|
| 520 |
+
"n": 948
|
| 521 |
+
},
|
| 522 |
+
"captcha": {
|
| 523 |
+
"p": 0.9627118644067797,
|
| 524 |
+
"r": 0.9435215946843853,
|
| 525 |
+
"f1": 0.953020134228188,
|
| 526 |
+
"n": 301
|
| 527 |
+
},
|
| 528 |
+
"other": {
|
| 529 |
+
"p": 0.9695132221660772,
|
| 530 |
+
"r": 0.9665827036104114,
|
| 531 |
+
"f1": 0.9680457450386813,
|
| 532 |
+
"n": 5955
|
| 533 |
+
}
|
| 534 |
+
},
|
| 535 |
+
"per_form": {
|
| 536 |
+
"login": {
|
| 537 |
+
"p": 0.9891774891774892,
|
| 538 |
+
"r": 0.9956427015250545,
|
| 539 |
+
"f1": 0.992399565689468,
|
| 540 |
+
"n": 1836
|
| 541 |
+
},
|
| 542 |
+
"signup": {
|
| 543 |
+
"p": 0.9777227722772277,
|
| 544 |
+
"r": 0.947242206235012,
|
| 545 |
+
"f1": 0.9622411693057246,
|
| 546 |
+
"n": 417
|
| 547 |
+
},
|
| 548 |
+
"password-change": {
|
| 549 |
+
"p": 0.8928571428571429,
|
| 550 |
+
"r": 1.0,
|
| 551 |
+
"f1": 0.9433962264150945,
|
| 552 |
+
"n": 50
|
| 553 |
+
},
|
| 554 |
+
"password-recovery": {
|
| 555 |
+
"p": 0.9459459459459459,
|
| 556 |
+
"r": 0.9210526315789473,
|
| 557 |
+
"f1": 0.9333333333333332,
|
| 558 |
+
"n": 38
|
| 559 |
+
},
|
| 560 |
+
"otp": {
|
| 561 |
+
"p": 0.9210526315789473,
|
| 562 |
+
"r": 0.875,
|
| 563 |
+
"f1": 0.8974358974358975,
|
| 564 |
+
"n": 40
|
| 565 |
+
},
|
| 566 |
+
"payment": {
|
| 567 |
+
"p": 0.7142857142857143,
|
| 568 |
+
"r": 0.625,
|
| 569 |
+
"f1": 0.6666666666666666,
|
| 570 |
+
"n": 8
|
| 571 |
+
},
|
| 572 |
+
"address": {
|
| 573 |
+
"p": 0.7941176470588235,
|
| 574 |
+
"r": 0.7714285714285715,
|
| 575 |
+
"f1": 0.782608695652174,
|
| 576 |
+
"n": 35
|
| 577 |
+
},
|
| 578 |
+
"newsletter": {
|
| 579 |
+
"p": 0.9725609756097561,
|
| 580 |
+
"r": 0.9565217391304348,
|
| 581 |
+
"f1": 0.9644746787603932,
|
| 582 |
+
"n": 667
|
| 583 |
+
},
|
| 584 |
+
"contact": {
|
| 585 |
+
"p": 0.9453262786596119,
|
| 586 |
+
"r": 0.9804878048780488,
|
| 587 |
+
"f1": 0.9625860520802155,
|
| 588 |
+
"n": 1640
|
| 589 |
+
},
|
| 590 |
+
"search": {
|
| 591 |
+
"p": 0.967487684729064,
|
| 592 |
+
"r": 0.9415148609779482,
|
| 593 |
+
"f1": 0.9543245869776482,
|
| 594 |
+
"n": 1043
|
| 595 |
+
},
|
| 596 |
+
"other": {
|
| 597 |
+
"p": 0.9021113243761996,
|
| 598 |
+
"r": 0.8834586466165414,
|
| 599 |
+
"f1": 0.892687559354226,
|
| 600 |
+
"n": 1064
|
| 601 |
+
}
|
| 602 |
+
},
|
| 603 |
+
"top_confusions": [
|
| 604 |
+
[
|
| 605 |
+
"given-name",
|
| 606 |
+
"name",
|
| 607 |
+
48
|
| 608 |
+
],
|
| 609 |
+
[
|
| 610 |
+
"search",
|
| 611 |
+
"other",
|
| 612 |
+
39
|
| 613 |
+
],
|
| 614 |
+
[
|
| 615 |
+
"address-level2",
|
| 616 |
+
"other",
|
| 617 |
+
27
|
| 618 |
+
],
|
| 619 |
+
[
|
| 620 |
+
"username",
|
| 621 |
+
"email",
|
| 622 |
+
26
|
| 623 |
+
],
|
| 624 |
+
[
|
| 625 |
+
"other",
|
| 626 |
+
"search",
|
| 627 |
+
25
|
| 628 |
+
],
|
| 629 |
+
[
|
| 630 |
+
"other",
|
| 631 |
+
"address-level2",
|
| 632 |
+
22
|
| 633 |
+
],
|
| 634 |
+
[
|
| 635 |
+
"other",
|
| 636 |
+
"country",
|
| 637 |
+
19
|
| 638 |
+
],
|
| 639 |
+
[
|
| 640 |
+
"address-level1",
|
| 641 |
+
"other",
|
| 642 |
+
18
|
| 643 |
+
],
|
| 644 |
+
[
|
| 645 |
+
"name",
|
| 646 |
+
"other",
|
| 647 |
+
15
|
| 648 |
+
],
|
| 649 |
+
[
|
| 650 |
+
"other",
|
| 651 |
+
"address-level1",
|
| 652 |
+
15
|
| 653 |
+
],
|
| 654 |
+
[
|
| 655 |
+
"other",
|
| 656 |
+
"tel",
|
| 657 |
+
15
|
| 658 |
+
],
|
| 659 |
+
[
|
| 660 |
+
"captcha",
|
| 661 |
+
"other",
|
| 662 |
+
14
|
| 663 |
+
],
|
| 664 |
+
[
|
| 665 |
+
"new-password",
|
| 666 |
+
"current-password",
|
| 667 |
+
13
|
| 668 |
+
],
|
| 669 |
+
[
|
| 670 |
+
"other",
|
| 671 |
+
"username",
|
| 672 |
+
13
|
| 673 |
+
],
|
| 674 |
+
[
|
| 675 |
+
"other",
|
| 676 |
+
"name",
|
| 677 |
+
12
|
| 678 |
+
],
|
| 679 |
+
[
|
| 680 |
+
"other",
|
| 681 |
+
"email",
|
| 682 |
+
11
|
| 683 |
+
],
|
| 684 |
+
[
|
| 685 |
+
"name",
|
| 686 |
+
"given-name",
|
| 687 |
+
11
|
| 688 |
+
],
|
| 689 |
+
[
|
| 690 |
+
"address-line2",
|
| 691 |
+
"other",
|
| 692 |
+
11
|
| 693 |
+
],
|
| 694 |
+
[
|
| 695 |
+
"other",
|
| 696 |
+
"street-address",
|
| 697 |
+
11
|
| 698 |
+
],
|
| 699 |
+
[
|
| 700 |
+
"other",
|
| 701 |
+
"current-password",
|
| 702 |
+
10
|
| 703 |
+
],
|
| 704 |
+
[
|
| 705 |
+
"other",
|
| 706 |
+
"captcha",
|
| 707 |
+
9
|
| 708 |
+
],
|
| 709 |
+
[
|
| 710 |
+
"other",
|
| 711 |
+
"postal-code",
|
| 712 |
+
9
|
| 713 |
+
],
|
| 714 |
+
[
|
| 715 |
+
"email",
|
| 716 |
+
"username",
|
| 717 |
+
8
|
| 718 |
+
],
|
| 719 |
+
[
|
| 720 |
+
"email",
|
| 721 |
+
"other",
|
| 722 |
+
8
|
| 723 |
+
],
|
| 724 |
+
[
|
| 725 |
+
"organization",
|
| 726 |
+
"other",
|
| 727 |
+
8
|
| 728 |
+
]
|
| 729 |
+
]
|
| 730 |
+
},
|
| 731 |
+
"model_synthetic": {
|
| 732 |
+
"n_forms": 84,
|
| 733 |
+
"n_fields": 319,
|
| 734 |
+
"field_acc": 0.9780564263322884,
|
| 735 |
+
"field_macro_f1": 0.9765257496746269,
|
| 736 |
+
"form_acc": 0.9642857142857143,
|
| 737 |
+
"form_macro_f1": 0.9578655564477768,
|
| 738 |
+
"password_precision": 1.0,
|
| 739 |
+
"password_recall": 1.0,
|
| 740 |
+
"password_f1": 1.0,
|
| 741 |
+
"cur_vs_new_acc": 0.9743589743589743,
|
| 742 |
+
"identifier_f1": 1.0,
|
| 743 |
+
"per_field": {
|
| 744 |
+
"username": {
|
| 745 |
+
"p": 1.0,
|
| 746 |
+
"r": 1.0,
|
| 747 |
+
"f1": 1.0,
|
| 748 |
+
"n": 7
|
| 749 |
+
},
|
| 750 |
+
"email": {
|
| 751 |
+
"p": 1.0,
|
| 752 |
+
"r": 1.0,
|
| 753 |
+
"f1": 1.0,
|
| 754 |
+
"n": 16
|
| 755 |
+
},
|
| 756 |
+
"current-password": {
|
| 757 |
+
"p": 0.9333333333333333,
|
| 758 |
+
"r": 1.0,
|
| 759 |
+
"f1": 0.9655172413793104,
|
| 760 |
+
"n": 14
|
| 761 |
+
},
|
| 762 |
+
"new-password": {
|
| 763 |
+
"p": 1.0,
|
| 764 |
+
"r": 0.96,
|
| 765 |
+
"f1": 0.9795918367346939,
|
| 766 |
+
"n": 25
|
| 767 |
+
},
|
| 768 |
+
"one-time-code": {
|
| 769 |
+
"p": 0.9285714285714286,
|
| 770 |
+
"r": 1.0,
|
| 771 |
+
"f1": 0.962962962962963,
|
| 772 |
+
"n": 39
|
| 773 |
+
},
|
| 774 |
+
"tel": {
|
| 775 |
+
"p": 0.8666666666666667,
|
| 776 |
+
"r": 1.0,
|
| 777 |
+
"f1": 0.9285714285714286,
|
| 778 |
+
"n": 13
|
| 779 |
+
},
|
| 780 |
+
"name": {
|
| 781 |
+
"p": 1.0,
|
| 782 |
+
"r": 1.0,
|
| 783 |
+
"f1": 1.0,
|
| 784 |
+
"n": 6
|
| 785 |
+
},
|
| 786 |
+
"given-name": {
|
| 787 |
+
"p": 1.0,
|
| 788 |
+
"r": 1.0,
|
| 789 |
+
"f1": 1.0,
|
| 790 |
+
"n": 6
|
| 791 |
+
},
|
| 792 |
+
"family-name": {
|
| 793 |
+
"p": 1.0,
|
| 794 |
+
"r": 1.0,
|
| 795 |
+
"f1": 1.0,
|
| 796 |
+
"n": 6
|
| 797 |
+
},
|
| 798 |
+
"organization": {
|
| 799 |
+
"p": 1.0,
|
| 800 |
+
"r": 1.0,
|
| 801 |
+
"f1": 1.0,
|
| 802 |
+
"n": 4
|
| 803 |
+
},
|
| 804 |
+
"street-address": {
|
| 805 |
+
"p": 1.0,
|
| 806 |
+
"r": 0.9333333333333333,
|
| 807 |
+
"f1": 0.9655172413793104,
|
| 808 |
+
"n": 15
|
| 809 |
+
},
|
| 810 |
+
"address-line2": {
|
| 811 |
+
"p": 1.0,
|
| 812 |
+
"r": 0.8,
|
| 813 |
+
"f1": 0.888888888888889,
|
| 814 |
+
"n": 5
|
| 815 |
+
},
|
| 816 |
+
"address-level2": {
|
| 817 |
+
"p": 1.0,
|
| 818 |
+
"r": 0.9166666666666666,
|
| 819 |
+
"f1": 0.9565217391304348,
|
| 820 |
+
"n": 12
|
| 821 |
+
},
|
| 822 |
+
"address-level1": {
|
| 823 |
+
"p": 1.0,
|
| 824 |
+
"r": 0.8888888888888888,
|
| 825 |
+
"f1": 0.9411764705882353,
|
| 826 |
+
"n": 9
|
| 827 |
+
},
|
| 828 |
+
"postal-code": {
|
| 829 |
+
"p": 1.0,
|
| 830 |
+
"r": 0.9333333333333333,
|
| 831 |
+
"f1": 0.9655172413793104,
|
| 832 |
+
"n": 15
|
| 833 |
+
},
|
| 834 |
+
"country": {
|
| 835 |
+
"p": 1.0,
|
| 836 |
+
"r": 0.8888888888888888,
|
| 837 |
+
"f1": 0.9411764705882353,
|
| 838 |
+
"n": 9
|
| 839 |
+
},
|
| 840 |
+
"cc-name": {
|
| 841 |
+
"p": 1.0,
|
| 842 |
+
"r": 1.0,
|
| 843 |
+
"f1": 1.0,
|
| 844 |
+
"n": 21
|
| 845 |
+
},
|
| 846 |
+
"cc-number": {
|
| 847 |
+
"p": 1.0,
|
| 848 |
+
"r": 1.0,
|
| 849 |
+
"f1": 1.0,
|
| 850 |
+
"n": 26
|
| 851 |
+
},
|
| 852 |
+
"cc-exp": {
|
| 853 |
+
"p": 1.0,
|
| 854 |
+
"r": 1.0,
|
| 855 |
+
"f1": 1.0,
|
| 856 |
+
"n": 18
|
| 857 |
+
},
|
| 858 |
+
"cc-exp-month": {
|
| 859 |
+
"p": 1.0,
|
| 860 |
+
"r": 1.0,
|
| 861 |
+
"f1": 1.0,
|
| 862 |
+
"n": 8
|
| 863 |
+
},
|
| 864 |
+
"cc-exp-year": {
|
| 865 |
+
"p": 1.0,
|
| 866 |
+
"r": 1.0,
|
| 867 |
+
"f1": 1.0,
|
| 868 |
+
"n": 8
|
| 869 |
+
},
|
| 870 |
+
"cc-csc": {
|
| 871 |
+
"p": 1.0,
|
| 872 |
+
"r": 1.0,
|
| 873 |
+
"f1": 1.0,
|
| 874 |
+
"n": 25
|
| 875 |
+
},
|
| 876 |
+
"captcha": {
|
| 877 |
+
"p": 1.0,
|
| 878 |
+
"r": 1.0,
|
| 879 |
+
"f1": 1.0,
|
| 880 |
+
"n": 4
|
| 881 |
+
},
|
| 882 |
+
"other": {
|
| 883 |
+
"p": 0.8888888888888888,
|
| 884 |
+
"r": 1.0,
|
| 885 |
+
"f1": 0.9411764705882353,
|
| 886 |
+
"n": 8
|
| 887 |
+
}
|
| 888 |
+
},
|
| 889 |
+
"per_form": {
|
| 890 |
+
"login": {
|
| 891 |
+
"p": 0.8888888888888888,
|
| 892 |
+
"r": 1.0,
|
| 893 |
+
"f1": 0.9411764705882353,
|
| 894 |
+
"n": 8
|
| 895 |
+
},
|
| 896 |
+
"password-change": {
|
| 897 |
+
"p": 1.0,
|
| 898 |
+
"r": 0.9375,
|
| 899 |
+
"f1": 0.967741935483871,
|
| 900 |
+
"n": 16
|
| 901 |
+
},
|
| 902 |
+
"password-recovery": {
|
| 903 |
+
"p": 1.0,
|
| 904 |
+
"r": 1.0,
|
| 905 |
+
"f1": 1.0,
|
| 906 |
+
"n": 8
|
| 907 |
+
},
|
| 908 |
+
"otp": {
|
| 909 |
+
"p": 1.0,
|
| 910 |
+
"r": 1.0,
|
| 911 |
+
"f1": 1.0,
|
| 912 |
+
"n": 18
|
| 913 |
+
},
|
| 914 |
+
"payment": {
|
| 915 |
+
"p": 1.0,
|
| 916 |
+
"r": 0.9629629629629629,
|
| 917 |
+
"f1": 0.9811320754716981,
|
| 918 |
+
"n": 27
|
| 919 |
+
},
|
| 920 |
+
"address": {
|
| 921 |
+
"p": 0.8571428571428571,
|
| 922 |
+
"r": 0.8571428571428571,
|
| 923 |
+
"f1": 0.8571428571428571,
|
| 924 |
+
"n": 7
|
| 925 |
+
}
|
| 926 |
+
},
|
| 927 |
+
"top_confusions": [
|
| 928 |
+
[
|
| 929 |
+
"new-password",
|
| 930 |
+
"current-password",
|
| 931 |
+
1
|
| 932 |
+
],
|
| 933 |
+
[
|
| 934 |
+
"street-address",
|
| 935 |
+
"tel",
|
| 936 |
+
1
|
| 937 |
+
],
|
| 938 |
+
[
|
| 939 |
+
"address-line2",
|
| 940 |
+
"one-time-code",
|
| 941 |
+
1
|
| 942 |
+
],
|
| 943 |
+
[
|
| 944 |
+
"address-level2",
|
| 945 |
+
"other",
|
| 946 |
+
1
|
| 947 |
+
],
|
| 948 |
+
[
|
| 949 |
+
"address-level1",
|
| 950 |
+
"one-time-code",
|
| 951 |
+
1
|
| 952 |
+
],
|
| 953 |
+
[
|
| 954 |
+
"postal-code",
|
| 955 |
+
"one-time-code",
|
| 956 |
+
1
|
| 957 |
+
],
|
| 958 |
+
[
|
| 959 |
+
"country",
|
| 960 |
+
"tel",
|
| 961 |
+
1
|
| 962 |
+
]
|
| 963 |
+
]
|
| 964 |
+
},
|
| 965 |
+
"heuristic_baseline_real": {
|
| 966 |
+
"n_forms": 6838,
|
| 967 |
+
"n_fields": 19811,
|
| 968 |
+
"field_acc": 0.827368633587401,
|
| 969 |
+
"field_macro_f1": 0.7233335250918939,
|
| 970 |
+
"form_acc": 0.7157063468850541,
|
| 971 |
+
"form_macro_f1": 0.48529320010648663,
|
| 972 |
+
"password_precision": 0.9391143911439115,
|
| 973 |
+
"password_recall": 0.9980392156862745,
|
| 974 |
+
"password_f1": 0.967680608365019,
|
| 975 |
+
"cur_vs_new_acc": 0.8545098039215686,
|
| 976 |
+
"identifier_f1": 0.9453604914322663,
|
| 977 |
+
"per_field": {
|
| 978 |
+
"username": {
|
| 979 |
+
"p": 0.7456213511259383,
|
| 980 |
+
"r": 0.6962616822429907,
|
| 981 |
+
"f1": 0.7200966572694322,
|
| 982 |
+
"n": 1284
|
| 983 |
+
},
|
| 984 |
+
"email": {
|
| 985 |
+
"p": 0.8662225922842076,
|
| 986 |
+
"r": 0.9774506733479487,
|
| 987 |
+
"f1": 0.918481459682166,
|
| 988 |
+
"n": 3193
|
| 989 |
+
},
|
| 990 |
+
"current-password": {
|
| 991 |
+
"p": 0.8279176201372997,
|
| 992 |
+
"r": 0.984221980413493,
|
| 993 |
+
"f1": 0.8993288590604027,
|
| 994 |
+
"n": 1838
|
| 995 |
+
},
|
| 996 |
+
"new-password": {
|
| 997 |
+
"p": 0.7047619047619048,
|
| 998 |
+
"r": 0.5196629213483146,
|
| 999 |
+
"f1": 0.5982215036378336,
|
| 1000 |
+
"n": 712
|
| 1001 |
+
},
|
| 1002 |
+
"one-time-code": {
|
| 1003 |
+
"p": 0.6486486486486487,
|
| 1004 |
+
"r": 0.35555555555555557,
|
| 1005 |
+
"f1": 0.45933014354066987,
|
| 1006 |
+
"n": 135
|
| 1007 |
+
},
|
| 1008 |
+
"tel": {
|
| 1009 |
+
"p": 0.8867791842475387,
|
| 1010 |
+
"r": 0.9431563201196709,
|
| 1011 |
+
"f1": 0.91409931134469,
|
| 1012 |
+
"n": 1337
|
| 1013 |
+
},
|
| 1014 |
+
"name": {
|
| 1015 |
+
"p": 0.8157695223654283,
|
| 1016 |
+
"r": 0.7364818617385352,
|
| 1017 |
+
"f1": 0.7741007194244605,
|
| 1018 |
+
"n": 1461
|
| 1019 |
+
},
|
| 1020 |
+
"given-name": {
|
| 1021 |
+
"p": 0.6842105263157895,
|
| 1022 |
+
"r": 0.7114427860696517,
|
| 1023 |
+
"f1": 0.697560975609756,
|
| 1024 |
+
"n": 603
|
| 1025 |
+
},
|
| 1026 |
+
"family-name": {
|
| 1027 |
+
"p": 0.9160671462829736,
|
| 1028 |
+
"r": 0.7549407114624506,
|
| 1029 |
+
"f1": 0.8277356446370531,
|
| 1030 |
+
"n": 506
|
| 1031 |
+
},
|
| 1032 |
+
"organization": {
|
| 1033 |
+
"p": 0.8041666666666667,
|
| 1034 |
+
"r": 0.75390625,
|
| 1035 |
+
"f1": 0.778225806451613,
|
| 1036 |
+
"n": 256
|
| 1037 |
+
},
|
| 1038 |
+
"street-address": {
|
| 1039 |
+
"p": 0.6370370370370371,
|
| 1040 |
+
"r": 0.8269230769230769,
|
| 1041 |
+
"f1": 0.7196652719665272,
|
| 1042 |
+
"n": 208
|
| 1043 |
+
},
|
| 1044 |
+
"address-line2": {
|
| 1045 |
+
"p": 0.453125,
|
| 1046 |
+
"r": 0.6041666666666666,
|
| 1047 |
+
"f1": 0.5178571428571429,
|
| 1048 |
+
"n": 48
|
| 1049 |
+
},
|
| 1050 |
+
"address-level2": {
|
| 1051 |
+
"p": 0.6318537859007833,
|
| 1052 |
+
"r": 0.8287671232876712,
|
| 1053 |
+
"f1": 0.717037037037037,
|
| 1054 |
+
"n": 292
|
| 1055 |
+
},
|
| 1056 |
+
"address-level1": {
|
| 1057 |
+
"p": 0.5258964143426295,
|
| 1058 |
+
"r": 0.7542857142857143,
|
| 1059 |
+
"f1": 0.6197183098591549,
|
| 1060 |
+
"n": 175
|
| 1061 |
+
},
|
| 1062 |
+
"postal-code": {
|
| 1063 |
+
"p": 0.925,
|
| 1064 |
+
"r": 0.8043478260869565,
|
| 1065 |
+
"f1": 0.8604651162790697,
|
| 1066 |
+
"n": 322
|
| 1067 |
+
},
|
| 1068 |
+
"country": {
|
| 1069 |
+
"p": 0.8226950354609929,
|
| 1070 |
+
"r": 0.7341772151898734,
|
| 1071 |
+
"f1": 0.7759197324414716,
|
| 1072 |
+
"n": 158
|
| 1073 |
+
},
|
| 1074 |
+
"cc-name": {
|
| 1075 |
+
"p": 0.5,
|
| 1076 |
+
"r": 1.0,
|
| 1077 |
+
"f1": 0.6666666666666666,
|
| 1078 |
+
"n": 2
|
| 1079 |
+
},
|
| 1080 |
+
"cc-number": {
|
| 1081 |
+
"p": 0.5,
|
| 1082 |
+
"r": 1.0,
|
| 1083 |
+
"f1": 0.6666666666666666,
|
| 1084 |
+
"n": 6
|
| 1085 |
+
},
|
| 1086 |
+
"cc-exp-month": {
|
| 1087 |
+
"p": 1.0,
|
| 1088 |
+
"r": 0.5,
|
| 1089 |
+
"f1": 0.6666666666666666,
|
| 1090 |
+
"n": 6
|
| 1091 |
+
},
|
| 1092 |
+
"cc-exp-year": {
|
| 1093 |
+
"p": 0.6,
|
| 1094 |
+
"r": 0.5,
|
| 1095 |
+
"f1": 0.5454545454545454,
|
| 1096 |
+
"n": 6
|
| 1097 |
+
},
|
| 1098 |
+
"cc-csc": {
|
| 1099 |
+
"p": 0.4166666666666667,
|
| 1100 |
+
"r": 0.8333333333333334,
|
| 1101 |
+
"f1": 0.5555555555555556,
|
| 1102 |
+
"n": 6
|
| 1103 |
+
},
|
| 1104 |
+
"bday": {
|
| 1105 |
+
"p": 0.9318181818181818,
|
| 1106 |
+
"r": 0.7735849056603774,
|
| 1107 |
+
"f1": 0.8453608247422681,
|
| 1108 |
+
"n": 53
|
| 1109 |
+
},
|
| 1110 |
+
"search": {
|
| 1111 |
+
"p": 0.8284714119019837,
|
| 1112 |
+
"r": 0.7489451476793249,
|
| 1113 |
+
"f1": 0.7867036011080333,
|
| 1114 |
+
"n": 948
|
| 1115 |
+
},
|
| 1116 |
+
"captcha": {
|
| 1117 |
+
"p": 0.9297297297297298,
|
| 1118 |
+
"r": 0.5714285714285714,
|
| 1119 |
+
"f1": 0.7078189300411523,
|
| 1120 |
+
"n": 301
|
| 1121 |
+
},
|
| 1122 |
+
"other": {
|
| 1123 |
+
"p": 0.8645796693633486,
|
| 1124 |
+
"r": 0.8255247691015953,
|
| 1125 |
+
"f1": 0.8446009792973114,
|
| 1126 |
+
"n": 5955
|
| 1127 |
+
}
|
| 1128 |
+
},
|
| 1129 |
+
"per_form": {
|
| 1130 |
+
"login": {
|
| 1131 |
+
"p": 0.94069640914037,
|
| 1132 |
+
"r": 0.9417211328976035,
|
| 1133 |
+
"f1": 0.9412084921066958,
|
| 1134 |
+
"n": 1836
|
| 1135 |
+
},
|
| 1136 |
+
"signup": {
|
| 1137 |
+
"p": 0.5643224699828473,
|
| 1138 |
+
"r": 0.7889688249400479,
|
| 1139 |
+
"f1": 0.6579999999999999,
|
| 1140 |
+
"n": 417
|
| 1141 |
+
},
|
| 1142 |
+
"password-change": {
|
| 1143 |
+
"p": 0.0,
|
| 1144 |
+
"r": 0.0,
|
| 1145 |
+
"f1": 0.0,
|
| 1146 |
+
"n": 50
|
| 1147 |
+
},
|
| 1148 |
+
"password-recovery": {
|
| 1149 |
+
"p": 0.26666666666666666,
|
| 1150 |
+
"r": 0.21052631578947367,
|
| 1151 |
+
"f1": 0.23529411764705882,
|
| 1152 |
+
"n": 38
|
| 1153 |
+
},
|
| 1154 |
+
"otp": {
|
| 1155 |
+
"p": 0.3,
|
| 1156 |
+
"r": 0.375,
|
| 1157 |
+
"f1": 0.33333333333333326,
|
| 1158 |
+
"n": 40
|
| 1159 |
+
},
|
| 1160 |
+
"payment": {
|
| 1161 |
+
"p": 0.21052631578947367,
|
| 1162 |
+
"r": 0.5,
|
| 1163 |
+
"f1": 0.2962962962962963,
|
| 1164 |
+
"n": 8
|
| 1165 |
+
},
|
| 1166 |
+
"address": {
|
| 1167 |
+
"p": 0.03722504230118443,
|
| 1168 |
+
"r": 0.6285714285714286,
|
| 1169 |
+
"f1": 0.07028753993610225,
|
| 1170 |
+
"n": 35
|
| 1171 |
+
},
|
| 1172 |
+
"newsletter": {
|
| 1173 |
+
"p": 0.8864774624373957,
|
| 1174 |
+
"r": 0.7961019490254873,
|
| 1175 |
+
"f1": 0.8388625592417062,
|
| 1176 |
+
"n": 667
|
| 1177 |
+
},
|
| 1178 |
+
"contact": {
|
| 1179 |
+
"p": 0.9479371316306483,
|
| 1180 |
+
"r": 0.5884146341463414,
|
| 1181 |
+
"f1": 0.7261098570353649,
|
| 1182 |
+
"n": 1640
|
| 1183 |
+
},
|
| 1184 |
+
"search": {
|
| 1185 |
+
"p": 0.984313725490196,
|
| 1186 |
+
"r": 0.4813039309683605,
|
| 1187 |
+
"f1": 0.6464906632324533,
|
| 1188 |
+
"n": 1043
|
| 1189 |
+
},
|
| 1190 |
+
"other": {
|
| 1191 |
+
"p": 0.493125,
|
| 1192 |
+
"r": 0.7415413533834586,
|
| 1193 |
+
"f1": 0.5923423423423424,
|
| 1194 |
+
"n": 1064
|
| 1195 |
+
}
|
| 1196 |
+
},
|
| 1197 |
+
"top_confusions": [
|
| 1198 |
+
[
|
| 1199 |
+
"new-password",
|
| 1200 |
+
"current-password",
|
| 1201 |
+
339
|
| 1202 |
+
],
|
| 1203 |
+
[
|
| 1204 |
+
"username",
|
| 1205 |
+
"email",
|
| 1206 |
+
309
|
| 1207 |
+
],
|
| 1208 |
+
[
|
| 1209 |
+
"name",
|
| 1210 |
+
"given-name",
|
| 1211 |
+
148
|
| 1212 |
+
],
|
| 1213 |
+
[
|
| 1214 |
+
"name",
|
| 1215 |
+
"other",
|
| 1216 |
+
135
|
| 1217 |
+
],
|
| 1218 |
+
[
|
| 1219 |
+
"other",
|
| 1220 |
+
"search",
|
| 1221 |
+
134
|
| 1222 |
+
],
|
| 1223 |
+
[
|
| 1224 |
+
"search",
|
| 1225 |
+
"other",
|
| 1226 |
+
128
|
| 1227 |
+
],
|
| 1228 |
+
[
|
| 1229 |
+
"other",
|
| 1230 |
+
"email",
|
| 1231 |
+
118
|
| 1232 |
+
],
|
| 1233 |
+
[
|
| 1234 |
+
"other",
|
| 1235 |
+
"new-password",
|
| 1236 |
+
117
|
| 1237 |
+
],
|
| 1238 |
+
[
|
| 1239 |
+
"other",
|
| 1240 |
+
"address-level2",
|
| 1241 |
+
108
|
| 1242 |
+
],
|
| 1243 |
+
[
|
| 1244 |
+
"captcha",
|
| 1245 |
+
"other",
|
| 1246 |
+
100
|
| 1247 |
+
],
|
| 1248 |
+
[
|
| 1249 |
+
"given-name",
|
| 1250 |
+
"name",
|
| 1251 |
+
91
|
| 1252 |
+
],
|
| 1253 |
+
[
|
| 1254 |
+
"other",
|
| 1255 |
+
"name",
|
| 1256 |
+
89
|
| 1257 |
+
],
|
| 1258 |
+
[
|
| 1259 |
+
"other",
|
| 1260 |
+
"username",
|
| 1261 |
+
84
|
| 1262 |
+
],
|
| 1263 |
+
[
|
| 1264 |
+
"other",
|
| 1265 |
+
"tel",
|
| 1266 |
+
84
|
| 1267 |
+
],
|
| 1268 |
+
[
|
| 1269 |
+
"other",
|
| 1270 |
+
"address-level1",
|
| 1271 |
+
82
|
| 1272 |
+
],
|
| 1273 |
+
[
|
| 1274 |
+
"family-name",
|
| 1275 |
+
"other",
|
| 1276 |
+
63
|
| 1277 |
+
],
|
| 1278 |
+
[
|
| 1279 |
+
"email",
|
| 1280 |
+
"username",
|
| 1281 |
+
62
|
| 1282 |
+
],
|
| 1283 |
+
[
|
| 1284 |
+
"tel",
|
| 1285 |
+
"other",
|
| 1286 |
+
54
|
| 1287 |
+
],
|
| 1288 |
+
[
|
| 1289 |
+
"name",
|
| 1290 |
+
"username",
|
| 1291 |
+
52
|
| 1292 |
+
],
|
| 1293 |
+
[
|
| 1294 |
+
"username",
|
| 1295 |
+
"other",
|
| 1296 |
+
49
|
| 1297 |
+
],
|
| 1298 |
+
[
|
| 1299 |
+
"given-name",
|
| 1300 |
+
"other",
|
| 1301 |
+
39
|
| 1302 |
+
],
|
| 1303 |
+
[
|
| 1304 |
+
"other",
|
| 1305 |
+
"street-address",
|
| 1306 |
+
39
|
| 1307 |
+
],
|
| 1308 |
+
[
|
| 1309 |
+
"one-time-code",
|
| 1310 |
+
"other",
|
| 1311 |
+
39
|
| 1312 |
+
],
|
| 1313 |
+
[
|
| 1314 |
+
"other",
|
| 1315 |
+
"current-password",
|
| 1316 |
+
32
|
| 1317 |
+
],
|
| 1318 |
+
[
|
| 1319 |
+
"other",
|
| 1320 |
+
"address-line2",
|
| 1321 |
+
31
|
| 1322 |
+
]
|
| 1323 |
+
]
|
| 1324 |
+
},
|
| 1325 |
+
"autocomplete_masked": {
|
| 1326 |
+
"n_fields": 1622,
|
| 1327 |
+
"acc": 0.9784217016029593,
|
| 1328 |
+
"per_class": {
|
| 1329 |
+
"address-level1": "4/4",
|
| 1330 |
+
"address-level2": "12/14",
|
| 1331 |
+
"address-line2": "0/3",
|
| 1332 |
+
"bday": "5/5",
|
| 1333 |
+
"cc-csc": "1/1",
|
| 1334 |
+
"cc-exp-month": "1/1",
|
| 1335 |
+
"cc-exp-year": "1/1",
|
| 1336 |
+
"cc-number": "1/1",
|
| 1337 |
+
"country": "6/6",
|
| 1338 |
+
"current-password": "323/324",
|
| 1339 |
+
"email": "417/418",
|
| 1340 |
+
"family-name": "34/37",
|
| 1341 |
+
"given-name": "39/42",
|
| 1342 |
+
"name": "182/187",
|
| 1343 |
+
"new-password": "180/180",
|
| 1344 |
+
"one-time-code": "7/7",
|
| 1345 |
+
"organization": "16/16",
|
| 1346 |
+
"postal-code": "26/30",
|
| 1347 |
+
"street-address": "11/13",
|
| 1348 |
+
"tel": "111/111",
|
| 1349 |
+
"username": "210/221"
|
| 1350 |
+
},
|
| 1351 |
+
"heuristic_acc": 0.7866831072749692
|
| 1352 |
+
}
|
| 1353 |
+
}
|
eval/eval_val_bfloat16.json
ADDED
|
@@ -0,0 +1,1338 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"split": "val",
|
| 3 |
+
"dtype": "bfloat16",
|
| 4 |
+
"forms_per_s_gpu": 2872.840982485025,
|
| 5 |
+
"model": {
|
| 6 |
+
"n_forms": 3408,
|
| 7 |
+
"n_fields": 9819,
|
| 8 |
+
"field_acc": 0.9747428455036155,
|
| 9 |
+
"field_macro_f1": 0.9662059517142381,
|
| 10 |
+
"form_acc": 0.9568661971830986,
|
| 11 |
+
"form_macro_f1": 0.9375185465621217,
|
| 12 |
+
"password_precision": 0.9898358092259578,
|
| 13 |
+
"password_recall": 0.998422712933754,
|
| 14 |
+
"password_f1": 0.994110718492344,
|
| 15 |
+
"cur_vs_new_acc": 0.9944794952681388,
|
| 16 |
+
"identifier_f1": 0.9945380063723259,
|
| 17 |
+
"per_field": {
|
| 18 |
+
"username": {
|
| 19 |
+
"p": 0.9921752738654147,
|
| 20 |
+
"r": 0.9768875192604006,
|
| 21 |
+
"f1": 0.984472049689441,
|
| 22 |
+
"n": 649
|
| 23 |
+
},
|
| 24 |
+
"email": {
|
| 25 |
+
"p": 0.9884541372674791,
|
| 26 |
+
"r": 0.9961215255332903,
|
| 27 |
+
"f1": 0.9922730199613651,
|
| 28 |
+
"n": 1547
|
| 29 |
+
},
|
| 30 |
+
"current-password": {
|
| 31 |
+
"p": 0.9814814814814815,
|
| 32 |
+
"r": 0.9977851605758582,
|
| 33 |
+
"f1": 0.9895661724327293,
|
| 34 |
+
"n": 903
|
| 35 |
+
},
|
| 36 |
+
"new-password": {
|
| 37 |
+
"p": 0.997229916897507,
|
| 38 |
+
"r": 0.9863013698630136,
|
| 39 |
+
"f1": 0.9917355371900827,
|
| 40 |
+
"n": 365
|
| 41 |
+
},
|
| 42 |
+
"one-time-code": {
|
| 43 |
+
"p": 0.9901960784313726,
|
| 44 |
+
"r": 0.9805825242718447,
|
| 45 |
+
"f1": 0.9853658536585367,
|
| 46 |
+
"n": 103
|
| 47 |
+
},
|
| 48 |
+
"tel": {
|
| 49 |
+
"p": 0.9815950920245399,
|
| 50 |
+
"r": 0.9937888198757764,
|
| 51 |
+
"f1": 0.9876543209876544,
|
| 52 |
+
"n": 644
|
| 53 |
+
},
|
| 54 |
+
"name": {
|
| 55 |
+
"p": 0.9592123769338959,
|
| 56 |
+
"r": 0.9855491329479769,
|
| 57 |
+
"f1": 0.9722024233784746,
|
| 58 |
+
"n": 692
|
| 59 |
+
},
|
| 60 |
+
"given-name": {
|
| 61 |
+
"p": 0.978494623655914,
|
| 62 |
+
"r": 0.9161073825503355,
|
| 63 |
+
"f1": 0.9462738301559791,
|
| 64 |
+
"n": 298
|
| 65 |
+
},
|
| 66 |
+
"family-name": {
|
| 67 |
+
"p": 0.9961089494163424,
|
| 68 |
+
"r": 0.9808429118773946,
|
| 69 |
+
"f1": 0.9884169884169886,
|
| 70 |
+
"n": 261
|
| 71 |
+
},
|
| 72 |
+
"organization": {
|
| 73 |
+
"p": 0.9761904761904762,
|
| 74 |
+
"r": 0.9318181818181818,
|
| 75 |
+
"f1": 0.9534883720930233,
|
| 76 |
+
"n": 132
|
| 77 |
+
},
|
| 78 |
+
"street-address": {
|
| 79 |
+
"p": 0.9369369369369369,
|
| 80 |
+
"r": 0.9122807017543859,
|
| 81 |
+
"f1": 0.9244444444444444,
|
| 82 |
+
"n": 114
|
| 83 |
+
},
|
| 84 |
+
"address-line2": {
|
| 85 |
+
"p": 1.0,
|
| 86 |
+
"r": 0.9090909090909091,
|
| 87 |
+
"f1": 0.9523809523809523,
|
| 88 |
+
"n": 22
|
| 89 |
+
},
|
| 90 |
+
"address-level2": {
|
| 91 |
+
"p": 0.8843537414965986,
|
| 92 |
+
"r": 0.8904109589041096,
|
| 93 |
+
"f1": 0.887372013651877,
|
| 94 |
+
"n": 146
|
| 95 |
+
},
|
| 96 |
+
"address-level1": {
|
| 97 |
+
"p": 0.898989898989899,
|
| 98 |
+
"r": 0.9368421052631579,
|
| 99 |
+
"f1": 0.9175257731958762,
|
| 100 |
+
"n": 95
|
| 101 |
+
},
|
| 102 |
+
"postal-code": {
|
| 103 |
+
"p": 0.9724137931034482,
|
| 104 |
+
"r": 0.9724137931034482,
|
| 105 |
+
"f1": 0.9724137931034482,
|
| 106 |
+
"n": 145
|
| 107 |
+
},
|
| 108 |
+
"country": {
|
| 109 |
+
"p": 0.8863636363636364,
|
| 110 |
+
"r": 0.9873417721518988,
|
| 111 |
+
"f1": 0.9341317365269461,
|
| 112 |
+
"n": 79
|
| 113 |
+
},
|
| 114 |
+
"cc-name": {
|
| 115 |
+
"p": 1.0,
|
| 116 |
+
"r": 0.95,
|
| 117 |
+
"f1": 0.9743589743589743,
|
| 118 |
+
"n": 20
|
| 119 |
+
},
|
| 120 |
+
"cc-number": {
|
| 121 |
+
"p": 0.9565217391304348,
|
| 122 |
+
"r": 1.0,
|
| 123 |
+
"f1": 0.9777777777777777,
|
| 124 |
+
"n": 22
|
| 125 |
+
},
|
| 126 |
+
"cc-exp": {
|
| 127 |
+
"p": 1.0,
|
| 128 |
+
"r": 1.0,
|
| 129 |
+
"f1": 1.0,
|
| 130 |
+
"n": 13
|
| 131 |
+
},
|
| 132 |
+
"cc-exp-month": {
|
| 133 |
+
"p": 1.0,
|
| 134 |
+
"r": 1.0,
|
| 135 |
+
"f1": 1.0,
|
| 136 |
+
"n": 9
|
| 137 |
+
},
|
| 138 |
+
"cc-exp-year": {
|
| 139 |
+
"p": 1.0,
|
| 140 |
+
"r": 1.0,
|
| 141 |
+
"f1": 1.0,
|
| 142 |
+
"n": 8
|
| 143 |
+
},
|
| 144 |
+
"cc-csc": {
|
| 145 |
+
"p": 0.9523809523809523,
|
| 146 |
+
"r": 1.0,
|
| 147 |
+
"f1": 0.975609756097561,
|
| 148 |
+
"n": 20
|
| 149 |
+
},
|
| 150 |
+
"bday": {
|
| 151 |
+
"p": 1.0,
|
| 152 |
+
"r": 0.8461538461538461,
|
| 153 |
+
"f1": 0.9166666666666666,
|
| 154 |
+
"n": 39
|
| 155 |
+
},
|
| 156 |
+
"search": {
|
| 157 |
+
"p": 0.9624217118997912,
|
| 158 |
+
"r": 0.9808510638297873,
|
| 159 |
+
"f1": 0.9715489989462592,
|
| 160 |
+
"n": 470
|
| 161 |
+
},
|
| 162 |
+
"captcha": {
|
| 163 |
+
"p": 0.9814814814814815,
|
| 164 |
+
"r": 0.9352941176470588,
|
| 165 |
+
"f1": 0.9578313253012049,
|
| 166 |
+
"n": 170
|
| 167 |
+
},
|
| 168 |
+
"other": {
|
| 169 |
+
"p": 0.9704016913319239,
|
| 170 |
+
"r": 0.9652996845425867,
|
| 171 |
+
"f1": 0.9678439641539274,
|
| 172 |
+
"n": 2853
|
| 173 |
+
}
|
| 174 |
+
},
|
| 175 |
+
"per_form": {
|
| 176 |
+
"login": {
|
| 177 |
+
"p": 0.9900552486187846,
|
| 178 |
+
"r": 0.9933481152993349,
|
| 179 |
+
"f1": 0.991698948533481,
|
| 180 |
+
"n": 902
|
| 181 |
+
},
|
| 182 |
+
"signup": {
|
| 183 |
+
"p": 0.9951456310679612,
|
| 184 |
+
"r": 0.9855769230769231,
|
| 185 |
+
"f1": 0.9903381642512078,
|
| 186 |
+
"n": 208
|
| 187 |
+
},
|
| 188 |
+
"password-change": {
|
| 189 |
+
"p": 1.0,
|
| 190 |
+
"r": 1.0,
|
| 191 |
+
"f1": 1.0,
|
| 192 |
+
"n": 22
|
| 193 |
+
},
|
| 194 |
+
"password-recovery": {
|
| 195 |
+
"p": 0.8461538461538461,
|
| 196 |
+
"r": 0.9565217391304348,
|
| 197 |
+
"f1": 0.8979591836734695,
|
| 198 |
+
"n": 23
|
| 199 |
+
},
|
| 200 |
+
"otp": {
|
| 201 |
+
"p": 0.8888888888888888,
|
| 202 |
+
"r": 0.8571428571428571,
|
| 203 |
+
"f1": 0.8727272727272727,
|
| 204 |
+
"n": 28
|
| 205 |
+
},
|
| 206 |
+
"payment": {
|
| 207 |
+
"p": 1.0,
|
| 208 |
+
"r": 1.0,
|
| 209 |
+
"f1": 1.0,
|
| 210 |
+
"n": 22
|
| 211 |
+
},
|
| 212 |
+
"address": {
|
| 213 |
+
"p": 0.8666666666666667,
|
| 214 |
+
"r": 0.7222222222222222,
|
| 215 |
+
"f1": 0.7878787878787877,
|
| 216 |
+
"n": 36
|
| 217 |
+
},
|
| 218 |
+
"newsletter": {
|
| 219 |
+
"p": 0.9774193548387097,
|
| 220 |
+
"r": 0.9439252336448598,
|
| 221 |
+
"f1": 0.9603803486529319,
|
| 222 |
+
"n": 321
|
| 223 |
+
},
|
| 224 |
+
"contact": {
|
| 225 |
+
"p": 0.9369482976040353,
|
| 226 |
+
"r": 0.9750656167979003,
|
| 227 |
+
"f1": 0.9556270096463023,
|
| 228 |
+
"n": 762
|
| 229 |
+
},
|
| 230 |
+
"search": {
|
| 231 |
+
"p": 0.9554730983302412,
|
| 232 |
+
"r": 0.9662288930581614,
|
| 233 |
+
"f1": 0.9608208955223881,
|
| 234 |
+
"n": 533
|
| 235 |
+
},
|
| 236 |
+
"other": {
|
| 237 |
+
"p": 0.9147727272727273,
|
| 238 |
+
"r": 0.8765880217785844,
|
| 239 |
+
"f1": 0.8952734012974978,
|
| 240 |
+
"n": 551
|
| 241 |
+
}
|
| 242 |
+
},
|
| 243 |
+
"top_confusions": [
|
| 244 |
+
[
|
| 245 |
+
"given-name",
|
| 246 |
+
"name",
|
| 247 |
+
22
|
| 248 |
+
],
|
| 249 |
+
[
|
| 250 |
+
"other",
|
| 251 |
+
"address-level2",
|
| 252 |
+
16
|
| 253 |
+
],
|
| 254 |
+
[
|
| 255 |
+
"other",
|
| 256 |
+
"search",
|
| 257 |
+
16
|
| 258 |
+
],
|
| 259 |
+
[
|
| 260 |
+
"address-level2",
|
| 261 |
+
"other",
|
| 262 |
+
15
|
| 263 |
+
],
|
| 264 |
+
[
|
| 265 |
+
"other",
|
| 266 |
+
"current-password",
|
| 267 |
+
11
|
| 268 |
+
],
|
| 269 |
+
[
|
| 270 |
+
"other",
|
| 271 |
+
"address-level1",
|
| 272 |
+
10
|
| 273 |
+
],
|
| 274 |
+
[
|
| 275 |
+
"captcha",
|
| 276 |
+
"other",
|
| 277 |
+
9
|
| 278 |
+
],
|
| 279 |
+
[
|
| 280 |
+
"other",
|
| 281 |
+
"country",
|
| 282 |
+
9
|
| 283 |
+
],
|
| 284 |
+
[
|
| 285 |
+
"search",
|
| 286 |
+
"other",
|
| 287 |
+
9
|
| 288 |
+
],
|
| 289 |
+
[
|
| 290 |
+
"other",
|
| 291 |
+
"email",
|
| 292 |
+
9
|
| 293 |
+
],
|
| 294 |
+
[
|
| 295 |
+
"username",
|
| 296 |
+
"email",
|
| 297 |
+
8
|
| 298 |
+
],
|
| 299 |
+
[
|
| 300 |
+
"street-address",
|
| 301 |
+
"other",
|
| 302 |
+
7
|
| 303 |
+
],
|
| 304 |
+
[
|
| 305 |
+
"organization",
|
| 306 |
+
"other",
|
| 307 |
+
7
|
| 308 |
+
],
|
| 309 |
+
[
|
| 310 |
+
"address-level1",
|
| 311 |
+
"other",
|
| 312 |
+
6
|
| 313 |
+
],
|
| 314 |
+
[
|
| 315 |
+
"name",
|
| 316 |
+
"given-name",
|
| 317 |
+
5
|
| 318 |
+
],
|
| 319 |
+
[
|
| 320 |
+
"username",
|
| 321 |
+
"other",
|
| 322 |
+
5
|
| 323 |
+
],
|
| 324 |
+
[
|
| 325 |
+
"other",
|
| 326 |
+
"tel",
|
| 327 |
+
5
|
| 328 |
+
],
|
| 329 |
+
[
|
| 330 |
+
"other",
|
| 331 |
+
"street-address",
|
| 332 |
+
5
|
| 333 |
+
],
|
| 334 |
+
[
|
| 335 |
+
"bday",
|
| 336 |
+
"other",
|
| 337 |
+
5
|
| 338 |
+
],
|
| 339 |
+
[
|
| 340 |
+
"new-password",
|
| 341 |
+
"current-password",
|
| 342 |
+
4
|
| 343 |
+
],
|
| 344 |
+
[
|
| 345 |
+
"name",
|
| 346 |
+
"other",
|
| 347 |
+
4
|
| 348 |
+
],
|
| 349 |
+
[
|
| 350 |
+
"other",
|
| 351 |
+
"postal-code",
|
| 352 |
+
4
|
| 353 |
+
],
|
| 354 |
+
[
|
| 355 |
+
"postal-code",
|
| 356 |
+
"other",
|
| 357 |
+
4
|
| 358 |
+
],
|
| 359 |
+
[
|
| 360 |
+
"other",
|
| 361 |
+
"name",
|
| 362 |
+
3
|
| 363 |
+
],
|
| 364 |
+
[
|
| 365 |
+
"family-name",
|
| 366 |
+
"name",
|
| 367 |
+
3
|
| 368 |
+
]
|
| 369 |
+
]
|
| 370 |
+
},
|
| 371 |
+
"model_real": {
|
| 372 |
+
"n_forms": 3362,
|
| 373 |
+
"n_fields": 9619,
|
| 374 |
+
"field_acc": 0.9747374987004886,
|
| 375 |
+
"field_macro_f1": 0.9605143456394969,
|
| 376 |
+
"form_acc": 0.9562760261748959,
|
| 377 |
+
"form_macro_f1": 0.9290284576847291,
|
| 378 |
+
"password_precision": 0.9912490055688147,
|
| 379 |
+
"password_recall": 0.9983974358974359,
|
| 380 |
+
"password_f1": 0.9948103792415169,
|
| 381 |
+
"cur_vs_new_acc": 0.9951923076923077,
|
| 382 |
+
"identifier_f1": 0.9945054945054944,
|
| 383 |
+
"per_field": {
|
| 384 |
+
"username": {
|
| 385 |
+
"p": 0.9921135646687698,
|
| 386 |
+
"r": 0.9767080745341615,
|
| 387 |
+
"f1": 0.9843505477308295,
|
| 388 |
+
"n": 644
|
| 389 |
+
},
|
| 390 |
+
"email": {
|
| 391 |
+
"p": 0.988394584139265,
|
| 392 |
+
"r": 0.9961013645224172,
|
| 393 |
+
"f1": 0.992233009708738,
|
| 394 |
+
"n": 1539
|
| 395 |
+
},
|
| 396 |
+
"current-password": {
|
| 397 |
+
"p": 0.9835706462212487,
|
| 398 |
+
"r": 0.9988876529477196,
|
| 399 |
+
"f1": 0.9911699779249449,
|
| 400 |
+
"n": 899
|
| 401 |
+
},
|
| 402 |
+
"new-password": {
|
| 403 |
+
"p": 1.0,
|
| 404 |
+
"r": 0.9856733524355301,
|
| 405 |
+
"f1": 0.9927849927849928,
|
| 406 |
+
"n": 349
|
| 407 |
+
},
|
| 408 |
+
"one-time-code": {
|
| 409 |
+
"p": 0.9868421052631579,
|
| 410 |
+
"r": 0.974025974025974,
|
| 411 |
+
"f1": 0.9803921568627451,
|
| 412 |
+
"n": 77
|
| 413 |
+
},
|
| 414 |
+
"tel": {
|
| 415 |
+
"p": 0.9813953488372092,
|
| 416 |
+
"r": 0.9937205651491365,
|
| 417 |
+
"f1": 0.987519500780031,
|
| 418 |
+
"n": 637
|
| 419 |
+
},
|
| 420 |
+
"name": {
|
| 421 |
+
"p": 0.9588652482269504,
|
| 422 |
+
"r": 0.9854227405247813,
|
| 423 |
+
"f1": 0.9719626168224299,
|
| 424 |
+
"n": 686
|
| 425 |
+
},
|
| 426 |
+
"given-name": {
|
| 427 |
+
"p": 0.9783393501805054,
|
| 428 |
+
"r": 0.9155405405405406,
|
| 429 |
+
"f1": 0.9458987783595114,
|
| 430 |
+
"n": 296
|
| 431 |
+
},
|
| 432 |
+
"family-name": {
|
| 433 |
+
"p": 0.996078431372549,
|
| 434 |
+
"r": 0.9806949806949807,
|
| 435 |
+
"f1": 0.9883268482490273,
|
| 436 |
+
"n": 259
|
| 437 |
+
},
|
| 438 |
+
"organization": {
|
| 439 |
+
"p": 0.9758064516129032,
|
| 440 |
+
"r": 0.9307692307692308,
|
| 441 |
+
"f1": 0.952755905511811,
|
| 442 |
+
"n": 130
|
| 443 |
+
},
|
| 444 |
+
"street-address": {
|
| 445 |
+
"p": 0.9326923076923077,
|
| 446 |
+
"r": 0.9065420560747663,
|
| 447 |
+
"f1": 0.919431279620853,
|
| 448 |
+
"n": 107
|
| 449 |
+
},
|
| 450 |
+
"address-line2": {
|
| 451 |
+
"p": 1.0,
|
| 452 |
+
"r": 0.8823529411764706,
|
| 453 |
+
"f1": 0.9375,
|
| 454 |
+
"n": 17
|
| 455 |
+
},
|
| 456 |
+
"address-level2": {
|
| 457 |
+
"p": 0.8794326241134752,
|
| 458 |
+
"r": 0.8857142857142857,
|
| 459 |
+
"f1": 0.8825622775800712,
|
| 460 |
+
"n": 140
|
| 461 |
+
},
|
| 462 |
+
"address-level1": {
|
| 463 |
+
"p": 0.8947368421052632,
|
| 464 |
+
"r": 0.9340659340659341,
|
| 465 |
+
"f1": 0.913978494623656,
|
| 466 |
+
"n": 91
|
| 467 |
+
},
|
| 468 |
+
"postal-code": {
|
| 469 |
+
"p": 0.9708029197080292,
|
| 470 |
+
"r": 0.9708029197080292,
|
| 471 |
+
"f1": 0.9708029197080292,
|
| 472 |
+
"n": 137
|
| 473 |
+
},
|
| 474 |
+
"country": {
|
| 475 |
+
"p": 0.8809523809523809,
|
| 476 |
+
"r": 0.9866666666666667,
|
| 477 |
+
"f1": 0.930817610062893,
|
| 478 |
+
"n": 75
|
| 479 |
+
},
|
| 480 |
+
"cc-name": {
|
| 481 |
+
"p": 1.0,
|
| 482 |
+
"r": 1.0,
|
| 483 |
+
"f1": 1.0,
|
| 484 |
+
"n": 3
|
| 485 |
+
},
|
| 486 |
+
"cc-number": {
|
| 487 |
+
"p": 0.75,
|
| 488 |
+
"r": 1.0,
|
| 489 |
+
"f1": 0.8571428571428571,
|
| 490 |
+
"n": 3
|
| 491 |
+
},
|
| 492 |
+
"cc-exp-month": {
|
| 493 |
+
"p": 1.0,
|
| 494 |
+
"r": 1.0,
|
| 495 |
+
"f1": 1.0,
|
| 496 |
+
"n": 3
|
| 497 |
+
},
|
| 498 |
+
"cc-exp-year": {
|
| 499 |
+
"p": 1.0,
|
| 500 |
+
"r": 1.0,
|
| 501 |
+
"f1": 1.0,
|
| 502 |
+
"n": 3
|
| 503 |
+
},
|
| 504 |
+
"cc-csc": {
|
| 505 |
+
"p": 1.0,
|
| 506 |
+
"r": 1.0,
|
| 507 |
+
"f1": 1.0,
|
| 508 |
+
"n": 3
|
| 509 |
+
},
|
| 510 |
+
"bday": {
|
| 511 |
+
"p": 1.0,
|
| 512 |
+
"r": 0.8461538461538461,
|
| 513 |
+
"f1": 0.9166666666666666,
|
| 514 |
+
"n": 39
|
| 515 |
+
},
|
| 516 |
+
"search": {
|
| 517 |
+
"p": 0.9624217118997912,
|
| 518 |
+
"r": 0.9808510638297873,
|
| 519 |
+
"f1": 0.9715489989462592,
|
| 520 |
+
"n": 470
|
| 521 |
+
},
|
| 522 |
+
"captcha": {
|
| 523 |
+
"p": 0.9808917197452229,
|
| 524 |
+
"r": 0.9333333333333333,
|
| 525 |
+
"f1": 0.9565217391304348,
|
| 526 |
+
"n": 165
|
| 527 |
+
},
|
| 528 |
+
"other": {
|
| 529 |
+
"p": 0.970712773465067,
|
| 530 |
+
"r": 0.9662802950474183,
|
| 531 |
+
"f1": 0.968491462770639,
|
| 532 |
+
"n": 2847
|
| 533 |
+
}
|
| 534 |
+
},
|
| 535 |
+
"per_form": {
|
| 536 |
+
"login": {
|
| 537 |
+
"p": 0.9900110987791343,
|
| 538 |
+
"r": 0.9933184855233853,
|
| 539 |
+
"f1": 0.9916620344635908,
|
| 540 |
+
"n": 898
|
| 541 |
+
},
|
| 542 |
+
"signup": {
|
| 543 |
+
"p": 0.9951219512195122,
|
| 544 |
+
"r": 0.9855072463768116,
|
| 545 |
+
"f1": 0.9902912621359223,
|
| 546 |
+
"n": 207
|
| 547 |
+
},
|
| 548 |
+
"password-change": {
|
| 549 |
+
"p": 1.0,
|
| 550 |
+
"r": 1.0,
|
| 551 |
+
"f1": 1.0,
|
| 552 |
+
"n": 15
|
| 553 |
+
},
|
| 554 |
+
"password-recovery": {
|
| 555 |
+
"p": 0.8181818181818182,
|
| 556 |
+
"r": 0.9473684210526315,
|
| 557 |
+
"f1": 0.8780487804878049,
|
| 558 |
+
"n": 19
|
| 559 |
+
},
|
| 560 |
+
"otp": {
|
| 561 |
+
"p": 0.8571428571428571,
|
| 562 |
+
"r": 0.8181818181818182,
|
| 563 |
+
"f1": 0.8372093023255814,
|
| 564 |
+
"n": 22
|
| 565 |
+
},
|
| 566 |
+
"payment": {
|
| 567 |
+
"p": 1.0,
|
| 568 |
+
"r": 1.0,
|
| 569 |
+
"f1": 1.0,
|
| 570 |
+
"n": 3
|
| 571 |
+
},
|
| 572 |
+
"address": {
|
| 573 |
+
"p": 0.84,
|
| 574 |
+
"r": 0.6774193548387096,
|
| 575 |
+
"f1": 0.75,
|
| 576 |
+
"n": 31
|
| 577 |
+
},
|
| 578 |
+
"newsletter": {
|
| 579 |
+
"p": 0.9774193548387097,
|
| 580 |
+
"r": 0.9439252336448598,
|
| 581 |
+
"f1": 0.9603803486529319,
|
| 582 |
+
"n": 321
|
| 583 |
+
},
|
| 584 |
+
"contact": {
|
| 585 |
+
"p": 0.9369482976040353,
|
| 586 |
+
"r": 0.9750656167979003,
|
| 587 |
+
"f1": 0.9556270096463023,
|
| 588 |
+
"n": 762
|
| 589 |
+
},
|
| 590 |
+
"search": {
|
| 591 |
+
"p": 0.9554730983302412,
|
| 592 |
+
"r": 0.9662288930581614,
|
| 593 |
+
"f1": 0.9608208955223881,
|
| 594 |
+
"n": 533
|
| 595 |
+
},
|
| 596 |
+
"other": {
|
| 597 |
+
"p": 0.9147727272727273,
|
| 598 |
+
"r": 0.8765880217785844,
|
| 599 |
+
"f1": 0.8952734012974978,
|
| 600 |
+
"n": 551
|
| 601 |
+
}
|
| 602 |
+
},
|
| 603 |
+
"top_confusions": [
|
| 604 |
+
[
|
| 605 |
+
"given-name",
|
| 606 |
+
"name",
|
| 607 |
+
22
|
| 608 |
+
],
|
| 609 |
+
[
|
| 610 |
+
"other",
|
| 611 |
+
"address-level2",
|
| 612 |
+
16
|
| 613 |
+
],
|
| 614 |
+
[
|
| 615 |
+
"other",
|
| 616 |
+
"search",
|
| 617 |
+
16
|
| 618 |
+
],
|
| 619 |
+
[
|
| 620 |
+
"address-level2",
|
| 621 |
+
"other",
|
| 622 |
+
15
|
| 623 |
+
],
|
| 624 |
+
[
|
| 625 |
+
"other",
|
| 626 |
+
"address-level1",
|
| 627 |
+
10
|
| 628 |
+
],
|
| 629 |
+
[
|
| 630 |
+
"captcha",
|
| 631 |
+
"other",
|
| 632 |
+
9
|
| 633 |
+
],
|
| 634 |
+
[
|
| 635 |
+
"other",
|
| 636 |
+
"country",
|
| 637 |
+
9
|
| 638 |
+
],
|
| 639 |
+
[
|
| 640 |
+
"search",
|
| 641 |
+
"other",
|
| 642 |
+
9
|
| 643 |
+
],
|
| 644 |
+
[
|
| 645 |
+
"other",
|
| 646 |
+
"email",
|
| 647 |
+
9
|
| 648 |
+
],
|
| 649 |
+
[
|
| 650 |
+
"other",
|
| 651 |
+
"current-password",
|
| 652 |
+
9
|
| 653 |
+
],
|
| 654 |
+
[
|
| 655 |
+
"username",
|
| 656 |
+
"email",
|
| 657 |
+
8
|
| 658 |
+
],
|
| 659 |
+
[
|
| 660 |
+
"street-address",
|
| 661 |
+
"other",
|
| 662 |
+
7
|
| 663 |
+
],
|
| 664 |
+
[
|
| 665 |
+
"organization",
|
| 666 |
+
"other",
|
| 667 |
+
7
|
| 668 |
+
],
|
| 669 |
+
[
|
| 670 |
+
"address-level1",
|
| 671 |
+
"other",
|
| 672 |
+
6
|
| 673 |
+
],
|
| 674 |
+
[
|
| 675 |
+
"name",
|
| 676 |
+
"given-name",
|
| 677 |
+
5
|
| 678 |
+
],
|
| 679 |
+
[
|
| 680 |
+
"username",
|
| 681 |
+
"other",
|
| 682 |
+
5
|
| 683 |
+
],
|
| 684 |
+
[
|
| 685 |
+
"other",
|
| 686 |
+
"tel",
|
| 687 |
+
5
|
| 688 |
+
],
|
| 689 |
+
[
|
| 690 |
+
"other",
|
| 691 |
+
"street-address",
|
| 692 |
+
5
|
| 693 |
+
],
|
| 694 |
+
[
|
| 695 |
+
"bday",
|
| 696 |
+
"other",
|
| 697 |
+
5
|
| 698 |
+
],
|
| 699 |
+
[
|
| 700 |
+
"new-password",
|
| 701 |
+
"current-password",
|
| 702 |
+
4
|
| 703 |
+
],
|
| 704 |
+
[
|
| 705 |
+
"name",
|
| 706 |
+
"other",
|
| 707 |
+
4
|
| 708 |
+
],
|
| 709 |
+
[
|
| 710 |
+
"other",
|
| 711 |
+
"postal-code",
|
| 712 |
+
4
|
| 713 |
+
],
|
| 714 |
+
[
|
| 715 |
+
"postal-code",
|
| 716 |
+
"other",
|
| 717 |
+
4
|
| 718 |
+
],
|
| 719 |
+
[
|
| 720 |
+
"other",
|
| 721 |
+
"name",
|
| 722 |
+
3
|
| 723 |
+
],
|
| 724 |
+
[
|
| 725 |
+
"family-name",
|
| 726 |
+
"name",
|
| 727 |
+
3
|
| 728 |
+
]
|
| 729 |
+
]
|
| 730 |
+
},
|
| 731 |
+
"model_synthetic": {
|
| 732 |
+
"n_forms": 46,
|
| 733 |
+
"n_fields": 200,
|
| 734 |
+
"field_acc": 0.975,
|
| 735 |
+
"field_macro_f1": 0.9657287157287158,
|
| 736 |
+
"form_acc": 1.0,
|
| 737 |
+
"form_macro_f1": 1.0,
|
| 738 |
+
"password_precision": 0.9090909090909091,
|
| 739 |
+
"password_recall": 1.0,
|
| 740 |
+
"password_f1": 0.9523809523809523,
|
| 741 |
+
"cur_vs_new_acc": 0.95,
|
| 742 |
+
"identifier_f1": 1.0,
|
| 743 |
+
"per_field": {
|
| 744 |
+
"username": {
|
| 745 |
+
"p": 1.0,
|
| 746 |
+
"r": 1.0,
|
| 747 |
+
"f1": 1.0,
|
| 748 |
+
"n": 5
|
| 749 |
+
},
|
| 750 |
+
"email": {
|
| 751 |
+
"p": 1.0,
|
| 752 |
+
"r": 1.0,
|
| 753 |
+
"f1": 1.0,
|
| 754 |
+
"n": 8
|
| 755 |
+
},
|
| 756 |
+
"current-password": {
|
| 757 |
+
"p": 0.6,
|
| 758 |
+
"r": 0.75,
|
| 759 |
+
"f1": 0.6666666666666665,
|
| 760 |
+
"n": 4
|
| 761 |
+
},
|
| 762 |
+
"new-password": {
|
| 763 |
+
"p": 0.9411764705882353,
|
| 764 |
+
"r": 1.0,
|
| 765 |
+
"f1": 0.9696969696969697,
|
| 766 |
+
"n": 16
|
| 767 |
+
},
|
| 768 |
+
"one-time-code": {
|
| 769 |
+
"p": 1.0,
|
| 770 |
+
"r": 1.0,
|
| 771 |
+
"f1": 1.0,
|
| 772 |
+
"n": 26
|
| 773 |
+
},
|
| 774 |
+
"tel": {
|
| 775 |
+
"p": 1.0,
|
| 776 |
+
"r": 1.0,
|
| 777 |
+
"f1": 1.0,
|
| 778 |
+
"n": 7
|
| 779 |
+
},
|
| 780 |
+
"name": {
|
| 781 |
+
"p": 1.0,
|
| 782 |
+
"r": 1.0,
|
| 783 |
+
"f1": 1.0,
|
| 784 |
+
"n": 6
|
| 785 |
+
},
|
| 786 |
+
"given-name": {
|
| 787 |
+
"p": 1.0,
|
| 788 |
+
"r": 1.0,
|
| 789 |
+
"f1": 1.0,
|
| 790 |
+
"n": 2
|
| 791 |
+
},
|
| 792 |
+
"family-name": {
|
| 793 |
+
"p": 1.0,
|
| 794 |
+
"r": 1.0,
|
| 795 |
+
"f1": 1.0,
|
| 796 |
+
"n": 2
|
| 797 |
+
},
|
| 798 |
+
"organization": {
|
| 799 |
+
"p": 1.0,
|
| 800 |
+
"r": 1.0,
|
| 801 |
+
"f1": 1.0,
|
| 802 |
+
"n": 2
|
| 803 |
+
},
|
| 804 |
+
"street-address": {
|
| 805 |
+
"p": 1.0,
|
| 806 |
+
"r": 1.0,
|
| 807 |
+
"f1": 1.0,
|
| 808 |
+
"n": 7
|
| 809 |
+
},
|
| 810 |
+
"address-line2": {
|
| 811 |
+
"p": 1.0,
|
| 812 |
+
"r": 1.0,
|
| 813 |
+
"f1": 1.0,
|
| 814 |
+
"n": 5
|
| 815 |
+
},
|
| 816 |
+
"address-level2": {
|
| 817 |
+
"p": 1.0,
|
| 818 |
+
"r": 1.0,
|
| 819 |
+
"f1": 1.0,
|
| 820 |
+
"n": 6
|
| 821 |
+
},
|
| 822 |
+
"address-level1": {
|
| 823 |
+
"p": 1.0,
|
| 824 |
+
"r": 1.0,
|
| 825 |
+
"f1": 1.0,
|
| 826 |
+
"n": 4
|
| 827 |
+
},
|
| 828 |
+
"postal-code": {
|
| 829 |
+
"p": 1.0,
|
| 830 |
+
"r": 1.0,
|
| 831 |
+
"f1": 1.0,
|
| 832 |
+
"n": 8
|
| 833 |
+
},
|
| 834 |
+
"country": {
|
| 835 |
+
"p": 1.0,
|
| 836 |
+
"r": 1.0,
|
| 837 |
+
"f1": 1.0,
|
| 838 |
+
"n": 4
|
| 839 |
+
},
|
| 840 |
+
"cc-name": {
|
| 841 |
+
"p": 1.0,
|
| 842 |
+
"r": 0.9411764705882353,
|
| 843 |
+
"f1": 0.9696969696969697,
|
| 844 |
+
"n": 17
|
| 845 |
+
},
|
| 846 |
+
"cc-number": {
|
| 847 |
+
"p": 1.0,
|
| 848 |
+
"r": 1.0,
|
| 849 |
+
"f1": 1.0,
|
| 850 |
+
"n": 19
|
| 851 |
+
},
|
| 852 |
+
"cc-exp": {
|
| 853 |
+
"p": 1.0,
|
| 854 |
+
"r": 1.0,
|
| 855 |
+
"f1": 1.0,
|
| 856 |
+
"n": 13
|
| 857 |
+
},
|
| 858 |
+
"cc-exp-month": {
|
| 859 |
+
"p": 1.0,
|
| 860 |
+
"r": 1.0,
|
| 861 |
+
"f1": 1.0,
|
| 862 |
+
"n": 6
|
| 863 |
+
},
|
| 864 |
+
"cc-exp-year": {
|
| 865 |
+
"p": 1.0,
|
| 866 |
+
"r": 1.0,
|
| 867 |
+
"f1": 1.0,
|
| 868 |
+
"n": 5
|
| 869 |
+
},
|
| 870 |
+
"cc-csc": {
|
| 871 |
+
"p": 0.9444444444444444,
|
| 872 |
+
"r": 1.0,
|
| 873 |
+
"f1": 0.9714285714285714,
|
| 874 |
+
"n": 17
|
| 875 |
+
},
|
| 876 |
+
"captcha": {
|
| 877 |
+
"p": 1.0,
|
| 878 |
+
"r": 1.0,
|
| 879 |
+
"f1": 1.0,
|
| 880 |
+
"n": 5
|
| 881 |
+
},
|
| 882 |
+
"other": {
|
| 883 |
+
"p": 0.75,
|
| 884 |
+
"r": 0.5,
|
| 885 |
+
"f1": 0.6,
|
| 886 |
+
"n": 6
|
| 887 |
+
}
|
| 888 |
+
},
|
| 889 |
+
"per_form": {
|
| 890 |
+
"login": {
|
| 891 |
+
"p": 1.0,
|
| 892 |
+
"r": 1.0,
|
| 893 |
+
"f1": 1.0,
|
| 894 |
+
"n": 4
|
| 895 |
+
},
|
| 896 |
+
"signup": {
|
| 897 |
+
"p": 1.0,
|
| 898 |
+
"r": 1.0,
|
| 899 |
+
"f1": 1.0,
|
| 900 |
+
"n": 1
|
| 901 |
+
},
|
| 902 |
+
"password-change": {
|
| 903 |
+
"p": 1.0,
|
| 904 |
+
"r": 1.0,
|
| 905 |
+
"f1": 1.0,
|
| 906 |
+
"n": 7
|
| 907 |
+
},
|
| 908 |
+
"password-recovery": {
|
| 909 |
+
"p": 1.0,
|
| 910 |
+
"r": 1.0,
|
| 911 |
+
"f1": 1.0,
|
| 912 |
+
"n": 4
|
| 913 |
+
},
|
| 914 |
+
"otp": {
|
| 915 |
+
"p": 1.0,
|
| 916 |
+
"r": 1.0,
|
| 917 |
+
"f1": 1.0,
|
| 918 |
+
"n": 6
|
| 919 |
+
},
|
| 920 |
+
"payment": {
|
| 921 |
+
"p": 1.0,
|
| 922 |
+
"r": 1.0,
|
| 923 |
+
"f1": 1.0,
|
| 924 |
+
"n": 19
|
| 925 |
+
},
|
| 926 |
+
"address": {
|
| 927 |
+
"p": 1.0,
|
| 928 |
+
"r": 1.0,
|
| 929 |
+
"f1": 1.0,
|
| 930 |
+
"n": 5
|
| 931 |
+
}
|
| 932 |
+
},
|
| 933 |
+
"top_confusions": [
|
| 934 |
+
[
|
| 935 |
+
"other",
|
| 936 |
+
"current-password",
|
| 937 |
+
2
|
| 938 |
+
],
|
| 939 |
+
[
|
| 940 |
+
"cc-name",
|
| 941 |
+
"other",
|
| 942 |
+
1
|
| 943 |
+
],
|
| 944 |
+
[
|
| 945 |
+
"current-password",
|
| 946 |
+
"new-password",
|
| 947 |
+
1
|
| 948 |
+
],
|
| 949 |
+
[
|
| 950 |
+
"other",
|
| 951 |
+
"cc-csc",
|
| 952 |
+
1
|
| 953 |
+
]
|
| 954 |
+
]
|
| 955 |
+
},
|
| 956 |
+
"heuristic_baseline_real": {
|
| 957 |
+
"n_forms": 3362,
|
| 958 |
+
"n_fields": 9619,
|
| 959 |
+
"field_acc": 0.819523859029005,
|
| 960 |
+
"field_macro_f1": 0.7496204195913461,
|
| 961 |
+
"form_acc": 0.7245687091017252,
|
| 962 |
+
"form_macro_f1": 0.5030916170134985,
|
| 963 |
+
"password_precision": 0.9431818181818182,
|
| 964 |
+
"password_recall": 0.9975961538461539,
|
| 965 |
+
"password_f1": 0.9696261682242991,
|
| 966 |
+
"cur_vs_new_acc": 0.8333333333333334,
|
| 967 |
+
"identifier_f1": 0.9389296024872307,
|
| 968 |
+
"per_field": {
|
| 969 |
+
"username": {
|
| 970 |
+
"p": 0.7634782608695653,
|
| 971 |
+
"r": 0.6816770186335404,
|
| 972 |
+
"f1": 0.7202625102543068,
|
| 973 |
+
"n": 644
|
| 974 |
+
},
|
| 975 |
+
"email": {
|
| 976 |
+
"p": 0.8578796561604585,
|
| 977 |
+
"r": 0.9727095516569201,
|
| 978 |
+
"f1": 0.9116930572472595,
|
| 979 |
+
"n": 1539
|
| 980 |
+
},
|
| 981 |
+
"current-password": {
|
| 982 |
+
"p": 0.8096980786825252,
|
| 983 |
+
"r": 0.9844271412680756,
|
| 984 |
+
"f1": 0.8885542168674699,
|
| 985 |
+
"n": 899
|
| 986 |
+
},
|
| 987 |
+
"new-password": {
|
| 988 |
+
"p": 0.6828193832599119,
|
| 989 |
+
"r": 0.44412607449856734,
|
| 990 |
+
"f1": 0.5381944444444444,
|
| 991 |
+
"n": 349
|
| 992 |
+
},
|
| 993 |
+
"one-time-code": {
|
| 994 |
+
"p": 0.7538461538461538,
|
| 995 |
+
"r": 0.6363636363636364,
|
| 996 |
+
"f1": 0.6901408450704226,
|
| 997 |
+
"n": 77
|
| 998 |
+
},
|
| 999 |
+
"tel": {
|
| 1000 |
+
"p": 0.8990963855421686,
|
| 1001 |
+
"r": 0.9372056514913658,
|
| 1002 |
+
"f1": 0.9177555726364336,
|
| 1003 |
+
"n": 637
|
| 1004 |
+
},
|
| 1005 |
+
"name": {
|
| 1006 |
+
"p": 0.7942583732057417,
|
| 1007 |
+
"r": 0.7259475218658892,
|
| 1008 |
+
"f1": 0.7585681645087586,
|
| 1009 |
+
"n": 686
|
| 1010 |
+
},
|
| 1011 |
+
"given-name": {
|
| 1012 |
+
"p": 0.6656050955414012,
|
| 1013 |
+
"r": 0.706081081081081,
|
| 1014 |
+
"f1": 0.6852459016393442,
|
| 1015 |
+
"n": 296
|
| 1016 |
+
},
|
| 1017 |
+
"family-name": {
|
| 1018 |
+
"p": 0.9074074074074074,
|
| 1019 |
+
"r": 0.7567567567567568,
|
| 1020 |
+
"f1": 0.8252631578947369,
|
| 1021 |
+
"n": 259
|
| 1022 |
+
},
|
| 1023 |
+
"organization": {
|
| 1024 |
+
"p": 0.744,
|
| 1025 |
+
"r": 0.7153846153846154,
|
| 1026 |
+
"f1": 0.7294117647058823,
|
| 1027 |
+
"n": 130
|
| 1028 |
+
},
|
| 1029 |
+
"street-address": {
|
| 1030 |
+
"p": 0.7105263157894737,
|
| 1031 |
+
"r": 0.7570093457943925,
|
| 1032 |
+
"f1": 0.7330316742081447,
|
| 1033 |
+
"n": 107
|
| 1034 |
+
},
|
| 1035 |
+
"address-line2": {
|
| 1036 |
+
"p": 0.4782608695652174,
|
| 1037 |
+
"r": 0.6470588235294118,
|
| 1038 |
+
"f1": 0.55,
|
| 1039 |
+
"n": 17
|
| 1040 |
+
},
|
| 1041 |
+
"address-level2": {
|
| 1042 |
+
"p": 0.5699481865284974,
|
| 1043 |
+
"r": 0.7857142857142857,
|
| 1044 |
+
"f1": 0.6606606606606606,
|
| 1045 |
+
"n": 140
|
| 1046 |
+
},
|
| 1047 |
+
"address-level1": {
|
| 1048 |
+
"p": 0.5511811023622047,
|
| 1049 |
+
"r": 0.7692307692307693,
|
| 1050 |
+
"f1": 0.6422018348623854,
|
| 1051 |
+
"n": 91
|
| 1052 |
+
},
|
| 1053 |
+
"postal-code": {
|
| 1054 |
+
"p": 0.957983193277311,
|
| 1055 |
+
"r": 0.8321167883211679,
|
| 1056 |
+
"f1": 0.890625,
|
| 1057 |
+
"n": 137
|
| 1058 |
+
},
|
| 1059 |
+
"country": {
|
| 1060 |
+
"p": 0.8028169014084507,
|
| 1061 |
+
"r": 0.76,
|
| 1062 |
+
"f1": 0.7808219178082192,
|
| 1063 |
+
"n": 75
|
| 1064 |
+
},
|
| 1065 |
+
"cc-name": {
|
| 1066 |
+
"p": 1.0,
|
| 1067 |
+
"r": 1.0,
|
| 1068 |
+
"f1": 1.0,
|
| 1069 |
+
"n": 3
|
| 1070 |
+
},
|
| 1071 |
+
"cc-number": {
|
| 1072 |
+
"p": 0.6,
|
| 1073 |
+
"r": 1.0,
|
| 1074 |
+
"f1": 0.7499999999999999,
|
| 1075 |
+
"n": 3
|
| 1076 |
+
},
|
| 1077 |
+
"cc-exp-month": {
|
| 1078 |
+
"p": 1.0,
|
| 1079 |
+
"r": 1.0,
|
| 1080 |
+
"f1": 1.0,
|
| 1081 |
+
"n": 3
|
| 1082 |
+
},
|
| 1083 |
+
"cc-exp-year": {
|
| 1084 |
+
"p": 1.0,
|
| 1085 |
+
"r": 1.0,
|
| 1086 |
+
"f1": 1.0,
|
| 1087 |
+
"n": 3
|
| 1088 |
+
},
|
| 1089 |
+
"cc-csc": {
|
| 1090 |
+
"p": 0.0,
|
| 1091 |
+
"r": 0.0,
|
| 1092 |
+
"f1": 0.0,
|
| 1093 |
+
"n": 3
|
| 1094 |
+
},
|
| 1095 |
+
"bday": {
|
| 1096 |
+
"p": 0.8125,
|
| 1097 |
+
"r": 0.6666666666666666,
|
| 1098 |
+
"f1": 0.7323943661971831,
|
| 1099 |
+
"n": 39
|
| 1100 |
+
},
|
| 1101 |
+
"search": {
|
| 1102 |
+
"p": 0.8271889400921659,
|
| 1103 |
+
"r": 0.7638297872340426,
|
| 1104 |
+
"f1": 0.7942477876106194,
|
| 1105 |
+
"n": 470
|
| 1106 |
+
},
|
| 1107 |
+
"captcha": {
|
| 1108 |
+
"p": 0.9393939393939394,
|
| 1109 |
+
"r": 0.5636363636363636,
|
| 1110 |
+
"f1": 0.7045454545454546,
|
| 1111 |
+
"n": 165
|
| 1112 |
+
},
|
| 1113 |
+
"other": {
|
| 1114 |
+
"p": 0.855465884079237,
|
| 1115 |
+
"r": 0.8191078328064629,
|
| 1116 |
+
"f1": 0.8368921586219272,
|
| 1117 |
+
"n": 2847
|
| 1118 |
+
}
|
| 1119 |
+
},
|
| 1120 |
+
"per_form": {
|
| 1121 |
+
"login": {
|
| 1122 |
+
"p": 0.9395973154362416,
|
| 1123 |
+
"r": 0.9354120267260579,
|
| 1124 |
+
"f1": 0.9375,
|
| 1125 |
+
"n": 898
|
| 1126 |
+
},
|
| 1127 |
+
"signup": {
|
| 1128 |
+
"p": 0.5955882352941176,
|
| 1129 |
+
"r": 0.782608695652174,
|
| 1130 |
+
"f1": 0.6764091858037579,
|
| 1131 |
+
"n": 207
|
| 1132 |
+
},
|
| 1133 |
+
"password-change": {
|
| 1134 |
+
"p": 0.0,
|
| 1135 |
+
"r": 0.0,
|
| 1136 |
+
"f1": 0.0,
|
| 1137 |
+
"n": 15
|
| 1138 |
+
},
|
| 1139 |
+
"password-recovery": {
|
| 1140 |
+
"p": 0.13333333333333333,
|
| 1141 |
+
"r": 0.10526315789473684,
|
| 1142 |
+
"f1": 0.11764705882352941,
|
| 1143 |
+
"n": 19
|
| 1144 |
+
},
|
| 1145 |
+
"otp": {
|
| 1146 |
+
"p": 0.4482758620689655,
|
| 1147 |
+
"r": 0.5909090909090909,
|
| 1148 |
+
"f1": 0.5098039215686274,
|
| 1149 |
+
"n": 22
|
| 1150 |
+
},
|
| 1151 |
+
"payment": {
|
| 1152 |
+
"p": 0.1875,
|
| 1153 |
+
"r": 1.0,
|
| 1154 |
+
"f1": 0.3157894736842105,
|
| 1155 |
+
"n": 3
|
| 1156 |
+
},
|
| 1157 |
+
"address": {
|
| 1158 |
+
"p": 0.06713780918727916,
|
| 1159 |
+
"r": 0.6129032258064516,
|
| 1160 |
+
"f1": 0.12101910828025478,
|
| 1161 |
+
"n": 31
|
| 1162 |
+
},
|
| 1163 |
+
"newsletter": {
|
| 1164 |
+
"p": 0.8866666666666667,
|
| 1165 |
+
"r": 0.8286604361370716,
|
| 1166 |
+
"f1": 0.856682769726248,
|
| 1167 |
+
"n": 321
|
| 1168 |
+
},
|
| 1169 |
+
"contact": {
|
| 1170 |
+
"p": 0.9390756302521008,
|
| 1171 |
+
"r": 0.5866141732283464,
|
| 1172 |
+
"f1": 0.7221324717285944,
|
| 1173 |
+
"n": 762
|
| 1174 |
+
},
|
| 1175 |
+
"search": {
|
| 1176 |
+
"p": 0.9743589743589743,
|
| 1177 |
+
"r": 0.49906191369606,
|
| 1178 |
+
"f1": 0.6600496277915633,
|
| 1179 |
+
"n": 533
|
| 1180 |
+
},
|
| 1181 |
+
"other": {
|
| 1182 |
+
"p": 0.5199004975124378,
|
| 1183 |
+
"r": 0.7586206896551724,
|
| 1184 |
+
"f1": 0.6169741697416974,
|
| 1185 |
+
"n": 551
|
| 1186 |
+
}
|
| 1187 |
+
},
|
| 1188 |
+
"top_confusions": [
|
| 1189 |
+
[
|
| 1190 |
+
"new-password",
|
| 1191 |
+
"current-password",
|
| 1192 |
+
192
|
| 1193 |
+
],
|
| 1194 |
+
[
|
| 1195 |
+
"username",
|
| 1196 |
+
"email",
|
| 1197 |
+
155
|
| 1198 |
+
],
|
| 1199 |
+
[
|
| 1200 |
+
"name",
|
| 1201 |
+
"given-name",
|
| 1202 |
+
78
|
| 1203 |
+
],
|
| 1204 |
+
[
|
| 1205 |
+
"other",
|
| 1206 |
+
"email",
|
| 1207 |
+
71
|
| 1208 |
+
],
|
| 1209 |
+
[
|
| 1210 |
+
"other",
|
| 1211 |
+
"search",
|
| 1212 |
+
71
|
| 1213 |
+
],
|
| 1214 |
+
[
|
| 1215 |
+
"search",
|
| 1216 |
+
"other",
|
| 1217 |
+
69
|
| 1218 |
+
],
|
| 1219 |
+
[
|
| 1220 |
+
"other",
|
| 1221 |
+
"address-level2",
|
| 1222 |
+
62
|
| 1223 |
+
],
|
| 1224 |
+
[
|
| 1225 |
+
"name",
|
| 1226 |
+
"other",
|
| 1227 |
+
62
|
| 1228 |
+
],
|
| 1229 |
+
[
|
| 1230 |
+
"given-name",
|
| 1231 |
+
"name",
|
| 1232 |
+
51
|
| 1233 |
+
],
|
| 1234 |
+
[
|
| 1235 |
+
"captcha",
|
| 1236 |
+
"other",
|
| 1237 |
+
50
|
| 1238 |
+
],
|
| 1239 |
+
[
|
| 1240 |
+
"other",
|
| 1241 |
+
"username",
|
| 1242 |
+
48
|
| 1243 |
+
],
|
| 1244 |
+
[
|
| 1245 |
+
"other",
|
| 1246 |
+
"address-level1",
|
| 1247 |
+
47
|
| 1248 |
+
],
|
| 1249 |
+
[
|
| 1250 |
+
"other",
|
| 1251 |
+
"new-password",
|
| 1252 |
+
45
|
| 1253 |
+
],
|
| 1254 |
+
[
|
| 1255 |
+
"other",
|
| 1256 |
+
"name",
|
| 1257 |
+
28
|
| 1258 |
+
],
|
| 1259 |
+
[
|
| 1260 |
+
"other",
|
| 1261 |
+
"tel",
|
| 1262 |
+
28
|
| 1263 |
+
],
|
| 1264 |
+
[
|
| 1265 |
+
"tel",
|
| 1266 |
+
"other",
|
| 1267 |
+
28
|
| 1268 |
+
],
|
| 1269 |
+
[
|
| 1270 |
+
"name",
|
| 1271 |
+
"username",
|
| 1272 |
+
25
|
| 1273 |
+
],
|
| 1274 |
+
[
|
| 1275 |
+
"one-time-code",
|
| 1276 |
+
"other",
|
| 1277 |
+
24
|
| 1278 |
+
],
|
| 1279 |
+
[
|
| 1280 |
+
"email",
|
| 1281 |
+
"username",
|
| 1282 |
+
23
|
| 1283 |
+
],
|
| 1284 |
+
[
|
| 1285 |
+
"other",
|
| 1286 |
+
"organization",
|
| 1287 |
+
23
|
| 1288 |
+
],
|
| 1289 |
+
[
|
| 1290 |
+
"username",
|
| 1291 |
+
"other",
|
| 1292 |
+
23
|
| 1293 |
+
],
|
| 1294 |
+
[
|
| 1295 |
+
"family-name",
|
| 1296 |
+
"other",
|
| 1297 |
+
22
|
| 1298 |
+
],
|
| 1299 |
+
[
|
| 1300 |
+
"family-name",
|
| 1301 |
+
"name",
|
| 1302 |
+
19
|
| 1303 |
+
],
|
| 1304 |
+
[
|
| 1305 |
+
"organization",
|
| 1306 |
+
"other",
|
| 1307 |
+
17
|
| 1308 |
+
],
|
| 1309 |
+
[
|
| 1310 |
+
"street-address",
|
| 1311 |
+
"other",
|
| 1312 |
+
16
|
| 1313 |
+
]
|
| 1314 |
+
]
|
| 1315 |
+
},
|
| 1316 |
+
"autocomplete_masked": {
|
| 1317 |
+
"n_fields": 641,
|
| 1318 |
+
"acc": 0.9859594383775351,
|
| 1319 |
+
"per_class": {
|
| 1320 |
+
"address-level2": "3/4",
|
| 1321 |
+
"address-line2": "2/2",
|
| 1322 |
+
"country": "1/1",
|
| 1323 |
+
"current-password": "157/157",
|
| 1324 |
+
"email": "161/161",
|
| 1325 |
+
"family-name": "11/11",
|
| 1326 |
+
"given-name": "14/15",
|
| 1327 |
+
"name": "74/75",
|
| 1328 |
+
"new-password": "55/58",
|
| 1329 |
+
"one-time-code": "13/13",
|
| 1330 |
+
"organization": "5/5",
|
| 1331 |
+
"postal-code": "6/6",
|
| 1332 |
+
"street-address": "4/4",
|
| 1333 |
+
"tel": "26/26",
|
| 1334 |
+
"username": "100/103"
|
| 1335 |
+
},
|
| 1336 |
+
"heuristic_acc": 0.7550702028081123
|
| 1337 |
+
}
|
| 1338 |
+
}
|
eval/export_parity.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"positions": 11605,
|
| 3 |
+
"fp32_onnx_max_abs_logit_diff": 0.00010442733764648438,
|
| 4 |
+
"bf16_torch_argmax_agreement": 0.9999138302455838,
|
| 5 |
+
"fp32_onnx_argmax_agreement": 1.0,
|
| 6 |
+
"int8_onnx_argmax_agreement": 0.999138302455838
|
| 7 |
+
}
|
eval/val_metrics.json
ADDED
|
@@ -0,0 +1,367 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"epoch": 2,
|
| 3 |
+
"n_forms": 3408,
|
| 4 |
+
"n_fields": 9819,
|
| 5 |
+
"field_acc": 0.9746410021387106,
|
| 6 |
+
"field_macro_f1": 0.9661696867284804,
|
| 7 |
+
"form_acc": 0.9571596244131455,
|
| 8 |
+
"form_macro_f1": 0.9376820897318026,
|
| 9 |
+
"password_precision": 0.9898358092259578,
|
| 10 |
+
"password_recall": 0.998422712933754,
|
| 11 |
+
"password_f1": 0.994110718492344,
|
| 12 |
+
"cur_vs_new_acc": 0.9944794952681388,
|
| 13 |
+
"identifier_f1": 0.9945380063723259,
|
| 14 |
+
"per_field": {
|
| 15 |
+
"username": {
|
| 16 |
+
"p": 0.9921752738654147,
|
| 17 |
+
"r": 0.9768875192604006,
|
| 18 |
+
"f1": 0.984472049689441,
|
| 19 |
+
"n": 649
|
| 20 |
+
},
|
| 21 |
+
"email": {
|
| 22 |
+
"p": 0.9884541372674791,
|
| 23 |
+
"r": 0.9961215255332903,
|
| 24 |
+
"f1": 0.9922730199613651,
|
| 25 |
+
"n": 1547
|
| 26 |
+
},
|
| 27 |
+
"current-password": {
|
| 28 |
+
"p": 0.9814814814814815,
|
| 29 |
+
"r": 0.9977851605758582,
|
| 30 |
+
"f1": 0.9895661724327293,
|
| 31 |
+
"n": 903
|
| 32 |
+
},
|
| 33 |
+
"new-password": {
|
| 34 |
+
"p": 0.997229916897507,
|
| 35 |
+
"r": 0.9863013698630136,
|
| 36 |
+
"f1": 0.9917355371900827,
|
| 37 |
+
"n": 365
|
| 38 |
+
},
|
| 39 |
+
"one-time-code": {
|
| 40 |
+
"p": 0.9901960784313726,
|
| 41 |
+
"r": 0.9805825242718447,
|
| 42 |
+
"f1": 0.9853658536585367,
|
| 43 |
+
"n": 103
|
| 44 |
+
},
|
| 45 |
+
"tel": {
|
| 46 |
+
"p": 0.9800918836140888,
|
| 47 |
+
"r": 0.9937888198757764,
|
| 48 |
+
"f1": 0.9868928296067848,
|
| 49 |
+
"n": 644
|
| 50 |
+
},
|
| 51 |
+
"name": {
|
| 52 |
+
"p": 0.9592123769338959,
|
| 53 |
+
"r": 0.9855491329479769,
|
| 54 |
+
"f1": 0.9722024233784746,
|
| 55 |
+
"n": 692
|
| 56 |
+
},
|
| 57 |
+
"given-name": {
|
| 58 |
+
"p": 0.978494623655914,
|
| 59 |
+
"r": 0.9161073825503355,
|
| 60 |
+
"f1": 0.9462738301559791,
|
| 61 |
+
"n": 298
|
| 62 |
+
},
|
| 63 |
+
"family-name": {
|
| 64 |
+
"p": 0.9961089494163424,
|
| 65 |
+
"r": 0.9808429118773946,
|
| 66 |
+
"f1": 0.9884169884169886,
|
| 67 |
+
"n": 261
|
| 68 |
+
},
|
| 69 |
+
"organization": {
|
| 70 |
+
"p": 0.9761904761904762,
|
| 71 |
+
"r": 0.9318181818181818,
|
| 72 |
+
"f1": 0.9534883720930233,
|
| 73 |
+
"n": 132
|
| 74 |
+
},
|
| 75 |
+
"street-address": {
|
| 76 |
+
"p": 0.9369369369369369,
|
| 77 |
+
"r": 0.9122807017543859,
|
| 78 |
+
"f1": 0.9244444444444444,
|
| 79 |
+
"n": 114
|
| 80 |
+
},
|
| 81 |
+
"address-line2": {
|
| 82 |
+
"p": 1.0,
|
| 83 |
+
"r": 0.9090909090909091,
|
| 84 |
+
"f1": 0.9523809523809523,
|
| 85 |
+
"n": 22
|
| 86 |
+
},
|
| 87 |
+
"address-level2": {
|
| 88 |
+
"p": 0.8843537414965986,
|
| 89 |
+
"r": 0.8904109589041096,
|
| 90 |
+
"f1": 0.887372013651877,
|
| 91 |
+
"n": 146
|
| 92 |
+
},
|
| 93 |
+
"address-level1": {
|
| 94 |
+
"p": 0.898989898989899,
|
| 95 |
+
"r": 0.9368421052631579,
|
| 96 |
+
"f1": 0.9175257731958762,
|
| 97 |
+
"n": 95
|
| 98 |
+
},
|
| 99 |
+
"postal-code": {
|
| 100 |
+
"p": 0.9724137931034482,
|
| 101 |
+
"r": 0.9724137931034482,
|
| 102 |
+
"f1": 0.9724137931034482,
|
| 103 |
+
"n": 145
|
| 104 |
+
},
|
| 105 |
+
"country": {
|
| 106 |
+
"p": 0.8863636363636364,
|
| 107 |
+
"r": 0.9873417721518988,
|
| 108 |
+
"f1": 0.9341317365269461,
|
| 109 |
+
"n": 79
|
| 110 |
+
},
|
| 111 |
+
"cc-name": {
|
| 112 |
+
"p": 1.0,
|
| 113 |
+
"r": 0.95,
|
| 114 |
+
"f1": 0.9743589743589743,
|
| 115 |
+
"n": 20
|
| 116 |
+
},
|
| 117 |
+
"cc-number": {
|
| 118 |
+
"p": 0.9565217391304348,
|
| 119 |
+
"r": 1.0,
|
| 120 |
+
"f1": 0.9777777777777777,
|
| 121 |
+
"n": 22
|
| 122 |
+
},
|
| 123 |
+
"cc-exp": {
|
| 124 |
+
"p": 1.0,
|
| 125 |
+
"r": 1.0,
|
| 126 |
+
"f1": 1.0,
|
| 127 |
+
"n": 13
|
| 128 |
+
},
|
| 129 |
+
"cc-exp-month": {
|
| 130 |
+
"p": 1.0,
|
| 131 |
+
"r": 1.0,
|
| 132 |
+
"f1": 1.0,
|
| 133 |
+
"n": 9
|
| 134 |
+
},
|
| 135 |
+
"cc-exp-year": {
|
| 136 |
+
"p": 1.0,
|
| 137 |
+
"r": 1.0,
|
| 138 |
+
"f1": 1.0,
|
| 139 |
+
"n": 8
|
| 140 |
+
},
|
| 141 |
+
"cc-csc": {
|
| 142 |
+
"p": 0.9523809523809523,
|
| 143 |
+
"r": 1.0,
|
| 144 |
+
"f1": 0.975609756097561,
|
| 145 |
+
"n": 20
|
| 146 |
+
},
|
| 147 |
+
"bday": {
|
| 148 |
+
"p": 1.0,
|
| 149 |
+
"r": 0.8461538461538461,
|
| 150 |
+
"f1": 0.9166666666666666,
|
| 151 |
+
"n": 39
|
| 152 |
+
},
|
| 153 |
+
"search": {
|
| 154 |
+
"p": 0.9624217118997912,
|
| 155 |
+
"r": 0.9808510638297873,
|
| 156 |
+
"f1": 0.9715489989462592,
|
| 157 |
+
"n": 470
|
| 158 |
+
},
|
| 159 |
+
"captcha": {
|
| 160 |
+
"p": 0.9814814814814815,
|
| 161 |
+
"r": 0.9352941176470588,
|
| 162 |
+
"f1": 0.9578313253012049,
|
| 163 |
+
"n": 170
|
| 164 |
+
},
|
| 165 |
+
"other": {
|
| 166 |
+
"p": 0.9703912583715192,
|
| 167 |
+
"r": 0.9649491763056431,
|
| 168 |
+
"f1": 0.9676625659050966,
|
| 169 |
+
"n": 2853
|
| 170 |
+
}
|
| 171 |
+
},
|
| 172 |
+
"per_form": {
|
| 173 |
+
"login": {
|
| 174 |
+
"p": 0.9900552486187846,
|
| 175 |
+
"r": 0.9933481152993349,
|
| 176 |
+
"f1": 0.991698948533481,
|
| 177 |
+
"n": 902
|
| 178 |
+
},
|
| 179 |
+
"signup": {
|
| 180 |
+
"p": 0.9951456310679612,
|
| 181 |
+
"r": 0.9855769230769231,
|
| 182 |
+
"f1": 0.9903381642512078,
|
| 183 |
+
"n": 208
|
| 184 |
+
},
|
| 185 |
+
"password-change": {
|
| 186 |
+
"p": 1.0,
|
| 187 |
+
"r": 1.0,
|
| 188 |
+
"f1": 1.0,
|
| 189 |
+
"n": 22
|
| 190 |
+
},
|
| 191 |
+
"password-recovery": {
|
| 192 |
+
"p": 0.8461538461538461,
|
| 193 |
+
"r": 0.9565217391304348,
|
| 194 |
+
"f1": 0.8979591836734695,
|
| 195 |
+
"n": 23
|
| 196 |
+
},
|
| 197 |
+
"otp": {
|
| 198 |
+
"p": 0.8888888888888888,
|
| 199 |
+
"r": 0.8571428571428571,
|
| 200 |
+
"f1": 0.8727272727272727,
|
| 201 |
+
"n": 28
|
| 202 |
+
},
|
| 203 |
+
"payment": {
|
| 204 |
+
"p": 1.0,
|
| 205 |
+
"r": 1.0,
|
| 206 |
+
"f1": 1.0,
|
| 207 |
+
"n": 22
|
| 208 |
+
},
|
| 209 |
+
"address": {
|
| 210 |
+
"p": 0.8666666666666667,
|
| 211 |
+
"r": 0.7222222222222222,
|
| 212 |
+
"f1": 0.7878787878787877,
|
| 213 |
+
"n": 36
|
| 214 |
+
},
|
| 215 |
+
"newsletter": {
|
| 216 |
+
"p": 0.9774193548387097,
|
| 217 |
+
"r": 0.9439252336448598,
|
| 218 |
+
"f1": 0.9603803486529319,
|
| 219 |
+
"n": 321
|
| 220 |
+
},
|
| 221 |
+
"contact": {
|
| 222 |
+
"p": 0.9369482976040353,
|
| 223 |
+
"r": 0.9750656167979003,
|
| 224 |
+
"f1": 0.9556270096463023,
|
| 225 |
+
"n": 762
|
| 226 |
+
},
|
| 227 |
+
"search": {
|
| 228 |
+
"p": 0.9555555555555556,
|
| 229 |
+
"r": 0.9681050656660413,
|
| 230 |
+
"f1": 0.961789375582479,
|
| 231 |
+
"n": 533
|
| 232 |
+
},
|
| 233 |
+
"other": {
|
| 234 |
+
"p": 0.9165085388994307,
|
| 235 |
+
"r": 0.8765880217785844,
|
| 236 |
+
"f1": 0.8961038961038961,
|
| 237 |
+
"n": 551
|
| 238 |
+
}
|
| 239 |
+
},
|
| 240 |
+
"top_confusions": [
|
| 241 |
+
[
|
| 242 |
+
"given-name",
|
| 243 |
+
"name",
|
| 244 |
+
22
|
| 245 |
+
],
|
| 246 |
+
[
|
| 247 |
+
"other",
|
| 248 |
+
"address-level2",
|
| 249 |
+
16
|
| 250 |
+
],
|
| 251 |
+
[
|
| 252 |
+
"other",
|
| 253 |
+
"search",
|
| 254 |
+
16
|
| 255 |
+
],
|
| 256 |
+
[
|
| 257 |
+
"address-level2",
|
| 258 |
+
"other",
|
| 259 |
+
15
|
| 260 |
+
],
|
| 261 |
+
[
|
| 262 |
+
"other",
|
| 263 |
+
"current-password",
|
| 264 |
+
11
|
| 265 |
+
],
|
| 266 |
+
[
|
| 267 |
+
"other",
|
| 268 |
+
"address-level1",
|
| 269 |
+
10
|
| 270 |
+
],
|
| 271 |
+
[
|
| 272 |
+
"captcha",
|
| 273 |
+
"other",
|
| 274 |
+
9
|
| 275 |
+
],
|
| 276 |
+
[
|
| 277 |
+
"other",
|
| 278 |
+
"country",
|
| 279 |
+
9
|
| 280 |
+
],
|
| 281 |
+
[
|
| 282 |
+
"search",
|
| 283 |
+
"other",
|
| 284 |
+
9
|
| 285 |
+
],
|
| 286 |
+
[
|
| 287 |
+
"other",
|
| 288 |
+
"email",
|
| 289 |
+
9
|
| 290 |
+
],
|
| 291 |
+
[
|
| 292 |
+
"username",
|
| 293 |
+
"email",
|
| 294 |
+
8
|
| 295 |
+
],
|
| 296 |
+
[
|
| 297 |
+
"street-address",
|
| 298 |
+
"other",
|
| 299 |
+
7
|
| 300 |
+
],
|
| 301 |
+
[
|
| 302 |
+
"organization",
|
| 303 |
+
"other",
|
| 304 |
+
7
|
| 305 |
+
],
|
| 306 |
+
[
|
| 307 |
+
"address-level1",
|
| 308 |
+
"other",
|
| 309 |
+
6
|
| 310 |
+
],
|
| 311 |
+
[
|
| 312 |
+
"other",
|
| 313 |
+
"tel",
|
| 314 |
+
6
|
| 315 |
+
],
|
| 316 |
+
[
|
| 317 |
+
"name",
|
| 318 |
+
"given-name",
|
| 319 |
+
5
|
| 320 |
+
],
|
| 321 |
+
[
|
| 322 |
+
"username",
|
| 323 |
+
"other",
|
| 324 |
+
5
|
| 325 |
+
],
|
| 326 |
+
[
|
| 327 |
+
"other",
|
| 328 |
+
"street-address",
|
| 329 |
+
5
|
| 330 |
+
],
|
| 331 |
+
[
|
| 332 |
+
"bday",
|
| 333 |
+
"other",
|
| 334 |
+
5
|
| 335 |
+
],
|
| 336 |
+
[
|
| 337 |
+
"new-password",
|
| 338 |
+
"current-password",
|
| 339 |
+
4
|
| 340 |
+
],
|
| 341 |
+
[
|
| 342 |
+
"name",
|
| 343 |
+
"other",
|
| 344 |
+
4
|
| 345 |
+
],
|
| 346 |
+
[
|
| 347 |
+
"other",
|
| 348 |
+
"postal-code",
|
| 349 |
+
4
|
| 350 |
+
],
|
| 351 |
+
[
|
| 352 |
+
"postal-code",
|
| 353 |
+
"other",
|
| 354 |
+
4
|
| 355 |
+
],
|
| 356 |
+
[
|
| 357 |
+
"other",
|
| 358 |
+
"name",
|
| 359 |
+
3
|
| 360 |
+
],
|
| 361 |
+
[
|
| 362 |
+
"family-name",
|
| 363 |
+
"name",
|
| 364 |
+
3
|
| 365 |
+
]
|
| 366 |
+
]
|
| 367 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7c4424bdc370732e57af4c22907420b23f2cb0ea649db4a4b4ff67aa85b22695
|
| 3 |
+
size 41414010
|
onnx/model.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8c24a379a5d484cab9ab82ff1a3a420987c9fd751cf10a2143c33aacef4a2645
|
| 3 |
+
size 82955341
|
onnx/model_quantized.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fd39c40e43437a726f4e82691e36c3cbfdc36b8269dd528e1ceb2d8ee5bbcbf4
|
| 3 |
+
size 21107484
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"cls_token": "[CLS]",
|
| 3 |
+
"mask_token": "[MASK]",
|
| 4 |
+
"model_max_length": 512,
|
| 5 |
+
"pad_token": "[PAD]",
|
| 6 |
+
"sep_token": "[SEP]",
|
| 7 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 8 |
+
"unk_token": "[UNK]"
|
| 9 |
+
}
|
training/card/infer_parity_py.py
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Python side of the JS/Python inference parity test: classify saved pages with zap_infer.Zap."""
|
| 2 |
+
import json, sys, os
|
| 3 |
+
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
| 4 |
+
from zap_infer import Zap
|
| 5 |
+
model_dir, pdir, out, n = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
|
| 6 |
+
zap = Zap.from_pretrained(model_dir)
|
| 7 |
+
pages = json.load(open(os.path.join(pdir, "py.json")))[:n]
|
| 8 |
+
res = []
|
| 9 |
+
for p in pages:
|
| 10 |
+
r = zap.classify_html(open(os.path.join(pdir, p["file"])).read(), p["url"])
|
| 11 |
+
res.append({"file": p["file"], "forms": [{"form": f["form"], "fields": [x["label"] for x in f["fields"]]} for f in r]})
|
| 12 |
+
json.dump(res, open(out, "w"))
|
| 13 |
+
print("ok", sum(len(r["forms"]) for r in res))
|
training/card/zap_extract.py
ADDED
|
@@ -0,0 +1,580 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Zap form extractor (canonical Python implementation).
|
| 2 |
+
|
| 3 |
+
Finds fillable fields in an HTML page, groups them into forms, and produces
|
| 4 |
+
* the serialized text the Zap model reads (``serialize``), and
|
| 5 |
+
* a cleaned, index-annotated HTML snippet for the teacher LLM (``teacher_html``).
|
| 6 |
+
|
| 7 |
+
The JavaScript port in ``js/zap-extract.js`` implements the same rules on a live DOM.
|
| 8 |
+
Every rule here is expressible with standard DOM APIs, so keep it that way.
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
import copy
|
| 13 |
+
import hashlib
|
| 14 |
+
import re
|
| 15 |
+
import unicodedata
|
| 16 |
+
from urllib.parse import unquote, urlsplit
|
| 17 |
+
|
| 18 |
+
import lxml.html
|
| 19 |
+
from lxml import etree
|
| 20 |
+
|
| 21 |
+
SKIP_INPUT_TYPES = {"hidden", "submit", "button", "reset", "image", "file", "checkbox", "radio", "range", "color"}
|
| 22 |
+
FIELD_TAGS = {"input", "select", "textarea"}
|
| 23 |
+
NO_TEXT_TAGS = {"script", "style", "noscript", "template", "svg", "select", "textarea", "option", "head"}
|
| 24 |
+
HEADING_TAGS = {"h1", "h2", "h3", "h4", "h5", "h6", "legend"}
|
| 25 |
+
DATA_ATTRS = ["data-testid", "data-test", "data-qa", "data-cy", "formcontrolname", "data-field", "data-name", "data-type"]
|
| 26 |
+
MAX_FIELDS = 40
|
| 27 |
+
MAX_FORMS = 10
|
| 28 |
+
|
| 29 |
+
# Key tokens (special tokens of the Zap tokenizer).
|
| 30 |
+
CTX_KEYS = ["[TITLE]", "[URL]", "[FORM]", "[HEAD]", "[BTN]", "[LINK]", "[TXT]", "[HID]"]
|
| 31 |
+
FIELD_KEYS = ["[FLD]", "[TYPE]", "[NAME]", "[ID]", "[CLASS]", "[AC]", "[PH]", "[LBL]", "[ARIA]", "[TIP]",
|
| 32 |
+
"[NEAR]", "[AFTER]", "[MODE]", "[MAX]", "[REQ]", "[OPT]", "[DATA]"]
|
| 33 |
+
SPECIAL_TOKENS = CTX_KEYS + FIELD_KEYS
|
| 34 |
+
|
| 35 |
+
# Character budgets (Unicode code points).
|
| 36 |
+
CTX_LIMITS = {"title": 60, "url": 60, "form": 60, "head": 60, "btn": 60, "link": 80, "txt": 120, "hid": 40}
|
| 37 |
+
FIELD_LIMITS = {"name": 40, "id": 40, "class": 30, "ac": 30, "ph": 50, "lbl": 60, "aria": 50, "tip": 40,
|
| 38 |
+
"near": 50, "after": 30, "mode": 12, "max": 6, "opt": 60, "data": 30}
|
| 39 |
+
CTX_ORDER = [("title", "[TITLE]"), ("url", "[URL]"), ("form", "[FORM]"), ("head", "[HEAD]"), ("btn", "[BTN]"),
|
| 40 |
+
("link", "[LINK]"), ("txt", "[TXT]"), ("hid", "[HID]")]
|
| 41 |
+
FIELD_ORDER = [("type", "[TYPE]"), ("name", "[NAME]"), ("id", "[ID]"), ("class", "[CLASS]"), ("ac", "[AC]"),
|
| 42 |
+
("ph", "[PH]"), ("lbl", "[LBL]"), ("aria", "[ARIA]"), ("tip", "[TIP]"), ("near", "[NEAR]"),
|
| 43 |
+
("after", "[AFTER]"), ("mode", "[MODE]"), ("max", "[MAX]"), ("req", "[REQ]"), ("opt", "[OPT]"),
|
| 44 |
+
("data", "[DATA]")]
|
| 45 |
+
|
| 46 |
+
_WS = re.compile(r"\s+")
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def norm(s: str | None) -> str:
|
| 50 |
+
"""NFKC + collapse whitespace + strip."""
|
| 51 |
+
if not s:
|
| 52 |
+
return ""
|
| 53 |
+
return _WS.sub(" ", unicodedata.normalize("NFKC", s)).strip()
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def cut(s: str, n: int) -> str:
|
| 57 |
+
return s[:n].rstrip()
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def words(s: str | None) -> str:
|
| 61 |
+
"""Identifier -> lowercase words: 'loginForm_Email2' -> 'login form email 2'."""
|
| 62 |
+
if not s:
|
| 63 |
+
return ""
|
| 64 |
+
s = unicodedata.normalize("NFKC", s)
|
| 65 |
+
s = re.sub(r"([a-z])([A-Z])", r"\1 \2", s)
|
| 66 |
+
s = re.sub(r"([A-Z]+)([A-Z][a-z])", r"\1 \2", s)
|
| 67 |
+
s = re.sub(r"([A-Za-z])([0-9])", r"\1 \2", s)
|
| 68 |
+
s = re.sub(r"([0-9])([A-Za-z])", r"\1 \2", s)
|
| 69 |
+
out = "".join(c if unicodedata.category(c)[0] in "LNM" else " " for c in s)
|
| 70 |
+
return _WS.sub(" ", out).strip().lower()
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
def tag(el) -> str:
|
| 74 |
+
t = el.tag
|
| 75 |
+
return t.lower() if isinstance(t, str) else ""
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def attr(el, name: str) -> str:
|
| 79 |
+
v = el.get(name)
|
| 80 |
+
return v if v is not None else ""
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def is_element(node) -> bool:
|
| 84 |
+
return isinstance(node.tag, str)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def text(el) -> str:
|
| 88 |
+
"""textContent without script/style/select/textarea/option/svg/template/noscript, normalized."""
|
| 89 |
+
parts: list[str] = []
|
| 90 |
+
|
| 91 |
+
def walk(e):
|
| 92 |
+
if tag(e) in NO_TEXT_TAGS:
|
| 93 |
+
return
|
| 94 |
+
if e.text and is_element(e):
|
| 95 |
+
parts.append(e.text)
|
| 96 |
+
for c in e:
|
| 97 |
+
if is_element(c):
|
| 98 |
+
walk(c)
|
| 99 |
+
if c.tail:
|
| 100 |
+
parts.append(c.tail)
|
| 101 |
+
|
| 102 |
+
if is_element(el):
|
| 103 |
+
walk(el)
|
| 104 |
+
return norm(" ".join(parts))
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def input_type(el) -> str:
|
| 108 |
+
t = tag(el)
|
| 109 |
+
if t == "input":
|
| 110 |
+
return (attr(el, "type").strip().lower() or "text")
|
| 111 |
+
return t
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def hidden_attr(el) -> bool:
|
| 115 |
+
if el.get("hidden") is not None or attr(el, "aria-hidden").strip().lower() == "true":
|
| 116 |
+
return True
|
| 117 |
+
style = attr(el, "style").replace(" ", "").lower()
|
| 118 |
+
return "display:none" in style or "visibility:hidden" in style
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def is_field(el) -> bool:
|
| 122 |
+
t = tag(el)
|
| 123 |
+
if t not in FIELD_TAGS:
|
| 124 |
+
return False
|
| 125 |
+
if t == "input" and input_type(el) in SKIP_INPUT_TYPES:
|
| 126 |
+
return False
|
| 127 |
+
return el.get("disabled") is None
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def visible_within(el, root) -> bool:
|
| 131 |
+
node = el
|
| 132 |
+
while node is not None:
|
| 133 |
+
if hidden_attr(node):
|
| 134 |
+
return False
|
| 135 |
+
if node is root:
|
| 136 |
+
return True
|
| 137 |
+
node = node.getparent()
|
| 138 |
+
return True
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
def contains_field(el) -> bool:
|
| 142 |
+
if tag(el) in FIELD_TAGS:
|
| 143 |
+
return True
|
| 144 |
+
return any(tag(d) in FIELD_TAGS for d in el.iter() if d is not el)
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
# ---------------------------------------------------------------- sibling walks (DOM node order)
|
| 148 |
+
|
| 149 |
+
def prev_nodes(el):
|
| 150 |
+
"""Previous siblings of el in DOM order, nearest first: ('t', str) or ('e', element)."""
|
| 151 |
+
parent = el.getparent()
|
| 152 |
+
if parent is None:
|
| 153 |
+
return
|
| 154 |
+
kids = list(parent)
|
| 155 |
+
i = kids.index(el)
|
| 156 |
+
for j in range(i - 1, -1, -1):
|
| 157 |
+
k = kids[j]
|
| 158 |
+
if k.tail:
|
| 159 |
+
yield ("t", k.tail)
|
| 160 |
+
if is_element(k):
|
| 161 |
+
yield ("e", k)
|
| 162 |
+
if parent.text:
|
| 163 |
+
yield ("t", parent.text)
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
def next_nodes(el):
|
| 167 |
+
parent = el.getparent()
|
| 168 |
+
if parent is None:
|
| 169 |
+
return
|
| 170 |
+
if el.tail:
|
| 171 |
+
yield ("t", el.tail)
|
| 172 |
+
kids = list(parent)
|
| 173 |
+
i = kids.index(el)
|
| 174 |
+
for k in kids[i + 1:]:
|
| 175 |
+
if is_element(k):
|
| 176 |
+
yield ("e", k)
|
| 177 |
+
if k.tail:
|
| 178 |
+
yield ("t", k.tail)
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def near_text(el, root) -> str:
|
| 182 |
+
node = el
|
| 183 |
+
for _level in range(3):
|
| 184 |
+
seen = 0
|
| 185 |
+
for kind, v in prev_nodes(node):
|
| 186 |
+
if kind == "t":
|
| 187 |
+
t = norm(v)
|
| 188 |
+
else:
|
| 189 |
+
if contains_field(v):
|
| 190 |
+
return ""
|
| 191 |
+
t = text(v)
|
| 192 |
+
if t:
|
| 193 |
+
return t
|
| 194 |
+
if kind == "e":
|
| 195 |
+
seen += 1
|
| 196 |
+
if seen >= 3:
|
| 197 |
+
break
|
| 198 |
+
if node is root:
|
| 199 |
+
break
|
| 200 |
+
node = node.getparent()
|
| 201 |
+
if node is None or node is root:
|
| 202 |
+
break
|
| 203 |
+
return ""
|
| 204 |
+
|
| 205 |
+
|
| 206 |
+
def after_text(el) -> str:
|
| 207 |
+
seen = 0
|
| 208 |
+
for kind, v in next_nodes(el):
|
| 209 |
+
if kind == "t":
|
| 210 |
+
t = norm(v)
|
| 211 |
+
else:
|
| 212 |
+
if contains_field(v):
|
| 213 |
+
return ""
|
| 214 |
+
t = text(v)
|
| 215 |
+
if t:
|
| 216 |
+
return t
|
| 217 |
+
if kind == "e":
|
| 218 |
+
seen += 1
|
| 219 |
+
if seen >= 2:
|
| 220 |
+
break
|
| 221 |
+
return ""
|
| 222 |
+
|
| 223 |
+
|
| 224 |
+
# ---------------------------------------------------------------- labels
|
| 225 |
+
|
| 226 |
+
class Page:
|
| 227 |
+
def __init__(self, doc, url: str):
|
| 228 |
+
self.doc = doc
|
| 229 |
+
self.url = url
|
| 230 |
+
self.by_id: dict[str, object] = {}
|
| 231 |
+
self.label_for: dict[str, list] = {}
|
| 232 |
+
for e in doc.iter():
|
| 233 |
+
if not is_element(e):
|
| 234 |
+
continue
|
| 235 |
+
i = e.get("id")
|
| 236 |
+
if i and i not in self.by_id:
|
| 237 |
+
self.by_id[i] = e
|
| 238 |
+
if tag(e) == "label" and e.get("for"):
|
| 239 |
+
self.label_for.setdefault(e.get("for"), []).append(e)
|
| 240 |
+
t = doc.find(".//title")
|
| 241 |
+
self.title = norm(t.text_content()) if t is not None else ""
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def label_text(el, page: Page) -> str:
|
| 245 |
+
out: list[str] = []
|
| 246 |
+
for ref in attr(el, "aria-labelledby").split():
|
| 247 |
+
e = page.by_id.get(ref)
|
| 248 |
+
if e is not None:
|
| 249 |
+
out.append(text(e))
|
| 250 |
+
i = el.get("id")
|
| 251 |
+
if i:
|
| 252 |
+
for lab in page.label_for.get(i, []):
|
| 253 |
+
out.append(text(lab))
|
| 254 |
+
node = el.getparent()
|
| 255 |
+
while node is not None:
|
| 256 |
+
if tag(node) == "label":
|
| 257 |
+
out.append(text(node))
|
| 258 |
+
break
|
| 259 |
+
node = node.getparent()
|
| 260 |
+
seen, uniq = set(), []
|
| 261 |
+
for t in out:
|
| 262 |
+
if t and t not in seen:
|
| 263 |
+
seen.add(t)
|
| 264 |
+
uniq.append(t)
|
| 265 |
+
return " | ".join(uniq)
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
# ---------------------------------------------------------------- form context
|
| 269 |
+
|
| 270 |
+
def is_buttonish(e) -> bool:
|
| 271 |
+
t = tag(e)
|
| 272 |
+
if t == "button":
|
| 273 |
+
return True
|
| 274 |
+
if t == "input" and input_type(e) in ("submit", "button", "image"):
|
| 275 |
+
return True
|
| 276 |
+
return attr(e, "role").lower() == "button"
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
def heading(root) -> str:
|
| 280 |
+
for e in root.iter():
|
| 281 |
+
if is_element(e) and (tag(e) in HEADING_TAGS or attr(e, "role").lower() == "heading"):
|
| 282 |
+
t = text(e)
|
| 283 |
+
if t:
|
| 284 |
+
return t
|
| 285 |
+
node = root
|
| 286 |
+
for _level in range(4):
|
| 287 |
+
n = 0
|
| 288 |
+
for kind, sib in prev_nodes(node):
|
| 289 |
+
if kind != "e":
|
| 290 |
+
continue
|
| 291 |
+
n += 1
|
| 292 |
+
if n > 5:
|
| 293 |
+
break
|
| 294 |
+
if tag(sib) in HEADING_TAGS or attr(sib, "role").lower() == "heading":
|
| 295 |
+
t = text(sib)
|
| 296 |
+
if t:
|
| 297 |
+
return t
|
| 298 |
+
hs = [d for d in sib.iter() if is_element(d) and d is not sib and
|
| 299 |
+
(tag(d) in HEADING_TAGS or attr(d, "role").lower() == "heading")]
|
| 300 |
+
for d in reversed(hs):
|
| 301 |
+
t = text(d)
|
| 302 |
+
if t:
|
| 303 |
+
return t
|
| 304 |
+
node = node.getparent()
|
| 305 |
+
if node is None or tag(node) in ("body", "html"):
|
| 306 |
+
break
|
| 307 |
+
return ""
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
def button_texts(root) -> str:
|
| 311 |
+
out: list[str] = []
|
| 312 |
+
for e in root.iter():
|
| 313 |
+
if not is_element(e) or not is_buttonish(e):
|
| 314 |
+
continue
|
| 315 |
+
t = text(e) or norm(attr(e, "value")) or norm(attr(e, "aria-label")) or norm(attr(e, "alt")) or norm(attr(e, "title"))
|
| 316 |
+
if t and t not in out:
|
| 317 |
+
out.append(cut(t, 30))
|
| 318 |
+
if len(out) >= 3:
|
| 319 |
+
break
|
| 320 |
+
return " | ".join(out)
|
| 321 |
+
|
| 322 |
+
|
| 323 |
+
def link_texts(root) -> str:
|
| 324 |
+
out: list[str] = []
|
| 325 |
+
for e in root.iter():
|
| 326 |
+
if not is_element(e) or tag(e) != "a":
|
| 327 |
+
continue
|
| 328 |
+
t = text(e) or norm(attr(e, "aria-label")) or norm(attr(e, "title"))
|
| 329 |
+
if t and t not in out:
|
| 330 |
+
out.append(cut(t, 30))
|
| 331 |
+
if len(out) >= 4:
|
| 332 |
+
break
|
| 333 |
+
return " | ".join(out)
|
| 334 |
+
|
| 335 |
+
|
| 336 |
+
def hidden_names(root) -> str:
|
| 337 |
+
out: list[str] = []
|
| 338 |
+
for e in root.iter():
|
| 339 |
+
if is_element(e) and tag(e) == "input" and input_type(e) == "hidden":
|
| 340 |
+
w = words(attr(e, "name"))
|
| 341 |
+
if w and w not in out:
|
| 342 |
+
out.append(w)
|
| 343 |
+
if len(out) >= 4:
|
| 344 |
+
break
|
| 345 |
+
return " ".join(out)
|
| 346 |
+
|
| 347 |
+
|
| 348 |
+
def url_words(url: str) -> str:
|
| 349 |
+
try:
|
| 350 |
+
u = urlsplit(url)
|
| 351 |
+
except ValueError:
|
| 352 |
+
return ""
|
| 353 |
+
host = (u.hostname or "").split(".")
|
| 354 |
+
lead = host[0] if len(host) >= 3 and host[0] != "www" else ""
|
| 355 |
+
return words(lead + " " + unquote(u.path))
|
| 356 |
+
|
| 357 |
+
|
| 358 |
+
def form_words(root) -> str:
|
| 359 |
+
parts = [attr(root, "id"), attr(root, "name"), attr(root, "class")]
|
| 360 |
+
if tag(root) == "form":
|
| 361 |
+
try:
|
| 362 |
+
parts.append(unquote(urlsplit(attr(root, "action")).path))
|
| 363 |
+
except ValueError:
|
| 364 |
+
pass
|
| 365 |
+
else:
|
| 366 |
+
parts.append(attr(root, "role"))
|
| 367 |
+
return (words(" ".join(parts)) + " " + norm(attr(root, "aria-label")).lower()).strip()
|
| 368 |
+
|
| 369 |
+
|
| 370 |
+
# ---------------------------------------------------------------- fields
|
| 371 |
+
|
| 372 |
+
def field_features(el, root, page: Page) -> dict:
|
| 373 |
+
t = tag(el)
|
| 374 |
+
f: dict = {"type": t if t != "input" else "input " + input_type(el)}
|
| 375 |
+
f["name"] = words(attr(el, "name"))
|
| 376 |
+
f["id"] = words(attr(el, "id"))
|
| 377 |
+
f["class"] = words(attr(el, "class"))
|
| 378 |
+
f["ac"] = norm(attr(el, "autocomplete")).lower()
|
| 379 |
+
f["ph"] = norm(attr(el, "placeholder"))
|
| 380 |
+
f["lbl"] = label_text(el, page)
|
| 381 |
+
f["aria"] = norm(attr(el, "aria-label"))
|
| 382 |
+
f["tip"] = norm(attr(el, "title"))
|
| 383 |
+
near = near_text(el, root)
|
| 384 |
+
f["near"] = "" if near in (f["lbl"], f["ph"], f["aria"]) else near
|
| 385 |
+
f["after"] = after_text(el)
|
| 386 |
+
f["mode"] = norm(attr(el, "inputmode")).lower()
|
| 387 |
+
f["max"] = norm(attr(el, "maxlength"))
|
| 388 |
+
f["req"] = el.get("required") is not None or attr(el, "aria-required").lower() == "true"
|
| 389 |
+
if t == "select":
|
| 390 |
+
opts = [text(o) or norm(o.text_content()) for o in el.iter() if is_element(o) and tag(o) == "option"]
|
| 391 |
+
opts = [o for o in opts if o]
|
| 392 |
+
f["opt"] = (str(len(opts)) + ": " + " | ".join(opts[:6])) if opts else ""
|
| 393 |
+
else:
|
| 394 |
+
f["opt"] = ""
|
| 395 |
+
data = [norm(attr(el, a)) for a in DATA_ATTRS if attr(el, a).strip()]
|
| 396 |
+
f["data"] = words(" ".join(data[:2]))
|
| 397 |
+
for k, n in FIELD_LIMITS.items():
|
| 398 |
+
if isinstance(f.get(k), str):
|
| 399 |
+
f[k] = cut(f[k], n)
|
| 400 |
+
return f
|
| 401 |
+
|
| 402 |
+
|
| 403 |
+
def form_context(root, page: Page) -> dict:
|
| 404 |
+
c = {"title": page.title, "url": url_words(page.url), "form": form_words(root), "head": heading(root),
|
| 405 |
+
"btn": button_texts(root), "link": link_texts(root), "txt": text(root), "hid": hidden_names(root)}
|
| 406 |
+
return {k: cut(v, CTX_LIMITS[k]) for k, v in c.items()}
|
| 407 |
+
|
| 408 |
+
|
| 409 |
+
def group_root(el, body, in_form):
|
| 410 |
+
node = el.getparent()
|
| 411 |
+
for _level in range(6):
|
| 412 |
+
if node is None or node is body or tag(node) in ("body", "html"):
|
| 413 |
+
break
|
| 414 |
+
if any(is_element(d) and is_buttonish(d) and id(d) not in in_form for d in node.iter()):
|
| 415 |
+
return node
|
| 416 |
+
node = node.getparent()
|
| 417 |
+
node = el
|
| 418 |
+
for _level in range(3):
|
| 419 |
+
p = node.getparent()
|
| 420 |
+
if p is None or p is body or tag(p) in ("body", "html"):
|
| 421 |
+
break
|
| 422 |
+
node = p
|
| 423 |
+
return node
|
| 424 |
+
|
| 425 |
+
|
| 426 |
+
def find_forms(doc) -> list[tuple[object, list]]:
|
| 427 |
+
"""[(root, [field elements])], document order, <form>s first-seen order."""
|
| 428 |
+
body = doc.find(".//body")
|
| 429 |
+
if body is None:
|
| 430 |
+
body = doc
|
| 431 |
+
groups: dict[int, tuple[object, list]] = {}
|
| 432 |
+
order: list[int] = []
|
| 433 |
+
# keep the proxies alive: lxml recycles element proxies, so id() is only stable while referenced
|
| 434 |
+
form_nodes = [d for f in body.iter("form") for d in f.iter()]
|
| 435 |
+
in_form = {id(d) for d in form_nodes}
|
| 436 |
+
for el in body.iter():
|
| 437 |
+
if not is_element(el) or not is_field(el):
|
| 438 |
+
continue
|
| 439 |
+
root = None
|
| 440 |
+
node = el.getparent()
|
| 441 |
+
while node is not None:
|
| 442 |
+
if tag(node) == "form":
|
| 443 |
+
root = node
|
| 444 |
+
break
|
| 445 |
+
node = node.getparent()
|
| 446 |
+
if root is None:
|
| 447 |
+
root = group_root(el, body, in_form)
|
| 448 |
+
if not visible_within(el, root):
|
| 449 |
+
continue
|
| 450 |
+
key = id(root)
|
| 451 |
+
if key not in groups:
|
| 452 |
+
groups[key] = (root, [])
|
| 453 |
+
order.append(key)
|
| 454 |
+
if len(groups[key][1]) < MAX_FIELDS:
|
| 455 |
+
groups[key][1].append(el)
|
| 456 |
+
return [groups[k] for k in order][:MAX_FORMS]
|
| 457 |
+
|
| 458 |
+
|
| 459 |
+
# ---------------------------------------------------------------- serialization
|
| 460 |
+
|
| 461 |
+
def serialize_ctx(ctx: dict) -> str:
|
| 462 |
+
return " ".join(f"{key} {ctx[k]}" for k, key in CTX_ORDER if ctx.get(k))
|
| 463 |
+
|
| 464 |
+
|
| 465 |
+
def serialize_field(f: dict) -> str:
|
| 466 |
+
out = ["[FLD]"]
|
| 467 |
+
for k, key in FIELD_ORDER:
|
| 468 |
+
v = f.get(k)
|
| 469 |
+
if k == "req":
|
| 470 |
+
if v:
|
| 471 |
+
out.append("[REQ]")
|
| 472 |
+
elif v:
|
| 473 |
+
out.append(f"{key} {v}")
|
| 474 |
+
return " ".join(out)
|
| 475 |
+
|
| 476 |
+
|
| 477 |
+
def serialize(ctx: dict, fields: list[dict]) -> str:
|
| 478 |
+
return " ".join([serialize_ctx(ctx)] + [serialize_field(f) for f in fields])
|
| 479 |
+
|
| 480 |
+
|
| 481 |
+
# ---------------------------------------------------------------- teacher view
|
| 482 |
+
|
| 483 |
+
KEEP_ATTRS = {"id", "name", "type", "class", "placeholder", "autocomplete", "title", "for", "role", "action", "method",
|
| 484 |
+
"inputmode", "maxlength", "minlength", "required", "value", "href", "data-zap", "pattern", "alt", "label",
|
| 485 |
+
"multiple", "readonly"} | set(DATA_ATTRS)
|
| 486 |
+
DROP_TAGS = {"script", "style", "noscript", "template", "svg", "iframe", "img", "picture", "video", "audio", "canvas",
|
| 487 |
+
"link", "meta", "object", "embed", "path", "source", "use", "head"}
|
| 488 |
+
|
| 489 |
+
|
| 490 |
+
def teacher_html(root, field_els, max_chars=9000) -> str | None:
|
| 491 |
+
for i, el in enumerate(field_els, 1):
|
| 492 |
+
el.set("data-zap", str(i))
|
| 493 |
+
try:
|
| 494 |
+
r = copy.deepcopy(root)
|
| 495 |
+
finally:
|
| 496 |
+
for el in field_els:
|
| 497 |
+
el.attrib.pop("data-zap", None)
|
| 498 |
+
r.tail = None
|
| 499 |
+
for e in list(r.iter()):
|
| 500 |
+
if not is_element(e): # comments / PIs
|
| 501 |
+
p = e.getparent()
|
| 502 |
+
if p is not None:
|
| 503 |
+
e.drop_tree() if hasattr(e, "drop_tree") else p.remove(e)
|
| 504 |
+
continue
|
| 505 |
+
if tag(e) in DROP_TAGS and e is not r:
|
| 506 |
+
e.drop_tree()
|
| 507 |
+
for e in r.iter():
|
| 508 |
+
if not is_element(e):
|
| 509 |
+
continue
|
| 510 |
+
for a in list(e.attrib):
|
| 511 |
+
if a.startswith("aria-") or a in KEEP_ATTRS:
|
| 512 |
+
v = e.attrib[a]
|
| 513 |
+
if tag(e) == "input" and a == "value" and input_type(e) not in ("submit", "button", "hidden"):
|
| 514 |
+
del e.attrib[a]
|
| 515 |
+
continue
|
| 516 |
+
lim = 60 if a == "class" else 80
|
| 517 |
+
if a == "href":
|
| 518 |
+
try:
|
| 519 |
+
v = urlsplit(v).path
|
| 520 |
+
except ValueError:
|
| 521 |
+
v = ""
|
| 522 |
+
if len(v) > lim:
|
| 523 |
+
e.attrib[a] = v[:lim]
|
| 524 |
+
elif a == "href":
|
| 525 |
+
e.attrib[a] = v
|
| 526 |
+
else:
|
| 527 |
+
del e.attrib[a]
|
| 528 |
+
if tag(e) == "select":
|
| 529 |
+
opts = [o for o in e if is_element(o) and tag(o) in ("option", "optgroup")]
|
| 530 |
+
if len(opts) > 6:
|
| 531 |
+
for o in opts[6:]:
|
| 532 |
+
e.remove(o)
|
| 533 |
+
e.append(etree.Comment(f" +{len(opts) - 6} more options "))
|
| 534 |
+
if e.text:
|
| 535 |
+
e.text = cut(norm(e.text), 200) + (" " if e.text[-1:].isspace() else "")
|
| 536 |
+
if e.tail:
|
| 537 |
+
e.tail = " " + cut(norm(e.tail), 200) + " "
|
| 538 |
+
html = lxml.html.tostring(r, encoding="unicode", method="html")
|
| 539 |
+
html = re.sub(r">\s+<", "> <", html)
|
| 540 |
+
html = _WS.sub(" ", html).strip()
|
| 541 |
+
return html if len(html) <= max_chars else None
|
| 542 |
+
|
| 543 |
+
|
| 544 |
+
def signature(ctx: dict, fields: list[dict]) -> str:
|
| 545 |
+
key = "|".join([ctx.get("form", ""), ctx.get("btn", "")] +
|
| 546 |
+
[f"{f['type']}/{f['name']}/{f['id']}/{f['ac']}/{f['ph']}/{f['lbl']}" for f in fields])
|
| 547 |
+
return hashlib.sha1(key.encode()).hexdigest()[:16]
|
| 548 |
+
|
| 549 |
+
|
| 550 |
+
def html5_normalize(html: str) -> str:
|
| 551 |
+
"""Re-serialize through an HTML5-spec parser (lexbor) so the tree matches what browsers build
|
| 552 |
+
(e.g. <form> inside <table>); lxml's libxml2 parser nests malformed markup differently."""
|
| 553 |
+
try:
|
| 554 |
+
from selectolax.lexbor import LexborHTMLParser
|
| 555 |
+
except ImportError:
|
| 556 |
+
return html
|
| 557 |
+
try:
|
| 558 |
+
return LexborHTMLParser(html).html or html
|
| 559 |
+
except Exception: # noqa: BLE001
|
| 560 |
+
return html
|
| 561 |
+
|
| 562 |
+
|
| 563 |
+
def extract_page(html: str | bytes, url: str, want_teacher: bool = True, html5: bool = True) -> list[dict]:
|
| 564 |
+
if html5 and isinstance(html, str):
|
| 565 |
+
html = html5_normalize(html)
|
| 566 |
+
try:
|
| 567 |
+
doc = lxml.html.document_fromstring(html)
|
| 568 |
+
except (etree.ParserError, ValueError):
|
| 569 |
+
return []
|
| 570 |
+
page = Page(doc, url)
|
| 571 |
+
out = []
|
| 572 |
+
for root, els in find_forms(doc):
|
| 573 |
+
fields = [field_features(el, root, page) for el in els]
|
| 574 |
+
ctx = form_context(root, page)
|
| 575 |
+
rec = {"kind": "form" if tag(root) == "form" else "group", "ctx": ctx, "fields": fields,
|
| 576 |
+
"sig": signature(ctx, fields)}
|
| 577 |
+
if want_teacher:
|
| 578 |
+
rec["html"] = teacher_html(root, els)
|
| 579 |
+
out.append(rec)
|
| 580 |
+
return out
|
training/card/zap_infer.py
ADDED
|
@@ -0,0 +1,99 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Zap inference helper: classify every fillable field of a web page (and each form's purpose).
|
| 2 |
+
|
| 3 |
+
from zap_infer import Zap
|
| 4 |
+
zap = Zap.from_pretrained("ProCreations/Zap") # or a local directory
|
| 5 |
+
for form in zap.classify_html(html, url):
|
| 6 |
+
print(form["form"], [(f["label"], round(f["score"], 3)) for f in form["fields"]])
|
| 7 |
+
|
| 8 |
+
In a browser extension, run zap-extract.js on the live DOM and feed each form's `ctx`/`fields` to the model
|
| 9 |
+
through the same windowing (see README); `Zap.classify_forms` accepts the same dicts.
|
| 10 |
+
"""
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import torch
|
| 14 |
+
|
| 15 |
+
from zap_extract import extract_page, serialize_ctx, serialize_field
|
| 16 |
+
|
| 17 |
+
MAX_LEN = 512
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def windows(tok, ctx, fields, max_len=MAX_LEN):
|
| 21 |
+
"""[CLS] ctx + as many whole fields as fit + [SEP]; returns [(input_ids, field_positions, field_indices)]."""
|
| 22 |
+
ctx_ids = tok(serialize_ctx(ctx), add_special_tokens=False)["input_ids"][:160]
|
| 23 |
+
f_ids = [tok(serialize_field(f), add_special_tokens=False)["input_ids"][:120] for f in fields]
|
| 24 |
+
out, cur, cur_idx, used = [], [], [], 0
|
| 25 |
+
budget = max_len - 2 - len(ctx_ids)
|
| 26 |
+
for i, ids in enumerate(f_ids):
|
| 27 |
+
if cur and used + len(ids) > budget:
|
| 28 |
+
out.append((cur, cur_idx))
|
| 29 |
+
cur, cur_idx, used = [], [], 0
|
| 30 |
+
cur.append(ids)
|
| 31 |
+
cur_idx.append(i)
|
| 32 |
+
used += len(ids)
|
| 33 |
+
if cur or not out:
|
| 34 |
+
out.append((cur, cur_idx))
|
| 35 |
+
res = []
|
| 36 |
+
for chunk, idx in out:
|
| 37 |
+
ids, pos = [tok.cls_token_id] + ctx_ids, []
|
| 38 |
+
for f in chunk:
|
| 39 |
+
pos.append(len(ids))
|
| 40 |
+
ids += f
|
| 41 |
+
ids.append(tok.sep_token_id)
|
| 42 |
+
res.append((ids, pos, idx))
|
| 43 |
+
return res
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
class Zap:
|
| 47 |
+
def __init__(self, model, tok, device="cpu"):
|
| 48 |
+
self.model, self.tok, self.device = model.eval().to(device), tok, device
|
| 49 |
+
labels = [model.config.id2label[i] for i in range(model.config.num_labels)]
|
| 50 |
+
self.form_ids = [i for i, l in enumerate(labels) if l.startswith("form:")]
|
| 51 |
+
self.field_ids = [i for i, l in enumerate(labels) if l.startswith("field:")]
|
| 52 |
+
self.form_names = [labels[i][5:] for i in self.form_ids]
|
| 53 |
+
self.field_names = [labels[i][6:] for i in self.field_ids]
|
| 54 |
+
|
| 55 |
+
@classmethod
|
| 56 |
+
def from_pretrained(cls, path="ProCreations/Zap", device="cpu", dtype=torch.float32):
|
| 57 |
+
from transformers import AutoModelForTokenClassification, AutoTokenizer
|
| 58 |
+
model = AutoModelForTokenClassification.from_pretrained(path, dtype=dtype)
|
| 59 |
+
return cls(model, AutoTokenizer.from_pretrained(path), device)
|
| 60 |
+
|
| 61 |
+
@torch.no_grad()
|
| 62 |
+
def classify_forms(self, forms):
|
| 63 |
+
"""forms: [{"ctx": {...}, "fields": [{...}, ...]}] as produced by zap_extract / zap-extract.js."""
|
| 64 |
+
results = []
|
| 65 |
+
for form in forms:
|
| 66 |
+
wins = windows(self.tok, form["ctx"], form["fields"])
|
| 67 |
+
n = max(len(w[0]) for w in wins)
|
| 68 |
+
ids = torch.tensor([w[0] + [self.tok.pad_token_id] * (n - len(w[0])) for w in wins], device=self.device)
|
| 69 |
+
logits = self.model(input_ids=ids, attention_mask=(ids != self.tok.pad_token_id).long()).logits.float()
|
| 70 |
+
form_p = torch.softmax(logits[:, 0, self.form_ids].mean(0), -1)
|
| 71 |
+
fields = [None] * len(form["fields"])
|
| 72 |
+
for b, (_, pos, idx) in enumerate(wins):
|
| 73 |
+
for p, k in zip(pos, idx):
|
| 74 |
+
pr = torch.softmax(logits[b, p, self.field_ids], -1)
|
| 75 |
+
top = int(pr.argmax())
|
| 76 |
+
fields[k] = {"label": self.field_names[top], "score": float(pr[top]),
|
| 77 |
+
"probs": {self.field_names[j]: float(pr[j]) for j in pr.argsort(descending=True)[:3].tolist()}}
|
| 78 |
+
top = int(form_p.argmax())
|
| 79 |
+
results.append({"form": self.form_names[top], "form_score": float(form_p[top]), "fields": fields})
|
| 80 |
+
return results
|
| 81 |
+
|
| 82 |
+
def classify_html(self, html, url):
|
| 83 |
+
forms = extract_page(html, url, want_teacher=False)
|
| 84 |
+
out = self.classify_forms(forms)
|
| 85 |
+
for f, r in zip(forms, out):
|
| 86 |
+
r["kind"] = f["kind"]
|
| 87 |
+
for fld, res in zip(f["fields"], r["fields"]):
|
| 88 |
+
res["name"], res["id"], res["type"] = fld["name"], fld["id"], fld["type"]
|
| 89 |
+
return out
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
if __name__ == "__main__":
|
| 93 |
+
import json
|
| 94 |
+
import sys
|
| 95 |
+
import urllib.request
|
| 96 |
+
zap = Zap.from_pretrained(sys.argv[2] if len(sys.argv) > 2 else "ProCreations/Zap")
|
| 97 |
+
url = sys.argv[1]
|
| 98 |
+
html = urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})).read().decode("utf-8", "replace")
|
| 99 |
+
print(json.dumps(zap.classify_html(html, url), indent=1, ensure_ascii=False))
|
training/harvest/cc_candidates.py
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Stage A: sample CDX index blocks of a Common Crawl crawl and keep form-likely URLs.
|
| 2 |
+
|
| 3 |
+
Usage: python cc_candidates.py OUT.jsonl NUM_BLOCKS [SEED]
|
| 4 |
+
"""
|
| 5 |
+
import asyncio
|
| 6 |
+
import gzip
|
| 7 |
+
import json
|
| 8 |
+
import os
|
| 9 |
+
import random
|
| 10 |
+
import re
|
| 11 |
+
import sys
|
| 12 |
+
import time
|
| 13 |
+
from collections import Counter, defaultdict
|
| 14 |
+
|
| 15 |
+
import aiohttp
|
| 16 |
+
|
| 17 |
+
CRAWL = "CC-MAIN-2026-39"
|
| 18 |
+
BASE = "https://data.commoncrawl.org/"
|
| 19 |
+
IDX = f"{BASE}cc-index/collections/{CRAWL}/indexes/"
|
| 20 |
+
|
| 21 |
+
KW = re.compile(
|
| 22 |
+
r"log-?_?in|sign-?_?in|signon|logon|auth|sso|account|register|registration|sign-?_?up|join|create-?_?account|"
|
| 23 |
+
r"passw|passwd|forgot|reset|recover|checkout|cart|basket|billing|payment|subscribe|newsletter|contact|profile|"
|
| 24 |
+
r"member|wp-login|/user|customer|my-?account|anmelden|registrier|konto|passwort|connexion|inscription|compte|"
|
| 25 |
+
r"mot-de-passe|iniciar|sesion|registro|cuenta|contrase|entrar|cadastro|conta|senha|accedi|registrati|accesso|"
|
| 26 |
+
r"inloggen|registreren|wachtwoord|logowanie|rejestracja|zaloguj|haslo|giris|kayit|uye|sifre|dang-nhap|dang-ky|"
|
| 27 |
+
r"masuk|daftar|vhod|voiti|registraci|parol|2fa|verify|verification|otp|mfa|shipping|address|booking|reserv|"
|
| 28 |
+
r"quote|apply|enquiry|inquiry|order|signin|login|kirjaudu|logga|tilmeld|registr|prihlas|belepes|autentic|"
|
| 29 |
+
r"%e7%99%bb|%e6%b3%a8%e5%86%8c|%e3%83%ad%e3%82%b0|%d0%b2%d1%85%d0%be%d0%b4|%eb%a1%9c%ea%b7%b8")
|
| 30 |
+
HTML_MIMES = ("text/html", "application/xhtml+xml")
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
async def fetch(session, sem, shard, off, length, tries=6):
|
| 34 |
+
url = IDX + shard
|
| 35 |
+
for t in range(tries):
|
| 36 |
+
try:
|
| 37 |
+
async with sem:
|
| 38 |
+
async with session.get(url, headers={"Range": f"bytes={off}-{off + length - 1}"},
|
| 39 |
+
timeout=aiohttp.ClientTimeout(total=120)) as r:
|
| 40 |
+
if r.status in (200, 206):
|
| 41 |
+
return await r.read()
|
| 42 |
+
if r.status in (429, 503, 500, 502, 504):
|
| 43 |
+
await asyncio.sleep(2 ** t + random.random())
|
| 44 |
+
continue
|
| 45 |
+
return None
|
| 46 |
+
except (aiohttp.ClientError, asyncio.TimeoutError):
|
| 47 |
+
await asyncio.sleep(2 ** t + random.random())
|
| 48 |
+
return None
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
async def main(out_path, n_blocks, seed):
|
| 52 |
+
rng = random.Random(seed)
|
| 53 |
+
async with aiohttp.ClientSession() as s:
|
| 54 |
+
cache = os.path.join(os.path.dirname(os.path.abspath(out_path)), "cluster.idx")
|
| 55 |
+
for attempt in range(8):
|
| 56 |
+
if os.path.exists(cache):
|
| 57 |
+
break
|
| 58 |
+
async with s.get(IDX + "cluster.idx") as r:
|
| 59 |
+
if r.status == 200:
|
| 60 |
+
body = await r.read()
|
| 61 |
+
if len(body) > 10_000_000:
|
| 62 |
+
open(cache, "wb").write(body)
|
| 63 |
+
break
|
| 64 |
+
print(f"cluster.idx status {r.status}, retrying", flush=True)
|
| 65 |
+
await asyncio.sleep(5 * (attempt + 1))
|
| 66 |
+
idx = open(cache).read().splitlines()
|
| 67 |
+
blocks = []
|
| 68 |
+
for line in idx:
|
| 69 |
+
parts = line.split("\t")
|
| 70 |
+
if len(parts) >= 4:
|
| 71 |
+
blocks.append((parts[1], int(parts[2]), int(parts[3])))
|
| 72 |
+
print(f"{len(blocks)} blocks in index", flush=True)
|
| 73 |
+
pick = rng.sample(blocks, n_blocks)
|
| 74 |
+
sem = asyncio.Semaphore(48)
|
| 75 |
+
per_host_kw, per_host_rand = defaultdict(int), defaultdict(int)
|
| 76 |
+
stats = Counter()
|
| 77 |
+
t0 = time.time()
|
| 78 |
+
with open(out_path, "w") as out:
|
| 79 |
+
for i in range(0, len(pick), 480):
|
| 80 |
+
chunk = pick[i:i + 480]
|
| 81 |
+
datas = await asyncio.gather(*[fetch(s, sem, *b) for b in chunk])
|
| 82 |
+
for data in datas:
|
| 83 |
+
if not data:
|
| 84 |
+
stats["block_fail"] += 1
|
| 85 |
+
continue
|
| 86 |
+
try:
|
| 87 |
+
text = gzip.decompress(data).decode("utf-8", "replace")
|
| 88 |
+
except OSError:
|
| 89 |
+
stats["gz_fail"] += 1
|
| 90 |
+
continue
|
| 91 |
+
for line in text.splitlines():
|
| 92 |
+
sp = line.find(" {")
|
| 93 |
+
if sp < 0:
|
| 94 |
+
continue
|
| 95 |
+
surt = line[:line.find(" ")]
|
| 96 |
+
try:
|
| 97 |
+
j = json.loads(line[sp + 1:])
|
| 98 |
+
except json.JSONDecodeError:
|
| 99 |
+
continue
|
| 100 |
+
stats["records"] += 1
|
| 101 |
+
if j.get("status") != "200" or j.get("mime-detected", j.get("mime")) not in HTML_MIMES:
|
| 102 |
+
continue
|
| 103 |
+
stats["html"] += 1
|
| 104 |
+
path = surt[surt.find(")") + 1:].lower()
|
| 105 |
+
host = surt[:surt.find(")")]
|
| 106 |
+
rec = {"url": j["url"], "filename": j["filename"], "offset": int(j["offset"]),
|
| 107 |
+
"length": int(j["length"]), "lang": j.get("languages", ""), "host": host}
|
| 108 |
+
if KW.search(path):
|
| 109 |
+
if per_host_kw[host] < 6:
|
| 110 |
+
per_host_kw[host] += 1
|
| 111 |
+
rec["src"] = "kw"
|
| 112 |
+
out.write(json.dumps(rec) + "\n")
|
| 113 |
+
stats["kw"] += 1
|
| 114 |
+
elif rng.random() < 1 / 60 and per_host_rand[host] < 2:
|
| 115 |
+
per_host_rand[host] += 1
|
| 116 |
+
rec["src"] = "rand"
|
| 117 |
+
out.write(json.dumps(rec) + "\n")
|
| 118 |
+
stats["rand"] += 1
|
| 119 |
+
el = time.time() - t0
|
| 120 |
+
print(f"{i + len(chunk)}/{len(pick)} blocks {el:.0f}s {dict(stats)}", flush=True)
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
if __name__ == "__main__":
|
| 124 |
+
asyncio.run(main(sys.argv[1], int(sys.argv[2]), int(sys.argv[3]) if len(sys.argv) > 3 else 1234))
|
training/harvest/fetch_extract.py
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Stage B: fetch WARC records by byte range and extract forms.
|
| 2 |
+
|
| 3 |
+
Usage: python fetch_extract.py CANDIDATES.jsonl OUT_DIR N_PROCS [CONC_PER_PROC]
|
| 4 |
+
Writes OUT_DIR/forms-XX.jsonl.gz (one form per line, with page metadata).
|
| 5 |
+
"""
|
| 6 |
+
import asyncio
|
| 7 |
+
import gzip
|
| 8 |
+
import json
|
| 9 |
+
import multiprocessing as mp
|
| 10 |
+
import os
|
| 11 |
+
import random
|
| 12 |
+
import re
|
| 13 |
+
import sys
|
| 14 |
+
import time
|
| 15 |
+
|
| 16 |
+
import aiohttp
|
| 17 |
+
|
| 18 |
+
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
| 19 |
+
from zap.extract import extract_page # noqa: E402
|
| 20 |
+
|
| 21 |
+
BASE = "https://data.commoncrawl.org/"
|
| 22 |
+
CHARSET_RE = re.compile(rb"""<meta[^>]+charset=["']?([A-Za-z0-9_\-]+)""", re.I)
|
| 23 |
+
XMLDECL_RE = re.compile(r"^\s*<\?xml[^>]*\?>", re.I)
|
| 24 |
+
FORMISH = re.compile(rb"<(input|select|textarea)\b", re.I)
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
def decode_record(raw: bytes):
|
| 28 |
+
raw = gzip.decompress(raw)
|
| 29 |
+
i = raw.find(b"\r\n\r\n")
|
| 30 |
+
rest = raw[i + 4:]
|
| 31 |
+
j = rest.find(b"\r\n\r\n")
|
| 32 |
+
if j < 0:
|
| 33 |
+
return None
|
| 34 |
+
http_h, body = rest[:j].decode("latin-1").lower(), rest[j + 4:]
|
| 35 |
+
if "\ncontent-encoding: gzip" in http_h and body[:2] == b"\x1f\x8b":
|
| 36 |
+
try:
|
| 37 |
+
body = gzip.decompress(body)
|
| 38 |
+
except OSError:
|
| 39 |
+
return None
|
| 40 |
+
if len(body) > 3_000_000 or not FORMISH.search(body):
|
| 41 |
+
return None
|
| 42 |
+
cs = None
|
| 43 |
+
m = re.search(r"content-type:[^\n]*charset=([a-z0-9_\-]+)", http_h)
|
| 44 |
+
if m:
|
| 45 |
+
cs = m.group(1)
|
| 46 |
+
else:
|
| 47 |
+
m = CHARSET_RE.search(body[:4096])
|
| 48 |
+
if m:
|
| 49 |
+
cs = m.group(1).decode("ascii", "ignore").lower()
|
| 50 |
+
for enc in [cs, "utf-8", "cp1252"]:
|
| 51 |
+
if not enc:
|
| 52 |
+
continue
|
| 53 |
+
try:
|
| 54 |
+
html = body.decode(enc)
|
| 55 |
+
break
|
| 56 |
+
except (LookupError, UnicodeDecodeError):
|
| 57 |
+
continue
|
| 58 |
+
else:
|
| 59 |
+
html = body.decode("utf-8", "replace")
|
| 60 |
+
return XMLDECL_RE.sub("", html)
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
async def fetch(session, rec, tries=6):
|
| 64 |
+
for t in range(tries):
|
| 65 |
+
try:
|
| 66 |
+
async with session.get(BASE + rec["filename"],
|
| 67 |
+
headers={"Range": f"bytes={rec['offset']}-{rec['offset'] + rec['length'] - 1}"},
|
| 68 |
+
timeout=aiohttp.ClientTimeout(total=60)) as r:
|
| 69 |
+
if r.status in (200, 206):
|
| 70 |
+
return await r.read()
|
| 71 |
+
if r.status in (429, 503, 500, 502, 504):
|
| 72 |
+
await asyncio.sleep(2 ** t + random.random())
|
| 73 |
+
continue
|
| 74 |
+
return None
|
| 75 |
+
except (aiohttp.ClientError, asyncio.TimeoutError):
|
| 76 |
+
await asyncio.sleep(2 ** t + random.random())
|
| 77 |
+
return None
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
async def run_worker(wid, recs, out_dir, conc):
|
| 81 |
+
out = gzip.open(os.path.join(out_dir, f"forms-{wid:02d}.jsonl.gz"), "wt")
|
| 82 |
+
stats = {"pages": 0, "fetched": 0, "with_forms": 0, "forms": 0, "fail": 0}
|
| 83 |
+
q: asyncio.Queue = asyncio.Queue()
|
| 84 |
+
for r in recs:
|
| 85 |
+
q.put_nowait(r)
|
| 86 |
+
t0 = time.time()
|
| 87 |
+
|
| 88 |
+
async def consumer(session):
|
| 89 |
+
while True:
|
| 90 |
+
try:
|
| 91 |
+
rec = q.get_nowait()
|
| 92 |
+
except asyncio.QueueEmpty:
|
| 93 |
+
return
|
| 94 |
+
data = await fetch(session, rec)
|
| 95 |
+
stats["pages"] += 1
|
| 96 |
+
if data is None:
|
| 97 |
+
stats["fail"] += 1
|
| 98 |
+
continue
|
| 99 |
+
stats["fetched"] += 1
|
| 100 |
+
try:
|
| 101 |
+
html = decode_record(data)
|
| 102 |
+
forms = extract_page(html, rec["url"]) if html else []
|
| 103 |
+
except Exception as e: # noqa: BLE001 - malformed pages must not kill the worker
|
| 104 |
+
stats.setdefault("err", 0)
|
| 105 |
+
stats["err"] += 1
|
| 106 |
+
forms = []
|
| 107 |
+
if forms:
|
| 108 |
+
stats["with_forms"] += 1
|
| 109 |
+
for k, f in enumerate(forms):
|
| 110 |
+
f.update(url=rec["url"], host=rec["host"], lang=rec["lang"], src=rec["src"], fidx=k)
|
| 111 |
+
out.write(json.dumps(f, ensure_ascii=False) + "\n")
|
| 112 |
+
stats["forms"] += 1
|
| 113 |
+
if wid == 0 and stats["pages"] % 500 == 0:
|
| 114 |
+
el = time.time() - t0
|
| 115 |
+
print(f"w0 {stats} {stats['pages'] / el:.1f} pages/s", flush=True)
|
| 116 |
+
await asyncio.sleep(0)
|
| 117 |
+
|
| 118 |
+
conn = aiohttp.TCPConnector(limit=conc)
|
| 119 |
+
async with aiohttp.ClientSession(connector=conn) as session:
|
| 120 |
+
await asyncio.gather(*[consumer(session) for _ in range(conc)])
|
| 121 |
+
out.close()
|
| 122 |
+
return stats
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def worker(args):
|
| 126 |
+
wid, recs, out_dir, conc = args
|
| 127 |
+
return asyncio.run(run_worker(wid, recs, out_dir, conc))
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
if __name__ == "__main__":
|
| 131 |
+
cand, out_dir, nproc = sys.argv[1], sys.argv[2], int(sys.argv[3])
|
| 132 |
+
conc = int(sys.argv[4]) if len(sys.argv) > 4 else 4
|
| 133 |
+
os.makedirs(out_dir, exist_ok=True)
|
| 134 |
+
recs = [json.loads(l) for l in open(cand)]
|
| 135 |
+
random.Random(7).shuffle(recs)
|
| 136 |
+
chunks = [(i, recs[i::nproc], out_dir, conc) for i in range(nproc)]
|
| 137 |
+
t0 = time.time()
|
| 138 |
+
with mp.Pool(nproc) as pool:
|
| 139 |
+
results = pool.map(worker, chunks)
|
| 140 |
+
tot = {}
|
| 141 |
+
for r in results:
|
| 142 |
+
for k, v in r.items():
|
| 143 |
+
tot[k] = tot.get(k, 0) + v
|
| 144 |
+
print(f"DONE {time.time() - t0:.0f}s {tot}", flush=True)
|
training/harvest/parity_py.py
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import json, sys
|
| 2 |
+
sys.path.insert(0, "/job/work")
|
| 3 |
+
from zap.extract import extract_page, serialize
|
| 4 |
+
d = sys.argv[1]
|
| 5 |
+
py = json.load(open(f"{d}/py.json"))
|
| 6 |
+
for p in py:
|
| 7 |
+
html = open(f"{d}/{p['file']}").read()
|
| 8 |
+
p["texts"] = [serialize(f["ctx"], f["fields"]) for f in extract_page(html, p["url"], want_teacher=False)]
|
| 9 |
+
json.dump(py, open(f"{d}/py5.json", "w"), ensure_ascii=False)
|
| 10 |
+
print("ok", sum(len(p["texts"]) for p in py))
|
training/harvest/polite_harvest.py
ADDED
|
@@ -0,0 +1,254 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Polite streaming Common Crawl harvester (replaces the burst-y two-stage pilot).
|
| 2 |
+
|
| 3 |
+
One asyncio fetcher with an adaptive global request rate (backs off hard on 403/429/503),
|
| 4 |
+
CDX index blocks fetched on demand, form-likely URLs prioritized, parsing in a process pool.
|
| 5 |
+
Forms are written to rotating shards OUT/part-NNNN.jsonl.gz (a shard is complete once renamed from .tmp).
|
| 6 |
+
|
| 7 |
+
Usage: python polite_harvest.py OUT_DIR MAX_PAGES [--rate 15] [--max-rate 40] [--procs 12]
|
| 8 |
+
"""
|
| 9 |
+
import argparse
|
| 10 |
+
import asyncio
|
| 11 |
+
import gzip
|
| 12 |
+
import json
|
| 13 |
+
import os
|
| 14 |
+
import random
|
| 15 |
+
import re
|
| 16 |
+
import sys
|
| 17 |
+
import time
|
| 18 |
+
from collections import defaultdict
|
| 19 |
+
from concurrent.futures import ProcessPoolExecutor
|
| 20 |
+
|
| 21 |
+
import aiohttp
|
| 22 |
+
|
| 23 |
+
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
| 24 |
+
from harvest.fetch_extract import decode_record # noqa: E402
|
| 25 |
+
from zap.extract import extract_page # noqa: E402
|
| 26 |
+
|
| 27 |
+
CRAWL = "CC-MAIN-2026-39"
|
| 28 |
+
BASE = "https://data.commoncrawl.org/"
|
| 29 |
+
IDX = f"{BASE}cc-index/collections/{CRAWL}/indexes/"
|
| 30 |
+
HTML_MIMES = ("text/html", "application/xhtml+xml")
|
| 31 |
+
TIER1 = re.compile(
|
| 32 |
+
r"log-?_?in|sign-?_?in|signon|logon|/auth|sso|register|registration|sign-?_?up|create-?_?account|passw|forgot|"
|
| 33 |
+
r"reset|recover|checkout|wp-login|my-?account|/account|anmelden|registrier|passwort|connexion|inscription|"
|
| 34 |
+
r"mot-de-passe|iniciar-sesion|registro|contrase|entrar|cadastro|senha|accedi|registrati|inloggen|registreren|"
|
| 35 |
+
r"wachtwoord|logowanie|rejestracja|zaloguj|giris|kayit|sifre|dang-nhap|dang-ky|masuk|daftar|vhod|parol|2fa|"
|
| 36 |
+
r"verify|otp|mfa|billing|payment|%e7%99%bb|%e6%b3%a8%e5%86%8c|%e3%83%ad%e3%82%b0|%eb%a1%9c%ea%b7%b8")
|
| 37 |
+
TIER2 = re.compile(
|
| 38 |
+
r"account|join|subscribe|newsletter|contact|profile|member|/user|customer|konto|compte|cuenta|conta|accesso|"
|
| 39 |
+
r"uye|verification|shipping|address|booking|reserv|quote|apply|enquiry|inquiry|order|cart|basket|kirjaudu|"
|
| 40 |
+
r"logga|tilmeld|registr|prihlas|belepes|autentic|%d0%b2%d1%85%d0%be%d0%b4")
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
class Limiter:
|
| 44 |
+
def __init__(self, rate, max_rate):
|
| 45 |
+
self.rate, self.max_rate = rate, max_rate
|
| 46 |
+
self.next_t = time.monotonic()
|
| 47 |
+
self.pause_until = 0.0
|
| 48 |
+
self.ok_streak = 0
|
| 49 |
+
self.blocks = 0
|
| 50 |
+
|
| 51 |
+
async def wait(self):
|
| 52 |
+
while True:
|
| 53 |
+
now = time.monotonic()
|
| 54 |
+
if now < self.pause_until:
|
| 55 |
+
await asyncio.sleep(self.pause_until - now)
|
| 56 |
+
continue
|
| 57 |
+
if now >= self.next_t:
|
| 58 |
+
self.next_t = max(self.next_t, now - 0.5) + 1.0 / self.rate
|
| 59 |
+
return
|
| 60 |
+
await asyncio.sleep(self.next_t - now)
|
| 61 |
+
|
| 62 |
+
def ok(self):
|
| 63 |
+
self.ok_streak += 1
|
| 64 |
+
if self.ok_streak >= 400 and self.rate < self.max_rate:
|
| 65 |
+
self.rate = min(self.max_rate, self.rate * 1.1)
|
| 66 |
+
self.ok_streak = 0
|
| 67 |
+
|
| 68 |
+
def throttled(self, status):
|
| 69 |
+
self.blocks += 1
|
| 70 |
+
self.ok_streak = 0
|
| 71 |
+
self.rate = max(2.0, self.rate / 2)
|
| 72 |
+
pause = 90 if status == 403 else 20
|
| 73 |
+
self.pause_until = max(self.pause_until, time.monotonic() + pause)
|
| 74 |
+
print(f"throttled {status}: pausing {pause}s, rate -> {self.rate:.1f}/s", flush=True)
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
async def get(session, lim, url, rng=None, tries=8):
|
| 78 |
+
for _ in range(tries):
|
| 79 |
+
await lim.wait()
|
| 80 |
+
try:
|
| 81 |
+
headers = {"Range": f"bytes={rng[0]}-{rng[1]}"} if rng else {}
|
| 82 |
+
async with session.get(url, headers=headers, timeout=aiohttp.ClientTimeout(total=90)) as r:
|
| 83 |
+
if r.status in (200, 206):
|
| 84 |
+
data = await r.read()
|
| 85 |
+
lim.ok()
|
| 86 |
+
return data
|
| 87 |
+
if r.status in (403, 429, 503):
|
| 88 |
+
lim.throttled(r.status)
|
| 89 |
+
continue
|
| 90 |
+
if r.status >= 500:
|
| 91 |
+
await asyncio.sleep(3)
|
| 92 |
+
continue
|
| 93 |
+
return None
|
| 94 |
+
except (aiohttp.ClientError, asyncio.TimeoutError):
|
| 95 |
+
await asyncio.sleep(3)
|
| 96 |
+
return None
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def parse_job(args):
|
| 100 |
+
rec, data = args
|
| 101 |
+
try:
|
| 102 |
+
html = decode_record(data)
|
| 103 |
+
return rec, (extract_page(html, rec["url"]) if html else [])
|
| 104 |
+
except Exception: # noqa: BLE001
|
| 105 |
+
return rec, []
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
class Writer:
|
| 109 |
+
def __init__(self, out_dir, per_shard=5000):
|
| 110 |
+
self.out_dir, self.per_shard = out_dir, per_shard
|
| 111 |
+
self.n_shard = len([f for f in os.listdir(out_dir) if f.startswith("part-")])
|
| 112 |
+
self.f, self.count, self.total = None, 0, 0
|
| 113 |
+
|
| 114 |
+
def write(self, row):
|
| 115 |
+
if self.f is None:
|
| 116 |
+
self.path = os.path.join(self.out_dir, f"part-{self.n_shard:04d}.jsonl.gz")
|
| 117 |
+
self.f = gzip.open(self.path + ".tmp", "wt")
|
| 118 |
+
self.f.write(json.dumps(row, ensure_ascii=False) + "\n")
|
| 119 |
+
self.count += 1
|
| 120 |
+
self.total += 1
|
| 121 |
+
if self.count >= self.per_shard:
|
| 122 |
+
self.close()
|
| 123 |
+
|
| 124 |
+
def close(self):
|
| 125 |
+
if self.f:
|
| 126 |
+
self.f.close()
|
| 127 |
+
os.rename(self.path + ".tmp", self.path)
|
| 128 |
+
self.f, self.count = None, 0
|
| 129 |
+
self.n_shard += 1
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
async def main(a):
|
| 133 |
+
os.makedirs(a.out, exist_ok=True)
|
| 134 |
+
rnd = random.Random(a.seed)
|
| 135 |
+
lim = Limiter(a.rate, a.max_rate)
|
| 136 |
+
writer = Writer(a.out)
|
| 137 |
+
pool = ProcessPoolExecutor(a.procs)
|
| 138 |
+
loop = asyncio.get_running_loop()
|
| 139 |
+
queues = {"t1": [], "t2": [], "rand": []}
|
| 140 |
+
per_host = defaultdict(int)
|
| 141 |
+
stats = defaultdict(int)
|
| 142 |
+
t0 = time.time()
|
| 143 |
+
cache = os.path.join(a.out, "..", "cluster.idx")
|
| 144 |
+
async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=a.conc)) as s:
|
| 145 |
+
if not os.path.exists(cache):
|
| 146 |
+
data = await get(s, lim, IDX + "cluster.idx")
|
| 147 |
+
if not data:
|
| 148 |
+
raise SystemExit("cannot fetch cluster.idx")
|
| 149 |
+
open(cache, "wb").write(data)
|
| 150 |
+
blocks = []
|
| 151 |
+
for line in open(cache):
|
| 152 |
+
p = line.rstrip("\n").split("\t")
|
| 153 |
+
if len(p) >= 4:
|
| 154 |
+
blocks.append((p[1], int(p[2]), int(p[3])))
|
| 155 |
+
rnd.shuffle(blocks)
|
| 156 |
+
print(f"{len(blocks)} index blocks", flush=True)
|
| 157 |
+
block_i = 0
|
| 158 |
+
refill_lock = asyncio.Lock()
|
| 159 |
+
|
| 160 |
+
async def refill():
|
| 161 |
+
nonlocal block_i
|
| 162 |
+
async with refill_lock:
|
| 163 |
+
if len(queues["t1"]) > 300 or block_i >= len(blocks):
|
| 164 |
+
return
|
| 165 |
+
shard, off, ln = blocks[block_i]
|
| 166 |
+
block_i += 1
|
| 167 |
+
data = await get(s, lim, IDX + shard, (off, off + ln - 1))
|
| 168 |
+
stats["blocks"] += 1
|
| 169 |
+
if not data:
|
| 170 |
+
return
|
| 171 |
+
for line in gzip.decompress(data).decode("utf-8", "replace").splitlines():
|
| 172 |
+
sp = line.find(" {")
|
| 173 |
+
if sp < 0:
|
| 174 |
+
continue
|
| 175 |
+
surt = line[:line.find(" ")]
|
| 176 |
+
try:
|
| 177 |
+
j = json.loads(line[sp + 1:])
|
| 178 |
+
except json.JSONDecodeError:
|
| 179 |
+
continue
|
| 180 |
+
if j.get("status") != "200" or j.get("mime-detected", j.get("mime")) not in HTML_MIMES:
|
| 181 |
+
continue
|
| 182 |
+
host = surt[:surt.find(")")]
|
| 183 |
+
path = surt[surt.find(")") + 1:].lower()
|
| 184 |
+
rec = {"url": j["url"], "filename": j["filename"], "offset": int(j["offset"]),
|
| 185 |
+
"length": int(j["length"]), "lang": j.get("languages", ""), "host": host}
|
| 186 |
+
if TIER1.search(path):
|
| 187 |
+
tier = "t1"
|
| 188 |
+
elif TIER2.search(path):
|
| 189 |
+
tier = "t2"
|
| 190 |
+
elif rnd.random() < 1 / 150:
|
| 191 |
+
tier = "rand"
|
| 192 |
+
else:
|
| 193 |
+
continue
|
| 194 |
+
cap = 2 if tier == "rand" else 5
|
| 195 |
+
if per_host[(host, tier)] >= cap:
|
| 196 |
+
continue
|
| 197 |
+
per_host[(host, tier)] += 1
|
| 198 |
+
rec["src"] = tier
|
| 199 |
+
queues[tier].append(rec)
|
| 200 |
+
for q in queues.values():
|
| 201 |
+
rnd.shuffle(q)
|
| 202 |
+
|
| 203 |
+
async def next_rec():
|
| 204 |
+
while True:
|
| 205 |
+
if len(queues["t1"]) < 200:
|
| 206 |
+
await refill()
|
| 207 |
+
x = rnd.random()
|
| 208 |
+
order = ["t1", "t2", "rand"] if x < 0.6 else (["t2", "t1", "rand"] if x < 0.85 else ["rand", "t1", "t2"])
|
| 209 |
+
for t in order:
|
| 210 |
+
if queues[t]:
|
| 211 |
+
return queues[t].pop()
|
| 212 |
+
if block_i >= len(blocks):
|
| 213 |
+
return None
|
| 214 |
+
|
| 215 |
+
pages = 0
|
| 216 |
+
|
| 217 |
+
async def fetcher():
|
| 218 |
+
nonlocal pages
|
| 219 |
+
while pages < a.max_pages:
|
| 220 |
+
rec = await next_rec()
|
| 221 |
+
if rec is None:
|
| 222 |
+
return
|
| 223 |
+
pages += 1
|
| 224 |
+
data = await get(s, lim, BASE + rec["filename"], (rec["offset"], rec["offset"] + rec["length"] - 1))
|
| 225 |
+
if not data:
|
| 226 |
+
stats["fail"] += 1
|
| 227 |
+
continue
|
| 228 |
+
rec2, forms = await loop.run_in_executor(pool, parse_job, (rec, data))
|
| 229 |
+
stats["pages_" + rec["src"]] += 1
|
| 230 |
+
for k, f in enumerate(forms):
|
| 231 |
+
f.update(url=rec["url"], host=rec["host"], lang=rec["lang"], src=rec["src"], fidx=k)
|
| 232 |
+
writer.write(f)
|
| 233 |
+
stats["forms"] += 1
|
| 234 |
+
if pages % 1000 == 0:
|
| 235 |
+
el = time.time() - t0
|
| 236 |
+
print(f"{pages} pages {el:.0f}s {pages / el:.1f}/s rate {lim.rate:.1f} throttles {lim.blocks} "
|
| 237 |
+
f"{dict(stats)} queued {[len(q) for q in queues.values()]}", flush=True)
|
| 238 |
+
|
| 239 |
+
await asyncio.gather(*[fetcher() for _ in range(a.conc)])
|
| 240 |
+
writer.close()
|
| 241 |
+
pool.shutdown()
|
| 242 |
+
print(f"DONE {time.time() - t0:.0f}s pages {pages} {dict(stats)}", flush=True)
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
if __name__ == "__main__":
|
| 246 |
+
ap = argparse.ArgumentParser()
|
| 247 |
+
ap.add_argument("out")
|
| 248 |
+
ap.add_argument("max_pages", type=int)
|
| 249 |
+
ap.add_argument("--rate", type=float, default=15)
|
| 250 |
+
ap.add_argument("--max-rate", type=float, default=40)
|
| 251 |
+
ap.add_argument("--conc", type=int, default=24)
|
| 252 |
+
ap.add_argument("--procs", type=int, default=12)
|
| 253 |
+
ap.add_argument("--seed", type=int, default=2026)
|
| 254 |
+
asyncio.run(main(ap.parse_args()))
|
training/harvest/sample_pages.py
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Save N decoded HTML pages (with form fields) from one WARC + the Python extractor output, for JS parity tests."""
|
| 2 |
+
import json, os, sys, urllib.request
|
| 3 |
+
sys.path.insert(0, "/job/work")
|
| 4 |
+
from harvest.warc_stream import BASE, members, parse_record
|
| 5 |
+
from zap.extract import extract_page, serialize
|
| 6 |
+
out, n, path = sys.argv[1], int(sys.argv[2]), sys.argv[3]
|
| 7 |
+
os.makedirs(out, exist_ok=True)
|
| 8 |
+
res, k = [], 0
|
| 9 |
+
with urllib.request.urlopen(BASE + path, timeout=300) as r:
|
| 10 |
+
for raw in members(r):
|
| 11 |
+
rec = parse_record(raw)
|
| 12 |
+
if not rec:
|
| 13 |
+
continue
|
| 14 |
+
url, lang, http_h, body = rec
|
| 15 |
+
import re
|
| 16 |
+
m = re.search(r"charset=([a-z0-9_\-]+)", http_h)
|
| 17 |
+
try:
|
| 18 |
+
html = body.decode(m.group(1) if m else "utf-8")
|
| 19 |
+
except (LookupError, UnicodeDecodeError):
|
| 20 |
+
continue
|
| 21 |
+
forms = extract_page(html, url, want_teacher=False)
|
| 22 |
+
if not forms:
|
| 23 |
+
continue
|
| 24 |
+
open(f"{out}/{k:04d}.html", "w").write(html)
|
| 25 |
+
res.append({"file": f"{k:04d}.html", "url": url, "texts": [serialize(f["ctx"], f["fields"]) for f in forms]})
|
| 26 |
+
k += 1
|
| 27 |
+
if k >= n:
|
| 28 |
+
break
|
| 29 |
+
json.dump(res, open(f"{out}/py.json", "w"), ensure_ascii=False)
|
| 30 |
+
print("saved", k)
|
training/harvest/warc_stream.py
ADDED
|
@@ -0,0 +1,181 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Stream whole Common Crawl WARC files (one HTTP request per file) and extract forms.
|
| 2 |
+
|
| 3 |
+
A WARC.gz is a concatenation of per-record gzip members; we split members on the fly and hand
|
| 4 |
+
HTML response records to a process pool. Far fewer requests than per-page range fetches.
|
| 5 |
+
|
| 6 |
+
Usage: python warc_stream.py OUT_DIR N_FILES [--procs 24] [--parallel 3] [--seed 7]
|
| 7 |
+
"""
|
| 8 |
+
import argparse
|
| 9 |
+
import gzip
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
import random
|
| 13 |
+
import sys
|
| 14 |
+
import time
|
| 15 |
+
import urllib.request
|
| 16 |
+
import zlib
|
| 17 |
+
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
|
| 18 |
+
from threading import Lock
|
| 19 |
+
|
| 20 |
+
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
| 21 |
+
from harvest.fetch_extract import FORMISH # noqa: E402
|
| 22 |
+
from harvest.polite_harvest import Writer # noqa: E402
|
| 23 |
+
from zap.extract import extract_page # noqa: E402
|
| 24 |
+
|
| 25 |
+
BASE = "https://data.commoncrawl.org/"
|
| 26 |
+
CRAWL = "CC-MAIN-2026-39"
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def members(stream):
|
| 30 |
+
"""Yield decompressed gzip members from a byte stream."""
|
| 31 |
+
buf = b""
|
| 32 |
+
d = zlib.decompressobj(31)
|
| 33 |
+
out = []
|
| 34 |
+
while True:
|
| 35 |
+
chunk = stream.read(1 << 20)
|
| 36 |
+
if not chunk:
|
| 37 |
+
break
|
| 38 |
+
buf += chunk
|
| 39 |
+
while buf:
|
| 40 |
+
data = d.decompress(buf)
|
| 41 |
+
out.append(data)
|
| 42 |
+
if d.eof:
|
| 43 |
+
yield b"".join(out)
|
| 44 |
+
out = []
|
| 45 |
+
buf = d.unused_data
|
| 46 |
+
d = zlib.decompressobj(31)
|
| 47 |
+
else:
|
| 48 |
+
buf = b""
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def parse_record(raw):
|
| 52 |
+
i = raw.find(b"\r\n\r\n")
|
| 53 |
+
if i < 0:
|
| 54 |
+
return None
|
| 55 |
+
wh = raw[:i].decode("latin-1")
|
| 56 |
+
if "WARC-Type: response" not in wh:
|
| 57 |
+
return None
|
| 58 |
+
url = ""
|
| 59 |
+
for line in wh.split("\r\n"):
|
| 60 |
+
if line.startswith("WARC-Target-URI:"):
|
| 61 |
+
url = line.split(":", 1)[1].strip()
|
| 62 |
+
lang = ""
|
| 63 |
+
for line in wh.split("\r\n"):
|
| 64 |
+
if line.startswith("WARC-Identified-Content-Language:"):
|
| 65 |
+
lang = line.split(":", 1)[1].strip()
|
| 66 |
+
rest = raw[i + 4:]
|
| 67 |
+
j = rest.find(b"\r\n\r\n")
|
| 68 |
+
if j < 0:
|
| 69 |
+
return None
|
| 70 |
+
http_h = rest[:j].decode("latin-1").lower()
|
| 71 |
+
if "content-type: text/html" not in http_h and "application/xhtml" not in http_h:
|
| 72 |
+
return None
|
| 73 |
+
body = rest[j + 4:]
|
| 74 |
+
if len(body) > 3_000_000 or not FORMISH.search(body):
|
| 75 |
+
return None
|
| 76 |
+
return url, lang, http_h, body
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def extract_job(item):
|
| 80 |
+
url, lang, http_h, body = item
|
| 81 |
+
import re
|
| 82 |
+
cs = None
|
| 83 |
+
m = re.search(r"content-type:[^\n]*charset=([a-z0-9_\-]+)", http_h)
|
| 84 |
+
if m:
|
| 85 |
+
cs = m.group(1)
|
| 86 |
+
else:
|
| 87 |
+
m = re.search(rb"""<meta[^>]+charset=["']?([A-Za-z0-9_\-]+)""", body[:4096], re.I)
|
| 88 |
+
if m:
|
| 89 |
+
cs = m.group(1).decode("ascii", "ignore").lower()
|
| 90 |
+
html = None
|
| 91 |
+
for enc in [cs, "utf-8", "cp1252"]:
|
| 92 |
+
if not enc:
|
| 93 |
+
continue
|
| 94 |
+
try:
|
| 95 |
+
html = body.decode(enc)
|
| 96 |
+
break
|
| 97 |
+
except (LookupError, UnicodeDecodeError):
|
| 98 |
+
continue
|
| 99 |
+
if html is None:
|
| 100 |
+
html = body.decode("utf-8", "replace")
|
| 101 |
+
html = re.sub(r"^\s*<\?xml[^>]*\?>", "", html, flags=re.I)
|
| 102 |
+
try:
|
| 103 |
+
return url, lang, extract_page(html, url)
|
| 104 |
+
except Exception: # noqa: BLE001
|
| 105 |
+
return url, lang, []
|
| 106 |
+
|
| 107 |
+
|
| 108 |
+
def host_of(url):
|
| 109 |
+
try:
|
| 110 |
+
h = url.split("/")[2].lower()
|
| 111 |
+
except IndexError:
|
| 112 |
+
return ""
|
| 113 |
+
return ",".join(reversed(h.split(":")[0].split(".")))
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def process_file(path, pool, writer, lock, stats):
|
| 117 |
+
t0 = time.time()
|
| 118 |
+
req = urllib.request.Request(BASE + path)
|
| 119 |
+
batch, futures = [], []
|
| 120 |
+
with urllib.request.urlopen(req, timeout=300) as r:
|
| 121 |
+
for raw in members(r):
|
| 122 |
+
rec = parse_record(raw)
|
| 123 |
+
if rec:
|
| 124 |
+
batch.append(rec)
|
| 125 |
+
if len(batch) >= 64:
|
| 126 |
+
futures.append(pool.submit(_batch, batch))
|
| 127 |
+
batch = []
|
| 128 |
+
if batch:
|
| 129 |
+
futures.append(pool.submit(_batch, batch))
|
| 130 |
+
n_pages = n_forms = 0
|
| 131 |
+
for fu in futures:
|
| 132 |
+
for url, lang, forms in fu.result():
|
| 133 |
+
n_pages += 1
|
| 134 |
+
host = host_of(url)
|
| 135 |
+
with lock:
|
| 136 |
+
for k, f in enumerate(forms):
|
| 137 |
+
f.update(url=url, host=host, lang=lang, src="warc", fidx=k)
|
| 138 |
+
writer.write(f)
|
| 139 |
+
n_forms += 1
|
| 140 |
+
with lock:
|
| 141 |
+
stats["files"] += 1
|
| 142 |
+
stats["pages"] += n_pages
|
| 143 |
+
stats["forms"] += n_forms
|
| 144 |
+
print(f"{path.split('/')[-1]} {n_pages} form-pages {n_forms} forms {time.time() - t0:.0f}s total {stats}", flush=True)
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def _batch(items):
|
| 148 |
+
return [extract_job(it) for it in items]
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def main(a):
|
| 152 |
+
os.makedirs(a.out, exist_ok=True)
|
| 153 |
+
paths_file = os.path.join(a.out, "..", "warc.paths")
|
| 154 |
+
if not os.path.exists(paths_file):
|
| 155 |
+
with urllib.request.urlopen(f"{BASE}crawl-data/{CRAWL}/warc.paths.gz") as r:
|
| 156 |
+
open(paths_file, "wb").write(gzip.decompress(r.read()))
|
| 157 |
+
paths = open(paths_file).read().split()
|
| 158 |
+
random.Random(a.seed).shuffle(paths)
|
| 159 |
+
paths = paths[:a.n_files]
|
| 160 |
+
writer = Writer(a.out, per_shard=10000)
|
| 161 |
+
lock = Lock()
|
| 162 |
+
stats = {"files": 0, "pages": 0, "forms": 0}
|
| 163 |
+
with ProcessPoolExecutor(a.procs) as pool, ThreadPoolExecutor(a.parallel) as tp:
|
| 164 |
+
futs = [tp.submit(process_file, p, pool, writer, lock, stats) for p in paths]
|
| 165 |
+
for f in futs:
|
| 166 |
+
try:
|
| 167 |
+
f.result()
|
| 168 |
+
except Exception as e: # noqa: BLE001
|
| 169 |
+
print("file failed:", e, flush=True)
|
| 170 |
+
writer.close()
|
| 171 |
+
print("DONE", stats, flush=True)
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
if __name__ == "__main__":
|
| 175 |
+
ap = argparse.ArgumentParser()
|
| 176 |
+
ap.add_argument("out")
|
| 177 |
+
ap.add_argument("n_files", type=int)
|
| 178 |
+
ap.add_argument("--procs", type=int, default=24)
|
| 179 |
+
ap.add_argument("--parallel", type=int, default=3)
|
| 180 |
+
ap.add_argument("--seed", type=int, default=7)
|
| 181 |
+
main(ap.parse_args())
|
training/js/bench.mjs
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// CPU latency benchmark: model-only time per form (extraction excluded), onnxruntime-node via transformers.js
|
| 2 |
+
import fs from "fs"; import path from "path"; import { createRequire } from "module";
|
| 3 |
+
const require = createRequire(import.meta.url);
|
| 4 |
+
const tf = require("@huggingface/transformers");
|
| 5 |
+
const { JSDOM, VirtualConsole } = require("jsdom");
|
| 6 |
+
const { zapExtract } = require("./zap-extract.js");
|
| 7 |
+
const { ZapModel } = require("./zap-infer.js");
|
| 8 |
+
const [modelDir, dir, dtype] = process.argv.slice(2);
|
| 9 |
+
tf.env.allowRemoteModels = false; tf.env.localModelPath = path.dirname(path.resolve(modelDir)) + "/";
|
| 10 |
+
const zap = await ZapModel.load(tf, path.basename(modelDir), { dtype: dtype || "fp32" });
|
| 11 |
+
const pages = JSON.parse(fs.readFileSync(path.join(dir, "py.json"), "utf8")).slice(0, 200);
|
| 12 |
+
const vc = new VirtualConsole(); const forms = []; let tx = 0;
|
| 13 |
+
for (const p of pages) { const d = new JSDOM(fs.readFileSync(path.join(dir, p.file), "utf8"), { url: p.url, virtualConsole: vc });
|
| 14 |
+
const t = performance.now(); forms.push(...zapExtract(d.window.document, p.url)); tx += performance.now() - t; }
|
| 15 |
+
await zap.classify(forms.slice(0, 20)); // warmup
|
| 16 |
+
const t0 = performance.now(); const res = await zap.classify(forms); const dt = performance.now() - t0;
|
| 17 |
+
const nf = forms.reduce((a, f) => a + f.fields.length, 0);
|
| 18 |
+
console.log(JSON.stringify({ dtype: dtype || "fp32", forms: forms.length, fields: nf, ms_per_form: +(dt / forms.length).toFixed(2), extract_ms_per_form: +(tx / forms.length).toFixed(2) }));
|
training/js/infer-parity-test.mjs
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// End-to-end JS inference (jsdom + zap-extract.js + zap-infer.js + transformers.js/onnxruntime-node)
|
| 2 |
+
// usage: node infer-parity-test.mjs MODEL_DIR PARITY_DIR OUT.json
|
| 3 |
+
import fs from "fs";
|
| 4 |
+
import path from "path";
|
| 5 |
+
import { createRequire } from "module";
|
| 6 |
+
const require = createRequire(import.meta.url);
|
| 7 |
+
const tf = require("@huggingface/transformers");
|
| 8 |
+
const { JSDOM, VirtualConsole } = require("jsdom");
|
| 9 |
+
const { zapExtract } = require("./zap-extract.js");
|
| 10 |
+
const { ZapModel } = require("./zap-infer.js");
|
| 11 |
+
const [modelDir, dir, outFile] = process.argv.slice(2);
|
| 12 |
+
tf.env.allowRemoteModels = false;
|
| 13 |
+
tf.env.localModelPath = path.dirname(path.resolve(modelDir)) + "/";
|
| 14 |
+
const zap = await ZapModel.load(tf, path.basename(modelDir));
|
| 15 |
+
const pages = JSON.parse(fs.readFileSync(path.join(dir, "py.json"), "utf8")).slice(0, Number(process.env.N || 150));
|
| 16 |
+
const vc = new VirtualConsole();
|
| 17 |
+
const out = [];
|
| 18 |
+
const t0 = Date.now();
|
| 19 |
+
let nf = 0;
|
| 20 |
+
for (const p of pages) {
|
| 21 |
+
const dom = new JSDOM(fs.readFileSync(path.join(dir, p.file), "utf8"), { url: p.url, virtualConsole: vc });
|
| 22 |
+
const forms = zapExtract(dom.window.document, p.url);
|
| 23 |
+
const res = await zap.classify(forms);
|
| 24 |
+
nf += forms.length;
|
| 25 |
+
out.push({ file: p.file, forms: res.map((r) => ({ form: r.form, fields: r.fields.map((f) => f.label) })) });
|
| 26 |
+
dom.window.close();
|
| 27 |
+
}
|
| 28 |
+
console.log(`classified ${nf} forms on ${pages.length} pages in ${Date.now() - t0} ms`);
|
| 29 |
+
fs.writeFileSync(outFile, JSON.stringify(out));
|
training/js/parity-test.js
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Parity test: JS extractor (jsdom) vs Python extractor on saved real pages.
|
| 2 |
+
// usage: NODE_PATH=<dir with jsdom> node parity-test.js PARITY_DIR
|
| 3 |
+
const fs = require("fs");
|
| 4 |
+
const path = require("path");
|
| 5 |
+
const { JSDOM, VirtualConsole } = require("jsdom");
|
| 6 |
+
const vc = new VirtualConsole();
|
| 7 |
+
const { zapExtract } = require("./zap-extract.js");
|
| 8 |
+
const dir = process.argv[2];
|
| 9 |
+
const py = JSON.parse(fs.readFileSync(path.join(dir, process.argv[3] || "py.json"), "utf8"));
|
| 10 |
+
let pages = 0, samePages = 0, forms = 0, sameForms = 0, pyForms = 0;
|
| 11 |
+
const diffs = [];
|
| 12 |
+
for (const p of py) {
|
| 13 |
+
const html = fs.readFileSync(path.join(dir, p.file), "utf8");
|
| 14 |
+
const dom = new JSDOM(html, { url: p.url, virtualConsole: vc });
|
| 15 |
+
const js = zapExtract(dom.window.document, p.url).map((f) => f.text);
|
| 16 |
+
pages++; pyForms += p.texts.length; forms += js.length;
|
| 17 |
+
const pySet = new Set(p.texts);
|
| 18 |
+
let same = 0;
|
| 19 |
+
for (const t of js) if (pySet.has(t)) same++;
|
| 20 |
+
sameForms += same;
|
| 21 |
+
if (same === p.texts.length && js.length === p.texts.length) samePages++;
|
| 22 |
+
else if (diffs.length < 12) {
|
| 23 |
+
const a = p.texts.find((t) => !js.includes(t)) || "", b = js.find((t) => !pySet.has(t)) || "";
|
| 24 |
+
let i = 0; while (i < Math.min(a.length, b.length) && a[i] === b[i]) i++;
|
| 25 |
+
diffs.push({ file: p.file, py: p.texts.length, js: js.length, at: i, pyText: a.slice(Math.max(0, i - 60), i + 100), jsText: b.slice(Math.max(0, i - 60), i + 100) });
|
| 26 |
+
}
|
| 27 |
+
dom.window.close();
|
| 28 |
+
}
|
| 29 |
+
console.log(JSON.stringify({ pages, samePages, pyForms, jsForms: forms, identicalForms: sameForms }, null, 1));
|
| 30 |
+
for (const d of diffs) console.log(JSON.stringify(d));
|
training/js/zap-extract.js
ADDED
|
@@ -0,0 +1,326 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Zap form extractor — JavaScript port of zap/extract.py (keep the two in lockstep).
|
| 2 |
+
// Runs on a live DOM (browser extension content script) or jsdom.
|
| 3 |
+
// const forms = zapExtract(document, location.href);
|
| 4 |
+
// forms[i] = { root, elements, ctx, fields, text } // `text` is what the Zap model reads
|
| 5 |
+
(function (global) {
|
| 6 |
+
"use strict";
|
| 7 |
+
|
| 8 |
+
const SKIP_INPUT_TYPES = new Set(["hidden", "submit", "button", "reset", "image", "file", "checkbox", "radio", "range", "color"]);
|
| 9 |
+
const FIELD_TAGS = new Set(["input", "select", "textarea"]);
|
| 10 |
+
const NO_TEXT_TAGS = new Set(["script", "style", "noscript", "template", "svg", "select", "textarea", "option", "head"]);
|
| 11 |
+
const HEADING_TAGS = new Set(["h1", "h2", "h3", "h4", "h5", "h6", "legend"]);
|
| 12 |
+
const DATA_ATTRS = ["data-testid", "data-test", "data-qa", "data-cy", "formcontrolname", "data-field", "data-name", "data-type"];
|
| 13 |
+
const MAX_FIELDS = 40;
|
| 14 |
+
const MAX_FORMS = 10;
|
| 15 |
+
const CTX_LIMITS = { title: 60, url: 60, form: 60, head: 60, btn: 60, link: 80, txt: 120, hid: 40 };
|
| 16 |
+
const FIELD_LIMITS = { name: 40, id: 40, class: 30, ac: 30, ph: 50, lbl: 60, aria: 50, tip: 40, near: 50, after: 30, mode: 12, max: 6, opt: 60, data: 30 };
|
| 17 |
+
const CTX_ORDER = [["title", "[TITLE]"], ["url", "[URL]"], ["form", "[FORM]"], ["head", "[HEAD]"], ["btn", "[BTN]"], ["link", "[LINK]"], ["txt", "[TXT]"], ["hid", "[HID]"]];
|
| 18 |
+
const FIELD_ORDER = [["type", "[TYPE]"], ["name", "[NAME]"], ["id", "[ID]"], ["class", "[CLASS]"], ["ac", "[AC]"], ["ph", "[PH]"], ["lbl", "[LBL]"], ["aria", "[ARIA]"], ["tip", "[TIP]"], ["near", "[NEAR]"], ["after", "[AFTER]"], ["mode", "[MODE]"], ["max", "[MAX]"], ["req", "[REQ]"], ["opt", "[OPT]"], ["data", "[DATA]"]];
|
| 19 |
+
|
| 20 |
+
const ELEMENT = 1, TEXT = 3, CDATA = 4;
|
| 21 |
+
|
| 22 |
+
function norm(s) {
|
| 23 |
+
if (!s) return "";
|
| 24 |
+
return s.normalize("NFKC").replace(/\s+/g, " ").trim();
|
| 25 |
+
}
|
| 26 |
+
function cut(s, n) {
|
| 27 |
+
const cps = Array.from(s);
|
| 28 |
+
return cps.length <= n ? s.trimEnd() : cps.slice(0, n).join("").trimEnd();
|
| 29 |
+
}
|
| 30 |
+
function words(s) {
|
| 31 |
+
if (!s) return "";
|
| 32 |
+
s = s.normalize("NFKC");
|
| 33 |
+
s = s.replace(/([a-z])([A-Z])/g, "$1 $2");
|
| 34 |
+
s = s.replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2");
|
| 35 |
+
s = s.replace(/([A-Za-z])([0-9])/g, "$1 $2");
|
| 36 |
+
s = s.replace(/([0-9])([A-Za-z])/g, "$1 $2");
|
| 37 |
+
s = s.replace(/[^\p{L}\p{N}\p{M}]/gu, " ");
|
| 38 |
+
return s.replace(/\s+/g, " ").trim().toLowerCase();
|
| 39 |
+
}
|
| 40 |
+
function tag(el) { return el && el.nodeType === ELEMENT ? el.localName.toLowerCase() : ""; }
|
| 41 |
+
function attr(el, name) { const v = el.getAttribute(name); return v === null ? "" : v; }
|
| 42 |
+
function has(el, name) { return el.hasAttribute(name); }
|
| 43 |
+
function unquote(p) { try { return decodeURIComponent(p); } catch (e) { return p; } }
|
| 44 |
+
|
| 45 |
+
function text(el) {
|
| 46 |
+
const parts = [];
|
| 47 |
+
(function walk(e) {
|
| 48 |
+
if (NO_TEXT_TAGS.has(tag(e))) return;
|
| 49 |
+
for (let c = e.firstChild; c; c = c.nextSibling) {
|
| 50 |
+
if (c.nodeType === ELEMENT) walk(c);
|
| 51 |
+
else if (c.nodeType === TEXT || c.nodeType === CDATA) parts.push(c.data);
|
| 52 |
+
}
|
| 53 |
+
})(el);
|
| 54 |
+
return norm(parts.join(" "));
|
| 55 |
+
}
|
| 56 |
+
function inputType(el) {
|
| 57 |
+
const t = tag(el);
|
| 58 |
+
if (t === "input") return attr(el, "type").trim().toLowerCase() || "text";
|
| 59 |
+
return t;
|
| 60 |
+
}
|
| 61 |
+
function hiddenAttr(el) {
|
| 62 |
+
if (has(el, "hidden") || attr(el, "aria-hidden").trim().toLowerCase() === "true") return true;
|
| 63 |
+
const style = attr(el, "style").replace(/ /g, "").toLowerCase();
|
| 64 |
+
return style.includes("display:none") || style.includes("visibility:hidden");
|
| 65 |
+
}
|
| 66 |
+
function isField(el) {
|
| 67 |
+
const t = tag(el);
|
| 68 |
+
if (!FIELD_TAGS.has(t)) return false;
|
| 69 |
+
if (t === "input" && SKIP_INPUT_TYPES.has(inputType(el))) return false;
|
| 70 |
+
return !has(el, "disabled");
|
| 71 |
+
}
|
| 72 |
+
function visibleWithin(el, root) {
|
| 73 |
+
for (let n = el; n; n = n.parentElement) {
|
| 74 |
+
if (hiddenAttr(n)) return false;
|
| 75 |
+
if (n === root) return true;
|
| 76 |
+
}
|
| 77 |
+
return true;
|
| 78 |
+
}
|
| 79 |
+
function containsField(el) {
|
| 80 |
+
return FIELD_TAGS.has(tag(el)) || !!el.querySelector("input,select,textarea");
|
| 81 |
+
}
|
| 82 |
+
|
| 83 |
+
// Previous / next siblings in DOM order, nearest first: ["t", string] or ["e", element]
|
| 84 |
+
function* prevNodes(el) {
|
| 85 |
+
for (let n = el.previousSibling; n; n = n.previousSibling) {
|
| 86 |
+
if (n.nodeType === TEXT || n.nodeType === CDATA) yield ["t", n.data];
|
| 87 |
+
else if (n.nodeType === ELEMENT) yield ["e", n];
|
| 88 |
+
}
|
| 89 |
+
}
|
| 90 |
+
function* nextNodes(el) {
|
| 91 |
+
for (let n = el.nextSibling; n; n = n.nextSibling) {
|
| 92 |
+
if (n.nodeType === TEXT || n.nodeType === CDATA) yield ["t", n.data];
|
| 93 |
+
else if (n.nodeType === ELEMENT) yield ["e", n];
|
| 94 |
+
}
|
| 95 |
+
}
|
| 96 |
+
function nearText(el, root) {
|
| 97 |
+
let node = el;
|
| 98 |
+
for (let level = 0; level < 3; level++) {
|
| 99 |
+
let seen = 0;
|
| 100 |
+
for (const [kind, v] of prevNodes(node)) {
|
| 101 |
+
let t;
|
| 102 |
+
if (kind === "t") t = norm(v);
|
| 103 |
+
else { if (containsField(v)) return ""; t = text(v); }
|
| 104 |
+
if (t) return t;
|
| 105 |
+
if (kind === "e") { seen++; if (seen >= 3) break; }
|
| 106 |
+
}
|
| 107 |
+
if (node === root) break;
|
| 108 |
+
node = node.parentElement;
|
| 109 |
+
if (!node || node === root) break;
|
| 110 |
+
}
|
| 111 |
+
return "";
|
| 112 |
+
}
|
| 113 |
+
function afterText(el) {
|
| 114 |
+
let seen = 0;
|
| 115 |
+
for (const [kind, v] of nextNodes(el)) {
|
| 116 |
+
let t;
|
| 117 |
+
if (kind === "t") t = norm(v);
|
| 118 |
+
else { if (containsField(v)) return ""; t = text(v); }
|
| 119 |
+
if (t) return t;
|
| 120 |
+
if (kind === "e") { seen++; if (seen >= 2) break; }
|
| 121 |
+
}
|
| 122 |
+
return "";
|
| 123 |
+
}
|
| 124 |
+
|
| 125 |
+
function makePage(doc, url) {
|
| 126 |
+
const byId = new Map(), labelFor = new Map();
|
| 127 |
+
for (const e of doc.getElementsByTagName("*")) {
|
| 128 |
+
const i = e.getAttribute("id");
|
| 129 |
+
if (i && !byId.has(i)) byId.set(i, e);
|
| 130 |
+
if (tag(e) === "label" && e.getAttribute("for")) {
|
| 131 |
+
const f = e.getAttribute("for");
|
| 132 |
+
if (!labelFor.has(f)) labelFor.set(f, []);
|
| 133 |
+
labelFor.get(f).push(e);
|
| 134 |
+
}
|
| 135 |
+
}
|
| 136 |
+
const t = doc.querySelector("title");
|
| 137 |
+
return { doc, url, byId, labelFor, title: t ? norm(t.textContent) : "" };
|
| 138 |
+
}
|
| 139 |
+
function labelText(el, page) {
|
| 140 |
+
const out = [];
|
| 141 |
+
for (const ref of attr(el, "aria-labelledby").split(/\s+/).filter(Boolean)) {
|
| 142 |
+
const e = page.byId.get(ref);
|
| 143 |
+
if (e) out.push(text(e));
|
| 144 |
+
}
|
| 145 |
+
const i = el.getAttribute("id");
|
| 146 |
+
if (i) for (const lab of page.labelFor.get(i) || []) out.push(text(lab));
|
| 147 |
+
for (let n = el.parentElement; n; n = n.parentElement) {
|
| 148 |
+
if (tag(n) === "label") { out.push(text(n)); break; }
|
| 149 |
+
}
|
| 150 |
+
const uniq = [];
|
| 151 |
+
for (const t of out) if (t && !uniq.includes(t)) uniq.push(t);
|
| 152 |
+
return uniq.join(" | ");
|
| 153 |
+
}
|
| 154 |
+
|
| 155 |
+
function isButtonish(e) {
|
| 156 |
+
const t = tag(e);
|
| 157 |
+
if (t === "button") return true;
|
| 158 |
+
if (t === "input" && ["submit", "button", "image"].includes(inputType(e))) return true;
|
| 159 |
+
return attr(e, "role").toLowerCase() === "button";
|
| 160 |
+
}
|
| 161 |
+
function isHeading(e) { return HEADING_TAGS.has(tag(e)) || attr(e, "role").toLowerCase() === "heading"; }
|
| 162 |
+
function* iterSelf(root) { yield root; yield* root.getElementsByTagName("*"); }
|
| 163 |
+
|
| 164 |
+
function heading(root) {
|
| 165 |
+
for (const e of iterSelf(root)) {
|
| 166 |
+
if (isHeading(e)) { const t = text(e); if (t) return t; }
|
| 167 |
+
}
|
| 168 |
+
let node = root;
|
| 169 |
+
for (let level = 0; level < 4; level++) {
|
| 170 |
+
let n = 0;
|
| 171 |
+
for (const [kind, sib] of prevNodes(node)) {
|
| 172 |
+
if (kind !== "e") continue;
|
| 173 |
+
n++;
|
| 174 |
+
if (n > 5) break;
|
| 175 |
+
if (isHeading(sib)) { const t = text(sib); if (t) return t; }
|
| 176 |
+
const hs = Array.from(sib.getElementsByTagName("*")).filter(isHeading);
|
| 177 |
+
for (let k = hs.length - 1; k >= 0; k--) { const t = text(hs[k]); if (t) return t; }
|
| 178 |
+
}
|
| 179 |
+
node = node.parentElement;
|
| 180 |
+
if (!node || tag(node) === "body" || tag(node) === "html") break;
|
| 181 |
+
}
|
| 182 |
+
return "";
|
| 183 |
+
}
|
| 184 |
+
function buttonTexts(root) {
|
| 185 |
+
const out = [];
|
| 186 |
+
for (const e of iterSelf(root)) {
|
| 187 |
+
if (!isButtonish(e)) continue;
|
| 188 |
+
const t = text(e) || norm(attr(e, "value")) || norm(attr(e, "aria-label")) || norm(attr(e, "alt")) || norm(attr(e, "title"));
|
| 189 |
+
if (t && !out.includes(t)) out.push(cut(t, 30));
|
| 190 |
+
if (out.length >= 3) break;
|
| 191 |
+
}
|
| 192 |
+
return out.join(" | ");
|
| 193 |
+
}
|
| 194 |
+
function linkTexts(root) {
|
| 195 |
+
const out = [];
|
| 196 |
+
for (const e of iterSelf(root)) {
|
| 197 |
+
if (tag(e) !== "a") continue;
|
| 198 |
+
const t = text(e) || norm(attr(e, "aria-label")) || norm(attr(e, "title"));
|
| 199 |
+
if (t && !out.includes(t)) out.push(cut(t, 30));
|
| 200 |
+
if (out.length >= 4) break;
|
| 201 |
+
}
|
| 202 |
+
return out.join(" | ");
|
| 203 |
+
}
|
| 204 |
+
function hiddenNames(root) {
|
| 205 |
+
const out = [];
|
| 206 |
+
for (const e of iterSelf(root)) {
|
| 207 |
+
if (tag(e) === "input" && inputType(e) === "hidden") {
|
| 208 |
+
const w = words(attr(e, "name"));
|
| 209 |
+
if (w && !out.includes(w)) out.push(w);
|
| 210 |
+
if (out.length >= 4) break;
|
| 211 |
+
}
|
| 212 |
+
}
|
| 213 |
+
return out.join(" ");
|
| 214 |
+
}
|
| 215 |
+
function urlPath(u, base) {
|
| 216 |
+
try { const x = base ? new URL(u, base) : new URL(u); return { host: x.hostname, path: x.pathname }; } catch (e) { return null; }
|
| 217 |
+
}
|
| 218 |
+
function urlWords(url) {
|
| 219 |
+
const u = urlPath(url);
|
| 220 |
+
if (!u) return "";
|
| 221 |
+
const host = u.host.split(".");
|
| 222 |
+
const lead = host.length >= 3 && host[0] !== "www" ? host[0] : "";
|
| 223 |
+
return words(lead + " " + unquote(u.path));
|
| 224 |
+
}
|
| 225 |
+
function formWords(root) {
|
| 226 |
+
const parts = [attr(root, "id"), attr(root, "name"), attr(root, "class")];
|
| 227 |
+
if (tag(root) === "form") {
|
| 228 |
+
const a = attr(root, "action");
|
| 229 |
+
const u = a ? urlPath(a, "http://x/") : null;
|
| 230 |
+
parts.push(u ? unquote(u.path) : "");
|
| 231 |
+
} else parts.push(attr(root, "role"));
|
| 232 |
+
return (words(parts.join(" ")) + " " + norm(attr(root, "aria-label")).toLowerCase()).trim();
|
| 233 |
+
}
|
| 234 |
+
|
| 235 |
+
function fieldFeatures(el, root, page) {
|
| 236 |
+
const t = tag(el);
|
| 237 |
+
const f = { type: t !== "input" ? t : "input " + inputType(el) };
|
| 238 |
+
f.name = words(attr(el, "name"));
|
| 239 |
+
f.id = words(attr(el, "id"));
|
| 240 |
+
f.class = words(attr(el, "class"));
|
| 241 |
+
f.ac = norm(attr(el, "autocomplete")).toLowerCase();
|
| 242 |
+
f.ph = norm(attr(el, "placeholder"));
|
| 243 |
+
f.lbl = labelText(el, page);
|
| 244 |
+
f.aria = norm(attr(el, "aria-label"));
|
| 245 |
+
f.tip = norm(attr(el, "title"));
|
| 246 |
+
const near = nearText(el, root);
|
| 247 |
+
f.near = near === f.lbl || near === f.ph || near === f.aria ? "" : near;
|
| 248 |
+
f.after = afterText(el);
|
| 249 |
+
f.mode = norm(attr(el, "inputmode")).toLowerCase();
|
| 250 |
+
f.max = norm(attr(el, "maxlength"));
|
| 251 |
+
f.req = has(el, "required") || attr(el, "aria-required").toLowerCase() === "true";
|
| 252 |
+
if (t === "select") {
|
| 253 |
+
const opts = Array.from(el.getElementsByTagName("option")).map((o) => norm(o.textContent)).filter(Boolean);
|
| 254 |
+
f.opt = opts.length ? opts.length + ": " + opts.slice(0, 6).join(" | ") : "";
|
| 255 |
+
} else f.opt = "";
|
| 256 |
+
const data = DATA_ATTRS.filter((a) => attr(el, a).trim()).map((a) => norm(attr(el, a)));
|
| 257 |
+
f.data = words(data.slice(0, 2).join(" "));
|
| 258 |
+
for (const [k, n] of Object.entries(FIELD_LIMITS)) if (typeof f[k] === "string") f[k] = cut(f[k], n);
|
| 259 |
+
return f;
|
| 260 |
+
}
|
| 261 |
+
function formContext(root, page) {
|
| 262 |
+
const c = { title: page.title, url: urlWords(page.url), form: formWords(root), head: heading(root),
|
| 263 |
+
btn: buttonTexts(root), link: linkTexts(root), txt: text(root), hid: hiddenNames(root) };
|
| 264 |
+
for (const k of Object.keys(c)) c[k] = cut(c[k], CTX_LIMITS[k]);
|
| 265 |
+
return c;
|
| 266 |
+
}
|
| 267 |
+
function groupRoot(el, body) {
|
| 268 |
+
let node = el.parentElement;
|
| 269 |
+
for (let level = 0; level < 6; level++) {
|
| 270 |
+
if (!node || node === body || tag(node) === "body" || tag(node) === "html") break;
|
| 271 |
+
for (const d of iterSelf(node)) {
|
| 272 |
+
if (isButtonish(d) && !d.closest("form")) return node;
|
| 273 |
+
}
|
| 274 |
+
node = node.parentElement;
|
| 275 |
+
}
|
| 276 |
+
node = el;
|
| 277 |
+
for (let level = 0; level < 3; level++) {
|
| 278 |
+
const p = node.parentElement;
|
| 279 |
+
if (!p || p === body || tag(p) === "body" || tag(p) === "html") break;
|
| 280 |
+
node = p;
|
| 281 |
+
}
|
| 282 |
+
return node;
|
| 283 |
+
}
|
| 284 |
+
function findForms(doc) {
|
| 285 |
+
const body = doc.body || doc.documentElement;
|
| 286 |
+
const groups = new Map();
|
| 287 |
+
for (const el of body.getElementsByTagName("*")) {
|
| 288 |
+
if (!isField(el)) continue;
|
| 289 |
+
let root = el.parentElement ? el.parentElement.closest("form") : null;
|
| 290 |
+
if (!root) root = groupRoot(el, body);
|
| 291 |
+
if (!visibleWithin(el, root)) continue;
|
| 292 |
+
if (!groups.has(root)) groups.set(root, []);
|
| 293 |
+
const g = groups.get(root);
|
| 294 |
+
if (g.length < MAX_FIELDS) g.push(el);
|
| 295 |
+
}
|
| 296 |
+
return Array.from(groups.entries()).slice(0, MAX_FORMS);
|
| 297 |
+
}
|
| 298 |
+
|
| 299 |
+
function serializeCtx(ctx) {
|
| 300 |
+
return CTX_ORDER.filter(([k]) => ctx[k]).map(([k, key]) => key + " " + ctx[k]).join(" ");
|
| 301 |
+
}
|
| 302 |
+
function serializeField(f) {
|
| 303 |
+
const out = ["[FLD]"];
|
| 304 |
+
for (const [k, key] of FIELD_ORDER) {
|
| 305 |
+
const v = f[k];
|
| 306 |
+
if (k === "req") { if (v) out.push("[REQ]"); } else if (v) out.push(key + " " + v);
|
| 307 |
+
}
|
| 308 |
+
return out.join(" ");
|
| 309 |
+
}
|
| 310 |
+
function serialize(ctx, fields) {
|
| 311 |
+
return [serializeCtx(ctx)].concat(fields.map(serializeField)).join(" ");
|
| 312 |
+
}
|
| 313 |
+
|
| 314 |
+
function zapExtract(doc, url) {
|
| 315 |
+
const page = makePage(doc, url);
|
| 316 |
+
return findForms(doc).map(([root, elements]) => {
|
| 317 |
+
const fields = elements.map((el) => fieldFeatures(el, root, page));
|
| 318 |
+
const ctx = formContext(root, page);
|
| 319 |
+
return { kind: tag(root) === "form" ? "form" : "group", root, elements, ctx, fields, text: serialize(ctx, fields) };
|
| 320 |
+
});
|
| 321 |
+
}
|
| 322 |
+
|
| 323 |
+
const api = { zapExtract, serialize, serializeCtx, serializeField, words, norm };
|
| 324 |
+
if (typeof module !== "undefined" && module.exports) module.exports = api;
|
| 325 |
+
else global.Zap = Object.assign(global.Zap || {}, api);
|
| 326 |
+
})(typeof globalThis !== "undefined" ? globalThis : this);
|
training/js/zap-infer.js
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Zap inference in JavaScript (browser extension / Node) with transformers.js.
|
| 2 |
+
// import * as transformers from "@huggingface/transformers";
|
| 3 |
+
// const zap = await ZapModel.load(transformers, "ProCreations/Zap");
|
| 4 |
+
// const results = await zap.classify(Zap.zapExtract(document, location.href));
|
| 5 |
+
// Mirrors zap_infer.py: [CLS] + form context + as many whole fields as fit in 512 tokens + [SEP] per window;
|
| 6 |
+
// form = mean of [CLS] form logits over windows, each field = its [FLD] token restricted to field labels.
|
| 7 |
+
(function (global) {
|
| 8 |
+
"use strict";
|
| 9 |
+
const MAX_LEN = 512;
|
| 10 |
+
const extract = () => (global.Zap || require("./zap-extract.js"));
|
| 11 |
+
|
| 12 |
+
class ZapModel {
|
| 13 |
+
constructor(tf, tok, model) {
|
| 14 |
+
this.Tensor = tf.Tensor;
|
| 15 |
+
this.tok = tok;
|
| 16 |
+
this.model = model;
|
| 17 |
+
const id2label = model.config.id2label;
|
| 18 |
+
this.labels = Object.keys(id2label).sort((a, b) => a - b).map((k) => id2label[k]);
|
| 19 |
+
this.formIdx = this.labels.map((l, i) => (l.startsWith("form:") ? i : -1)).filter((i) => i >= 0);
|
| 20 |
+
this.fieldIdx = this.labels.map((l, i) => (l.startsWith("field:") ? i : -1)).filter((i) => i >= 0);
|
| 21 |
+
}
|
| 22 |
+
static async load(tf, path = "ProCreations/Zap", opts = {}) {
|
| 23 |
+
return new ZapModel(tf, await tf.AutoTokenizer.from_pretrained(path),
|
| 24 |
+
await tf.AutoModelForTokenClassification.from_pretrained(path, { dtype: "fp32", ...opts }));
|
| 25 |
+
}
|
| 26 |
+
ids(text) {
|
| 27 |
+
return Array.from(this.tok.encode(text, { add_special_tokens: false }));
|
| 28 |
+
}
|
| 29 |
+
windows(form) {
|
| 30 |
+
const { serializeCtx, serializeField } = extract();
|
| 31 |
+
const ctx = this.ids(serializeCtx(form.ctx)).slice(0, 160);
|
| 32 |
+
const fields = form.fields.map((f) => this.ids(serializeField(f)).slice(0, 120));
|
| 33 |
+
const cls = this.tok.cls_token_id ?? this.tok.model.tokens_to_ids.get("[CLS]");
|
| 34 |
+
const sep = this.tok.sep_token_id ?? this.tok.model.tokens_to_ids.get("[SEP]");
|
| 35 |
+
const budget = MAX_LEN - 2 - ctx.length;
|
| 36 |
+
const groups = [];
|
| 37 |
+
let cur = [], used = 0;
|
| 38 |
+
fields.forEach((f, i) => {
|
| 39 |
+
if (cur.length && used + f.length > budget) { groups.push(cur); cur = []; used = 0; }
|
| 40 |
+
cur.push(i); used += f.length;
|
| 41 |
+
});
|
| 42 |
+
if (cur.length || !groups.length) groups.push(cur);
|
| 43 |
+
return groups.map((idx) => {
|
| 44 |
+
const ids = [cls, ...ctx], pos = [];
|
| 45 |
+
for (const i of idx) { pos.push(ids.length); ids.push(...fields[i]); }
|
| 46 |
+
ids.push(sep);
|
| 47 |
+
return { ids, pos, idx };
|
| 48 |
+
});
|
| 49 |
+
}
|
| 50 |
+
softmax(row, idx) {
|
| 51 |
+
const v = idx.map((i) => row[i]);
|
| 52 |
+
const m = Math.max(...v);
|
| 53 |
+
const e = v.map((x) => Math.exp(x - m));
|
| 54 |
+
const s = e.reduce((a, b) => a + b, 0);
|
| 55 |
+
return e.map((x) => x / s);
|
| 56 |
+
}
|
| 57 |
+
async classify(forms) {
|
| 58 |
+
const Tensor = this.Tensor;
|
| 59 |
+
const out = [];
|
| 60 |
+
for (const form of forms) {
|
| 61 |
+
const wins = this.windows(form);
|
| 62 |
+
const n = Math.max(...wins.map((w) => w.ids.length));
|
| 63 |
+
const B = wins.length, L = this.labels.length;
|
| 64 |
+
const ids = new BigInt64Array(B * n), mask = new BigInt64Array(B * n);
|
| 65 |
+
wins.forEach((w, b) => w.ids.forEach((t, j) => { ids[b * n + j] = BigInt(t); mask[b * n + j] = 1n; }));
|
| 66 |
+
const { logits } = await this.model({ input_ids: new Tensor("int64", ids, [B, n]), attention_mask: new Tensor("int64", mask, [B, n]) });
|
| 67 |
+
const data = logits.data;
|
| 68 |
+
const row = (b, p) => data.subarray((b * n + p) * L, (b * n + p + 1) * L);
|
| 69 |
+
const formLogits = new Array(this.formIdx.length).fill(0);
|
| 70 |
+
const fields = new Array(form.fields.length);
|
| 71 |
+
wins.forEach((w, b) => {
|
| 72 |
+
const r0 = row(b, 0);
|
| 73 |
+
this.formIdx.forEach((li, k) => { formLogits[k] += r0[li] / B; });
|
| 74 |
+
w.pos.forEach((p, k) => {
|
| 75 |
+
const pr = this.softmax(row(b, p), this.fieldIdx);
|
| 76 |
+
const top = pr.indexOf(Math.max(...pr));
|
| 77 |
+
fields[w.idx[k]] = { label: this.labels[this.fieldIdx[top]].slice(6), score: pr[top], element: form.elements && form.elements[w.idx[k]] };
|
| 78 |
+
});
|
| 79 |
+
});
|
| 80 |
+
const fp = this.softmax(formLogits, formLogits.map((_, i) => i));
|
| 81 |
+
const ft = fp.indexOf(Math.max(...fp));
|
| 82 |
+
out.push({ form: this.labels[this.formIdx[ft]].slice(5), formScore: fp[ft], fields });
|
| 83 |
+
}
|
| 84 |
+
return out;
|
| 85 |
+
}
|
| 86 |
+
}
|
| 87 |
+
if (typeof module !== "undefined" && module.exports) module.exports = { ZapModel };
|
| 88 |
+
else global.ZapModel = ZapModel;
|
| 89 |
+
})(typeof globalThis !== "undefined" ? globalThis : this);
|
training/label/common.py
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Shared helpers: autocomplete-token mapping, uids, loading harvested forms."""
|
| 2 |
+
import glob
|
| 3 |
+
import gzip
|
| 4 |
+
import hashlib
|
| 5 |
+
import json
|
| 6 |
+
|
| 7 |
+
AC_MAP = {
|
| 8 |
+
"username": "username", "email": "email", "current-password": "current-password",
|
| 9 |
+
"new-password": "new-password", "one-time-code": "one-time-code",
|
| 10 |
+
"tel": "tel", "tel-national": "tel", "tel-local": "tel", "tel-country-code": "tel", "tel-area-code": "tel",
|
| 11 |
+
"tel-extension": "tel", "tel-local-prefix": "tel", "tel-local-suffix": "tel",
|
| 12 |
+
"name": "name", "given-name": "given-name", "additional-name": "given-name", "family-name": "family-name",
|
| 13 |
+
"organization": "organization", "street-address": "street-address", "address-line1": "street-address",
|
| 14 |
+
"address-line2": "address-line2", "address-line3": "address-line2", "address-level2": "address-level2",
|
| 15 |
+
"address-level1": "address-level1", "postal-code": "postal-code", "country": "country", "country-name": "country",
|
| 16 |
+
"cc-name": "cc-name", "cc-given-name": "cc-name", "cc-family-name": "cc-name", "cc-number": "cc-number",
|
| 17 |
+
"cc-exp": "cc-exp", "cc-exp-month": "cc-exp-month", "cc-exp-year": "cc-exp-year", "cc-csc": "cc-csc",
|
| 18 |
+
"bday": "bday", "bday-day": "bday", "bday-month": "bday", "bday-year": "bday",
|
| 19 |
+
}
|
| 20 |
+
AC_SKIP = {"section-", "shipping", "billing", "home", "work", "mobile", "fax", "pager", "webauthn"}
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def ac_label(ac: str):
|
| 24 |
+
"""Map an autocomplete attribute to a Zap field label, or None when it carries no field meaning."""
|
| 25 |
+
toks = [t for t in (ac or "").lower().replace('"', "").split() if t]
|
| 26 |
+
toks = [t for t in toks if not t.startswith("section-") and t not in AC_SKIP]
|
| 27 |
+
if len(toks) != 1:
|
| 28 |
+
return None
|
| 29 |
+
return AC_MAP.get(toks[0])
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def uid(row) -> str:
|
| 33 |
+
return hashlib.sha1(f"{row['url']}#{row['fidx']}".encode()).hexdigest()[:16]
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def load_forms(pattern):
|
| 37 |
+
for path in sorted(glob.glob(pattern)):
|
| 38 |
+
with gzip.open(path, "rt") as f:
|
| 39 |
+
for line in f:
|
| 40 |
+
row = json.loads(line)
|
| 41 |
+
row["uid"] = uid(row)
|
| 42 |
+
yield row
|
training/label/eval_calib.py
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Score teacher labels against developer autocomplete tokens (the teacher may or may not have seen them).
|
| 2 |
+
|
| 3 |
+
Usage: python eval_calib.py CALIB.jsonl TEACHER.jsonl [N_EXAMPLES]
|
| 4 |
+
"""
|
| 5 |
+
import json
|
| 6 |
+
import sys
|
| 7 |
+
from collections import Counter, defaultdict
|
| 8 |
+
|
| 9 |
+
sys.path.insert(0, "/job/work")
|
| 10 |
+
from label.common import ac_label # noqa: E402
|
| 11 |
+
from zap.extract import serialize_field # noqa: E402
|
| 12 |
+
|
| 13 |
+
# Format-based semantics: an identifier field with autocomplete=username may legitimately be email/tel.
|
| 14 |
+
ALSO_OK = {"username": {"email", "tel"}}
|
| 15 |
+
|
| 16 |
+
forms = {json.loads(l)["uid"]: json.loads(l) for l in open(sys.argv[1])}
|
| 17 |
+
teach = {}
|
| 18 |
+
for l in open(sys.argv[2]):
|
| 19 |
+
j = json.loads(l)
|
| 20 |
+
teach[j["uid"]] = j
|
| 21 |
+
n_ex = int(sys.argv[3]) if len(sys.argv) > 3 else 25
|
| 22 |
+
per = defaultdict(Counter)
|
| 23 |
+
conf = Counter()
|
| 24 |
+
bad = []
|
| 25 |
+
fail = 0
|
| 26 |
+
for uid, f in forms.items():
|
| 27 |
+
t = teach.get(uid)
|
| 28 |
+
if not t or not t["fields"]:
|
| 29 |
+
fail += 1
|
| 30 |
+
continue
|
| 31 |
+
for fld, tl in zip(f["fields"], t["fields"]):
|
| 32 |
+
g = ac_label(fld["ac"])
|
| 33 |
+
if not g:
|
| 34 |
+
continue
|
| 35 |
+
ok = tl == g or tl in ALSO_OK.get(g, ())
|
| 36 |
+
per[g]["ok" if ok else "bad"] += 1
|
| 37 |
+
if not ok:
|
| 38 |
+
conf[(g, tl)] += 1
|
| 39 |
+
bad.append((g, tl, t["form"], f["url"], serialize_field(fld)))
|
| 40 |
+
tot_ok = sum(c["ok"] for c in per.values())
|
| 41 |
+
tot = sum(c["ok"] + c["bad"] for c in per.values())
|
| 42 |
+
print(f"teacher failures {fail}/{len(forms)}; autocomplete agreement {tot_ok}/{tot} = {tot_ok / max(tot, 1):.3f}")
|
| 43 |
+
for g, c in sorted(per.items(), key=lambda x: -(x[1]["ok"] + x[1]["bad"])):
|
| 44 |
+
print(f" {g:18s} {c['ok']:4d}/{c['ok'] + c['bad']:4d}")
|
| 45 |
+
print("confusions:", conf.most_common(15))
|
| 46 |
+
for b in bad[:n_ex]:
|
| 47 |
+
print(" AC", b[0], "-> T", b[1], "| form", b[2], "|", b[3][:80], "|", b[4][:200])
|
| 48 |
+
print("form labels:", Counter(t["form"] for t in teach.values()).most_common())
|
training/label/prep_calib.py
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Calibration set: deduped pilot forms with >=1 mappable autocomplete token."""
|
| 2 |
+
import json, sys
|
| 3 |
+
sys.path.insert(0, "/job/work")
|
| 4 |
+
from label.common import load_forms, ac_label
|
| 5 |
+
seen, out = set(), []
|
| 6 |
+
for r in load_forms(sys.argv[1]):
|
| 7 |
+
if not r.get("html") or r["sig"] in seen:
|
| 8 |
+
continue
|
| 9 |
+
seen.add(r["sig"])
|
| 10 |
+
if any(ac_label(f["ac"]) for f in r["fields"]):
|
| 11 |
+
out.append(r)
|
| 12 |
+
with open(sys.argv[2], "w") as f:
|
| 13 |
+
for r in out:
|
| 14 |
+
f.write(json.dumps(r, ensure_ascii=False) + "\n")
|
| 15 |
+
print(len(out), "calibration forms")
|
training/label/select.py
ADDED
|
@@ -0,0 +1,126 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Dedup harvested forms, assign host-level splits, and build prioritized teacher-labeling chunks.
|
| 2 |
+
|
| 3 |
+
Usage: python select.py OUT_DIR PATTERN [PATTERN ...] [--labeled DIR ...]
|
| 4 |
+
Signatures already labeled in a previous round (DIR/queue-*.jsonl + DIR/labels-*.jsonl) are not queued again.
|
| 5 |
+
Writes OUT_DIR/queue-NNN.jsonl (teacher input, 10k rows each, interleaved priorities),
|
| 6 |
+
OUT_DIR/reps.jsonl.gz (one representative per signature, with split), OUT_DIR/select_stats.json.
|
| 7 |
+
"""
|
| 8 |
+
import gzip
|
| 9 |
+
import hashlib
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
import random
|
| 13 |
+
import sys
|
| 14 |
+
from collections import Counter
|
| 15 |
+
|
| 16 |
+
sys.path.insert(0, "/job/work")
|
| 17 |
+
from label.common import load_forms # noqa: E402
|
| 18 |
+
|
| 19 |
+
SEARCHY = {"q", "s", "search", "query", "keyword", "keywords", "search query", "k", "term", "search term"}
|
| 20 |
+
MIX = [(0, 0.40), (1, 0.25), (2, 0.17), (3, 0.12), (4, 0.06)]
|
| 21 |
+
CAPS = {3: 60000, 4: 20000}
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def split_of(host):
|
| 25 |
+
h = int(hashlib.md5(host.encode()).hexdigest()[:8], 16) % 100
|
| 26 |
+
return "test" if h < 4 else ("val" if h < 6 else "train")
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def prio(r):
|
| 30 |
+
fs = r["fields"]
|
| 31 |
+
types = [f["type"] for f in fs]
|
| 32 |
+
if "input password" in types:
|
| 33 |
+
return 0
|
| 34 |
+
if len(fs) >= 3:
|
| 35 |
+
return 1
|
| 36 |
+
if len(fs) == 2 or any(t in ("input email", "input tel") for t in types):
|
| 37 |
+
return 2
|
| 38 |
+
f = fs[0]
|
| 39 |
+
if f["type"] == "input search" or f["name"] in SEARCHY or f["id"] in SEARCHY:
|
| 40 |
+
return 4
|
| 41 |
+
return 3
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def labeled_sigs(dirs):
|
| 45 |
+
import glob
|
| 46 |
+
out = {}
|
| 47 |
+
for d in dirs:
|
| 48 |
+
uid2sig = {}
|
| 49 |
+
for q in glob.glob(os.path.join(d, "queue-*.jsonl")):
|
| 50 |
+
for line in open(q):
|
| 51 |
+
j = json.loads(line)
|
| 52 |
+
uid2sig[j["uid"]] = j["sig"]
|
| 53 |
+
for lf in glob.glob(os.path.join(d, "labels-*.jsonl")):
|
| 54 |
+
for line in open(lf):
|
| 55 |
+
try:
|
| 56 |
+
j = json.loads(line)
|
| 57 |
+
except json.JSONDecodeError: # partial last line of an interrupted run
|
| 58 |
+
continue
|
| 59 |
+
if j.get("form") and j.get("fields") and j["uid"] in uid2sig:
|
| 60 |
+
out[uid2sig[j["uid"]]] = j
|
| 61 |
+
return out
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def main(out_dir, patterns, labeled_dirs=()):
|
| 65 |
+
os.makedirs(out_dir, exist_ok=True)
|
| 66 |
+
done = labeled_sigs(labeled_dirs)
|
| 67 |
+
print(f"{len(done)} signatures already labeled", flush=True)
|
| 68 |
+
reps, copies, seen = {}, Counter(), set()
|
| 69 |
+
n = 0
|
| 70 |
+
for pat in patterns:
|
| 71 |
+
for r in load_forms(pat):
|
| 72 |
+
n += 1
|
| 73 |
+
if r["uid"] in seen:
|
| 74 |
+
continue
|
| 75 |
+
seen.add(r["uid"])
|
| 76 |
+
copies[r["sig"]] += 1
|
| 77 |
+
cur = reps.get(r["sig"])
|
| 78 |
+
if cur is None or (not cur.get("html") and r.get("html")):
|
| 79 |
+
reps[r["sig"]] = r
|
| 80 |
+
print(f"{n} rows, {len(seen)} unique uids, {len(reps)} unique signatures", flush=True)
|
| 81 |
+
rng = random.Random(13)
|
| 82 |
+
pools = {p: [] for p, _ in MIX}
|
| 83 |
+
with gzip.open(os.path.join(out_dir, "reps.jsonl.gz"), "wt") as g:
|
| 84 |
+
for sig, r in reps.items():
|
| 85 |
+
r["split"] = split_of(r["host"])
|
| 86 |
+
r["copies"] = copies[sig]
|
| 87 |
+
r["prio"] = prio(r)
|
| 88 |
+
g.write(json.dumps(r, ensure_ascii=False) + "\n")
|
| 89 |
+
if r.get("html") and sig not in done:
|
| 90 |
+
pools[r["prio"]].append(r)
|
| 91 |
+
for p in pools:
|
| 92 |
+
rng.shuffle(pools[p])
|
| 93 |
+
if p in CAPS:
|
| 94 |
+
pools[p] = pools[p][:CAPS[p]]
|
| 95 |
+
stats = {"rows": n, "uids": len(seen), "sigs": len(reps), "pools": {p: len(v) for p, v in pools.items()},
|
| 96 |
+
"splits": Counter(split_of(r["host"]) for r in reps.values())}
|
| 97 |
+
print(stats, flush=True)
|
| 98 |
+
order, idx = [], {p: 0 for p in pools}
|
| 99 |
+
while any(idx[p] < len(pools[p]) for p in pools):
|
| 100 |
+
x, acc = rng.random(), 0.0
|
| 101 |
+
live = [(p, w) for p, w in MIX if idx[p] < len(pools[p])]
|
| 102 |
+
tot = sum(w for _, w in live)
|
| 103 |
+
for p, w in live:
|
| 104 |
+
acc += w / tot
|
| 105 |
+
if x <= acc:
|
| 106 |
+
order.append(pools[p][idx[p]])
|
| 107 |
+
idx[p] += 1
|
| 108 |
+
break
|
| 109 |
+
keep = ("uid", "url", "host", "ctx", "fields", "html", "sig", "split", "prio")
|
| 110 |
+
for c in range(0, len(order), 10000):
|
| 111 |
+
with open(os.path.join(out_dir, f"queue-{c // 10000:03d}.jsonl"), "w") as f:
|
| 112 |
+
for r in order[c:c + 10000]:
|
| 113 |
+
f.write(json.dumps({k: r[k] for k in keep}, ensure_ascii=False) + "\n")
|
| 114 |
+
stats["queued"] = len(order)
|
| 115 |
+
json.dump(stats, open(os.path.join(out_dir, "select_stats.json"), "w"), indent=1)
|
| 116 |
+
print("queued", len(order), flush=True)
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
if __name__ == "__main__":
|
| 120 |
+
import argparse
|
| 121 |
+
ap = argparse.ArgumentParser()
|
| 122 |
+
ap.add_argument("out")
|
| 123 |
+
ap.add_argument("patterns", nargs="+")
|
| 124 |
+
ap.add_argument("--labeled", action="append", default=[])
|
| 125 |
+
a = ap.parse_args()
|
| 126 |
+
main(a.out, a.patterns, a.labeled)
|
training/label/select_rare.py
ADDED
|
@@ -0,0 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Round 3 selection: unlabeled real forms whose fields look like rare classes (card payment, OTP, address parts).
|
| 2 |
+
|
| 3 |
+
Usage: python select_rare.py REPS_DIR OUT_DIR --labeled DIR [--labeled DIR ...]
|
| 4 |
+
"""
|
| 5 |
+
import argparse
|
| 6 |
+
import gzip
|
| 7 |
+
import json
|
| 8 |
+
import os
|
| 9 |
+
import random
|
| 10 |
+
import re
|
| 11 |
+
import sys
|
| 12 |
+
|
| 13 |
+
sys.path.insert(0, "/job/work")
|
| 14 |
+
from label.select import labeled_sigs # noqa: E402
|
| 15 |
+
|
| 16 |
+
POOLS = {
|
| 17 |
+
"payment": (re.compile(r"card.?(number|no\b|num)|\bcc.?(num|number|exp|cvv|csc|name)|\bcvv|\bcvc|\bcsc\b|expir|exp.?(date|month|year)|"
|
| 18 |
+
r"kartennummer|tarjeta|carte|cartão|cardholder|name.?on.?card|security.?code|card.?code|kreditkart"), 20000),
|
| 19 |
+
"otp": (re.compile(r"\botp\b|one.?time|verification.?code|verify.?code|sms.?code|\b2fa\b|\bmfa\b|totp|authenticat\w*.?code|"
|
| 20 |
+
r"passcode|auth.?code|confirmation.?code|code.?(sent|sms|e.?mail)|pin.?code|digit|認証コード|验证码|код"), 20000),
|
| 21 |
+
"address": (re.compile(r"address.?(line)?.?[23]|\bapt\b|apartment|suite|\bstate\b|province|region|county|\bcity\b|town|\bzip|postal|"
|
| 22 |
+
r"postcode|\bplz\b|street|house.?(number|no)|hausnummer|ciudad|\bcep\b|district|stadt|ville|provincia"), 16000),
|
| 23 |
+
}
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def field_text(f):
|
| 27 |
+
return " ".join(str(f.get(k, "")) for k in ("name", "id", "ac", "ph", "lbl", "aria", "near", "data")).lower()
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def main(a):
|
| 31 |
+
os.makedirs(a.out, exist_ok=True)
|
| 32 |
+
done = labeled_sigs(a.labeled)
|
| 33 |
+
rng = random.Random(21)
|
| 34 |
+
picked = {p: [] for p in POOLS}
|
| 35 |
+
with gzip.open(os.path.join(a.reps, "reps.jsonl.gz"), "rt") as f:
|
| 36 |
+
for line in f:
|
| 37 |
+
r = json.loads(line)
|
| 38 |
+
if r["sig"] in done or not r.get("html"):
|
| 39 |
+
continue
|
| 40 |
+
txt = " ".join(field_text(x) for x in r["fields"])
|
| 41 |
+
for p, (rx, _) in POOLS.items():
|
| 42 |
+
if rx.search(txt):
|
| 43 |
+
picked[p].append(r)
|
| 44 |
+
break
|
| 45 |
+
keep = ("uid", "url", "host", "ctx", "fields", "html", "sig", "split", "prio")
|
| 46 |
+
rows = []
|
| 47 |
+
for p, (_, cap) in POOLS.items():
|
| 48 |
+
rng.shuffle(picked[p])
|
| 49 |
+
rows += picked[p][:cap]
|
| 50 |
+
print(p, len(picked[p]), "->", min(cap, len(picked[p])), flush=True)
|
| 51 |
+
rng.shuffle(rows)
|
| 52 |
+
for c in range(0, len(rows), 10000):
|
| 53 |
+
with open(os.path.join(a.out, f"queue-{c // 10000:03d}.jsonl"), "w") as f:
|
| 54 |
+
for r in rows[c:c + 10000]:
|
| 55 |
+
f.write(json.dumps({k: r[k] for k in keep}, ensure_ascii=False) + "\n")
|
| 56 |
+
print("queued", len(rows), flush=True)
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
if __name__ == "__main__":
|
| 60 |
+
ap = argparse.ArgumentParser()
|
| 61 |
+
ap.add_argument("reps")
|
| 62 |
+
ap.add_argument("out")
|
| 63 |
+
ap.add_argument("--labeled", action="append", default=[])
|
| 64 |
+
main(ap.parse_args())
|
training/label/stats.py
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import sys, collections
|
| 2 |
+
sys.path.insert(0, "/job/work")
|
| 3 |
+
from label.common import load_forms
|
| 4 |
+
C = collections.Counter
|
| 5 |
+
rows = list(load_forms(sys.argv[1]))
|
| 6 |
+
sigs = C(r["sig"] for r in rows)
|
| 7 |
+
hosts = C(r["host"] for r in rows)
|
| 8 |
+
print("forms", len(rows), "unique sigs", len(sigs), "hosts", len(hosts))
|
| 9 |
+
uniq = {}
|
| 10 |
+
for r in rows:
|
| 11 |
+
uniq.setdefault(r["sig"], r)
|
| 12 |
+
u = list(uniq.values())
|
| 13 |
+
pw = [r for r in u if any(f["type"] == "input password" for f in r["fields"])]
|
| 14 |
+
print("unique with password", len(pw), "html ok", sum(1 for r in u if r.get("html")))
|
| 15 |
+
print("nfields", sorted(C(min(len(r["fields"]), 12) for r in u).items()))
|
| 16 |
+
print("types", C(f["type"] for r in u for f in r["fields"]).most_common(12))
|
| 17 |
+
print("top hosts", hosts.most_common(5))
|
| 18 |
+
single_search = sum(1 for r in u if len(r["fields"]) == 1 and (r["fields"][0]["type"] == "input search" or r["fields"][0]["name"] in ("q", "s", "search", "query", "keyword", "keywords")))
|
| 19 |
+
print("single-field search-like", single_search)
|
training/label/synth.py
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Generate synthetic forms for classes the static web crawl under-represents (card payment, OTP, password
|
| 2 |
+
change/reset, checkout address, identifier-first logins, SPA div-soup markup), then extract them with the
|
| 3 |
+
standard extractor. They are labeled later by the same teacher pass as real forms.
|
| 4 |
+
|
| 5 |
+
Usage: python synth.py OUT_DIR N [--conc 32] [--seed 5]
|
| 6 |
+
"""
|
| 7 |
+
import argparse
|
| 8 |
+
import asyncio
|
| 9 |
+
import gzip
|
| 10 |
+
import json
|
| 11 |
+
import os
|
| 12 |
+
import random
|
| 13 |
+
import re
|
| 14 |
+
import sys
|
| 15 |
+
|
| 16 |
+
import aiohttp
|
| 17 |
+
|
| 18 |
+
sys.path.insert(0, "/job/work")
|
| 19 |
+
from zap.extract import extract_page # noqa: E402
|
| 20 |
+
|
| 21 |
+
SCENARIOS = [
|
| 22 |
+
(8, "checkout payment step with credit/debit card fields (card number, name on card, expiry, CVC)"),
|
| 23 |
+
(4, "checkout page with shipping address AND card payment fields in one form"),
|
| 24 |
+
(4, "card payment form where expiry month and expiry year are two separate <select> dropdowns"),
|
| 25 |
+
(2, "donation form with amount, name, email and card details"),
|
| 26 |
+
(3, "subscription billing form with card number, expiry (MM/YY single field), CVC and billing ZIP"),
|
| 27 |
+
(2, "add a new payment card form in account settings (cardholder name, number, expiry, security code)"),
|
| 28 |
+
(4, "two-factor authentication page asking for a 6-digit code from an authenticator app (one input)"),
|
| 29 |
+
(4, "SMS verification page where the 6-digit code is split into six single-character inputs"),
|
| 30 |
+
(3, "email verification page asking for a code sent by email, with a resend link"),
|
| 31 |
+
(1, "2FA page asking for a backup/recovery code"),
|
| 32 |
+
(4, "change password page for a logged-in user: current password, new password, confirm new password"),
|
| 33 |
+
(3, "reset password page reached from an email link: new password and confirm password only"),
|
| 34 |
+
(2, "forgot password page asking for the account email address"),
|
| 35 |
+
(1, "forgot password page asking for username or email"),
|
| 36 |
+
(3, "identifier-first sign-in step 1: only an email or phone field and a Next button"),
|
| 37 |
+
(3, "sign-in step 2: the username is shown as text and only the password field is present"),
|
| 38 |
+
(1, "login form with phone number and password"),
|
| 39 |
+
(4, "shipping address form: full name, address line 1, address line 2, city, state/province, postal code, country, phone"),
|
| 40 |
+
(2, "billing address form with first name, last name, company, street, house number, city, ZIP, country"),
|
| 41 |
+
(2, "account profile / billing details edit form with company name and VAT field"),
|
| 42 |
+
(1, "login form inside a modal dialog on a shop homepage"),
|
| 43 |
+
(1, "registration form with username, email, password and repeat password"),
|
| 44 |
+
]
|
| 45 |
+
STYLES = [
|
| 46 |
+
"plain semantic HTML with <form>, <label for> and ids",
|
| 47 |
+
"Bootstrap 5 markup",
|
| 48 |
+
"Tailwind CSS utility classes and no <form> element (a React SPA, fields inside divs, button type=button)",
|
| 49 |
+
"Material UI style: floating labels placed after the input, generated class names like css-1x2y3z",
|
| 50 |
+
"Angular reactive form with formcontrolname attributes and mat-form-field wrappers",
|
| 51 |
+
"Vue app with data-v-xxxx attributes and v-model style names",
|
| 52 |
+
"WooCommerce / WordPress markup",
|
| 53 |
+
"Shopify-like checkout markup with data-testid attributes",
|
| 54 |
+
"legacy table-based layout with <td> labels",
|
| 55 |
+
"Stripe Elements-like structure where each card field is its own div with an input",
|
| 56 |
+
"minimal markup where inputs have only placeholders and no labels",
|
| 57 |
+
"inputs with aria-label only and obfuscated names like field_7f3a2 or input_12",
|
| 58 |
+
]
|
| 59 |
+
QUIRKS = [
|
| 60 |
+
"no autocomplete attributes at all",
|
| 61 |
+
"autocomplete=\"off\" on every input",
|
| 62 |
+
"correct HTML autocomplete tokens on every input",
|
| 63 |
+
"some inputs use type=\"text\" even for email/phone/number",
|
| 64 |
+
"the password inputs have a show/hide toggle button next to them",
|
| 65 |
+
"field names are in the page language (not English)",
|
| 66 |
+
"a hidden honeypot text input styled off-screen",
|
| 67 |
+
"a CAPTCHA text input near the submit button",
|
| 68 |
+
"inputmode and maxlength attributes where appropriate",
|
| 69 |
+
"",
|
| 70 |
+
]
|
| 71 |
+
LANGS = ["English"] * 8 + ["Spanish", "French", "German", "Portuguese (Brazil)", "Italian", "Dutch", "Polish",
|
| 72 |
+
"Russian", "Japanese", "Chinese (Simplified)", "Korean", "Turkish", "Arabic", "Hindi",
|
| 73 |
+
"Indonesian", "Vietnamese", "Swedish", "Czech", "Ukrainian", "Greek", "Hebrew", "Thai"]
|
| 74 |
+
SITES = ["online store", "bank", "airline", "streaming service", "SaaS dashboard", "forum", "news site", "university portal",
|
| 75 |
+
"government service", "hotel booking site", "food delivery app", "crypto exchange", "telecom operator",
|
| 76 |
+
"insurance company", "game platform", "email provider", "charity", "gym", "car rental", "marketplace",
|
| 77 |
+
"developer platform", "social network", "utility company", "ticketing site", "healthcare portal"]
|
| 78 |
+
|
| 79 |
+
PROMPT = """Write the HTML of a realistic web page section for this form.
|
| 80 |
+
|
| 81 |
+
Form: {scenario}
|
| 82 |
+
Website: a {site} (invent a plausible brand name)
|
| 83 |
+
Language of all visible text: {lang}
|
| 84 |
+
Markup style: {style}
|
| 85 |
+
Detail: {quirk}
|
| 86 |
+
|
| 87 |
+
Output exactly:
|
| 88 |
+
TITLE: <the page's <title> text>
|
| 89 |
+
URL: <a plausible https URL of the page>
|
| 90 |
+
<the HTML of the page section containing the form: include a heading, the fields, buttons and links; no <html>, <head>, <script> or <style>>"""
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
async def gen(session, url, prompt, sem):
|
| 94 |
+
body = {"model": "qwen", "messages": [{"role": "user", "content": prompt}], "temperature": 1.0, "top_p": 0.95,
|
| 95 |
+
"max_tokens": 1800, "chat_template_kwargs": {"enable_thinking": False}}
|
| 96 |
+
async with sem:
|
| 97 |
+
for _ in range(3):
|
| 98 |
+
try:
|
| 99 |
+
async with session.post(url, json=body, timeout=aiohttp.ClientTimeout(total=900)) as r:
|
| 100 |
+
j = await r.json()
|
| 101 |
+
return j["choices"][0]["message"]["content"] or ""
|
| 102 |
+
except Exception: # noqa: BLE001
|
| 103 |
+
await asyncio.sleep(3)
|
| 104 |
+
return ""
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def parse(text):
|
| 108 |
+
t = re.search(r"TITLE:\s*(.+)", text)
|
| 109 |
+
u = re.search(r"URL:\s*(\S+)", text)
|
| 110 |
+
m = re.search(r"<[a-zA-Z]", text[u.end():] if u else text)
|
| 111 |
+
if not (t and u and m):
|
| 112 |
+
return None
|
| 113 |
+
html = (text[u.end():] if u else text)[m.start():]
|
| 114 |
+
html = re.sub(r"```\s*$", "", html.strip()).strip()
|
| 115 |
+
return t.group(1).strip()[:200], u.group(1).strip(), html
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
async def main(a):
|
| 119 |
+
os.makedirs(a.out, exist_ok=True)
|
| 120 |
+
rng = random.Random(a.seed)
|
| 121 |
+
weights = [w for w, _ in SCENARIOS]
|
| 122 |
+
jobs = []
|
| 123 |
+
for i in range(a.n):
|
| 124 |
+
sc = rng.choices(SCENARIOS, weights)[0][1]
|
| 125 |
+
jobs.append({"i": i, "scenario": sc, "site": rng.choice(SITES), "lang": rng.choice(LANGS),
|
| 126 |
+
"style": rng.choice(STYLES), "quirk": rng.choice(QUIRKS)})
|
| 127 |
+
url = f"http://127.0.0.1:{a.port}/v1/chat/completions"
|
| 128 |
+
sem = asyncio.Semaphore(a.conc)
|
| 129 |
+
out = gzip.open(os.path.join(a.out, f"part-synth-{a.seed}.jsonl.gz.tmp"), "wt")
|
| 130 |
+
stats = {"ok": 0, "bad": 0, "forms": 0}
|
| 131 |
+
async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(limit=a.conc)) as s:
|
| 132 |
+
async def one(job):
|
| 133 |
+
text = await gen(s, url, PROMPT.format(**job), sem)
|
| 134 |
+
p = parse(text)
|
| 135 |
+
if not p:
|
| 136 |
+
stats["bad"] += 1
|
| 137 |
+
return
|
| 138 |
+
title, page_url, html = p
|
| 139 |
+
doc = f"<html><head><title>{title}</title></head><body>{html}</body></html>"
|
| 140 |
+
forms = extract_page(doc, page_url)
|
| 141 |
+
stats["ok"] += 1
|
| 142 |
+
for k, f in enumerate(forms):
|
| 143 |
+
f.update(url=page_url + f"#synth{job['i']}", host=f"synth,{a.seed},{job['i']}", lang=job["lang"],
|
| 144 |
+
src="synth", fidx=k, synth=job)
|
| 145 |
+
out.write(json.dumps(f, ensure_ascii=False) + "\n")
|
| 146 |
+
stats["forms"] += 1
|
| 147 |
+
if (stats["ok"] + stats["bad"]) % 200 == 0:
|
| 148 |
+
print(stats, flush=True)
|
| 149 |
+
await asyncio.gather(*[one(j) for j in jobs])
|
| 150 |
+
out.close()
|
| 151 |
+
os.rename(os.path.join(a.out, f"part-synth-{a.seed}.jsonl.gz.tmp"), os.path.join(a.out, f"part-synth-{a.seed}.jsonl.gz"))
|
| 152 |
+
print("DONE", stats, flush=True)
|
| 153 |
+
|
| 154 |
+
|
| 155 |
+
if __name__ == "__main__":
|
| 156 |
+
ap = argparse.ArgumentParser()
|
| 157 |
+
ap.add_argument("out")
|
| 158 |
+
ap.add_argument("n", type=int)
|
| 159 |
+
ap.add_argument("--conc", type=int, default=32)
|
| 160 |
+
ap.add_argument("--seed", type=int, default=5)
|
| 161 |
+
ap.add_argument("--port", type=int, default=30020)
|
| 162 |
+
asyncio.run(main(ap.parse_args()))
|
training/label/teacher.py
ADDED
|
@@ -0,0 +1,191 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Teacher labeling: Qwen3.8-Flash-Next labels every field and the form purpose.
|
| 2 |
+
|
| 3 |
+
Usage: python teacher.py IN.jsonl OUT.jsonl [--conc 128] [--mask-ac] [--port 30020] [--limit N]
|
| 4 |
+
IN rows need: uid, url, ctx.title, html (with data-zap indices), fields.
|
| 5 |
+
"""
|
| 6 |
+
import argparse
|
| 7 |
+
import asyncio
|
| 8 |
+
import json
|
| 9 |
+
import re
|
| 10 |
+
import time
|
| 11 |
+
|
| 12 |
+
import aiohttp
|
| 13 |
+
|
| 14 |
+
FIELD_LABELS = ["username", "email", "current-password", "new-password", "one-time-code", "tel", "name",
|
| 15 |
+
"given-name", "family-name", "organization", "street-address", "address-line2", "address-level2",
|
| 16 |
+
"address-level1", "postal-code", "country", "cc-name", "cc-number", "cc-exp", "cc-exp-month",
|
| 17 |
+
"cc-exp-year", "cc-csc", "bday", "search", "captcha", "other"]
|
| 18 |
+
FORM_LABELS = ["login", "signup", "password-change", "password-recovery", "otp", "payment", "address", "newsletter",
|
| 19 |
+
"contact", "search", "other"]
|
| 20 |
+
|
| 21 |
+
SYSTEM = """You label HTML form fields for a password manager's autofill engine.
|
| 22 |
+
|
| 23 |
+
You get one web form: the page URL, the page title, and the cleaned HTML of the form (or of the page region that holds the fields). Every fillable field carries data-zap="N". Give every numbered field exactly one FIELD label, and the whole form one FORM label.
|
| 24 |
+
|
| 25 |
+
FIELD labels (what data goes INTO the field):
|
| 26 |
+
username - account identifier that is not strictly an email address: user name, login name, user ID, member/customer/account number used to sign in, "email or username", "email or phone", "login".
|
| 27 |
+
email - an email address, in any context: login "Email", signup email, newsletter, contact, checkout, "confirm email".
|
| 28 |
+
current-password - the user's EXISTING password or sign-in PIN/passcode: login password, "current/old password" in a change-password form, password re-entry to confirm identity.
|
| 29 |
+
new-password - a password being CREATED or SET: signup password, "new password", "confirm/repeat/retype password" on signup, change or reset forms.
|
| 30 |
+
one-time-code - one-time code: 2FA/OTP, SMS or email verification code, authenticator code, backup/recovery code, activation code (a single box or one of several digit boxes).
|
| 31 |
+
tel - phone/mobile number or part of one (country dialing code, area code, extension).
|
| 32 |
+
name - full name in a single field.
|
| 33 |
+
given-name - first name (also middle name).
|
| 34 |
+
family-name - last name / surname.
|
| 35 |
+
organization - company / organization / business name.
|
| 36 |
+
street-address - street address, address line 1, street name, house number.
|
| 37 |
+
address-line2 - apartment, suite, unit, floor, building, address line 2 or 3.
|
| 38 |
+
address-level2 - city / town / locality.
|
| 39 |
+
address-level1 - state / province / region / county.
|
| 40 |
+
postal-code - ZIP / postal code / postcode.
|
| 41 |
+
country - country or country/region.
|
| 42 |
+
cc-name - name on the payment card.
|
| 43 |
+
cc-number - payment card number.
|
| 44 |
+
cc-exp - card expiry date in one field (MM/YY).
|
| 45 |
+
cc-exp-month - card expiry month only.
|
| 46 |
+
cc-exp-year - card expiry year only.
|
| 47 |
+
cc-csc - card security code (CVV/CVC/CID).
|
| 48 |
+
bday - date of birth or one part of it (birth day/month/year).
|
| 49 |
+
search - site search / search query box.
|
| 50 |
+
captcha - CAPTCHA or anti-spam challenge answer (e.g. "what is 3+4?").
|
| 51 |
+
other - anything else: message/comment, subject, quantity, amount, coupon/promo code, booking dates, gender, website URL, security-question answers, IBAN/tax/ID numbers, nickname/display name on a non-login form, honeypot fields meant to stay empty, filters, sorting, etc.
|
| 52 |
+
|
| 53 |
+
FORM labels (what the whole form is for):
|
| 54 |
+
login - signing in to an existing account, including the "enter your email/username" first step of a multi-step sign-in.
|
| 55 |
+
signup - creating a new account / registration (may also ask for name, address, phone...).
|
| 56 |
+
password-change - setting a new password for an existing account (change-password page, or the reset-link page that asks for the new password).
|
| 57 |
+
password-recovery - requesting a password reset / "forgot password" / "forgot username" (asks only for an identifier).
|
| 58 |
+
otp - entering a one-time / verification / 2FA code.
|
| 59 |
+
payment - entering payment card details (checkout payment, donation by card, billing with card number).
|
| 60 |
+
address - shipping/billing/contact details without a card: checkout address, profile/account details, delivery info.
|
| 61 |
+
newsletter - subscribing an email (or phone) to a newsletter or updates, possibly with a name.
|
| 62 |
+
contact - sending a message: contact form, comment form, feedback, quote/enquiry, support ticket, review.
|
| 63 |
+
search - searching the site (search box, optionally with filters).
|
| 64 |
+
other - everything else: booking/availability search, calculators, filter-only forms, polls, job applications, settings, age gates, language/currency pickers, etc.
|
| 65 |
+
|
| 66 |
+
Rules:
|
| 67 |
+
- Decide from ALL evidence: type, name, id, autocomplete, placeholder, labels, nearby text, buttons, headings, page title and URL. Text can be in any language.
|
| 68 |
+
- autocomplete attributes are strong hints but can be wrong (e.g. "new-password" or "off" set only to block autofill). Follow what the field really asks for.
|
| 69 |
+
- A type="password" field is never username or email. Decide current-password vs new-password from the form's purpose. A masked field for a code sent by SMS/email is one-time-code.
|
| 70 |
+
- On a login form: a field accepting "email or username" (or "email or phone") is username; a field asking only for an email is email; a phone-only field is tel.
|
| 71 |
+
- If a form mixes login and signup, pick the FORM label of the action its submit button performs.
|
| 72 |
+
- If the numbered fields are unrelated bits of a page, use FORM "other" unless one clear purpose exists.
|
| 73 |
+
|
| 74 |
+
Output ONLY this format, no explanations:
|
| 75 |
+
FORM: <form label>
|
| 76 |
+
1: <field label>
|
| 77 |
+
2: <field label>
|
| 78 |
+
..."""
|
| 79 |
+
|
| 80 |
+
AC_RE = re.compile(r'\sautocomplete="[^"]*"')
|
| 81 |
+
LINE_RE = re.compile(r"^\s*(\d+)\s*[:.)-]\s*([a-z0-9\-]+)\s*$")
|
| 82 |
+
FORM_RE = re.compile(r"^\s*FORM\s*:\s*([a-z\-]+)\s*$", re.I)
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def build_user(row, mask_ac=False):
|
| 86 |
+
html = row["html"]
|
| 87 |
+
if mask_ac:
|
| 88 |
+
html = AC_RE.sub("", html)
|
| 89 |
+
return f"URL: {row['url']}\nTitle: {row['ctx'].get('title', '')}\nHTML:\n{html}"
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def parse(text, n_fields):
|
| 93 |
+
form, fields = None, {}
|
| 94 |
+
for line in (text or "").splitlines():
|
| 95 |
+
m = FORM_RE.match(line)
|
| 96 |
+
if m:
|
| 97 |
+
form = m.group(1).lower()
|
| 98 |
+
continue
|
| 99 |
+
m = LINE_RE.match(line)
|
| 100 |
+
if m:
|
| 101 |
+
fields[int(m.group(1))] = m.group(2).lower()
|
| 102 |
+
if form not in FORM_LABELS:
|
| 103 |
+
return None
|
| 104 |
+
labels = [fields.get(i) for i in range(1, n_fields + 1)]
|
| 105 |
+
if any(l not in FIELD_LABELS for l in labels):
|
| 106 |
+
return None
|
| 107 |
+
return form, labels
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
async def label_one(session, url, row, args, stats):
|
| 111 |
+
n = len(row["fields"])
|
| 112 |
+
body = {"model": "qwen", "messages": [{"role": "system", "content": SYSTEM},
|
| 113 |
+
{"role": "user", "content": build_user(row, args.mask_ac)}],
|
| 114 |
+
"temperature": 0.0, "max_tokens": 24 + 10 * n,
|
| 115 |
+
"chat_template_kwargs": {"enable_thinking": False}}
|
| 116 |
+
for attempt in range(3):
|
| 117 |
+
try:
|
| 118 |
+
async with session.post(url, json=body, timeout=aiohttp.ClientTimeout(total=600)) as r:
|
| 119 |
+
j = await r.json()
|
| 120 |
+
msg = j["choices"][0]["message"]
|
| 121 |
+
text = msg.get("content") or ""
|
| 122 |
+
usage = j.get("usage", {})
|
| 123 |
+
stats["prompt"] += usage.get("prompt_tokens", 0)
|
| 124 |
+
stats["completion"] += usage.get("completion_tokens", 0)
|
| 125 |
+
p = parse(text, n)
|
| 126 |
+
if p:
|
| 127 |
+
return {"uid": row["uid"], "form": p[0], "fields": p[1], "raw": text}
|
| 128 |
+
stats["parse_fail"] += 1
|
| 129 |
+
body["temperature"] = 0.3
|
| 130 |
+
except Exception as e: # noqa: BLE001
|
| 131 |
+
stats["http_fail"] += 1
|
| 132 |
+
await asyncio.sleep(2 + attempt * 3)
|
| 133 |
+
return {"uid": row["uid"], "form": None, "fields": None, "raw": None}
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
async def main(args):
|
| 137 |
+
rows = [json.loads(l) for l in open(args.inp)]
|
| 138 |
+
if args.limit:
|
| 139 |
+
rows = rows[:args.limit]
|
| 140 |
+
done = set()
|
| 141 |
+
try:
|
| 142 |
+
for l in open(args.out):
|
| 143 |
+
try:
|
| 144 |
+
done.add(json.loads(l)["uid"])
|
| 145 |
+
except json.JSONDecodeError:
|
| 146 |
+
pass
|
| 147 |
+
except FileNotFoundError:
|
| 148 |
+
pass
|
| 149 |
+
rows = [r for r in rows if r["uid"] not in done]
|
| 150 |
+
print(f"{len(rows)} to label ({len(done)} already done)", flush=True)
|
| 151 |
+
url = f"http://127.0.0.1:{args.port}/v1/chat/completions"
|
| 152 |
+
stats = {"prompt": 0, "completion": 0, "parse_fail": 0, "http_fail": 0, "ok": 0, "bad": 0}
|
| 153 |
+
q: asyncio.Queue = asyncio.Queue()
|
| 154 |
+
for r in rows:
|
| 155 |
+
q.put_nowait(r)
|
| 156 |
+
t0 = time.time()
|
| 157 |
+
out = open(args.out, "a")
|
| 158 |
+
|
| 159 |
+
async def worker(session):
|
| 160 |
+
while True:
|
| 161 |
+
try:
|
| 162 |
+
row = q.get_nowait()
|
| 163 |
+
except asyncio.QueueEmpty:
|
| 164 |
+
return
|
| 165 |
+
res = await label_one(session, url, row, args, stats)
|
| 166 |
+
stats["ok" if res["form"] else "bad"] += 1
|
| 167 |
+
out.write(json.dumps(res, ensure_ascii=False) + "\n")
|
| 168 |
+
n = stats["ok"] + stats["bad"]
|
| 169 |
+
if n % 500 == 0:
|
| 170 |
+
out.flush()
|
| 171 |
+
el = time.time() - t0
|
| 172 |
+
print(f"{n}/{len(rows)} {el:.0f}s {n / el:.1f} forms/s prompt {stats['prompt'] / el:.0f} tok/s "
|
| 173 |
+
f"gen {stats['completion'] / el:.0f} tok/s {stats}", flush=True)
|
| 174 |
+
|
| 175 |
+
conn = aiohttp.TCPConnector(limit=args.conc)
|
| 176 |
+
async with aiohttp.ClientSession(connector=conn) as session:
|
| 177 |
+
await asyncio.gather(*[worker(session) for _ in range(args.conc)])
|
| 178 |
+
out.close()
|
| 179 |
+
el = time.time() - t0
|
| 180 |
+
print(f"DONE {el:.0f}s {stats}", flush=True)
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
if __name__ == "__main__":
|
| 184 |
+
ap = argparse.ArgumentParser()
|
| 185 |
+
ap.add_argument("inp")
|
| 186 |
+
ap.add_argument("out")
|
| 187 |
+
ap.add_argument("--conc", type=int, default=128)
|
| 188 |
+
ap.add_argument("--mask-ac", action="store_true")
|
| 189 |
+
ap.add_argument("--port", type=int, default=30020)
|
| 190 |
+
ap.add_argument("--limit", type=int, default=0)
|
| 191 |
+
asyncio.run(main(ap.parse_args()))
|
training/runner/run.sh
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/bin/bash
|
| 2 |
+
# Queue runner for the Zap temporary job.
|
| 3 |
+
# Every *.sh dropped into /job/work/queue/ is launched in the background (tasks may run in parallel;
|
| 4 |
+
# a task that needs ordering waits on another task's /job/work/state/NAME.exit file itself).
|
| 5 |
+
# /job/work/queue/STOP -> stop launching, wait for running tasks, exit.
|
| 6 |
+
set -u
|
| 7 |
+
Q=/job/work/queue; S=/job/work/state; L=/job/logs/tasks
|
| 8 |
+
mkdir -p "$Q" "$S" "$L"
|
| 9 |
+
echo "$(date -Is) runner up"
|
| 10 |
+
declare -A PIDS
|
| 11 |
+
while true; do
|
| 12 |
+
for f in $(ls "$Q"/*.sh 2>/dev/null | sort); do
|
| 13 |
+
name=$(basename "$f" .sh)
|
| 14 |
+
mv "$f" "$S/$name.sh"
|
| 15 |
+
( bash "$S/$name.sh" > "$L/$name.log" 2>&1; echo $? > "$S/$name.exit" ) &
|
| 16 |
+
PIDS[$name]=$!
|
| 17 |
+
echo "$(date -Is) launched $name pid ${PIDS[$name]}"
|
| 18 |
+
done
|
| 19 |
+
for name in "${!PIDS[@]}"; do
|
| 20 |
+
if ! kill -0 "${PIDS[$name]}" 2>/dev/null; then
|
| 21 |
+
echo "$(date -Is) finished $name exit $(cat "$S/$name.exit" 2>/dev/null)"
|
| 22 |
+
unset "PIDS[$name]"
|
| 23 |
+
fi
|
| 24 |
+
done
|
| 25 |
+
if [ -f "$Q/STOP" ] && [ ${#PIDS[@]} -eq 0 ]; then echo "$(date -Is) STOP"; break; fi
|
| 26 |
+
sleep 5
|
| 27 |
+
done
|
training/runner/t00_setup.sh
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
set -x
|
| 2 |
+
cd /job/work
|
| 3 |
+
python3 -m venv --system-site-packages /job/work/.venv
|
| 4 |
+
. /job/work/.venv/bin/activate
|
| 5 |
+
pip install -q selectolax warcio onnx onnxruntime onnxscript tldextract aiohttp 2>&1 | tail -3
|
| 6 |
+
python3 -c "import selectolax, onnx, onnxruntime, aiohttp, torch; print('ok', torch.cuda.is_available())"
|
| 7 |
+
# compare baked overlays in image vs production overlay files
|
| 8 |
+
P=/home/user/.local/share/rtx-pro-apps/qwen3.8-flash-next/runtime-speed-20260924
|
| 9 |
+
for pair in "qwen_sparse_attn_backend.py:layers/attention/qwen_sparse_attn_backend.py" "qsa_sparse_attn.py:layers/attention/qsa/sparse_attn.py" "qwen4_exp_mtp_hotmap.py:models/qwen4_exp_mtp.py" "eagle_worker_v2.py:speculative/eagle_worker_v2.py" "metadata.py:layers/attention/qsa/metadata.py" "graph_metadata.py:layers/attention/qsa/graph_metadata.py" "qsa_indexer.py:layers/attention/qsa/qsa_indexer.py" "qsa_kv_pool.py:mem_cache/qsa_kv_pool.py" "eagle_utils.py:speculative/eagle_utils.py" "qwenfast.py:qwenfast.py" "qwen4_exp_startup.py:models/qwen4_exp.py" "loader_startup.py:model_loader/loader.py"; do
|
| 10 |
+
a=${pair%%:*}; b=${pair#*:}
|
| 11 |
+
echo "$a $(sha256sum $P/$a | cut -c1-12) $(sha256sum /sgl-workspace/sglang/python/sglang/srt/$b | cut -c1-12)"
|
| 12 |
+
done
|
| 13 |
+
ls /opt/ | head; ls /opt/qwen* 2>/dev/null
|
| 14 |
+
curl -sI https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-39/warc.paths.gz | head -3
|
| 15 |
+
df -h /job | tail -1
|
training/runner/t01_pylib.sh
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
cd /job/work
|
| 2 |
+
rm -rf /job/work/.venv
|
| 3 |
+
pip install -q --no-deps --target /job/work/pylib selectolax onnx onnxruntime onnxscript onnx_ir coloredlogs humanfriendly flatbuffers 2>&1 | tail -3
|
| 4 |
+
export PYTHONPATH=/job/work/pylib
|
| 5 |
+
python3 -c "
|
| 6 |
+
import selectolax, onnx, onnxruntime, onnxscript, aiohttp, torch, numpy, lxml, tokenizers, transformers
|
| 7 |
+
print('ok', torch.__version__, numpy.__version__, onnx.__version__, onnxruntime.__version__, torch.cuda.is_available())
|
| 8 |
+
"
|
training/runner/t02_pilot.sh
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/pilot
|
| 3 |
+
python3 harvest/cc_candidates.py data/pilot/cand.jsonl 200 99 2>&1 | tail -3
|
| 4 |
+
wc -l data/pilot/cand.jsonl
|
| 5 |
+
head -3000 data/pilot/cand.jsonl > data/pilot/cand3k.jsonl
|
| 6 |
+
python3 harvest/fetch_extract.py data/pilot/cand3k.jsonl data/pilot/forms 28 4 2>&1 | tail -5
|
training/runner/t03_harvest.sh
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/cc
|
| 3 |
+
python3 harvest/cc_candidates.py data/cc/cand.jsonl 16000 2026 > /job/logs/tasks/cand.log 2>&1
|
| 4 |
+
tail -3 /job/logs/tasks/cand.log
|
| 5 |
+
wc -l data/cc/cand.jsonl || exit 1
|
| 6 |
+
python3 harvest/fetch_extract.py data/cc/cand.jsonl data/cc/forms 28 6 > /job/logs/tasks/fetch.log 2>&1
|
| 7 |
+
tail -3 /job/logs/tasks/fetch.log
|
| 8 |
+
du -sh data/cc
|
training/runner/t04_sglang.sh
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# private high-batch teacher server (loopback inside the job container only)
|
| 2 |
+
mkdir -p /job/cache /job/home
|
| 3 |
+
[ -e /job/home/.cache ] || ln -s /job/cache /job/home/.cache
|
| 4 |
+
RC=/home/user/ai/maintenance/20260919-upgrades/runtime-cache/qwen38-flash-next-sglang
|
| 5 |
+
rm -rf /job/cache/sglang; mkdir -p /job/cache/sglang/inductor
|
| 6 |
+
export SGLANG_CACHE_DIR=/job/cache/sglang TORCHINDUCTOR_CACHE_DIR=/job/cache/sglang/inductor
|
| 7 |
+
[ -d /job/cache/triton/cache ] || cp -r $RC/triton/. /job/cache/triton/ 2>/dev/null
|
| 8 |
+
[ -d /job/home/.tilelang ] || cp -r $RC/.tilelang /job/home/.tilelang 2>/dev/null
|
| 9 |
+
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
| 10 |
+
M=/home/user/models/huggingface/hub/models--RadixArk--Qwen3.8-Flash-Next-NVFP4/snapshots/7b719225242aacd3dbd3f9407468c2ee9a9d2594
|
| 11 |
+
exec python3 -m sglang.launch_server --model-path $M --served-model-name qwen --tp 1 --trust-remote-code \
|
| 12 |
+
--quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cutlass --context-length 16384 \
|
| 13 |
+
--max-total-tokens 131072 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.97 --page-size 64 \
|
| 14 |
+
--chunked-prefill-size 8192 --ple-offload-embedding --linear-attn-prefill-backend triton \
|
| 15 |
+
--linear-attn-decode-backend flashinfer --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer \
|
| 16 |
+
--mamba-track-interval 64 --max-mamba-cache-size 128 --max-running-requests 64 --cuda-graph-max-bs-decode 64 \
|
| 17 |
+
--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":16}' \
|
| 18 |
+
--reasoning-parser auto --sampling-defaults model --host 127.0.0.1 --port 30020
|
training/runner/t05_grep.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
env | grep -i "cache\|sglang" | head -20
|
| 2 |
+
grep -rn "\.cache/sglang\|inductor" /sgl-workspace/sglang/python/sglang/srt --include=*.py | grep -i "cache_dir\|environ\|getenv\|expanduser\|/root" | head -20
|
| 3 |
+
grep -rn "root/.cache" /sgl-workspace/sglang/python/sglang/srt --include=*.py | head
|
training/runner/t06_harvest.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/cc/parts
|
| 3 |
+
python3 harvest/polite_harvest.py data/cc/parts 250000 --rate 10 --max-rate 25 --conc 24 --procs 12
|
training/runner/t07_calib_prep.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/calib
|
| 3 |
+
python3 label/prep_calib.py 'data/pilot/forms/*.jsonl.gz' data/calib/calib.jsonl
|
training/runner/t08_calib.sh
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work
|
| 3 |
+
curl -s http://127.0.0.1:30020/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5,"chat_template_kwargs":{"enable_thinking":false}}'; echo
|
| 4 |
+
python3 label/teacher.py data/calib/calib.jsonl data/calib/teacher_maskac.jsonl --conc 64 --mask-ac
|
| 5 |
+
python3 label/teacher.py data/calib/calib.jsonl data/calib/teacher_full.jsonl --conc 64
|
training/runner/t09_warc_test.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/warc/parts
|
| 3 |
+
python3 harvest/warc_stream.py data/warc/parts 2 --procs 16 --parallel 2 --seed 11
|
training/runner/t10_warc.sh
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/warc/parts
|
| 3 |
+
python3 harvest/warc_stream.py data/warc/parts 50 --procs 26 --parallel 4 --seed 11 2>&1 | grep -v "^CC-MAIN" ; ls data/warc/parts | wc -l
|
training/runner/t11_label.sh
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
export PYTHONPATH=/job/work/pylib:/job/work
|
| 2 |
+
cd /job/work && mkdir -p data/label
|
| 3 |
+
python3 label/select.py data/label 'data/pilot/forms/*.jsonl.gz' 'data/warc/parts/*.jsonl.gz' 'data/cc/parts/*.jsonl.gz'
|
| 4 |
+
for q in data/label/queue-*.jsonl; do
|
| 5 |
+
python3 label/teacher.py $q data/label/labels-all.jsonl --conc 64 2>&1 | grep -v "^\s*$"
|
| 6 |
+
done
|