File size: 2,403 Bytes
795f737 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | # Synthetic v1 schema and limitations
Each JSONL row has schema_version, source, license, split, template, seed,
sample_id, goal, elements, action, target, argument, execution_verified and
content_hash. Element fields are ref, role, name, path, visible, enabled, sensitive.
Target is a zero-based index into the row's elements; action is a supported DSL
opcode. Arguments are synthetic values and never production secrets.
Seeds 101/202/303/404 generate training/validation/test/novel-wording data.
Training has 2,400 rows and each evaluation split has 480. There are no imported
third-party records. `CC0-1.0` describes this generated fixture data, not any future
third-party dataset. Model-visible rows omit oracle metadata and target labels.
The table/nested HTML templates are excluded from training; semantic label
vocabulary is shared. Novel-wording phrases are excluded from training. These
splits test generated layouts and phrase shifts, not unseen real websites.
Candidate selection uses visibility, enablement, sensitivity, roles and lexical
overlap. It caps candidates at 40; supervised reports measure candidate recall.
Important weakness: action and target role are strongly coupled in v1. A lexical
and role baseline can exploit this shortcut. Training improvements on v1 alone
are inadequate evidence of browser intelligence. Multi-step goals, history,
ordinals, negation, distractors, task interruption, dynamic workflows, extraction,
and unsupported-action detection still need harder datasets and evaluations.
`execution_verified=false` means the generated labels were not individually
executed before training. Browser evaluation separately executes sampled fixtures
and checks independent page-side outcomes; it does not silently relabel the full
training set as verified. No unverified teacher-generated data has been used.
Checkpoint selection uses validation joint accuracy, taking the earliest epoch
on a tie. Temperature scaling also uses validation only. Calibrated marginals are
not reliable under the observed wording distribution shift. The inference
threshold of 0.85 is a provisional research setting, not an established safe gate.
These evaluation splits have now been inspected for architecture development.
A future final claim requires an additional untouched audit set. Generic task
success, production security, and real-site generalization remain unproven.
|