Synthetic v1 schema and limitations
Each JSONL row has schema_version, source, license, split, template, seed, sample_id, goal, elements, action, target, argument, execution_verified and content_hash. Element fields are ref, role, name, path, visible, enabled, sensitive. Target is a zero-based index into the row's elements; action is a supported DSL opcode. Arguments are synthetic values and never production secrets.
Seeds 101/202/303/404 generate training/validation/test/novel-wording data.
Training has 2,400 rows and each evaluation split has 480. There are no imported
third-party records. CC0-1.0 describes this generated fixture data, not any future
third-party dataset. Model-visible rows omit oracle metadata and target labels.
The table/nested HTML templates are excluded from training; semantic label vocabulary is shared. Novel-wording phrases are excluded from training. These splits test generated layouts and phrase shifts, not unseen real websites. Candidate selection uses visibility, enablement, sensitivity, roles and lexical overlap. It caps candidates at 40; supervised reports measure candidate recall.
Important weakness: action and target role are strongly coupled in v1. A lexical and role baseline can exploit this shortcut. Training improvements on v1 alone are inadequate evidence of browser intelligence. Multi-step goals, history, ordinals, negation, distractors, task interruption, dynamic workflows, extraction, and unsupported-action detection still need harder datasets and evaluations.
execution_verified=false means the generated labels were not individually
executed before training. Browser evaluation separately executes sampled fixtures
and checks independent page-side outcomes; it does not silently relabel the full
training set as verified. No unverified teacher-generated data has been used.
Checkpoint selection uses validation joint accuracy, taking the earliest epoch on a tie. Temperature scaling also uses validation only. Calibrated marginals are not reliable under the observed wording distribution shift. The inference threshold of 0.85 is a provisional research setting, not an established safe gate.
These evaluation splits have now been inspected for architecture development. A future final claim requires an additional untouched audit set. Generic task success, production security, and real-site generalization remain unproven.