| # Synthetic v1 schema and limitations |
|
|
| Each JSONL row has schema_version, source, license, split, template, seed, |
| sample_id, goal, elements, action, target, argument, execution_verified and |
| content_hash. Element fields are ref, role, name, path, visible, enabled, sensitive. |
| Target is a zero-based index into the row's elements; action is a supported DSL |
| opcode. Arguments are synthetic values and never production secrets. |
|
|
| Seeds 101/202/303/404 generate training/validation/test/novel-wording data. |
| Training has 2,400 rows and each evaluation split has 480. There are no imported |
| third-party records. `CC0-1.0` describes this generated fixture data, not any future |
| third-party dataset. Model-visible rows omit oracle metadata and target labels. |
|
|
| The table/nested HTML templates are excluded from training; semantic label |
| vocabulary is shared. Novel-wording phrases are excluded from training. These |
| splits test generated layouts and phrase shifts, not unseen real websites. |
| Candidate selection uses visibility, enablement, sensitivity, roles and lexical |
| overlap. It caps candidates at 40; supervised reports measure candidate recall. |
|
|
| Important weakness: action and target role are strongly coupled in v1. A lexical |
| and role baseline can exploit this shortcut. Training improvements on v1 alone |
| are inadequate evidence of browser intelligence. Multi-step goals, history, |
| ordinals, negation, distractors, task interruption, dynamic workflows, extraction, |
| and unsupported-action detection still need harder datasets and evaluations. |
|
|
| `execution_verified=false` means the generated labels were not individually |
| executed before training. Browser evaluation separately executes sampled fixtures |
| and checks independent page-side outcomes; it does not silently relabel the full |
| training set as verified. No unverified teacher-generated data has been used. |
|
|
| Checkpoint selection uses validation joint accuracy, taking the earliest epoch |
| on a tie. Temperature scaling also uses validation only. Calibrated marginals are |
| not reliable under the observed wording distribution shift. The inference |
| threshold of 0.85 is a provisional research setting, not an established safe gate. |
|
|
| These evaluation splits have now been inspected for architecture development. |
| A future final claim requires an additional untouched audit set. Generic task |
| success, production security, and real-site generalization remain unproven. |
|
|
