File size: 5,538 Bytes
795f737 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | # Research status
Active engineering goal; not a completed or deployable browser model.
The complete user requirements are preserved in requirements.txt.
## Environment audit
2026-09-27: Windows host, AMD Ryzen 7 7435HS, 8 physical cores / 16 logical processors,
approximately 15.8 GiB total RAM and 0.7 GiB free at initial inspection. Python 3.12
and 3.14 are installed. Python 3.12 has Playwright and psutil, but no PyTorch,
NumPy, ONNX Runtime, or Transformers. Hugging Face CLI authentication verified.
No repository implementation existed in the task workspace.
Target remains Linux, two EPYC 9354P vCPUs, approximately 5 GiB available RAM.
Local performance must not be presented as target hardware performance.
## Initial experiment order
1. DOM observation, ephemeral references, validated actions, trajectory capture.
2. Randomized local browser environments and independent outcome oracles.
3. Learned action classifier and candidate pointer compared with lexical baseline.
4. Held-out template evaluation, confidence calibration, recovery and memory.
5. Quantization and architecture comparisons with identical evaluation data.
These are planned experiments, not five completed iterations. Select the final
architecture only after measurements. Conventional LLM necessity is unresolved.
Local small-policy training has now been executed. Arbitrary-site task success,
visual fallback and target VPS latency remain UNTESTED. No checkpoint is a
production champion.
## Implemented foundation
Strict action wire parser for all 18 requested action types; task authority tickets;
DOM/semantic observation including frames and open shadow roots; actual Chromium
executor; stale-node/document and loop rejection; host permission and completion
callbacks; metadata-only SQLite recorder with HMAC identifiers. Tests use fixture
actions and fixture completion checks, not a learned policy.
The first measured iteration reduced 2,000-element observation median from
1134.41 ms to 254.24 ms on the local Windows host, with a small-page regression.
See ITERATIONS.md for exact scope, regressions, and report filenames. General
completion verification, learned planning, experience retrieval, full data
redaction and target-VPS validation remain open.
Memory availability recovered to about 5.2 GiB later in the session, so the initial
low-memory reading is not a continuing blocker to a small local training run.
## Learned baseline experiments
Trained four CPU checkpoints on 2,400 synthetic samples: mean encoder with lexical
features (128,900 parameters), mean without lexical features (128,900), GRU
(153,860), and Transformer (163,908). Each has 480-row validation, familiar-wording
test and novel-wording evaluations. Familiar-wording prediction accuracy is near
100%, but novel-wording joint accuracy is only 55.83%, 50.63%, 35.83%, and 42.71%
respectively. These are deliberately tiny engineering baselines, not the requested
50M–1.5B model-range study or a final model selection.
The mean policy executed 120/120 familiar-wording browser fixtures successfully,
and 74/120 novel-wording fixtures with 19 abstentions. Synthetic v1 has shortcuts;
its high familiar-wording score is not evidence of general browser intelligence.
A pinned 28 MB Mind2Web training shard was downloaded to the intermediate work
directory. Offline diagnostic: 9 tasks, 3 websites, 49 steps, 46 scorable. The
synthetic-trained mean model achieved 1/46 joint action/target matches; candidate
recall was 22/46. No downloaded HTML was executed and no real data used in training.
This is a small training-shard diagnostic, not an official benchmark result.
Immutable checkpoint registration, integrity verification, transactional promotion
and rollback have been implemented and tested. Production gates require full
target-hardware and task/recovery/security evidence absent from current results.
Next: measure quantization and model costs, then prioritize real-data candidate
coverage, semantic priors, history/task decomposition and harder synthetic tasks.
The quantization and isolated model comparisons are now complete. Linear INT8
reduced the mean checkpoint from 516,312 to 459,286 bytes but regressed p95 policy
latency from 2.847 to 4.170 ms. It is not selected. The deterministic comparator
solved 120/120 novel-wording browser fixtures, exposing the synthetic shortcut.
Five measured experiments are documented in ITERATIONS.md; the design is still
immature. All 22 tests pass. Four candidates are registered, champion is null.
## Pretrained retrieval diagnostic
A sixth experiment found 11/46 real-data targets excluded by the original role
filter. Broad BM25 did not improve top-40 recall (20/46 versus 22/46). A pinned
22.7M-parameter MiniLM cross-encoder reranking 80 candidates achieved 6/46 top-one
target matches with prior-action history, at 1381.79 ms median rerank cost. The
runtime defaults are unchanged because the gain does not justify this cost yet.
All 24 tests pass, including current/future-label exclusion from history. See
PRETRAINED.md and reports/retrieval-audit.json for reproducibility and scope.
Pinned Qwen2.5-0.5B-Instruct was tested locally in FP32 on eight offline steps.
It produced zero valid DSL actions; median generation time was about 6.81 seconds
with approximately 2.5 GB observed Python RSS. Some regression testing overlapped,
so this is a smoke test rather than isolated timing evidence. No prediction was
executed. Constrained decoding and optimized/quantized backends remain pending.
|