docs/ITERATIONS.md

# Measured iterations

## 1 — Compute structural paths once during traversal

Hypothesis: repeated sibling enumeration makes observation quadratic in the
number of siblings. Computing paths during traversal should lower large-page
latency without changing model-visible state.

Alternatives considered: omit structural paths (faster but weakens cache matching
and stale-target checks); cache snapshots across mutations (potentially larger
gain but requires reliable invalidation); compute paths once (small implementation
cost, linear extra storage, no intended semantic change). Chose the third.

Change: DOM traversal returns paths alongside retained node handles. Single-target
revalidation still recomputes its path. The original path recomputation remains
available through `--path-mode naive` for future ablations.

Measured on Windows Ryzen 7 7435HS, unrestricted CPU, 20 observations/clicks per
page size. Reports: `reports/runtime-baseline.json` and
`reports/runtime-paths-v1.json`. These are runtime fixture microbenchmarks, not
learned-policy task success or EPYC VPS results.

| Elements | Before observation median | After observation median | Before click median | After click median |
|---|---:|---:|---:|---:|
| 40 | 15.01 ms | 28.97 ms | 64.33 ms | 57.04 ms |
| 200 | 41.18 ms | 43.16 ms | 55.38 ms | 55.33 ms |
| 2000 | 1134.41 ms | 254.24 ms | 70.84 ms | 63.70 ms |

Large-page observation improved 4.46x in these runs. Small-page observation
regressed by 13.96 ms; added handle processing and host variability are possible
contributors, not established causes. No claim of statistical significance.
Observed process-tree RSS peaks were 600,313,856 and 581,799,936 bytes respectively;
these are 10 ms samples and sum shared pages, not exact unique peak memory.

Regression analysis: the initial change broke target revalidation because
Playwright supplied null rather than undefined for an omitted argument. Tests
caught it before accepting the experiment. The null handling was fixed; all
15 then-existing tests passed before the after benchmark. An additional equality
test now checks optimized versus recomputed paths, including frames and shadow DOM.

Decision: provisionally retain the change for the substantial large-page gain;
the absolute small-page penalty is below the one-second action target but merits
further profiling. Do not declare the architecture mature after this iteration.

Next bottleneck: 2,000 elements still serialize to about 271 KB. Candidate reduction,
compact model-facing fields and measured tokenizer costs are next. Path output
alone is not adequate context reduction, and raw DOM collection remains unbounded.

## Common evidence for iterations 2–5

All models use the same 2,400 generated training rows, 480 validation rows, seed
1729, 16 epochs, AdamW, and two PyTorch CPU threads. Checkpoints are selected on
validation joint accuracy, earliest epoch on a tie. Each also has 480-row
familiar-wording and novel-wording predictions. The inference comparisons run in
separate sequential processes. This is one seed, not a statistical study.

| Configuration | Parameters before quantization | Familiar joint | Novel joint | p95 policy time | Observed Python RSS | Artifact bytes |
|---|---:|---:|---:|---:|---:|---:|
| Mean + lexical FP32 | 128,900 | 100% | 55.83% | 2.847 ms | 275,996,672 | 516,312 |
| Mean, lexical features zeroed | 128,900 | 99.79% | 50.63% | 2.787 ms | 276,901,888 | 516,312 |
| GRU + lexical FP32 | 153,860 | 100% | 35.83% | 6.874 ms | 297,365,504 | 616,488 |
| Transformer + lexical FP32 | 163,908 | 100% | 42.71% | 5.411 ms | 373,432,320 | 657,592 |
| Mean + lexical, Linear INT8 | 128,900 | 100% | 55.62% | 4.170 ms | 295,170,048 | 459,286 |

Policy time includes feature encoding and neural inference, not browser work.
RSS includes Python, training-library imports and evaluation tensors; it is not
browser-inclusive service memory or exact peak RSS. These are Windows Ryzen
measurements, not EPYC VPS results. Reports are `reports/bench-*.json`;
`reports/model-comparison.json` contains the exact diagnostic scoring formula.

Actual 120-fixture novel-wording Chromium evaluations, with confidence abstention:

| Policy | Successful | Abstained | Median policy latency |
|---|---:|---:|---:|
| Mean + lexical | 74/120 | 19/120 | 2.642 ms |
| Mean without lexical | 70/120 | 21/120 | 2.590 ms |
| GRU | 47/120 | 48/120 | 5.643 ms |
| Transformer | 59/120 | 18/120 | 4.209 ms |
| Deterministic lexical/role comparator | 120/120 | 0/120 | 0.654 ms |

Reports are `reports/policy-browser-novel-v*.json` and
`reports/baseline-browser-novel.json`. Browser fixtures use independent outcome
checks; model inputs do not include expected indices. The comparator's perfect
score exposes a dataset shortcut. It is a baseline, not the final architecture.

## 2 — Remove lexical overlap features

Hypothesis: learned embeddings might replace deterministic lexical overlap,
simplifying the model inputs. Alternative: retain lexical features as a cheap
generalization aid; introduce a pretrained semantic encoder at higher cost.
Expected gain was architectural simplicity, with an uncertain generalization
risk and negligible RAM savings. The ablation zeroes five lexical features; it
does not reduce the parameter count, so it isolates their information contribution.

Before/after: novel joint accuracy fell from 55.83% to 50.63%; browser success fell
from 74/120 to 70/120; p95 policy time changed from 2.847 to 2.787 ms. That small
latency difference is not enough evidence of a real speed benefit. Unit checks
cover candidate filtering and pointer permutation equivariance.

Decision: reject removing lexical features. Next bottleneck: general semantics,
not their small compute cost.

## 3 — Replace mean encoding with a GRU

Hypothesis: word order and recurrent context may improve grounding. Alternatives
were mean pooling or self-attention. Complexity and serial CPU work increase;
expected accuracy gain was uncertain, and synthetic overfitting was a risk.
Changed both goal and candidate encoders, preserving classifier/pointer heads.

Before/after: novel joint accuracy fell from 55.83% to 35.83%; novel browser success
fell from 74/120 to 47/120; p95 policy time rose from 2.847 to 6.874 ms. Parameters
rose by 24,960 and observed RSS by about 21 MB. Local training took 149.73 seconds
versus 12.39 for mean pooling; host load was not controlled for training timing.

Decision: reject this GRU configuration. Regression: substantially worse
distribution-shift behavior despite perfect familiar-wording accuracy. Next
bottleneck: data diversity and semantic priors; recurrent architecture alone did
not solve it. Padding inefficiency is also a possible optimization target.

## 4 — Replace mean encoding with a tiny Transformer

Hypothesis: attention could model phrase relationships better than pooling or
recurrence. Estimated tradeoff: more compute and activation memory, uncertain
accuracy benefit. Changed to one 64-wide, four-head encoder layer with positions;
the action/pointer structure stayed unchanged.

Before/after: novel joint accuracy fell from 55.83% to 42.71%; browser success fell
from 74/120 to 59/120; p95 policy time rose from 2.847 to 5.411 ms. Observed Python
RSS increased from 276 MB to 373 MB. Familiar-wording accuracy remained 100%.
All encoder variants pass finite-output/padding tests.

Decision: reject this scratch-trained Transformer configuration. This does not
reject pretrained Transformers, larger models, or attention generally. Next
bottleneck: pretraining/data/task representation rather than choosing an
architecture by popularity.

## 5 — Dynamic INT8 Linear quantization

Hypothesis: quantized matrix operations might reduce disk/RAM and latency.
Alternatives are FP32, embedding quantization, or ONNX/other CPU runtimes. Expected
benefit was uncertain for such small layers; dispatch overhead can dominate.
Changed Linear layers only; embeddings remain FP32. The serialized INT8 state
was reloaded with `weights_only=True` and re-evaluated.

Before/after: artifact size fell from 516,312 to 459,286 bytes, but p95 policy time
rose from 2.847 to 4.170 ms. Neural-forward median rose from 0.491 to 1.220 ms.
Novel joint accuracy fell from 55.83% to 55.62%. Familiar-wording Chromium success
was 120/120 for both FP32 and INT8. Observed Python RSS was higher for INT8, so the
smaller file is not evidence of runtime RAM reduction.

Decision: retain the export and report, reject INT8 as the preferred runtime for
this model. Next bottleneck: encoding/observation costs and model quality; revisit
quantization for larger models and alternative kernels. PyTorch emitted API
deprecation warnings, recorded in the benchmark log.

## Maturity decision and real-data diagnostic

These five measured engineering experiments **do not establish a mature design**.
The synthetic suite is inadequate as a generalization acceptance test. A read-only
Mind2Web diagnostic over nine tasks from three websites produced only 1/46 correct
scorable action/target pairs and 22/46 candidate recall. Three steps had no usable
positive label and are reported separately. This is a pinned training shard, not
the official benchmark, and no source HTML was executed. See
`reports/mind2web-smoke-v000.json` for provenance and preprocessing limitations.

Next cycle must improve real candidate coverage, preserve task/history semantics,
add multi-step/hard-negative data, investigate pretrained small encoders/language
models, and reserve a fresh untouched audit set. A simpler deterministic baseline
currently beats every learned model on the easy fixtures. No superiority or
production-readiness claim is justified. All 22 regression tests pass; four model
versions are registered and the production champion remains null.

## 6 — Broad lexical retrieval and pretrained semantic reranking

Hypothesis: role filtering and lack of semantic priors contribute to the real-data
failure. Alternatives: expand the original role filter, use BM25 over the annotated
candidate pool, or rerank that pool using a pretrained cross-encoder. The last
option costs substantially more CPU; all are measured before changing runtime
defaults. Labels remain evaluator-only and previous-action history excludes the
current action. Two focused tests verify these boundaries.

Audit: original role eligibility excluded 11/46 targets even without a top-k
limit (10 generic, one tab). Original recall was 22/46 at 40 and 30/46 at 80.
Broad BM25 recall was 20/46 at 40, 26/46 at 80 and 37/46 at 200. Removing the role
exclusion therefore did not automatically improve recall at the relevant budget.

Change: 22,713,601-parameter pretrained MiniLM cross-encoder reranks 80 BM25
candidates with either the full goal or goal plus up to four prior human actions.
Top-one target accuracy was 4/46 for BM25, 5/46 for semantic goal only, and 6/46
with history. Goal-only semantic top-40 recall was 25/46, versus 20/46 for BM25
top-40; the semantic path initially sees 80 candidates. History reduced semantic
top-40 recall to 24/46. Median rerank time over both settings was 1381.79 ms.

Regression/decision: retain diagnostic tooling, do not replace production
retrieval with this configuration. The small top-one gain is insufficient for
the CPU cost and target coverage remains poor. No checkpoint was trained on this
source. This is a 46-step training-shard diagnostic, not statistical or official
benchmark evidence. Report: `reports/retrieval-audit.json`.

Next bottleneck: goal-to-next-step decomposition, history representation, better
accessible names and supervised real-world grounding. Naively appending previous
actions to a passage-ranking query is not a learned planner.

## 7 — Qwen 0.5B autoregressive smoke baseline

Hypothesis: a pretrained instruction model may provide task interpretation missing
from the synthetic-trained policies. Change: pinned Qwen2.5-0.5B-Instruct in local
FP32 CPU inference, two torch threads, 20 BM25 candidates, up to four prior human
actions, greedy generation capped at 48 tokens. This is an offline eight-step
smoke test; outputs never reach the browser executor.

Before: synthetic-trained policy matched 1/46 action/target pairs in the larger
diagnostic. After: Qwen returned zero valid DSL actions and zero correct pairs in
the first eight steps; only five of those targets were in its candidate pool.
Different sample counts prohibit a direct quality ranking. Input lengths were
656–887 tokens, output lengths 5–9 tokens. Median observed generation latency was
about 6.81 seconds, and observed Python RSS was approximately 2.5 GB. Regression
tests ran concurrently during part of this smoke test, so latency is indicative,
not an isolated comparative benchmark. No target-VPS claim is made.

Decision: reject this unconstrained configuration as a fallback. Next bottleneck:
grammar-constrained decoding and better prompt/task representation, followed by
an isolated rerun and quantized CPU backend comparison. Invalid outputs were
rejected rather than heuristically repaired or executed. Report:
`reports/qwen-baseline.json`. This result does not establish that conventional
LLMs are unnecessary; it establishes that this particular naive integration fails.