src/miniweb/evaluation/README.md

src/miniweb/evaluation/

Runs browser agents on the annotated MiniWeb tasks and grades them. The harness is browser-use. Grading (LLM macro judge by default, per-task verifier alongside) is described in docs/VERIFIER_DESIGN.md.

Use the conda env: ~/.conda/envs/miniweb/bin/python (written python below). Gemini judge calls need Vertex credentials in .env.

Files

File	Role
run_agent_verify.py	One agent on one task: start a MiniWeb server, run the agent, collect the trajectory, grade it, write the artifacts.
run_study.py	A study from one config: N models x M tasks x repeats, with parallel workers, resume and a summary.
agents.py	AgentRunner protocol, BrowserUseAgent, MockAgent (no LLM, no browser), ChatLLM (routes the agent through helpers.llm), build_agent(model) factory and MODEL_ALIASES.
macro_judge.py	The LLM macro judge (default grader); also a CLI to judge a whole study.
verifiers.py, evidence_checks.py	The deterministic verifier: verify_task and the check types.
trajectory.py	merge_server_log (adds the server request log to the recorder stream), filter_log_to_window, extract_final_reasoning.
server.py	Start, wait for and stop a MiniWeb server on a port.
xray.py	Web viewer for result folders.
generate_fixtures.py	Lists the real files of the simulated file system (src/miniweb/filesystem/) that uploads use.
configs/	Study configs. qwen35_4b_all_x3.json is the current reference config.
results/	Run outputs (git-ignored).

One task

python src/miniweb/evaluation/run_agent_verify.py --task-id crm_bf9346 --model mock --grader verifier   # pipeline smoke test, no LLM
python src/miniweb/evaluation/run_agent_verify.py --task-id banking_357033 --model gemini-flash --obs visual
python src/miniweb/evaluation/run_agent_verify.py --task-id qa-knowledge_471ddc --no-headless      # watch the browser

Flag	Default	Meaning
--task-id	(required)	<task_id> or <annotator>/<task_id>
--model	gemini-flash	model id, an alias from agents.MODEL_ALIASES, or mock
--obs	axtree	visual = screenshots (use_vision=True); axtree / html = browser-use's text DOM
--native-llm	off	use browser-use's provider-native LLM class instead of ChatLLM
--start-from	recorded	recorded = first page of the human recording; starting_url = the task's field
--grader	judge	judge = macro judge decides, verifier reported; verifier = verifier only
--port	8099	server port
--max-steps, --timeout	50, 300	step and wall-clock limits
--no-headless	off	show the browser
--out	src/miniweb/evaluation/results/agentverify_<task>_<model>_<time>/	artifact folder

What a run does:

1. Starts a fresh server. Tasks with authenticate_by_form start it with MINIWEB_NO_AUTOLOGIN=1 so the agent begins logged out.
2. Picks the start page. A single-site task starts where the human recording started. A multi-site task starts at the portal home (/), and the prompt ends with the names and URLs of the apps involved (app_directory; recorded as prompt_extra).
3. Runs the agent, then pulls /_admin/record (recorder actions and observations) and /_admin/log (server requests) into one trajectory.
4. Grades with the verifier and, with --grader judge, the macro judge.
5. Writes result.json, trajectory.json, server_log.json, verify_report.json, judge_report.json, plus browser-use's history.json, screenshots/ and conversations/.

A study

python src/miniweb/evaluation/run_study.py --config src/miniweb/evaluation/configs/qwen35_4b_all_x3.json --dry-run   # print the plan
python src/miniweb/evaluation/run_study.py --config src/miniweb/evaluation/configs/qwen35_4b_all_x3.json --workers 8
python src/miniweb/evaluation/run_study.py --summarize-only src/miniweb/evaluation/results/qwen35_4b_all_x3

Config (YAML or JSON):

name: qwen35_4b_all_x3
out: src/miniweb/evaluation/results/qwen35_4b_all_x3
tasks: all                  # or ["<annotator>/banking_357033", "site:dating", ...]
exclude: []
resume: true                # skip episodes that already have result.json
repeats: 3
workers: 12                 # worker i uses port episode.port + i and every N-th episode
episode:
  max_steps: 30
  timeout: 1800
  obs: visual
  start_from: recorded
  headless: true
  port: 8200
  browser: ~/.cache/ms-playwright/chromium-1117/chrome-linux/chrome
  # grader: judge           # or verifier
models:
  - label: qwen3.5-4b
    provider: ollama        # openai | anthropic | gemini | ollama | huggingface | openai-compatible
    model: qwen3.5:4b-t2
    host: http://127.0.0.1:11435
    hosts: [http://127.0.0.1:11435, http://127.0.0.1:11436]   # optional; worker i uses hosts[i % n]

Episode defaults: max_steps 30, timeout 600, obs visual, start_from recorded, headless true, port 8099. Model keys: temperature, max_tokens, base_url, api_key_env. API keys come from OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY / GEMINI_API_KEY, HF_TOKEN.

Each episode lands in <out>/<label>__<annotator>-<task_id>[__r<repeat>]/. Workers log to <out>/worker_<i>.log. The summary (overall pass rate, by chain length, by macro) is printed and saved to <out>/study_summary.json.

To re-judge a finished study, see macro_judge.py --results in VERIFIER_DESIGN.md.

Harness behaviour that affects results

• Offline. Chromium runs with --host-resolver-rules=MAP * ~NOTFOUND , EXCLUDE localhost; only the local server is reachable.
• Pinned browser. Set episode.browser or MINIWEB_BROWSER. Playwright's Chrome for Testing 149 never finishes loading http pages headless on the lab machine; Chromium build 1117 works.
• Clipboard. Headless pages are never focused, so site Copy buttons failed. BrowserUseAgent grants clipboard permissions and emulates focus on every step.
• Uploads. browser-use's upload_file shortcut is removed. The agent clicks the file input and picks the file in MiniWeb's in-page file picker, as a person would.
• Logs. The server request log is stamped in UTC and filtered to the episode's time window.

Results from before these dates are not comparable for the affected tasks:

Date	Change	Affected tasks
2026-09-27	recorder kept observations of pages over 64 KB	checks on large pages
2026-09-27	clipboard fix	copy tasks
2026-09-28	Leaflet served locally (maps were blank offline)	map-services pan/zoom, markers, routes
2026-09-28	uploads through the file picker	the 25 upload tasks

Known gap: the recorder cannot tell a site Copy button from an agent writing the clipboard itself with a script. Value checks and the judge catch most of these.

Latest reference run

src/miniweb/evaluation/results/qwen35_4b_all_x3: qwen3.5:4b (ollama), browser-use, visual, 376 tasks x 3, re-graded 2026-09-27 (judge v3, macro_judge_flash_v3.jsonl):

Grader	pass@1	pass@3	all 3
macro judge (gemini-3.5-flash)	41.8%	58.0%	26.9%
verifier	41.3%	54.8%	26.9%

Episode-level agreement between the two: 92.4%. The run predates the 2026-09-28 map and upload changes.

X-Ray viewer

python src/miniweb/evaluation/xray.py --port 8125

Browses src/miniweb/evaluation/results/<run>/<episode>/: instruction, expected vs agent answer, pass/fail, the verifier check tree from verify_report.json, step screenshots with the actions taken, any note.md, and a feedback box saved to feedback.md in the episode folder.

Other harnesses

• src/miniwebagent/evaluate.py is the WebMix agent's MiniWeb evaluator (vLLM-served LoRA arms on the same browser-use harness). See src/miniwebagent/README.md.
• src/miniweb/browsergym/ runs the tasks under BrowserGym + AgentLab, verifier-only. See docs/BROWSERGYM_AGENTLAB_MIGRATION.md.