scripts/README.md

Reproducing the paper

python scripts/reproduce.py is the one entry point; scripts/paper.yaml is its registry. REPRODUCE.md is the walkthrough.

Command	What it does
python scripts/reproduce.py status	Checks every run, RQ1 run and adapter the paper reads: present, and complete for its benchmark.
python scripts/reproduce.py table	Task success of every arm on every benchmark, mean over the runs present, next to the paper's number.
python scripts/reproduce.py show <arm>	Prints the model-server line and the run + grading commands behind every run of an arm. Arms: base, pooled, planner, planner_pooled (MiniWebAgent), taskcentric (the task-centric executor of Tab. 3), agent (specialists), nodone (stop-early control); rq1 for the ten RQ1 agents.
python scripts/reproduce.py run <arm> --bench <b> --seed <k> [--execute]	One run (benchmarks: single, heldout, wa, wv, om2w). Prints unless --execute.
python scripts/reproduce.py train <adapter> [--execute]	Builds the adapter's training folder from its id list and prints the torchrun line with the paper's recipe.
python scripts/reproduce.py figures [--only <name>]	Regenerates every figure and table into docs/figures/ and copies them into the paper source (needs the figure scripts, which are not in the public repository).
python scripts/skill_report.py <run> [--by primitive] [--failures N]	Per-skill results of any run: task success, instances passed per family or primitive (judge and verifiers), failed instances with the judge's reason.

What lives where

• Which runs the paper reads. paper.yaml: each arm's runs: (one list per benchmark, one run per seed; the paper reports the mean over the list) and paper: (the reported numbers). The figure scripts' configs list the same runs.
• How each run is made. paper.yaml: the model server, the agent configuration per benchmark kind, the arms, the RQ1 agents, the adapters and their training recipe, the data pipeline, and the protocol deviations as they ran.
• The training data. scripts/pools/: pool3k_plain_ids.txt (the 3,128 generated trajectories), pool3k_dirs.txt (the 4,614 replayed episodes of the 27.6K-step set: plain, chains, recoveries). data/webmix/replay_v4 holds them, so an adapter is trained on a folder of symlinks to its list (scripts/reproduce.py train builds it). The task-centric executor trains on data/webmix/taskgen/rows_27k.
• Cluster wrappers. src/miniwebagent/scripts/cluster/: train.sh, eval.sh (MiniWeb), live.sh (WebVoyager, Online-Mind2Web), ownloop.sh (RQ1 own-loop agents), retry_timeouts.py (sets aside live-web episodes ended by model-server timeouts, for a rerun), opencua/ (OpenCUA-7B tokenizer + vLLM plugin). Set WEBMIX_ROOT (see cluster/env.sh). The cluster holds no API keys: MiniWeb runs are re-graded with miniwebagent.evaluate --regrade and live-web runs judged with miniwebagent.om2w_judge where the keys are.
• WebArena-Lite. src/miniwebagent/scripts/webarena/ready.sh resets the four instances and waits until every site serves (WA_RESET_TOKEN, WA_HOST).

Environments

scripts/envs/ pins the three environments the paper's runs used (pip list --format=freeze):

• miniweb.txt (Python 3.11): the app, annotation tool, MiniWebGen, the browser-use harness, grading, figures. Playwright 1.44 with Chromium build 1117.
• webmix.txt (Python 3.12): vLLM 0.30 serving (src/miniwebagent/scripts/serve.sh) and LoRA training (torch 2.13, transformers 5.17, peft 0.21).
• fara.txt (Python 3.11): Fara1.5-4B's released loop (microsoft/fara at a675d6d).

The repo's requirements.txt (environment.yml) installs current versions for using MiniWeb; these files are what produced the numbers. Each <name>.yml creates its environment from <name>.txt and names the systems it ran on (REPRODUCE.md).

Grading

• MiniWeb: the task verifiers (src/miniweb/evaluation/verifiers.py); their answer checks fall back to Gemini 3.5 Flash-Lite (miniwebagent.evaluate.QA_MODEL) when the regex, exact and precise matchers cannot decide. RQ1 also uses the VLM judge (python -m miniweb.evaluation.judge_runs, Gemini 3.5 Flash).
• WebArena-Lite: the benchmark's evaluators; the fuzzy answer matches of 43 tasks use Gemini 3.5 Flash-Lite instead of GPT-4o (miniwebagent.wa --judge).
• WebVoyager: its GPT-4o judge; Online-Mind2Web: WebJudge (o4-mini) (python -m miniwebagent.om2w_judge).