Spaces:
Running
Running
| # Reproducing the paper | |
| `python scripts/reproduce.py` is the one entry point; `scripts/paper.yaml` is its registry. [REPRODUCE.md](../REPRODUCE.md) | |
| is the walkthrough. | |
| | Command | What it does | | |
| |---|---| | |
| | `python scripts/reproduce.py status` | Checks every run, RQ1 run and adapter the paper reads: present, and complete for its benchmark. | | |
| | `python scripts/reproduce.py table` | Task success of every arm on every benchmark, mean over the runs present, next to the paper's number. | | |
| | `python scripts/reproduce.py show <arm>` | Prints the model-server line and the run + grading commands behind every run of an arm. Arms: `base`, `pooled`, `planner`, `planner_pooled` (MiniWebAgent), `taskcentric` (the task-centric executor of Tab. 3), `agent` (specialists), `nodone` (stop-early control); `rq1` for the ten RQ1 agents. | | |
| | `python scripts/reproduce.py run <arm> --bench <b> --seed <k> [--execute]` | One run (benchmarks: `single`, `heldout`, `wa`, `wv`, `om2w`). Prints unless `--execute`. | | |
| | `python scripts/reproduce.py train <adapter> [--execute]` | Builds the adapter's training folder from its id list and prints the `torchrun` line with the paper's recipe. | | |
| | `python scripts/reproduce.py figures [--only <name>]` | Regenerates every figure and table into `docs/figures/` and copies them into the paper source (needs the figure scripts, which are not in the public repository). | | |
| | `python scripts/skill_report.py <run> [--by primitive] [--failures N]` | Per-skill results of any run: task success, instances passed per family or primitive (judge and verifiers), failed instances with the judge's reason. | | |
| ## What lives where | |
| - **Which runs the paper reads.** `paper.yaml`: each arm's `runs:` (one list per benchmark, one run per seed; the paper | |
| reports the mean over the list) and `paper:` (the reported numbers). The figure scripts' configs list the same runs. | |
| - **How each run is made.** `paper.yaml`: the model server, the agent configuration per benchmark kind, the arms, the | |
| RQ1 agents, the adapters and their training recipe, the data pipeline, and the protocol deviations as they ran. | |
| - **The training data.** `scripts/pools/`: `pool3k_plain_ids.txt` (the 3,128 generated trajectories), | |
| `pool3k_dirs.txt` (the 4,614 replayed episodes of the 27.6K-step set: plain, chains, recoveries). | |
| `data/webmix/replay_v4` holds them, so an adapter is trained on a folder of symlinks to its list | |
| (`scripts/reproduce.py train` builds it). The task-centric executor trains on `data/webmix/taskgen/rows_27k`. | |
| - **Cluster wrappers.** `src/miniwebagent/scripts/cluster/`: `train.sh`, `eval.sh` (MiniWeb), `live.sh` (WebVoyager, | |
| Online-Mind2Web), `ownloop.sh` (RQ1 own-loop agents), `retry_timeouts.py` (sets aside live-web episodes ended by | |
| model-server timeouts, for a rerun), `opencua/` (OpenCUA-7B tokenizer + vLLM plugin). Set `WEBMIX_ROOT` (see | |
| `cluster/env.sh`). The cluster holds no API keys: MiniWeb runs are re-graded with `miniwebagent.evaluate --regrade` and | |
| live-web runs judged with `miniwebagent.om2w_judge` where the keys are. | |
| - **WebArena-Lite.** `src/miniwebagent/scripts/webarena/ready.sh` resets the four instances and waits until every site serves | |
| (`WA_RESET_TOKEN`, `WA_HOST`). | |
| ## Environments | |
| `scripts/envs/` pins the three environments the paper's runs used (`pip list --format=freeze`): | |
| - `miniweb.txt` (Python 3.11): the app, annotation tool, MiniWebGen, the browser-use harness, grading, figures. | |
| Playwright 1.44 with Chromium build 1117. | |
| - `webmix.txt` (Python 3.12): vLLM 0.30 serving (`src/miniwebagent/scripts/serve.sh`) and LoRA training (torch 2.13, transformers | |
| 5.17, peft 0.21). | |
| - `fara.txt` (Python 3.11): Fara1.5-4B's released loop (`microsoft/fara` at a675d6d). | |
| The repo's `requirements.txt` (`environment.yml`) installs current versions for using MiniWeb; these files are what produced the numbers. Each `<name>.yml` creates its environment from `<name>.txt` and names the systems it ran on (REPRODUCE.md). | |
| ## Grading | |
| - MiniWeb: the task verifiers (`src/miniweb/evaluation/verifiers.py`); their answer checks fall back to Gemini 3.5 Flash-Lite | |
| (`miniwebagent.evaluate.QA_MODEL`) when the regex, exact and precise matchers cannot decide. RQ1 also uses the VLM judge | |
| (`python -m miniweb.evaluation.judge_runs`, Gemini 3.5 Flash). | |
| - WebArena-Lite: the benchmark's evaluators; the fuzzy answer matches of 43 tasks use Gemini 3.5 Flash-Lite instead of | |
| GPT-4o (`miniwebagent.wa --judge`). | |
| - WebVoyager: its GPT-4o judge; Online-Mind2Web: WebJudge (o4-mini) (`python -m miniwebagent.om2w_judge`). | |