| --- |
| license: apache-2.0 |
| tags: |
| - agent |
| - evaluation |
| - monitoring |
| --- |
| |
| # Squares Progress Agent |
|
|
| A small, offline-first agent that monitors public claims about packing n unit squares into the smallest square. It extracts claimed container sides from pages, validates geometry when coordinates are given, compares claims to a local bound table, and reports improvements, candidates, conflicts and errors. |
|
|
| This repository contains code only. There are no model weights. |
|
|
| The point of the project is the evaluation: which parts of the pipeline cause or prevent false "improvement" reports. |
|
|
| ## Read first |
|
|
| - The synthetic scenarios were written by the same author as the agent. They test the design on its own fixtures and are not an independent test. |
| - Real-page results (below) come from private local runs on private captures. The raw pages, source URLs and quote-bearing labels are withheld, so those tables cannot be reproduced from this repository. |
| - LLM extractor numbers (below) also come from private local runs. They are not produced by `make test`, were not re-run after a later code-hash change, and cannot be reproduced from this repository without a live model endpoint. |
| - The held-out sets are small, so the confidence intervals are wide. |
| - The agent has not discovered or improved any packing record. |
|
|
| ## Reproducible result (synthetic, `make test`) |
|
|
| Numbers below are from `results/synthetic_publish_check.txt` and `results/eval.json`, and are reproduced by `make test`. |
|
|
| | system | pass (23 scenarios) | false improvements | |
| |---|---|---| |
| | agent (rule extractor) | 22/23 | 0 | |
| | regex + same guards | 12/23 | 0 | |
| | plain regex baseline | 4/23 | 8 | |
| | agent without retry | 20/23 | 0 | |
|
|
| The one agent miss is a LaTeX-table scenario, which is a deliberate distribution-shift case. |
|
|
| `results/budget_curve.csv` is produced by a script that forces the same `max_steps` on every scenario. `s09_budget_tight` expects exhaustion at its native budget of 2, so the agent score of 21/23 (91%) at budget 8 is expected. |
|
|
| ## Local results (not reproducible from this repo) |
|
|
| Real-page and LLM numbers come from private local runs. They are recorded in this repository but cannot be reproduced from the publishable tree alone. |
|
|
| Real batch 1 is taken from `results/real_publish_check.txt` (scenarios real01–real15). Real batch 2 is taken from `results/batch2_eval.txt`. The LLM sample is taken from `results/llm_sample7_arms.txt` (and the narrative in `results/llm_summary.md`). `results/heldout_scored.json` and `results/batch2_eval.txt` record code hash `eac25bad096c9a9c`; the published code differs because scrubbing, and a later change that made the model id required, changed the hashed files. |
|
|
| | set | agent | regex + guards | plain regex | |
| |---|---|---|---| |
| | real batch 1 (15 pages) | 15/15 | 14/15 | 9/15 (6 false improvements) | |
| | real batch 2 (12 pages, one source type) | 12/12 | 12/12 | 10/12 (2 false improvements) | |
|
|
| Batch 2 cannot tell the rule extractor and regex+guards apart. An optional LLM extractor — a free-tier hosted model (arm name flint); endpoint settings are not published — was tried on 7 scenarios and passed 6, with no fallback to the rules. The rule extractor passed all 7, so there is no evidence the LLM helps. |
|
|
| ## What it does |
|
|
| search → fetch → extract → drop rejected or stale claims → validate coordinates → compare to bounds → report |
|
|
| - An improvement needs coordinates that validate. A lower number without coordinates is only a candidate. |
| - Exact rational packings are proofs. Float validation is not a proof, and near-bound float claims stay candidates. |
| - Hedged or retracted lines are skipped, which costs some recall. |
| - The LLM extractor proposes only (n, quote). The side value is parsed from a verbatim quote and goes through the same guards. |
|
|
| ## Run |
|
|
| make test # synthetic scenarios only, no network, no API key |
| |
| The real-page evaluation skips with a message when the private fixtures are missing. Live LLM calls need `SQ_LLM_API_KEY`, `SQ_LLM_BASE_URL` and `SQ_LLM_MODEL` in the environment. |
|
|
| ## Known limits |
|
|
| - Lists like `5,6` are ambiguous with comma decimals. |
| - No Kaggle CSV adapter; the column and angle conventions are unconfirmed. |
| - `bounds.json` covers n = 1..29, 68, 102, 299 from a published catalogue. For n = 102 and 299, community claims may be lower than the catalogue. |
|
|
| ## Next steps |
|
|
| Real search through official APIs, interval-arithmetic certification for near-bound float claims, and a fresh held-out batch. |
|
|
| ## License |
|
|
| Apache-2.0 |
|
|
