File size: 4,533 Bytes
04104cd 6cb442b 04104cd 6cb442b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: apache-2.0
tags:
- agent
- evaluation
- monitoring
---
# Squares Progress Agent
A small, offline-first agent that monitors public claims about packing n unit squares into the smallest square. It extracts claimed container sides from pages, validates geometry when coordinates are given, compares claims to a local bound table, and reports improvements, candidates, conflicts and errors.
This repository contains code only. There are no model weights.
The point of the project is the evaluation: which parts of the pipeline cause or prevent false "improvement" reports.
## Read first
- The synthetic scenarios were written by the same author as the agent. They test the design on its own fixtures and are not an independent test.
- Real-page results (below) come from private local runs on private captures. The raw pages, source URLs and quote-bearing labels are withheld, so those tables cannot be reproduced from this repository.
- LLM extractor numbers (below) also come from private local runs. They are not produced by `make test`, were not re-run after a later code-hash change, and cannot be reproduced from this repository without a live model endpoint.
- The held-out sets are small, so the confidence intervals are wide.
- The agent has not discovered or improved any packing record.
## Reproducible result (synthetic, `make test`)
Numbers below are from `results/synthetic_publish_check.txt` and `results/eval.json`, and are reproduced by `make test`.
| system | pass (23 scenarios) | false improvements |
|---|---|---|
| agent (rule extractor) | 22/23 | 0 |
| regex + same guards | 12/23 | 0 |
| plain regex baseline | 4/23 | 8 |
| agent without retry | 20/23 | 0 |
The one agent miss is a LaTeX-table scenario, which is a deliberate distribution-shift case.
`results/budget_curve.csv` is produced by a script that forces the same `max_steps` on every scenario. `s09_budget_tight` expects exhaustion at its native budget of 2, so the agent score of 21/23 (91%) at budget 8 is expected.
## Local results (not reproducible from this repo)
Real-page and LLM numbers come from private local runs. They are recorded in this repository but cannot be reproduced from the publishable tree alone.
Real batch 1 is taken from `results/real_publish_check.txt` (scenarios real01–real15). Real batch 2 is taken from `results/batch2_eval.txt`. The LLM sample is taken from `results/llm_sample7_arms.txt` (and the narrative in `results/llm_summary.md`). `results/heldout_scored.json` and `results/batch2_eval.txt` record code hash `eac25bad096c9a9c`; the published code differs because scrubbing, and a later change that made the model id required, changed the hashed files.
| set | agent | regex + guards | plain regex |
|---|---|---|---|
| real batch 1 (15 pages) | 15/15 | 14/15 | 9/15 (6 false improvements) |
| real batch 2 (12 pages, one source type) | 12/12 | 12/12 | 10/12 (2 false improvements) |
Batch 2 cannot tell the rule extractor and regex+guards apart. An optional LLM extractor — a free-tier hosted model (arm name flint); endpoint settings are not published — was tried on 7 scenarios and passed 6, with no fallback to the rules. The rule extractor passed all 7, so there is no evidence the LLM helps.
## What it does
search → fetch → extract → drop rejected or stale claims → validate coordinates → compare to bounds → report
- An improvement needs coordinates that validate. A lower number without coordinates is only a candidate.
- Exact rational packings are proofs. Float validation is not a proof, and near-bound float claims stay candidates.
- Hedged or retracted lines are skipped, which costs some recall.
- The LLM extractor proposes only (n, quote). The side value is parsed from a verbatim quote and goes through the same guards.
## Run
make test # synthetic scenarios only, no network, no API key
The real-page evaluation skips with a message when the private fixtures are missing. Live LLM calls need `SQ_LLM_API_KEY`, `SQ_LLM_BASE_URL` and `SQ_LLM_MODEL` in the environment.
## Known limits
- Lists like `5,6` are ambiguous with comma decimals.
- No Kaggle CSV adapter; the column and angle conventions are unconfirmed.
- `bounds.json` covers n = 1..29, 68, 102, 299 from a published catalogue. For n = 102 and 299, community claims may be lower than the catalogue.
## Next steps
Real search through official APIs, interval-arithmetic certification for near-bound float claims, and a fresh held-out batch.
## License
Apache-2.0
|