---
license: apache-2.0
tags:
  - agent
  - evaluation
  - monitoring
---

# Squares Progress Agent

A small, offline-first agent that monitors public claims about packing n unit squares into the smallest square. It extracts claimed container sides from pages, validates geometry when coordinates are given, compares claims to a local bound table, and reports improvements, candidates, conflicts and errors.

This repository contains code only. There are no model weights.

The point of the project is the evaluation: which parts of the pipeline cause or prevent false "improvement" reports.

## Read first

- The synthetic scenarios were written by the same author as the agent. They test the design on its own fixtures and are not an independent test.
- Real-page results (below) come from private local runs on private captures. The raw pages, source URLs and quote-bearing labels are withheld, so those tables cannot be reproduced from this repository.
- LLM extractor numbers (below) also come from private local runs. They are not produced by `make test`, were not re-run after a later code-hash change, and cannot be reproduced from this repository without a live model endpoint.
- The held-out sets are small, so the confidence intervals are wide.
- The agent has not discovered or improved any packing record.

## Reproducible result (synthetic, `make test`)

Numbers below are from `results/synthetic_publish_check.txt` and `results/eval.json`, and are reproduced by `make test`.

| system | pass (23 scenarios) | false improvements |
|---|---|---|
| agent (rule extractor) | 22/23 | 0 |
| regex + same guards | 12/23 | 0 |
| plain regex baseline | 4/23 | 8 |
| agent without retry | 20/23 | 0 |

The one agent miss is a LaTeX-table scenario, which is a deliberate distribution-shift case.

`results/budget_curve.csv` is produced by a script that forces the same `max_steps` on every scenario. `s09_budget_tight` expects exhaustion at its native budget of 2, so the agent score of 21/23 (91%) at budget 8 is expected.

## Local results (not reproducible from this repo)

Real-page and LLM numbers come from private local runs. They are recorded in this repository but cannot be reproduced from the publishable tree alone.

Real batch 1 is taken from `results/real_publish_check.txt` (scenarios real01–real15). Real batch 2 is taken from `results/batch2_eval.txt`. The LLM sample is taken from `results/llm_sample7_arms.txt` (and the narrative in `results/llm_summary.md`). `results/heldout_scored.json` and `results/batch2_eval.txt` record code hash `eac25bad096c9a9c`; the published code differs because scrubbing, and a later change that made the model id required, changed the hashed files.

| set | agent | regex + guards | plain regex |
|---|---|---|---|
| real batch 1 (15 pages) | 15/15 | 14/15 | 9/15 (6 false improvements) |
| real batch 2 (12 pages, one source type) | 12/12 | 12/12 | 10/12 (2 false improvements) |

Batch 2 cannot tell the rule extractor and regex+guards apart. An optional LLM extractor — a free-tier hosted model (arm name flint); endpoint settings are not published — was tried on 7 scenarios and passed 6, with no fallback to the rules. The rule extractor passed all 7, so there is no evidence the LLM helps.

## What it does

search → fetch → extract → drop rejected or stale claims → validate coordinates → compare to bounds → report

- An improvement needs coordinates that validate. A lower number without coordinates is only a candidate.
- Exact rational packings are proofs. Float validation is not a proof, and near-bound float claims stay candidates.
- Hedged or retracted lines are skipped, which costs some recall.
- The LLM extractor proposes only (n, quote). The side value is parsed from a verbatim quote and goes through the same guards.

## Run

    make test        # synthetic scenarios only, no network, no API key

The real-page evaluation skips with a message when the private fixtures are missing. Live LLM calls need `SQ_LLM_API_KEY`, `SQ_LLM_BASE_URL` and `SQ_LLM_MODEL` in the environment.

## Known limits

- Lists like `5,6` are ambiguous with comma decimals.
- No Kaggle CSV adapter; the column and angle conventions are unconfirmed.
- `bounds.json` covers n = 1..29, 68, 102, 299 from a published catalogue. For n = 102 and 299, community claims may be lower than the catalogue.

## Next steps

Real search through official APIs, interval-arithmetic certification for near-bound float claims, and a fresh held-out batch.

## License

Apache-2.0