README.md

metadata

license: apache-2.0
tags:
  - agent
  - evaluation
  - monitoring

Squares Progress Agent

A small, offline-first agent that monitors public claims about packing n unit squares into the smallest square. It extracts claimed container sides from pages, validates geometry when coordinates are given, compares claims to a local bound table, and reports improvements, candidates, conflicts and errors.

This repository contains code only. There are no model weights.

The point of the project is the evaluation: which parts of the pipeline cause or prevent false "improvement" reports.

Read first

• The synthetic scenarios were written by the same author as the agent. They test the design on its own fixtures and are not an independent test.
• Real-page results (below) come from private local runs on private captures. The raw pages, source URLs and quote-bearing labels are withheld, so those tables cannot be reproduced from this repository.
• LLM extractor numbers (below) also come from private local runs. They are not produced by make test, were not re-run after a later code-hash change, and cannot be reproduced from this repository without a live model endpoint.
• The held-out sets are small, so the confidence intervals are wide.
• The agent has not discovered or improved any packing record.

Reproducible result (synthetic, make test)

Numbers below are from results/synthetic_publish_check.txt and results/eval.json, and are reproduced by make test.

system	pass (23 scenarios)	false improvements
agent (rule extractor)	22/23	0
regex + same guards	12/23	0
plain regex baseline	4/23	8
agent without retry	20/23	0

The one agent miss is a LaTeX-table scenario, which is a deliberate distribution-shift case.

results/budget_curve.csv is produced by a script that forces the same max_steps on every scenario. s09_budget_tight expects exhaustion at its native budget of 2, so the agent score of 21/23 (91%) at budget 8 is expected.

Local results (not reproducible from this repo)

Real-page and LLM numbers come from private local runs. They are recorded in this repository but cannot be reproduced from the publishable tree alone.

Real batch 1 is taken from results/real_publish_check.txt (scenarios real01–real15). Real batch 2 is taken from results/batch2_eval.txt. The LLM sample is taken from results/llm_sample7_arms.txt (and the narrative in results/llm_summary.md). results/heldout_scored.json and results/batch2_eval.txt record code hash eac25bad096c9a9c; the published code differs because scrubbing, and a later change that made the model id required, changed the hashed files.

set	agent	regex + guards	plain regex
real batch 1 (15 pages)	15/15	14/15	9/15 (6 false improvements)
real batch 2 (12 pages, one source type)	12/12	12/12	10/12 (2 false improvements)

Batch 2 cannot tell the rule extractor and regex+guards apart. An optional LLM extractor — a free-tier hosted model (arm name flint); endpoint settings are not published — was tried on 7 scenarios and passed 6, with no fallback to the rules. The rule extractor passed all 7, so there is no evidence the LLM helps.

What it does

search → fetch → extract → drop rejected or stale claims → validate coordinates → compare to bounds → report

• An improvement needs coordinates that validate. A lower number without coordinates is only a candidate.
• Exact rational packings are proofs. Float validation is not a proof, and near-bound float claims stay candidates.
• Hedged or retracted lines are skipped, which costs some recall.
• The LLM extractor proposes only (n, quote). The side value is parsed from a verbatim quote and goes through the same guards.

Run

make test        # synthetic scenarios only, no network, no API key

The real-page evaluation skips with a message when the private fixtures are missing. Live LLM calls need SQ_LLM_API_KEY, SQ_LLM_BASE_URL and SQ_LLM_MODEL in the environment.

Known limits

• Lists like 5,6 are ambiguous with comma decimals.
• No Kaggle CSV adapter; the column and angle conventions are unconfirmed.
• bounds.json covers n = 1..29, 68, 102, 299 from a published catalogue. For n = 102 and 299, community claims may be lower than the catalogue.

Next steps

Real search through official APIs, interval-arithmetic certification for near-bound float claims, and a fresh held-out batch.

License

Apache-2.0