README.md

metadata

title: AIM Autonomous Ingestion Agent
emoji: 🤖
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: true
short_description: Autonomous DOI-deduped literature harvesting for AIM DB

AIM Autonomous Ingestion Agent

An autonomous agent that continuously grows the AIM Composites Materials Database — the same Postgres the MaterialsDatabase Space serves — by harvesting open-access literature with DOI provenance:

        plan ─▶ discover ─▶ dedupe ─▶ download ─▶ ingest ─▶ report
                (OpenAlex,   (by DOI,   (validated  (Gemini →   (metrics,
                 arXiv,       URL, and    %PDF,      grounding,   run log,
                 Unpaywall)   sha256)     size caps) unit-norm,   query
                                                     status)      rotation)

• discover — query rotation over an editable frontier (8 builtin intents + a ~450-query materials × reinforcement × property/process grid seeded at boot); per query: OpenAlex full-text search (OA gold/hybrid/green), arXiv (ANDed terms) and Semantic Scholar bulk search (up to 1,000 OA-PDF papers per call); per cycle: one page of the OpenAlex topic walk (1-credit filter calls over the fibre-reinforced-composites topics, cursor persisted) and, when due, a manufacturer datasheet seed crawl.
• dedupe — a durable agent_doi_seen registry: a paper already harvested (by DOI, then URL, then content sha256) is never downloaded twice.
• download — polite (rate-limited, robots-aware), magic-byte + PyMuPDF validated, size-bounded; URL chain per work: repository copies first, then the publisher PDF, then Unpaywall by DOI, then the landing page's citation_pdf_url.
• ingest — extraction.py (grounded, unit-aware Gemini extraction, prompt v2.1: every value also carries the processing route of its specimen) via batch_ingest.process_pdf with the pg_mirror Postgres backend: every value is grounded in the PDF text, unit-normalized (value_si, unit_canonical), classified, deduped at measurement grain, and inserted with a status — flagged rows are quarantined for review, never dropped.
• report — per-cycle metrics in agent_runs (dashboards + paper §8), event stream in agent_events, optional Gemini query expansion when yields dry up.

The graph runs on LangGraph; the agent loop runs in-process on an APScheduler interval, so the Space works autonomously while it is up — no viewers needed.

Pages

Page	What it shows
Dashboard	agent status pills, KPI tiles, rows/PDFs per cycle charts, latest activity feed, latest DOIs, recent cycles
Run Control	autonomy on/off, interval, per-cycle budget, source toggles, figure-mining toggles, query frontier editor, manual "run one cycle"
Recently Added	newest ingested documents with DOI links + verified-row counts
Review Queue	all status != 'ok' rows with origin filter, the figure crops behind figure-estimate rows, CSV export, origin-scoped human promote-to-ok
Runs & Logs	cycle history (status chips, durations, live-refreshing while a cycle runs) with per-PDF results, event stream grouped per run
Database	material-first browser (one expander per material with the process type and conditions of each property; 📈 marks figure-derived estimates) + raw-table browser (incl. the figures provenance table) with filters, pagination, CSV export
Knowledge Graph	every row of the three material tables as one graph, the review queue included and marked as such: whole-database map, matrix and fiber pairs, single-node view with the measured values; each value opens to its page, quoted sentence, flag reason, figure picture, processing route and date added; rebuilt when the tables change and swapped into the open page; JSON and Neo4j exports

UI: single light theme + design system in agent/ui_theme.py (tokens, CSS, altair theme, KPI/pill/chip/empty-state components); pages compose those helpers and contain no styling of their own.

Since 28 Sep the Dashboard carries a "Candidates queued" tile, Run Control a "Candidate queue" card (toggle, pass interval, low-water mark, per-pass budgets, queue status by source, "Run a discovery pass now") and Runs & Logs a Kind column (cycle / discovery) plus a Queued counter for passes.

Deploy (≈5 minutes)

1. Create the Space: huggingface.co → New Space → owner aim4composites, name e.g. AutonomousAgent, SDK Docker, visibility your call.

2. Push this folder to it:

   git clone https://huggingface.co/spaces/aim4composites/AutonomousAgent
   cp -r <this folder>/* <this folder>/.streamlit <this folder>/.gitignore AutonomousAgent/
   cd AutonomousAgent && git add -A && git commit -m "Autonomous ingestion agent" && git push
   

   (or upload the files through the web UI — Files → Add file.)

3. Set Space secrets (Settings → Variables and secrets):

   Secret	Value
   DB_HOST / DB_PORT / DB_NAME / DB_USER / DB_PASSWORD	same values the MaterialsDatabase Space uses
   GEMINI_API_KEY	a fresh Gemini key (rotate the leaked ones!)
   AGENT_ADMIN_PASSWORD	password that unlocks Run Control / promote
   CONTACT_EMAIL (variable, optional)	polite-crawling contact for OpenAlex/Unpaywall
   OPENALEX_API_KEY	required since 2026 — OpenAlex bills per call: anonymous = $0.10/day per IP (shared with every other Space on the egress IP), a free account key = $1/day ≈ 1,000 searches or 10,000 filter calls. Get it at openalex.org → Settings → API
   S2_API_KEY (optional)	dedicated Semantic Scholar quota (the bulk lane works without a key on the shared bucket; a free key is a form away)

4. One-time schema check: the shared Postgres must already have the hardening columns + sources table (python pg_migrate.py dry-run, then --apply, from any machine with the DB env set). If the July pipeline runs already did this, there is nothing to do — the agent refuses to write until the schema is migrated, so it fails safe either way.

5. Open the Space → Run Control → unlock admin → Run one cycle now to smoke-test → toggle Autonomous crawling ON.

Process type and conditions per property (prompt 2.1, Oct 2026)

A composite's properties depend on how the part was made, so every property row carries the processing route of the specimen it was measured on:

Column	Content
process_type	route, normalized to extraction.PROCESS_TYPE_ENUM (injection molding, compression molding / hot press, autoclave, out-of-autoclave, automated fiber placement / tape laying, thermoforming / stamp forming, filament winding, pultrusion, extrusion / compounding, additive manufacturing, liquid molding, welding / joining, fiber spinning, film / solution casting, electrospinning, heat treatment / annealing, other)
process_name	the route as the document names it
process_conditions	processing parameters as printed, ; -joined (temperature, pressure, hold time, cooling rate, nozzle/bed temperature, print speed, ...)
process_quote, process_page	the verbatim sentence that names the route, and its page
process_status	grounded / grounded_off_page (sentence found in the PDF text and every number of the conditions found on digit boundaries), conditions_unverified (sentence found, a number is not), ungrounded (sentence not found), unchecked (no text layer)

• The model returns the routes once per document (processes, ids P1, P2, ...) and each property points at one through process_id, so the output grows by one short field per property, not by a repeated paragraph. The same route at two settings is two routes.
• All six columns are NULL when the document does not say how the material was made (most vendor datasheets, as-received constituents, values quoted from other literature). The prompt forbids guessing.
• The route is context, not a value: its check never changes a row's status. An unverified route stays attached and marked (⚠ in the Database page).
• test_condition is still the test condition (23 °C, 50% RH); the Processing section still holds process settings a document reports as values of their own.
• Schema: the columns are appended to migrate.EXTRA_COLUMNS, so the boot migration adds them to the three material tables (ADD COLUMN IF NOT EXISTS); nothing is run by hand. The dedup grain is unchanged.
• Rows stored before this change (prompt 2.0) keep an empty route. The source PDFs are deleted after ingest, so filling them needs a re-download pass; that is not part of this change.
• Pages: Database (Process and Process conditions columns per material, a Process filter, a "With processing route" tile, the columns in the raw browser and its CSV), Review Queue (three process columns, also in the CSV), Dashboard ("Processing routes" card: rows per route, share grounded).
• Tests: test_process_e2e.py (parse, grounding, rows, boot migration of an old database, a stubbed cycle into Postgres, the three pages).

Knowledge graph (live, Oct 2026)

The Knowledge Graph page shows the whole database as one graph and keeps it current without anyone running anything.

• What is in it. agent/kg.py reads every row of Polymers, Fibers and Composites_materials through a server-side cursor (no row limit, the image bytes left out) and every row of figures, and builds the graph: materials, properties, property categories, polymer and fiber families, manufacturers, source documents (DOI, year and lane from agent_doi_seen), figures and process types. Every measured value keeps its status, page, SI value, route, the day it was added and the figure it was read off or cites.
• The review queue is in it. Rows with status != 'ok' are not left out: each is marked "review queue" with its status and flag_reason, and the switch above the graph shows all values, the verified ones, or only the queue (counts, map and lists follow the switch). Promoting a row in the Review Queue page changes the fingerprint, so the graph follows.
• Where a value was found. Opening a value (the arrow at the start of its row) shows the page, the quoted sentence (source_quote), the model's comment, the figure with its picture, the route's own name, quote, page and grounding status, and the date, model and prompt version of the record. These fields live in separate detail files of 2,000 values each (static/kg/d/<build>/<n>.json.gz), fetched when a value is opened, so the main file stays small as the database grows.
• Figures. Every harvested figure is a node, linked to its document and to the values read off it or citing it. The pictures of figures with values are copied from figures.image_bytes to static/kg/fig/<figure_id>.png in the background (20 per query, at most 4 minutes per run of the timer job, only what is not on disk yet; after a restart this takes a few minutes). AGENT_KG_FIGURES = linked (default) | all (every stored crop) | off.
• When it rebuilds. A fingerprint (per table: rows, verified rows, newest extracted_at, rows with a process type; and the number of figures) is compared with the one the current graph was built from. It is read right after every cycle, every AGENT_KG_REFRESH_MINUTES (default 10; 0 switches the timer off) by the scheduler job kg-refresh, and when someone opens the page (at most every 20 s). The graph is rebuilt only when the fingerprint changed, so an idle database costs four small aggregate queries per check.
• How the open page follows. The graph is written to static/kg/ (kg-data.json, kg-data.json.gz, kg-meta.json) and served by Streamlit at /app/static/kg/... (enableStaticServing in .streamlit/config.toml; the Dockerfile makes the folder writable; AGENT_KG_STATIC_DIR moves it). The explorer (agent/kg_explorer.html, canvas + d3 from cdnjs) asks for kg-meta.json every AGENT_KG_POLL_SECONDS (default 60) and swaps the new graph in, keeping the open node and the zoom. Each build has its own detail folder (the one before is kept for pages still showing it). If the folder cannot be written, the graph and the value details are embedded in the page instead, a reload picks up changes, and figure pictures are not shown.
• What is inferred. Polymer and fiber families are assigned by pattern rules, and property names are merged for case, spelling and a short synonym list. Everything else is the database as stored. The map is computed from names and counts, not simulated: the same database always gives the same picture, and it shifts only where families grow.
• Exports. The published JSON can be read from outside the Space (https://<space host>/app/static/kg/kg-data.json; detail.dir in it names the folder of the detail files). The page also offers a Neo4j zip (nodes.csv, relationships.csv, load script; one Measurement node per value with its quote, flag reason, day added and review-queue mark; Figure nodes). python -m agent.kg --out DIR writes the same files, the figure pictures and a stand-alone explorer.html from the configured database.
• Cost. On the 1 Oct snapshot (34,768 rows, 1,093 figures in the test copy): build about 3 s, 5.6 MB of JSON, 0.9 MB compressed. At ten times the rows: build about 20 s, 46 MB / 7 MB, about 0.7 GB of memory at the peak. No model calls.

Figure & graph mining (in-cycle)

Every ingested PDF also passes through figures.py (the same module the local pipeline uses): plots and table images are harvested from the PDF's layout (embedded rasters + vector clusters, junk filters, caption pairing), classified with one Gemini vision call per PDF, and mineable kinds (property_plot, table_image) are read for axis ranges and salient values.

A number read off a graph is an estimate, not a grounded fact. Figure rows are inserted as origin='figure', status='figure_estimate' — they never publish (status='ok') at insert, whatever the caller does (batch_ingest._row_values() downgrades them), and only a human promote in the Review Queue can bless one. The crop PNG itself is stored in the figures table (image_bytes, capped at ~1.5 MB) so the Review Queue can show the reviewer the exact plot behind every estimate — the Space's disk is ephemeral, the database is not.

Knobs (Run Control, or env for first-boot defaults): AGENT_FIGURES (on by default), AGENT_FIGURE_MINING (off = harvest+classify only), AGENT_MAX_FIGURES (per-PDF cap, default 12, hard cap 20). Per-run counters (figures_found, figures_mined, figure_rows, vision_calls) land in agent_runs for the Runs & Logs table and the paper's cycle accounting.

Operations notes

• Budgets: per-cycle caps (default 5 PDFs) bound Gemini spend; a hard ceiling of 25 is compiled in. Cycle lock is durable and self-expiring, so overlapping schedules or container restarts can't double-run.
• Keeps running on its own: the container entry point (run_space.py) boots the agent (schema check, stale-run sweep, scheduler) at process start, so cycles run from the moment the container is up, with or without a visitor. A keep-alive thread GETs the Space's own URL (https://$SPACE_HOST/_stcore/health) every AGENT_KEEPALIVE_MINUTES (default 10; 0 disables; AGENT_KEEPALIVE_URL overrides) so a free Space never reaches its 48-h idle sleep. Keep an external hourly pinger as a backup: if the container ever does sleep, any request wakes it and the agent boots itself.
• Watchdog: every cycle runs under a wall-clock budget (AGENT_CYCLE_TIMEOUT_MINUTES, default 60). A cycle still running past it is marked timeout in Runs & Logs, its lock is released and the next scheduled cycle proceeds. Without this, one cycle stuck in a network wait blocked the scheduler for good (the single-instance job never returned), which is what the long runs of stale cycles in August/September were. The crawler's Retry-After sleeps are capped at 300 s (the cap had been lost in the Aug 25 crawler sync), so that wait should no longer happen.
• Stall detector (22 Sep): besides the hard budget, a cycle whose last agent event is older than AGENT_CYCLE_STALL_MINUTES (default 15) is abandoned with the stall written into its verdict; a slow but reporting cycle is bound only by the hard budget. heartbeat_at on agent_runs is the liveness signal (every event touches it).
• Budgeted sources (22 Sep): a 429 whose Retry-After exceeds the 300 s cap, or that reports a zero remaining budget (OpenAlex's per-request billing), closes that host for its window with no retry and no sleep; the lane is skipped with one warn event per cycle and the reason lands in agent_runs.sources. Set the Space secret OPENALEX_API_KEY (a free account key is ten times the shared anonymous allowance). AGENT_USE_NTRS (default on) adds the NASA Technical Reports Server lane: free, no key, direct PDFs.
• Cost column (22 Sep): Gemini usageMetadata is summed per cycle into tokens_in / tokens_out (text + vision calls); Runs & Logs derives dollars at read time from AGENT_PRICE_IN_PER_M / AGENT_PRICE_OUT_PER_M (defaults 0.30 / 2.50 USD per million, Gemini 2.5 Flash list price).
• Figure citation linking (22 Sep): AGENT_LINK_FIGURES (default on, Run Control toggle) links each text row to the figure its evidence cites and embeds the PNG in the row's image column (no S3 needed). Adds three columns (figure_ref, figure_link_score, figure_link_signals) that the boot migration creates; rows_linked is counted per cycle.
• Frontier rotation on dry cycles (28 Sep): every query a cycle searched is marked used in node_report, whether or not the cycle downloaded anything. Until then the bookkeeping sat behind node_ingest's early return, so a dry cycle left last_used_at untouched and next_queries (LRU, never-used first) handed the same 20 queries back every half hour: from 22 to 28 Sep every cycle rediscovered the identical 109 exhausted candidates (42 DOIs already registered, 66 URLs already final) while the Gemini-expanded queries piled up unreached. Symptom to watch for in Runs & Logs: byte-identical Candidates / Skipped / URL seen columns across consecutive cycles. The harvest loop also no longer fetches (and marks) a frontier slice past its last round.
• Downloads that never got ingested (28 Sep): a cycle that finds the material schema unmigrated ends as error (not crashed) and keeps its PDFs and their downloaded registry rows; the next cycle adopts and ingests whatever is still on disk (adopted N PDF(s) event). At boot, downloaded rows older than 3 h whose file is gone are deleted and their URLs re-opened in the crawler seen-set (boot repair: re-queued ...), so discovery finds and fetches those papers again — this is what frees the five PDFs runs #135/#139 lost on 15–16 Sep. The Inspect-run panel now shows why a run ended (crash traceback, watchdog note or cycle error).
• More papers (28 Sep): discovery lanes on by default — OpenAlex phrase search and topic walk (AGENT_OPENALEX_TOPICS, 1-credit filter calls with persisted cursors, round-robin), Semantic Scholar bulk (≤1,000 OA-PDF papers per call; S2_API_KEY optional), arXiv with ANDed terms, NASA NTRS, and a datasheet round over the manufacturer seed pages (each seed ≤ once a day); the query frontier is seeded once with the 455-intent grid (agent/query_grid.py, origin grid). Downloads try repository copies before publisher URLs and fall back to the landing page's citation_pdf_url; records without an abstract get relevance-gate relief. Note that publisher hosts (MDPI, Elsevier, Wiley, …) 403 any cloud egress address, so the yield comes from repositories, NTRS, arXiv and datasheets.
• Parallel ingestion (28 Sep): a cycle extracts its PDFs on a thread pool (ingest_workers in Run Control, default 3, hard max AGENT_HARD_MAX_INGEST_WORKERS = 6), one Postgres connection per worker; counters, events and the registry are written by the cycle thread as each PDF completes. Run Control also exposes the cycle's target (Target new PDFs per cycle, default 10: below it the cycle takes another frontier slice, up to Max rounds per cycle) — together with the 25-PDF cap these are the throughput knobs. A cycle's OpenAlex cost is 10 credits per query per round plus 1 per topic page, so 4 queries × 5 rounds × 48 cycles is ≈ 9,700 credits/day: right at the free key's allowance; prepaid credits or a longer interval buy headroom.
• Candidate queue (28 Sep): discovery and download are decoupled. A discovery pass (agent/discovery.py, an agent_runs row of kind discovery) searches the next discovery_queries frontier queries on every lane — Semantic Scholar bulk, arXiv, NTRS, UDSpace (UD's DSpace theses, direct bitstreams), CORE (needs the free CORE_API_KEY secret; inactive without it) and, for the first discovery_openalex_queries, the 10-credit OpenAlex phrase search — walks discovery_topic_pages pages per OpenAlex topic (1 credit each) and crawls every datasheet seed that is due, then queues every relevant candidate in agent_candidates (a DOI already in the registry or a URL the crawler has given up on lands as seen). It runs every discovery_interval_hours (24), right after any cycle that finds the queue below queue_low_water (150) or drains it to that, at boot when the queue is empty, and on demand (Run Control → "Run a discovery pass now"). Cycles pop the queue best-score-first (repository-hosted copies rank ahead of publisher URLs) and download with no search calls at all; they search inline only when the queue is empty. Every popped row is settled: ingested, seen (registry / final URL), back to queued after a transient failure (never twice in one cycle; failed after 3 attempts). Turn it off with the Run Control toggle to get the pre-28 Sep inline behaviour. OpenAlex spend drops from up to ~9,700 credits/day to a few hundred per pass.
• Secrets never reach the database (28 Sep): agdb.redact() strips credential query parameters (?key=, &api_key=, token=) and bearer tokens from every event, run report, registry status and candidate error before it is stored, and extraction raises Gemini errors that name the endpoint and the API's own detail instead of the keyed URL (requests' HTTPError / urllib3's connection errors quote it). At boot, agdb.redact_stored_secrets scrubs rows written before this existed and logs boot repair: redacted credential parameters … — if that event ever appears, rotate the key it refers to: it was readable on Runs & Logs.
• Gemini budget refusals (28 Sep): a 429 whose body says RESOURCE_EXHAUSTED / "monthly spending cap" is not retried (it cost 14 s per PDF and cleared nothing). The PDF stays on disk with its registry row downloaded, the cycle's remaining PDFs are not attempted, one warn event says Gemini refused for budget reasons (…); N PDF(s) kept on disk for the next cycle, and the run ends error only if nothing was ingested at all. The next cycle adopts the kept PDFs (or the boot repair re-queues them after a restart). Raise the cap at ai.studio/spend. While the refusal lasts the agent is on a budget hold (agent_state gemini_budget_hold): cycles download nothing and probe with one kept PDF (or one fresh download if the disk was wiped); the first successful extraction lifts the hold and downloads resume next cycle. Adoption of kept PDFs is capped per cycle at the download cap, so a backlog is worked off over several cycles.
• Cost (28 Sep): Gemini's thinking tokens are billed as output at 6× the input price, and with no thinkingConfig the model thinks with an unbounded dynamic budget: a 25-paper cycle cost $7.50, $6.69 of it output (30k tokens per PDF for a ~4k-token extraction). gemini_thinking (Run Control → Gemini cost; env GEMINI_THINKING; default low) is applied to every Gemini call — dynamic | low | minimal | high | off | an integer budget; a model that rejects the option gets the call repeated without it and the option is dropped for the process. Figure mining (one thinking vision call per mineable figure, into quarantined figure_estimate rows) is off by default; harvest + classify (one call per PDF) and figure linking stay on. Both defaults are applied once to a stored config at boot (cost defaults applied: … event) and can be changed back in Run Control. Expected: roughly a third of the previous bill per paper — watch the Cost column, and rows per paper / flagged share for recall.
• Cost (30 Sep) — real prices, models, near-duplicates: the cost column used to price every token at Gemini 2.5 Flash rates ($0.30 / $2.50 per M) while the agent runs gemini-3.5-flash ($1.50 / $9.00): it showed about a quarter of the bill (runs 544–593: $12.98 shown, ~$48.50 real, ~$0.41 per PDF on dynamic thinking with mining on). Runs now store cost_usd, priced per model (agent/config.py MODEL_PRICES), text and vision calls each at their own model; older runs are re-priced at gemini-3.5-flash in Runs & Logs, which also shows thinking tokens, the model, $/PDF and a spend line. Run Control → Gemini cost adds the text model (every row is stamped with it: change it only after cost_ab.py on the gold set), a vision model for the figure calls (default: same), and the near-duplicate gate (agent/textdedupe.py, default 0.9): a PDF whose text is that similar to an ingested one — regional datasheet variants, a preprint and its published version — is skipped before any Gemini call (duplicate_text in the registry and the Near-dup column).
• Ephemeral disk: PDFs live under /tmp only until ingested; provenance (DOI, sha256, title, URL, query) is durable in agent_doi_seen.
• Safety: agent bookkeeping is in namespaced agent_* tables; controls are password-gated; read-only without the password. At boot the Space additively migrates the connected database (hardening columns incl. origin/figure_id, the origin-aware dedup index, the figures table) — aim_agent is the agent's own DB, so this is on by default. Before pointing DB_NAME at the team's shared database, either agree on the additive migration or set AUTO_MIGRATE_MATERIAL_SCHEMA=0 (the cycle then refuses to write until pg_migrate.py --apply is run by hand).

Repo map

Vendored pipeline (single source of truth, synced from the project repo — the only Space-side deltas are the PG keepalive/statement-timeout hardening and the Retry-After cap): extraction.py, batch_ingest.py, figures.py, pdf_crawler.py, pg_mirror.py, pg_migrate.py, migrate.py, generate_queries.py. FIGURES.md documents the figure stage.

Agent layer (new): agent/ (orchestrator, scheduler with watchdog, autostart, keepalive, DB, config, query_grid), run_space.py (container entry point), app.py + page_files/ (Streamlit UI), agent/kg.py + agent/kg_explorer.html (knowledge graph builder and explorer; test_kg.py). Tests: test_agent_state.py, test_harvest_loop.py, test_sources.py (no DB, no network), test_e2e.py, test_figures_e2e.py, test_keep_running.py, test_counters_e2e.py, test_frontier_rotation_e2e.py, test_more_papers_e2e.py, test_parallel_ingest_e2e.py, test_candidate_queue_e2e.py (scratch Postgres via DB_*; the network sources are switched off by env), test_gemini_errors.py (no DB, no network).