---
title: AIM Autonomous Ingestion Agent
emoji: 🤖
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: true
short_description: Autonomous DOI-deduped literature harvesting for AIM DB
---

# AIM Autonomous Ingestion Agent

An autonomous agent that continuously grows the **AIM Composites Materials
Database** — the same Postgres the
[MaterialsDatabase Space](https://huggingface.co/spaces/aim4composites/MaterialsDatabase)
serves — by harvesting open-access literature **with DOI provenance**:

```
        plan ─▶ discover ─▶ dedupe ─▶ download ─▶ ingest ─▶ report
                (OpenAlex,   (by DOI,   (validated  (Gemini →   (metrics,
                 arXiv,       URL, and    %PDF,      grounding,   run log,
                 Unpaywall)   sha256)     size caps) unit-norm,   query
                                                     status)      rotation)
```

* **discover** — query rotation over an editable frontier (8 builtin intents +
  a ~450-query materials × reinforcement × property/process grid seeded at
  boot); per query: OpenAlex full-text search (OA gold/hybrid/green), arXiv
  (ANDed terms) and Semantic Scholar **bulk** search (up to 1,000 OA-PDF
  papers per call); per cycle: one page of the **OpenAlex topic walk**
  (1-credit filter calls over the fibre-reinforced-composites topics, cursor
  persisted) and, when due, a **manufacturer datasheet** seed crawl.
* **dedupe** — a durable `agent_doi_seen` registry: a paper already harvested
  (by DOI, then URL, then content sha256) is never downloaded twice.
* **download** — polite (rate-limited, robots-aware), magic-byte + PyMuPDF
  validated, size-bounded; URL chain per work: repository copies first, then
  the publisher PDF, then Unpaywall by DOI, then the landing page's
  `citation_pdf_url`.
* **ingest** — `extraction.py` (grounded, unit-aware Gemini extraction, prompt
  v2.1: every value also carries the processing route of its specimen) via `batch_ingest.process_pdf` with the `pg_mirror` Postgres backend:
  every value is grounded in the PDF text, unit-normalized (`value_si`,
  `unit_canonical`), classified, deduped at measurement grain, and inserted
  with a `status` — flagged rows are quarantined for review, never dropped.
* **report** — per-cycle metrics in `agent_runs` (dashboards + paper §8), event
  stream in `agent_events`, optional Gemini query expansion when yields dry up.

The graph runs on LangGraph; the agent loop runs in-process on an APScheduler
interval, so the Space works autonomously while it is up — no viewers needed.

## Pages

| Page | What it shows |
|---|---|
| **Dashboard** | agent status pills, KPI tiles, rows/PDFs per cycle charts, latest activity feed, latest DOIs, recent cycles |
| **Run Control** | autonomy on/off, interval, per-cycle budget, source toggles, figure-mining toggles, query frontier editor, manual "run one cycle" |
| **Recently Added** | newest ingested documents with DOI links + verified-row counts |
| **Review Queue** | all `status != 'ok'` rows with origin filter, the figure crops behind figure-estimate rows, CSV export, origin-scoped human promote-to-ok |
| **Runs & Logs** | cycle history (status chips, durations, live-refreshing while a cycle runs) with per-PDF results, event stream grouped per run |
| **Database** | material-first browser (one expander per material with the process type and conditions of each property; 📈 marks figure-derived estimates) + raw-table browser (incl. the `figures` provenance table) with filters, pagination, CSV export |
| **Knowledge Graph** | every row of the three material tables as one graph, the review queue included and marked as such: whole-database map, matrix and fiber pairs, single-node view with the measured values; each value opens to its page, quoted sentence, flag reason, figure picture, processing route and date added; rebuilt when the tables change and swapped into the open page; JSON and Neo4j exports |

UI: single light theme + design system in `agent/ui_theme.py` (tokens, CSS,
altair theme, KPI/pill/chip/empty-state components); pages compose those
helpers and contain no styling of their own.

Since 28 Sep the Dashboard carries a "Candidates queued" tile, Run Control a
"Candidate queue" card (toggle, pass interval, low-water mark, per-pass budgets,
queue status by source, "Run a discovery pass now") and Runs & Logs a `Kind`
column (`cycle` / `discovery`) plus a `Queued` counter for passes.

## Deploy (≈5 minutes)

1. **Create the Space**: huggingface.co → New Space → owner `aim4composites`,
   name e.g. `AutonomousAgent`, SDK **Docker**, visibility your call.
2. **Push this folder** to it:
   ```bash
   git clone https://huggingface.co/spaces/aim4composites/AutonomousAgent
   cp -r <this folder>/* <this folder>/.streamlit <this folder>/.gitignore AutonomousAgent/
   cd AutonomousAgent && git add -A && git commit -m "Autonomous ingestion agent" && git push
   ```
   (or upload the files through the web UI — Files → Add file.)
3. **Set Space secrets** (Settings → Variables and secrets):

   | Secret | Value |
   |---|---|
   | `DB_HOST` / `DB_PORT` / `DB_NAME` / `DB_USER` / `DB_PASSWORD` | same values the MaterialsDatabase Space uses |
   | `GEMINI_API_KEY` | a **fresh** Gemini key (rotate the leaked ones!) |
   | `AGENT_ADMIN_PASSWORD` | password that unlocks Run Control / promote |
   | `CONTACT_EMAIL` *(variable, optional)* | polite-crawling contact for OpenAlex/Unpaywall |
   | `OPENALEX_API_KEY` | **required since 2026** — OpenAlex bills per call: anonymous = $0.10/day *per IP* (shared with every other Space on the egress IP), a free account key = $1/day ≈ 1,000 searches or 10,000 filter calls. Get it at openalex.org → Settings → API |
   | `S2_API_KEY` *(optional)* | dedicated Semantic Scholar quota (the bulk lane works without a key on the shared bucket; a free key is a form away) |

4. **One-time schema check**: the shared Postgres must already have the
   hardening columns + `sources` table (`python pg_migrate.py` dry-run, then
   `--apply`, from any machine with the DB env set). If the July pipeline runs
   already did this, there is nothing to do — the agent refuses to write until
   the schema is migrated, so it fails safe either way.
5. Open the Space → **Run Control** → unlock admin → *Run one cycle now* to
   smoke-test → toggle **Autonomous crawling** ON.

## Process type and conditions per property (prompt 2.1, Oct 2026)

A composite's properties depend on how the part was made, so every property
row carries the processing route of the specimen it was measured on:

| Column | Content |
|---|---|
| `process_type` | route, normalized to `extraction.PROCESS_TYPE_ENUM` (injection molding, compression molding / hot press, autoclave, out-of-autoclave, automated fiber placement / tape laying, thermoforming / stamp forming, filament winding, pultrusion, extrusion / compounding, additive manufacturing, liquid molding, welding / joining, fiber spinning, film / solution casting, electrospinning, heat treatment / annealing, other) |
| `process_name` | the route as the document names it |
| `process_conditions` | processing parameters as printed, `; `-joined (temperature, pressure, hold time, cooling rate, nozzle/bed temperature, print speed, ...) |
| `process_quote`, `process_page` | the verbatim sentence that names the route, and its page |
| `process_status` | `grounded` / `grounded_off_page` (sentence found in the PDF text and every number of the conditions found on digit boundaries), `conditions_unverified` (sentence found, a number is not), `ungrounded` (sentence not found), `unchecked` (no text layer) |

* The model returns the routes once per document (`processes`, ids `P1`,
  `P2`, ...) and each property points at one through `process_id`, so the
  output grows by one short field per property, not by a repeated paragraph.
  The same route at two settings is two routes.
* All six columns are NULL when the document does not say how the material
  was made (most vendor datasheets, as-received constituents, values quoted
  from other literature). The prompt forbids guessing.
* The route is context, not a value: its check never changes a row's `status`.
  An unverified route stays attached and marked (⚠ in the Database page).
* `test_condition` is still the *test* condition (23 °C, 50% RH); the
  `Processing` section still holds process settings a document reports as
  values of their own.
* Schema: the columns are appended to `migrate.EXTRA_COLUMNS`, so the boot
  migration adds them to the three material tables (`ADD COLUMN IF NOT
  EXISTS`); nothing is run by hand. The dedup grain is unchanged.
* Rows stored before this change (prompt 2.0) keep an empty route. The
  source PDFs are deleted after ingest, so filling them needs a re-download
  pass; that is not part of this change.
* Pages: Database (Process and Process conditions columns per material, a
  Process filter, a "With processing route" tile, the columns in the raw
  browser and its CSV), Review Queue (three process columns, also in the CSV),
  Dashboard ("Processing routes" card: rows per route, share grounded).
* Tests: `test_process_e2e.py` (parse, grounding, rows, boot migration of an
  old database, a stubbed cycle into Postgres, the three pages).

## Knowledge graph (live, Oct 2026)

The **Knowledge Graph** page shows the whole database as one graph and keeps it
current without anyone running anything.

- **What is in it.** `agent/kg.py` reads every row of `Polymers`, `Fibers` and
  `Composites_materials` through a server-side cursor (no row limit, the
  `image` bytes left out) and every row of `figures`, and builds the graph:
  materials, properties, property categories, polymer and fiber families,
  manufacturers, source documents (DOI, year and lane from `agent_doi_seen`),
  figures and process types. Every measured value keeps its status, page, SI
  value, route, the day it was added and the figure it was read off or cites.
- **The review queue is in it.** Rows with `status != 'ok'` are not left out:
  each is marked "review queue" with its status and `flag_reason`, and the
  switch above the graph shows all values, the verified ones, or only the
  queue (counts, map and lists follow the switch). Promoting a row in the
  Review Queue page changes the fingerprint, so the graph follows.
- **Where a value was found.** Opening a value (the arrow at the start of its
  row) shows the page, the quoted sentence (`source_quote`), the model's
  comment, the figure with its picture, the route's own name, quote, page and
  grounding status, and the date, model and prompt version of the record.
  These fields live in separate detail files of 2,000 values each
  (`static/kg/d/<build>/<n>.json.gz`), fetched when a value is opened, so the
  main file stays small as the database grows.
- **Figures.** Every harvested figure is a node, linked to its document and to
  the values read off it or citing it. The pictures of figures with values
  are copied from `figures.image_bytes` to `static/kg/fig/<figure_id>.png` in
  the background (20 per query, at most 4 minutes per run of the timer job,
  only what is not on disk yet; after a restart this takes a few minutes).
  `AGENT_KG_FIGURES` = `linked` (default) | `all` (every stored crop) | `off`.
- **When it rebuilds.** A fingerprint (per table: rows, verified rows, newest
  `extracted_at`, rows with a process type; and the number of figures) is
  compared with the one the current graph was built from. It is read right
  after every cycle, every `AGENT_KG_REFRESH_MINUTES` (default 10; 0 switches
  the timer off) by the scheduler job `kg-refresh`, and when someone opens the
  page (at most every 20 s). The graph is rebuilt only when the fingerprint
  changed, so an idle database costs four small aggregate queries per check.
- **How the open page follows.** The graph is written to `static/kg/`
  (`kg-data.json`, `kg-data.json.gz`, `kg-meta.json`) and served by Streamlit
  at `/app/static/kg/...` (`enableStaticServing` in `.streamlit/config.toml`;
  the Dockerfile makes the folder writable; `AGENT_KG_STATIC_DIR` moves it).
  The explorer (`agent/kg_explorer.html`, canvas + d3 from cdnjs) asks for
  `kg-meta.json` every `AGENT_KG_POLL_SECONDS` (default 60) and swaps the new
  graph in, keeping the open node and the zoom. Each build has its own detail
  folder (the one before is kept for pages still showing it). If the folder
  cannot be written, the graph and the value details are embedded in the page
  instead, a reload picks up changes, and figure pictures are not shown.
- **What is inferred.** Polymer and fiber families are assigned by pattern
  rules, and property names are merged for case, spelling and a short synonym
  list. Everything else is the database as stored. The map is computed from
  names and counts, not simulated: the same database always gives the same
  picture, and it shifts only where families grow.
- **Exports.** The published JSON can be read from outside the Space
  (`https://<space host>/app/static/kg/kg-data.json`; `detail.dir` in it
  names the folder of the detail files). The page also offers a Neo4j zip
  (nodes.csv, relationships.csv, load script; one Measurement node per value
  with its quote, flag reason, day added and review-queue mark; Figure nodes).
  `python -m agent.kg --out DIR` writes the same files, the figure pictures
  and a stand-alone explorer.html from the configured database.
- **Cost.** On the 1 Oct snapshot (34,768 rows, 1,093 figures in the test
  copy): build about 3 s, 5.6 MB of JSON, 0.9 MB compressed. At ten times the
  rows: build about 20 s, 46 MB / 7 MB, about 0.7 GB of memory at the peak.
  No model calls.

## Figure & graph mining (in-cycle)

Every ingested PDF also passes through `figures.py` (the same module the
local pipeline uses): plots and table images are harvested from the PDF's
layout (embedded rasters + vector clusters, junk filters, caption pairing),
classified with **one** Gemini vision call per PDF, and mineable kinds
(`property_plot`, `table_image`) are read for axis ranges and salient values.

**A number read off a graph is an estimate, not a grounded fact.** Figure
rows are inserted as `origin='figure'`, `status='figure_estimate'` — they
never publish (`status='ok'`) at insert, whatever the caller does
(`batch_ingest._row_values()` downgrades them), and only a human promote in
the Review Queue can bless one. The crop PNG itself is stored in the
`figures` table (`image_bytes`, capped at ~1.5 MB) so the Review Queue can
show the reviewer the exact plot behind every estimate — the Space's disk is
ephemeral, the database is not.

Knobs (Run Control, or env for first-boot defaults): `AGENT_FIGURES` (on by
default), `AGENT_FIGURE_MINING` (off = harvest+classify only),
`AGENT_MAX_FIGURES` (per-PDF cap, default 12, hard cap 20). Per-run counters
(`figures_found`, `figures_mined`, `figure_rows`, `vision_calls`) land in
`agent_runs` for the Runs & Logs table and the paper's cycle accounting.

## Operations notes

* **Budgets**: per-cycle caps (default 5 PDFs) bound Gemini spend; a hard
  ceiling of 25 is compiled in. Cycle lock is durable and self-expiring, so
  overlapping schedules or container restarts can't double-run.
* **Keeps running on its own**: the container entry point (`run_space.py`)
  boots the agent (schema check, stale-run sweep, scheduler) at process
  start, so cycles run from the moment the container is up, with or without
  a visitor. A keep-alive thread GETs the Space's own URL
  (`https://$SPACE_HOST/_stcore/health`) every `AGENT_KEEPALIVE_MINUTES`
  (default 10; 0 disables; `AGENT_KEEPALIVE_URL` overrides) so a free Space
  never reaches its 48-h idle sleep. Keep an external hourly pinger as a
  backup: if the container ever does sleep, any request wakes it and the
  agent boots itself.
* **Watchdog**: every cycle runs under a wall-clock budget
  (`AGENT_CYCLE_TIMEOUT_MINUTES`, default 60). A cycle still running past it
  is marked `timeout` in Runs & Logs, its lock is released and the next
  scheduled cycle proceeds. Without this, one cycle stuck in a network wait
  blocked the scheduler for good (the single-instance job never returned),
  which is what the long runs of `stale` cycles in August/September were.
  The crawler's `Retry-After` sleeps are capped at 300 s (the cap had been
  lost in the Aug 25 crawler sync), so that wait should no longer happen.
* **Stall detector (22 Sep)**: besides the hard budget, a cycle whose last
  agent event is older than `AGENT_CYCLE_STALL_MINUTES` (default 15) is
  abandoned with the stall written into its verdict; a slow but reporting
  cycle is bound only by the hard budget. `heartbeat_at` on `agent_runs` is
  the liveness signal (every event touches it).
* **Budgeted sources (22 Sep)**: a 429 whose `Retry-After` exceeds the 300 s
  cap, or that reports a zero remaining budget (OpenAlex's per-request
  billing), closes that host for its window with no retry and no sleep; the
  lane is skipped with one warn event per cycle and the reason lands in
  `agent_runs.sources`. Set the Space secret `OPENALEX_API_KEY` (a free
  account key is ten times the shared anonymous allowance). `AGENT_USE_NTRS`
  (default on) adds the NASA Technical Reports Server lane: free, no key,
  direct PDFs.
* **Cost column (22 Sep)**: Gemini `usageMetadata` is summed per cycle into
  `tokens_in` / `tokens_out` (text + vision calls); Runs & Logs derives
  dollars at read time from `AGENT_PRICE_IN_PER_M` / `AGENT_PRICE_OUT_PER_M`
  (defaults 0.30 / 2.50 USD per million, Gemini 2.5 Flash list price).
* **Figure citation linking (22 Sep)**: `AGENT_LINK_FIGURES` (default on,
  Run Control toggle) links each text row to the figure its evidence cites
  and embeds the PNG in the row's `image` column (no S3 needed). Adds three
  columns (`figure_ref`, `figure_link_score`, `figure_link_signals`) that
  the boot migration creates; `rows_linked` is counted per cycle.
* **Frontier rotation on dry cycles (28 Sep)**: every query a cycle searched
  is marked used in `node_report`, whether or not the cycle downloaded
  anything. Until then the bookkeeping sat behind `node_ingest`'s early
  return, so a dry cycle left `last_used_at` untouched and `next_queries`
  (LRU, never-used first) handed the same 20 queries back every half hour:
  from 22 to 28 Sep every cycle rediscovered the identical 109 exhausted
  candidates (42 DOIs already registered, 66 URLs already final) while the
  Gemini-expanded queries piled up unreached. Symptom to watch for in
  Runs & Logs: byte-identical Candidates / Skipped / URL seen columns across
  consecutive cycles. The harvest loop also no longer fetches (and marks) a
  frontier slice past its last round.
* **Downloads that never got ingested (28 Sep)**: a cycle that finds the
  material schema unmigrated ends as `error` (not `crashed`) and keeps its
  PDFs and their `downloaded` registry rows; the next cycle adopts and
  ingests whatever is still on disk (`adopted N PDF(s)` event). At boot,
  `downloaded` rows older than 3 h whose file is gone are deleted and their
  URLs re-opened in the crawler seen-set (`boot repair: re-queued ...`), so
  discovery finds and fetches those papers again — this is what frees the
  five PDFs runs #135/#139 lost on 15–16 Sep. The Inspect-run panel now
  shows why a run ended (crash traceback, watchdog note or cycle error).
* **More papers (28 Sep)**: discovery lanes on by default — OpenAlex phrase
  search *and* topic walk (`AGENT_OPENALEX_TOPICS`, 1-credit filter calls with
  persisted cursors, round-robin), Semantic Scholar bulk (≤1,000 OA-PDF papers
  per call; `S2_API_KEY` optional), arXiv with ANDed terms, NASA NTRS, and a
  datasheet round over the manufacturer seed pages (each seed ≤ once a day);
  the query frontier is seeded once with the 455-intent grid
  (`agent/query_grid.py`, origin `grid`). Downloads try repository copies
  before publisher URLs and fall back to the landing page's
  `citation_pdf_url`; records without an abstract get relevance-gate relief.
  Note that publisher hosts (MDPI, Elsevier, Wiley, …) 403 any cloud egress
  address, so the yield comes from repositories, NTRS, arXiv and datasheets.
* **Parallel ingestion (28 Sep)**: a cycle extracts its PDFs on a thread pool
  (`ingest_workers` in Run Control, default 3, hard max
  `AGENT_HARD_MAX_INGEST_WORKERS` = 6), one Postgres connection per worker;
  counters, events and the registry are written by the cycle thread as each
  PDF completes. Run Control also exposes the cycle's target (`Target new
  PDFs per cycle`, default 10: below it the cycle takes another frontier
  slice, up to `Max rounds per cycle`) — together with the 25-PDF cap these
  are the throughput knobs. A cycle's OpenAlex cost is 10 credits per query
  per round plus 1 per topic page, so 4 queries × 5 rounds × 48 cycles is
  ≈ 9,700 credits/day: right at the free key's allowance; prepaid credits
  or a longer interval buy headroom.
* **Candidate queue (28 Sep)**: discovery and download are decoupled. A
  *discovery pass* (`agent/discovery.py`, an `agent_runs` row of kind
  `discovery`) searches the next `discovery_queries` frontier queries on
  every lane — Semantic Scholar bulk, arXiv, NTRS, UDSpace (UD's DSpace
  theses, direct bitstreams), CORE (needs the free `CORE_API_KEY` secret;
  inactive without it) and, for the first `discovery_openalex_queries`, the
  10-credit OpenAlex phrase search — walks `discovery_topic_pages` pages per
  OpenAlex topic (1 credit each) and crawls every datasheet seed that is due,
  then queues every relevant candidate in `agent_candidates` (a DOI already in
  the registry or a URL the crawler has given up on lands as `seen`). It runs
  every `discovery_interval_hours` (24), right after any cycle that finds the
  queue below `queue_low_water` (150) or drains it to that, at boot when the
  queue is empty, and on demand (Run Control → "Run a discovery pass now").
  Cycles pop the queue best-score-first (repository-hosted copies rank ahead
  of publisher URLs) and download with no search calls at all; they search
  inline only when the queue is empty. Every popped row is settled:
  `ingested`, `seen` (registry / final URL), back to `queued` after a
  transient failure (never twice in one cycle; `failed` after 3 attempts).
  Turn it off with the Run Control toggle to get the pre-28 Sep inline
  behaviour. OpenAlex spend drops from up to ~9,700 credits/day to a few
  hundred per pass.
* **Secrets never reach the database (28 Sep)**: `agdb.redact()` strips
  credential query parameters (`?key=`, `&api_key=`, `token=`) and bearer
  tokens from every event, run report, registry status and candidate error
  before it is stored, and `extraction` raises Gemini errors that name the
  endpoint and the API's own detail instead of the keyed URL (requests'
  `HTTPError` / urllib3's connection errors quote it). At boot,
  `agdb.redact_stored_secrets` scrubs rows written before this existed and
  logs `boot repair: redacted credential parameters …` — if that event ever
  appears, rotate the key it refers to: it was readable on Runs & Logs.
* **Gemini budget refusals (28 Sep)**: a 429 whose body says
  `RESOURCE_EXHAUSTED` / "monthly spending cap" is not retried (it cost
  14 s per PDF and cleared nothing). The PDF stays on disk with its
  registry row `downloaded`, the cycle's remaining PDFs are not attempted,
  one warn event says `Gemini refused for budget reasons (…); N PDF(s) kept
  on disk for the next cycle`, and the run ends `error` only if nothing was
  ingested at all. The next cycle adopts the kept PDFs (or the boot repair
  re-queues them after a restart). Raise the cap at ai.studio/spend.
  While the refusal lasts the agent is on a **budget hold** (`agent_state
  gemini_budget_hold`): cycles download nothing and probe with one kept PDF
  (or one fresh download if the disk was wiped); the first successful
  extraction lifts the hold and downloads resume next cycle. Adoption of
  kept PDFs is capped per cycle at the download cap, so a backlog is worked
  off over several cycles.
* **Cost (28 Sep)**: Gemini's thinking tokens are billed as output at 6× the
  input price, and with no `thinkingConfig` the model thinks with an
  unbounded dynamic budget: a 25-paper cycle cost ~$7.50, $6.69 of it output
  (~30k tokens per PDF for a ~4k-token extraction). `gemini_thinking`
  (Run Control → Gemini cost; env `GEMINI_THINKING`; default `low`) is
  applied to every Gemini call — `dynamic` | `low` | `minimal` | `high` |
  `off` | an integer budget; a model that rejects the option gets the call
  repeated without it and the option is dropped for the process. Figure
  *mining* (one thinking vision call per mineable figure, into quarantined
  `figure_estimate` rows) is off by default; harvest + classify (one call per
  PDF) and figure linking stay on. Both defaults are applied once to a stored
  config at boot (`cost defaults applied: …` event) and can be changed back in
  Run Control. Expected: roughly a third of the previous bill per paper —
  watch the Cost column, and rows per paper / flagged share for recall.
* **Cost (30 Sep) — real prices, models, near-duplicates**: the cost column
  used to price every token at Gemini 2.5 Flash rates ($0.30 / $2.50 per M)
  while the agent runs `gemini-3.5-flash` ($1.50 / $9.00): it showed about a
  quarter of the bill (runs 544–593: $12.98 shown, ~$48.50 real, ~$0.41 per
  PDF on dynamic thinking with mining on). Runs now store `cost_usd`, priced
  per model (`agent/config.py` `MODEL_PRICES`), text and vision calls each at
  their own model; older runs are re-priced at gemini-3.5-flash in Runs &
  Logs, which also shows thinking tokens, the model, $/PDF and a spend line.
  Run Control → Gemini cost adds the **text model** (every row is stamped
  with it: change it only after `cost_ab.py` on the gold set), a **vision
  model** for the figure calls (default: same), and the **near-duplicate
  gate** (`agent/textdedupe.py`, default 0.9): a PDF whose text is that
  similar to an ingested one — regional datasheet variants, a preprint and
  its published version — is skipped before any Gemini call
  (`duplicate_text` in the registry and the Near-dup column).
* **Ephemeral disk**: PDFs live under `/tmp` only until ingested; provenance
  (DOI, sha256, title, URL, query) is durable in `agent_doi_seen`.
* **Safety**: agent bookkeeping is in namespaced `agent_*` tables; controls
  are password-gated; read-only without the password. At boot the Space
  additively migrates the **connected** database (hardening columns incl.
  `origin`/`figure_id`, the origin-aware dedup index, the `figures` table) —
  `aim_agent` is the agent's own DB, so this is on by default. Before
  pointing `DB_NAME` at the team's shared database, either agree on the
  additive migration or set `AUTO_MIGRATE_MATERIAL_SCHEMA=0` (the cycle then
  refuses to write until `pg_migrate.py --apply` is run by hand).

## Repo map

Vendored pipeline (single source of truth, synced from the project repo —
the only Space-side deltas are the PG keepalive/statement-timeout hardening
and the Retry-After cap): `extraction.py`, `batch_ingest.py`, `figures.py`,
`pdf_crawler.py`, `pg_mirror.py`, `pg_migrate.py`, `migrate.py`,
`generate_queries.py`. `FIGURES.md` documents the figure stage.

Agent layer (new): `agent/` (orchestrator, scheduler with watchdog, autostart,
keepalive, DB, config, `query_grid`), `run_space.py` (container entry point),
`app.py` + `page_files/` (Streamlit UI), `agent/kg.py` + `agent/kg_explorer.html`
(knowledge graph builder and explorer; `test_kg.py`). Tests: `test_agent_state.py`,
`test_harvest_loop.py`, `test_sources.py` (no DB, no network), `test_e2e.py`,
`test_figures_e2e.py`, `test_keep_running.py`, `test_counters_e2e.py`,
`test_frontier_rotation_e2e.py`, `test_more_papers_e2e.py`, `test_parallel_ingest_e2e.py`,
`test_candidate_queue_e2e.py` (scratch Postgres via `DB_*`; the network sources are
switched off by env), `test_gemini_errors.py` (no DB, no network).