perf(consumer): materialize observation boundaries

README.md

@@ -1,46 +1,250 @@
----
-title: xtap-pool
-emoji: 🐦
-colorFrom: gray
-colorTo: blue
-sdk: docker
-app_port: 7860
-hf_oauth: true
-hf_oauth_expiration_minutes: 43200
-pinned: false
----
-
 # xtap-pool
 
-Private tweet pool for a group of friends running the
-[xtap-pool extension](https://github.com/osolmaz/xtap-pool).
-
-- `POST /api/ingest` — extension submissions (Bearer pool token)
-- `GET /connect` — sign in with Hugging Face to connect the extension
-- `GET /api/units` — revision-consistent enriched units for scoped service accounts
-- `/` — tweet explorer and pool administration
-- `GET /healthz` and `GET /readyz` — machine-readable runtime health
-
-Required Space secrets: `HF_TOKEN` (fine-grained, read/write access to the raw
-and index Buckets only), `POOL_SIGNING_SECRET`, and `SESSION_SECRET`.
-Production classification does not run in the Space, and `ENRICH_ENABLED` must
-remain `false` there.
-
-Required Space variables: `RAW_BUCKET`, `INDEX_BUCKET`, `ALLOWED_USERS`
-(initial comma-separated HF usernames), `POOL_ADMINS` (bootstrap admins),
-`SPACE_HOST` (auto-injected by HF), and the bounded enrichment configuration
-reconciled by setup. Doctor creates a separate suspended Hugging Face Job with
-encrypted Bucket-writer and inference secrets. The Job refuses missing pricing,
-cost ceilings, or a source revision that differs from the deployed Space image.
-
-After setup, admins manage individual members and one allowed member
-organization in the Space Admin tab. Durable membership is stored in the raw
-Bucket at `config/pool.json`; the Space variables are bootstrap and recovery
-inputs. The `member_orgs` config key is an array, but only one organization
-grant can be active.
-
-Admins issue read-only machine credentials from the Admin tab. Only credential
-hashes are stored in `config/service-accounts.json`; raw credentials are shown
-once. `units:read` grants `GET /api/units`, while `taxonomy:read` grants the
-label, free-label, and graph read endpoints. These credentials cannot ingest or
-administer the pool.
+Pool [xTap](https://github.com/mkubicek/xTap) captures with a group of friends.
+
+xtap-pool has these parts:
+
+- **`extension/`** — a vendored xTap Chrome extension that keeps local captures
+  and syncs them to a shared Hugging Face Space.
+- **`space/`** — a Hugging Face Docker Space that authenticates contributors,
+  validates and deduplicates submissions, and commits them to an immutable raw
+  Bucket log before it returns success.
+- **`explorer/`** — a React web interface for browsing, filtering, searching,
+  and administering the pool.
+- **Consumer API** — revision-consistent enriched conversation-author units for
+  downstream applications.
+
+The private raw Bucket is the system of record. A separate private Bucket holds
+replaceable, checksum-verified SQLite index generations.
+
+See [Move pool storage to an immutable Bucket log](docs/2026-08-12-bucket-object-log-plan.md)
+for the storage contract and cutover procedure.
+[Incremental consumer reads and engagement history](docs/2026-09-07-incremental-consumer-and-engagement-history-plan.md)
+describes cursor-based changes and bounded observation history for downstream applications.
+The other implementation plans under [`docs/`](docs/) record earlier design work.
+
+## Set up a pool
+
+You need Node.js 22 or newer, npm, the Hugging Face CLI, and an active
+`hf auth login` with write access to the target namespace.
+
+```sh
+npm ci
+npm run setup
+```
+
+Setup creates or verifies the private raw and index Buckets and the public
+Docker Space. It configures the Space, publishes the first SQLite index, and
+verifies a storage token with read/write access to exactly both Buckets.
+
+New production pools write their initial configuration through the Bucket log.
+The migration commands below are only for the one-time dataset cutover.
+
+## Diagnose and repair
+
+Inspect an existing pool without changing secrets or restarting it:
+
+```sh
+npm run doctor -- osolmaz/xtap-pool
+npm run --silent doctor -- osolmaz/xtap-pool --json
+```
+
+Run bounded repair when deployment configuration or the scheduled enrichment
+Job is missing or invalid:
+
+```sh
+npm run doctor -- osolmaz/xtap-pool --fix
+```
+
+The repair flow prompts privately for replacement credentials when required.
+The storage token must cover the raw and index Buckets. The inference token must
+be a separate token with the `Make calls to Inference Providers` permission.
+The flow does not copy credentials from another store, print them, or save them
+locally.
+
+Run the two-Job recovery canary while the schedule remains suspended. The
+default 32-call configuration has a combined hard ceiling above $5, so record
+the operator's explicit cumulative approval in the command:
+
+```sh
+npm run doctor -- osolmaz/xtap-pool --fix --canary \
+  --approved-cost-ceiling-usd=25
+```
+
+Enable the recurring schedule only after the canary passes and the operator
+confirms the displayed recurring cost:
+
+```sh
+npm run doctor -- osolmaz/xtap-pool --fix --canary \
+  --approved-cost-ceiling-usd=25 --enable-schedule
+```
+
+The approved ceiling must cover the two-Job canary. It does not change either
+Job's configured run limit.
+
+## Update an existing pool
+
+```sh
+npm run update
+npm run update -- osolmaz/xtap-pool
+# Only after the pinned import and index verification:
+npm run update -- osolmaz/xtap-pool \
+  --verified-bucket-cutover=/secure/path/import-report.json
+```
+
+Update reads the current `RAW_BUCKET` and `INDEX_BUCKET` variables, keeps all
+secrets, uploads the current Space code, and reconciles missing variables. It
+removes the retired `DATASET_REPO` variable. It does not create or rotate
+signing or session secrets.
+
+A reviewed deployment can continue an unfinished enrichment plan from an older
+worker revision only through one exact immutable revision handoff. The updater
+must first prove that the canonical schedule is suspended, no enrichment Job is
+active, and the active plan and checkpoint fully verify. It records the
+secret-free handoff in `.xtap-deployment.json` in the Space commit. It does not
+change any plan, checkpoint, result, claim, receipt, pointer, index, database,
+or manifest in either Bucket.
+
+The worker normally requires the plan worker revision to equal the deployed
+source revision. A mismatch is valid only when the deployment manifest pins the
+exact active generation, activation, run, plan, contract, source snapshot,
+predecessor and target worker revisions, checkpoint pointer, sequence, object,
+digest, and size. The worker verifies the complete handoff and restores the
+pinned checkpoint before it creates a provider or any write-capable runtime
+object. Missing, stale, conflicting, malformed, or forked identities fail
+closed. There is no revision list, wildcard, fallback, or environment override.
+
+Keep the schedule suspended and do not run a paid Job while preparing,
+reviewing, deploying, or restore-testing this handoff. After deployment, verify
+the exact Space repository and runtime revisions, manifest, health, one
+suspended non-concurrent schedule, zero active Jobs, exact restore, zero orphan
+segments, zero provider calls, and unchanged pointers and object listings. A
+later paid retry needs its own cost and non-overlap gate. See
+[Add small worker checkpoints](docs/2026-08-19-small-worker-checkpoints-plan.md#reviewed-worker-revision-handoff)
+for the complete contract and repair plan.
+
+The lower-level deployment script is also available. Supplying
+`XTAP_STORAGE_TOKEN` authorizes the script to bootstrap the index and install
+that value as the Space `HF_TOKEN` secret. An existing pool also requires the
+verified report from the pinned final dataset revision. Run it only after the
+coordinated cutover has paused legacy writers and removed `DATASET_REPO`. It
+checks that the dataset head still equals the imported revision before it can
+bootstrap the index. Use `ALLOW_EMPTY_POOL=1` only for a confirmed new pool.
+
+```sh
+scripts/deploy-space.sh <namespace> # creates resources, then stops for credentials
+PINNED_IMPORT_REPORT=/secure/path/import-report.json \
+  XTAP_STORAGE_TOKEN=... scripts/deploy-space.sh <namespace>
+```
+
+## Import the retired dataset
+
+These commands read one explicit 40-character dataset commit only. They are
+migration tools and are not part of the runtime storage path.
+
+```sh
+npm run storage:import -- \
+  --dataset osolmaz/xtap-pool-data \
+  --revision <40-character-sha> \
+  --raw-bucket osolmaz/xtap-pool-data \
+  --report /secure/path/import-report.json \
+  --work-dir /secure/path/work
+
+npm run storage:verify -- \
+  --dataset osolmaz/xtap-pool-data \
+  --revision <same-sha> \
+  --raw-bucket osolmaz/xtap-pool-data \
+  --report /secure/path/verify-report.json \
+  --work-dir /secure/path/work
+```
+
+The importer validates every approved record, writes deterministic immutable
+segments, creates an exact snapshot, and compares source and target counts and
+digests. Reports contain hashes and counts, not post text. Existing report paths
+are never overwritten.
+
+Build a fresh SQLite projection from only the two Buckets:
+
+```sh
+RAW_BUCKET=osolmaz/xtap-pool-data \
+INDEX_BUCKET=osolmaz/xtap-pool-bucket \
+HF_TOKEN=... \
+npm run index:bootstrap
+```
+
+## Pool administration
+
+Admins manage members and one allowed Hugging Face organization from the Space
+**Admin** tab. Membership is stored in the raw Bucket at `config/pool.json`.
+The Space bootstrap variables remain recovery inputs.
+
+Admins can also issue read-only service credentials. The raw credential is
+shown once; only its hash is stored in `config/service-accounts.json`.
+
+## Scheduled enrichment
+
+Production enrichment runs in one non-concurrent scheduled Hugging Face Job.
+The private raw Bucket is the system of record. The SQLite index is a
+replaceable public read model.
+
+A logical run freezes one source snapshot, queue order, registry order, and
+enrichment contract. Each physical Job restores the small active plan and
+compact checkpoint, then processes only missing ordinals. It does not download
+the public SQLite database during normal queue or registry work. The completed
+logical run restores the frozen base once, applies its verified result batches,
+and publishes a new checksum-verified index.
+
+After publication activates a verified successor, the physical Job stops.
+The next scheduled Job restores that successor and processes only its unresolved
+queue or registry work. This keeps one physical Job tied to one logical run and
+prevents a completed run from starting work that cannot finish before the
+platform timeout.
+
+Run enrichment manually only for local development:
+
+```sh
+npm run enrich --workspace space
+```
+
+The scheduled production command is `space/dist/src/enrich-job-main.js`. The
+schedule must remain non-concurrent and use only `HF_TOKEN` and
+`INFERENCE_TOKEN`. Setup doctor owns schedule creation, replacement, canaries,
+and activation. Keep a replacement schedule suspended until bounded restore,
+interruption recovery, final publication, successor activation, and the
+required two-Job recovery canary have passed.
+
+The checked-in defaults give each Job a 40-minute worker budget, a 45-minute
+platform timeout, and a $10 inference limit. Each scheduled Job runs one logical
+plan and leaves its verified successor for the next Job. The operator-approved
+cumulative ceiling must cover both canary Jobs before either starts. The web
+Space keeps enrichment disabled, and GitHub Actions remains CI-only.
+
+Use an explicit rebuild only as a recovery operation:
+
+```sh
+npm run index:bootstrap
+```
+
+See [Add small worker checkpoints](docs/2026-08-19-small-worker-checkpoints-plan.md)
+for the logical-run contract, production transition gates, and failure rules.
+
+## Join a pool
+
+1. Load `extension/` unpacked from `chrome://extensions` in Developer mode.
+2. Select the extension icon, select **Connect**, and sign in with Hugging Face.
+3. Browse X. Captures sync to the pool automatically.
+
+Local JSONL saving stays available when the optional daemon/native host under
+`extension/native-host/` is installed. Image downloads remain disabled in the
+pool deployment.
+
+## Development
+
+```sh
+npm ci
+npm run check
+```
+
+## License
+
+[MIT](LICENSE)

docs/2026-07-06-implementation-plan.md

@@ -0,0 +1,292 @@
+---
+title: "xtap-pool — Implementation Plan"
+author: "Bob <dutifulbob@gmail.com>"
+date: "2026-07-06"
+---
+
+# xtap-pool — Implementation Plan
+
+## Problem
+
+A group of friends each run [xTap](https://github.com/mkubicek/xTap) (a Chrome
+MV3 extension that passively captures X/Twitter tweets from GraphQL responses
+into daily JSONL files). We want to pool everyone's captures in one private,
+centralized place where:
+
+- uploading happens through each person's own Hugging Face account,
+- every tweet is attributed to whoever captured it,
+- the pool is browsable and filterable (by contributor, author, date, text),
+- nobody can damage anyone else's data (no shared write access, no force
+  pushes),
+- and nobody runs or maintains a server.
+
+## Architecture
+
+Three components in this monorepo, one durable store on the HF Hub:
+
+```
+┌─────────────────────┐
+│ extension/          │  vendored xTap fork ("xtap-pool")
+│ captures tweets     │  keeps local JSONL saving (optional daemon),
+│ on x.com            │  adds a sync queue → POST /api/ingest
+└─────────┬───────────┘
+          │ HTTPS, Bearer pool token (per user, signed)
+          ▼
+┌─────────────────────┐      ┌──────────────────────────────┐
+│ space/              │      │ HF dataset repo (private)     │
+│ private HF Docker   │─────▶│ data/<user>/YYYY/MM/          │
+│ Space (Hono, TS)    │commit│   tweets-YYYY-MM-DD.jsonl     │
+│ the ONLY writer     │      │ system of record, git-backed  │
+└─────────┬───────────┘      └──────────────────────────────┘
+          │ serves
+          ▼
+┌─────────────────────┐
+│ explorer/           │  Vite + React + shadcn/ui
+│ browse & filter UI  │  tweet cards styled after solmaz.io
+└─────────────────────┘
+```
+
+Design invariants:
+
+1. **The dataset repo is the system of record.** The Space is stateless glue:
+   everything it holds locally (SQLite index) is a cache rebuilt from the
+   dataset snapshot on boot. The Space can be deleted and recreated without
+   data loss.
+2. **The Space is the only writer.** Friends never get write access to the
+   dataset repo; a fine-grained HF token scoped to the one dataset repo lives
+   as a Space secret. Force pushes by contributors are structurally
+   impossible.
+3. **Attribution is enforced server-side.** The contributor identity comes
+   from the verified HF OAuth login / signed pool token, never from
+   client-supplied fields.
+4. **Local capture behavior is unchanged.** The vendored extension saves
+   tweets exactly like upstream xTap when the local daemon/native host is
+   installed. Pool sync is additive. Friends who don't want local copies can
+   skip the daemon entirely and run pool-only.
+
+### Why no Gradio
+
+The explorer is a real TypeScript/shadcn UI, and the ingest API is a plain
+HTTP endpoint. Wrapping either in Gradio would add a Python layer whose only
+job is to host things we build elsewhere. A **Docker Space** running a single
+Node process (Hono) serves the API and the built explorer with less moving
+machinery, one language across the repo, and full control over auth/session
+behavior. HF OAuth (`hf_oauth: true`) works identically for Docker Spaces.
+
+## Data model
+
+### Dataset repo layout
+
+```
+data/<hf-username>/YYYY/MM/tweets-YYYY-MM-DD.jsonl
+```
+
+- One JSONL line per captured tweet, in the **exact xTap output format**
+  (see upstream README "Output Format"), plus two stamped fields:
+  - `contributed_by`: HF username (verified server-side)
+  - `pooled_at`: ISO timestamp of ingestion
+- Daily file selected by tweet `captured_at` (UTC), matching xtap-sync
+  conventions.
+- Per-user dedup at ingest: a `(contributed_by, id)` pair is stored at most
+  once; re-submissions with newer `captured_at` update the row (fresher
+  metrics) — implemented as append + latest-wins on read, with periodic
+  compaction as a later optimization (not in MVP).
+- Cross-user dedup happens at query time (same tweet captured by several
+  friends collapses into one card with contributor chips).
+
+Attribution therefore exists at three levels: file path, `contributed_by`
+field, and Space commit messages (`pool: <user> +N tweets (YYYY-MM-DD)`).
+
+### Shared schema
+
+`shared/` package: TypeScript types + [zod] validation for the xTap tweet
+object (superset-tolerant: unknown fields pass through untouched so upstream
+xTap format evolution doesn't break ingestion; only structurally required
+fields are enforced: `id`, `url`, `author.username`, `captured_at`, `text`).
+Used by the extension (pre-send sanity check), Space (ingest validation), and
+explorer (rendering types).
+
+## Auth & UX — the click budget
+
+Target friend onboarding: **install extension, one OAuth authorize, done.**
+
+1. Friend loads the extension (unpacked from a GitHub release zip).
+2. Extension popup shows **"Connect to pool"** → opens the Space's
+   `/connect` page.
+3. The Space app itself is public (a private Space would gate friends who
+   are not repo collaborators before the app even runs) → the app shows
+   **"Sign in with Hugging Face"** (one click if already logged into HF).
+   Everything beyond the sign-in page is enforced in-app by the allowlist;
+   the dataset repo stays private.
+4. `/connect` verifies the HF identity against the pool membership config
+   (individual users plus one allowed HF organization), mints a **pool token**,
+   and renders it in the page DOM.
+5. A content script (matching the Space origin only) picks the token up
+   automatically and stores it in `chrome.storage.local`. Popup flips to
+   "Connected as @user". **No copy-paste, no manual HF token creation.**
+
+Pool token design: stateless signed token (HMAC-SHA256 with a Space secret;
+payload = username + expiry + an optional OAuth-proven organization ID). The
+Space verifies signatures against current pool membership. Explicit user tokens
+last ~180 days; organization-derived tokens are shorter-lived so org removals
+take effect without keeping HF access tokens. Revocation = remove the user/org
+grant or rotate the signing secret (re-connect is one click). A "paste token
+manually" field in extension options is the fallback for browsers where the
+content-script handoff fails.
+
+Only one organization grant is active. Multiple organization grants are
+deprecated because Hugging Face OAuth `orgIds` behaves like a required-org check
+rather than an any-of-orgs check. The `member_orgs` config field stays as an
+array for compatibility, but admin updates replace the active organization.
+
+Explorer access: same HF login; session cookie (signed, httpOnly) issued after
+OAuth. Friends need **zero** repo permissions — the app-level pool membership
+config is the entire access system. Optionally friends can also be given read
+access on the dataset repo for `load_dataset`/DuckDB power use; not required for
+any flow.
+
+## Components
+
+### `extension/` — vendored xTap fork
+
+- Vendor `mkubicek/xTap` at a pinned commit; record SHA + local modifications
+  in `extension/VENDORED.md`. Keep upstream MIT `LICENSE` in place. Diet:
+  keep everything (incl. native host + daemon for local saving); rename
+  extension to "xtap-pool", new icons deferred.
+- Keep upstream code style (vanilla JS, MV3, no bundler) to minimize the
+  vendor diff and keep future re-syncs cheap.
+- New module `lib/pool-sync.js` wired into `background.js` after the
+  existing parse/dedup step:
+  - append captured tweets to a persistent **sync queue**
+    (`chrome.storage.local`, `unlimitedStorage` permission);
+  - flush in batches (≤500 tweets or 20s debounce) via
+    `fetch(POOL_URL + '/api/ingest', {Authorization: Bearer <pool token>})`;
+  - at-least-once delivery: batch removed from queue only on 2xx; server
+    dedup makes retries safe; exponential backoff on failure; queue survives
+    service-worker restarts and offline periods.
+- Popup additions: connect button, sync status (queued / synced counts,
+  last error), pause pool sync independently of capture.
+- Options: Space URL (default baked in), manual token fallback.
+- Manifest additions: `unlimitedStorage`; host permission + content script
+  for the Space origin (token handoff).
+
+### `space/` — HF Docker Space (TypeScript, Hono, Node 22)
+
+Endpoints:
+
+| Route                                               | Auth                                       | Purpose                                                                                                                   |
+| --------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- |
+| `GET /` + static                                    | session                                    | serves built explorer                                                                                                     |
+| `GET /oauth/login` → `/oauth/callback`              | —                                          | HF OIDC code flow, sets session cookie, enforces pool membership                                                          |
+| `GET /connect`                                      | session                                    | mints + renders pool token for extension pickup                                                                           |
+| `POST /api/ingest`                                  | pool token                                 | validate (zod), stamp, dedup, persist                                                                                     |
+| `GET /api/tweets`                                   | session                                    | filters: `contributors`, `author`, `q` (FTS), `since`/`until`, `has_media`, `is_article`, `dedup=true`, cursor pagination |
+| `GET /api/units`                                    | session or `units:read` service account    | complete enriched conversation-author units with revision-bound cursor pagination                                         |
+| `GET /api/labels`, `/api/free-labels`, `/api/graph` | session or `taxonomy:read` service account | revision-consistent enrichment vocabulary and graph reads                                                                 |
+| `/api/admin/service-accounts/*`                     | admin session                              | issue, rotate, and revoke hashed, scoped machine credentials                                                              |
+| `GET /api/contributors`                             | session                                    | per-user counts, last sync                                                                                                |
+| `GET /healthz`                                      | —                                          | liveness                                                                                                                  |
+
+Storage engine inside the Space: **SQLite** (`better-sqlite3`) on ephemeral
+disk + FTS5 for text search. Boot: download dataset snapshot (`@huggingface/hub`)
+→ rebuild index (idempotent). Ingest: insert new rows → append to the day's
+JSONL → **commit to the dataset repo in the same request** (no local-only
+buffering; ephemeral disk must never hold unpersisted data). Commits use
+`@huggingface/hub` `commit()` with retry; concurrent ingests serialized
+through a single writer queue.
+
+Config via Space secrets/variables: `HF_TOKEN` (fine-grained, read/write on
+the one dataset repo), `POOL_SIGNING_SECRET`, `SESSION_SECRET`,
+`ALLOWED_USERS` (initial comma-separated HF usernames), `POOL_ADMINS`
+(bootstrap admins), `DATASET_REPO`. After bootstrap, durable pool membership
+lives in `config/pool.json` in the private dataset repo and is managed from the
+Space Admin tab.
+
+README metadata: `sdk: docker`, `hf_oauth: true`, scopes `openid profile`.
+
+### `explorer/` — Vite + React + TS + Tailwind + shadcn/ui
+
+Views:
+
+- **Feed** — infinite scroll over `/api/tweets`; left filter rail:
+  contributor checkboxes, author filter, date range, media/article toggles,
+  search box; dedup toggle (default on) collapsing cross-user duplicates
+  into one card with contributor chips.
+- **Tweet detail** — full card incl. quoted tweet, article rendering, link
+  out to x.com.
+- **Stats** — per-contributor counts, capture timeline (simple bars).
+
+Tweet rendering ported from the solmaz.io blog's X display (author's own
+prior art, visually faithful to X):
+`~/repos/solmazio/_sass/minima/_x_cards.scss`, `_x_shell.scss`,
+`_x_responsive.scss`, and `_includes/tweet.html` / `x_quoted_tweet.html` /
+`x_tweet_media.html` — avatar/content grid, identity row, media grid,
+quoted-tweet nesting, metrics row; translated to Tailwind + CSS variables,
+light/dark via `prefers-color-scheme` + toggle.
+
+### Quality gates — Slophammer
+
+- Root `slophammer.yml` with `typescript` targets (`shared`, `space`,
+  `explorer`) per the slophammer TS template: strict `tsconfig`, ESLint,
+  Vitest with coverage thresholds, duplication (dry) budget.
+- The vendored `extension/` is excluded from slophammer structural rules
+  (upstream style preserved) but its parser/sync tests run in CI.
+- CI (GitHub Actions): typecheck, lint, tests + coverage for all workspaces,
+  explorer + space build, pinned `slophammer` GitHub Action.
+- Conventional Commits throughout.
+
+## Delivery phases
+
+Each phase lands as commits on this PR's branch; the PR stays green.
+
+1. **Scaffolding** — pnpm workspaces (`shared`, `space`, `explorer`),
+   slophammer config + CI, strict TS/ESLint/Vitest baselines.
+2. **Shared schema** — tweet types + zod validators + fixtures drawn from
+   real xTap output data (sanitized); unit tests.
+3. **Space backend** — OAuth + sessions + allowlist, pool tokens, ingest
+   pipeline (validate → stamp → dedup → JSONL append → hub commit), SQLite
+   index + query API, boot-time rebuild. Hub layer mocked in tests; real
+   round-trip in phase 6.
+4. **Explorer** — components, feed/filters/detail/stats, API client; built
+   output served by the Space.
+5. **Extension vendoring** — vendor xTap at pinned SHA, rename, `pool-sync`
+   queue + flush + backoff, connect flow (content-script token handoff),
+   popup/options additions; Vitest/node tests for queue and flush logic.
+6. **Deploy + E2E** — create private dataset repo + private Docker Space
+   under the `dutifuldev` HF org (exists already; friends never need org
+   membership — the allowlist governs access), set secrets, import existing
+   xTap output under `data/osolmaz/`, live round-trip:
+   connect → ingest fixture batch → verify dataset commit → explore in UI.
+
+## Testing
+
+- Unit: schema validation, token sign/verify, dedup, JSONL day-bucketing,
+  queue flush/backoff, ingest handler (hub mocked), query filters (SQLite
+  in-memory), React component tests for tweet card edge cases (RT, quote,
+  article, media grid).
+- Integration (local): full Space against a temp dir + mocked hub remote;
+  curl ingest with a real signed token; explorer `vite build` served by the
+  Space; manual browser pass.
+- Live (after deploy): real OAuth + ingest + commit verified on the Hub;
+  documented in the PR. Chrome extension manual test on x.com (capture →
+  queue → sync → visible in explorer).
+
+**Known non-automatable pieces:** the HF OAuth browser dance and the Chrome
+extension's on-x.com capture path need manual verification; everything below
+them is covered by tests.
+
+## Risks / notes
+
+- **Space sleep**: free CPU Spaces sleep when idle; first hit wakes in
+  ~30–60 s. Extension backoff absorbs this transparently (retry until 2xx).
+  Upgrade to always-on hardware later if it annoys anyone.
+- **Commit contention**: single-writer queue in the Space serializes hub
+  commits; friend-scale volume is far below HF rate limits.
+- **Upstream xTap drift**: vendored at a SHA with `VENDORED.md`; re-sync is
+  a deliberate manual step. Long-term option: upstream the sync feature into
+  xTap and shrink this vendor copy.
+- **Token leak blast radius**: a pool token only allows _adding_ tweets as
+  that user; dataset write token never leaves the Space; both rotate via
+  Space secrets.
+- **Ephemeral disk**: no unpersisted state by construction (commit per
+  ingest request).

docs/2026-07-27-unit-consumer-api.md

@@ -0,0 +1,104 @@
+---
+title: "Unit consumer API"
+author: "Onur Solmaz <2453968+osolmaz@users.noreply.github.com>"
+date: "2026-07-27"
+---
+
+# Unit consumer API
+
+The machine contract below is superseded by the [incremental HTTP contract](2026-09-07-consumer-http-handoff.md). `/api/units` now requires explicit pinned bootstrap; it no longer returns the old full-read envelope. Explorer whole-unit browsing uses the separate session-only `/api/explorer/units` route.
+
+The [incremental consumer and engagement-history plan](2026-09-07-incremental-consumer-and-engagement-history-plan.md) records the planned replacement for the normal full-read and boot-revision pagination procedures below. It includes repeated-observation delivery, historical counts, removals, and stable source cursors. These changes are not implemented yet; this document continues to describe the existing consumer path.
+
+xtap-pool is the semantic source of truth for downstream applications. Consumers read enriched conversation-author units instead of scanning the private dataset or rebuilding labels, threads, free-label vocabulary, and graph edges.
+
+This document describes the target consumer contract in [Labels and Free Labels Implementation Plan](2026-07-28-labels-and-free-labels-implementation-plan.md) and [Durable Enrichment Implementation Plan](2026-07-28-durable-enrichment-implementation-plan.md). Production keeps serving the previous contract until the replacement reader and worker are complete and the migration recovery canary passes.
+
+## Service accounts
+
+Admins issue service accounts from the explorer's **Admin** tab. A service account has a stable name and explicit read scopes. One or more rotatable keys authenticate it:
+
+- `units:read` authorizes `GET /api/units`.
+- `taxonomy:read` authorizes `GET /api/labels` and `GET /api/free-labels/:name`. It also authorizes `GET /api/graph`.
+
+The raw credential is shown once. xtap-pool stores only its SHA-256 hash in `config/service-accounts.json` in the private dataset. Keys use 256 bits of random secret material, expire after 365 days, can overlap during rotation, and can be revoked individually. Revoking an account removes every active key. Service credentials never authorize ingest, enrichment runs, membership changes, or administration.
+
+Store the raw credential only in the consumer's secret store and send it as:
+
+```http
+Authorization: Bearer xtap_sa_...
+```
+
+Creating or copying the credential into another secret store is an explicit operator action. Do not reuse `HF_TOKEN`, `INFERENCE_TOKEN`, or a contributor pool token.
+
+## Enriched units
+
+```http
+GET /api/units?author_ids=123,456&labels=ai,local-models&label_mode=any&publication=public-original&limit=200
+```
+
+The endpoint accepts contributor and author filters. It also supports text and date filters, media and article filters, and both label kinds. It also accepts `unlabeled` and `limit`. `author_ids` is a comma-separated exact allowlist of immutable X user IDs. It does not fuzzy-match handles. `label_mode` is `any` or `all`; consumers should set it explicitly.
+
+`publication=public-original` excludes an entire unit if any member post is subscriber-only and excludes units containing only retweets. Use this filter for public projections. Apply the same parameter to label and graph reads so label metadata and edge weights come from exactly the publishable unit set.
+
+Each item is one complete conversation-author unit:
+
+```json
+{
+  "revision": "boot-epoch:revision",
+  "units": [
+    {
+      "id": "conversation-id:author",
+      "posts": [],
+      "contributors": ["osolmaz"],
+      "preset_labels": [
+        {
+          "name": "ai",
+          "evidence": [{ "tweet_id": "123", "quote": "frontier AI" }]
+        }
+      ],
+      "free_labels": [
+        {
+          "name": "open-weight-model",
+          "evidence": [{ "tweet_id": "123", "quote": "open-weight frontier model" }]
+        }
+      ]
+    }
+  ],
+  "next_cursor": "opaque"
+}
+```
+
+Posts are ordered oldest first. Units are ordered by their latest post, newest first. Only units enriched under the active contract are returned. Consumer responses include approved free labels only; candidate and rejected labels remain private to administration reads.
+
+## Consistent pagination
+
+The first page returns a result `revision`. Every cursor embeds that revision. A write that changes served tweets, label assignments, free-label registry state, or graph data advances it. Continuing an older cursor returns HTTP `409`:
+
+```json
+{
+  "error": "the unit result changed during pagination; restart from the first page",
+  "current_revision": "..."
+}
+```
+
+Discard the partial result and restart from page one. Invalid cursors return `400`.
+
+The label, free-label detail, and graph responses also return `revision`, `contract_hash`, and numeric `free_label_registry_revision`. When supplied, they echo the shared `cutoff`. Pass the unit revision as their `revision` query parameter. Free-label and graph reads accept the same `author_ids=<csv>`, `labels=<csv>`, `label_mode=any|all`, and `publication=public-original` selection. They return only approved free-label data from matching publishable units. A revision mismatch returns `409`, preventing one publication from mixing source states or counting authors outside the selected boundary.
+
+`GET /api/enrichment/status` returns the same revision and contract metadata, the normalized exact `author_ids` selection, queue totals, and the latest `complete_through` cutoff. Consumers must use that exact cutoff and revision for every subsequent unit or taxonomy read. They do not need to wait for queue totals to reach zero. The totals and cutoff age show how far the projection has caught up.
+
+## Static publication
+
+A static downstream application should:
+
+1. Require `GET /readyz` to be ready.
+2. Load and validate the consumer's exact author-ID allowlist.
+3. Read enrichment status and choose its returned revision and `complete_through` cutoff even when backlog counts are nonzero.
+4. Download every unit page at that cutoff with the exact `author_ids` selection into a temporary local result.
+5. Fetch approved free-label and graph data at the same revision and cutoff with the same author IDs.
+6. Validate the complete response and reject any unit outside the allowlist.
+7. Upload an immutable snapshot.
+8. Replace a small current-manifest object only after the snapshot is uploaded and verified.
+
+On any error, leave the previous manifest unchanged. A later successful run advances the cutoff as scheduled enrichment catches up. Never expose the service credential to browser code and do not silently fall back to raw dataset scanning.

docs/2026-07-28-durable-enrichment-implementation-plan.md

@@ -0,0 +1,263 @@
+---
+title: "Durable Enrichment Implementation Plan"
+author: "Onur Solmaz <2453968+osolmaz@users.noreply.github.com>"
+date: "2026-07-28"
+---
+
+# Durable Enrichment Implementation Plan
+
+## Status
+
+Implemented. Production schedule activation remains gated on deployment, purpose-scoped Job credentials, the live two-Job recovery canary, and explicit approval for recurring paid work. The existing capture and membership contracts stay in place, along with service accounts and consumer authorization.
+
+## Problem
+
+The dataset contains durable tweets and enrichment results, but the work queue lives only in the Space's SQLite cache. A unit is currently considered complete when a result covers its member tweet IDs under the active taxonomy. The result does not bind the text, links, prompt, output schema, or processor version that produced it.
+
+A failed unit stops after three attempts and stays failed until the Space restarts or its membership changes. The scheduler chooses the oldest queue update, so a large historical backlog can keep recent captures out of downstream feeds. Deployments also interrupt the worker because it runs inside the web process.
+
+The system needs one stronger guarantee: every current unit is either backed by a matching durable result or appears as pending, retrying, or blocked work. No unit may disappear from the backlog.
+
+## Boundaries
+
+xtap-pool remains responsible for conversation-author units, preset labels, free labels, the approved free-label graph, privacy filtering, and revision-consistent consumer reads. The classifier produces only preset labels and free labels, as specified in [Labels and Free Labels Implementation Plan](labels-and-free-labels-implementation-plan.md).
+
+This work does not:
+
+- hardcode Local Frontier accounts or any other consumer selection;
+- move model-to-canonical-Hugging-Face association into xtap-pool;
+- change capture attribution, contributor authorization, or dataset ownership;
+- expose private posts or credentials;
+- make SQLite a durable source of truth.
+
+## Completion contract
+
+A boolean `processed` field is not sufficient because unit contents and processor behavior can change.
+
+Every unit gets two deterministic digests:
+
+- `input_hash` covers the ordered semantic input: unit ID, member tweet IDs, text, expanded URLs, reply and quote references, privacy flags, and other fields used by the prompt. Capture metrics that do not affect enrichment stay outside the hash.
+- `contract_hash` covers the taxonomy, prompt template, output schema, normalization rules, configured model, and processor implementation version.
+
+An enrichment result is current only when:
+
+1. its `unit_id` matches the unit;
+2. its `tweet_ids` cover the current membership;
+3. its `input_hash` matches the current semantic input;
+4. its `contract_hash` matches the active processor contract;
+5. its taxonomy version matches the active taxonomy.
+
+Any mismatch creates pending work automatically. Consumer endpoints continue returning only current completed units.
+
+## Durable records
+
+### Enrichment results
+
+Append-only enrichment rows gain `input_hash` and `contract_hash`. They store exactly two classifier outputs. Both may be empty, and every assignment carries evidence from a member tweet:
+
+```json
+{
+  "unit_id": "conversation-id:author",
+  "tweet_ids": ["123"],
+  "input_hash": "<sha256>",
+  "contract_hash": "<sha256>",
+  "preset_labels": [
+    {
+      "name": "ai",
+      "evidence": [{ "tweet_id": "123", "quote": "frontier AI" }]
+    }
+  ],
+  "free_labels": [
+    {
+      "name": "open-weight-model",
+      "evidence": [{ "tweet_id": "123", "quote": "open-weight frontier model" }]
+    }
+  ],
+  "model": "provider/model",
+  "taxonomy_version": 2,
+  "enriched_at": "2026-07-27T00:00:00.000Z"
+}
+```
+
+The worker accepts a preset label only when its name is configured and its evidence is valid. A free label must have a normalized name, valid evidence, and a registry state that permits assignment. New free labels enter the registry as private candidates. Approved labels may appear in consumer reads, while rejected labels are discarded and fed back to the prompt as negative examples.
+
+The dataset commit must succeed before SQLite marks the unit complete. Replaying the dataset reconstructs the same completed set after a crash or replacement deployment.
+
+### Free-label registry events
+
+The dataset also stores append-only discovery and lifecycle events for free labels. Each decision records its rule, supporting counts, representative quotes, timestamp, and registry revision. SQLite rebuilds all registry states from these events and current enrichment rows.
+
+Registry state is operational metadata rather than another classifier output. The model still returns only preset labels and free labels. Candidate and rejected free labels cannot contribute to consumer units, counts, filters, or graph data.
+
+### Attempt events
+
+Batch commits also append compact attempt events containing the unit ID, both hashes, attempt number, outcome, timestamp, retry time, and a bounded error classification. These events preserve retry and blocked state across restarts without storing credentials or provider response bodies.
+
+The SQLite queue remains a projection. It holds the current status, attempts, next retry time, lease owner and expiry, first queued time, latest unit activity time, and bounded last error. It is rebuilt from tweets and durable enrichment records on boot.
+
+## Worker command
+
+Enrichment moves behind a standalone bounded worker command. The API Space may invoke that command during development, but production scheduling must not depend on the web server's lifetime.
+
+One run follows this order:
+
+1. Rebuild or refresh the desired work set.
+2. Recover expired leases.
+3. Claim a bounded batch with an expiring lease.
+4. Build prompts from the claimed input hashes.
+5. Call the configured inference provider with a request timeout.
+6. Validate and normalize the complete response.
+7. Commit results and attempt events to the dataset.
+8. Verify the commit, apply it to SQLite, and release the leases.
+9. Stop at the configured unit, token, time, error-rate, or cost ceiling.
+
+The first production version uses one worker. More workers require a durable shared lease implementation and a separate cost review.
+
+## Hugging Face deployment and repair
+
+Production execution stays on Hugging Face. The Docker Space serves capture, administration, and read APIs. A non-concurrent scheduled Hugging Face Job runs the standalone worker on `cpu-basic` with an explicit timeout. The private dataset remains the durable source of truth. GitHub Actions is limited to CI and cannot run, schedule, or hold credentials for production enrichment.
+
+The setup doctor and repair command owns the scheduled Job as part of deployment reconciliation. It must:
+
+1. inspect active and scheduled Jobs before changing state;
+2. validate the exact Space revision, dataset repository, worker command, model, taxonomy, pricing, ceilings, timeout, and schedule;
+3. accept dataset-writer and inference tokens only through hidden interactive input or explicitly named environment inputs;
+4. verify dataset read/write access and inference-provider permission before creating a Job;
+5. create or replace the schedule with concurrency disabled and the schedule suspended;
+6. attach the two purpose-scoped values as encrypted Hugging Face Job secrets without printing or persisting them locally;
+7. trigger the two-Job bounded recovery canary through that suspended schedule;
+8. verify durable outputs, strict `JOB_ID`-bound receipts, source revision, contract continuity, cost accounting, continuation in a second physical Job, and absence of overlap; and
+9. resume the schedule only after every canary check succeeds and recurring paid work is explicitly approved.
+
+Space secrets are scoped to the Space and their values are write-only. Repair cannot recover them by name, silently copy them into a Job, or mint access tokens. When Job secrets are missing or need rotation, the operator supplies the original values or purpose-scoped replacements. If required input is unavailable, capability validation fails, a transient Hub error prevents a trustworthy read, required approval is absent, or an active worker cannot be safely stopped at a durable boundary, repair leaves scheduling suspended and reports the blocker.
+
+Hugging Face Job configuration is immutable for a physical run. Repair suspends the existing schedule and verifies that no matching Job is active before replacing a mismatched schedule. It does not mutate unrelated Jobs. Manual canaries and automatic runs use the same schedule identity and non-concurrency policy.
+
+`doctor --fix` creates or repairs the suspended schedule. `doctor --fix --canary` runs two physical Jobs without resuming cron. `doctor --fix --canary --enable-schedule` presents the schedule and per-run ceiling, then requires explicit approval before recurring execution. The default ceiling is $2 of inference and 400,000 total tokens per physical Job. Including worst-case `cpu-basic` time, the two-Job canary has a cumulative hard ceiling below $4.02.
+
+## Scheduling and retries
+
+Each claim reserves equal bounded capacity for recent and old work:
+
+- newest pending units by latest unit activity;
+- oldest pending units by first queued time.
+
+Unused capacity from either side can be filled by the other. This keeps recent feeds moving while guaranteeing that continuous ingest cannot starve historical work.
+
+Transient failures such as timeouts, rate limits, and provider 5xx responses use exponential backoff with jitter. Invalid model output gets a small bounded retry budget. A unit becomes `blocked` after the configured retry policy is exhausted. Blocked units remain visible, are not counted as complete, and receive infrequent bounded retries. A changed input or processor contract resets their retry state.
+
+There is no silent terminal drop. The operational guarantee is that every unit is either current and complete or visible with a reason that prevents completion.
+
+## Freshness and status API
+
+Add a revision-consistent endpoint authorized by `taxonomy:read`:
+
+```http
+GET /api/enrichment/status?author_ids=123,456&publication=public-original
+```
+
+The response reports:
+
+- total, pending, running, retrying, blocked, and completed units;
+- oldest pending and newest completed activity times;
+- a `complete_through` timestamp for which every selected unit at or before that time has a current result;
+- the current result revision and processor contract hash.
+
+`author_ids` uses the same exact immutable-ID semantics as unit and taxonomy reads. The endpoint cannot filter by enrichment labels because pending units do not have current labels yet.
+
+Unit, label, free-label detail, and graph reads gain a shared activity cutoff so a consumer can request one complete window ending at `complete_through`. The cutoff participates in the revision and must be applied identically to every response. Consumers may publish that completed window while queue totals remain nonzero. Backlog size and cutoff age describe staleness; they do not invalidate the window.
+
+## Existing-result migration
+
+The migration is a hard replacement of the current completion rule.
+
+An existing result can be adopted without another inference call only when all of the following are provable:
+
+- its tweet IDs exactly cover current membership;
+- no semantic capture is newer than the result;
+- its taxonomy and model match the contract being adopted;
+- it contains only preset and free-label assignments;
+- every assignment carries evidence that passes the current schema and normalization rules.
+
+All earlier model-generated free labels and concepts are discarded from the active projection. They are never adopted, copied into the new assignment arrays, promoted, or used to seed candidate records. Legacy rows remain only as append-only history and cannot contribute evidence, counts, filters, or graph edges. Curated preset taxonomy entries remain because their source is `config/labels.json`. Every source unit must receive a new evidence-bearing result under the active contract.
+
+The migration appends a current row with both hashes. Every other result is queued for processing. After migration, runtime reads do not accept rows without the new hashes.
+
+Before reprocessing any unverifiable rows, record their count, measured throughput, projected inference cost, the configured full-run ceiling, and the outputs that will be reused. Once the implementation checks and recovery canary pass, the worker continues through all queued units instead of stopping at a pilot-only spending boundary. Consumers can roll out against the current completed cutoff while this background convergence continues.
+
+## Observability and controls
+
+Run receipts record successful units, retries, blocked units, provider calls, tokens, elapsed time, and cost when the provider reports it. They also record emitted assignments, labels discarded for missing evidence or rejected names, new candidates, and registry decisions. The Admin UI shows queue counts, cutoff age, the active contract hash, recent error classes, whether a worker is active, and the candidate, approved, and rejected free-label counts. Queue counts and cutoff age are operational signals rather than publication gates.
+
+The worker stops automatically when:
+
+- the configured cost or runtime ceiling is reached;
+- the batch error rate crosses its limit;
+- invalid or ungrounded label output crosses its limit;
+- the dataset commit or result verification fails;
+- the model, method, credential state, contract, or observed cost differs from the recorded full-backlog run.
+
+A deterministic shared defect pauses further claims until the worker code and affected outputs are checked.
+
+## Tests
+
+### State and hashing
+
+- Stable semantic input produces a stable `input_hash`.
+- Text, links, privacy flags, or membership changes invalidate completion.
+- Metrics-only changes do not trigger semantic reprocessing.
+- Taxonomy, prompt, schema, registry revision, model, or processor changes alter `contract_hash`.
+- Rows with missing or mismatched hashes are never served as current.
+- Empty preset and free-label arrays are valid completed output.
+- Preset and free-label evidence must name a member tweet and quote its text.
+- `deixis`, unsupported `manufacturing`, and `quality-philosophy` regression fixtures cannot reach public reads.
+
+### Queue and recovery
+
+- New and stale units enter the queue exactly once.
+- A successful dataset commit happens before `done` is visible.
+- A crash before commit retries the unit.
+- A crash after commit but before SQLite apply converges on replay without another model call.
+- Expired leases return to pending work.
+- Retry timing survives restart.
+- Blocked work stays visible and resets only under the stated rules.
+- Newest and oldest work both progress under continuous ingest.
+- Doctor reports missing, suspended, active, and mismatched Hugging Face Job schedules without exposing secrets.
+- Repair creates replacement schedules suspended and enables them only after a successful bounded canary.
+- Repair refuses missing credentials, invalid capabilities, transient Hub failures, overlapping workers, and changed paid-run assumptions.
+
+### API and consumer consistency
+
+- Status counts match the selected immutable author IDs.
+- Mixed-author and missing-author-ID units fail the selection.
+- `complete_through` never passes pending or blocked selected work.
+- Unit, label, free-label detail, and graph reads apply the same cutoff and revision.
+- A consumer can publish the completed cutoff while selected pending, retrying, or blocked work remains after it.
+- Candidate and rejected free labels cannot affect consumer units, counts, filters, nodes, or edges.
+- A stale cutoff or revision returns `409` instead of mixed data.
+
+### Live canary
+
+The production canary processes representative old, recent, changed, failing, and multilingual units. It must persist at least two batches, survive an interruption, resume without duplicates, and prove checksums and receipts. When it passes, the same worker resumes and processes the remaining backlog under the recorded full-run ceiling. Consumer publication may start from the current completed cutoff during that work.
+
+## Delivery order
+
+1. Add hash and state-machine tests around the current store.
+2. Replace the enrichment schema with evidence-bearing preset and free-label assignments, including valid empty output.
+3. Add attempt and free-label registry event schemas plus replay support.
+4. Replace completion checks with the two-hash contract.
+5. Add leases, retry timing, fair claims, and the bounded worker command.
+6. Restrict consumer reads and graph materialization to approved free labels.
+7. Add the status endpoint, shared cutoff, registry metrics, and Admin UI reporting.
+8. Add doctor and repair reconciliation for a non-concurrent suspended Hugging Face scheduled Job; remove production scheduling and credentials from GitHub Actions.
+9. Run the no-inference migration analysis, record the reuse and cost report, and configure the full-run ceiling.
+10. Run the bounded recovery canary through the suspended schedule, including the known loose-label regression fixtures.
+11. Deploy the new reader and worker together, remove the old queue and generated-concept paths, and resume the schedule only after the canary passes.
+12. Publish and verify a consumer snapshot at the current completed cutoff without waiting for the backlog to drain.
+13. Monitor queue convergence, cutoff advancement, registry replay, consumer revisions, dataset receipts, cost, Job receipts, and restart recovery in production.
+
+## Completion criteria
+
+The work is complete when every captured unit is either backed by a verified current result or remains visible as pending, retrying, or blocked work, and consumers can publish an internally complete revision and cutoff. No earlier model-generated label or concept assignment may enter the replacement registry or consumer projection. Candidate and rejected labels cannot enter public reads. Registry replay must be deterministic, and restarts or deployments cannot lose work. Recent and historical queues must both advance. The full repository checks and live recovery canary must pass.
+
+If a durable lease, resumable output, defensible per-run ceiling, or consistent consumer cutoff cannot be established, the implementation is not complete. Report that blocker instead of claiming readiness. Once those safeguards pass, scheduled Jobs continue the historical backlog after rollout.

docs/2026-07-28-labels-and-free-labels-implementation-plan.md

@@ -0,0 +1,177 @@
+---
+title: "Labels and Free Labels Implementation Plan"
+author: "Onur Solmaz <2453968+osolmaz@users.noreply.github.com>"
+date: "2026-07-28"
+---
+
+# Labels and Free Labels Implementation Plan
+
+## Status
+
+Planned. This replaces the current three-output enrichment contract. The classifier will produce only preset labels and free labels. The durable worker, retries, completion hashes, and freshness API are specified in [Durable Enrichment Implementation Plan](durable-enrichment-implementation-plan.md).
+
+## Purpose
+
+xtap-pool classifies each conversation-author unit with two kinds of labels:
+
+- Preset labels come from the configured taxonomy.
+- Free labels describe specific subjects that the preset taxonomy does not cover.
+
+Both may be empty. Every assignment carries evidence from the source unit. The model does not generate concepts, entities, candidates, or any other classification output.
+
+The system keeps discovering free labels without publishing one-off mistakes as navigation categories. New free labels accumulate evidence in a private registry before they can appear in public filters or the graph.
+
+## Classification contract
+
+The model returns exactly two arrays per unit:
+
+```json
+{
+  "units": {
+    "conversation-id:author": {
+      "preset_labels": [
+        {
+          "name": "ai",
+          "evidence": [{ "tweet_id": "123", "quote": "frontier AI" }]
+        }
+      ],
+      "free_labels": [
+        {
+          "name": "open-weight-model",
+          "evidence": [{ "tweet_id": "123", "quote": "open-weight frontier model" }]
+        }
+      ]
+    }
+  }
+}
+```
+
+A label assignment contains a normalized name and one or more evidence records. Each evidence record identifies a member tweet and quotes text from it. Evidence is metadata on a label assignment; it is not another kind of label.
+
+The response rules are:
+
+- `preset_labels` may contain only exact names from `config/labels.json`.
+- `free_labels` contains at most five lowercase-dash names.
+- Either array may be empty. Thin replies, emoji-only posts, and units without a substantive subject should normally return no free labels.
+- Every quote must be a verbatim substring of the named tweet. Code checks membership and the quote before accepting the assignment.
+- A free label must name a specific subject supported by its evidence. Grammatical features, discourse forms, pronoun categories, generic nouns, and unsupported abstractions are invalid.
+- Free labels that match a rejected registry entry are discarded.
+- One invalid assignment does not invalidate valid assignments in the same unit. The worker records every discarded assignment and its reason.
+
+The prompt, output schema, normalization rules, rejection registry, preset taxonomy, and configured model participate in `contract_hash`.
+
+## Free-label lifecycle
+
+Preset labels are approved by configuration. Free labels use three internal lifecycle states:
+
+- `candidate` means the label has valid unit evidence but has not earned public use.
+- `approved` means the label may appear in consumer responses and their filters, counts, or graph.
+- `rejected` means the label cannot be assigned or rediscovered under the current registry revision.
+
+Registry state stays outside the classifier response. The model still returns only preset labels and free labels.
+
+A new valid free label enters the registry as `candidate`. Candidate assignments remain durable and visible to administrators, but consumer APIs omit them. The registry stores append-only discovery and lifecycle events. SQLite projects the current state and evidence counts from enrichment rows and registry events.
+
+Rejected labels that overlap the current batch are included in the prompt as negative examples. This prevents mistakes from feeding themselves back into later calls. Normalization also rejects known generic abstractions and unsupported suffix patterns. A rejected label can return only through an explicit registry event under a new registry revision.
+
+## Promotion and rejection
+
+Promotion requires independent evidence from more than one model response.
+
+A free label backed by a verified exact Hugging Face model or dataset repository may be approved from one unit. Other surface-grounded names must appear in at least five units from three authors on two days. Labels whose names are not directly present in their evidence require at least fifteen units from eight authors on two days and a constrained review call over the stored quotes.
+
+The review call decides only whether the supplied candidate is a specific, useful AI or local-model subject. It cannot rename the label or introduce another label. Invalid output leaves the candidate unchanged.
+
+A candidate is rejected when it matches a deterministic rejection rule, fails constrained review, or remains below the evidence threshold for 30 days. Every decision records the rule and counts plus representative quotes. It also records the timestamp and contract revision. Thresholds are configuration covered by tests and included in the registry revision.
+
+Approved recurring free labels are reported as possible additions to the preset taxonomy. Changing the preset taxonomy remains a deliberate configuration change because it redefines broad relevance and triggers a new `contract_hash`.
+
+## Derived graph
+
+The old model-generated `concepts` output and global concept vocabulary are removed. Graph nodes come from approved free labels, and graph edges count their co-occurrence in current units. Candidate and rejected labels never contribute public nodes, counts, edges, filters, or inline links.
+
+The API exposes approved free-label vocabulary and graph data through revision-consistent reads. Display names and aliases, when needed for presentation, belong to registry metadata and never create another classifier output.
+
+## Durable data
+
+An enrichment result stores the two evidence-bearing assignment arrays:
+
+```json
+{
+  "unit_id": "conversation-id:author",
+  "tweet_ids": ["123"],
+  "input_hash": "<sha256>",
+  "contract_hash": "<sha256>",
+  "preset_labels": [
+    {
+      "name": "ai",
+      "evidence": [{ "tweet_id": "123", "quote": "frontier AI" }]
+    }
+  ],
+  "free_labels": [
+    {
+      "name": "open-weight-model",
+      "evidence": [{ "tweet_id": "123", "quote": "open-weight frontier model" }]
+    }
+  ],
+  "model": "provider/model",
+  "taxonomy_version": 2,
+  "enriched_at": "2026-07-27T00:00:00.000Z"
+}
+```
+
+The dataset remains the system of record. SQLite is rebuilt from tweets, enrichment rows, attempt events, and free-label registry events. A result is durable before the queue marks its unit complete.
+
+## API and explorer
+
+`GET /api/units` returns evidence-bearing preset and approved free-label assignments. Label filters use assignment names. Candidate free labels are available only through authenticated administration endpoints.
+
+`GET /api/labels` returns preset-label counts, approved free-label counts, queue depth, and enrichment coverage. `GET /api/free-labels/:name` returns one approved free label, its unit count, and related approved labels. `GET /api/graph` returns approved free-label nodes and co-occurrence edges.
+
+The explorer shows preset labels and approved free labels on units. Its graph and detail pages use the approved free-label projection. The Admin UI shows candidate evidence, rejected labels, lifecycle events, and promotion progress.
+
+## Observability
+
+Each worker receipt records:
+
+- emitted preset and free-label assignments;
+- assignments discarded for missing evidence, invalid names, or rejection state;
+- candidate labels first observed;
+- candidates approved or rejected;
+- provider calls, token use, elapsed time, and cost.
+
+The worker stops when invalid or ungrounded output crosses its configured rate ceiling. Admin status reports the registry state counts, top candidates with representative quotes, recent decisions, and candidate creation per thousand units.
+
+## Migration
+
+This is a hard replacement of the current enrichment contract.
+
+All earlier model-generated free labels and concepts are discarded during migration. None are grandfathered, promoted, copied into the new assignment arrays, or used to seed candidate records. Old enrichment rows remain in the append-only dataset as historical records, but the replacement reader ignores their label output. The new registry starts without approved or candidate entries inherited from model output. Curated preset labels remain because their source is `config/labels.json`.
+
+Known bad names such as `deixis`, `quality-philosophy`, and unsupported `manufacturing` assignments become rejection rules and regression fixtures. This prevents rediscovery without treating the old assignments as evidence.
+
+The changed schema and prompt plus the normalization rules and registry revision produce a new `contract_hash`. The durable worker reprocesses every source unit under the measured full-run ceiling. Labels enter the new registry only from evidence validated under this contract. Public consumers may publish the completed prefix under the new contract while the worker continues through later units. Each replacement snapshot uses one revision and cutoff.
+
+## Tests
+
+The contract tests cover empty arrays, exact preset-name validation, free-label limits, tweet membership, verbatim evidence, duplicate normalization, and rejected names. Regression fixtures require an emoji-only reply to produce no free labels, a DGX enclosure post to avoid unsupported `manufacturing`, and the word `frontier` to avoid `quality-philosophy`.
+
+Lifecycle tests cover deterministic replay, each legal state transition, threshold boundaries, the verified Hub fast path, constrained-review abstention, stale-candidate rejection, and reopening under a changed registry revision. Migration tests prove that old free labels and concepts do not seed registry entries or assignments. Public API tests prove that legacy, candidate, and rejected labels cannot affect units, counts, filters, graph nodes, or edges.
+
+Recovery tests prove that evidence-bearing rows and registry events survive interruption, replay to the same state, and become visible only after their dataset commits succeed.
+
+## Delivery order
+
+1. Replace the model response and enrichment-row schemas with the two evidence-bearing label arrays.
+2. Add validation, empty-output fixtures, and the known-bad regression cases.
+3. Add the append-only free-label registry and deterministic replay.
+4. Restrict consumer reads and graph materialization to approved free labels.
+5. Add promotion, rejection, negative-prompt retrieval, and Admin reporting.
+6. Remove model-generated concepts and their vocabulary path.
+7. Integrate the new contract with durable hashes and queue recovery, including receipts and freshness reads.
+8. Run a no-inference migration report, then the bounded pause-resume canary.
+9. Reprocess the backlog under the recorded cost ceiling, publish the current completed cutoff, and let scheduled runs advance it as more work finishes.
+
+## Completion criteria
+
+The work is complete when every current unit is either backed by a durable evidence-bearing result or remains visible in the durable queue, and consumers can read a complete prefix at one revision and cutoff. No legacy model-generated label or concept assignment may contribute to the new registry, consumer reads, counts, filters, or graph. The model must accept empty arrays, and candidate or rejected free labels cannot enter public reads. Registry replay must be deterministic. The old generated-concept path must be absent. All local and production checks, including recovery, must pass.

docs/2026-07-29-parallel-enrichment-plan.md

@@ -0,0 +1,193 @@
+---
+title: Parallel enrichment plan
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-07-29
+---
+
+# Parallel enrichment plan
+
+## Status
+
+Implemented. The worker supports up to 32 provider calls in one physical Job, and production has completed 32-way runs with durable receipts. This update makes 32 the persistent setup default. The Hugging Face schedule remains physically non-concurrent so one coordinator owns queue selection, cost limits, registry order, Bucket commits, and receipts.
+
+The normal six-hour schedule uses a $10 run limit. That limit covers the $8 reservation bound for one full 32-call wave. A backlog repair may temporarily use a $100 run limit and a 5.5-hour worker limit under an explicit cumulative budget. The normal limits return after the backlog clears, while concurrency stays at 32.
+
+## Current performance
+
+The worker groups six conversation-author units into each call to `zai-org/GLM-5.2:fireworks-ai`. Forty-five production receipts with concurrency 32 completed 111,079 units in 52.42 worker-hours for $179.171605. The aggregate rate was 2,119 durable units per hour at $1.61 per thousand units.
+
+The August 15 backlog contains 48,495 pending units. The aggregate history projects about 23 worker-hours and $78 of inference. The observed rate varies widely with provider errors and input mix, so the operating range is 15 to 40 worker-hours with a $100 expected repair ceiling. Each useful result and attempt event is durable before a worker exits.
+
+## Decision
+
+The implementation will support a configurable maximum from 1 through 32 concurrent inference calls. Production will target 32 immediately, as approved, unless provider errors require a reduction.
+
+Each request keeps the existing six-unit prompt contract. Sixteen requests place at most 96 units in inference at once. Thirty-two requests place at most 192 units in inference at once.
+
+The primary metric is durable completed units per wall-clock hour. Moving from sequential execution must produce at least twice the baseline goodput to justify the additional coordinator logic. Cost per durable completed unit may rise by no more than 5 percent. Provider failures, invalid output, evidence rejection, and dataset commit failures remain veto gates.
+
+Under ideal provider behavior, 16 concurrent calls would process the current queue in under two hours. A practical estimate is one to three hours after Hub commit delays and provider backoff. This estimate is provisional until the production-parity canary measures it.
+
+## Process boundary
+
+Parallel Hugging Face Jobs are outside this design. Every Job currently rebuilds a private SQLite projection and obtains leases from that local database. Separate Jobs cannot see one another's leases. They can select the same units, pay for duplicate calls, and race while replacing shared JSONL shards.
+
+One physical Job will retain the following responsibilities:
+
+- rebuild the dataset snapshot and claim one fair bounded queue.
+- admit inference calls under every shared run ceiling.
+- serialize durable writes and registry revisions.
+- release unprocessed leases before exit.
+- publish one strict receipt tied to its Hugging Face `JOB_ID`.
+
+The scheduled Job keeps `concurrency: false`, `attempts: 1`, and its revision-pinned image. GitHub Actions remains CI-only.
+
+## Request coordinator
+
+`runEnrichTick` will replace the sequential call loop with a coordinator that has separate inference and commit lanes.
+
+The inference lane may have several HTTP requests in flight. Each dispatch gets a monotonically increasing sequence number and an immutable reservation containing:
+
+- the exact unit IDs and input hashes.
+- the serialized prompt digest and prompt byte count.
+- the completion-token limit.
+- the maximum charge permitted for the call.
+- the dispatch time and current processor contract hash.
+
+The coordinator records the reservation before sending the request. Missing usage, missing pricing, a timeout after dispatch, or an orphaned request is charged conservatively against the reservation. There are no hidden HTTP retries. A later unit retry is another durable attempt and needs a new reservation.
+
+The commit lane receives completed calls through a bounded queue. It commits them in dispatch order so free-label registry revisions and attempt events remain stable. One call can finish while an earlier call is still running, but its result cannot update SQLite or release its reservation until every earlier sequence has reached a durable terminal event.
+
+Dataset writes remain serialized through `DatasetMirror.commitBatch`. The dataset commit must succeed before the local queue marks any unit complete. A failed commit writes the existing durable failure form and leaves the affected units eligible for replay under the current policy.
+
+## Admission control
+
+A concurrency maximum caps active calls. The worker repeatedly claims one bounded wave and starts it immediately, subject to the shared cost and runtime ceilings. It does not stop after an arbitrary unit count or aggregate token count.
+
+Provider pressure reduces later waves quickly:
+
+- a rate limit, timeout, or provider 5xx response halves the active limit, rounded down to at least one.
+- a reduced limit remains in force for the rest of that Job.
+- reaching the run error ceiling closes admissions immediately.
+- admitted calls finish and produce durable success or failure events before the worker exits.
+
+A full 32-call wave can reserve at most $8 under the current $0.25 per-call bound. Provider errors reduce the next wave before further requests are admitted.
+
+The coordinator checks capacity before every dispatch. It includes all active reservations when evaluating the run ceilings. With the current `$0.25` per-call bound, 16 active calls reserve $4 and 32 active calls reserve $8. The worker cannot admit a call that would exceed the remaining cost capacity.
+
+Elapsed time stops new admissions. It does not abandon in-flight calls or skip their durable accounting. A platform timeout remains an emergency limit and must leave enough time for the request timeout, ordered commit queue, receipt write, and lease release.
+
+## Configuration
+
+Add `ENRICH_MAX_CONCURRENT_CALLS` as a strict positive integer in the following locations:
+
+- `space/src/config.ts` for worker configuration.
+- `space/src/enrich-command.ts` when constructing worker dependencies.
+- `setup/src/enrichment-job.ts` for defaults, validation, desired schedule state, and exact schedule matching.
+- deployment and doctor output so the effective value is visible before activation.
+
+The setup schedule template defaults to `32`. Setup stores this value as a Space variable, and doctor includes it in the exact schedule contract. Deployments and schedule repair therefore keep 32 unless an operator deliberately chooses a lower safe value. The generic worker parser keeps its fail-safe default of 1 when no deployment setting exists.
+
+The normal run limit is $10 because a full wave reserves at most $8. Backlog repair uses temporary $100 and 5.5-hour limits. The replacement schedule is always created suspended, checked with the bounded recovery canary, and enabled only after the canary passes. Canary commands whose hard ceiling is $5 or more must include `--approved-cost-ceiling-usd` with the operator's approved cumulative limit.
+
+Values above 32 fail configuration validation. A 32-call run also requires at least $8 of available reservation capacity under the current per-call bound. The configured run ceiling must cover in-flight reservations plus final commit and receipt work.
+
+Extend the strict enrichment receipt in `shared/src/enrichment.ts` with:
+
+```json
+{
+  "configured_concurrency": 16,
+  "peak_concurrency": 16,
+  "provider_backoffs": 2,
+  "reservation_peak_usd": 4,
+  "commit_queue_peak": 11
+}
+```
+
+These fields are required for the new processor revision. Earlier receipts remain historical and do not satisfy current worker-heartbeat checks after the contract change.
+
+## Correctness rules
+
+Parallel execution must preserve every current contract:
+
+1. The classifier still returns exactly `preset_labels` and `free_labels`.
+2. Every accepted assignment still cites an exact retained member-tweet quote.
+3. The semantic input hash and reservation bind the prompt content. They also bind its candidates and serialized bytes.
+4. Dataset rows, attempt events, registry events, and receipts become visible only after verified Hub commits.
+5. Queue status and leases remain a rebuildable projection of durable records.
+6. Every registry state stays outside model output, including candidate and approved or rejected states.
+7. Every current run ceiling includes all in-flight work.
+8. A changed contract invalidates stale projections and requeues affected units.
+9. Unprocessed claims are released at every normal or exceptional exit.
+10. Replay produces the same completed set without duplicate current rows or missing attempts.
+
+A deterministic defect in prompt construction, parsing, evidence validation, registry sequencing, or dataset commits closes admissions for the whole Job. Provider-specific transient errors use the bounded reduction and retry policy described above.
+
+## Tests
+
+Unit tests will use deferred fake inference promises so completion order can differ from dispatch order. They must prove that durable order, registry revisions, and receipt totals remain deterministic.
+
+Coverage must include:
+
+- concurrency values from 1 through 32 plus invalid zero and over-cap values.
+- peak in-flight requests never exceeding the configured or dynamically reduced limit.
+- exact prompt and cost reservations before dispatch.
+- conservative charging when usage or pricing is absent.
+- token and cost ceilings counting every active reservation.
+- out-of-order model responses committed in dispatch order.
+- one slow request applying bounded backpressure to the commit queue.
+- timeout, 429, provider 5xx, malformed output, and missing-unit responses.
+- no further admission after the error, elapsed-time, cost, token, or discard ceiling.
+- all admitted calls producing durable terminal events before exit.
+- commit failure leaving SQLite incomplete and replay recovering correctly.
+- registry events keeping unique consecutive revisions.
+- receipt replay rejecting malformed or wrong-contract concurrency fields.
+- schedule repair treating a concurrency-setting mismatch as a replacement.
+- production schedules remaining non-concurrent with zero platform retries.
+
+The repository quality gate remains `npm run check`. Coverage, mutation testing, setup tests, Space tests, and the two-Job recovery canary must pass for the new processor revision.
+
+## Throughput canary
+
+The canary uses the production model with six units per call. It keeps the current taxonomy and prompt contract while sampling representative old and recent units. It records every request because aggregate Job duration alone is insufficient.
+
+Run a bounded production-parity validation at concurrency 32. The worker records peak concurrency, reservations, ordered commits, and provider backoffs in its receipt. Any timeout, rate limit, or provider 5xx halves subsequent waves.
+
+The autonomous canary keeps its cumulative hard ceiling below $5. It publishes durable results and attempt events after each completed call, then proves replay from the Hub snapshot. The production run does not start from an unverified local checkpoint.
+
+Continuing at 32 requires all of the following evidence:
+
+- at least 2 times the baseline durable units per hour.
+- cost per durable completed unit within 5 percent of baseline.
+- no duplicate current rows, registry revision gaps, missing attempt events, or receipt disagreement.
+- provider error rate below the configured ceiling and no meaningful increase over baseline.
+- successful crash recovery with in-flight calls and an occupied commit queue.
+- exact read-back of every cache object and each shard, receipt or registry object used by the verifier.
+
+If the bounded run has provider failures or violates a ceiling, the replacement schedule is reduced to the highest stable setting before another paid run.
+
+## Deployment
+
+Implementation and deployment follow this order:
+
+1. Add coordinator tests and the new strict receipt fields.
+2. Implement bounded inference admission and ordered commits with the persistent default set to 32.
+3. Run the full repository checks and review the change against `main`.
+4. Deploy the exact Space revision without changing the active schedule.
+5. Replace the schedule in suspended state at concurrency 32 and run the bounded recovery canary.
+6. Inspect every measured cost and error. Check request latency, commit latency, replay behavior and current API projections.
+7. Trigger the approved production run and verify its receipt before enabling cron.
+8. Resume the single exact schedule and continue publishing completed cutoffs while backfill drains.
+
+Rollback creates a suspended replacement schedule at the highest measured stable concurrency under the same source revision, or deploys the previous reviewed revision. It never overlaps physical Jobs, deletes durable enrichment records, or rewrites historical receipts.
+
+## External limits
+
+Hugging Face does not publish one fixed concurrency limit for routed Fireworks requests. Fireworks documents a 600 requests-per-minute serverless account rate together with adaptive token limits and overload responses. At the measured 19 to 42 second latency, 16 active calls imply roughly 23 to 51 requests per minute. That leaves substantial request-rate headroom, while token limits and model capacity still require measurement.
+
+References:
+
+- [Hugging Face Inference Providers](https://huggingface.co/docs/inference-providers/en/index)
+- [Hugging Face Fireworks provider](https://huggingface.co/docs/inference-providers/en/providers/fireworks-ai)
+- [Fireworks serverless rate limits](https://docs.fireworks.ai/serverless/rate-limits)
+- [Fireworks inference error codes](https://docs.fireworks.ai/guides/inference-error-codes)

docs/2026-08-04-durable-enrichment-index-plan.md

@@ -0,0 +1,219 @@
+---
+date: 2026-08-04
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+title: Add a durable enrichment index
+tags: [enrichment, sqlite, hugging-face, operations]
+---
+
+# Add a durable enrichment index
+
+## Goal
+
+Every fresh xtap-pool process currently reconstructs SQLite from the complete private dataset. One production Job scanned about 90,000 tweet rows, 74,000 enrichment rows, and 134,000 attempt events and spent about three hours before useful work began. A later rebuild was much faster, so the exact delay varies, but the amount of replay work grows with the full history.
+
+Replace full replay with a checksum-verified SQLite index in the private Hugging Face Bucket `osolmaz/xtap-pool-bucket`. The private dataset `osolmaz/xtap-pool-data` remains authoritative. The index is a disposable projection that the Space and enrichment Job can restore and advance from strict JSONL append suffixes.
+
+## Requirements
+
+- Use the existing purpose-scoped `HF_TOKEN`, which has read and write access to exactly `osolmaz/xtap-pool-data` and `osolmaz/xtap-pool-bucket`.
+- Keep one non-concurrent enrichment Job and the current paid inference, reservation, retry, quality, and runtime limits.
+- Bind every index generation to one exact dataset Git revision and enrichment contract hash.
+- Upload an immutable database before replacing the current manifest.
+- Verify the database checksum, SQLite integrity, provenance, and physical counts before use and before publication.
+- Apply only new JSONL files or strict byte-for-byte append suffixes during normal startup.
+- Reject deleted files, changed prefixes, unknown paths, malformed manifests, and incompatible database schemas.
+- Preserve durable batch commits and receipts when a Job exits before publishing a new index. The next process must recover them from the dataset tail.
+- Keep the Space available as a public service and keep the dataset and Bucket private.
+- Replace full replay in production. Do not keep an automatic legacy fallback after the initial index has been seeded.
+
+## Scope
+
+This work covers:
+
+- the Bucket manifest and immutable SQLite files;
+- source-file inventory and exact dataset revision tracking;
+- index restore, incremental replay, validation, publication, and retention;
+- Space startup and enrichment Job startup;
+- setup configuration, token validation, schedule hashing, doctor checks, and deployment staging;
+- bootstrap and repair commands for an explicit full rebuild;
+- local, CI, and live recovery verification.
+
+This work does not change:
+
+- tweet, unit, enrichment, attempt, registry, or receipt contracts;
+- classifier prompts or labels;
+- inference providers, pricing, or concurrency;
+- the dataset's role as the system of record;
+- Our Models storage or publication.
+
+## Storage model
+
+### Current manifest
+
+The mutable dataset file `index/current.json` points to one immutable SQLite file in the private Bucket:
+
+```json
+{
+  "schema_version": 1,
+  "dataset": {
+    "repo": "osolmaz/xtap-pool-data",
+    "revision": "0123456789abcdef"
+  },
+  "projection": {
+    "contract_hash": "<sha256>"
+  },
+  "database": {
+    "key": "index/databases/<sha256>.sqlite",
+    "sha256": "<sha256>",
+    "predecessors": ["index/databases/<previous-sha256>.sqlite"]
+  },
+  "counts": {
+    "tweets": 84571,
+    "units": 73374,
+    "enrichments": 74325,
+    "attempt_events": 134568,
+    "registry_events": 1
+  }
+}
+```
+
+The manifest is strict and versioned. Publishing it uses a dataset commit whose `parentCommit` is the exact revision recorded in the database. Hugging Face rejects the commit if ingest or another publisher advances the dataset first, so two publishers cannot replace the pointer concurrently. Keys outside `index/` are outside this feature's control.
+
+### SQLite metadata
+
+The index keeps the existing tweet and enrichment tables and adds two small metadata tables:
+
+```sql
+CREATE TABLE index_metadata (
+  singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
+  schema_version INTEGER NOT NULL,
+  dataset_repo TEXT NOT NULL,
+  dataset_revision TEXT NOT NULL,
+  contract_hash TEXT NOT NULL
+);
+
+CREATE TABLE source_files (
+  path TEXT PRIMARY KEY,
+  kind TEXT NOT NULL,
+  oid TEXT NOT NULL,
+  byte_length INTEGER NOT NULL,
+  content_sha256 TEXT NOT NULL,
+  row_count INTEGER NOT NULL
+);
+```
+
+`source_files` covers authoritative JSONL inputs only. The content hash describes exactly `byte_length` bytes. When a file grows, the updater hashes the same prefix in the new object and parses only the remaining complete lines. A shorter object, a prefix mismatch, or a deleted path requires an explicit rebuild.
+
+Schemator reviewed the draft model and converged after two iterations. The accepted simplifications remove timestamps and duplicated taxonomy metadata from the manifest, use `dataset` instead of `source`, and use `enrichments` for the physical result count.
+
+## Runtime flow
+
+### Restore and advance
+
+1. Read and strictly parse `index/current.json` from one exact dataset revision.
+2. Download the referenced immutable SQLite object to a temporary path.
+3. Verify its SHA-256 digest, `PRAGMA quick_check`, metadata, contract hash, and physical counts.
+4. Pin the current dataset Git revision.
+5. List authoritative JSONL files at that revision with their Git object IDs and sizes.
+6. Skip unchanged files, parse complete new files, and parse only verified append suffixes.
+7. Commit all database and source inventory changes in transactions.
+8. Re-run integrity, metadata, and count checks before serving or claiming work.
+
+The Space opens a writable local copy because successful ingest requests still update the local projection after the dataset commit. The published Bucket object remains immutable. The Job uses the same restore path before it claims enrichment work.
+
+### Publish
+
+After the worker writes its final durable receipt, it pins a dataset revision that includes every worker output, advances the local index through that revision, and then:
+
+1. writes exact metadata and counts;
+2. checkpoints SQLite and creates a compact standalone copy;
+3. hashes and validates that copy;
+4. uploads it under `index/databases/<sha256>.sqlite` in the Bucket;
+5. downloads and validates the uploaded object;
+6. commits `index/current.json` to the dataset with the indexed dataset revision as its required parent commit;
+7. reads the manifest back from the returned commit and verifies the active generation; and
+8. removes old unreferenced Bucket generations while retaining the current generation and three recent predecessors.
+
+A crash before the manifest commit leaves the previous generation active. The next Job loads it and replays the durable dataset tail, including partial work committed by the crashed Job. A concurrent dataset write makes the parent-commit check fail and leaves the uploaded database unreferenced; the next successful publication removes that orphan during retention cleanup.
+
+### Bootstrap and repair
+
+A separate command performs the intentional full replay and publishes the first generation. Production startup requires a valid manifest after bootstrap. Missing or invalid index state fails closed with a direct repair command; it does not silently spend hours rebuilding history.
+
+The first deployment order is:
+
+1. build and validate an index from the current dataset without changing production;
+2. upload and read back the first generation;
+3. deploy the Space reader;
+4. create the exact replacement schedule suspended;
+5. run the required two-Job recovery canary;
+6. compare the second Job's restored counts and tail size with the first;
+7. activate scheduling only after all checks pass.
+
+## Setup and credentials
+
+Add `INDEX_BUCKET=osolmaz/xtap-pool-bucket` to Space and Job configuration. Include it in schedule identity and doctor checks.
+
+The token validator must accept one fine-grained token scoped to exactly the dataset and index Bucket, with read and write permissions on both and no unrelated global or repository permissions. It must perform direct authenticated reads against both resources without printing the token. The setup prompts continue to install this value as `HF_TOKEN` in the Space and scheduled Job.
+
+The Bucket client uses the official `@huggingface/hub` release already pinned by the project. No external service, source build, third-party runtime, or credential copy is required.
+
+## Validation
+
+### Unit and integration tests
+
+- Strict manifest parsing rejects unknown fields, bad hashes, wrong repositories, and unsafe object keys.
+- Database validation rejects checksum, integrity, provenance, contract, and count mismatches.
+- A new source file applies once.
+- A strict append applies only its suffix.
+- Reapplying the same dataset revision changes nothing.
+- Prefix edits, truncations, deletions, malformed final lines, and unknown source kinds fail closed.
+- Tweet, enrichment, attempt, registry, and receipt suffixes reproduce full replay results.
+- A crash before database upload, before manifest replacement, and after manifest replacement recovers safely.
+- The Space and Job restore the same generation and counts.
+- Token checks require both exact scopes.
+- Schedule hashes change with `INDEX_BUCKET`.
+- Retention cannot delete the active database or any retained predecessor.
+
+### Local checks
+
+Run:
+
+```bash
+npx -y @simpledoc/simpledoc check
+npm run check
+```
+
+Also run a local full-replay-versus-incremental comparison over representative tweet and enrichment fixtures. Compare every relevant SQLite table, queue total, contract hash, and physical count.
+
+### Live checks
+
+- Publish a bootstrap generation and download it with the purpose-scoped token.
+- Verify anonymous Bucket access remains `401`.
+- Confirm the Space reaches ready state from the index.
+- Run two non-overlapping Jobs from the suspended exact schedule.
+- Verify both receipts, source revisions, index manifests, database hashes, integrity results, counts, tail rows, and costs.
+- Confirm the second Job restores the first generation rather than replaying all history.
+- Resume cron and verify the next scheduled Job uses the same source revision and index contract.
+
+## Implementation measurements
+
+A local read-only production-data check pinned dataset revision `a4bbebdba5cde0c508bd4365480d32b488974da6`. The clean replay read 91,384 tweet rows, 74,913 enrichment rows, 135,744 attempt events, and 13,808 registry events in 120.351 seconds. Loading the resulting index and checking for an unchanged dataset revision took 1.417 seconds, a reduction of 118.934 seconds, or 98.8%.
+
+The durable and clean projections matched for tweets, unit membership, enrichment results, label assignments, label evidence, the free-label registry, recent errors, and the registry revision. Queue rows also matched except for `updated_at`, which records the local rebuild time and is not part of completion, retry, or consumer semantics.
+
+## Acceptance criteria
+
+The feature is complete when:
+
+- normal Space and Job startup does not replay unchanged historical JSONL rows;
+- the dataset manifest references a checksum-verified Bucket database covering one exact dataset revision;
+- incremental state matches a clean full replay in tests and the live canary;
+- a stale or corrupt index cannot become active;
+- a crashed Job's durable results are recovered from the dataset tail;
+- production has one exact non-concurrent active schedule;
+- the Bucket and dataset remain private;
+- CI is green and Pi Reviewer reports no P0 or P1 findings; and
+- the pull request is merged with a final validation report.
+
+If exact source provenance, strict append detection, deterministic replay, or safe manifest replacement cannot be established, leave production on the suspended schedule and report the blocker.

docs/2026-08-06-scrape-job-receipts-plan.md

@@ -0,0 +1,99 @@
+---
+date: 2026-08-06
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+title: Add durable receipts for browser scrape jobs
+tags: [extension, scraping, receipts, chrome]
+---
+
+# Durable scrape-job receipts
+
+This plan adds a browser-local record of which posts xTap observed during an Infinite Feed Scroller job. The result should let one command open an X list, capture posts through xTap, stop when coverage is complete, recover after an extension restart, and explain why the run stopped.
+
+xTap remains passive outside an explicitly opened scrape run. Infinite Feed Scroller remains responsible for movement and retry handling. It also owns job state. Tweet contents continue to live in xTap's local output and the pool dataset.
+
+## Requirements
+
+- Support steady catch-up runs and explicit historical date ranges.
+- Use xTap observations as the source of truth for coverage.
+- Preserve enough browser-local state to replay observations after an MV3 worker or page restart.
+- Record the list, post ID, post time, first-seen time, and capture sequence without copying tweet contents into the scroller.
+- Leave a durable run report with start and finish times, counts, final confidence, and stop reason.
+- Stop with a clear blocked or failed state when xTap is unavailable. Scrape jobs must not silently use DOM timestamps as replacement evidence.
+- Keep normal recording, playback, loop, and live-generator behavior independent of xTap.
+
+## Boundaries
+
+This change does not add a scheduler or system service. It does not change pool credentials, send new requests to X, or copy credentials between extensions. It does not move tweet contents into a second store.
+
+The first steady run for a list has no earlier list coverage. That run uses xTap post timestamps to cover the current UTC day. Later steady runs stop at posts that xTap observed for the same list before the run began.
+
+## Ownership
+
+xTap owns capture receipts and the external protocol. Infinite Feed Scroller owns scrape jobs and reports. The launcher starts a dedicated Chrome profile, loads both extensions, opens the xTap run, and then opens the list.
+
+The scroller has a stable unpacked-extension ID. xTap accepts the external port only from that ID. The xTap extension keeps its existing ID and storage, so its pool connection is not migrated.
+
+## xTap data
+
+xTap stores the following records in IndexedDB:
+
+- A receipt for each unique post with its post time, first-seen time, source endpoint, and global capture sequence.
+- A list receipt for each `(listId, postId)` pair. This prevents a post seen elsewhere on X from counting as prior coverage of the target list.
+- A run record with its list, baseline capture sequence, known-list count at start, last observation cursor, state, and timestamps.
+- Ordered run observations. Each observation records the post ID, post time, observation time, capture sequence, and whether that list-post pair was known before the run.
+
+Run observations contain no post text, author data, or pool token.
+
+## External protocol
+
+The protocol uses a named `chrome.runtime.connect()` port. Every message carries protocol version `1` and a run ID.
+
+The scroller opens a run before opening the X tab. xTap returns the durable run record and replays observations after the requested cursor. The list tab then reconnects with its last applied cursor and receives new observation batches. Duplicate batches are safe because observation cursors are monotonic.
+
+The scroller closes the xTap run with a terminal state. The state records successful completion, a failure, or a user abort. xTap retains the run and its observations for later inspection.
+
+## Completion rules
+
+A backfill job keeps the last 50 unique list observations. It gains confidence only while new observations arrive and the whole window is older than the requested start day. Three clean updates complete the job.
+
+A steady job uses the same window. After a list has earlier coverage, a clean window contains only list-post pairs known before the run. The first steady run for a list instead uses the current UTC day as its bootstrap range. Hard limits still stop a job after two hours or 3,600 update cycles.
+
+The exact window and streak remain visible in the job trace so later tuning can use observed runs.
+
+## Recovery
+
+The job page opens the xTap run before list navigation, which prevents the first timeline response from racing the receipt setup. A reconnect asks xTap to replay observations after the last applied cursor. The estimator can also rebuild from cursor zero after a full page restart.
+
+This one-run rule was replaced by the coordinated [concurrent scrape jobs plan](https://github.com/osolmaz/infinite-feed-scroller/blob/main/docs/2026-08-09-concurrent-scrape-jobs-plan.md). xTap now permits two tab-bound runs. Reopening the same run is idempotent, and a third run receives a capacity error.
+
+## Acceptance criteria
+
+- xTap records list observations before applying its normal capture deduplication.
+- A post seen on another timeline does not count as prior coverage of the target list.
+- Reopening a run returns the same baseline and ordered observations.
+- A sender other than the stable scroller extension ID cannot open the protocol.
+- The scroller opens the xTap run before creating the list tab.
+- Steady jobs stop on established list coverage and bootstrap from xTap timestamps when the list has no coverage.
+- Backfill jobs stop only after passing the requested range.
+- A missing bridge blocks the job and stops movement.
+- Completion and abort append one durable run report.
+- Both repositories pass their local quality gates and browser-independent protocol tests.
+
+## Verification
+
+In `xtap-pool`:
+
+```sh
+npm run test:extension
+npm run check
+```
+
+In `infinite-feed-scroller`:
+
+```sh
+npm run check
+npm run mutate
+npm run slophammer
+```
+
+A manual browser pass still needs a logged-in X profile. Run one bootstrap steady job, a second steady job against the same list, and a historical backfill. The second steady job should stop when its fresh window reaches posts recorded before that run.

docs/2026-08-12-bucket-object-log-plan.md

@@ -0,0 +1,361 @@
+---
+title: Move pool storage to an immutable Bucket log
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-08-12
+---
+
+# Move pool storage to an immutable Bucket log
+
+The pool dataset uses about 311.7 GB while its current files use about 528.8 MB. Each ingest rewrites one or more growing JSONL files, and the dataset keeps the old revisions. This change must preserve every pooled post and durable enrichment record while replacing that storage path in one coordinated deployment.
+
+The private Hugging Face Bucket `osolmaz/xtap-pool-data` will become the only raw system of record. The application will not keep dataset readers, dataset writers, dual writes, fallback reads, compatibility routes, or a legacy deployment period.
+
+## Scope
+
+This work will:
+
+- store raw pool transactions as immutable compressed Bucket objects;
+- store immutable source snapshots so any SQLite generation names its exact inputs;
+- keep replaceable SQLite generations and their active manifest in the separate private index Bucket;
+- import one explicitly pinned final dataset revision;
+- prove exact preservation before production starts on the new storage path;
+- update the Space, enrichment Job, setup, doctor, repair, deployment, and operator commands;
+- remove `DATASET_REPO` and all dataset-backed runtime code; and
+- leave the old dataset unchanged as a read-only cold backup.
+
+The first deployment keeps all owned wire and storage formats at version 1. This repository has no public release, so the coordinated replacement changes version 1 in place.
+
+## Non-goals
+
+- Do not store or download images.
+- Do not keep a queryable dataset projection.
+- Do not use Git history as runtime storage.
+- Do not add a service or another scheduler.
+- Do not delete, rewrite, squash, or reclaim the old dataset in this task.
+- Do not change pool identity, X capture, enrichment models, taxonomy meaning, or the Infinite Feed Scroller schedule.
+
+## Required data
+
+The import must preserve every valid record from these paths at the pinned revision:
+
+| Source                                                  | Durable value                      |
+| ------------------------------------------------------- | ---------------------------------- |
+| `data/<contributor>/YYYY/MM/tweets-YYYY-MM-DD.jsonl`    | Pooled tweets                      |
+| `enrichment/YYYY/MM/enrichment-YYYY-MM-DD.jsonl`        | Enrichment rows                    |
+| `enrichment/attempts/YYYY/MM/attempts-YYYY-MM-DD.jsonl` | Attempt events                     |
+| `enrichment/registry/YYYY/MM/registry-YYYY-MM-DD.jsonl` | Free-label registry events         |
+| `enrichment/receipts/YYYY-MM-DD.jsonl`                  | Worker receipts                    |
+| `config/pool.json`                                      | Pool membership and administrators |
+| `config/service-accounts.json`                          | Service accounts                   |
+| `config/labels.json`                                    | Taxonomy configuration             |
+| `enrichment/vocabulary.json`                            | Durable vocabulary configuration   |
+
+`index/current.json` is not imported. It belongs to the superseded index contract and is replaced by a new index manifest in the index Bucket.
+
+## Storage layout
+
+The raw Bucket contains only immutable version 1 objects:
+
+```text
+v1/segments/<category>/YYYY/MM/DD/<time-ms>-<uuid>-<sha256>.json.gz
+v1/snapshots/<sha256>.json
+```
+
+`category` is `tweet`, `enrichment`, `attempt`, `registry`, `receipt`, `config`, or `mixed`. It helps bounded configuration reads and operator inspection. It does not control validation; each operation path does.
+
+The index Bucket contains replaceable projections:
+
+```text
+index/databases/<sha256>.sqlite
+index/current.json
+```
+
+The raw and index Buckets have different deletion rules. Runtime code never deletes a raw segment or raw snapshot. Index publication may retain the active SQLite generation and three predecessors and delete older index generations.
+
+## Segment contract
+
+One segment records one successful application transaction:
+
+```json
+{
+  "schema_version": 1,
+  "transaction_id": "dfbc9fea-f42c-42d8-ae2d-44f909c417bf",
+  "created_at": "2026-08-12T12:34:56.789Z",
+  "operations": [
+    {
+      "path": "data/osolmaz/2026/08/tweets-2026-08-12.jsonl",
+      "mode": "append",
+      "lines": ["{\"id\":\"...\"}"]
+    }
+  ]
+}
+```
+
+An append operation contains one or more complete non-empty JSON lines for one approved record path. A write operation contains the complete UTF-8 value for one approved configuration path:
+
+```json
+{
+  "path": "config/pool.json",
+  "mode": "write",
+  "content": "{\n  \"version\": 1\n}\n"
+}
+```
+
+The segment object is canonical JSON with no insignificant whitespace and keys emitted in schema order. The SHA-256 in the key covers these uncompressed bytes. Gzip uses deterministic headers so a retry produces the same compressed bytes. The timestamp and UUID identify the transaction and give configuration writes a total order. A process must generate a timestamp greater than its prior transaction timestamp when two writes occur in the same millisecond or the clock moves backward.
+
+The writer validates every path and record before upload. Unknown fields, unknown paths, malformed JSON, invalid records, duplicate operation paths, empty append sets, unsupported versions, and invalid timestamps fail closed.
+
+The writer uploads to a new key, downloads it, decompresses it, verifies its uncompressed checksum and complete bytes, parses and validates it again, and only then returns success. A retry may reuse an existing key only when the stored object decodes to the exact same bytes. A conflicting object fails closed.
+
+Content-addressed keys make concurrent raw writes independent. The chance of a different valid payload using the same SHA-256 key is below the operational risk threshold. No mutable raw head or compare-and-swap credential is required.
+
+## Snapshot contract
+
+A raw snapshot names the exact immutable objects used to build an index:
+
+```json
+{
+  "schema_version": 1,
+  "bucket": "osolmaz/xtap-pool-data",
+  "files": [
+    {
+      "key": "v1/segments/tweet/2026/08/12/...json.gz",
+      "oid": "immutable-hub-object-id",
+      "size": 1234,
+      "content_sha256": "sha256-from-segment-key"
+    }
+  ]
+}
+```
+
+Files are sorted by key. The snapshot revision is the SHA-256 of the canonical snapshot bytes, and its key is `v1/snapshots/<revision>.json`. Snapshot creation lists raw segments, validates every key and immutable object ID, writes the content-addressed snapshot, reads it back, and verifies it.
+
+A snapshot can omit a transaction that finishes after the listing passes its key. This is safe because the snapshot explicitly lists its complete input set and the next advance includes the new segment. No transaction can appear partly: Bucket uploads publish complete objects, and replay verifies each complete object before changing SQLite.
+
+A reader never reconstructs an old snapshot from the current Bucket listing. It reads the exact immutable snapshot object named by the index manifest.
+
+## Configuration ordering
+
+Pool membership, service accounts, taxonomy, and vocabulary use write operations. The current value is the last valid write ordered by:
+
+1. `created_at`;
+2. `transaction_id`; and
+3. segment key as a final deterministic tie-breaker.
+
+Pool and service-account mutations remain serialized by the Space. The enrichment Job does not write these values. Taxonomy and vocabulary changes remain operator actions. A fresh restore derives configuration only from the raw Bucket.
+
+## Runtime storage boundary
+
+A `BucketLog` replaces `DatasetMirror`.
+
+It must provide:
+
+- validated and verified transaction upload;
+- stable snapshot creation and loading;
+- exact segment download and replay;
+- current configuration reads;
+- local bounded caching that is never authoritative; and
+- the latest valid worker receipt observed during replay.
+
+Ingest validates, stamps, and deduplicates a batch against SQLite. It writes one verified raw segment before updating SQLite or returning success. An upload or verification failure returns a storage error and leaves SQLite unchanged.
+
+The enrichment worker writes rows, attempt events, registry events, and receipts as immutable segments before applying them locally. Its existing single-Job schedule remains non-concurrent. The Space and Job may write different segments concurrently.
+
+## SQLite projection
+
+The SQLite database is a disposable projection. Its metadata records:
+
+```text
+schema_version = 1
+raw_bucket
+raw_snapshot_revision
+contract_hash
+```
+
+The source tables record each segment key, immutable Hub object ID, compressed byte length, uncompressed SHA-256, and row counts by record kind. A source object mutation, deletion from an exact snapshot, checksum mismatch, repeated key with changed identity, or replay count mismatch fails closed.
+
+Index bootstrap builds a fresh database from one new raw snapshot. Index advance creates a new snapshot, compares exact segment keys with the local source table, rejects mutation or deletion, validates all new segments before starting a SQLite transaction, and applies all new segments atomically.
+
+Index publication:
+
+1. advances to one exact raw snapshot;
+2. makes and validates a SQLite backup;
+3. uploads the content-addressed SQLite generation;
+4. downloads it and verifies its checksum, integrity, metadata, and counts;
+5. writes `index/current.json` in the index Bucket;
+6. reads the manifest back and verifies the complete value; and
+7. prunes only old index generations.
+
+The active manifest binds one exact raw snapshot and one exact SQLite generation:
+
+```json
+{
+  "schema_version": 1,
+  "source": {
+    "bucket": "osolmaz/xtap-pool-data",
+    "revision": "<snapshot-sha256>"
+  },
+  "projection": {
+    "contract_hash": "<sha256>"
+  },
+  "database": {
+    "key": "index/databases/<sha256>.sqlite",
+    "sha256": "<sha256>",
+    "predecessors": []
+  },
+  "counts": {
+    "tweets": 0,
+    "units": 0,
+    "enrichments": 0,
+    "attempt_events": 0,
+    "registry_events": 0,
+    "receipts": 0
+  }
+}
+```
+
+The manifest keeps physical counts because they detect the wrong or incomplete database before serving. The index Bucket has one authorized publisher at a time: the suspended setup/bootstrap command or the non-concurrent enrichment Job. The Space advances its local database after ingest but does not publish the shared generation.
+
+If a manifest write fails after the SQLite upload, the generation is unreferenced and harmless. The next publication may reuse it. If manifest read-back differs, publication fails and does not prune anything.
+
+## Import contract
+
+The importer requires:
+
+- an explicit dataset repository;
+- an explicit 40-character dataset revision;
+- the raw Bucket name;
+- an output report path; and
+- a local work directory with enough space for bounded files and verification data.
+
+It refuses `main`, a branch name, or an unpinned source. Import is resumable and idempotent. Each source file maps to one deterministic transaction ID and one content-addressed segment. Re-running the same pinned import verifies and reuses matching objects.
+
+The importer lists all approved source and configuration paths at the pinned revision. It downloads each file, validates UTF-8 and every record, and records:
+
+- source path and immutable object ID;
+- byte length and SHA-256;
+- valid non-empty line count;
+- record kind;
+- for tweets, the sorted `(id, contributed_by)` identity digest and per-contributor/per-capture-day counts; and
+- for configuration, the exact content SHA-256.
+
+The importer uploads and verifies each target segment. It then creates a target snapshot and replays only that snapshot into a fresh SQLite database.
+
+## No-loss reconciliation
+
+Cutover is blocked unless all checks pass:
+
+- every approved source path is represented once;
+- every source non-empty line is represented once with the exact UTF-8 line bytes;
+- source and target raw line counts match by kind and logical path;
+- source and target sorted line digests match by kind and logical path;
+- the complete sorted multiset of `(tweet id, contributor, exact line hash)` matches;
+- unique `(tweet id, contributor)` sets match;
+- per-contributor and per-capture-day tweet counts match;
+- enrichment unit IDs and exact row hashes match;
+- attempt event exact row hashes match;
+- registry `(name, revision)` values and exact row hashes match;
+- receipt exact row hashes match;
+- pool, service-account, taxonomy, and vocabulary bytes match exactly;
+- fresh Bucket-only replay produces the expected SQLite unique tweet and enrichment counts;
+- SQLite `PRAGMA integrity_check` returns `ok`; and
+- rebuilding a second time from the same snapshot produces the same logical table digests.
+
+The report contains repository and Bucket names, source and snapshot revisions, object counts, row counts, aggregate hashes, and pass/fail results. It does not contain tweet text, configuration contents, access tokens, or service-account secrets.
+
+## Fault tests
+
+Tests must prove that reconciliation stops for:
+
+- one deleted source line;
+- one duplicated source line;
+- one changed source line;
+- one truncated compressed object;
+- one object under the wrong key;
+- one key with the wrong checksum;
+- one source object with a changed immutable ID;
+- one missing configuration file;
+- one changed configuration byte;
+- one unsupported or malformed record;
+- one source object deleted after index state records it;
+- one failed upload;
+- one failed read-back; and
+- one manifest that names the wrong snapshot or SQLite generation.
+
+Tests must also prove retry idempotence, deterministic replay under shuffled listing order, safe concurrent segment creation, stable snapshot revisions, and a changed revision for any changed file set.
+
+## Setup and credentials
+
+Setup creates or verifies two private Buckets and one public Space:
+
+- raw: `<namespace>/xtap-pool-data`;
+- index: `<namespace>/xtap-pool-bucket`; and
+- Space: `<namespace>/xtap-pool`.
+
+`RAW_BUCKET` replaces `DATASET_REPO` everywhere. `INDEX_BUCKET` remains. The storage token validator requires read and write access to exactly both Buckets and rejects dataset-only scope. Setup, doctor, repair, and enrichment Job reconciliation use these names.
+
+The raw Bucket already exists for this deployment, and the existing xTap storage token has already been granted access in its current secret source. The implementation may use that token in place. It must not print it or copy it into a new store. If the Space and Job already hold that same token, they keep the existing secrets. If deployment needs a credential copied from `offline/secrets` into a new destination, work stops for explicit source-and-destination approval.
+
+## Deployment sequence
+
+1. Finish implementation, focused tests, full `npm run check`, SimpleDoc, Pi Reviewer, PR review, CI, merge, and deployment staging.
+2. Verify the new raw Bucket is private and empty or contains only objects from the same pinned import contract.
+3. Suspend the enrichment schedule and verify that no enrichment Job is running.
+4. Stop or pause the Space at a safe boundary. Extension durable staging queues must retain unsent captures while the ingest endpoint is unavailable.
+5. Resolve and record the final dataset commit SHA after the Space is stopped.
+6. Run the importer and no-loss reconciliation against that exact revision.
+7. Create and verify the final raw snapshot.
+8. Bootstrap the SQLite index from only that snapshot and publish its verified generation and index manifest.
+9. Configure `RAW_BUCKET` and retain `INDEX_BUCKET`; remove `DATASET_REPO` from Space and Job variables.
+10. Deploy the merged code. No production process may read the old dataset.
+11. Wait for the Space to report `RUNNING`, the intended source revision, valid Bucket credentials, the intended raw snapshot, and the expected counts.
+12. Let one already-staged real extension submission reach ingest. Verify its transaction segment, response, SQLite row, explorer/API visibility, pool sync receipt, and local xTap JSONL durability.
+13. Restart the Space once. Verify a fresh process restores the published index, advances from the raw Bucket, and serves that real post.
+14. Run the existing bounded enrichment canary. Verify raw enrichment segments, a new exact snapshot, a verified index generation, receipt durability, and matching source revision.
+15. Resume the enrichment schedule only after the canary passes.
+16. Verify pool sync and passive xTap capture without image downloads.
+17. Resume the historical Infinite Feed Scroller backfill from its deepest verified safe day with a one-day overlap and continue its separate recovery-capable monitoring.
+
+## Failure behavior
+
+Before the final dataset revision is pinned, a failure leaves production on the old code and no cutover has started.
+
+After the Space stops, any import, reconciliation, index, configuration, or deployment failure keeps ingestion offline. Operators fix the new Bucket path and retry from the same pinned source. They do not restart the legacy dataset runtime. Client staging queues retain captures during this bounded outage.
+
+After the new Space starts, a failed real ingest, restart proof, or enrichment canary suspends enrichment and stops the Space if continued writes could make diagnosis unsafe. Recovery stays on the Bucket contract. The old dataset is input to the pinned recovery importer only and never becomes a production fallback.
+
+A raw object conflict, checksum failure, missing pinned object, reconciliation mismatch, unknown record, invalid credential scope, or inconsistent index manifest is a material blocker. No cleanup follows it.
+
+## Verification commands
+
+The implementation must add documented commands for these operations:
+
+```sh
+npm run storage:import -- --dataset <repo> --revision <sha> --raw-bucket <bucket> --report <path>
+npm run storage:verify -- --dataset <repo> --revision <sha> --raw-bucket <bucket> --report <path>
+RAW_BUCKET=<bucket> INDEX_BUCKET=<bucket> npm run index:bootstrap
+npm run doctor -- <space>
+```
+
+Before merge:
+
+```sh
+npm run check
+npx -y @simpledoc/simpledoc check
+pi-reviewer --base main
+```
+
+Production evidence must include the pinned source SHA, raw snapshot revision, imported source and target object counts, all reconciliation counts and digests, SQLite manifest counts, deployed Git SHA, Space runtime SHA, real ingest segment key and receipt identifiers without post text, restart proof, and enrichment canary receipt.
+
+## Acceptance criteria
+
+- The plan, schema, implementation, operator commands, and deployment behavior agree.
+- The runtime contains no dataset source, dataset fallback, dataset variable, dataset write, dual read, or dual write.
+- Exact pinned-source reconciliation proves that no valid post or durable record was lost.
+- A fresh process builds and serves the same logical state using only the raw and index Buckets.
+- A real extension submission is durable before success and remains visible after restart.
+- The enrichment canary writes raw Bucket evidence and publishes a verified matching index.
+- Production remains on Hugging Face and uses one raw Bucket plus one index Bucket.
+- Images remain disabled.
+- The old dataset remains unchanged and unavailable to production runtime code.
+- Bob's historical backfill resumes only after storage cutover verification and preserves its existing receipts and local JSONL.

docs/2026-08-16-resumable-registry-review-plan.md

@@ -0,0 +1,96 @@
+---
+title: Make registry review resumable
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-08-16
+---
+
+# Make registry review resumable
+
+> **Historical plan:** [Add small worker checkpoints](2026-08-19-small-worker-checkpoints-plan.md)
+> replaces this receipt-cursor continuation path in production. Existing receipt
+> cursor fields remain valid historical evidence. New Jobs use a frozen logical
+> run, registry ordinals, immutable output claims, and compact checkpoints. Use
+> the newer plan for production recovery, spending limits, canaries, and schedule
+> activation.
+
+The enrichment worker must save useful work before a Hugging Face Job reaches its timeout. The current worker saves post enrichment as it goes, then reviews free-label candidates in one long pass. It keeps that pass in memory until the full scan ends. A timeout can therefore discard hours of registry review.
+
+This plan makes registry review resume from a durable cursor and makes the worker's elapsed-time limit include index restoration. It also removes the full mixed-segment scan from normal index restoration so the steady 45-minute Job can start useful work quickly.
+
+A production recovery later exposed a second timeout gap. The router deadline ended when response headers arrived, before the worker read the response body. Three requests therefore stayed dispatched for more than 62 minutes despite a 90-second timeout. The worker must keep one deadline active through the full response body read and release reservations after a timeout.
+
+## Requirements
+
+- Keep the current raw Bucket, index Bucket, model, provider, taxonomy, hardware class, and concurrency.
+- Preserve existing receipts and immutable segments.
+- Save registry decisions before a bounded stop.
+- Write a durable cursor in each receipt when a registry scan is incomplete.
+- Resume an incomplete scan only when no new enrichment work changed its inputs.
+- Start the worker elapsed-time budget before index restoration.
+- Restore the latest receipt without downloading every mixed segment.
+- Keep old receipts valid.
+- Keep one physical enrichment Job at a time.
+- Apply one router deadline to connection setup, response headers, the response body, and parsing.
+- Bound router response bodies before parsing them.
+- Record a timed-out body read as a normal timeout failure and clear its cost reservation.
+
+## Scope
+
+The change covers receipt metadata, receipt replay, registry settlement, worker timing, and regression tests. It also covers deployment, catch-up recovery, two steady canaries, and steady schedule activation.
+
+## Non-goals
+
+- No model, prompt, taxonomy, provider, storage, concurrency, or hardware change.
+- No new remote store.
+- No public API contract change beyond optional receipt fields used for operations.
+- No concurrent registry writes or unordered decisions. Pure Hub verification may run in bounded parallel, but registry decisions and commits must remain in candidate-name order.
+
+## Design
+
+1. Add optional registry-scan progress to the receipt schema. The progress records the last fully reviewed candidate, the number scanned, the candidate count, and whether the scan completed.
+2. Persist and apply collected registry decisions before any bounded stop. A continuation resumes after the last durable candidate only when the current run processed no new enrichment units.
+3. Restore the latest valid receipt from receipt-category segments, then inspect only mixed segments that are not older than the selected receipt segment. This preserves newer historical mixed receipts without downloading thousands of older mixed segments.
+4. Run pure Hub verification in bounded parallel. Settle decisions and write registry events in deterministic candidate-name order.
+5. Start the elapsed-time clock before index restoration. Pass only the remaining budget to enrichment processing. This preserves the configured gap between the worker limit and the platform timeout for receipt and index publication.
+6. Keep old receipt rows readable by making the new progress field optional.
+7. Run each router exchange inside one reusable deadline helper. The helper owns the abort timer until the callback finishes, so callers cannot return a live response body after the deadline ends.
+8. Read success and error response bodies through a byte-limited reader while the deadline remains active. Map a deadline to the existing timeout error class and reject an oversized or malformed response without keeping the worker blocked.
+
+## Acceptance criteria
+
+- A registry scan that reaches its elapsed ceiling writes a valid receipt with an incomplete cursor.
+- The next no-unit run resumes after that cursor and does not repeat prior external checks.
+- Decisions found before a ceiling are durable and replay correctly.
+- A run that processes new units starts a fresh registry scan.
+- A complete registry scan marks the receipt complete.
+- Index restoration reads the latest receipt without scanning mixed segments older than the selected receipt segment.
+- Restore time reduces the remaining worker processing budget.
+- Existing receipt fixtures and historical rows remain valid.
+- Repository checks, targeted tests, Pi Reviewer, and CI pass.
+- Catch-up completes within the approved $500 total budget.
+- Two steady canaries prove bounded continuation before schedule activation.
+- A router response that sends headers and then stalls its body fails within the configured deadline.
+- A stalled error body follows the same deadline.
+- An oversized router body fails before unbounded buffering.
+- A timed-out body read writes a durable timeout outcome, clears the cost reservation, and lets the worker finish its receipt.
+
+## Verification
+
+Run:
+
+```bash
+npm run check
+npm test --workspace @xtap-pool/shared
+npm test --workspace @xtap-pool/space -- enrich-worker enrich-job durable-index bucket-log
+npx -y @simpledoc/simpledoc check
+pi-reviewer --base main
+```
+
+After merge and deployment:
+
+1. Verify the deployed source revision.
+2. Run one suspended catch-up continuation with only the approved Job secrets.
+3. Validate the receipt hash, cursor, settled cost, zero reservations, registry events, and index pointer.
+4. Replace the catch-up schedule with the exact suspended steady profile.
+5. Run two non-overlapping steady canaries and validate continuation.
+6. Activate the steady schedule and verify production health, queue state, schedule count, and cleanup.

docs/2026-08-19-small-worker-checkpoints-plan.md

@@ -0,0 +1,1122 @@
+---
+title: Add small worker checkpoints
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-08-19
+updated: 2026-09-10
+tags: [enrichment, sqlite, checkpoints, hugging-face, resume]
+---
+
+# Add small worker checkpoints
+
+## Purpose
+
+A replacement enrichment Job must resume useful work without first restoring and
+later republishing the complete public SQLite index.
+Physical Jobs should be disposable. The logical run, fixed work plan, verified
+results, queue state, and registry cursor must survive them.
+
+The current durable index removed full raw-history replay, as described in
+[Add a durable enrichment index](2026-08-04-durable-enrichment-index-plan.md).
+The index is now about 1.3 GB, so making every physical Job restore it has become
+the next bottleneck. The existing
+[resumable registry plan](2026-08-16-resumable-registry-review-plan.md) saves a
+cursor, but new enrichment work can still reset that scan.
+
+This plan keeps the raw Bucket as the system of record and the public index as a
+replaceable read model. It adds a separate compact work plan and worker
+checkpoint. The shared checkpoint mechanics come from
+[HF Job Control](https://github.com/osolmaz/hf-job-control/blob/feat/durable-job-resume/docs/2026-08-19-durable-job-resume-plan.md).
+
+## Outcome
+
+A normal continuation will download one small work plan and checkpoint, then
+process only missing units or registry candidates. It will not open the public
+index. The large index will be restored and updated once, during final
+publication for the logical run.
+
+When a physical Job completes that logical run, it activates the verified
+successor and exits. The next scheduled Job restores that successor and
+processes only its unresolved work. One physical Job therefore runs one logical
+plan and does not start work that can exceed the platform deadline.
+
+## Requirements
+
+The implementation must:
+
+- Freeze one source snapshot, enrichment contract, queue order, and registry
+  candidate order for each logical run.
+- Keep one logical run ID across physical Jobs.
+- Keep each physical Job under a separate attempt ID.
+- Store completion with compact bitmaps instead of full completed queue rows.
+- Preserve retry and blocked details only for unresolved work.
+- Save a registry cursor that queue work cannot reset.
+- Persist one ordered registry event for every scanned candidate, including candidates that stay unchanged, so the cursor can always advance across exact names and ordinals.
+- Validate every orphan row and attempt against the frozen unit input hash, taxonomy version, and enrichment contract before recovery advances progress.
+- Publish and verify immutable result batches before checkpoint progress moves.
+- Treat a checkpoint failure after a durable success as an orphan-output recovery case, not as a failed provider result.
+- Use immutable sequence claims as the authoritative checkpoint history.
+- Use a contiguous immutable activation-claim chain as the authoritative active-run history.
+- Treat the mutable active-run pointer as a startup shortcut only.
+- Resolve the active run from that claim chain so the recurring schedule never pins one completed run forever.
+- Prepare the next compact work database from the already open final publication database, advance that local copy to the latest raw snapshot, carry the completed registry baseline forward while retaining candidates first observed after the frozen plan time, and activate the successor only after its plan and bootstrap checkpoint verify.
+- Stop the physical Job after it activates a verified successor, even when that successor has unresolved queue or registry work.
+- Give each physical Job one command start time, one logical run and one $10 inference ceiling.
+- Let the next non-concurrent scheduled Job restore the successor from its verified checkpoint.
+- Stop at a durable boundary without polling or spinning when the logical run completes or when the 40-minute worker or $10 inference ceiling is reached.
+- Repeat no more than one in-flight concurrency batch after interruption.
+- Keep progress monotonic across attempts in one logical run.
+- Build and fully verify the public SQLite index only after work completes.
+- Reuse an already uploaded final database after an interrupted publication.
+- Use the existing raw and index Buckets.
+- Preserve the current model and provider. Preserve taxonomy, privacy, receipt
+  semantics, and public index behavior.
+- Keep one physical enrichment Job at a time.
+
+## Boundaries
+
+This work does not change enrichment prompts, model selection, provider,
+taxonomy meaning, queue fairness, public routes, or capture storage. It does not
+create another Bucket, Dataset, service, scheduler, or credential. It also does
+not change the `cpu-upgrade` hardware, six-hour cadence, secret names or
+destinations, quality limits, $0.25 per-call limit, $10 physical-attempt
+inference ceiling, 40-minute worker ceiling, or 45-minute platform timeout.
+
+The current immutable recovery Jobs and their saved data remain untouched. New
+code is implemented and tested on a feature branch. Production deployment waits
+until recovery finishes and its final index validates.
+
+The deployment is a hard replacement. After it is active, normal worker code
+will not retain the old full-index startup path. Existing immutable databases,
+segments, and receipts remain available for audit and operator-directed
+rollback.
+
+## Reviewed worker revision handoff
+
+An unfinished frozen plan can outlive the worker revision that created it. A
+new worker revision must not edit that plan, ignore its worker identity, or use
+a list of accepted predecessor revisions. It can continue the same logical run
+only through one exact handoff in the immutable Space deployment commit.
+
+The handoff is part of `.xtap-deployment.json`. The file remains strict and
+secret-free. It retains `source_revision` and gains one required nullable field:
+
+```ts
+export type DeploymentManifest = {
+  source_revision: string;
+  enrichment_revision_handoff: null | {
+    active_generation: number;
+    activation_sha256: string;
+    run_id: string;
+    plan_sha256: string;
+    plan_worker_revision: string;
+    target_worker_revision: string;
+    contract_sha256: string;
+    source_snapshot_revision: string;
+    checkpoint_pointer_sha256: string;
+    checkpoint_sequence: number;
+    checkpoint_key: string;
+    checkpoint_sha256: string;
+    checkpoint_bytes: number;
+  };
+};
+```
+
+A null handoff means the active plan already names the deployed worker revision.
+A non-null handoff permits only the exact target revision in the same manifest.
+It binds that target to one active generation, activation chain, run, immutable
+plan, contract, source snapshot, and checkpoint boundary. It contains no time,
+secret, mutable URL, revision list, wildcard, fallback, environment override,
+or predecessor alias. The existing deployment manifest contract changes in
+place before the first public release. There is no second manifest version and
+no old-manifest reader.
+
+Canonical JSON makes the handoff and complete manifest deterministic. The
+schema rejects unknown fields, malformed hashes and IDs, self-handoffs,
+target/source disagreement, missing checkpoint identity, and extra predecessor
+revisions. The immutable Space Git commit is the handoff record. The repair does
+not add a handoff object, service, store, or coordination path to either Bucket.
+
+### Read-only preparation
+
+The checked-in updater prepares the handoff before it stages the Space upload.
+It must:
+
+1. Require a clean local Git revision and use it as the requested target.
+2. Read the current Space variables only to identify the existing index Bucket.
+3. Require one canonical schedule, require it to be suspended and
+   non-concurrent, and require no active enrichment Job.
+4. Resolve and hash the contiguous active-run activation chain.
+5. Parse and hash the exact immutable plan and verify its run, source, contract,
+   work order, and predecessor worker revision.
+6. Read the current checkpoint pointer and retain its exact bytes and digest.
+   Restore the newest checkpoint from the complete immutable claim chain.
+7. Require the pointer to name one checkpoint in that verified chain. A pointer
+   may lag the chain head after an interrupted write. Verify the checkpoint
+   sequence, key, bytes, digest, plan identity, output frontiers, queue bitmap,
+   registry cursor, claims, and absence of orphan output.
+8. Return deterministic handoff bytes without creating a provider, updating
+   progress, or exposing a write method.
+9. Re-read the activation, checkpoint pointer, schedule, and Job list
+   immediately before staging and again before upload. Stop on any drift.
+10. Embed the handoff in `.xtap-deployment.json`, upload the Space commit, and
+    make no Bucket write.
+
+If the active plan already names the target revision, the updater records null.
+If any identity is missing, stale, forked, corrupt, conflicting, or different,
+the updater stops before upload. A failed preflight must not change the Space,
+schedule, Jobs, Buckets, secrets, progress, or durable run state.
+
+### Worker preflight
+
+Direct equality between `plan.contract.worker_revision` and
+`XTAP_SOURCE_REVISION` remains the normal path. When they differ, the worker
+must read the deployment manifest from its image and require one exact handoff.
+It verifies:
+
+- The manifest source and handoff target equal `XTAP_SOURCE_REVISION`.
+- The current active generation and activation digest equal the handoff.
+- The run ID, plan digest, plan worker revision, contract digest, and source
+  snapshot equal the unchanged plan.
+- On the first start, the current checkpoint pointer either equals the pinned
+  pointer or names an older checkpoint in the same complete verified chain. The
+  worker restores the chain head and then repairs a lagging pointer before it
+  processes more work.
+- On a later attempt for the same plan, the current checkpoint is accepted only
+  when its sequence is higher and the complete verified claim chain descends
+  from the pinned handoff checkpoint without a gap, fork, or conflict.
+- The complete checkpoint bundle, immutable sequence claims, result claims,
+  bitmap, registry cursor, and output frontiers restore without a gap, fork,
+  conflict, duplicate ordinal, or orphan output.
+
+The worker performs this verification before it creates progress, an inference
+provider, a raw writer, or any other write-capable runtime object. A pointer
+that moves behind the handoff, changes plan identity, or does not descend from
+the pinned checkpoint stops the worker. Every missing, malformed, mismatched,
+or conflicting identity also stops before a provider call or remote write.
+
+After verification, the worker continues the same run, checkpoint chain, result
+claims, bitmap, queue ordinals, and registry ordinals. It processes only missing
+work. Checkpoint sequences stay monotonic. Existing immutable objects do not
+change. A successor plan records the deployed worker revision and enters the
+normal direct-revision path. Successor activation stays fenced. One physical
+attempt keeps one command start time and one inference budget across
+successors. The existing elapsed and cost ceilings, blocked-only stop,
+zero-work stop, and interruption recovery do not change.
+
+Restore-only mode uses the same manifest, handoff, activation, plan, pointer,
+and checkpoint verification. It receives no `INFERENCE_TOKEN`, creates no
+provider, writes nothing, and makes zero provider calls.
+
+### Bounded cutover repair
+
+The current repair starts from active run
+`xtap-1321e3a40c38f32c0c210d8859622379`. Its frozen plan records worker revision
+`aa7ccee986b7c96b494cc16e17a483740f3afb3d`. The healthy Space repository and
+runtime revision is `1ac51e39c73e871cfd5335cab8e5e3bbcc58fa76`, with deployment
+source `c6d07950b31eda8e0ecf74022fa72733a3bc87f8`. Canonical schedule
+`6a92f66145686a1580c1543c` is suspended and non-concurrent, and no enrichment
+Job is active. The first exact deployment probe stopped at the worker identity
+gate before execution. The preserved boundary has active generation 29,
+checkpoint sequence 807, checkpoint SHA-256
+`5ca69ab4853b0e0e462d49a6f69954867f4343fa8e71655a6715d4859425171f`,
+and progress sequence 834. The failed probe made zero provider calls, spent
+$0 on the provider, used $0.000125 of CPU, produced no receipt, and left zero
+active reservation exposure.
+
+Implement the repair in this order:
+
+1. Create a clean repair branch from the accepted `main` revision. Recheck the
+   exact Space revision and health, suspended schedule, empty active Job list,
+   active generation, plan digest, and checkpoint sequence, key, digest, and
+   size. Stop on drift.
+2. Update the strict deployment schema and canonical serializer in
+   `shared/src/deployment.ts`. Add schema round-trip and rejection tests.
+3. Add a focused read-only handoff preparation and verification module under
+   `space/src/`. Export only protocol helpers that the setup package needs. Do
+   not expose write methods or provider construction.
+4. Extend `setup/src/stage.ts` and the existing updater boundary to prepare,
+   revalidate, and embed the claim before upload. Keep upload as its only remote
+   mutation. Add setup process tests for ordering, drift, no upload after a
+   failed preflight, no Bucket writes, and no secrets in staged bytes.
+5. Refactor `space/src/enrich-planned-command.ts` so deployment, activation,
+   plan, handoff, pointer, and full checkpoint verification finishes before
+   progress, provider, raw writer, or another write-capable client exists.
+6. Add the production regression with the predecessor revision, active
+   generation 29, and checkpoint sequence 807 identity shape. It must fail
+   without the claim, accept only the exact claim, call no provider before all
+   identities verify, preserve claimed work, avoid duplicate queue and registry
+   ordinals, and resume safely after interruption.
+7. Extend the drain, runtime, and restore tests to prove exact result replay,
+   monotonic checkpoints, unchanged immutable objects, bounded repeated
+   in-flight work, successor fencing, cumulative elapsed and cost accounting,
+   blocked-only termination, zero-work termination, and a second clean restore.
+8. Run `npm run check`, `npm run mutate`, `npm run build`,
+   `npx slophammer-ts check .`, and
+   `npx -y @simpledoc/simpledoc check`. Review the complete diff. Run
+   pi-reviewer against `main` until no P0 or P1 finding remains, and address
+   proportionate P2 findings. Push one narrow branch, require green CI, and
+   rebase-merge the reviewed pull request.
+9. Deploy the exact merged revision only through
+   `npm run update -- osolmaz/xtap-pool`. Wait for the exact Space repository
+   and runtime revisions, manifest read-back, and healthy application. Replace
+   the canonical schedule only if its source differs. Keep exactly one schedule
+   suspended and non-concurrent, and require zero active Jobs.
+10. Run restore-only verification against the preserved production run with no
+    inference token. Require exact replay, zero provider calls, zero orphan
+    segments, and unchanged public pointer and complete run-object listing.
+    Do not trigger a paid Job during this repair.
+
+### Regression matrix
+
+Tests must cover:
+
+- Deterministic manifest and handoff bytes, strict field validation, and unknown
+  field rejection.
+- The exact predecessor-plan/current-worker mismatch before provider creation.
+- A valid sequence-807 handoff, safe recovery from a pointer that lags a unique
+  verified chain, and failures for a missing claim, wrong target, wrong plan,
+  wrong contract, wrong source snapshot, wrong activation digest, a pointer
+  outside the chain, wrong checkpoint sequence, key, size, bytes, or digest,
+  malformed claim, activation fork, or active writer.
+- Read-only preparation with zero object-store writes, zero progress updates,
+  and zero provider construction.
+- Complete checkpoint and result-claim replay with no gaps, hash differences,
+  duplicate ordinals, or orphan output.
+- No duplicate queue or registry work across the handoff and an interrupted
+  first post-handoff checkpoint.
+- Successor fencing, one physical-attempt start time, cumulative provider cost,
+  active-reservation admission, cost-overrun rejection, elapsed ceiling,
+  blocked-only stop, and finite zero-work stop.
+- A clean second restore with zero provider calls and unchanged pointer and
+  object listing.
+- Post-merge Space source and runtime identity, manifest, health, suspended
+  schedule, zero active Jobs, and production restore-only verification.
+
+### Risks and stop rules
+
+A stale handoff could bind an unrelated checkpoint. The updater prevents this
+by requiring writer exclusion, hashing the exact pointer bytes, and re-reading
+the activation, pointer, schedule, and Job list before staging and upload. The
+worker requires that exact pointer on the first start. On later attempts, it
+accepts only a higher checkpoint in the same fully verified claim chain whose
+immutable ancestry includes the handoff checkpoint.
+
+Moving revision verification later could permit an early provider call or
+write. The worker must instead move the complete identity and checkpoint
+verification earlier than every progress, provider, writer, and write-capable
+client construction.
+
+A handoff could become a compatibility path. The schema therefore permits one
+exact target for one exact active plan and checkpoint. It has no list, fallback,
+wildcard, environment override, predecessor alias, old-manifest path, or second
+format version.
+
+A bad Space commit cannot be edited in place. The updater prepares deterministic
+bytes locally, verifies all identities, rechecks writer exclusion, and requires
+exact deployment read-back. Any correction is another reviewed deployment, not
+a history rewrite.
+
+Existing results could repeat after the handoff. The worker must continue the
+same restored bitmap, result claims, checkpoint chain, queue ordinals, and
+registry ordinals. Tests cover interruption at the first post-handoff
+checkpoint and reject duplicate work.
+
+A full restore can make startup slower. Use the existing bounded restore and
+prefetch interfaces and require a cold restore below five minutes. Keep provider
+and writer initialization deferred until restore finishes.
+
+Credentials must not enter the handoff or logs. The schema accepts only IDs,
+hashes, revisions, keys, and counts. Approved credentials remain in the
+short-lived updater process and the existing approved Space and schedule secret
+destinations.
+
+Stop the repair if a paid Job becomes active, an immutable production identity
+drifts, history would need to change, the identity gate would need to weaken, a
+protected provider or runtime contract would need to change, review or CI leaves
+a material defect, deployment provenance is not exact, or restore-only
+verification writes or calls a provider. If the same worker-plan mismatch
+repeats after the exact repaired deployment, preserve the evidence and stop.
+
+This repair does not run a production probe, recovery canary, recurring Job, or
+OurModels publication. It does not change an existing plan, checkpoint, output,
+claim, receipt, activation, pointer, index, database, manifest, or other
+historical object. It does not change OurModels, publication locks, the pinned
+HF Job Control dependency, Hugging Face or Fireworks behavior, runtime images,
+provider, model, hardware, cadence, concurrency, timeout, elapsed or cost
+limits, taxonomy, quality rules, secret names, or credential destinations. It
+does not add a migration reader, dual path, compatibility alias or allowlist,
+new dependency, service, repository, store, Endpoint, or extra schedule.
+
+## Resume model
+
+Each completed batch is saved once. A small claimed checkpoint records the
+finished ordinals and unresolved work. `current.json` points to the latest known
+claim so startup is fast, but the immutable claims remain the source of truth.
+
+A normal continuation downloads the frozen plan and small checkpoint. It then
+replays only the raw output segments claimed by that checkpoint into the compact
+worker database so queue results, registry statuses, and registry revisions are
+exact before new work starts. It does not restore the public database. The Job
+builds that database once after queue and registry work is complete.
+
+The physical-attempt coordinator records the command start time once. It adds
+settled inference cost and active reservations from each logical run before it
+admits work in a successor. A logical-run receipt remains tied to its immutable
+plan, but moving to a successor does not create a new physical-attempt budget.
+After publication activates a successor, the coordinator verifies that plan and
+its bootstrap checkpoint, checks whether queue or registry work remains, and
+continues it only when both physical-attempt ceilings still admit work. A
+zero-work successor ends the physical Job cleanly.
+
+## Logical run plan
+
+Create one canonical run plan:
+
+```ts
+export type ObjectReference = {
+  key: string;
+  sha256: string;
+  bytes: number;
+};
+
+export type EnrichmentRunPlan = {
+  schema_version: 1;
+  run_id: string;
+  created_at: string;
+  source: {
+    bucket: string;
+    snapshot_revision: string;
+    ordered_segments: ObjectReference;
+  };
+  contract: {
+    worker_revision: string;
+    contract_sha256: string;
+    taxonomy_version: number;
+    model: string;
+    provider: string;
+  };
+  base_index: {
+    key: string;
+    sha256: string;
+    bytes: number;
+    source_revision: string;
+    source_segment_count: number;
+    receipt_count: number;
+    registry_revision: number;
+  };
+  work: ObjectReference & {
+    queue_total: number;
+    queue_baseline_done: number;
+    registry_total: number;
+    registry_baseline_scanned: number;
+  };
+};
+```
+
+Canonical JSON bytes without `run_id` determine the plan SHA-256. The run ID is
+`xtap-` plus the first 32 hexadecimal characters of that hash. Stored plan bytes
+include the derived run ID and remain stable across attempts.
+
+New raw segments and candidates wait for the next logical run. They cannot
+change the total or reset progress in an active run.
+
+## Immutable work plan
+
+Store the fixed work order in `work-plan.sqlite`:
+
+```sql
+CREATE TABLE worker_queue_plan (
+  ordinal INTEGER PRIMARY KEY,
+  unit_id TEXT NOT NULL UNIQUE,
+  input_hash TEXT NOT NULL,
+  taxonomy_version INTEGER NOT NULL,
+  initial_status TEXT NOT NULL,
+  attempts INTEGER NOT NULL,
+  next_retry_at TEXT
+);
+
+CREATE TABLE worker_registry_plan (
+  ordinal INTEGER PRIMARY KEY,
+  name TEXT NOT NULL UNIQUE,
+  evidence_hash TEXT NOT NULL
+);
+
+CREATE TABLE plan_metadata (
+  singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
+  run_id TEXT NOT NULL,
+  plan_sha256 TEXT NOT NULL,
+  queue_total INTEGER NOT NULL,
+  registry_total INTEGER NOT NULL
+);
+```
+
+Rows use deterministic ordinal order. The database retains only tweets needed
+by unresolved queue work and evidence needed by remaining registry candidates.
+Historical completed rows are represented by the baseline count and bitmap, so
+the worker database stays small. The plan is immutable and downloaded once per
+physical attempt.
+
+## Compact worker state
+
+The checkpoint payload contains compact application state:
+
+```ts
+export type OutputFrontier = {
+  sequence: number;
+  chain_sha256: string | null;
+};
+
+export type EnrichmentCheckpointState = {
+  schema_version: 1;
+  run_id: string;
+  plan_sha256: string;
+  sequence: number;
+  previous_checkpoint_sha256: string | null;
+  queue: {
+    completed_bitmap: Uint8Array;
+    done: number;
+    retrying: RetryRecord[];
+    blocked: BlockedRecord[];
+  };
+  registry: {
+    next_ordinal: number;
+    approved: number;
+    rejected: number;
+  };
+  outputs: {
+    enrichment: OutputFrontier;
+    attempt: OutputFrontier;
+    registry: OutputFrontier;
+    receipt: OutputFrontier;
+  };
+  publication: {
+    state: "pending" | "building" | "uploaded" | "verified" | "published";
+    database_key: string | null;
+    database_sha256: string | null;
+    database_bytes: number | null;
+    manifest: DurableIndexManifest | null;
+  };
+};
+```
+
+A 251,000-item completion bitmap is about 31 KB. Completed queue rows do not
+remain in mutable SQLite state. Retry and blocked arrays contain only unresolved
+records. Output chain frontiers replace an ever-growing list of result objects.
+
+The adapter serializes this state into the content-addressed HF Job Control
+checkpoint bundle. Before restore, it validates bitmap length, counts, ordinals,
+hashes, and output frontiers. Restore finds the latest committed state from
+immutable sequence claims. `current.json` only avoids listing those claims on a
+normal startup.
+
+## Deterministic batches
+
+Each batch has this identity:
+
+```text
+<run_id>:<phase>:<batch_sequence>
+```
+
+The batch intent names its exact ordinals. The result object names the intent,
+contract, predecessor result hash, unit outcomes, and exact immutable segment
+reference.
+
+For each batch, the worker:
+
+1. Reads missing ordinals from the frozen plan.
+2. Saves the batch identity and exact unit list locally.
+3. Runs at most the configured concurrency.
+4. Writes one immutable attempt segment, including concurrent dispatch
+   reservations before provider calls.
+5. Uploads the segment and downloads it for SHA-256 verification.
+6. Updates the local bitmap and retry details, then updates counters and the output chain.
+7. Commits and verifies a small checkpoint bundle.
+8. Publishes and verifies its immutable sequence claim.
+9. Updates and reads back the pointer hint.
+10. Publishes progress from that checkpoint sequence.
+
+A process that stops before the raw result upload may repeat that batch. On
+startup, recovery compares verified raw segments after the frozen source
+snapshot with deterministic result manifests. If a raw segment exists without a
+claimed result, recovery creates the same result manifest, applies it, and
+commits a checkpoint before any provider call.
+
+## Queue semantics
+
+The queue total comes from the frozen plan. A unit is done only when its result
+segment is durable and the verified checkpoint marks its ordinal complete.
+
+Retries retain attempts, error class, next retry time, and input identity.
+Blocked records retain the deterministic reason and evidence needed by the
+existing completion contract. Queue completion requires no pending or retrying
+records and only explicitly validated blocked records. Completion of one frozen
+queue is not a reason to end the physical Job: after verified publication and
+successor activation, the same Job continues the successor when it contains
+unresolved work and the shared physical-attempt budget remains.
+
+Leases remain physical-attempt state. They are not restored. The deterministic
+batch identity and result reconciliation prevent a cleared lease from losing a
+completed result.
+
+## Registry semantics
+
+Registry candidates have a fixed order in `worker_registry_plan`. The checkpoint
+stores `next_ordinal` and decision counters. Every decision is uploaded and
+verified before the cursor advances.
+
+Enrichment work cannot reset the cursor. New candidates wait for the next
+logical run. A completed scan means `next_ordinal === registry_total` and the
+registry output chain agrees with all decisions in that prefix.
+
+## Final public index
+
+Normal worker attempts do not call `DurableIndex.restore()`.
+
+After queue and registry completion, the publication attempt:
+
+1. Commits publication state `building`.
+2. Downloads the exact base public database named by the frozen plan.
+3. Verifies byte count, SHA-256, metadata, and `PRAGMA quick_check`.
+4. Applies only the verified raw output segments named by this run's result
+   manifests.
+5. Verifies queue totals, registry revision, source frontier, and physical row
+   counts.
+6. Runs `PRAGMA quick_check` and uploads the database under
+   `index/databases/<sha256>.sqlite`.
+7. Commits state `uploaded` with the immutable key, digest, byte count, and
+   complete manifest.
+8. Downloads it and verifies the bytes and digest. It then verifies metadata and
+   counts, checks integrity, and commits state `verified`.
+9. Confirms that `index/current.json` still has the frozen predecessor bytes.
+10. Replaces the public pointer, reads it back, and commits state `published`.
+11. Publishes the final receipt from the claimed `published` checkpoint.
+
+Each state change has its own checkpoint claim. A replacement reuses an uploaded
+or verified database and continues at the next step. If the public pointer was
+written before the `published` checkpoint, recovery recognizes the same
+manifest and commits that state without another pointer write.
+
+## Storage layout
+
+Use the existing index Bucket:
+
+```text
+operations/enrichment/runs/<run_id>/plan.json
+operations/enrichment/runs/<run_id>/work-plan.sqlite
+operations/enrichment/runs/<run_id>/checkpoints/sha256-<hash>/checkpoint.hfjob
+operations/enrichment/runs/<run_id>/checkpoints/claims/sequence-<n>/<attempt_id>.json
+operations/enrichment/runs/<run_id>/batches/<phase>/<sequence>/result.json
+operations/enrichment/runs/<run_id>/current.json
+index/databases/<sha256>.sqlite
+index/current.json
+```
+
+All paths except the two current pointers are immutable. The run pointer is a
+startup hint. `index/current.json` keeps its existing public read contract and
+changes only after a verified immutable publication claim.
+
+## Bootstrap preparation and commit
+
+Bootstrap has two phases. Preparation is read-only. Commit owns every remote
+write.
+
+This boundary fixes a defect found during the production transition. The first
+bootstrap resolved taxonomy before it loaded the validated raw snapshot. An
+empty `BucketLog` therefore treated all 19,140 historical segment files as new,
+downloaded them serially, retained them in memory, and reported no exact
+progress. The repair must load known snapshot identities before taxonomy
+resolution.
+
+### Bucket capabilities
+
+Reader interfaces expose only listing, download, and text reads. Writer
+interfaces add upload, deletion, checkpoint publication, activation, and
+pointer replacement. Preparation accepts readers only. The read-only shadow
+binary must not import a provider, checkpoint writer, activation writer, upload
+method, delete method, or schedule method.
+
+A validated snapshot can seed `BucketLog` only after all of these checks pass:
+
+- The snapshot bytes hash to the source revision in `index/current.json`.
+- The snapshot names the expected raw Bucket.
+- Every segment key, size, content hash, and listed object identity is valid.
+- No key or identity is duplicated.
+
+Seeding records known immutable identities. It does not claim that the current
+process downloaded those segment bodies. Head discovery still lists all keys
+so it can detect deletion, size drift, or identity drift. It downloads and
+fully verifies bodies only for new or conflicting tail objects.
+
+Tail body verification can use a small fixed worker pool. Verification results
+must be retained by immutable key and applied in sorted key order, so request
+completion order cannot change the projection. The first missing, deleted,
+changed, duplicate, or invalid object fails preparation.
+
+### Read-only preparation
+
+Preparation does this work in order:
+
+1. Read and parse `index/current.json`. Record its exact bytes and SHA-256.
+2. Require the expected raw Bucket, database key and SHA-256, source snapshot
+   revision, projection contract hash, predecessor identities, and counts.
+3. Use the manifest contract hash only to restore the already validated
+   database and raw snapshot. Verify the 1,328,619,520-byte database, its
+   SHA-256, provenance, schema, and `PRAGMA quick_check` result.
+4. Seed `BucketLog` with the validated snapshot before it reads taxonomy.
+5. Resolve taxonomy from the seeded log. Derive the taxonomy and model contract
+   and require exact equality with the manifest contract before tail work.
+6. Discover the current raw head against the known snapshot. Check every known
+   listing identity and download only the tail.
+7. Verify tail bodies with bounded concurrency and apply them in deterministic
+   order. Durable registry revision 31,201 must appear exactly once after base
+   revision 31,200.
+8. Advance a disposable local database. Verify source continuity, receipt
+   continuity, queue state, registry state, leases, retry timestamps, and all
+   physical row counts.
+9. Export queue IDs and input hashes in deterministic order. Build the completion
+   bitmap and preserve unresolved retry or blocked details.
+10. Build the registry candidate plan with the production discovery algorithm.
+    Verify the saved cursor by scanned count, last name, and receipt time.
+    Exclude old candidates at or before that name, but retain candidates first
+    observed after the receipt even when their names sort before the cursor.
+11. Build and verify the compact local work database. Run its SQLite integrity
+    and provenance checks.
+12. Return a content-addressed `BootstrapCandidate`. Do not write a remote
+    object.
+
+The candidate records:
+
+- The source pointer bytes and digest.
+- The base database key, size, digest, and source revision.
+- The validated raw snapshot and discovered head revisions.
+- The ordered tail digest and counts.
+- The taxonomy, model, provider, worker revision, and contract digest.
+- Queue and registry totals, baselines, ordinals, cursor, and unresolved state.
+- The compact local database path, size, and digest.
+- Exact progress totals and elapsed time for each preparation stage.
+
+Observational timestamps do not contribute to semantic identity. Repeated
+preparation from the same inputs must produce the same candidate identities and
+local artifact digests.
+
+### Exclusive commit
+
+`commitBootstrapCandidate` is the only bootstrap operation that can write the
+plan, work database, initial checkpoint, activation claim, or active-run hint.
+It must revalidate the candidate digest, source head, public pointer bytes, and
+exclusive-writer admission immediately before its first write. It uses the
+existing `bootstrapEnrichmentRun` and `activateEnrichmentRun` formats. The repair
+must not change compact-run keys, checkpoint formats, claims, or public pointer
+formats.
+
+The production bootstrap command composes preparation and commit. The shadow
+command calls preparation only. A candidate prepared while another writer is
+active cannot commit until a later exclusive-writer check passes.
+
+### Progress
+
+Preparation reports these stable stages:
+
+- Manifest and pointer validation.
+- Database download and byte verification.
+- Snapshot load and known-identity seeding.
+- Head listing and known-identity checks.
+- Tail body verification.
+- Deterministic tail application.
+- Queue and registry projection.
+- Compact database validation.
+- Candidate validation.
+
+Each stage reports exact completed and total items or bytes and elapsed
+milliseconds. These events are local observations. They do not add a remote
+progress API or a new durable format.
+
+### Read-only shadow validation
+
+While the original production importer is active, a shadow run must remain
+structurally read-only. For this transition, session 216 and PID 3252076 identify
+the original importer and only possible writer:
+
+1. Create a clean repair worktree from merged revision
+   `c0a1a95ebf6a4209f9f4d9505e9b6748af08216b`. Leave the dirty documentation
+   tree and exact deployment worktree unchanged.
+2. Record the original process state and I/O, active xTap Jobs, Space revision,
+   enrichment state, suspended schedule, public pointer bytes, and complete
+   `operations/enrichment/runs` object listing.
+3. Run all deterministic tests, repository gates, mutation tests, and review in
+   the repair worktree. Fix all P0 and P1 findings before a live shadow run.
+4. Start the shadow in a fresh local directory with only the approved storage
+   credential mapped to `HF_TOKEN`. Do not load `INFERENCE_TOKEN`.
+5. Use conservative tail concurrency. Stop only the shadow if it harms original
+   process I/O or host health.
+6. Monitor exact shadow progress and the original process independently. Do not
+   signal, cancel, restart, or replace the original process while it progresses.
+7. If the original writes during shadow preparation, treat that as observed
+   source drift. The shadow remains read-only and must revalidate its candidate
+   before later use.
+8. Verify the candidate schema and digest, local artifact sizes and hashes,
+   SQLite integrity and provenance, source and ordinal continuity, queue and
+   registry parity, and exact single application of revision 31,201.
+9. Prove provider calls are zero.
+10. Re-read the public pointer, active-run object listing, Jobs, Space, and
+    schedule after the shadow exits. Require no shadow-caused change.
+
+The shadow writes only local candidate and evidence files. It cannot write a
+plan, checkpoint, activation claim, publication, pointer, Space setting, or
+schedule. It cannot launch a Hugging Face Job.
+
+Do not commit, push, open, merge, or deploy the repair during shadow validation.
+If the original importer completes validly, compare its immutable result with
+the shadow candidate and prefer the accepted production result. If the original
+fails without writes, the production transition can consider the reviewed
+repair only after it proves that no writer is active and repeats all admission
+checks.
+
+## Files
+
+Add:
+
+```text
+space/src/enrich-run-plan.ts
+space/src/enrich-checkpoint.ts
+space/src/enrich-state.ts
+space/src/enrich-batch.ts
+space/src/enrich-planned-command.ts
+space/src/bootstrap-enrichment-run.ts
+space/src/prepare-enrichment-bootstrap.ts
+space/src/verify-enrichment-bootstrap-main.ts
+space/src/bootstrap-enrichment-main.ts
+```
+
+Change:
+
+```text
+space/src/enrich-command.ts
+space/src/enrich-worker.ts
+space/src/enrich-store.ts
+space/src/durable-index.ts
+space/src/job-progress.ts
+space/src/bucket-log.ts
+space/package.json
+package.json
+```
+
+`durable-index.ts` remains the public-index import and publication boundary.
+Normal queue and registry work uses the work plan and checkpoint plus immutable
+raw output segments. The new preparation module is the shared read-only import
+path. The production bootstrap adds commit capability at its composition root.
+The shadow composition root has read capability only.
+
+## Forced-stop tests
+
+Stop the worker at each boundary:
+
+- Before result upload.
+- After result upload and before verification.
+- After result verification and before checkpoint upload.
+- After checkpoint upload and before claim publication.
+- After claim publication and before pointer-hint publication.
+- After pointer-hint publication and before progress publication.
+- During public database build.
+- During public database upload.
+- After the `uploaded` checkpoint and before verification.
+- After the `verified` checkpoint and before `index/current.json` replacement.
+- After pointer replacement and before the `published` checkpoint.
+- After the `published` checkpoint and before the final receipt.
+
+Each replacement must restore verified state, avoid repeating durable batches,
+and produce the same final output as an uninterrupted reference.
+
+Corruption tests cover wrong run and plan identities, wrong hashes or sizes,
+missing predecessors, multiple roots, different claims for one sequence,
+malformed bitmaps, inconsistent counters, changed source snapshots, missing
+result segments, broken output chains, invalid SQLite files, pointers outside
+the verified chain, and conflicting public publication claims. A separate crash
+test leaves the pointer one checkpoint behind and proves that the next worker
+repairs it without repeating completed work.
+
+## Verification
+
+Run:
+
+```bash
+npm run check
+npm run coverage
+npm run mutate
+npx slophammer-ts check .
+npm run doctor
+npm run storage:verify
+npx -y @simpledoc/simpledoc check
+```
+
+Add tests for plan determinism, production import, checkpoint round trips,
+missing-only queue work, registry cursor continuity, deterministic batch replay,
+publication recovery, and resumed-versus-uninterrupted equivalence.
+
+The single-run Job change also needs direct tests for:
+
+- Running only one logical plan when its verified successor has work.
+- Passing the physical Job's full time and cost limits to that plan.
+- Propagating interruption without starting another logical run.
+- Restoring the successor in the next scheduled Job without duplicate results,
+  checkpoints, receipts, schedules or physical Jobs.
+
+The bootstrap repair also needs tests for:
+
+- Validated snapshot seeding, known-body skipping, and rejection of duplicate,
+  missing, deleted, changed, malformed, or wrong-Bucket identities.
+- The tail verification concurrency ceiling, deterministic application order,
+  fail-fast behavior, and exact progress totals.
+- Manifest-first database and snapshot restore followed by independent taxonomy
+  and model contract verification.
+- Queue and registry parity, source continuity, and exactly one application of
+  registry revision 31,201.
+- Read-only capability enforcement, zero provider construction, and zero remote
+  writes from the shadow binary.
+- `BootstrapCandidate` schema, canonical bytes, digest, local artifact hashes,
+  and repeated-run determinism.
+- Prepared-state equivalence between seeded-tail preparation and full replay.
+- Live before-and-after evidence for pointer bytes, active-run objects, Jobs,
+  Space state, original process state, progress counts, and provider calls.
+
+## Production baseline
+
+The final legacy Job completed on 2026-08-20. Use this generation as the only
+bootstrap input:
+
+- Public pointer SHA-256:
+  `5a259cc95a3437d1c983d7d123efe476cf6bf3d48e14f97fec31bb4067834740`.
+- Database SHA-256:
+  `9e9940b063f615dae42ed86ba08ec6aab27873b61b8da8b7ad42ee8fff413c12`.
+- The database is 1,328,619,520 bytes.
+- The queue has 252,042 completed records out of 252,504. It has 462 pending
+  records, no leases, and no retry timestamps.
+- The registry scan has checked 15,840 of 25,675 candidates.
+- The database contains registry revision 31,200.
+- The later durable registry revision is 31,201. The importer must apply it
+  exactly once after the base revision.
+- Settled restoration spend is $102.3746080.
+- Conservative exposure before the new canaries is $602.3746080.
+- The approved cumulative restoration ceiling is $2,000.
+- The merged worker revision is
+  `c0a1a95ebf6a4209f9f4d9505e9b6748af08216b`.
+
+The final receipt, pointer, full database hash, `PRAGMA quick_check`, source
+identity, queue state, registry revision, and raw segment hashes were verified.
+The legacy schedule is suspended and no enrichment Job is active.
+
+## Production transition
+
+This is a hard replacement. Do not start another legacy full-index worker.
+Keep all schedules suspended until the final activation step.
+
+1. Re-read the final receipt, pointer, database, raw segment frontier, Job list,
+   and schedule. Stop if any identity differs from the production baseline.
+2. Create or verify a clean deployment worktree at the exact merged worker
+   revision. Keep documentation updates outside that worktree. Do not deploy
+   uncommitted files or a later revision, and do not merge or change the pinned
+   HF Job Control dependency during this transition.
+3. Before each paid action, add settled spend, prior unreceipted attempt caps,
+   active reservations, and the next worst-case action. Stop before an action
+   that can take conservative exposure above $2,000.
+4. Load only the approved storage and inference credentials in a short-lived,
+   non-traced process. The Space, canary Jobs, and scheduled Job may contain
+   only `HF_TOKEN` and `INFERENCE_TOKEN` where each interface requires them.
+5. During the bounded bootstrap repair, leave the original importer running as
+   the only possible writer. Run the repaired shadow with `HF_TOKEN` only. It
+   prepares and verifies a local candidate but cannot commit or activate it.
+   If the original importer completes validly, use its immutable result and
+   compare it with the candidate. If it fails without writes, use the repaired
+   commit path only after all tests and review pass and a new admission check
+   proves no writer is active.
+6. Run `npm run enrich:bootstrap-run` from the accepted exact worktree only when
+   exclusive-writer admission passes. The importer must restore the exact base,
+   advance over later valid raw segments including registry revision 31,201,
+   write the frozen plan, compact work database, bootstrap checkpoint,
+   activation claim, and active-run hint, then read them back. A repaired
+   bootstrap must prepare the candidate first and revalidate the pointer,
+   source head, candidate digest, and writer exclusion before commit.
+7. Treat `operations/enrichment/runs` as the isolated future-runtime namespace.
+   The suspended legacy schedule and old deployed worker do not read it.
+   Capture `index/current.json` before and after import and require byte equality.
+8. Validate the imported plan SHA, run ID, source snapshot, worker revision,
+   contract hash, base reference, queue and registry order, cursor, ordinal
+   coverage, checkpoint sequence, activation chain, claims, output frontiers,
+   object hashes, and unresolved rows. Revision 31,201 must be neither missing
+   nor applied twice.
+9. Restore the active plan and checkpoint twice into clean directories, with
+   the second run using no local cache. Both restores must return the same state
+   in less than five minutes, make zero provider calls, and leave all production
+   pointers unchanged.
+10. Deploy the merged revision from the exact deployment worktree with:
+
+```sh
+npm run update -- osolmaz/xtap-pool
+```
+
+Wait for the exact repository and runtime revision to become `RUNNING` and
+`READY`. Require HTTP 200 from `/readyz`, ready storage, and enrichment
+disabled before schedule repair.
+
+11. Inspect the schedule contract without mutation:
+
+    ```sh
+    npm run --silent doctor -- osolmaz/xtap-pool --json
+    ```
+
+    Then use `npm run doctor -- osolmaz/xtap-pool --fix` to replace the legacy
+    schedule. Read back exactly one suspended, non-concurrent `cpu-upgrade`
+    schedule at `17 */6 * * *`. It must run
+    `space/dist/src/enrich-job-main.js`, use the merged deployment contract,
+    have a 2,700-second platform timeout and 2,400,000-millisecond worker budget,
+    keep the checked-in $10 run ceiling, and use only `HF_TOKEN` and
+    `INFERENCE_TOKEN`.
+
+12. Recheck cost admission and trigger one physical Job from the suspended
+    schedule. Require restore under five minutes and wait for a verified output
+    claim and checkpoint boundary before any interruption.
+13. Save the Job inspection, logs, active run, plan, checkpoint hashes, output
+    frontier, provider receipt frontier, and cost evidence. Cancel only that
+    physical Job. Do not create a receipt for work that did not publish one.
+14. After the canceled Job is terminal and no xTap Job is active, trigger the
+    same suspended schedule once. The replacement must use the same run ID and
+    plan SHA, restore the prior checkpoint, reconcile orphan outputs, process
+    only missing ordinals, and keep progress monotonic. At most the documented
+    in-flight concurrency batch may repeat.
+15. Continue the frozen run one physical Job at a time. After each attempt,
+    validate the terminal state, exact receipt, referenced outputs, checkpoint
+    chain, claims, cost, and no-overlap state before another trigger. A valid
+    deterministic blocked record is data, not a shared failure.
+16. After queue and registry completion, validate resumable publication. The
+    database key must equal its SHA-256, `PRAGMA quick_check` must return `ok`,
+    metadata and counts must match the frozen plan and completed checkpoint,
+    and every manifest and receipt reference must exist. Move
+    `index/current.json` only after full read-back.
+17. Resolve every activation claim and require one contiguous chain. The
+    completed plan must create exactly one verified successor, with no competing
+    plan or work object. The active hint must match the last valid claim.
+18. Run the two-Job steady canary while the schedule is suspended. Use the hard
+    ceiling reported by setup doctor and pass the approved ceiling explicitly:
+
+    ```sh
+    npm run doctor -- osolmaz/xtap-pool --fix --canary \
+      --approved-cost-ceiling-usd=<approved-canary-ceiling>
+    ```
+
+    Validate both receipts and every referenced output. Both Jobs must use the
+    successor, restore bounded state, and remain non-overlapping.
+
+19. Activate only after every earlier gate passes:
+
+    ```sh
+    npm run doctor -- osolmaz/xtap-pool --fix --canary \
+      --approved-cost-ceiling-usd=<approved-canary-ceiling> --enable-schedule
+    ```
+
+    Read back exactly one active canonical schedule. Remove the legacy schedule
+    and source revision. Leave no physical xTap Job active at handoff.
+
+## Backlog-drain rollout
+
+The same-attempt successor change uses the existing production contract. Do not
+deploy it while physical Job `6a92b7ae45686a1580c14a67` or another matching
+Job is active. First wait for the Job to become terminal, then validate its
+receipt, result batches, checkpoint, public index, manifest, and active
+successor. Do not discard or repeat its durable work.
+
+Use this order:
+
+1. Run the complete repository checks and recovery tests on the final branch.
+2. Review and merge one narrow pull request.
+3. Wait for the xTap Space to report `RUNNING` at the exact merged revision.
+4. Use setup doctor to reconcile one exact suspended schedule. Run the required
+   sequential canary before activation. Keep the schedule non-concurrent at
+   `17 */6 * * *` on `cpu-upgrade` with the existing secrets and limits.
+5. Trigger or resume only one physical Job at a time. After each terminal
+   attempt, validate its cumulative physical-attempt cost, receipts, checkpoint
+   chain, result batches, index read-back, manifest, active successor, and zero
+   overlap before another attempt.
+6. Continue until one captured enrichment status reports `pending=0`,
+   `running=0`, `retrying=0`, and `blocked=0`. Record that source revision and
+   cutoff with the verified active successor and public index.
+7. Use the existing OurModels publication tooling only after that empty xTap
+   revision verifies. Reconcile its exact producer, generate and verify one
+   fresh publication from the captured revision, and leave one active
+   non-concurrent six-hour OurModels schedule.
+
+This rollout does not add a cross-service trigger or a new runtime contract.
+The recurring xTap Job drains successors within its existing physical-attempt
+bounds. The recurring OurModels Job keeps its own frozen-plan and cost
+contracts.
+
+## Failure handling
+
+Fail closed before new provider calls or pointer writes when an identity, hash,
+ordinal, predecessor, cursor, source revision, or count differs.
+
+If import writes only part of the new immutable graph, keep the public pointer
+and schedules unchanged. Re-run only after every existing immutable object
+matches the intended bytes.
+
+If a read-only shadow detects pointer, source, or object drift, stop only the
+shadow and preserve its candidate and evidence. It must not signal the original
+importer or commit remote state. Revalidate the candidate only after the source
+and writer state are stable.
+
+If shadow CPU, memory, disk, or network use harms the original importer or host,
+stop only the shadow. Keep its bounded concurrency low and preserve the original
+as the only writer.
+
+If deployment fails, keep the schedule suspended. Repair only the bounded defect
+inside this repository, run the full quality gates and review, and repeat the
+failed gate. Do not substitute an upstream dependency or unreviewed runtime.
+
+If a Job stops outside a verified boundary, preserve its inspection and logs,
+then restore from the last valid checkpoint and reconcile orphan outputs. Do not
+fabricate a receipt or mark unverified work complete.
+
+If repeated work exceeds one in-flight concurrency batch, stop the run and
+investigate the claim frontier before another paid attempt.
+
+If the next action can take conservative exposure above $2,000, do not launch
+it. Preserve the current state and report the exact cost evidence.
+
+## Acceptance criteria
+
+The work is complete when:
+
+- Read-only preparation seeds the validated snapshot before taxonomy lookup,
+  downloads no known historical body, verifies only the tail, and reports exact
+  progress.
+- Shadow validation makes zero provider calls and leaves the public pointer,
+  active-run objects, Jobs, Space, and schedule unchanged.
+- The exact legacy generation and revision 31,201 are imported without changing
+  the public pointer or any raw object.
+- A normal continuation does not download the public SQLite database.
+- Two cold checkpoint restores take less than five minutes and make zero
+  provider calls.
+- Checkpoint restore takes less than five minutes at current production size.
+- Queue and registry totals remain fixed for one logical run.
+- Completed queue work and registry decisions are not repeated.
+- At most one in-flight concurrency batch repeats after interruption.
+- Progress stays monotonic across physical attempts.
+- Registry work never resets because queue work or new source data appeared.
+- A completed run activates exactly one verified successor. The same physical
+  Job uses it while unresolved work and the original time and cost budgets
+  remain; otherwise the next scheduled Job resumes it without schedule
+  replacement.
+- One physical Job uses one command start time and one cumulative $10 inference
+  ceiling across every successor it processes.
+- A zero-work successor stops the physical Job without polling, spinning, or
+  creating duplicate work.
+- Interrupted final publication resumes from its last claimed state and reuses
+  uploaded or verified immutable artifacts.
+- Resumed and uninterrupted runs produce the same logical database and manifest.
+- Invalid, duplicate, overlapping, or conflicting ordinal state fails before
+  new provider calls or pointer writes.
+- Two steady canary receipts and every referenced output validate.
+- Settled spend and conservative exposure stay at or below $2,000.
+- The Space is healthy on the merged runtime with enrichment disabled.
+- Production uses one active non-concurrent schedule at `17 */6 * * *`, with
+  only `HF_TOKEN` and `INFERENCE_TOKEN`, and no legacy runtime path.
+- No physical xTap Job is active at final handoff.

docs/2026-09-07-consumer-http-handoff.md

@@ -0,0 +1,252 @@
+---
+title: Incremental consumer HTTP contract
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-09-07
+updated: 2026-09-10
+tags: [consumer-api, implementation]
+---
+
+# Incremental consumer HTTP contract
+
+This is the HTTP handoff for the [incremental consumer plan](2026-09-07-incremental-consumer-and-engagement-history-plan.md). The source schemas are `space/src/consumer-http-contract.ts`, `consumer-page.ts`, `consumer-observations.ts`, `consumer-privacy.ts`, and `consumer-query.ts`. Runtime and route code are `consumer-runtime.ts` and `consumer-routes.ts`. This replaces the version-1 machine contract in place. Deployment and the Our Models cutover remain operator gates.
+
+## Requests and grants
+
+| Operation                 | Request                                                                                 | Service-account scopes                             |
+| ------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------- |
+| Start bootstrap           | `/api/units?author_ids=123,456&labels=ai&label_mode=any&publication=public-original`    | `units:read`, `taxonomy:read`                      |
+| Continue bootstrap        | `/api/units?cursor=<cursor>`                                                            | Same                                               |
+| Start or continue changes | `/api/changes?after=<cursor>`                                                           | `units:read`, `taxonomy:read`, `observations:read` |
+| Start history             | `/api/observations?at=<fully-consumed-cursor>&post_ids=123,456&since=<UTC>&until=<UTC>` | `units:read`, `observations:read`                  |
+| Reconcile privacy         | `/api/reconcile?cursor=<recovery-cursor>`                                               | `units:read`                                       |
+| Continue history          | `/api/observations?cursor=<next_cursor>`                                                | `units:read`, `observations:read`                  |
+
+Every request checks current authorization. An authorized member session also permits these reads. Existing credentials gain no scopes. An operator must issue an explicit grant for `observations:read`. Default grants stay unchanged. Explorer whole-unit browsing uses `/api/explorer/units` and requires a member session; it is not a machine recovery route. The explorer's separate `/api/tweets` route is unchanged.
+
+The selection is fixed: sorted, unique exact `author_ids`, sorted unique `labels` (default empty), `label_mode` (`any` by default, or `all`), optional `free_label`, and required `publication=public-original`. Repeated or unsupported query options fail. Continuations may repeat matching selection fields; conflicting fields fail. Names are case-sensitive. History continuation binds source, IDs, range, and selection; optional repeated fields must agree. History IDs are distinct and sorted. Times must be UTC with `Z`; they normalize to millisecond precision.
+
+## Change envelope
+
+All bootstrap and change responses have exactly these fields:
+
+```json
+{
+  "schema_version": 1,
+  "source": "<full raw snapshot SHA-256>",
+  "complete_through": null,
+  "observations_through": null,
+  "history_since": "<inclusive UTC>",
+  "history_until": "<exclusive UTC>",
+  "cursor": "<signed next cursor>",
+  "has_more": true,
+  "changes": []
+}
+```
+
+The two coverage timestamps are null or UTC. Observation freshness does not promise complete observation coverage. The source identifies the exact raw object set, including late arrivals.
+
+Changes are the existing `metadata`, `unit_upsert`, `unit_remove`, and `observation` variants in `consumer-page.ts`. Bootstrap starts with a metadata-only page, even for an empty selection. Continue that page to receive upserts or the empty final page. The signed `metadata_sent` field prevents repeats at `limit=1`. Changes emit metadata only when the frozen taxonomy or approved registry state differs. Registry counters and coverage movement alone do not cause metadata changes.
+
+Each `unit_upsert` has exactly `type`, `content_hash`, `post_content_hashes`, and `unit`:
+
+```json
+{
+  "type": "unit_upsert",
+  "content_hash": "<canonical unit SHA-256>",
+  "post_content_hashes": [
+    "<canonical post SHA-256 for unit.posts[0]>",
+    "<canonical post SHA-256 for unit.posts[1]>"
+  ],
+  "unit": {
+    "id": "...",
+    "posts": ["<post body 0>", "<post body 1>"],
+    "contributors": ["..."],
+    "preset_labels": [],
+    "free_labels": []
+  }
+}
+```
+
+The example abbreviates post bodies. `post_content_hashes` is a required array of 64-character lowercase hex SHA-256 strings, with exactly one entry per `unit.posts` entry, in the same order. Entry `i` equals the shared source `contentHash(unit.posts[i])` and the `content_hash` from `normalizeObservation` for that exact body. It is present even when the post has no observations in the advertised hot window. The API validates each hash against the returned body; the historical reader also checks the reconstructed body against its stored source hash. No client normalization or observation lookup is required.
+
+Pair each body with its hash before indexing or sorting. Same-ID contributor copies do not collapse into a hash map: if multiple copies are returned, each has its own aligned entry. The current unit reader selects one body per post ID with the existing deterministic observed-time, privacy, content-hash, and contributor order. The upsert hash identifies that selected body, not an arbitrary other contributor's copy or all past versions. Observations can identify other retained versions; compare their hash with the saved body's supplied hash. Sample counters, capture/receipt times, attribution, transport metadata, and follower count do not change the post content hash. Content edits do. The new array is outside the post bodies and does not change either canonical hashing rule. It is included in complete-item and final response byte limits.
+
+Empty pages can have `has_more=true`. Continue until false. The final cursor is an idle cursor for the fully consumed target. Commit each page and cursor together. Do not poll changes or start history with an unfinished bootstrap cursor. Bootstrap does not deliver observations; use the final cursor for initial bounded history. Activation and fresh change phases can deliver the same logical observation. Deduplicate by observation `id`.
+
+## History envelope
+
+History has `schema_version: 1`, `source`, `history_since`, `history_until`, `observations`, `coverage`, `has_more`, and nullable `next_cursor`. Its bounds are the requested inclusive/exclusive range, checked against the source's advertised range. Each coverage item has `post_id`, `state` (`available`, `no_history`, or `unavailable`), `count`, `earliest`, and `latest`. Unavailable counts and times are null. Coverage describes the complete requested range, not only the current page. Pages sort by `(post_id, observed_at, id)`.
+
+A final history page has `has_more=false` and `next_cursor=null`. A history cursor cannot become a global or idle cursor. Keep the original fully consumed source cursor separately.
+
+## Limits and recovery
+
+Default page limit is 200; maximum is 500. History permits at most 100 IDs and 30 days. The target response size is 2 MiB; the hard serialized limit is 8 MiB. A complete oversized item fails without an acknowledgment cursor. Page sequences last at most 48 hours; sources last at most 30 days from context creation. The earlier deadline applies when a history or change sequence uses an older source. Context cleanup waits 32 days. Raw snapshot bases and raw segments are retained; old full databases are not needed.
+
+Each read has a 60-second wall deadline, including source/context I/O and mutex wait. A request that reaches an exact full-selection coverage calculation gets a five-minute absolute deadline measured from the original request start. No other stage extends its deadline. Clients that can request a fresh source context must allow at least 315 seconds for a response. SQLite runs in at most two persistent read-only child processes. Deadline cancellation kills a running child, including native SQLite. Requests do not copy or rebuild the database. The database must contain every exact pinned source descriptor and the current verified source boundary.
+
+Errors use `{"error":{"code":"...","message":"...","recovery":{}}}`. The recovery field is present only when needed. No error includes an acknowledgment cursor.
+
+| HTTP status | Meaning and action                                                                                                        |
+| ----------- | ------------------------------------------------------------------------------------------------------------------------- |
+| 400         | Invalid, conflicting, unsupported, or wrong-kind query/cursor. Correct the request.                                       |
+| 401         | Missing, invalid, expired, or revoked authentication.                                                                     |
+| 403         | Valid service credential lacks an explicit required scope.                                                                |
+| 409         | `privacy_changed`: apply the removal procedure below. `contract_changed`: use an explicit bootstrap for the new contract. |
+| 410         | Cursor/context expired or history outside its retained range. Use explicit bootstrap or operator history recovery.        |
+| 413         | A complete source item or final serialized response exceeds its bound. Do not advance.                                    |
+| 429         | Consumer capacity is full. Retry the same cursor after five seconds.                                                      |
+| 502         | Source/context read or verification failed. Retry the same cursor; persistent failures need operator repair.              |
+| 503         | Source projection unavailable, database replaced, worker failed, or deadline exceeded. Retry the same cursor.             |
+
+## Coverage maintenance
+
+Idle changes polls still pin a fresh immutable context under the existing mutex. The source, current metadata, context lease, and history window are refreshed. Every page still checks authorization, current privacy, database identity, and exact pinned source containment. HTTP response shapes are unchanged.
+
+`consumer-coverage-update.ts` selects one of three internal coverage paths:
+
+- **Reuse:** the base has the same selection, contract, projection, taxonomy version, and approval membership for its selected free label. Exact source-set differences and the existing `source_segments` counts and source-effect indexes show no relevant post or new result effects. Identical sources, bookkeeping writes, ignored/replayed attempts, duplicate results, and unrelated author units retain both clocks without reading post bodies. Attempt replay uses the existing queue rules: completed work stays done, and retry/blocked states remain incomplete. There is no second queue projection.
+- **Metrics:** source-index proofs compare affected posts' latest contributor-copy hash sets, the selected content hash, and the start of the content run at the base with current `unit_members` facts. A same-time hash conflict uses the existing privacy tie-break for that affected post. Once the entire delta passes, observation coverage reads only affected posts and their units' privacy/eligibility facts, in batches of at most 100 post IDs. It retains the prior maximum and includes any later permitted observation. Delayed samples cannot move the maximum backwards. No unrelated cohort body is read.
+- **Semantic recalculation:** new selected results, changed content or membership, a corrected content-run clock, a changed taxonomy version, or a selected free-label approval change require a current selected-coverage calculation. This can read the selected cohort. It computes the new maximum directly, so withdrawal of the previous maximum can lower it or return null. Private, unselected, pending, and currently retweeted post samples do not advance observation coverage. This is a coverage calculation, not a fallback content download.
+
+Taxonomy descriptions and unrelated approval changes can leave coverage unchanged while still emitting the required metadata change. Registry revision numbers alone do not determine either decision. The page privacy and fresh-observation queries use fixed join order from their bounded source candidates; SQLite cannot start by evaluating privacy on the whole content table.
+
+There is no new persistence, table, index, projection identity, or classifier change. `eligibleUnits` in `enrich-store.ts` gains only an optional internal unit-ID filter for the affected-post query. Keep that filter and the source-driven join order when combining later parent changes. The running source-index preparation does not need a restart or rebuild.
+
+The regression fixture measures the real forked worker. Test-only wrappers count evaluated JSON body predicates and returned post-body fields, including reads outside the coverage operation. The bootstrap is a positive control for the counter. With 250 posts, twenty identical-source polls and twenty harmless-revision comparisons read **zero post bodies**; all 40 coverage evaluations use the reuse path. This includes a fresh worker process and context-store restart, blocked/dispatched attempt replay, and an unpublished registry candidate. With 260 posts, ten batches of ten metric observations update ten posts: only those ten post IDs are read; the 250 unrelated post IDs are not read. The clocks are checked against an independent full-coverage calculation. Additional tests cover delayed samples, corrected clocks, mixed edits and counters, pending-to-eligible changes, private/retweet maximum withdrawal, contributor winner changes, and actual metadata changes.
+
+These are small-fixture read counts, not a completed production-size performance check. They measure logical body access, not SQLite cache-page I/O. The full-source local canary below adds real reader timings. Cold restore and remote transport still need separate measurements.
+
+Local commands for this branch are `npm run build --workspace shared`, `npm run build --workspace space`, `npx --no-install vitest run space/tests/consumer-coverage-update.test.ts`, `UV_OFFLINE=1 npm run check`, and `npm_config_offline=true npx --no-install --package=@simpledoc/simpledoc simpledoc check`. The merged build hooks also build space before tests and coverage. The counts-only ingest acknowledgment, extension version 0.26.0, and bootstrap implementation are unchanged by the initial-coverage fix.
+
+## Full-source initial coverage fix
+
+The September 7 local canary used the immutable prepared source `36f27e68bf5ba7cfa53a6fedd7ba6fa66906113d058a048dd1384e7aa37034aa`: 38,219 segments, 689,421 logical observations, and a 4,811,792,384-byte SQLite database. The current Our Models publication account file selects 123 exact author IDs. Their source history contains 114,603 distinct unit candidates before current eligibility and label filtering. The source database was opened read-only. No index rebuild, schema migration, source copy, or publication was required.
+
+The original first request reached the 30-second deadline in `coverage` (30.012 seconds in the parent log). `EXPLAIN QUERY PLAN` showed observation membership driven by `idx_observation_source_segment (segment_key=? AND observation_id=?)`. This probes the large segment selection for each observation. Content completeness also took two bounded queries of 6.05 and 4.43 seconds in the diagnostic process; the following observation query had not finished when that process reached its 35-second diagnostic limit.
+
+The fix keeps every eligibility, privacy, and exact source predicate:
+
+- `consumer-coverage.ts` and `historical-unit-reader.ts` use the existing observation-ID index for source membership. The membership set is still the exact pinned source. The fixed observation-coverage query took 10.688, 4.673, and 5.496 seconds in three local reads and returned the same maximum each time.
+- `source-effect-store.ts` uses the existing unit-ID index for bootstrap order. SQLite can stop at the next page boundary instead of sorting all remaining author matches. A measured 33-ID candidate query fell from 1.747 seconds to 0.011–0.016 seconds across three repetitions, with identical rows.
+- Historical observation reconstruction for the same measured slice fell from 0.413 seconds to 0.023 seconds across three repetitions, with identical rows.
+- `recorded-result.ts` batches exact-input result lookups for the requested unit slice. It builds source membership once, probes by result hash, and keeps deterministic result order, quote validation, and the 50-candidate limit for each unit. The single-unit caller uses the same implementation. No classifier input, model, prompt, source effect, or projection identity changes.
+- `consumer-observations.ts` uses observation-ID lookups for history rows and history coverage. The same 100-post source query spent 5.338 and 5.344 seconds in these two statements before the fix. The final history canary returned 133 observations in three pages, each taking 0.610–0.760 seconds, with identical coverage across pages and a null final history cursor.
+- An exact no-op returns an empty page with an idle cursor. It skips candidate SQL when there are no source additions or changed approvals, after validating source containment. The earlier implementation returned an unnecessary empty observation continuation. This was detected after the complete bootstrap, and the saved global cursor was reused for the fixed probes.
+- Bootstrap uses the historical reader's existing 200-unit request bound. Content deltas retain their 32-candidate bound. The 2,000-post reconstruction bound, complete-item byte checks, response limits, and 30-second request deadline remain in force. A measured page processed 200 candidates and returned 59 eligible units in 0.761 seconds after these changes.
+
+These query timings are diagnostics on one host, not cold-storage or cloud measurements. The meaningful acceptance test is a complete, correct local source read within the unchanged per-request deadline, followed by warm no-op and restart probes. The full sequence uses local immutable context files and the local pinned snapshot in place of Bucket metadata transport. It does not measure remote metadata latency, source download, browser capture, metric-delta publication, or website freshness. SQL plan tests and the existing body-read tests check the access paths separately.
+
+The private harness is `/tmp/xtap-consumer-initial-coverage/consumer-read-canary.mjs`; its imports point to this worktree. The parent's harness is unchanged. Reproduce with `timeout -k 3s 5450s node /tmp/xtap-consumer-initial-coverage/consumer-read-canary.mjs <prepared-bootstrap.sqlite> <private-output-directory>`. It checkpoints every page locally, limits the complete run to 5,000 pages and 90 minutes, replays page 10 after a worker/context-store restart, polls the idle cursor twenty times, and performs another restart probe. These harness limits do not change the production deadline. The compiled source was built with `npm run build --workspace shared` and `npm run build --workspace space`.
+
+The successful report is `/tmp/xtap-consumer-initial-coverage/run-3/report.json`; bounded history is in `run-3/history-report.json`. The bootstrap log is `canary-3.log`, and the resumed no-op/restart log is `canary-final.log` under the same private root. The bootstrap query implementation stayed fixed during those 575 pages. The later no-op transition and history lookup fixes were tested by resuming that completed checkpoint; no accepted bootstrap page was discarded or read again as a new bootstrap.
+
+| Full-source local measurement                  |                                   Result |
+| ---------------------------------------------- | ---------------------------------------: |
+| Bootstrap pages                                |                                      575 |
+| Unit upserts                                   |                                   34,743 |
+| Distinct post IDs                              |                                   68,291 |
+| Serialized bootstrap bytes                     |                               90,116,606 |
+| Largest serialized page                        |                            326,282 bytes |
+| Sum of bootstrap request times                 |                          523.959 seconds |
+| Bootstrap request median / p95 / maximum       |           0.916 / 1.092 / 13.755 seconds |
+| First coverage worker / complete first request |                  13.203 / 13.755 seconds |
+| Identical page-10 replay after restart         |                            0.788 seconds |
+| Each no-op response body                       |                  598 bytes; zero changes |
+| Twenty no-op polls: minimum / p95 / maximum    |            1.463 / 1.609 / 1.682 seconds |
+| Final restart poll                             |              1.682 seconds; zero changes |
+| Bounded history                                | 100 posts; 133 observations; three pages |
+| History total invocation time                  |                            2.137 seconds |
+| Resumed no-op/restart invocation time          |                           33.301 seconds |
+
+**523.959 seconds is the sum of all 575 saved bootstrap page timings. The resumed report's 33.301 seconds covers only its no-op/restart invocation.** Neither value includes all diagnostic work or time between invocations. The source file was preserved: device, inode, byte size, and modification time match the recorded pre-canary values. Context transport was local immutable files and a local pinned snapshot, with zero network reads/writes and no credentials. The canary's authorization callback is local; deployed grants remain an operator gate.
+
+The host exposed 20 ARM64 CPUs and 130,596,577,280 bytes of RAM; the CPU model was unavailable. The database page cache was warm from the read-only diagnostics. Some early bootstrap pages overlapped local checks. The no-op timing run followed those checks. The resumed main process reached 564,800 KiB peak RSS; this is neither bootstrap peak memory nor combined worker memory. These are one-host measurements, with repeated query/no-op probes rather than a statistical confidence interval. All measured requests stayed below the unchanged 30-second deadline, and the uninstrumented warm no-op p95 met the two-second local target.
+
+A separate real-database worker probe counted 8,011 body evaluations across 319 post IDs as a positive control. Twenty no-op polls and one restart poll then produced **zero post-body evaluations** across 42 worker operations; all 21 coverage operations reused their base clocks. The private result is `/tmp/xtap-consumer-initial-coverage/body-read-report.json`. Its instrumented timings are not used for the latency target. It measures logical JSON-body evaluation and returned body fields, not physical SQLite cache-page I/O.
+
+The bounded history harness is `/tmp/xtap-consumer-initial-coverage/consumer-history-canary.mjs`, run with the same database and completed checkpoint directory under a 335-second outer diagnostic bound. Each HTTP request still has the production 30-second deadline. It verifies all 133 distinct observations against the returned coverage, uses history continuations for all three pages, and rejects a history cursor that becomes global/idle. The body probe uses `consumer-noop-probe.mjs` with a separate copy of local context metadata and the completed source cursor; it does not copy the database or restart bootstrap.
+
+The report records base commit `b229b70` plus hashes of the final changed source files. The canonical JSON hash of that source-file map is `d21b9a78cadb5039fe11b7f5be6c8ab268c5943c47a13db36aa307cc5f7ef273`. The publication-account file hash is `850bb1009aaa5f4deb40055cc062161111d6cec6498b2ea132ef085f320d3949`. Source coverage remained `complete_through=2026-09-06T22:21:24.712Z` and `observations_through=2026-09-06T22:56:43.226Z`.
+
+### Final-code repeat and candidate bound
+
+A fresh run at committed `532668f` repeated all 575 bootstrap pages without changing source or code during the run. It returned the same 34,743 units, 68,291 distinct posts, and 90,116,606 response bytes. Bootstrap request time totaled 614.507 seconds; the longest request took 17.268 seconds. Page 10 replayed identically after worker restart. Twenty no-op requests and another restart also passed. The full invocation took 666.467 seconds, including local metadata and checkpoint work. The report is `/tmp/xtap-consumer-initial-coverage/run-final-532668f/report.json`.
+
+This repeat measured no-op p95 at 2.630 seconds and restart at 2.640 seconds, compared with 1.609 seconds in the earlier warm run. Other local implementation work was active. The two runs do not establish consistent compliance with the two-second local target, and neither measures cloud transport or website freshness. All requests remained within the unchanged 30-second deadline. Do not report only the faster run.
+
+Follow-up review identified that the batched SQL limited results after its window function, which could rank an entire unit's history. The corrected query limits each indexed unit lookup to 51 candidates before ranking. A regression uses 500 additional valid results and counts actual result-body evaluations entering the window: only 51 may be evaluated. Existing tests still reject a unit when all first 50 candidates fail validation. A valid early result remains accepted even when older results exist, as in the previous single-unit implementation; the bound limits validation work, not the total retained history. The fresh full-run timing above predates this final candidate-bound correction. A separate read-only parity probe compared both queries for the 100 real units with the most retained results. All 100 returned rows matched exactly; the largest retained history in this source was 10. The report is `/tmp/xtap-consumer-initial-coverage/result-bound-report.json`. The 500-result regression covers the longer-history case that this real source does not contain.
+
+## Privacy reconciliation
+
+A withdrawal after an earlier accepted page stops that sequence with HTTP 409:
+
+```json
+{
+  "error": {
+    "code": "privacy_changed",
+    "message": "Apply the affected removals before resuming this page.",
+    "recovery": {
+      "action": "reconcile",
+      "path": "/api/reconcile",
+      "cursor": "<signed removal cursor; no acknowledgment>"
+    }
+  }
+}
+```
+
+Call `/api/reconcile?cursor=<recovery.cursor>&limit=200`. Every successful page has exactly:
+
+```json
+{
+  "schema_version": 1,
+  "source": "<full raw snapshot SHA-256 for this removal range>",
+  "removals": [
+    { "type": "post_remove", "post_id": "123", "reason": "not_available" },
+    { "type": "unit_remove", "unit_id": "123:author", "post_id": "123", "reason": "not_available" }
+  ],
+  "has_more": true,
+  "next_cursor": "<signed next removal cursor>",
+  "resume_cursor": null,
+  "resume_path": null
+}
+```
+
+The final page has `has_more=false`, `next_cursor=null`, a signed `resume_cursor`, and `resume_path` equal to `/api/units`, `/api/changes`, or `/api/observations`. Use `cursor=<resume_cursor>` for units/history, or `after=<resume_cursor>` for changes. The resume cursor retains the original source comparison, selection, history bounds, and page position. It also records the applied privacy checkpoint. It does not restart bootstrap. A history resume cursor remains a history cursor.
+
+Apply these rules in the Our Models client:
+
+1. Keep the unfinished content/history cursor and unrelated accepted and published content. Stop that sequence and follow the removal cursor. Do not treat the 409 cursor or the response `source` as applied progress.
+2. For `post_remove`, remove that post's saved body, observations, and dependent published stories. Use the maintained post-to-unit index to remove accepted units that contain the post. For `unit_remove`, remove the named saved unit **only if it still contains `post_id`**. That condition protects a later version which reuses an old unit ID without the withdrawn post. Remove orphaned bodies and observations when their last permitted unit reference is removed. Keep public content which still has a separate unaffected unit reference.
+3. Save each complete removal page and its next cursor in one transaction. Publish affected removals through the existing publication writer before marking that privacy checkpoint published. A failed transaction retries the same cursor. Both removal variants are idempotent. Never clear the entire selection.
+4. Follow `next_cursor` while `has_more=true`, including empty pages. Only after the final page's removals are applied may the client save and use `resume_cursor`. Do not feed a pending removal cursor to a content/history route or use it as a history `at` cursor.
+5. Keep the last completed `resume_cursor` as the selection's privacy receipt, in addition to the normal global source cursor. When starting a separate changes poll, send `/api/changes?after=<fully-consumed-global-cursor>&reconciled=<privacy-receipt>`. A separate initial history read can also send `reconciled=<privacy-receipt>`. This transfers only the applied privacy checkpoint; it never turns a history cursor into a global cursor. The receipt must be signed, completed, unexpired, and for the same selection. `reconciled` is rejected on continuations. A resumed sequence already carries its checkpoint.
+6. Continue the original sequence, then poll changes normally. The engine re-emits affected restored units even when their content hash equals the old public hash. It also recovers their permitted retained observations. The next fully consumed source which contains the privacy checkpoint clears that pending repair state. Once all active streams carry that checkpoint or a later source, the client can stop sending the separate receipt.
+
+Removal records sort by `(post_id, unit_id)`, with the direct post removal first. They come from restricted source events after the last applied privacy checkpoint (or the pinned target), and exact historical post/unit dependencies in the original base and target. A restriction event remains in its fixed range even if the post becomes public again during recovery. Historical dependencies are qualified by `post_id`; they are not unconditional deletion of every past unit ID. The query returns IDs only and does not reconstruct historical bodies. Repeated restriction events can produce idempotent removals in later ranges.
+
+If new source writes arrive during removal paging, the server finishes the fixed range and returns a next cursor for another pinned range. New withdrawals which sort before a previous page position are included in that new range. No cursor acknowledges a range before all its removal records have been returned. Replay and restart preserve the range and position. New withdrawals after the final recovery page cause another 409 on resume. Current privacy checks continue to suppress all forbidden old-source bodies and observations.
+
+The normal page limits and wall deadline apply. Signed cursors are limited to 8 KiB, including 100 long history IDs and the removal position. Recovery keeps the original 48-hour page lease and uses immutable metadata in the existing context store. Expiry is explicit; it does not silently clear content or start a full read.
+
+## Exact production-selection follow-up
+
+The earlier complete read used only `ai`. A later run used the actual Our Models selection: 123 authors, `ai,local-models`, `label_mode=any`, and `publication=public-original`. At frozen source `b5ab0341f57f931ea975ff72c1c2e25b5f0e135caa833619889cc10ae3862981`, its first coverage request hit the unchanged 30-second deadline.
+
+The query joined each conversation member to each matching label before checking post bodies. Thus a conversation with both labels repeated the same author and publication checks. `eligibleUnits` now materializes distinct candidate IDs before those checks. The explicit ID bound remains inside the candidate query; cutoff, author, publication, and all-label requirements remain enforced. No schema, cursor, source contract, or deadline changed.
+
+The regression uses a 20-post conversation with both production labels. It verifies identical rows and body-check counts for one label, either label, both required labels, and no labels. A missing required label still excludes the conversation. The existing 12 coverage tests also pass, including metric-only, no-op, privacy, and restart behavior. `npm run check` passed after the fix.
+
+On the same local frozen database, the fixed initial coverage took 14.597 seconds. The earlier unbounded diagnostic took 4.459 and 4.330 seconds for completion queries and 24.792 seconds for observation coverage. These are single local measurements, not cloud latency estimates. The decision boundary remains the existing 30-second request limit, with unchanged row/privacy checks. The complete corrected production-selection canary passed: 576 pages, 42,633 conversations, 84,570 distinct posts, and 111,763,946 response bytes. Request time totalled 702.085 seconds; median was 1.114 seconds, p95 was 1.736 seconds, and the maximum was 15.496 seconds. Twenty unchanged polls and a reader restart emitted zero changes; unchanged p95 was 2.598 seconds. Selected content was complete through `2026-09-07T11:02:05.771Z`. This is one local frozen-source run, not cloud or live-website performance. No source write, paid call, or Job was required.
+
+Review at `ac8f29102706926fc302fba1cca14aae2771f000` found no issues. [CI run 34148170156](https://github.com/osolmaz/xtap-pool/actions/runs/34148170156) passed. The follow-up is [pull request 48](https://github.com/osolmaz/xtap-pool/pull/48).
+
+## Remaining live gates
+
+Local checks passed: `npm run check` ran 761 Vitest tests in 78 files, 189 extension tests, and 180 native extension tests. This includes 12 coverage-maintenance tests, 13 paged privacy recovery tests, and 14 post-hash contract tests. Coverage was 88.04% lines/statements, 85.75% branches, and 92.08% functions. The duplicate-code check reported zero candidates. The parent must implement and test the Our Models transaction, removal, history, and cursor rules against these schemas. An operator must verify purpose-scoped grants, deployed projection readiness, raw-base retention, current source containment after restore, production unit sizes and deadlines, and the approved history window. The coordinated deployment, live restart/privacy canary, cold/cloud and real metric-delta measurements, and website publication proof remain pending. No production completion is claimed.
+
+## Parent integration
+
+This branch changes no classifier model, prompt, or semantic contract. It does not change `index-bootstrap.ts`, `index-command.ts`, command entry points, or the setup wizard. In `durable-index.ts`, it adds only the optional fetch adapter to `createDurableIndexBucketClient`; in `bucket-log.ts`, it adds that adapter to the existing raw Bucket clients. Keep those options and their read/write/list propagation when merging the parent's resumable CPU bootstrap and tail verification work. Preserve the parent's `compareSegmentKeys` export. No HTTP path publishes `index/current.json`. The bounded engine in `space/src/consumer-changes.ts` now accepts an optional applied privacy context to repair same-hash restoration. Preserve that argument, the cursor privacy fields in `consumer-cursor.ts`, and worker source-containment checks when combining later core changes. Keep the per-post hash field in `consumer-page.ts` and the exact source-hash check in `historical-unit-reader.ts` when combining later core changes. No source-effect table or classifier contract changed.
+
+The requested documentation command is `npx --no-install @simpledoc/simpledoc check`. In this local npm installation, that form returned exit 127 (`@simpledoc/simpledoc: not found`). The same scoped checker, version 0.1.6, passed with `npx --no-install --package=@simpledoc/simpledoc simpledoc check`. No untracked generated `index.html` was present. The unscoped npm package is unrelated.

docs/2026-09-07-incremental-consumer-and-engagement-history-plan.md

@@ -0,0 +1,558 @@
+---
+title: Incremental consumer reads and engagement history plan
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-09-07
+tags: [consumer-api, observations, engagement, incremental, extension, storage]
+---
+
+# Incremental consumer reads and engagement history plan
+
+Our Models needs two source capabilities from xTap: efficient access to changed content, and timestamped engagement observations for story ranking. Implement them together in xtap-pool before resuming the dependent story work.
+
+The earlier Our Models work stopped repeated AI processing and made checks cheap when xTap's revision stayed unchanged. It did not eliminate the complete source read after a revision change. The reported failure case was a 56-minute run over roughly 42,550 source entries that found no new, changed, or deleted entries and needed no AI calls. Another reported changed-data run took 86 minutes. These are incident inputs, not measurements repeated for this document. Verify the entry-versus-individual-post counts during the baseline measurement.
+
+The required result is specific: a harmless source revision change must not cause a complete post download, and a new likes/views observation must not rerun semantic processing. Saving future counts in Our Models alone would leave both historical access and the content-change problem unresolved.
+
+Status: **implementation in progress** on `feat/incremental-consumers`. The bounded raw-history audit, repeat delivery, indexed source effects, historical point reads, content clocks, bootstrap readiness, signed cursors, and bounded HTTP change/history reads pass local tests. The HTTP contract also includes source-authored post hashes and paged privacy removals that preserve unrelated consumer data. Production bootstrap cutover, Our Models integration, and live verification are not complete. No replacement extension, Space, Job, schedule, credential, resource, or spending limit has been deployed by this implementation. A read-only check on September 7 confirmed that the canonical six-hour enrichment and five-minute publication schedules remain active.
+
+## Ownership and related work
+
+This is the canonical cross-repository explanation of the xTap prerequisite. Keep source collection, provenance, observation history, and incremental source reads here. Our Models owns model eligibility, story grouping and writing, attention calculations, ranking, and its website publication.
+
+Related documents:
+
+- [Unit consumer API](2026-07-27-unit-consumer-api.md): the current full-read contract. Its normal full-download procedure will be replaced when the new reader and consumers pass the checks below.
+- [Immutable Bucket log](2026-08-12-bucket-object-log-plan.md): the authoritative storage and replay boundary. This plan extends that design; it does not restore Dataset/Git runtime storage.
+- [Small worker checkpoints](2026-08-19-small-worker-checkpoints-plan.md): logical runs, durable results, revision handoff, and scheduled Job recovery remain applicable.
+- [Browser scrape-job receipts](2026-08-06-scrape-job-receipts-plan.md): browser-local coverage receipts are distinct from engagement observations.
+- [Our Models story plan](https://huggingface.co/spaces/osolmaz/ourmodels/blob/main/docs/2026-09-06-daily-feed-and-organizations-plan.md): the consumer's model-only story requirements. This is an operational source link; the public website is [ourmodels.cc](https://ourmodels.cc).
+
+Do not copy this plan into Our Models. Link to it from the consumer implementation record. Keep the separate model-profile work independent.
+
+## Current implementation evidence
+
+The following facts were checked in the repository at `3945d92`. They describe the checked-in implementation, not proof of the installed browser version or the amount of historical production data.
+
+| Area                                                | Current behavior                                                                                                                                                              | Consequence                                                                                                                       |
+| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| `shared/src/tweet.ts`                               | Validates basic tweet fields with a loose schema. Extra fields, including supplied metrics, survive validation.                                                               | Historical counter values can exist in stored records, but the schema does not establish complete or well-typed counter coverage. |
+| `space/src/ingest.ts`                               | Appends accepted records to the verified raw Bucket log before updating the local index or returning success.                                                                 | Accepted repeat observations are durable.                                                                                         |
+| `space/src/store.ts`                                | Accepts only a newer `captured_at` for each post/contributor. Keeps only the latest record within a submitted batch and in its current SQLite row.                            | Older arrivals and intermediate same-batch observations can be discarded before reaching the raw log.                             |
+| `space/src/bucket-log.ts`                           | Writes complete accepted records into immutable, verified segments.                                                                                                           | The raw log is the existing history source. No additional raw archive is needed.                                                  |
+| `extension/background.js`, `extension/lib/dedup.js` | Uses persisted seen post IDs to skip ordinary repeats before `poolEnqueue`. Article and image-backfill exceptions are not a general engagement-history path.                  | Repeated browsing does not prove repeated metrics reached the pool.                                                               |
+| `extension/lib/pool-sync.js`                        | Uses a persistent, at-least-once queue. Its current overflow behavior drops the oldest entries.                                                                               | Increasing observation volume requires explicit queue bounds, recovery, and loss reporting.                                       |
+| `extension/lib/scrape-receipts.js`                  | Records list/run observations before ordinary post deduplication. The records contain post IDs, post times, observation times, and run positions, but no engagement counters. | A scroller coverage receipt cannot reconstruct past likes or views.                                                               |
+| `shared/src/hash.ts`                                | Excludes counters and capture timestamps from semantic input hashes.                                                                                                          | Preserve this separation. A repeat observation must not reclassify unchanged text.                                                |
+| `space/src/unit-store.ts`                           | Uses a process boot UUID plus SQL revision triggers. Many table writes invalidate cursors, including tweet and queue writes.                                                  | Revision changes are not a list of actual content changes. A restart can invalidate a consumer read without changing its content. |
+| `space/src/unit-store.ts`                           | The cutoff filters the latest stored capture for a unit. It is not a historical version lookup.                                                                               | A later count observation can affect cutoff membership unless content and observation times are separated.                        |
+| `space/src/durable-index.ts`, `space/src/server.ts` | Restore a verified index, discover raw segments, and apply a verified tail. The Space advances locally after ingest and checks external results every 5–60 seconds.           | Reuse this bounded source-advance path to build change indexes.                                                                   |
+| Existing index publication                          | The non-concurrent Job, or an authorized maintenance command, owns the shared `index/current.json` publication. The Space does not publish that shared generation.            | Do not introduce a second mutable publication authority for this feature.                                                         |
+
+The statement “xTap already saves history” therefore needs a qualification: its raw storage preserves accepted repeated records, but the current extension and ingest deduplication can prevent those records from arriving or being saved. Production history coverage remains unmeasured.
+
+## Requirements
+
+- Preserve useful observations of existing posts, including unchanged counter values at genuinely different observation times.
+- Keep exact content changes, observation changes, and internal bookkeeping changes separate.
+- Let consumers read only changes since a durable cursor, without scanning all post bodies to discover them.
+- Include removals, unit membership changes, label changes, registry visibility changes, and public-source eligibility changes. A new-observation endpoint alone does not satisfy this requirement.
+- Give a consumer one consistent source boundary for content, labels, related metadata, and observations. Identify observation freshness separately from content completeness.
+- Preserve the current author-ID allowlist and `public-original` restrictions. No subscriber-only content, retweet-only units, contributor credentials, or private administration data may enter a public consumer feed.
+- Make paging and replay safe across concurrent source writes, retries, Space restarts, and consumer crashes.
+- Reuse the private raw Bucket, private index Bucket, existing Space, existing extension storage, and existing Job/checkpoint machinery.
+- Keep observations useful to ordinary authorized consumers. Do not require a Story schema, an Our Models dependency, or an agent-specific progress protocol in xTap.
+- Validate both source efficiency and the final website update time. Fewer AI calls alone do not complete the work.
+
+## Scope and non-goals
+
+The implementation includes a production-history audit, the repeated-observation path, typed normalized history, indexed content differences, stable snapshot/cursor reads, bounded recovery, setup checks, and the coordinated Our Models reader change.
+
+It does not add a new Bucket, Dataset, database service, scheduler, queue service, vector database, or independent raw store. It does not change the enrichment model, taxonomy meaning, hardware, schedule, source account permissions, or spending limits as an incidental fix.
+
+There is no automatic creation of new X requests or a guarantee that a six-hour list visit observes every post again. A changed browsing/revisit policy is a separate, explicitly reviewed step after coverage is measured. Do not bypass platform access restrictions.
+
+The capture implementation must follow `AGENTS.md`: passive browser-level observation from extension-owned contexts. The existing main-world fetch/XHR interception must be replaced rather than extended or copied. First inventory any existing compliant capture adapter. If a replacement needs additional browser permissions or a debugger attachment, report those requirements and obtain approval before enabling it. Do not use page-owned API wrappers as a temporary fallback.
+
+## History and capture audit
+
+Perform this first, before promising growth measurements or freezing the history schema.
+
+1. Identify the actual installed extension build and the capture path used by the scheduled browser visits. Compare it with the checked-in deduplication and pool enqueue paths. Check whether another approved exporter sends repeats.
+2. Pin a verified raw Bucket snapshot and record its identity. Inspect a bounded sample across the last seven days and more than one completed visit. Start with at most 50 segments or 64 MiB compressed input, whichever is reached first. Report the actual files, bytes, and time. Expand only through an explicit bounded audit decision.
+3. Count physical records, distinct post IDs, distinct normalized observation identities, posts with two observations, and posts with three observations. Report counter availability separately for likes, replies, reposts, and views.
+4. Compare observation times with browser run receipts. A run receipt proves that a post was encountered, not that its metrics were retained or sent.
+5. Inspect time gaps, exact duplicates, same-time conflicting counts, counter decreases, invalid timestamps, and values that were missing, rounded, or represented as strings.
+6. Run a small browser canary using the approved capture path: observe a selected public post, observe it again, and confirm distinct observation times survive browser restart, queue delivery, raw persistence, and index replay. Use recorded fixtures for most tests; do not generate unnecessary X traffic.
+7. Record whether the first implementation can recover useful history, has only sparse history, or must begin measuring growth from deployment onward.
+
+Store raw audit inputs only in approved private working storage. Track the bounded audit procedure, aggregate counts, snapshot/checksum references, and conclusions in this repository. Do not commit captured text, account-specific browsing history, credentials, or a raw export.
+
+No repeat records means unknown historical growth. Do not fabricate an earlier count of zero. Do not infer metrics from scroller receipts, current totals, or a post's publication date.
+
+### September 7 audit and first implementation slice
+
+The audit used existing local Hugging Face authentication in place. It read only the canonical private raw and index Buckets. The source snapshot was `36f27e68bf5ba7cfa53a6fedd7ba6fa66906113d058a048dd1384e7aa37034aa`; the active manifest named database checksum `2083eaf860cc69d038371eb2e3adf83366411baf21fbcb83e0062d44d3f623e2` and projection contract `2a2814ad4162457c0b8fe6de9cc07e0813f9fffadd3bde21ca46c731cc58de69`.
+
+The selection took seven evenly spaced tweet segments per available UTC day from September 1 through September 6. The pinned snapshot had no September 7 tweet segment. Each selected object passed compressed-size and uncompressed-checksum verification. This was a bounded sample, not a complete history audit.
+
+| Measure                                              |             Observed value |
+| ---------------------------------------------------- | -------------------------: |
+| Selected segments                                    |                         42 |
+| Compressed bytes                                     |                    263,173 |
+| Physical records and distinct raw observation tuples |                      1,196 |
+| Distinct post IDs                                    |                      1,136 |
+| Posts with two distinct observation times            |                         60 |
+| Posts with three observation times                   |                          0 |
+| Repeat pairs from the same contributor               |                         60 |
+| Repeat pairs 1–18 hours apart                        |                         18 |
+| Minimum / median / maximum interval, hours           | 5.8992 / 25.6423 / 25.6427 |
+| Download and first-pass audit time, seconds          |                     33.682 |
+
+All 60 pairs had changed view counts. Likes changed in 40 pairs, replies in 29, and reposts in 16. Two pairs had decreasing reply counts. Among the 18 pairs 1–18 hours apart, nonzero, nondecreasing endpoints were available for likes in 16 pairs, replies in 10, reposts in 8, and views in all 18. This sample supports some historical growth measurements, but does not establish acceleration coverage.
+
+All 1,196 records had numeric values for the four counters. That does not prove that each counter was observed: the old parser replaced absent likes, replies, and reposts with zero, and partially parsed view strings. The sample contains 289 zero likes, 574 zero replies, and 918 zero reposts without a format marker. Their original meaning cannot be recovered from the saved parser output. Historical normalized zero values therefore remain null. Positive historical values retain their raw source reference; the original GraphQL counter was not independently verified.
+
+New parser output marks the metrics object with `format: "exact-v1"`, admits only exact nonnegative safe integers or exact decimal strings, and preserves actual zero separately from null. This marker is excluded from content identity with the other metric fields. It does not change historical raw records.
+
+The checked-in extension already uses passive Chrome Debugger Network capture in `lib/graphql-capture.js`. It does not need a new capture permission or a replacement main-world script for this task. A read-only process check found no local Chrome or Chromium browser, so the installed browser build and live browser canary remain unverified.
+
+The first source slice adds normalized observations, immutable content versions, and physical provenance keyed by segment, operation, logical path, and line position. Server admission keeps delayed and same-batch observations. Exact logical retries remain idempotent. Browser delivery occurs before unique-post export deduplication, and queue admission is saved with sampling state. A full queue leaves staged work pending; it does not discard the oldest observation. Staged responses now survive browser restart and retain their original observation time.
+
+A complete ingest acknowledgment accounts for every submitted observation as accepted, duplicate, or explicitly rejected. The browser saves explicit rejections in a local archive capped at 500 records; it continues valid work without silently dropping rejected data. Queue removal and rejection records are saved together. Archive overflow blocks removal and leaves the pending batch intact.
+
+Local validation of this slice passed `npm run check`: formatting, lint, TypeScript, 597 Vitest tests, 188 extension tests, 180 native Python tests, coverage, and the duplicate-code check. Raw replay tests compare the same physical source references against immediate ingest, including operation and line positions. These are local results, not evidence of deployment or end-to-end publication speed.
+
+The next committed slices add historical content/result dependencies and point reads for old and target source sets. Metric-only updates and exact result retries produce no content candidates. Membership changes, label approvals, delayed source keys, and content conflicts remain candidates for exact comparison. Historical reads reconstruct only the requested entries in a bounded temporary database, not a complete old database.
+
+Content coverage now uses the start of the current content version, not the latest metric observation. Tests cover all six delivery orders of a three-observation edit/revert sequence, late discovery of an intervening edit, UTC normalization, and same-time private changes. Current and historical result reads use the same deterministic tie rule.
+
+A separate consumer projection hash binds the derived index rules without changing the LLM contract. The saved consumer boundary contains the actual approved registry state and commits with the source inventory. Every new segment checks its observation, membership, and result-reference counts. Unsupported historical result rows retain a physical reference but cannot supply current labels. An old index with empty new tables remains unavailable to consumer reads and cannot publish a new index until explicit bootstrap. Output-segment application now stores the verified source snapshot before committing its rows and boundary; a storage failure does not acknowledge partially applied work.
+
+These slices passed `npm run check`: 635 Vitest tests, 189 extension tests, 180 native Python tests, formatting, lint, TypeScript, coverage, and the duplicate-code check. The overall line coverage was 86.11%. These are local checks, not live throughput measurements.
+
+The combined HTTP and bootstrap implementation subsequently passed `npm run check`: 744 Vitest tests in 77 files, 189 extension tests, and 180 native Python tests. Line coverage was 87.92%; the duplicate-code check found zero candidates. `npx --no-install @simpledoc/simpledoc check` and `git diff --check` passed. The exact consumer contract and required transaction rules are in the [HTTP handoff](2026-09-07-consumer-http-handoff.md). These checks include bounded native-query cancellation, pinned paging, targeted privacy recovery, and post-body/hash agreement. They are not deployment or live performance evidence.
+
+Pi Reviewer found one P1 in the first pass: the extension required an `ok` flag that the Space's successful ingest response does not return. The extension now validates the existing `added`, `duplicates`, and `rejected` fields, and its normal queue test uses that exact response shape. The second review found no issues. CI then exposed a separate clean-build error: forked consumer processes were not compiled before tests. Root and Space test/coverage commands now build them first. The combined checks and all 518 Space tests passed after that correction. The third review found no issues, and CI passed on `d573fd4`. The configured mutation targets also passed at 82.51%; these targets do not establish mutation coverage for the new consumer modules. Extension version 0.26.0 identifies the new repeat-observation behavior; it has not been installed in the live browser.
+
+A subsequent code audit found an acceptance gap: every idle change poll still requested a coverage calculation, and `selectedObservationThrough` read historical body JSON across the selected cohort. An empty HTTP response therefore did not prove zero unrelated body reads. The next correction must reuse proven unchanged coverage and calculate metric coverage from affected posts while preserving exact source, selection, privacy, and completion rules. This work is in progress; do not treat the passing review and CI as proof that the performance acceptance matrix is complete.
+
+The real local source preparation saved 10,368 of 38,219 segments before a network read failed. A bounded restart read four additional segments, not the completed prefix, and produced a valid paused checkpoint at 10,372 segments. Its database held 400,822 logical observations and occupied 3,222,917,120 bytes. Preparation resumed from that point. This is recovery evidence, not a completed bootstrap or a live deployment. Computing the classifier contract with the deployed GLM-5.2 model and built-in taxonomy gives the same `2a2814ad4162457c0b8fe6de9cc07e0813f9fffadd3bde21ca46c731cc58de69` hash as the incumbent index.
+
+The semantic schema checks are incomplete. Schemator's real Codex strategy timed out first at 120 seconds and again at 600 seconds per field. The longer attempts returned three of 39 SQL-field reviews and seven of ten observation-field reviews. Aggregation correctly reported incomplete coverage; no proposal was applied and no converged schema approval is claimed. In particular, the extracted `unknown` counter type is an extraction limitation: the production validators already require nullable, nonnegative safe integers. HTTP and Our Models schemas remain outside that partial result.
+
+These changes alone do not remove the Our Models full read. Explicit production bootstrap and incremental website publication remain required before this task is complete.
+
+## Repeated observations
+
+### Separate delivery from post-ID deduplication
+
+Keep the post-ID deduplication needed by local unique-post exports independent from pool observation delivery. A known post must still reach the observation path when it is observed again. Count unique posts and observations separately in local status.
+
+Create a stable observation identity before durable enqueue. Persist the observation and delivery state together before marking it handled. A worker restart or retry must resend the same observation identity and timestamp, not invent a fresh observation at recovery time. Staged browser responses must retain their original observation time as well; reparsing a staged response must not stamp it with the recovery time.
+
+Use the existing pool outbox where it can provide these guarantees. If its current storage cannot update queue and dedup state safely together, use a transaction in the extension's existing IndexedDB storage. Do not maintain two competing outboxes. Treat a full queue as an explicit degraded/blocked state; do not silently discard acknowledged work or persist a seen marker for an observation that was never saved.
+
+For burst control, start with at most one ordinary metric sample per post per five-minute sampling interval. Record its actual observation time, never the interval boundary. Six-hour visits must not be suppressed by lifetime post-ID deduplication. Content edits and visibility changes must be delivered immediately and must not wait for this metric sampling interval. Test and report any intentional sampling loss; do not describe sampled history as every browser response.
+
+A distinct later sample with unchanged counters is useful: it supports a measured zero increase. Within an interval, exact delivery replays are duplicates. Do not replace several independently queued observations with only the latest record before persistence.
+
+### Server acceptance
+
+Replace the current “latest timestamp per post/contributor” ingest admission rule for observations with observation-identity idempotence. Keep a separate latest-post projection for browsing.
+
+- Accept a valid delayed observation even if a newer one is already stored. It must not overwrite newer content or newer latest counts.
+- Preserve multiple valid observations for one post in the same batch.
+- Treat an exact retry as one logical observation. A successful retry returns the original result without writing duplicate logical work.
+- Preserve conflicting observations rather than silently selecting the larger count. Identical IDs with different bytes are an integrity error.
+- Normalize valid timestamps to UTC before comparison. Bound future clock skew and report invalid or unsupported timestamps. Receipt time and observation time remain distinct.
+- Keep contributor attribution server-controlled. Client observation IDs do not authorize another contributor's identity.
+- Persist accepted raw records before success. A projection failure after raw persistence remains recoverable by replay and does not require recapture.
+
+Historical raw objects remain immutable. The new reader normalizes old accepted records during the explicit backfill; it does not rewrite them to look like newly generated observations.
+
+## Observation data
+
+Use a small normalized observation record. The following is a proposed consumer example, not a currently available endpoint response:
+
+```json
+{
+  "id": "obs-example-001",
+  "post_id": "1234567890",
+  "observed_at": "2026-09-07T06:00:00.000Z",
+  "received_at": "2026-09-07T06:00:12.000Z",
+  "content_hash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+  "metrics": {
+    "likes": 400,
+    "replies": 18,
+    "reposts": 25,
+    "views": null
+  },
+  "source_ref": "opaque-source-record-reference"
+}
+```
+
+The example IDs are illustrative. Freeze the actual identity encoding and canonicalization vectors in shared schemas and tests during implementation.
+
+| Field          | Type           | Meaning and validation                                                                                                                                            |
+| -------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `id`           | string         | Stable logical observation identity. Deduplicate delivery by this value, independently of post identity.                                                          |
+| `post_id`      | string         | Immutable X post ID. Retweet wrappers must not add another copy of the original post's counters.                                                                  |
+| `observed_at`  | UTC timestamp  | When the source actually observed these counts. Preserve the original time through retries.                                                                       |
+| `received_at`  | UTC timestamp  | First durable pool receipt time for this logical observation. Useful for delay measurements, never a substitute for observation time.                             |
+| `content_hash` | SHA-256 string | Reference to the observed content version. A later content edit must not attach an old observation to an invented text version.                                   |
+| `metrics`      | object         | Required named counters, each an exact nonnegative safe integer or null. Unknown is null.                                                                         |
+| `source_ref`   | opaque string  | Resolves internally to immutable segment identity, operation, and record position. It does not expose a Bucket URL or contributor identity to the public website. |
+
+Use `reposts` in the new consumer contract and normalize the source's existing `retweets` field at the source-reader boundary. Do not maintain two synonymous consumer fields. Preserve supplied bookmarks, quotes, and other raw fields in the original record; they do not need to become ranking inputs in this task.
+
+Do not turn an abbreviated display such as `1.2K` into an exact count of 1,200. For the initial exact-count contract, report that metric as null and retain the original value and normalization outcome in private provenance. Invalid or negative historical fields are also unknown with a diagnostic, not silently clamped to zero.
+
+Define logical identity from stable normalized source identity, observation time, content identity, and counters. Preserve enough authenticated source context internally to distinguish independent observations and diagnose conflicts. Exclude receive/retry time from logical identity. Keep exact raw references for duplicate deliveries; the consumer must not count those deliveries separately. Historical normalization must produce the same IDs across shuffled replay, index rebuild, and process restart.
+
+## Indexed storage and change detection
+
+### Canonical storage
+
+Inventory and reuse the deployed `RAW_BUCKET` and `INDEX_BUCKET`; the existing design uses `osolmaz/xtap-pool-data` for raw segments and `osolmaz/xtap-pool-bucket` for replaceable indexes. Verify actual deployment variables during implementation. An access failure does not authorize replacement resources or credential copying.
+
+Retain the raw Bucket as the only complete source history. Extend the existing SQLite projection with normalized observations, distinct content versions, and reverse indexes for affected posts and units. Keep immutable source references and replay checkpoints in the existing storage scheme.
+
+Do not add a separately mutable consumer head or an independent feed publisher. The Space can derive and serve bounded changes from its verified local source advance, as it already does for current reads. The existing authorized index publisher remains the only writer of the shared index manifest.
+
+### Stable boundaries
+
+Use a content-addressed raw snapshot plus the semantic contract and normalized selection as the durable consumer boundary. The current boot UUID and SQL revision counter are unsuitable for this purpose.
+
+A consumer cursor must describe the exact raw object set already consumed, not a timestamp or a lexicographic segment-key high-water mark. Raw writers can run concurrently, and a newly discovered object can have an earlier timestamp. Compare verified snapshot membership to find new segments. Store the exact snapshot manifest before issuing a durable cursor that refers to it; an unpersisted local hash is insufficient.
+
+This reuses the existing immutable snapshot mechanism. Snapshot manifests are metadata, not duplicate post exports. Normal advance reads and applies only new source segments. Query handlers must not rescan historical segment bodies.
+
+The audited snapshot has 38,219 segment descriptors and takes 15,752,981 bytes as canonical JSON. A metadata-size diagnostic used that snapshot with synthetic tails of 1, 16, 128, and 1,024 files. A retained base reference plus those exact additions took 502, 6,682, 52,837, and 422,019 bytes, respectively. At 1,024 additions, it saves 15,330,962 bytes per boundary, or 97.32%, before amortizing a new full base. This exceeds the 90% minimum worthwhile reduction chosen for this extra metadata logic. These are deterministic size calculations, not historical source deltas, network throughput, USD cost estimates, or launch approval.
+
+The bounded page engine now compares exact old and target content, emits removals, and returns retained observations when content first becomes eligible. It filters by historical account membership before reading content and checks current privacy on every upsert and history read. Raw retries do not duplicate logical observations. Recorded classifier results can be reused after an exact content revert; an invalid later quote does not erase a valid prior result. Local validation passed after the resumable bootstrap changes: `npm run check` with 673 Vitest tests, 189 extension tests, 180 native extension tests, 87.60% line coverage, and `npx --no-install @simpledoc/simpledoc check`. HTTP authorization, worker deadlines, and live recovery remain separate unfinished gates.
+
+The implementation therefore stores an immutable full base through the existing raw snapshot store and places its hash plus explicit additions in the read context. It rolls the base after 1,024 additions or a 1 MiB descriptor. Reconstruction verifies the checksum of the complete source set. It rejects removed, mutated, or duplicated files and includes late keys; it does not use a key watermark. At most two loaded bases are cached in memory. Read contexts belong in the existing index Bucket and do not replace its shared manifest. HTTP deadlines, retained-context cleanup, and real publication timing still need their rollout checks.
+
+Separate the time of a substantive content version from the time of its latest metric observation. A count-only revisit must neither remove an already completed unit from cutoff selection nor advance content completeness by itself. Status and content reads must use the same completed-content boundary. An edit that invalidates the current enrichment must produce the documented pending/removal state until a matching result is ready; stale enrichment must not be attached to the edited text. Include this distinction in status, replay, and cutoff regression tests.
+
+### Derived indexes
+
+Settle exact SQL names in the shared schema review, but implement these distinct responsibilities:
+
+| Derived data            | Required key/index                                                            | Purpose                                                                            |
+| ----------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
+| Normalized observations | Unique observation ID; `(post_id, observed_at, id)`; source segment reference | Bounded post history and idempotent replay.                                        |
+| Content versions        | `(post_id, content_hash)` with source references and visibility metadata      | Separate text/context edits from metrics; evaluate a pinned source version.        |
+| Source effects          | `(segment_key, affected_entity_kind, affected_id)`                            | Find changed candidates from a raw snapshot difference without scanning all posts. |
+| Unit dependencies       | Post-to-unit and prior-to-current membership references                       | Handle added replies, splits, merges, author corrections, and removed membership.  |
+| Label dependencies      | Unit-to-label and label-to-unit references, including registry decisions      | Find the units whose consumer-visible labels or selection changed.                 |
+| Pinned read metadata    | Snapshot identity, contract identity, selection, bounded page position        | Reproduce a multi-page read without relying on process memory.                     |
+
+Build source-effect entries in the same SQLite transaction that applies a verified raw segment and advances the indexed source inventory. A crash must not mark a segment applied without its observations and change candidates. A schema/bootstrap change must not discard these indexes merely because the latest-post table can still be restored.
+
+For each candidate unit, compare the exact consumer-visible content at the old and target boundaries. Emit an upsert only if its content or permitted relationships changed. Exclude counts, observation time, pooled time, queue bookkeeping, and attempt receipts from that content identity. Include text, expanded links, media metadata used by consumers, authorship, membership, visibility, and published labels. Keep the narrower existing enrichment input hash where its semantics remain unchanged.
+
+An account rename can require display metadata to change without requiring an LLM call. A registry approval/rejection can change label selection without changing tweet text. Distinguish these dependencies explicitly rather than using one hash to invalidate every processing stage.
+
+An internal attempt, receipt, or queue write can advance the raw snapshot while producing zero consumer changes. It should advance the consumer cursor through a small empty result. A taxonomy/policy change that truly affects all selected units is broad work; report it explicitly and require the bounded re-evaluation or reset procedure. Do not disguise it as an ordinary small update.
+
+### Historical reads and bounded page construction
+
+The selected approach is indexed differences between pinned source snapshots, using stored content/observation versions and source effects. Do not create a globally increasing consumer sequence independently in each process. Such sequences can disagree after a Job restore or a Space restart.
+
+Before finalizing this implementation, prove that the old and target selected views can be read for affected IDs without reconstructing a full old database. Test delayed segments, unit regrouping, and registry changes. If the existing SQLite model cannot provide this, extend its version/dependency indexes before adding API handlers.
+
+A page must remain fixed while new source data arrives. Read the target version through indexed immutable versions, or freeze the necessary bounded response batch in checksum-addressed derived storage in the existing index Bucket. Use one mechanism after the prototype; do not ship parallel fallback readers. Any derived page cache is temporary, authorization-checked, and reconstructible. It must not become a second permanent source archive or a mutable publication authority.
+
+This proof is a design gate. “Fetch changed IDs and then read each ID's latest row” is insufficient: a concurrent edit could mix versions, and a removal may no longer have a current row.
+
+## Consumer API
+
+Change the owned version-1 contract in place and coordinate its consumers. Do not add a parallel `v2`, boot-epoch fallback, or automatic full-scan compatibility path. Keep current query routes that still serve independent browsing or administration uses; do not retain them as a hidden fallback for the incremental publisher.
+
+### Bootstrap
+
+A new consumer explicitly requests one pinned snapshot for its normalized selection. Continue using `/api/units` for this explicit bootstrap, with the new durable snapshot/cursor semantics. Use exact author IDs, label mode, and `publication=public-original` throughout.
+
+The bootstrap returns the matching complete content state, approved label/graph state or exact references to them, a declared observation-history range, and the resume cursor at the same boundary. Bootstrap pages must survive a Space restart and later source writes. The consumer does not advance its durable cursor until the snapshot is fully validated and saved.
+
+A pre-existing verified Our Models publication may be adopted as the bootstrap baseline only if its source snapshot, contract, selection, content digest, and cursor boundary are proven to match. Otherwise perform one explicit bootstrap. “It looks similar” and matching row counts are insufficient.
+
+### Changes
+
+Proposed route:
+
+```http
+GET /api/changes?after=<opaque-cursor>&limit=200
+Authorization: Bearer <configured-service-credential>
+```
+
+The cursor binds the normalized author/label/publication selection from bootstrap. Reject conflicting query filters. Do not expose raw segment URLs or ask the consumer to list the Bucket.
+
+Proposed response envelope:
+
+```json
+{
+  "schema_version": 1,
+  "source": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+  "complete_through": "2026-09-07T01:22:00.000Z",
+  "observations_through": "2026-09-07T06:00:00.000Z",
+  "cursor": "opaque-next-position",
+  "has_more": false,
+  "changes": []
+}
+```
+
+`source` names the pinned raw snapshot. `complete_through` describes completed content/enrichment coverage for the selected accounts. `observations_through` is the latest included observation time, or null if none is available; it is not a promise of complete observation coverage up to that time.
+
+The returned `cursor` is the position to save only after applying that page. While `has_more` is true, it remains inside the same fixed old-to-target snapshot comparison. After the final page it represents the fully consumed target boundary. New writes wait for the next comparison. A response with zero changes can still advance the source boundary.
+
+Provide typed changes for:
+
+- **Unit upsert:** complete selected content and published associations/labels for a new or changed unit, with a stable content hash. Separate metrics from this content identity.
+- **Unit removal:** the previously visible unit ID and an appropriate public removal reason, without the withdrawn text. Include old IDs after regrouping, author/filter changes, subscriber-only transitions, or label eligibility changes.
+- **Observation:** a normalized observation for a selected, permitted post. It must not imply that content changed.
+- **Shared metadata change:** approved taxonomy/registry/graph changes, or exact bounded references to matching state at this source boundary. They must not require a full post read to update metadata.
+
+Emit observations for currently completed, publishable content without waiting for another LLM pass on unchanged text. An observation associated with content that is not yet eligible must not leak through the public-original selection. When that content first becomes eligible, provide its permitted retained observations even if they were received before the consumer's previous cursor. Test this explicitly.
+
+Compute removals against both old and target membership. Filtering only the newest rows misses posts that left the selection. An absent post in a browsing session is not evidence of deletion; use an explicit source removal, an observed content/visibility change, or a documented membership/eligibility change.
+
+### Post history
+
+Proposed route:
+
+```http
+GET /api/observations?post_ids=1234567890,2345678901&since=2026-09-01T00:00:00Z&until=2026-09-08T00:00:00Z&at=<source-cursor>&limit=200
+Authorization: Bearer <configured-service-credential>
+```
+
+Use inclusive `since` and exclusive `until` on actual observation time. Require a bounded post-ID set and time range. Return stable pages ordered by `(post_id, observed_at, id)` at the pinned source boundary. Include observation counts/coverage and the earliest available history bound. Missing history must be explicit.
+
+This endpoint supports the initial bounded history recovery, audits, and later story activation. Normal operation uses `/api/changes`; it must not re-read the complete history of every story on every poll.
+
+### Bounds, validation, and errors
+
+Initial limits to verify against real unit sizes and throughput:
+
+- 200 changes or observations per page by default; 500 maximum.
+- 100 post IDs and a 30-day range per history request.
+- 2 MiB target response size and an 8 MiB hard page limit.
+- 30 seconds maximum server-side work per request; client deadlines must leave time to receive the response.
+- A 48-hour lease for an unfinished pinned page sequence; 30 days of supported incremental cursor recovery and hot observation history initially.
+
+These are proposed operational limits, not measured capacity or spending approval. Test unusually large threads and articles. Never silently truncate a complete unit or skip it and acknowledge its cursor. If a valid unit exceeds the hard limit, return a specific oversized-source error. If the audit finds such units in the required source set, bounded unit-member paging is an implementation prerequisite rather than a production surprise.
+
+Return structured errors:
+
+| Status        | Meaning                                                                         | Consumer action                                                                      |
+| ------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
+| `400`         | Malformed cursor, dates, IDs, limits, or conflicting selection                  | Correct the request. Do not retry unchanged.                                         |
+| `401` / `403` | Missing, expired, revoked, or insufficient credentials                          | Stop; preserve the last good publication.                                            |
+| `409`         | Contract/selection mismatch or a privacy change that invalidates an active page | Reconcile the explicit reason; do not silently restart a complete download.          |
+| `410`         | Cursor or pinned page lease is outside retained support                         | Use explicit bounded recovery or an approved bootstrap.                              |
+| `413`         | One valid content item exceeds the supported complete-item bound                | Stop that content advance with an actionable error; do not fabricate a partial item. |
+| `429`         | Capacity limit                                                                  | Retry the same cursor with bounded backoff and `Retry-After`.                        |
+| `502` / `503` | Temporary source/index unavailability                                           | Retry the same page with a finite elapsed-time and attempt limit.                    |
+
+Cursor decoding validates schema, size, source references, selection, contract, and position. A cursor is not an authorization token. Never trust a client-supplied Bucket path or fetch an arbitrary URL from it. Ordinary source writes and restarts do not invalidate a valid cursor.
+
+### Authorization and privacy
+
+Reuse service accounts and configured credentials in place. Define an explicit observation-history read permission alongside the current unit/taxonomy permissions. Confirm whether the existing Our Models account has the required grants; a broader grant or credential copy requires the named operator action before rollout.
+
+Apply current authorization on every page, including cached pages. Historical selection never grants access to subscriber-only, revoked, or otherwise withdrawn source bodies. If a privacy withdrawal invalidates an unfinished page, stop that page with a specific privacy-change response and provide a safe removal/reconciliation path. A consumer must not need to fetch the forbidden body to process its removal.
+
+Do not expose contributor IDs, raw private history, API credentials, or private Bucket URLs in Our Models. Its browser reads only the published public result.
+
+## Growth measurements and rechecks
+
+xTap supplies source observations and coverage facts. Our Models calculates attention using ordinary code, not an LLM.
+
+For a valid pair, retain both counts and both timestamps. Calculate the absolute difference and divide by actual elapsed hours. A change from 100 to 400 likes over six hours is an increase of 300, averaging 50 per hour. It does not identify when those likes arrived within the interval.
+
+Use at least three suitable observations for acceleration. Compare adjacent elapsed-time-adjusted rates, retaining both interval lengths. Do not mix a twelve-hour gap with a six-hour interval without accounting for the difference.
+
+Missing counters, a first observation, identical timestamps, excessive gaps, stale endpoints, and conflicting observations produce an unavailable signal for that metric. Counter decreases remain recorded as corrections/anomalies; do not convert them into positive growth or silently clamp them to zero. Missing intervals must not prevent other valid intervals from being inspected.
+
+Keep likes, replies, reposts, and views separate. Views are not unique people or endorsements. Compare similar post ages; use author baselines only when enough matched observations exist. Keep raw counts, intervals, coverage, and sample sizes alongside normalized scores. Cap individual-post and author contributions in Our Models, and treat copied/reposted sources as shared evidence rather than independent confirmation.
+
+Start with existing visits. If coverage is inadequate and a revisit policy is later approved, evaluate:
+
+- Broad coverage of new relevant posts during their first 48 hours, at the existing six-hour cadence.
+- Continued visits for posts in active discussions for a few more days.
+- Older posts only when new evidence or an explicit request justifies them.
+
+Do not select only already popular posts. Do not couple xTap to Our Models Story IDs; any requested post set must remain a normal, bounded source operation with its own authorization. Changes to Infinite Feed Scroller belong in that repository and need their own implementation/merge authority. This plan does not silently expand that work.
+
+## Our Models integration
+
+Implement this consumer phase only after xTap's source contract and recovery tests pass.
+
+1. Replace revision-triggered full reads with explicit bootstrap plus persisted change cursors. Reuse the existing frozen source work and publication coordination machinery.
+2. Apply each page's content, removals, observations, and metadata in a local transaction with its resume cursor. A failed transaction leaves the earlier cursor intact.
+3. Save the applied source state and cursor through the existing durable checkpoint mechanism. Distinguish the cursor already applied to a staged result from the cursor actually published to the website.
+4. Reuse cached results by their real dependencies. Text/model-relevant evidence changes can require association, claims, grouping, or writing. Display-only changes need only display updates. Metrics-only changes require **zero** association, claim, grouping, writing, or checking calls.
+5. Map changed observations to existing supporting post IDs, then recompute only affected stories' attention. New popularity cannot bypass model eligibility, change impact into a factual claim, duplicate a story, or rewrite its original date.
+6. Remove or recheck stories when their supporting source or eligibility changes. Keep the last verified story only when its evidence still remains valid.
+7. Retry a failed fetch or publication from the durable page/batch boundary. Never fall back to a complete read on ordinary `409`, `502`, or `503` responses.
+8. Keep exactly one publication writer. Publish a coherent verified result, then advance the live manifest and its published cursor together.
+
+Do not stop at an incremental API. The consumer must also stop decoding the complete result cache and rebuilding/verifying/uploading the full JSON and SQLite artifacts for every count update. Measure those stages separately. Reuse the existing accepted source state and update affected data with indexed writes; make the small public story/feed result independently publishable under the same authoritative manifest if measurements show the large commentary artifact still blocks freshness. This is a consumer publication change, not a new xTap store or competing writer. Select the exact artifact layout in the Our Models implementation plan and verify atomic consistency before shipping.
+
+Preserve one-time full build/bootstrap and explicit repair paths. Remove normal-path full scans after the coordinated replacement; do not retain them as indefinite compatibility code.
+
+## Retention and recovery
+
+Keep full source history in the immutable raw Bucket. Do not delete or rewrite raw segments for this task.
+
+Start with a rolling 30-day hot observation window and sufficient version/source-effect metadata to serve every supported cursor. Retain the predecessor observation needed at the beginning of a comparison window. The first backfill should cover the requested recent story window, with a 30-day maximum per bounded backfill run. Record exact dates and coverage before launch.
+
+An older history request can use an explicit resumable rebuild into the existing index storage. Bound source objects, bytes, elapsed time, and output size; save durable partial outputs and verified progress after each chunk. A request handler must never perform this historical rebuild implicitly.
+
+Current index retention keeps an active generation and three predecessors. That count alone cannot guarantee a 48-hour page lease or 30-day cursor support. Retention must preserve all versions/derived references required by the declared API window, or the API must advertise a smaller verified window. Do not promise a cursor lifetime that pruning can silently break.
+
+Cursors must survive ordinary index replacement and Space restart within that window. A full index rebuild from the same raw snapshot must reproduce observation IDs, content identities, source effects, and the same logical query results. Missing/corrupt raw objects or a mismatched contract fail closed with explicit repair evidence. A reset is an operator-visible exceptional path, not a hidden hourly bootstrap.
+
+Before any historical build or remote canary, measure a bounded local/CPU sample, estimate the complete runtime, disk use, low/high USD cost, and worst-case retry exposure. Reuse available canonical artifacts. Verify an applicable authorization and ceiling for changed work; the unchanged enrichment contract does not automatically authorize a different backfill Job, model, hardware, or schedule. Test a real interruption and resume before substantial work.
+
+### Local bootstrap recovery
+
+`index:bootstrap` now separates preparation from publication. Preparation freezes an exact source snapshot beside the working SQLite database and commits bounded, chronologically ordered chunks. A restart checks that the database is an exact prefix of that target and reads only the remaining segment bodies. An existing database is never silently replaced. An incomplete-bootstrap marker travels inside SQLite and blocks serving or publication, including after the database is copied without its companion snapshot file. Publication also rejects a candidate that omits a previously published source segment. The default chunk is 128 segments; `INDEX_BOOTSTRAP_MAX_SEGMENTS` bounds a diagnostic run. SIGINT and SIGTERM stop after a saved chunk. These are local SQLite checkpoints, not remote Job checkpoints.
+
+`INDEX_BOOTSTRAP_MODE=publish` is a separate operator action. It requires complete preparation and, when a live index exists, its exact `INDEX_BOOTSTRAP_EXPECTED_DATABASE` value. It applies the remaining source tail, verifies the full database, uploads and verifies it, and then replaces the pointer. Other publishers must be stopped for this cutover. Preparation does not grant publication approval or change a schedule. Setup uses the same two stages and retains partial work after failure.
+
+Full SQLite integrity checks now remain at restore, explicit verification, and publication. Normal tail transactions still verify raw identities and segment projection counts, but do not rescan every SQLite page, including after a harmless source check.
+
+A real local process-interruption canary used the audited 42 tweet segments. The process was killed with SIGKILL after 14 committed segments. Restart read only the remaining 28 segments (197,323 compressed bytes), preserved all 1,196 logical observations, and matched a fresh replay's observation, content, and physical-source digests. The fresh replay read 263,173 compressed bytes and produced a 7,405,568-byte database. It took 0.595 seconds on this 20-core ARM64 host with about 128 MiB peak process RSS; restart took 0.462 seconds for its remaining work. No remote state or live pointer changed, and no model or paid Job ran. This small tweet-only fixture proves local recovery, not production-sized performance, classifier-result coverage, or remote Job recovery. The full source includes mixed/result/registry records and 38,219 network objects, so these times must not be extrapolated as a production completion estimate.
+
+A read-only source pilot then prepared the first 128 real segments of the audited full snapshot. These contain 104,831,277 compressed bytes, 293,280 tweet records and logical observations, 196,276 raw enrichment rows, 372,876 attempt events, 22,122 registry events, and 76 receipts. These are whole-source counts, not the Our Models cohort or proof of its eligibility results. Two bounded invocations took 12.64 and 288.73 seconds. The saved database is 2,500,280,320 bytes; peak RSS in the larger invocation was 1,356,296 KiB. There were no model calls, paid Jobs, remote writes, or live pointer changes.
+
+The full pinned-source preparation completed on September 7. It contains all 38,219 segments, 689,421 logical observations, 326,102 raw enrichment rows, 636,129 attempt events, 48,081 registry events, and 195 receipts. The verified SQLite candidate is 4,811,792,384 bytes. These are whole-source counts, not the Our Models selection. The candidate remains local at `space/.data/consumer-bootstrap-2026-09-07/index/bootstrap.sqlite`; no production pointer changed.
+
+A network read failed after 10,368 saved segments. A four-segment recovery probe then completed, and the final continuation read only the remaining 27,847 segments (92,462,211 compressed bytes). That continuation, including final verification, took 6,213.060 seconds and reached 573,592 KiB peak process RSS. This is not the total time across the initial pilot, failed attempt, probe and continuation. Preparation used 128-segment checkpoints and eight concurrent reads on the same 20-core ARM64 host. Actual new HF Job and model-call cost is $0.00 because neither ran; ordinary host power and existing storage charges are excluded. No runtime or service was installed.
+
+Full-source reader tests exposed and corrected query-order and result-history bounds that smaller fixtures did not reveal. Both code changes are merged in [#46](https://github.com/osolmaz/xtap-pool/pull/46) and [#47](https://github.com/osolmaz/xtap-pool/pull/47), with passing review and CI. The [HTTP handoff](2026-09-07-consumer-http-handoff.md#full-source-initial-coverage-fix) records the complete local read, restart, no-op, history, body-read and final query-parity evidence, including the slower repeated timings. These tests do not establish cloud latency or website freshness. Deployment still requires source-tail catch-up, current predecessor checks, observation grants and the coordinated Our Models cutover. The canonical enrichment and publication schedules remained active and non-concurrent at 15:07 UTC.
+
+## Implementation sequence
+
+### Source audit and fixed tests
+
+- Complete the production-history/capture audit and the count-semantics baseline.
+- Freeze representative private fixtures and their public aggregate report.
+- Include exact repeats, unchanged counts, delayed observations, same-time conflicts, content edits, reply additions, restricted posts, label changes, and a source revision containing only receipts.
+- Fix acceptance limits before comparing implementations. Use practical significance and raw counts when measured differences determine architecture, scaling, or spending.
+
+### Observation delivery and storage
+
+- Establish the compliant passive capture path before changing extension delivery.
+- Separate unique-post export dedup from sampled pool observations.
+- Add durable identity/outbox behavior and replace latest-only ingest admission.
+- Add normalized observation/content-version schemas and deterministic historical normalization.
+- Preserve semantic hashes and prove metric-only input causes zero xTap enrichment calls.
+- Add extension and server failure tests; record vendored changes in `extension/VENDORED.md` and follow the repository's vendor-maintenance process.
+
+### Source effects and consistent reads
+
+- Add source-effect and dependency indexes to verified tail application.
+- Prove before/after membership for affected IDs without a full old-view rebuild.
+- Implement stable raw-snapshot cursors and one pinned pagination mechanism.
+- Handle unit removals/regrouping, metadata changes, completed-content gates, and observations that become visible with newly eligible content.
+- Update index checks, count/digest verification, bootstrap, retention, restore, setup doctor, and worker restore compatibility together.
+
+### API and bounded history recovery
+
+- Implement shared validators first, domain operations in `space/src/`, and thin routes in `space/src/app.ts`.
+- Add explicit bootstrap, changes, and history access with authorization and byte/time limits.
+- Backfill the approved recent history in bounded chunks; save receipts and run interruption/resume tests.
+- Complete local performance and crash tests before deploying the reader.
+
+### Consumer replacement and production proof
+
+- Prepare the Our Models reader and cursor checkpoints against fixed source fixtures.
+- Finish the remaining incremental publication work, including metric-only output publication.
+- Stage both repositories without exposing incompatible live readers. Use the exact predecessor and verified source boundary for transition.
+- During required maintenance, prevent overlap between the canonical worker and maintenance/index publication. Restore the validated canonical schedule immediately afterward.
+- Run the approved bounded production canary, including one source update and one zero-change poll after a restart.
+- Resume story generation and landing-page implementation only after these prerequisites pass.
+
+Do not merge or deploy a companion repository merely because this plan depends on it. Obtain the applicable authorization for each repository and changed operational contract.
+
+## Acceptance matrix
+
+| Case                                           | Required result                                                                                                                                       |
+| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Harmless raw revision change                   | Empty bounded change response; zero historical post-body reads/downloads and zero LLM calls. Cursor can advance.                                      |
+| Ten posts receive new counts                   | Exactly their new permitted observations, subject to documented sampling; no complete unit/post download and no semantic calls in either application. |
+| Unchanged counts at a later visit              | A distinct timestamped observation and a valid zero increase when both endpoints are usable.                                                          |
+| Content edit plus count change                 | One content-version update plus the observation; recompute only affected semantic dependencies.                                                       |
+| New reply in an old conversation               | Relevant unit membership update and current reply timestamp preserved; no unrelated units reread.                                                     |
+| Unit split or merge                            | Remove replaced IDs and upsert replacement membership coherently. No duplicate or stranded consumer entries.                                          |
+| A label loses approval                         | Affected labels/selections change, including removals where needed. No broad post scan merely to discover affected IDs.                               |
+| Newly completed content                        | Publish eligible content and its permitted earlier observations even when the observation arrival predates the old cursor.                            |
+| Restricted or withdrawn source                 | No forbidden body or counters leak through historical/cached pages; process a safe removal.                                                           |
+| New unrelated source write during pagination   | Remaining pages stay at the pinned target; the next cycle receives the later write.                                                                   |
+| Restart between pages                          | The same cursor resumes the same logical read within its advertised lifetime.                                                                         |
+| Late raw segment or delayed observation        | Arrives in a later snapshot difference and is delivered once; time-based ordering does not lose it.                                                   |
+| Duplicate delivery or worker replay            | Same observation IDs and same logical result; no duplicate growth contribution.                                                                       |
+| Failure after raw upload, before SQLite apply  | Verified tail replay restores observations and source effects together.                                                                               |
+| Consumer crash after page receipt              | Page replay is idempotent; its cursor was not acknowledged ahead of durable state.                                                                    |
+| Consumer upload succeeds, pointer update fails | Old website remains valid; retry reuses the prepared output and source cursor.                                                                        |
+| Expired cursor                                 | Explicit reset/recovery response. No silent complete scan.                                                                                            |
+| Counter missing/decreasing/conflicting         | Raw evidence retained; affected growth unavailable or marked anomalous, never invented or clamped.                                                    |
+| Six-hour versus twelve-hour interval           | Rates use actual elapsed time. No unsupported minute-level or daily-total claim.                                                                      |
+| Rebuild and shuffled replay                    | Matching logical digests, observation IDs, selection results, and source boundaries.                                                                  |
+| Queue pressure or interrupted browser          | Durable queued work survives; any sampling/backpressure is explicit and countable.                                                                    |
+
+## Performance and freshness proof
+
+Use a fixed production-sized fixture plus small controlled deltas. Record CPU/hardware, source size, individual-post and unit counts, observation count, cache state, and software revisions. Separate cold bootstrap/restore from warm steady-state reads. Do not include startup in one candidate and omit it from another.
+
+Proposed acceptance targets on the current supported CPU configuration:
+
+- Twenty warm no-change polls: no post-body reads, no inference, and p95 response time at most two seconds.
+- Twenty warm ten-observation updates: bounded observation payloads, zero content upserts, zero inference, and p95 source response time at most five seconds.
+- Small content updates: work and transferred bytes scale with changed units and required context, not the full 42,550-entry fixture.
+- Existing completed content's new metrics: visible through xTap within one successful ingest/index advance or external refresh, without waiting for the six-hour enrichment Job.
+- Our Models: an already eligible observation/content change reaches the public feed within ten minutes of source availability under the tested workload. This is an end-to-end target, not a result established by faster source reads alone.
+
+If a target is missed, report the stage and measured distribution. Do not rename a partial success as completion. Separate browser observation time, pool receipt, source availability, content complete-through, consumer apply time, and live publication time. The website's Today label cannot conceal missing content or old coverage.
+
+Track actual changed IDs, normalized observation count, source segment bodies read, source bytes transferred, SQL rows examined/query plans, output bytes, provider calls, build/verify/upload durations, and applied/published cursor identities. These are ordinary operational facts. No monitoring-specific API or separate telemetry service is required.
+
+Report absolute savings and uncertainty. A speed result alone cannot justify new infrastructure or paid scaling. Prefer the simpler implementation when measurements do not establish a worthwhile improvement.
+
+## Verification commands and production checks
+
+Repository checks for the implementation:
+
+```sh
+npm run test:extension
+npm run test:extension:native
+npm run check
+npm run build
+npm run mutate
+npx -y @simpledoc/simpledoc check
+```
+
+`npm run check` includes format, lint, type checks, tests, extension tests, coverage, and duplicate-code checks. Add named regression tests for each acceptance case; do not replace failures with broad exclusions. Keep changes to installed capture components separate from fixture-only tests.
+
+The documentation task runs `npm run check`, SimpleDoc, and `git diff --check`; implementation, mutation, live API, and production canary results must be recorded when they are actually run.
+
+The live report must identify the tested deployed builds, source snapshot, declared selection, observation coverage, canonical index generation, and exact consumer publication. It must include an interrupted/resumed source read, an interrupted/resumed consumer update, a repeated no-op, and a metric-only update with zero measured semantic calls. Validate the actual website response after publication, not only a successful Job exit.
+
+Never print credentials in the command line, logs, examples, or report. Use the installed CLI's configured authentication in place. New permissions, secret copies, runtime changes, and paid work require their own applicable authorization.
+
+## Remaining decisions and completion
+
+Settle these decisions with evidence during implementation:
+
+- Installed capture path, available historical counter coverage, and whether passive capture needs additional approved browser permissions.
+- Exact normalized identity vectors and raw-field precision handling.
+- Indexed historical-version reads versus one bounded frozen-page mechanism; prove correctness before choosing the simpler viable implementation.
+- Measured response, retention, backfill, and outbox limits, including maximum real unit size.
+- Existing service-account grants and the required observation-history permission.
+- The Our Models artifact layout that permits a small metric/story update without a complete commentary rebuild.
+- Approved history window, remote canary cost ceiling, and transition timing with current Jobs.
+
+The task is complete only when real repeat observations are preserved, historical coverage is honestly reported, content and metric changes can be consumed without normal full reads, recovery passes, and the website's measured update path meets the agreed freshness target. Story generation can then use this source boundary without rebuilding the missing foundation inside Our Models.

docs/2026-09-08-consumer-deployment.md

@@ -0,0 +1,144 @@
+---
+title: Consumer deployment and coverage deadline
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-09-08
+tags: [consumer, deployment, verification]
+---
+
+# Consumer deployment and coverage deadline
+
+The reviewed consumer source at `a56d5b0` was deployed to the existing Space as `ce0aad213f8bcd7fe48a4a9302046e147044adec` on September 8. Index `8cb6974753cd765c288e6fa2c551e3b9db028c346f8045c21d7038557fa5129f` was uploaded, downloaded, checksum-checked, and verified before replacing the pointer. Its 4,891,852,800-byte database preserves the prior index references and all raw records. The source revision is `5dc867c4665bf2360a95a24bca320dadba03fae68b11a528e878a29a152898b1`.
+
+The existing Our Models account now has `observations:read` in addition to its previous scopes. Keys and credential stores are unchanged. Space-side enrichment remains disabled. The canonical enrichment schedule was restored and Job `6a9fd0bd259f8e97255ee9e2` was triggered under its unchanged $10 inference and 2,700-second limits.
+
+## Live checks
+
+Readiness passed after the index and metadata restored. A bounded single-author selection returned 135 posts in three pages. Its history request returned 71 observations for 17 posts with repeated observations; eight of the 25 requested posts had no history in that window. A subsequent unchanged-source request returned zero changes and 598 bytes. The history request took 1.559 seconds and the unchanged request took 3.656 seconds. This is observed source history, not proof that the browser has loaded extension 0.26.0.
+
+The first full-selection Our Models recovery Job, `6a9fcfede686246ca69aa5bf`, then failed after three `503 deadline_exceeded` responses. No full-selection context was published. The failure occurred before source-page completion or model calls. Our Models scheduling remains suspended for repair; the website keeps its verified published database. The independent xTap enrichment Job is not affected by the consumer request deadline.
+
+## Completion boundary query
+
+The completion query repeated the same selected queue twice: once to find the earliest unfinished work, then again to find completed work before it. The replacement materializes that selection once in a single SQL statement. It retains the strict time boundary and handles every unfinished state, empty selections, and all-completed selections. It does not increase HTTP timeouts, change the classifier contract, or require another index migration.
+
+On the same verified local database, the old completion calculation took 7.322 seconds and the replacement took 3.669 seconds, 3.653 seconds less. The returned boundary was identical: September 7 at 22:52:28.518 UTC. Observation coverage remained identical and took 3.264 and 3.212 seconds respectively. These are single local measurements, not statistical or cloud performance claims. The release gate remains successful full-selection requests and real worker recovery on the deployed CPU hardware, with the existing 30-second request deadline unchanged.
+
+Local tests assert one prepared selection statement and check pending, running, retrying, and blocked work, including completed work at the same time as the unfinished boundary. Full validation and live recovery results must be recorded before claiming the consumer is ready for full catch-up.
+
+## Follow-up query and recovery checks
+
+The first completion optimization was not sufficient on the live CPU. A full-selection metadata page took 22.861 seconds, and a later attempt exceeded the unchanged 30-second deadline. The follow-up materializes distinct unit IDs before correlated eligibility checks, then reads each selected post's newest permitted observation through the existing post/time index. It still searches older observations when newer samples are outside the pinned source. No database migration or timeout change is required.
+
+On the same local database, completion took 1.766 seconds and observation coverage took 2.684 seconds. Both clocks matched the previous implementation exactly. These remain local single-run measurements; deployed first-page and content-page checks are required. Tests cover a pinned source with nine later observations, restricted post IDs, an empty selection, and indexed source membership checks.
+
+Activation history reads now use the existing bounded batch helper with concurrency eight. Every immutable plan and activation is still validated, and predecessor checks remain in generation order. Tests require every plan read, a maximum of eight simultaneous reads, unchanged read-only behavior, and rejection of a missing historical plan.
+
+The first enrichment Job failed because the initial deployment did not include the required revision handoff. Canonical stage preparation corrected that deployment error. The subsequent read-only Job `6a9fdb2b8e5f7b7fd14cb6d7` found that the active bootstrap plan still referenced the index from before the consumer deployment. It stopped before inference or output writes. Recovery must preserve the prior plan and checkpoint, verify whether any outputs exist, and prepare a successor from the verified current index. Both schedules remain suspended only for this active recovery.
+
+## Classifier work copy size
+
+Preparing a successor exposed another migration defect before activation: the compact classifier database still contained all consumer observation and change-history tables. Its 2,953,281,536-byte file exceeded Node's 2 GiB `readFile` limit. No successor was activated and no inference call was made.
+
+The worker compaction now excludes those eight consumer-only tables from its disposable copy. Publication still restores the verified full base index and applies worker outputs there; the public history and raw source are not removed. A regression test requires the public source file to remain byte-identical and preserves queue and registry work.
+
+On the verified prepared source, the work file falls to 60,850,176 bytes. The source database SHA-256 remains `5cf69cd60e424d83da3ac2e4b6d4161ca154a1873b392abdd56d361e15859eb8`; all 308,130 queue positions, 306,936 completed positions, and 29,629 registry positions remain represented. Full local checks pass with 771 tests, 88.12% configured coverage, and zero Slophammer DRY candidates. The corrected candidate still requires activation and cloud recovery verification.
+
+## Prepare source metadata before consumer requests
+
+The corrected worker copy was activated as generation 87, run `xtap-b0e01d0c7828c30def3948bcb4251e68`. Its read-only cloud restore passed. The following production retry failed with `fetch failed` after 243 seconds, before any provider calls or output changes. Its checkpoint remains at sequence one. The same run can resume; its plan and prior source files remain intact.
+
+Full-selection HTTP reads still exceeded the 30-second deadline after the SQL changes. A read-only cloud check on cpu-basic measured coverage at 8.220 seconds and privacy at 0.287 seconds against the published index. A separate local check of immutable source preparation uploaded and verified 16,260,675 bytes in 17.187 seconds. These are different executions, not an end-to-end latency measurement. They identify full snapshot preparation as additional work in the first HTTP request.
+
+Source bases are now prepared during verified index startup and the existing background refresh, before storage readiness is restored. Consumer requests only describe an already prepared base and its bounded additions. A missing or oversized base returns `503 source_not_ready` without uploading a snapshot. The existing background refresh prepares the next base; it does not require another API, store, schema version, or longer HTTP deadline. Ingest requests do not perform this preparation.
+
+Regression tests require no snapshot writes from cold, bounded, or oversized descriptions; explicit preparation and rollover; restart recovery; and unchanged immutable-file and checksum validation. Deployment must still prove full-selection metadata and content pages within the existing deadline before Our Models recovery Jobs resume.
+
+## Verified source-read deployment
+
+Source `35f8318026a3ad56868fd0fc57a42142a50bef3b` from PR 52 is deployed as Space revision `1b4196c0439563b00647610af118bb4d5a245513`. Local checks passed with 771 tests, 88.14% configured coverage, and zero DRY candidates. Pi Reviewer reported no findings and CI passed.
+
+The canonical revision handoff preserved active generation 87 and its sequence-one checkpoint. Read-only cloud Job `6a9ffc91b012ba1d5b8f2bc6` verified the handoff in 157.010 seconds. Restore Job `6a9ffdf48e5f7b7fd14cbca7` completed in 149.624 seconds with zero provider calls, zero orphan segments, and unchanged run objects and public index pointer.
+
+At 12:28 UTC, three full-selection metadata requests returned HTTP 200 in 13.899, 20.001, and 13.689 seconds. Their first content pages returned 112 changes each in 4.051, 5.215, and 5.049 seconds. All six requests stayed below the existing 30-second deadline. These tests used source `4b090d0a65daa4806ba97a5af15e97a00c716f8d5471503cd01602b6e7596711`; they prove bounded initial reads, not full backlog publication.
+
+Canonical schedule `6a9ffde3b012ba1d5b8f2bea` is active on the unchanged six-hour schedule and $10 inference/2,700-second bounds. Catch-up Job `6a9ffef6b012ba1d5b8f2c08` started at 12:26:40 UTC. Our Models cloud recovery Job `6a9fff9eb012ba1d5b8f2c1b` then completed in 82.331 seconds. A separate-directory restore recovered its two committed source pages, 112 pending source records, and all 1,117,337 cache entries without a provider call. The second physical recovery Job, `6aa000b9b012ba1d5b8f2c3e`, failed after two `503` responses and a `429 consumer_busy`; the first Job's saved pages remain intact.
+
+The live Space logs explain this failure: new mixed enrichment segments started full `configuration-refresh` scans at 12:34 UTC. Those scans held the same lock used by consumer reads. They continued while xTap made valid progress: its checkpoint reached sequence 41 and queue completion rose from 306,936 to 307,164. The reader must inspect only new segments for actual configuration writes, rather than treating every mixed segment as a configuration change, before retrying the Our Models recovery check.
+
+## Limit configuration refresh to real writes
+
+The configuration-change check now loads only newly verified config or mixed segments and examines their operations. Append-only enrichment, attempt, and registry batches do not start a full configuration refresh. Real configuration writes still take the existing validated reload path. Missing or corrupt new segments fail the check instead of being treated as unchanged.
+
+At 12:46 UTC, the raw log had 138 new mixed segments since 12:30. The first three and last three were checksum-verified and inspected; all six contained append operations only. This matches the failure in the live refresh logs. Regression tests cover append-only mixed batches, genuine dedicated and mixed configuration writes, late-arriving files, unchanged old files, irrelevant tweet files, and failed segment verification.
+
+## Final recovery and resumed schedules
+
+The configuration-tail fix passed 776 tests, 88.15% configured coverage, zero DRY candidates, review with no findings, and CI. Source `cfd46b156aeec2c30225e63f9873c295fb247731` is deployed as Space revision `2b422fa8b081389ab28f930053b3257e8564eee2`.
+
+The earlier catch-up completed all 1,194 pending entries in 202 provider calls, at a receipt cost of $2.0000476. All 308,130 queue positions and 29,629 registry positions were complete. Public index `8ec8e961756db809db65897a1129a7af00e7442ba626c8c699a47b25837ddc6e` was published, and checkpoint sequence 237 retained the outputs. The physical Job exceeded its configured 2,700-second timeout and was canceled. Hugging Face omitted its terminal timestamp; the observed cancellation bounds CPU cost conservatively at $0.025. This is not an invoice. The unused inference reservation was released.
+
+Canonical cloud handoff Job `6aa00b2c32d5d0c22c5ad683` passed. Its result was reused after checking the current activation, immutable plan, and exact checkpoint pointer hashes; replaying all checkpoint history again locally was unnecessary. The new image's read-only restore Job `6aa00e0b32d5d0c22c5ad6c9` recovered sequence 237, all completed work, and 232 claimed segments with zero orphan segments and zero provider calls.
+
+Three full-selection metadata requests then passed in 17.192, 17.950, and 14.115 seconds. Their content pages took 10.278, 4.472, and 11.999 seconds. All stayed below 30 seconds. The verified source was complete through September 8 at 10:14:55 UTC. This source boundary is distinct from the live Our Models publication boundary.
+
+Our Models recovery Job `6aa00f6632d5d0c22c5ad716` completed in 78.315 seconds. A separate-directory restore proved progression from source sequence 2 to 4 and from 112 to 302 source records, preserving the first page and all 1,117,337 cache entries with zero provider calls. The failed intermediate attempt remains recorded.
+
+Both schedules are active again. xTap schedule `6aa00dfa900620b5c77e22ed` started continuation Job `6aa00f2432d5d0c22c5ad70d`; Our Models schedule `6a9fcc6ae686246ca69aa578` started catch-up Job `6aa01065900620b5c77e2385`. The latter retains the approved $180 cumulative inference limit under the $200 task approval. Counted settled cost before these active attempts is $3.7983566, including provider receipts and conservative CPU estimates. Their active reservations are not money spent. Our Models backlog completion has not yet been established.
+
+The first continuation attempt, `6aa00f2432d5d0c22c5ad70d`, ended with `fatal: terminated` after 547.208 seconds. The active generation and checkpoint 237 were unchanged, with all work still complete and no successor activated. No AI work was repeated. Its $0.005 conservative CPU estimate was settled and its unused inference reservation released. The same validated schedule started retry `6aa0124d900620b5c77e23da`. Settled counted cost is now $3.8033566, excluding the active attempts. Our Models catch-up remains active and its observed source requests return HTTP 200 on their first attempt.
+
+## Saved continuation and browser delivery
+
+Job `6aa0124d900620b5c77e23da` reached its physical timeout at September 8 14:38:53 UTC. It had processed 84 additional entries through 14 provider calls for $0.1255376. Its conservative CPU estimate is $0.025 for 2,984.928 seconds. The unused inference and CPU reservations were released. Counted settled cost is $3.9538942, excluding the active Our Models receipt.
+
+The Job advanced to generation 89 and saved the initial checkpoint for `xtap-27c86045f33978c91c9279a0c193b425`. Outer and inner checkpoint hashes verified. All 308,214 queue entries were already complete; 11 registry entries and publication remained. The new run had no output segments or initial progress record. Worker code must publish that record before provider calls, so the preceding Job's final receipt accounts for its inference work. Resume Job `6aa02f9932d5d0c22c5adc0a` uses the existing schedule and checkpoint, with a $10 inference maximum and $0.03 CPU reservation. Those limits are not money spent.
+
+Chrome's registered service worker now reports xTap 0.26.0. After an unsuccessful logged-out test, the operator restored the login and a normal scraper retry recorded 242 observations across 237 posts, including 138 observations of previously known posts. This proves browser collection, not yet delivery of those particular observations through the history API to Our Models.
+
+## Changed-source request deadline
+
+A read-only production request from the Our Models saved idle cursor failed after the client deadline. A second diagnostic request, limited to one change, returned `503 deadline_exceeded` after 31.323 seconds. The Space health check remained successful. The initial full download had completed; this failure concerns the subsequent change-read path.
+
+Source pinning computes coverage before page construction. When taxonomy and approvals were unchanged, the first request also read changed bodies under that same 30-second deadline. The fix returns a privacy-checked boundary-only page after pinning a changed source, using the existing `metadata_sent` cursor marker. Unchanged metadata is not resent. The next request reads bodies against the saved source. An unchanged source still completes in one empty response. No response schema, deadline, compatibility reader, or new state store is added.
+
+Coverage candidate selection now collects distinct changed post IDs before checking selected unit membership. On the existing local September 8 database, a 1,000-segment window returned the same first 100 post IDs in all three comparisons. Baseline times were 2,778, 2,788, and 2,778 ms; candidate times were 980, 981, and 981 ms. The absolute reduction was 1,798–1,807 ms. These are local ARM64 query measurements, not cloud endpoint latency or proof of production recovery. Live change reads must still complete within the existing deadline after deployment.
+
+Regression tests cover boundary-only responses, restart and fixed-source behavior, unchanged-source completion, privacy changes between pages, and deduplicated candidate selection through the existing indexes.
+
+## September 9 verification
+
+At 02:17 UTC, the Space health check passed with storage ready. A change request from the actual Our Models saved cursor still returned `503 deadline_exceeded` after 31.568 seconds, before returning any page. PR #54 is therefore insufficient to establish bounded live change reads. Do not treat its deployment or recovery checks as end-to-end success. Further diagnosis must measure source preparation and coverage separately on the current source before another repair is accepted.
+
+Both earlier catch-up Jobs completed: xTap `6aa03fdb900620b5c77e2ee1` at September 8 17:46 UTC, and Our Models `6aa01065900620b5c77e2385` at 18:38 UTC. Our Models continued automatically under the same logical run. Checkpoint 8949, written September 9 at 02:16 UTC, reports 4,582 calls and $19.5950914 in cumulative inference cost, with one $0.051519 call reservation. Story output remains unset. Physical Job completion did not mean backlog publication.
+
+The live Our Models digest still reports artifact `4e0c744c880378c27391eb2d2bda65c32b9338e7e1a6454beb3439dc5930db9f`, generated September 8 at 06:46 UTC and complete through September 7 at 23:05:38.154 UTC. No current pending count or defensible completion ETA was established. No new paid Job or source reset was launched during this verification.
+
+## September 9 request repair
+
+Two read-only cpu-basic Jobs checked verified published database `2cae942abe77a575584fbdb709093324269b878c1bbca6a96e70e9b5e6b6444c`. Direct coverage took 8.611 seconds; the actual forked reader took 9.022 seconds. Saved-context loading took about 1.65 seconds. A separate local context-pin check took 6.555 seconds, including a 4.369-second metadata write and its read-back. A live history request completed, but full-selection and one-author bootstrap requests still hit the 30-second deadline. These checks do not identify a single proven cause on the live Space.
+
+Deadline errors now name the current request stage, without adding a metrics API or exposing request data, so a remaining production failure identifies the operation rather than inviting another speculative query rewrite. An attempted unchanged-context reuse was rejected because it prevented the rolling history window from advancing. Existing context and history behavior remains unchanged.
+
+Selection predicates now check author membership before public-content eligibility. Three local comparisons returned identical clocks and took 7.490, 7.421, and 7.488 seconds before the change, versus 6.546, 6.577, and 6.687 seconds after it. The absolute reduction was 0.801–0.944 seconds on 307,594 current units. A separate indexed-candidate approach needed 115,472 IDs but gave only a small additional benefit; it was dropped in favor of the simpler predicate order. A 64 MiB SQLite cache was also slower in all three comparisons and was not adopted. None of these local measurements establishes live request success; the saved-cursor production check remains mandatory.
+
+## Changed-source deployment
+
+PR [#54](https://github.com/osolmaz/xtap-pool/pull/54) passed `npm run check`, SimpleDoc, Pi Reviewer with no findings, and CI run `34251140734`. It merged as `853d3da38cef549f506a65453e452597bb36e763`. The existing Space received that source as `92c4e167b482e979a21ad30aba4938625301e664` at 16:57 UTC. Live change-read verification remains pending while the Space restores its index; the first probe received a database-not-restored response, not a change-read deadline result.
+
+The preceding Job `6aa02f9932d5d0c22c5adc0a` completed generation 89 and published verified index `31760d111d4790d90a3390c71d44467bda469b5efa630a1fa2dc6554525c037a`. It then processed 378 entries in generation 90 through 64 provider calls. Its final receipts report $0.698001 in inference; estimated CPU cost is $0.021. Unused reservations were released. Counted settled cost reached $4.6728952, excluding the active Our Models run.
+
+Read-only handoff Job `6aa03b0b32d5d0c22c5adde5` and new-image restore Job `6aa03ee732d5d0c22c5ade3f` both completed. Each added an estimated $0.0015 in CPU cost and zero provider calls. The restore verified checkpoint 73, 308,510 of 308,861 completed queue entries, all 29,717 registry entries, 72 claimed output segments, and zero orphan segments. It preserved run `xtap-8ad4b8908b14c51d353a14f08b4a9c8d`; no plan was reset.
+
+Canonical schedule `6aa03eb132d5d0c22c5ade37` is active after maintenance. Catch-up Job `6aa03fdb900620b5c77e2ee1` resumed that saved work with the unchanged $10 inference cap and 2,700-second physical limit. Settled cost before that Job is $4.6758952. Its maximum exposure is $10.0225, separate from the active Our Models cumulative $180 inference cap. All remain within the approved $200 task limit. Our Models Job `6aa01065900620b5c77e2385` remains active; these xTap results do not establish that its backlog is published.
+
+## September 9 live recovery
+
+PR [#55](https://github.com/osolmaz/xtap-pool/pull/55) passed the full check, final review with no findings, and CI run `34309536404`. Source `8ea1007bc4f562addb9a8c176d123aa8af58160a` was deployed as Space revision `158529d2eabfa6a746e9801bded6334646f00e22`. The change reports the stage of a request deadline and preserves selection semantics while reducing SQL work. It does not raise the deadline or change history-window behavior.
+
+Handoff Job `6aa0dcd4900620b5c77e516c` and new-image restore Job `6aa0de5332d5d0c22c5af3d0` passed without provider calls. The restore retained generation 94, checkpoint 131, 309,369 of 309,749 queue entries, all 29,788 registry entries, and 130 claimed segments with zero orphans. Canonical schedule `6aa0de4632d5d0c22c5af3ce` is active again. No extra manual enrichment Job was launched during this deployment.
+
+After the Space restored its index, the actual saved Our Models cursor returned three successful change pages in 17.414, 12.411, and 4.979 seconds. Another 60 consecutive pages passed in 2.709–11.076 seconds each, returning 29 changed entries and 1,110 observations. The response source was complete through September 8 at 22:06:11.884 UTC. More pages remained at the bounded test limit; this does not establish that the whole backlog was read or published. These results establish live success after deployment, not that predicate ordering alone caused the recovery.
+
+The responses contain repeat observations for 271 posts. For example, post `2097336073663684798` had 16 views and no replies at September 8 15:42:47.316 UTC, then 29 views and one reply at 22:33:35.137 UTC. This verifies delivery of real counter history, rather than only a history endpoint returning an empty response. The read-only checks made no provider calls and changed no remote source or consumer state.
+
+The restore Job used 169.656 seconds. Its conservative CPU estimate is $0.0015; the unused $0.021 hold was released. The earlier timed-out Job's unresolved provider exposure remains counted in the task budget. Our Models publication is a separate gate and is not claimed complete here.

extension/.coveragerc

@@ -0,0 +1,5 @@
+[run]
+include = native-host/xtap_core.py
+
+[report]
+show_missing = true

extension/.github/workflows/release.yml

@@ -0,0 +1,58 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build extension zip
+        run: |
+          VERSION="${GITHUB_REF_NAME#v}"
+          zip -r "xtap-pool-${VERSION}.zip" \
+            manifest.json \
+            background.js \
+            popup.html popup.js popup.css \
+            options.html options.js \
+            pool-connect.js \
+            reload.html reload.js \
+            cutover.html cutover.js \
+            debug.html debug.js debug.css \
+            icons/ \
+            lib/ \
+            native-host/xtap_core.py \
+            native-host/xtap_host.py \
+            native-host/xtap_daemon.py \
+            native-host/com.xtap.host.json \
+            native-host/com.xtap.daemon.plist \
+            native-host/com.xtap.daemon.service \
+            native-host/install.sh \
+            native-host/install.ps1 \
+            native-host/xtap_host.bat \
+            native-host/xtap_daemon.bat
+
+      - name: Publish release
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          VERSION="${GITHUB_REF_NAME#v}"
+          ASSET="xtap-pool-${VERSION}.zip#xTap Pool ${VERSION} (extension + native host)"
+          # Create the release if it doesn't exist yet (auto-generated notes
+          # from commits since the previous tag), otherwise just upload the
+          # zip into the release that was pre-created with hand-written notes.
+          if gh release view "$GITHUB_REF_NAME" >/dev/null 2>&1; then
+            gh release upload "$GITHUB_REF_NAME" "$ASSET" --clobber
+          else
+            gh release create "$GITHUB_REF_NAME" \
+              --title "xTap Pool ${VERSION}" \
+              --generate-notes \
+              "$ASSET"
+          fi

extension/.github/workflows/test.yml

@@ -0,0 +1,35 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install Python test dependencies
+        run: pip install pytest pytest-cov
+
+      - name: Run Python tests
+        run: python3 -m pytest tests/ -v --cov --cov-report=xml:coverage-python.xml
+
+      - name: Run JS tests with coverage
+        run: npx c8 --reporter=lcov --report-dir=coverage-js node --test tests/*.test.mjs
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v5
+        with:
+          files: coverage-python.xml,coverage-js/lcov.info

extension/.gitignore

@@ -0,0 +1,18 @@
+.DS_Store
+.claude/
+CLAUDE.md
+*.crx
+*.pem
+tweets*.jsonl
+node_modules/
+__pycache__
+.coverage
+coverage/
+coverage-*/
+.pytest_cache/
+tests/fixtures/private-raw/*
+!tests/fixtures/private-raw/CAPTURE.md
+tests/e2e/.extension-out/
+tests/e2e/certs/
+!tests/e2e/test-extension.pem
+native-host/*.generated.json

extension/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 xTap contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

extension/README.md

@@ -0,0 +1,404 @@
+<p align="center">
+  <img src="icons/icon128.png" alt="xTap logo" width="96" />
+</p>
+
+<h1 align="center">xTap</h1>
+
+<p align="center">
+  <strong>Passively capture tweets as you browse X/Twitter</strong>
+</p>
+
+<p align="center">
+  <a href="#installation">Installation</a> &middot;
+  <a href="#how-it-works">How It Works</a> &middot;
+  <a href="#is-this-safe-to-use">Stealth</a> &middot;
+  <a href="#output-format">Output Format</a> &middot;
+  <a href="#configuration">Configuration</a> &middot;
+  <a href="LICENSE">License</a>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/platform-macOS%20%7C%20Linux%20%7C%20Windows-blue" alt="Platform" />
+  <img src="https://img.shields.io/badge/browser-Chrome-green" alt="Chrome" />
+  <img src="https://img.shields.io/badge/license-MIT-yellow" alt="MIT License" />
+  <a href="https://codecov.io/gh/mkubicek/xTap"><img src="https://codecov.io/gh/mkubicek/xTap/graph/badge.svg" alt="codecov" /></a>
+</p>
+
+---
+
+xTap is a Chrome extension that passively reads the GraphQL responses X/Twitter already sends to your browser and saves captured tweets as structured JSONL. It does not patch X's page code or make extra X requests for ordinary capture.
+
+## Features
+
+- **Zero footprint** — no additional network requests; captures what your browser already receives
+- **Structured output** — each tweet saved as a clean JSON object with author, metrics, media, and more
+- **Article support** — long-form X articles are captured when X returns the full body, with inline image references and Draft.js block structure
+- **Video download** — download videos from tweets using yt-dlp (or direct MP4 fallback) via the extension popup. Requires the HTTP daemon. **Note:** unlike passive capture, video downloads make additional network requests to X and are not stealth.
+- **Image download** — opt-in toggle in the popup ("Download images automatically") fetches photos from `pbs.twimg.com` to `<output_dir>/media/<tweet_id>/<filename>` as you browse. If enabled after browsing, already-seen photo tweets are rechecked for media download without bumping the captured-tweet counter. Daemon-side; rate-limited; logs to `media-manifest.jsonl`. **Note:** also not stealth — adds requests to the Twitter CDN.
+- **Pause / resume** — click the extension icon to toggle capture on the fly
+- **Live counter** — badge on the extension icon shows tweets captured this session
+- **Multi-tab aware** — multiple X tabs feed into the same service worker with shared deduplication
+- **Debug logging** — optional toggle to write timestamped service worker logs to a date-rotated file
+- **Debug dashboard** — internal extension page with live capture events, transport health, and a parser sandbox for testing GraphQL response parsing
+- **Cross-platform** — works on macOS, Linux, and Windows
+
+## Scrape job receipts
+
+xTap can provide capture receipts to the allowlisted [Infinite Feed Scroller](https://github.com/osolmaz/infinite-feed-scroller) extension. The scroller binds a run to a blank source tab before navigating that tab to X. xTap stores each post ID, post time, first-seen time, source endpoint, and ordered run observation in browser IndexedDB.
+
+Live-list and search timelines use the same observation format. Up to two runs can be active. Each GraphQL response is routed only to the run bound to its Chrome tab, and finishing one run leaves the other running. The local index supports replay after an extension restart and records whether each list post was known before the run. The external port never exposes tweet text or the pool token.
+
+## How It Works
+
+```
+        X/Twitter GraphQL responses
+                    │
+                    ▼
+     ┌────────────────────────────┐
+     │ Chrome Debugger Network API│  Browser-owned
+     │ reads completed responses  │
+     └──────────────┬─────────────┘
+                    │
+                    ▼
+     ┌────────────────────────────┐
+     │     background.js          │  Service worker
+     │ parse, receipt, dedup, batch│
+     └──────────┬─────────┬───────┘
+                │         │
+          HTTP  │         │ native messaging
+     (all data) │         │ (token bootstrap only)
+                │         │
+                ▼         ▼
+     ┌──────────────┐  ┌──────────────┐
+     │ xtap_daemon  │  │ xtap_host.py │
+     │ (HTTP)       │  │ (stdio)      │
+     └──────┬───────┘  └──────────────┘
+            │
+            ▼
+       tweets-YYYY-MM-DD.jsonl
+```
+
+1. The service worker attaches Chrome's Debugger Network domain to open X tabs.
+2. It reads completed GraphQL response bodies without changing page JavaScript or the DOM.
+3. The existing parser normalizes tweets, records tab-bound scrape receipts, deduplicates normal capture, and builds upload batches.
+4. The HTTP daemon remains the primary disk transport. Native messaging bootstraps its token and provides the existing fallback.
+
+## Is This Safe to Use?
+
+xTap does not post, like, follow, scroll, or request extra X data. It records responses that Chrome received because of browsing in the attached tab.
+
+X can still enforce its rules based on account activity, browsing patterns, or automated scrolling performed by another tool. Review those rules and use a suitable account before running long scrapes.
+
+### Browser behavior
+
+xTap reads responses that Chrome already received and does not send extra X requests for capture. The extension no longer replaces page fetch or XHR functions and leaves the page DOM unchanged.
+
+- **No extra X requests** — debugger capture only reads responses the browser already received
+- **No page hooks** — xTap does not patch page-owned `fetch` or XHR APIs, inject page code, or add DOM signaling
+- **Extension-owned capture** — parsing, receipts, deduplication, staging, and transport stay in extension contexts
+
+Chrome shows its standard debugger notice while xTap is attached to an X tab. Opening DevTools or another debugger can detach xTap from that tab. Reload or revisit the tab to attach again.
+
+Capture still has account and platform risk. xTap can observe automated scrolling performed by another tool, and X can enforce its rules based on that browsing behavior.
+
+## Installation
+
+### Requirements
+
+| | Requirement |
+|---|---|
+| **Browser** | Google Chrome |
+| **Runtime** | Python 3 |
+| **OS** | macOS, Linux, or Windows |
+| [`yt-dlp`](https://github.com/yt-dlp/yt-dlp#installation) (optional) | For best-quality video downloads |
+
+### 1. Load the extension
+
+**Chrome:**
+1. Open `chrome://extensions`
+2. Enable **Developer mode** (top right)
+3. Click **Load unpacked** and select the `xtap/` directory
+4. Copy the extension ID shown on the card (used by native host install)
+
+### 2. Install the native host
+
+<details>
+<summary><strong>macOS</strong></summary>
+
+```bash
+cd native-host
+./install.sh <your-extension-id> chrome
+```
+
+This installs the native messaging host (for auth token bootstrap) and an HTTP daemon (`xtap_daemon.py`) that runs via launchd. The daemon runs independently of the browser process tree and has its own TCC permissions, so it can write to protected paths like `~/Documents` and iCloud Drive. The installer captures your current `PATH` so the daemon can find tools like `yt-dlp`.
+
+The extension automatically detects the daemon via the native host's auth token. If the daemon is not running, the extension will show a red "!" badge and an error in the popup.
+
+</details>
+
+<details>
+<summary><strong>Linux</strong></summary>
+
+```bash
+cd native-host
+./install.sh <your-extension-id> chrome
+```
+
+This installs the native messaging host and an HTTP daemon (`xtap_daemon.py`) that runs as a systemd user service. The daemon enables video downloads and provides the same HTTP transport as macOS.
+
+</details>
+
+<details>
+<summary><strong>Windows (PowerShell)</strong></summary>
+
+```powershell
+cd native-host
+.\install.ps1 -ExtensionId <your-extension-id> -Browser chrome
+```
+
+This installs the native messaging host and an HTTP daemon (`xtap_daemon.py`) as a Windows Scheduled Task that starts at logon. The daemon enables video downloads and provides the same HTTP transport as macOS/Linux.
+
+</details>
+
+### 3. Browse X
+
+Open [x.com](https://x.com) and browse normally. The badge counter on the extension icon shows how many tweets have been captured this session. Click the icon to see stats and pause/resume capture.
+
+> **After updating the extension:** Reload xTap at `chrome://extensions`, then reload any open X tabs. The extension must reattach its passive debugger capture to those tabs.
+
+### Upgrading from a previous version
+
+After updating the extension files:
+1. Re-run the installer (`install.sh` on macOS/Linux, `install.ps1` on Windows) — this updates the daemon configuration and picks up new Python code
+2. Reload the extension in your browser extension manager
+3. Hard-reload any open X tabs (`Cmd+Shift+R` / `Ctrl+Shift+R`)
+
+**From versions before v0.20.0 on macOS/Linux:** Re-running `install.sh` is **required** — the native messaging manifest now points to a wrapper script (`~/.xtap/xtap_host_wrapper.sh`) that uses an absolute Python path, fixing native host launch failures on macOS where Chrome's minimal environment couldn't find `python3`.
+
+**From versions before v0.19.0:** The native messaging host (`xtap_host.py`) no longer handles tweet writing — all data now flows through the HTTP daemon exclusively. Re-running `install.sh` is required to update the daemon's service configuration (adds `XTAP_LOG_LEVEL` support). The extension will show a red "!" badge if the daemon is not running, instead of silently falling back to native messaging.
+
+**From versions before v0.13.0 on macOS:** Re-running `install.sh` is **required** for video download support — the daemon needs an updated launchd configuration to find yt-dlp on your PATH.
+
+### Troubleshooting
+
+If the extension shows "Not connected" or a red "!" badge:
+
+1. **Check if the daemon is running:**
+   ```bash
+   curl http://127.0.0.1:17381/status
+   # Should return: {"ok": true, "version": "..."}
+   ```
+
+2. **Check daemon logs:**
+   ```bash
+   cat ~/.xtap/daemon-stderr.log
+   ```
+   The daemon logs startup diagnostics (Python version, output directory, token status) on every start. Common issues:
+   - `FATAL: ~/.xtap/secret not found` — run `install.sh` first
+   - `FATAL: Cannot bind to 127.0.0.1:17381` — another instance is already running
+   - Import errors — check Python version (`python3 --version`, requires 3.x)
+
+3. **Check native host errors** (token bootstrap failures):
+   ```bash
+   cat ~/.xtap/host-error.log
+   ```
+   This file is created when the native messaging host crashes. It includes the Python version, script path, and full traceback.
+
+4. **Enable debug logging** for detailed request-level daemon logs:
+   ```bash
+   export XTAP_LOG_LEVEL=debug
+   cd native-host && ./install.sh <extension-id> chrome
+   ```
+   Then check `~/.xtap/daemon-stderr.log` for per-request details (method, path, duration, tweet counts).
+
+5. **Verify the native messaging manifest** points to the correct path:
+   ```bash
+   # Chrome (macOS):
+   cat ~/Library/Application\ Support/Google/Chrome/NativeMessagingHosts/com.xtap.host.json
+   ```
+   The `path` field should point to `~/.xtap/xtap_host_wrapper.sh` (macOS/Linux) or `xtap_host.bat` (Windows). The wrapper uses an absolute Python path so native messaging works even in Chrome's minimal environment. If it still points directly at `xtap_host.py`, re-run `install.sh`.
+
+## Configuration
+
+### Output directory
+
+The easiest way to change where tweets are saved is through the extension popup — click the xTap icon and enter your preferred path in the **Output directory** field.
+
+Alternatively, set the `XTAP_OUTPUT_DIR` environment variable and re-run the installer — the daemon runs as a system service (launchd/systemd/Scheduled Task), so the variable must be baked into the service definition; exporting it in a shell or before launching the browser has no effect:
+
+```bash
+export XTAP_OUTPUT_DIR="$HOME/Documents/xtap-data"
+./native-host/install.sh <extension-id> [chrome|firefox]
+```
+
+On Windows, set it as a *user* environment variable (`[Environment]::SetEnvironmentVariable('XTAP_OUTPUT_DIR', 'D:\\path', 'User')`) and re-run `install.ps1`.
+
+| Setting | Default | Description |
+|---|---|---|
+| Popup "Output directory" | *(empty — uses default)* | Overrides the output path per-session |
+| `XTAP_OUTPUT_DIR` env var | `~/Downloads/xtap` | Fallback when no popup setting is configured (set at install time) |
+| Debug Dashboard | — | Accessible via popup link; shows live capture events, transport health, debug logging and discovery mode toggles, and parser sandbox |
+
+> **macOS note:** On macOS, the HTTP daemon (installed via `install.sh`) runs outside browser TCC sandboxes and can write to protected paths like `~/Documents` and iCloud Drive after a one-time macOS permission prompt.
+
+### Download tuning
+
+These daemon environment variables are optional. Re-run the installer after changing them so the service definition picks them up.
+
+| Setting | Default | Description |
+|---|---:|---|
+| `XTAP_IMAGE_DELAY_MS` | `100` | Delay between background image requests |
+| `XTAP_MAX_FILE_MB` | `50` | Max size for one downloaded image |
+| `XTAP_MAX_MEDIA_MB` | *(unlimited)* | Max cumulative image bytes per daemon process |
+| `XTAP_MAX_VIDEO_MB` | `500` | Max size for one direct MP4 fallback video download; set `0` or less to disable the cap |
+| `XTAP_CONN_TIMEOUT_S` | `15` | Daemon per-connection socket timeout — drops idle/stalled connections; bad or `<=0` values fall back to the default (the timeout is never disabled) |
+| `XTAP_SHUTDOWN_GRACE_S` | `10` | Max seconds the daemon waits for in-flight requests at shutdown before force-exiting; bad or `<=0` values fall back to the default |
+
+## Output Format
+
+Output is written to daily files (`tweets-YYYY-MM-DD.jsonl`). Each line is a self-contained JSON object:
+
+```jsonc
+{
+  "id": "1234567890",
+  "url": "https://x.com/handle/status/1234567890",
+  "created_at": "2024-01-01T00:00:00.000Z",
+  "author": {
+    "id": "987654321",
+    "username": "handle",
+    "display_name": "Display Name",
+    "verified": false,
+    "is_blue_verified": true,
+    "follower_count": 1234
+  },
+  "text": "Full tweet text...",
+  "lang": "en",
+  "metrics": {
+    "likes": 10,
+    "retweets": 5,
+    "replies": 2,
+    "views": 1000,
+    "bookmarks": 1,
+    "quotes": 0
+  },
+  "media": [],
+  "urls": [],
+  "hashtags": [],
+  "mentions": [],
+  "in_reply_to": null,
+  "quoted_tweet_id": null,
+  "conversation_id": "1234567890",
+  "is_retweet": false,
+  "retweeted_tweet_id": null,
+  "is_subscriber_only": false,          // true for subscriber-only tweets
+  "is_article": true,                   // present only for long-form articles
+  "article": {                          // present only for long-form articles
+    "title": "Article Title",
+    "text": "Rendered plain text with ![img](media/<id>/file.png) refs",
+    "blocks": [],                       // raw Draft.js content_state blocks
+    "media": [{                         // article image references
+      "id": "...",
+      "url": "https://pbs.twimg.com/...",  // original CDN URL
+      "filename": "image.png",
+      "local_path": "media/<tweet_id>/image.png",
+      "width": 1200,
+      "height": 800
+    }]
+  },
+  "source_endpoint": "HomeTimeline",    // which GraphQL endpoint
+  "captured_at": "2024-01-01T00:00:00.000Z"
+}
+```
+
+For regular tweets, `is_article` and `article` are absent. For articles, `text` contains a markdown-style rendering of the article with inline image references pointing to `media/<tweet_id>/`. Article stubs without `content_state.blocks` are skipped rather than saved as incomplete rows.
+
+### Media file convention
+
+When the "Download images automatically" toggle is on, the daemon writes photos to:
+
+```
+<output_dir>/media/<tweet_id>/<basename(media.url)>
+```
+
+Top-level photo `media[]` entries do **not** carry a `local_path` field — the path is derived by convention so consumers can reconstruct it directly from `tweet.id` + the URL basename. Article media (`tweet.article.media[]`) does include `local_path` because that path is also embedded in the rendered article markdown (`![](media/<id>/file.png)`) so the article body works as a self-contained document. Download status (success / 404 / quota / oversize / blocked) is appended to `<output_dir>/media-manifest.jsonl`.
+
+## Project Structure
+
+```
+xTap/
+├── manifest.json              # Chrome MV3 extension manifest
+├── background.js              # Service worker: capture, parsing, receipts, staging, transport
+├── lib/graphql-capture.js     # Passive Chrome Debugger Network capture
+├── lib/dedup.js               # Capture and image-backfill deduplication
+├── lib/tweet-parser.js        # GraphQL response parser
+├── popup.html/js/css          # Extension popup UI
+├── options.html/js            # Pool connection settings
+├── pool-connect.js            # Private pool token handoff
+├── reload.html/js             # Safe extension reload page
+├── cutover.html/js            # Coordinated active-run cutover page
+├── debug.html/js/css          # Debug dashboard (live events, transport health, parser sandbox)
+├── icons/                     # Extension icons
+└── native-host/
+    ├── xtap_core.py              # Shared file I/O logic
+    ├── xtap_host.py              # Native messaging host — token bootstrap only (Python, stdio)
+    ├── xtap_daemon.py            # HTTP daemon
+    ├── com.xtap.daemon.plist     # launchd plist template (macOS)
+    ├── com.xtap.daemon.service   # systemd unit template (Linux)
+    ├── com.xtap.host.json        # Native host manifest template (Chrome)
+    ├── install.sh                # Installer for macOS / Linux
+    ├── install.ps1               # Installer for Windows
+    ├── xtap_host.bat             # Windows native host wrapper
+    └── xtap_daemon.bat           # Windows daemon wrapper
+```
+
+## Development
+
+After modifying extension files (`background.js`, `lib/`, or `popup.*`), reload the extension at `chrome://extensions` and hard-reload any open X tabs. Chrome asks for the Debugger permission and shows its standard notice while an X tab is attached.
+
+**Debug dashboard:** Click "Debug Dashboard" in the popup to open a live view of capture events, transport health, and a parser sandbox for testing `extractTweets` against raw GraphQL JSON. Debug logging and discovery mode toggles are also here — enable debug logging to write timestamped service worker logs to `debug-YYYY-MM-DD.log`, or discovery mode to log endpoint response shapes to the console.
+
+**Dev mode:** When loaded unpacked (developer mode), the extension prefers `chrome.storage.session` for the `seenIds` dedup cache, and falls back to `chrome.storage.local` if session storage APIs are unavailable. When session storage is available, reloading the extension automatically clears the cache — no need to manually clear storage between test runs.
+
+After modifying Python host files (`xtap_core.py`, `xtap_host.py`, `xtap_daemon.py`), the native host picks up changes on next browser restart. To restart the HTTP daemon immediately:
+
+**macOS (launchd):**
+```bash
+launchctl kickstart -k gui/$(id -u)/com.xtap.daemon   # restart
+launchctl bootout gui/$(id -u)/com.xtap.daemon        # stop
+launchctl print gui/$(id -u)/com.xtap.daemon          # status
+tail -f ~/.xtap/daemon-stderr.log                     # logs
+```
+
+**Linux (systemd):**
+```bash
+systemctl --user restart com.xtap.daemon   # restart
+systemctl --user stop com.xtap.daemon      # stop
+systemctl --user status com.xtap.daemon    # status
+journalctl --user -u com.xtap.daemon -f    # logs
+```
+
+**Windows (Scheduled Task, PowerShell):**
+```powershell
+Stop-ScheduledTask -TaskName xTapDaemon; Start-ScheduledTask -TaskName xTapDaemon  # restart
+Stop-ScheduledTask -TaskName xTapDaemon                                            # stop
+Get-ScheduledTask -TaskName xTapDaemon                                             # status
+Get-Content ~\.xtap\daemon-stderr.log -Tail 50 -Wait                               # logs
+```
+
+## Testing
+
+```bash
+python3 -m pytest tests/ -v
+node --test tests/*.test.mjs
+```
+
+CI runs these on every push to `main` with coverage uploaded to [Codecov](https://codecov.io/gh/mkubicek/xTap).
+
+Parser fixture packs live under `tests/fixtures/`. Raw captures stay local in
+`tests/fixtures/private-raw/` (gitignored), while committed anonymized packs
+live in `tests/fixtures/sanitized/`. The anonymization methodology and review
+checklist are documented in `tests/fixtures/FIXTURES.md`.
+
+## License
+
+[MIT](LICENSE) — use it however you like.

extension/VENDORED.md

@@ -0,0 +1,81 @@
+# Vendored xTap
+
+This directory vendors the xTap Chrome extension.
+
+- Upstream: https://github.com/mkubicek/xTap
+- Vendored at commit: `9eba39a3c972649f07df98c3874cac6de38b383f` (`v0.24.0`)
+- Local extension version: `0.26.0`
+- License: MIT (see `LICENSE`, unchanged)
+
+Keep upstream code style (vanilla JS, MV3, no bundler) so future re-syncs
+stay cheap diffs. Re-syncing is a deliberate manual step: diff upstream against
+this directory, excluding the modifications below.
+
+## Local modifications
+
+- `manifest.json` — renamed to `xtap-pool`; added `debugger` and
+  `unlimitedStorage` permissions, the `https://*.hf.space/*` host permission,
+  and the `pool-connect.js` content script.
+- Firefox manifests, build scripts, page-interception tests, and browser E2E
+  harness are omitted. Firefox does not implement Chrome's extension debugger
+  API, and this fork does not fall back to page-owned fetch/XHR interception.
+- `lib/pool-sync.js` — **new**: persistent observation queue and batched flush to
+  the pool Space's `/api/ingest` with backoff. Queue admission and sampling state
+  are saved together. Storage errors and queue overflow leave staged work pending.
+  A batch leaves the queue only after the server's `added`, `duplicates`, and
+  `rejected` counts account for every record. No extra success flag is required.
+  Explicitly rejected records are saved in a bounded local rejection archive, so one invalid
+  post does not block valid work. A full rejection archive blocks removal instead
+  of discarding records.
+- `lib/observations.js` — **new**: stable local observation identities, exact
+  nullable counters, and five-minute sampling for unchanged content. Later visits
+  and immediate content or visibility edits are independent of unique-post export
+  deduplication.
+- `pool-connect.js` — **new**: content script for the Space's `/connect` page;
+  hands the pool token to the service worker (no copy-paste).
+- `background.js` — imports `lib/pool-sync.js`; new captures also feed the pool
+  queue before unique-post deduplication; new `POOL_*` message handlers; a
+  `chrome.alarms` periodic pool flush and staged-response retry;
+  `initPoolSync()` during startup. Staging uses local storage and preserves the
+  original observation time through a browser restart. Upstream's coupled
+  dedup/buffer persistence, image backfill, bounded splitting, startup flush,
+  and HTTP-only transport remain in place around the pool path.
+- `popup.html` / `popup.js` — added the "Pool sync" section (status, connect,
+  sync-now, pause) and an Options link.
+- `options.html` / `options.js` — **new**: configure the pool Space URL and
+  paste a token manually (fallback for the automatic handoff).
+- `tests/pool-sync.test.mjs` — **new**: node --test coverage for the queue,
+  flush, backoff and connect flows.
+- `lib/scrape-receipts.js` and `lib/scrape-bridge.js` — **new**: durable
+  IndexedDB receipts for X list observations plus a cursor-based external port
+  used by the allowlisted Infinite Feed Scroller extension.
+- `reload.html` and `reload.js` — **new**: extension-owned deployment page that
+  reloads the unpacked extension through `chrome.runtime.reload()`. The explicit
+  `?fail-active=1` cutover option gates new runs, fails active receipt runs,
+  and reloads; the new service worker clears the gate before accepting clients.
+- `cutover.html` and `cutover.js` — **new**: one-shot fresh-state page with a
+  distinct URL so a cached older reload page cannot bypass active-run cleanup.
+- `tests/graphql-capture.test.mjs` — **new**: debugger attachment, response-body
+  capture, operation extraction, and failed-request coverage. Receipt startup
+  retries passive debugger attachment for bounded transient failures.
+- `tests/scrape-receipts.test.mjs` — **new**: receipt persistence, per-list
+  coverage, search and live-list normalization, replay, typed errors, active-run,
+  and sender-allowlist coverage.
+- `lib/graphql-capture.js` — **new**: passively reads completed X GraphQL
+  responses through Chrome's Debugger Network domain without changing page-owned
+  JavaScript or the DOM. The old MAIN-world fetch/XHR patch and isolated bridge
+  were removed.
+- `background.js` sends debugger-captured list and search responses through one
+  parser and receipt path before normal capture deduplication. It stages the
+  exact request URL and Chrome tab ID so recovery replays tab-bound receipts
+  before clearing the write-ahead record. Observations go only to the matching
+  run. Up to four leased receipt runs may be active. Heartbeats, source-tab
+  closure, and lease expiry reclaim runs that no longer have a live client.
+- `lib/tweet-parser.js` — accepts object-shaped Draft.js `entityMap`s in
+  addition to X's array-of-pairs shape (+ regression test). New counter objects
+  carry `format: "exact-v1"`; absent, rounded, invalid, and unsafe counters are
+  null, while actual zero counts remain zero. Recovery passes the original response
+  time to `extractTweets`.
+- `.github/workflows/release.yml` — packages the pool settings, connection,
+  reload, and cutover pages; omits unsupported Firefox artifacts.
+- Removed upstream `AGENTS.md` / `CLAUDE.md` (superseded by the repo root's).

extension/background.js

@@ -0,0 +1,1111 @@
+// xTap — Service Worker (background)
+import { GraphqlCapture } from './lib/graphql-capture.js';
+import { extractTweets } from './lib/tweet-parser.js';
+import {
+  initPoolSync,
+  poolEnqueue,
+  poolFlush,
+  poolConnect,
+  poolSetConfig,
+  poolTogglePause,
+  poolStatus,
+} from './lib/pool-sync.js';
+import { ScrapeReceiptBridge } from './lib/scrape-bridge.js';
+import { dedupTweet } from './lib/dedup.js';
+
+const NATIVE_HOST = 'com.xtap.host';
+const BATCH_SIZE = 50;
+const FLUSH_INTERVAL_MS = 30_000;
+const MAX_SEEN_IDS = 50_000;
+const HTTP_TIMEOUT_MS = 10_000;
+const MAX_BUFFER_SIZE = 2000;
+const IMAGE_BACKFILL_FLAG = '__xtap_image_backfill';
+
+let captureEnabled = true;
+let buffer = [];
+let flushTimer = null;
+let seenIds = new Set();
+// Session-only set of tweet IDs already forwarded for image-backfill this SW
+// lifetime. Lets us re-enter the daemon once per duplicate-with-photos so
+// images skipped at original capture time get downloaded, without spamming
+// the daemon on every scroll/re-navigation. Cleared on SW restart by design
+// — the downloader's per-file os.path.exists is the source of truth.
+let imageCheckedIds = new Set();
+let sessionCount = 0;
+let allTimeCount = 0;
+let outputDir = '';
+let imageDownload = false;
+let debugLogging = false;
+let verboseLogging = false;
+let logBuffer = [];
+const isDevMode = !chrome.runtime.getManifest().update_url;
+const graphqlCapture = new GraphqlCapture({ onResponse: handleGraphqlResponse });
+const scrapeReceiptBridge = new ScrapeReceiptBridge({
+  ensureSourceCapture: (tabId) => graphqlCapture.ensureAttached(tabId),
+});
+scrapeReceiptBridge.attach();
+chrome.tabs?.onRemoved?.addListener((tabId) => {
+  void scrapeReceiptBridge.finishSourceTab(tabId).catch((error) => {
+    console.warn(`[xTap] Could not close scrape receipt for tab ${tabId}: ${error}`);
+  });
+});
+graphqlCapture.attach();
+const hasSessionStorage = !!chrome.storage.session;
+const traceStorage = chrome.storage.session || chrome.storage.local;
+let stageSeq = 0;
+let _saveChain = Promise.resolve();
+let readyResolve;
+const ready = new Promise(r => { readyResolve = r; });
+const autoDumpedThisSession = new Set();
+
+// --- Recent tweets cache (for video download lookup) ---
+const MAX_RECENT_TWEETS = 1000;
+const recentTweets = new Map();
+// tweetId → downloadId for in-progress downloads (so popup can resume polling)
+const activeDownloads = new Map();
+
+// --- Transport state ---
+// 'http' | 'none'
+let transport = 'none';
+let httpToken = null;
+let httpPort = null;
+
+// --- State persistence ---
+
+function seenIdsStorage() {
+  return (isDevMode && hasSessionStorage) ? chrome.storage.session : chrome.storage.local;
+}
+
+// Staged observations must survive a browser restart as well as worker sleep.
+function stagingStorage() {
+  return chrome.storage.local;
+}
+let recoveryRunning = false;
+const activeStagedPayloads = new Set();
+
+async function stagePayload(endpoint, data, metadata = {}) {
+  const key = `stg_${Date.now()}_${stageSeq++}`;
+  try {
+    await stagingStorage().set({
+      [key]: { endpoint, data, stagedAt: Date.now(), ...metadata },
+    });
+    return key;
+  } catch (e) {
+    console.warn('[xTap] Failed to stage payload (quota?):', e.message);
+    emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: null, status: 'STAGE_FAILED', reason: e.message });
+    throw e;
+  }
+}
+
+async function clearStagedPayload(key) {
+  if (!key) return;
+  try {
+    await stagingStorage().remove(key);
+  } catch (e) {
+    console.warn('[xTap] Failed to clear staged payload:', e.message);
+  }
+}
+
+async function recoverStagedPayloads() {
+  if (recoveryRunning) return;
+  recoveryRunning = true;
+  try { await replayStagedPayloads(); }
+  finally { recoveryRunning = false; }
+}
+
+async function replayStagedPayloads() {
+  let store;
+  try {
+    store = await stagingStorage().get(null);
+  } catch (e) {
+    console.warn('[xTap] Failed to read staging storage for recovery:', e.message);
+    return;
+  }
+  const keys = Object.keys(store).filter(k => k.startsWith('stg_')).sort((a, b) => {
+    const [, tsA, seqA] = a.split('_');
+    const [, tsB, seqB] = b.split('_');
+    return (tsA - tsB) || (seqA - seqB);
+  });
+  if (keys.length === 0) return;
+
+  let recoveredCount = 0;
+
+  for (const key of keys) {
+    if (activeStagedPayloads.has(key)) continue;
+    const entry = store[key];
+    let produced = false;
+    try {
+      if (!entry || !entry.data || !Number.isFinite(entry.stagedAt)) {
+        console.error(`[xTap] Invalid staged payload retained: ${key}`);
+        continue;
+      }
+      const tweets = extractTweets(entry.endpoint, entry.data, entry.stagedAt);
+      for (const tweet of tweets) tweet.source_endpoint = entry.endpoint;
+      if (tweets.length > 0) {
+        await recordScrapeReceipts({
+          endpoint: entry.endpoint,
+          url: entry.requestUrl,
+          sourceTabId: entry.sourceTabId,
+        }, tweets);
+        await enqueueTweets(tweets, entry.endpoint);
+        recoveredCount += tweets.length;
+        produced = true;
+      }
+    } catch (e) {
+      console.warn(`[xTap] Recovery failed for ${key}; staged data retained:`, e.message);
+      continue;
+    }
+    // Persist buffer before clearing WAL entry — if SW dies mid-recovery,
+    // already-cleared entries must have their tweets in durable storage.
+    // If saveState fails, keep the WAL entry for retry on next startup.
+    if (produced && !(await saveState())) continue;
+    await clearStagedPayload(key);
+  }
+
+  if (recoveredCount > 0) {
+    emitTraceEvent({ timestamp: Date.now(), endpoint: 'recovery', tweetId: null, status: 'RECOVERY_COMPLETE', reason: `recovered ${recoveredCount} tweets from ${keys.length} staged payloads` });
+    console.log(`[xTap] Recovery: ${recoveredCount} tweets from ${keys.length} staged payloads`);
+  }
+}
+
+// Serialized via _saveChain so concurrent callers can't interleave writes
+// (back-to-back handlers would otherwise race, and an earlier snapshot could
+// land after a later one, rolling back buffered tweets).  Returns true on
+// success, false on storage error — callers gate WAL clears on this.
+function saveState() {
+  const p = _saveChain.then(() => _saveStateImpl());
+  _saveChain = p.catch(() => {});
+  return p;
+}
+
+async function _saveStateImpl() {
+  // seenIds and tweetBuffer are coupled in one write so a quota failure
+  // loses both rather than persisting seenIds without the buffer (which
+  // would create ghost-dedup entries that permanently block those tweets).
+  const seenArr = [...seenIds].slice(-MAX_SEEN_IDS);
+  try {
+    if (isDevMode && hasSessionStorage) {
+      await Promise.all([
+        chrome.storage.session.set({ seenIds: seenArr, tweetBuffer: buffer }),
+        chrome.storage.local.set({ allTimeCount, captureEnabled }),
+      ]);
+    } else {
+      await chrome.storage.local.set({ seenIds: seenArr, tweetBuffer: buffer, allTimeCount, captureEnabled });
+    }
+    return true;
+  } catch (e) {
+    console.warn('[xTap] Failed to persist state:', e.message);
+    return false;
+  }
+}
+
+async function restoreState() {
+  const [seenStored, stored] = await Promise.all([
+    seenIdsStorage().get(['seenIds', 'tweetBuffer']),
+    chrome.storage.local.get(['allTimeCount', 'captureEnabled', 'outputDir', 'imageDownload', 'debugLogging', 'verboseLogging']),
+  ]);
+  if (seenStored.seenIds) seenIds = new Set(seenStored.seenIds.filter(Boolean));
+  if (Array.isArray(seenStored.tweetBuffer)) buffer = seenStored.tweetBuffer;
+  if (typeof stored.allTimeCount === 'number') allTimeCount = stored.allTimeCount;
+  if (typeof stored.captureEnabled === 'boolean') captureEnabled = stored.captureEnabled;
+  if (typeof stored.outputDir === 'string') outputDir = stored.outputDir;
+  if (typeof stored.imageDownload === 'boolean') imageDownload = stored.imageDownload;
+  if (typeof stored.debugLogging === 'boolean') debugLogging = stored.debugLogging;
+  if (typeof stored.verboseLogging === 'boolean') verboseLogging = stored.verboseLogging;
+}
+
+// --- Debug logging ---
+
+const _origLog = console.log;
+const _origWarn = console.warn;
+const _origError = console.error;
+
+const MAX_LOG_BUFFER = 5000;
+
+function debugLog(level, args) {
+  if (!debugLogging) return;
+  const ts = new Date().toISOString();
+  const text = args.map(a => {
+    if (typeof a === 'string') return a;
+    try { return JSON.stringify(a); } catch { return String(a); }
+  }).join(' ');
+  logBuffer.push(`${ts} [${level}] ${text}`);
+  // Cap so an unreachable daemon can't grow the buffer unboundedly.
+  if (logBuffer.length > MAX_LOG_BUFFER) {
+    logBuffer.splice(0, logBuffer.length - MAX_LOG_BUFFER);
+  }
+}
+
+console.log = (...args) => { _origLog(...args); debugLog('LOG', args); };
+console.warn = (...args) => { _origWarn(...args); debugLog('WARN', args); };
+console.error = (...args) => { _origError(...args); debugLog('ERROR', args); };
+
+// --- HTTP transport ---
+
+async function httpFetch(method, path, body) {
+  const url = `http://127.0.0.1:${httpPort}${path}`;
+  const opts = { method, headers: {} };
+  if (httpToken) {
+    opts.headers['Authorization'] = `Bearer ${httpToken}`;
+  }
+  if (body !== undefined) {
+    opts.headers['Content-Type'] = 'application/json';
+    opts.body = JSON.stringify(body);
+  }
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), HTTP_TIMEOUT_MS);
+  opts.signal = controller.signal;
+  try {
+    const resp = await fetch(url, opts);
+    return { status: resp.status, data: await resp.json() };
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+// AbortSignal.timeout may not exist in all MV3 runtimes (e.g. older Firefox)
+function makeTimeoutSignal(ms) {
+  if (typeof AbortSignal !== 'undefined' && typeof AbortSignal.timeout === 'function') {
+    return AbortSignal.timeout(ms);
+  }
+  const controller = new AbortController();
+  setTimeout(() => controller.abort(), ms); // timer self-clears when SW terminates
+  return controller.signal;
+}
+
+async function probeHttp(port, token) {
+  try {
+    const headers = {};
+    if (token) headers['Authorization'] = `Bearer ${token}`;
+    const resp = await fetch(`http://127.0.0.1:${port}/status`, {
+      headers,
+      signal: makeTimeoutSignal(3000)
+    });
+    const data = await resp.json();
+    return data.ok === true;
+  } catch {
+    return false;
+  }
+}
+
+async function getTokenViaNative() {
+  return new Promise((resolve) => {
+    let port;
+    try {
+      port = chrome.runtime.connectNative(NATIVE_HOST);
+    } catch {
+      resolve(null);
+      return;
+    }
+    let settled = false;
+    function finish(value) {
+      if (settled) return;
+      settled = true;
+      clearTimeout(timer);
+      try { port.disconnect(); } catch {}
+      resolve(value);
+    }
+    const timer = setTimeout(() => finish(null), 5000);
+    port.onMessage.addListener((msg) => {
+      if (msg.ok && msg.token) {
+        finish({ token: msg.token, port: msg.port });
+      } else {
+        finish(null);
+      }
+    });
+    port.onDisconnect.addListener(() => {
+      const err = chrome.runtime.lastError;
+      if (err) console.warn('[xTap] Native host disconnected:', err.message);
+      finish(null);
+    });
+    try {
+      port.postMessage({ type: 'GET_TOKEN' });
+    } catch {
+      finish(null);
+    }
+  });
+}
+
+async function initTransport() {
+  // 1. Check cached token
+  const cached = await chrome.storage.local.get(['httpToken', 'httpPort']);
+  if (cached.httpToken && cached.httpPort) {
+    const alive = await probeHttp(cached.httpPort, cached.httpToken);
+    if (alive) {
+      httpToken = cached.httpToken;
+      httpPort = cached.httpPort;
+      transport = 'http';
+      updateBadge(); // clear a stale '!' from a previous daemon-down session
+      console.log('[xTap] Using HTTP transport (cached token)');
+      return;
+    }
+  }
+
+  // 2. Try to get token from native host
+  const result = await getTokenViaNative();
+  if (result) {
+    const alive = await probeHttp(result.port, result.token);
+    if (alive) {
+      httpToken = result.token;
+      httpPort = result.port;
+      transport = 'http';
+      await chrome.storage.local.set({ httpToken, httpPort });
+      updateBadge(); // clear a stale '!' from a previous daemon-down session
+      console.log('[xTap] Using HTTP transport (token from native host)');
+      return;
+    }
+  }
+
+  // 3. No transport available
+  transport = 'none';
+  console.warn('[xTap] No transport available — daemon may not be running');
+  updateTransportBadge();
+}
+
+// --- Unified send ---
+
+async function sendToHost(msg) {
+  if (transport !== 'http') {
+    console.warn('[xTap] No transport available, message dropped');
+    return null;
+  }
+
+  let path, body;
+  if (msg.type === 'TEST_PATH') {
+    path = '/test-path';
+    body = { outputDir: msg.outputDir };
+  } else if (msg.type === 'LOG') {
+    path = '/log';
+    body = { lines: msg.lines };
+    if (msg.outputDir) body.outputDir = msg.outputDir;
+  } else if (msg.type === 'DUMP') {
+    path = '/dump';
+    body = { filename: msg.filename, content: msg.content };
+    if (msg.outputDir) body.outputDir = msg.outputDir;
+  } else if (msg.type === 'CHECK_YTDLP') {
+    path = '/check-ytdlp';
+    body = {};
+  } else if (msg.type === 'DOWNLOAD_VIDEO') {
+    path = '/download-video';
+    body = { tweetUrl: msg.tweetUrl, directUrl: msg.directUrl, postDate: msg.postDate };
+    if (msg.outputDir) body.outputDir = msg.outputDir;
+  } else if (msg.type === 'DOWNLOAD_STATUS') {
+    path = '/download-status';
+    body = { downloadId: msg.downloadId };
+  } else {
+    path = '/tweets';
+    body = { tweets: msg.tweets };
+    if (msg.outputDir) body.outputDir = msg.outputDir;
+    if (msg.imageDownload) body.image_download = true;
+  }
+
+  try {
+    const { status, data } = await httpFetch('POST', path, body);
+    if (status === 401 || status === 403) {
+      // The daemon is alive but our token is stale (e.g. rotated by a
+      // reinstall). Reset the transport so the next flush re-probes and
+      // fetches a fresh token via the native host — otherwise we'd retry
+      // the same dead credentials forever.
+      console.error(`[xTap] Daemon rejected credentials (HTTP ${status}), resetting transport`);
+      transport = 'none';
+      updateTransportBadge();
+      return null;
+    }
+    if (data && typeof data === 'object') data.httpStatus = status;
+    return data;
+  } catch (e) {
+    console.error('[xTap] HTTP send failed:', e.message);
+    transport = 'none';
+    updateTransportBadge();
+    return null;
+  }
+}
+
+// --- Batching & flushing ---
+
+function scheduledFlush() {
+  if (buffer.length > 0 || logBuffer.length > 0) flush();
+}
+
+// The setTimeout flush timer dies with the service worker (MV3 kills it ~30s
+// after the last event), stranding the final partial batch until a later
+// session. A chrome.alarms backstop survives SW termination and wakes the SW
+// to deliver whatever is still buffered, then clears itself.
+const FLUSH_ALARM = 'xtap-flush';
+let flushAlarmSet = false;
+
+function ensureFlushAlarm() {
+  if (flushAlarmSet || !chrome.alarms) return;
+  flushAlarmSet = true;
+  chrome.alarms.create(FLUSH_ALARM, { periodInMinutes: 1 });
+}
+
+chrome.alarms?.onAlarm.addListener((alarm) => {
+  if (alarm.name !== FLUSH_ALARM) return;
+  (async () => {
+    await ready;
+    if (buffer.length > 0 || logBuffer.length > 0) await flush();
+    if (buffer.length === 0 && logBuffer.length === 0) {
+      chrome.alarms.clear(FLUSH_ALARM);
+      flushAlarmSet = false;
+    }
+  })();
+});
+
+async function flushLogs() {
+  if (logBuffer.length === 0) return;
+  if (transport === 'none') return;
+  const lines = logBuffer.splice(0);
+  const message = { type: 'LOG', lines };
+  if (outputDir) message.outputDir = outputDir;
+  await sendToHost(message);
+}
+
+let lastReprobe = 0;
+const REPROBE_COOLDOWN_MS = 30_000;
+
+async function reprobeTransport() {
+  const now = Date.now();
+  if (now - lastReprobe < REPROBE_COOLDOWN_MS) return false;
+  lastReprobe = now;
+  console.log('[xTap] Re-probing HTTP daemon...');
+  // Try cached credentials first (fast path)
+  if (httpToken && httpPort) {
+    const alive = await probeHttp(httpPort, httpToken);
+    if (alive) {
+      transport = 'http';
+      updateBadge();
+      console.log('[xTap] HTTP daemon recovered (cached token)');
+      return true;
+    }
+  }
+  // Try getting a fresh token via native host
+  const result = await getTokenViaNative();
+  if (result) {
+    const alive = await probeHttp(result.port, result.token);
+    if (alive) {
+      httpToken = result.token;
+      httpPort = result.port;
+      transport = 'http';
+      await chrome.storage.local.set({ httpToken, httpPort });
+      updateBadge();
+      console.log('[xTap] HTTP daemon recovered (fresh token)');
+      return true;
+    }
+  }
+  // Fallback: check chrome.storage.local (token may have been written by
+  // a prior session or injected externally for testing)
+  const stored = await chrome.storage.local.get(['httpToken', 'httpPort']);
+  if (stored.httpToken && stored.httpPort) {
+    const alive = await probeHttp(stored.httpPort, stored.httpToken);
+    if (alive) {
+      httpToken = stored.httpToken;
+      httpPort = stored.httpPort;
+      transport = 'http';
+      updateBadge();
+      console.log('[xTap] HTTP daemon recovered (stored token)');
+      return true;
+    }
+  }
+  return false;
+}
+
+// Cap per POST so a backlog (e.g. after a daemon outage) can't exceed the
+// daemon's 10 MB body limit and wedge in a 413-retry loop.
+const MAX_TWEETS_PER_POST = 200;
+let flushInFlight = false;
+
+function tweetForHost(tweet) {
+  if (!tweet || !tweet[IMAGE_BACKFILL_FLAG]) return tweet;
+  const { [IMAGE_BACKFILL_FLAG]: _ignored, ...clean } = tweet;
+  return clean;
+}
+
+function isBufferedImageBackfill(tweet) {
+  return !!(tweet && tweet[IMAGE_BACKFILL_FLAG]);
+}
+
+// Remove the delivered/dropped tweets by object identity, never by position.
+// An overflow eviction in enqueueTweets can splice the buffer front while
+// flush() awaits the POST, so buffer[0..batch.length) may no longer be the
+// tweets we sent — a positional splice would then discard never-sent tweets
+// (their ids are already in seenIds, so the loss is permanent). Reassigning
+// `buffer` is safe: no other code runs between the await and here.
+function removeDelivered(batch) {
+  const delivered = new Set(batch);
+  buffer = buffer.filter(t => !delivered.has(t));
+}
+
+async function flush() {
+  if (buffer.length === 0 && logBuffer.length === 0) return;
+
+  if (transport === 'none') {
+    if (!(await reprobeTransport())) return;
+  }
+
+  if (buffer.length > 0 && !flushInFlight) {
+    flushInFlight = true;
+    try {
+      // Send from the front without removing — the batch stays in `buffer`
+      // (and thus in persisted state) until the daemon acks, so SW death
+      // mid-POST can't lose it. The daemon dedups by ID, so a death after
+      // the POST but before the ack only costs a duplicate send, not data.
+      let postCap = MAX_TWEETS_PER_POST;
+      let splitRemaining = 0;
+      while (buffer.length > 0) {
+        const batch = buffer.slice(0, postCap);
+        const message = { tweets: batch.map(tweetForHost) };
+        if (outputDir) message.outputDir = outputDir;
+        if (imageDownload) message.imageDownload = true;
+
+        const resp = await sendToHost(message);
+        if (!resp || !resp.ok) {
+          // 413: the batch is too large in bytes (count cap doesn't bound
+          // huge article tweets). Retrying it unchanged would wedge the
+          // queue forever — halve and retry; a single tweet that alone
+          // exceeds the daemon's body limit can never be delivered, so
+          // drop it with a trace event.
+          if (resp?.httpStatus === 413) {
+            if (batch.length > 1) {
+              postCap = Math.max(1, Math.floor(batch.length / 2));
+              splitRemaining = batch.length;
+              continue;
+            }
+            const oversized = batch[0];
+            removeDelivered([oversized]);
+            console.error(`[xTap] Dropping tweet ${oversized.id}: exceeds daemon body limit`);
+            emitTraceEvent({ timestamp: Date.now(), endpoint: 'flush', tweetId: oversized.id, status: 'DROPPED_OVERSIZED', reason: 'single tweet exceeds daemon body limit' });
+            postCap = MAX_TWEETS_PER_POST; // only the fat batch pays the split cost
+            splitRemaining = 0;
+            await saveState();
+            continue;
+          }
+          // Anything else: batch stays buffered for the next flush.
+          console.error('[xTap] Host rejected tweets:', resp?.error || 'no response');
+          break;
+        }
+        removeDelivered(batch);
+        if (splitRemaining > 0) {
+          splitRemaining -= batch.length;
+          if (splitRemaining <= 0) {
+            postCap = MAX_TWEETS_PER_POST; // restore after a fat batch forced a split
+            splitRemaining = 0;
+          }
+        }
+        await saveState();
+      }
+    } finally {
+      flushInFlight = false;
+    }
+  }
+
+  if (debugLogging) await flushLogs();
+}
+
+// --- Trace events ---
+
+const MAX_TRACE_EVENTS = 50;
+let traceEvents = [];
+let traceFlushTimer = null;
+
+function emitTraceEvent(event) {
+  traceEvents.push(event);
+  if (traceEvents.length > MAX_TRACE_EVENTS) {
+    traceEvents = traceEvents.slice(-MAX_TRACE_EVENTS);
+  }
+  if (!traceFlushTimer) {
+    traceFlushTimer = setTimeout(() => {
+      traceFlushTimer = null;
+      traceStorage.set({ lastEvents: traceEvents });
+    }, 500);
+  }
+}
+
+async function enqueueTweets(tweets, endpoint = 'unknown') {
+  // Persist observation delivery before lifetime post-ID dedup or local state changes.
+  await poolEnqueue(tweets);
+  let newCount = 0;
+  let queuedCount = 0;
+  let backfillCount = 0;
+  let skippedCount = 0;
+  let droppedBackfillCount = 0;
+  for (const tweet of tweets) {
+    // Always cache for video lookup (even dupes — updates with latest data)
+    if (tweet.id) {
+      recentTweets.set(tweet.id, tweet);
+      // FIFO eviction
+      if (recentTweets.size > MAX_RECENT_TWEETS) {
+        const oldest = recentTweets.keys().next().value;
+        recentTweets.delete(oldest);
+      }
+    }
+
+    const wasSeen = !!(tweet.id && seenIds.has(tweet.id));
+    if (!dedupTweet(tweet, seenIds, { imageBackfill: imageDownload, imageCheckedIds })) {
+      skippedCount++;
+      emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: tweet.id, status: 'DEDUPLICATED', reason: 'seenIds' });
+      continue;
+    }
+
+    const isImageBackfill = wasSeen && !tweet.is_article;
+    if (isImageBackfill) {
+      backfillCount++;
+      tweet[IMAGE_BACKFILL_FLAG] = true;
+    } else {
+      newCount++;
+    }
+    buffer.push(tweet);
+    queuedCount++;
+    emitTraceEvent({
+      timestamp: Date.now(),
+      endpoint,
+      tweetId: tweet.id,
+      status: isImageBackfill ? 'IMAGE_BACKFILL' : 'ACCEPTED',
+      reason: null,
+    });
+  }
+
+  // FIFO eviction if seenIds grows too large
+  if (seenIds.size > MAX_SEEN_IDS) {
+    const arr = [...seenIds];
+    seenIds = new Set(arr.slice(arr.length - MAX_SEEN_IDS));
+  }
+  if (imageCheckedIds.size > MAX_SEEN_IDS) {
+    const arr = [...imageCheckedIds];
+    imageCheckedIds = new Set(arr.slice(arr.length - MAX_SEEN_IDS));
+  }
+
+  sessionCount += newCount;
+  allTimeCount += newCount;
+  updateBadge();
+  if (queuedCount > 0) ensureFlushAlarm();
+
+  if (buffer.length > MAX_BUFFER_SIZE) {
+    let droppedOldestCount = 0;
+    while (buffer.length > MAX_BUFFER_SIZE) {
+      const backfillIndex = buffer.findIndex(isBufferedImageBackfill);
+      if (backfillIndex !== -1) {
+        const [dropped] = buffer.splice(backfillIndex, 1);
+        if (dropped?.id) imageCheckedIds.delete(dropped.id);
+        droppedBackfillCount++;
+      } else {
+        buffer.shift();
+        droppedOldestCount++;
+      }
+    }
+    const droppedTotal = droppedOldestCount + droppedBackfillCount;
+    console.warn(`[xTap] Buffer overflow: dropped ${droppedTotal} tweets (${droppedBackfillCount} image backfill, ${droppedOldestCount} oldest; cap: ${MAX_BUFFER_SIZE})`);
+    emitTraceEvent({ timestamp: Date.now(), endpoint, tweetId: null, status: 'BUFFER_OVERFLOW', reason: `dropped ${droppedTotal}` });
+  }
+
+  if (skippedCount > 0 || backfillCount > 0 || droppedBackfillCount > 0) {
+    console.log(`[xTap] Dedup: ${newCount} new, ${backfillCount} image backfill, ${skippedCount} duplicates skipped, ${droppedBackfillCount} backfill dropped (seenIds: ${seenIds.size})`);
+  }
+}
+
+// --- Badge ---
+
+function updateBadge() {
+  if (transport === 'none') return; // don't overwrite error badge
+  const text = sessionCount > 0 ? String(sessionCount) : '';
+  chrome.action.setBadgeText({ text });
+  chrome.action.setBadgeBackgroundColor({ color: '#1D9BF0' });
+}
+
+function updateTransportBadge() {
+  if (transport === 'none') {
+    chrome.action.setBadgeText({ text: '!' });
+    chrome.action.setBadgeBackgroundColor({ color: '#E0245E' });
+  }
+}
+
+// --- Verbose logging (discovery mode) ---
+
+function summarizeShape(obj, depth = 0, maxDepth = 3) {
+  if (depth >= maxDepth) return typeof obj === 'object' && obj !== null ? (Array.isArray(obj) ? '[…]' : '{…}') : typeof obj;
+  if (obj === null) return 'null';
+  if (Array.isArray(obj)) {
+    if (obj.length === 0) return '[]';
+    return `[${obj.length}× ${summarizeShape(obj[0], depth + 1, maxDepth)}]`;
+  }
+  if (typeof obj === 'object') {
+    const keys = Object.keys(obj);
+    if (keys.length === 0) return '{}';
+    const entries = keys.slice(0, 12).map(k => `${k}: ${summarizeShape(obj[k], depth + 1, maxDepth)}`);
+    if (keys.length > 12) entries.push(`…+${keys.length - 12} more`);
+    return `{ ${entries.join(', ')} }`;
+  }
+  if (typeof obj === 'string') return obj.length > 80 ? `str(${obj.length})` : JSON.stringify(obj);
+  return String(obj);
+}
+
+function verboseLog(endpoint, data) {
+  if (!verboseLogging) return;
+  const shape = summarizeShape(data);
+  console.log(`[xTap:verbose] ${endpoint} response shape: ${shape}`);
+
+  // Auto-dump first response per endpoint per session (for agentic fixture creation)
+  if (!autoDumpedThisSession.has(endpoint)) {
+    autoDumpedThisSession.add(endpoint);
+    const ts = Date.now();
+    const filename = `dump-${endpoint}-${ts}.json`;
+    const content = JSON.stringify({ endpoint, data }, null, 2);
+    sendToHost({ type: 'DUMP', filename, content, outputDir: outputDir || undefined });
+    console.log(`[xTap:autodump] ${endpoint} → ${filename}`);
+  }
+
+  // Manual dump: target a specific endpoint or tweet IDs for multi-sample capture.
+  // Configure via console:
+  //   chrome.storage.local.set({verboseDumpIds: ['1234567890']})   — dump responses containing these IDs
+  //   chrome.storage.local.set({verboseDumpEndpoint: 'TweetDetail'}) — dump all responses for this endpoint
+  // Dumps are written to <outputDir>/dump-<endpoint>-<timestamp>.json
+  chrome.storage.local.get(['verboseDumpIds', 'verboseDumpEndpoint'], (cfg) => {
+    let shouldDump = false;
+    let reason = '';
+
+    if (cfg.verboseDumpEndpoint === endpoint) {
+      shouldDump = true;
+      reason = `endpoint=${endpoint}`;
+    }
+    if (!shouldDump && cfg.verboseDumpIds?.length) {
+      const json = JSON.stringify(data);
+      for (const id of cfg.verboseDumpIds) {
+        if (json.includes(id)) {
+          shouldDump = true;
+          reason = `id=${id}`;
+          break;
+        }
+      }
+    }
+
+    if (shouldDump) {
+      const ts = Date.now();
+      const filename = `dump-${endpoint}-${ts}.json`;
+      const content = JSON.stringify({ endpoint, data }, null, 2);
+      sendToHost({ type: 'DUMP', filename, content, outputDir: outputDir || undefined });
+      console.log(`[xTap:dump] ${endpoint} (${reason}) → ${filename} (${content.length} chars)`);
+    }
+  });
+}
+
+// --- Message handling ---
+
+// Endpoints that use /i/api/graphql/ but never contain tweets
+const IGNORED_ENDPOINTS = new Set([
+  'DataSaverMode', 'getAltTextPromptPreference', 'useDirectCallSetupQuery',
+  'XChatDmSettingsQuery', 'useTotalAdCampaignsForUserQuery', 'useStoryTopicQuery',
+  'useSubscriptionsPaymentFailureQuery', 'PinnedTimelines', 'ExploreSidebar',
+  'SidebarUserRecommendations', 'useFetchProductSubscriptionsQuery',
+  'ExplorePage', 'UserByScreenName',
+  'ProfileSpotlightsQuery', 'useFetchProfileSections_canViewExpandedProfileQuery',
+  'UserSuperFollowTweets', 'NotificationsTimeline', 'AuthenticatePeriscope',
+  'BookmarkFoldersSlice', 'EditBookmarkFolder', 'fetchPostQuery',
+  'useReadableMessagesSnapshotMutation', 'UsersByRestIds',
+  'CreatorStudioTabBarItemQuery', 'DelegatedAccountListQuery',
+  'HandleShareBannerQuery', 'isEligibleForVoButtonUpsellQuery',
+  'useEligibleForHandleShareBannerQuery', 'useRelayDelegateDataPendingQuery',
+]);
+
+async function recordScrapeReceipts(msg, tweets) {
+  if (!Number.isInteger(msg.sourceTabId) || typeof msg.url !== 'string') return;
+  try {
+    await scrapeReceiptBridge.recordGraphqlResponse({
+      endpoint: msg.endpoint,
+      requestUrl: msg.url,
+      sourceTabId: msg.sourceTabId,
+      tweets,
+    });
+  } catch (error) {
+    console.error('[xTap] Failed to record scrape receipts:', error);
+  }
+}
+
+async function handleGraphqlResponse(msg) {
+  await ready;
+  verboseLog(msg.endpoint, msg.data);
+  if (!captureEnabled) return;
+  if (IGNORED_ENDPOINTS.has(msg.endpoint)) {
+    if (verboseLogging) console.log(`[xTap:verbose] ${msg.endpoint} (ignored)`);
+    return;
+  }
+
+  const observedAt = Date.now();
+  const stageKey = await stagePayload(msg.endpoint, msg.data, {
+    requestUrl: msg.url,
+    sourceTabId: msg.sourceTabId,
+    stagedAt: observedAt,
+  });
+  activeStagedPayloads.add(stageKey);
+  try {
+    const tweets = extractTweets(msg.endpoint, msg.data, observedAt);
+    for (const tweet of tweets) tweet.source_endpoint = msg.endpoint;
+    if (tweets.length > 0) {
+      const missingAuthor = tweets.filter(tweet => !tweet.author?.username).length;
+      const missingText = tweets.filter(tweet => !tweet.text).length;
+      let warning = '';
+      if (missingAuthor > 0) warning += ` | ${missingAuthor} missing username`;
+      if (missingText > 0) warning += ` | ${missingText} missing text`;
+      console.log(`[xTap] ${msg.endpoint}: ${tweets.length} tweets${warning}`);
+      await recordScrapeReceipts(msg, tweets);
+      await enqueueTweets(tweets, msg.endpoint);
+      if (await saveState()) await clearStagedPayload(stageKey);
+    } else {
+      await clearStagedPayload(stageKey);
+    }
+    if (buffer.length >= BATCH_SIZE) flush();
+  } catch (error) {
+    console.error(`[xTap] Parse error for ${msg.endpoint}:`, error, '| data keys:', Object.keys(msg.data || {}).join(', '));
+    emitTraceEvent({ timestamp: Date.now(), endpoint: msg.endpoint, tweetId: null, status: 'PARSER_ERROR', reason: error.message });
+    // Keep the original response and observation time until durable delivery succeeds.
+  } finally {
+    activeStagedPayloads.delete(stageKey);
+  }
+}
+
+chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
+  if (msg.type === 'POOL_CONNECT') {
+    (async () => {
+      const result = await poolConnect(msg, sender && sender.url ? sender.url : '');
+      sendResponse(result);
+    })();
+    return true;
+  }
+
+  if (msg.type === 'POOL_STATUS') {
+    sendResponse(poolStatus());
+    return;
+  }
+
+  if (msg.type === 'POOL_SET_CONFIG') {
+    (async () => {
+      await poolSetConfig(msg);
+      sendResponse(poolStatus());
+    })();
+    return true;
+  }
+
+  if (msg.type === 'POOL_TOGGLE_PAUSE') {
+    (async () => {
+      await poolTogglePause();
+      sendResponse(poolStatus());
+    })();
+    return true;
+  }
+
+  if (msg.type === 'POOL_FLUSH_NOW') {
+    (async () => {
+      try {
+        await ready;
+        await poolFlush();
+        await recoverStagedPayloads();
+        sendResponse(poolStatus());
+      } catch (error) {
+        sendResponse({ ...poolStatus(), lastError: error.message });
+      }
+    })();
+    return true;
+  }
+
+  if (msg.type === 'GET_STATUS') {
+    (async () => {
+      await ready;
+      sendResponse({
+        captureEnabled,
+        sessionCount,
+        allTimeCount,
+        connected: transport !== 'none',
+        buffered: buffer.length,
+        outputDir,
+        imageDownload,
+        debugLogging,
+        verboseLogging,
+        transport,
+        transportError: transport === 'none'
+          ? 'Daemon not running. Check ~/.xtap/daemon-stderr.log'
+          : null,
+        discoveredEndpoints: [...autoDumpedThisSession],
+      });
+    })();
+    return true;
+  }
+
+  if (msg.type === 'SET_DEBUG') {
+    debugLogging = !!msg.debugLogging;
+    chrome.storage.local.set({ debugLogging });
+    if (debugLogging) {
+      console.log('[xTap] Debug logging enabled');
+    } else {
+      logBuffer = [];
+    }
+    sendResponse({ debugLogging });
+    return true;
+  }
+
+  if (msg.type === 'SET_VERBOSE') {
+    verboseLogging = !!msg.verboseLogging;
+    chrome.storage.local.set({ verboseLogging });
+    console.log(`[xTap] Verbose logging ${verboseLogging ? 'enabled' : 'disabled'}`);
+    sendResponse({ verboseLogging });
+    return true;
+  }
+
+  if (msg.type === 'SET_OUTPUT_DIR') {
+    const newDir = msg.outputDir || '';
+    if (newDir && transport === 'http') {
+      sendToHost({ type: 'TEST_PATH', outputDir: newDir }).then((resp) => {
+        if (resp?.ok) {
+          outputDir = newDir;
+          chrome.storage.local.set({ outputDir });
+          sendResponse({ outputDir });
+        } else {
+          sendResponse({ error: resp?.error || 'Cannot write to that directory' });
+        }
+      }).catch((e) => {
+        sendResponse({ error: e.message });
+      });
+    } else if (newDir && transport === 'none') {
+      sendResponse({ error: 'Daemon not running' });
+    } else {
+      outputDir = newDir;
+      chrome.storage.local.set({ outputDir });
+      sendResponse({ outputDir });
+    }
+    return true;
+  }
+
+  if (msg.type === 'SET_IMAGE_DOWNLOAD') {
+    const wasOff = !imageDownload;
+    imageDownload = !!msg.imageDownload;
+    if (wasOff && imageDownload) imageCheckedIds.clear();
+    chrome.storage.local.set({ imageDownload });
+    sendResponse({ imageDownload });
+    return true;
+  }
+
+  if (msg.type === 'TOGGLE_CAPTURE') {
+    captureEnabled = !captureEnabled;
+    saveState();
+    sendResponse({ captureEnabled });
+    return true;
+  }
+
+  if (msg.type === 'CHECK_VIDEO') {
+    const tweet = recentTweets.get(msg.tweetId);
+    if (!tweet || !tweet.media || tweet.media.length === 0) {
+      sendResponse({ hasVideo: false });
+      return true;
+    }
+    const videoMedia = tweet.media.find(m => m.type === 'video' || m.type === 'animated_gif');
+    if (!videoMedia) {
+      sendResponse({ hasVideo: false });
+      return true;
+    }
+    sendResponse({
+      hasVideo: true,
+      tweetUrl: tweet.url || `https://x.com/i/status/${msg.tweetId}`,
+      directUrl: videoMedia.url || null,
+      mediaType: videoMedia.type,
+      durationMs: videoMedia.duration_ms || null,
+      postDate: tweet.created_at || null,
+      activeDownloadId: activeDownloads.get(msg.tweetId) || null,
+    });
+    return true;
+  }
+
+  if (msg.type === 'CHECK_YTDLP') {
+    (async () => {
+      try {
+        const resp = await sendToHost({ type: 'CHECK_YTDLP' });
+        sendResponse(resp || { ok: false, error: 'No transport' });
+      } catch (e) {
+        sendResponse({ ok: false, error: e.message });
+      }
+    })();
+    return true;
+  }
+
+  if (msg.type === 'DOWNLOAD_VIDEO') {
+    (async () => {
+      try {
+        const resp = await sendToHost({
+          type: 'DOWNLOAD_VIDEO',
+          tweetUrl: msg.tweetUrl,
+          directUrl: msg.directUrl,
+          postDate: msg.postDate,
+          outputDir: outputDir || undefined,
+        });
+        // Track active download so popup can resume polling after close/reopen
+        if (resp?.ok && resp.downloadId && msg.tweetId) {
+          activeDownloads.set(msg.tweetId, resp.downloadId);
+        }
+        sendResponse(resp || { ok: false, error: 'No transport' });
+      } catch (e) {
+        sendResponse({ ok: false, error: e.message });
+      }
+    })();
+    return true;
+  }
+
+  if (msg.type === 'DOWNLOAD_STATUS') {
+    (async () => {
+      try {
+        const resp = await sendToHost({
+          type: 'DOWNLOAD_STATUS',
+          downloadId: msg.downloadId,
+        });
+        // Clean up finished downloads from active map. 'unknown' means the
+        // daemon restarted and lost the download — stop resuming it.
+        if (resp?.status === 'done' || resp?.status === 'error' || resp?.status === 'unknown') {
+          for (const [tid, did] of activeDownloads) {
+            if (did === msg.downloadId) { activeDownloads.delete(tid); break; }
+          }
+        }
+        sendResponse(resp || { ok: false, error: 'No transport' });
+      } catch (e) {
+        sendResponse({ ok: false, error: e.message });
+      }
+    })();
+    return true;
+  }
+});
+
+// --- Init ---
+
+if (typeof chrome.storage.session?.setAccessLevel === 'function') {
+  chrome.storage.session.setAccessLevel({ accessLevel: 'TRUSTED_AND_UNTRUSTED_CONTEXTS' });
+}
+
+// Periodic pool flush that survives service-worker sleep.
+chrome.alarms.create('xtap-pool-flush', { periodInMinutes: 1 });
+chrome.alarms.onAlarm.addListener((alarm) => {
+  if (alarm.name === 'xtap-pool-flush') {
+    ready.then(async () => {
+      await poolFlush();
+      await recoverStagedPayloads();
+    }).catch(error => console.error('[xTap] Pool recovery failed:', error.message));
+  }
+});
+
+// Graceful degradation: if restoreState fails (e.g. storage unavailable), continue
+// with defaults so the extension still captures tweets.
+restoreState().catch((error) => {
+  console.error('[xTap] Failed to restore state:', error);
+}).then(async () => {
+  await initPoolSync();
+  await recoverStagedPayloads();
+  await initTransport();
+  readyResolve();
+  updateBadge();
+  // Deliver anything restored or recovered before MV3 idles the worker.
+  if (buffer.length > 0 || logBuffer.length > 0) {
+    if (buffer.length > 0) ensureFlushAlarm();
+    flush();
+  }
+  function scheduleNextFlush() {
+    const jitter = Math.random() * FLUSH_INTERVAL_MS * 0.5;
+    flushTimer = setTimeout(() => { scheduledFlush(); scheduleNextFlush(); }, FLUSH_INTERVAL_MS + jitter);
+  }
+  scheduleNextFlush();
+  const seenStorageLabel = (isDevMode && hasSessionStorage) ? 'session' : 'local';
+  console.log(`[xTap] Service worker started (${isDevMode ? 'dev' : 'production'} mode, seenIds in ${seenStorageLabel} storage)`);
+});

extension/cutover.html

@@ -0,0 +1,12 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <title>Cut over xTap</title>
+  </head>
+  <body>
+    <p>Ending active xTap runs and reloading…</p>
+    <script src="cutover.js" type="module"></script>
+  </body>
+</html>

extension/cutover.js

@@ -0,0 +1,10 @@
+import { ScrapeReceiptStore } from './lib/scrape-receipts.js';
+
+try {
+  const store = new ScrapeReceiptStore();
+  await store.beginCutover(Date.now());
+  await store.close();
+  chrome.runtime.reload();
+} catch {
+  document.body.textContent = 'xTap cutover failed.';
+}

extension/debug.css

@@ -0,0 +1,232 @@
+* {
+  margin: 0;
+  padding: 0;
+  box-sizing: border-box;
+}
+
+body {
+  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
+  background: #15202b;
+  color: #e7e9ea;
+  padding: 24px;
+}
+
+.container {
+  max-width: 960px;
+  margin: 0 auto;
+}
+
+h1 {
+  font-size: 20px;
+  font-weight: 700;
+  margin-bottom: 20px;
+  color: #1d9bf0;
+}
+
+h2 {
+  font-size: 14px;
+  font-weight: 600;
+  margin-bottom: 12px;
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+}
+
+.panel {
+  background: #192734;
+  border: 1px solid #2f3336;
+  border-radius: 8px;
+  padding: 16px;
+  margin-bottom: 16px;
+}
+
+.panel-actions {
+  display: flex;
+  align-items: center;
+  gap: 12px;
+  font-weight: 400;
+}
+
+.auto-scroll-label {
+  font-size: 12px;
+  color: #8b98a5;
+  cursor: pointer;
+  display: flex;
+  align-items: center;
+  gap: 4px;
+}
+
+.auto-scroll-label input {
+  accent-color: #1d9bf0;
+}
+
+.small-btn {
+  padding: 3px 10px;
+  font-size: 11px;
+  border: 1px solid #2f3336;
+  border-radius: 4px;
+  background: transparent;
+  color: #8b98a5;
+  cursor: pointer;
+}
+
+.small-btn:hover {
+  background: #2f3336;
+  color: #e7e9ea;
+}
+
+/* Health table */
+.health-table {
+  width: 100%;
+  font-size: 13px;
+}
+
+.health-table td {
+  padding: 4px 0;
+}
+
+.health-table .label {
+  color: #8b98a5;
+  width: 140px;
+}
+
+.status-connected { color: #4ade80; }
+.status-disconnected { color: #facc15; }
+.status-error { color: #f87171; }
+
+.toggle-row {
+  margin-top: 12px;
+  font-size: 13px;
+  color: #8b98a5;
+}
+
+.toggle-row label {
+  display: flex;
+  align-items: center;
+  gap: 8px;
+  cursor: pointer;
+}
+
+.toggle-row input[type="checkbox"] {
+  accent-color: #1d9bf0;
+}
+
+/* Discovered endpoints */
+.discovered-list {
+  font-family: "SF Mono", "Menlo", "Monaco", "Consolas", monospace;
+  font-size: 12px;
+  color: #8b98a5;
+}
+
+/* Events table */
+.events-wrap {
+  max-height: 400px;
+  overflow-y: auto;
+  border: 1px solid #2f3336;
+  border-radius: 4px;
+}
+
+.events-table {
+  width: 100%;
+  border-collapse: collapse;
+  font-size: 12px;
+  font-family: "SF Mono", "Menlo", "Monaco", "Consolas", monospace;
+}
+
+.events-table thead {
+  position: sticky;
+  top: 0;
+  background: #1e2a36;
+}
+
+.events-table th {
+  text-align: left;
+  padding: 6px 8px;
+  color: #8b98a5;
+  font-weight: 600;
+  border-bottom: 1px solid #2f3336;
+}
+
+.events-table td {
+  padding: 4px 8px;
+  border-bottom: 1px solid #1e2a36;
+}
+
+.events-table tr:hover {
+  background: #1e2a36;
+}
+
+.status-ACCEPTED { color: #4ade80; }
+.status-IMAGE_BACKFILL { color: #60a5fa; }
+.status-DEDUPLICATED { color: #8b98a5; }
+.status-PARSER_ERROR { color: #f87171; }
+.status-DROPPED_OVERSIZED { color: #f87171; }
+
+/* Sandbox */
+.sandbox-row {
+  margin-bottom: 10px;
+}
+
+.sandbox-row label {
+  display: block;
+  font-size: 12px;
+  color: #8b98a5;
+  margin-bottom: 4px;
+}
+
+.sandbox-row input,
+.sandbox-row textarea {
+  width: 100%;
+  padding: 8px;
+  border: 1px solid #2f3336;
+  border-radius: 6px;
+  background: #1e2a36;
+  color: #e7e9ea;
+  font-size: 12px;
+  font-family: "SF Mono", "Menlo", "Monaco", "Consolas", monospace;
+  outline: none;
+  resize: vertical;
+}
+
+.sandbox-row input:focus,
+.sandbox-row textarea:focus {
+  border-color: #1d9bf0;
+}
+
+.action-btn {
+  padding: 8px 16px;
+  border: none;
+  border-radius: 6px;
+  background: #1d9bf0;
+  color: #fff;
+  font-size: 13px;
+  font-weight: 600;
+  cursor: pointer;
+}
+
+.action-btn:hover {
+  opacity: 0.85;
+}
+
+.sandbox-output {
+  margin-top: 12px;
+  padding: 12px;
+  background: #1e2a36;
+  border: 1px solid #2f3336;
+  border-radius: 6px;
+  font-size: 12px;
+  font-family: "SF Mono", "Menlo", "Monaco", "Consolas", monospace;
+  white-space: pre-wrap;
+  word-break: break-all;
+  max-height: 400px;
+  overflow-y: auto;
+  display: none;
+}
+
+.sandbox-output.visible {
+  display: block;
+}
+
+.sandbox-output.error {
+  color: #f87171;
+}

extension/debug.html

@@ -0,0 +1,76 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <title>xTap Debug Dashboard</title>
+  <link rel="stylesheet" href="debug.css">
+</head>
+<body>
+  <div class="container">
+    <h1>xTap Debug Dashboard</h1>
+
+    <section class="panel">
+      <h2>Transport Health</h2>
+      <table class="health-table">
+        <tr><td class="label">Transport</td><td id="h-transport">—</td></tr>
+        <tr><td class="label">Status</td><td id="h-status">—</td></tr>
+        <tr><td class="label">Capture</td><td id="h-capture">—</td></tr>
+        <tr><td class="label">Session count</td><td id="h-session">—</td></tr>
+        <tr><td class="label">All-time count</td><td id="h-alltime">—</td></tr>
+        <tr><td class="label">Buffer size</td><td id="h-buffer">—</td></tr>
+        <tr id="h-error-row" style="display: none;"><td class="label">Error</td><td id="h-error" class="status-error">—</td></tr>
+      </table>
+      <div class="toggle-row">
+        <label><input type="checkbox" id="debug-toggle"> Debug logging to file</label>
+      </div>
+      <div class="toggle-row">
+        <label><input type="checkbox" id="verbose-toggle"> Discovery mode (verbose endpoint logging)</label>
+      </div>
+    </section>
+
+    <section class="panel" id="discovered-section" style="display: none;">
+      <h2>Discovered Endpoints</h2>
+      <div id="discovered-list" class="discovered-list"></div>
+    </section>
+
+    <section class="panel">
+      <h2>
+        Capture Events
+        <span class="panel-actions">
+          <label class="auto-scroll-label"><input type="checkbox" id="auto-scroll" checked> Auto-scroll</label>
+          <button id="clear-events" class="small-btn">Clear</button>
+        </span>
+      </h2>
+      <div class="events-wrap">
+        <table class="events-table">
+          <thead>
+            <tr>
+              <th>Time</th>
+              <th>Endpoint</th>
+              <th>Tweet ID</th>
+              <th>Status</th>
+              <th>Reason</th>
+            </tr>
+          </thead>
+          <tbody id="events-body"></tbody>
+        </table>
+      </div>
+    </section>
+
+    <section class="panel">
+      <h2>Parser Sandbox</h2>
+      <div class="sandbox-row">
+        <label for="sandbox-endpoint">Endpoint</label>
+        <input type="text" id="sandbox-endpoint" placeholder="e.g. HomeTimeline" spellcheck="false">
+      </div>
+      <div class="sandbox-row">
+        <label for="sandbox-json">Response JSON</label>
+        <textarea id="sandbox-json" rows="10" placeholder="Paste raw GraphQL JSON here..." spellcheck="false"></textarea>
+      </div>
+      <button id="sandbox-run" class="action-btn">Run Parser</button>
+      <pre id="sandbox-output" class="sandbox-output"></pre>
+    </section>
+  </div>
+  <script type="module" src="debug.js"></script>
+</body>
+</html>

extension/debug.js

@@ -0,0 +1,168 @@
+import { extractTweets } from './lib/tweet-parser.js';
+
+// --- Health polling ---
+
+const hTransport = document.getElementById('h-transport');
+const hStatus = document.getElementById('h-status');
+const hCapture = document.getElementById('h-capture');
+const hSession = document.getElementById('h-session');
+const hAlltime = document.getElementById('h-alltime');
+const hBuffer = document.getElementById('h-buffer');
+const hErrorRow = document.getElementById('h-error-row');
+const hError = document.getElementById('h-error');
+const debugToggle = document.getElementById('debug-toggle');
+const verboseToggle = document.getElementById('verbose-toggle');
+const discoveredSection = document.getElementById('discovered-section');
+const discoveredList = document.getElementById('discovered-list');
+
+function refreshHealth() {
+  chrome.runtime.sendMessage({ type: 'GET_STATUS' }, (resp) => {
+    if (!resp) return;
+    hTransport.textContent = resp.transport || 'none';
+    hStatus.textContent = resp.connected ? 'Connected' : 'Disconnected';
+    hStatus.className = resp.connected ? 'status-connected' : 'status-disconnected';
+    hCapture.textContent = resp.captureEnabled ? 'Enabled' : 'Paused';
+    hSession.textContent = resp.sessionCount.toLocaleString();
+    hAlltime.textContent = resp.allTimeCount.toLocaleString();
+    hBuffer.textContent = resp.buffered;
+    if (resp.transportError) {
+      hErrorRow.style.display = '';
+      hError.textContent = resp.transportError;
+    } else {
+      hErrorRow.style.display = 'none';
+    }
+    debugToggle.checked = !!resp.debugLogging;
+    verboseToggle.checked = !!resp.verboseLogging;
+    // Discovered endpoints
+    if (resp.discoveredEndpoints && resp.discoveredEndpoints.length > 0) {
+      discoveredSection.style.display = '';
+      discoveredList.textContent = resp.discoveredEndpoints.join(', ');
+    } else {
+      discoveredSection.style.display = 'none';
+    }
+  });
+}
+
+refreshHealth();
+setInterval(refreshHealth, 5000);
+
+debugToggle.addEventListener('change', () => {
+  chrome.runtime.sendMessage({ type: 'SET_DEBUG', debugLogging: debugToggle.checked }, () => {
+    refreshHealth();
+  });
+});
+
+verboseToggle.addEventListener('change', () => {
+  chrome.runtime.sendMessage({ type: 'SET_VERBOSE', verboseLogging: verboseToggle.checked }, () => {
+    refreshHealth();
+  });
+});
+
+// --- Capture events ---
+
+const eventsBody = document.getElementById('events-body');
+const autoScrollCheckbox = document.getElementById('auto-scroll');
+const clearBtn = document.getElementById('clear-events');
+const eventsWrap = document.querySelector('.events-wrap');
+const traceStorage = chrome.storage.session || chrome.storage.local;
+const traceArea = chrome.storage.session ? 'session' : 'local';
+
+let renderedCount = 0;
+
+function formatTime(ts) {
+  const d = new Date(ts);
+  return d.toLocaleTimeString('en-GB', { hour12: false }) + '.' + String(d.getMilliseconds()).padStart(3, '0');
+}
+
+function renderEvents(events) {
+  eventsBody.innerHTML = '';
+  renderedCount = 0;
+  for (const ev of events) {
+    appendEventRow(ev);
+  }
+}
+
+function appendEventRow(ev) {
+  const tr = document.createElement('tr');
+  const cells = [formatTime(ev.timestamp), ev.endpoint, ev.tweetId || '—', ev.status, ev.reason || ''];
+  for (const text of cells) {
+    const td = document.createElement('td');
+    td.textContent = text;
+    tr.appendChild(td);
+  }
+  tr.children[3].className = `status-${ev.status}`;
+  eventsBody.appendChild(tr);
+  renderedCount++;
+  if (autoScrollCheckbox.checked) {
+    eventsWrap.scrollTop = eventsWrap.scrollHeight;
+  }
+}
+
+// Load initial events
+traceStorage.get(['lastEvents'], (result) => {
+  if (result.lastEvents) renderEvents(result.lastEvents);
+});
+
+// Live updates
+chrome.storage.onChanged.addListener((changes, area) => {
+  if (area === traceArea && changes.lastEvents) {
+    const events = changes.lastEvents.newValue || [];
+    // Re-render if the new batch has fewer (was trimmed) or is a fresh set
+    if (events.length <= renderedCount || events.length === 0) {
+      renderEvents(events);
+    } else {
+      // Append only new events
+      const newEvents = events.slice(renderedCount);
+      for (const ev of newEvents) {
+        appendEventRow(ev);
+      }
+    }
+  }
+});
+
+clearBtn.addEventListener('click', () => {
+  eventsBody.innerHTML = '';
+  renderedCount = 0;
+  traceStorage.set({ lastEvents: [] });
+});
+
+// --- Parser sandbox ---
+
+const sandboxEndpoint = document.getElementById('sandbox-endpoint');
+const sandboxJson = document.getElementById('sandbox-json');
+const sandboxRun = document.getElementById('sandbox-run');
+const sandboxOutput = document.getElementById('sandbox-output');
+
+sandboxRun.addEventListener('click', () => {
+  const endpoint = sandboxEndpoint.value.trim() || 'unknown';
+  const raw = sandboxJson.value.trim();
+  sandboxOutput.classList.add('visible');
+  sandboxOutput.classList.remove('error');
+
+  if (!raw) {
+    sandboxOutput.classList.add('error');
+    sandboxOutput.textContent = 'Paste JSON above first.';
+    return;
+  }
+
+  let data;
+  try {
+    data = JSON.parse(raw);
+  } catch (e) {
+    sandboxOutput.classList.add('error');
+    sandboxOutput.textContent = `JSON parse error: ${e.message}`;
+    return;
+  }
+
+  try {
+    const tweets = extractTweets(endpoint, data);
+    if (tweets.length === 0) {
+      sandboxOutput.textContent = 'No tweets extracted.';
+    } else {
+      sandboxOutput.textContent = `${tweets.length} tweet(s) extracted:\n\n${JSON.stringify(tweets, null, 2)}`;
+    }
+  } catch (e) {
+    sandboxOutput.classList.add('error');
+    sandboxOutput.textContent = `Parser error: ${e.message}\n\n${e.stack}`;
+  }
+});

extension/lib/dedup.js

@@ -0,0 +1,60 @@
+/**
+ * Tweet deduplication logic.
+ *
+ * Extracted so both background.js and tests can use the same code path.
+ */
+
+/**
+ * Check whether a tweet is new and track it in seenIds.
+ *
+ * Returns true  → tweet is new, should be enqueued.
+ * Returns false → tweet is a duplicate, should be skipped.
+ *
+ * - Tweets without an id are always new (and not tracked).
+ * - Article tweets may bypass tweet-ID dedup once — they may enrich a
+ *   previously captured non-article row, but repeated full article payloads
+ *   should not be appended forever.
+ * - When `options.imageBackfill` is set and the tweet has photo media,
+ *   a duplicate is also let through once per session (tracked in
+ *   `options.imageCheckedIds`) so the daemon can download images that
+ *   were skipped when image-download was off at original capture time.
+ *   The downloader's per-file `os.path.exists` check makes repeats cheap.
+ */
+export function dedupTweet(tweet, seenIds, options = {}) {
+  const { imageBackfill = false, imageCheckedIds = null } = options;
+
+  const isDup = !!(tweet.id && seenIds.has(tweet.id));
+  const isArticle = !!tweet.is_article;
+  const articleKey = isArticle && tweet.id ? articleSeenKey(tweet.id) : null;
+  const isArticleDup = !!(articleKey && seenIds.has(articleKey));
+  const wantsImageBackfill = (
+    imageBackfill &&
+    !!tweet.id &&
+    imageCheckedIds &&
+    !imageCheckedIds.has(tweet.id) &&
+    hasPhotoMedia(tweet)
+  );
+
+  if (isDup && (!isArticle || isArticleDup) && !wantsImageBackfill) {
+    return false;
+  }
+
+  if (tweet.id) seenIds.add(tweet.id);
+  if (articleKey) seenIds.add(articleKey);
+  if (wantsImageBackfill) imageCheckedIds.add(tweet.id);
+
+  return true;
+}
+
+function articleSeenKey(id) {
+  return `article:${id}`;
+}
+
+function hasPhotoMedia(tweet) {
+  const media = tweet.media;
+  if (!Array.isArray(media)) return false;
+  for (const m of media) {
+    if (m && m.type === 'photo' && m.url) return true;
+  }
+  return false;
+}

extension/lib/graphql-capture.js

@@ -0,0 +1,214 @@
+const DEBUGGER_PROTOCOL_VERSION = '1.3';
+const GRAPHQL_PATH = '/i/api/graphql/';
+const MAX_PENDING_RESPONSES = 10_000;
+const X_URL_PATTERNS = ['*://*.x.com/*', '*://*.twitter.com/*'];
+
+export class GraphqlCapture {
+  constructor({
+    debuggerApi = globalThis.chrome?.debugger,
+    tabs = globalThis.chrome?.tabs,
+    onResponse,
+    logger = console,
+  } = {}) {
+    if (!debuggerApi || !tabs) throw new Error('Chrome debugger capture is unavailable');
+    if (typeof onResponse !== 'function') throw new Error('GraphQL response handler is required');
+    this.debuggerApi = debuggerApi;
+    this.tabs = tabs;
+    this.onResponse = onResponse;
+    this.logger = logger;
+    this.attachedTabs = new Set();
+    this.attachmentRevisions = new Map();
+    this.attachments = new Map();
+    this.pendingResponses = new Map();
+  }
+
+  attach() {
+    this.debuggerApi.onEvent.addListener((source, method, params) => {
+      void this.handleEvent(source, method, params).catch((error) => {
+        this.logger.error('[xTap] Passive GraphQL capture failed:', error);
+      });
+    });
+    this.debuggerApi.onDetach.addListener((source) => {
+      if (source.tabId === undefined) return;
+      this.attachedTabs.delete(source.tabId);
+      this.attachments.delete(source.tabId);
+      this.deletePendingForTab(source.tabId);
+    });
+    this.tabs.onUpdated.addListener((tabId, changeInfo, tab) => {
+      const url = changeInfo.url ?? tab.url;
+      if (isXUrl(url)) {
+        void this.ensureAttached(tabId);
+      } else if (changeInfo.url || changeInfo.status === 'loading') {
+        void this.detachTab(tabId);
+      }
+    });
+    this.tabs.onRemoved.addListener((tabId) => {
+      void this.detachTab(tabId);
+    });
+    void this.attachExistingTabs();
+  }
+
+  async attachExistingTabs() {
+    const tabs = await this.tabs.query({ url: X_URL_PATTERNS });
+    await Promise.all(
+      tabs
+        .map((tab) => tab.id)
+        .filter((tabId) => Number.isSafeInteger(tabId))
+        .map((tabId) => this.ensureAttached(tabId)),
+    );
+  }
+
+  async ensureAttached(tabId) {
+    if (!Number.isSafeInteger(tabId) || tabId < 0) return false;
+    if (this.attachedTabs.has(tabId)) return true;
+    const pending = this.attachments.get(tabId);
+    if (pending) return pending;
+    const revision = this.attachmentRevisions.get(tabId) ?? 0;
+    const attaching = this.attachTab(tabId, revision).finally(() => {
+      if (this.attachments.get(tabId) === attaching) this.attachments.delete(tabId);
+    });
+    this.attachments.set(tabId, attaching);
+    return attaching;
+  }
+
+  async attachTab(tabId, revision) {
+    let attachedHere = false;
+    try {
+      await this.debuggerApi.attach({ tabId }, DEBUGGER_PROTOCOL_VERSION);
+      attachedHere = true;
+    } catch (error) {
+      if (!isAlreadyAttachedError(error)) {
+        this.logger.warn(`[xTap] Could not observe X tab ${tabId}: ${errorMessage(error)}`);
+        return false;
+      }
+    }
+    try {
+      await this.debuggerApi.sendCommand({ tabId }, 'Network.enable');
+      if ((this.attachmentRevisions.get(tabId) ?? 0) !== revision) {
+        if (attachedHere) await this.safeDetach(tabId);
+        return false;
+      }
+      this.attachedTabs.add(tabId);
+      return true;
+    } catch (error) {
+      if (attachedHere) await this.safeDetach(tabId);
+      this.logger.warn(`[xTap] Could not enable capture for X tab ${tabId}: ${errorMessage(error)}`);
+      return false;
+    }
+  }
+
+  async detachTab(tabId) {
+    const revision = (this.attachmentRevisions.get(tabId) ?? 0) + 1;
+    this.attachmentRevisions.set(tabId, revision);
+    const pending = this.attachments.get(tabId);
+    if (pending) await pending;
+    this.deletePendingForTab(tabId);
+    if (!this.attachedTabs.delete(tabId)) return;
+    await this.safeDetach(tabId);
+  }
+
+  async safeDetach(tabId) {
+    try {
+      await this.debuggerApi.detach({ tabId });
+    } catch {
+      // Chrome may have already detached a closed or reassigned tab.
+    }
+  }
+
+  async handleEvent(source, method, params) {
+    const tabId = source.tabId;
+    if (!Number.isSafeInteger(tabId)) return;
+    const requestId = typeof params?.requestId === 'string' ? params.requestId : undefined;
+    if (!requestId) return;
+    const key = responseKey(tabId, requestId);
+
+    if (method === 'Network.responseReceived') {
+      const url = params?.response?.url;
+      const endpoint = extractGraphqlEndpoint(url);
+      if (endpoint) this.rememberPending(key, { endpoint, requestId, tabId, url });
+      return;
+    }
+    if (method === 'Network.loadingFailed') {
+      this.pendingResponses.delete(key);
+      return;
+    }
+    if (method !== 'Network.loadingFinished') return;
+
+    const pending = this.pendingResponses.get(key);
+    if (!pending) return;
+    this.pendingResponses.delete(key);
+    const body = await this.debuggerApi.sendCommand(
+      { tabId },
+      'Network.getResponseBody',
+      { requestId },
+    );
+    const text = decodeResponseBody(body);
+    const data = JSON.parse(text);
+    await this.onResponse({
+      data,
+      endpoint: pending.endpoint,
+      sourceTabId: tabId,
+      url: pending.url,
+    });
+  }
+
+  rememberPending(key, response) {
+    this.pendingResponses.set(key, response);
+    if (this.pendingResponses.size <= MAX_PENDING_RESPONSES) return;
+    const oldestKey = this.pendingResponses.keys().next().value;
+    if (oldestKey !== undefined) this.pendingResponses.delete(oldestKey);
+  }
+
+  deletePendingForTab(tabId) {
+    const prefix = `${tabId}:`;
+    for (const key of this.pendingResponses.keys()) {
+      if (key.startsWith(prefix)) this.pendingResponses.delete(key);
+    }
+  }
+}
+
+export function extractGraphqlEndpoint(url) {
+  if (typeof url !== 'string') return undefined;
+  try {
+    const parsed = new URL(url);
+    if (!isXHostname(parsed.hostname)) return undefined;
+    if (!parsed.pathname.includes(GRAPHQL_PATH)) return undefined;
+    const parts = parsed.pathname.split('/');
+    const graphqlIndex = parts.indexOf('graphql');
+    const endpoint = graphqlIndex >= 0 ? parts[graphqlIndex + 2] : undefined;
+    return typeof endpoint === 'string' && endpoint.length > 0 ? endpoint : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
+function decodeResponseBody(value) {
+  if (!value || typeof value.body !== 'string') throw new Error('GraphQL response body is unavailable');
+  return value.base64Encoded === true ? atob(value.body) : value.body;
+}
+
+function isXUrl(value) {
+  if (typeof value !== 'string') return false;
+  try {
+    return isXHostname(new URL(value).hostname);
+  } catch {
+    return false;
+  }
+}
+
+function isXHostname(hostname) {
+  return hostname === 'x.com' || hostname.endsWith('.x.com') ||
+    hostname === 'twitter.com' || hostname.endsWith('.twitter.com');
+}
+
+function isAlreadyAttachedError(error) {
+  return errorMessage(error).toLowerCase().includes('another debugger is already attached');
+}
+
+function errorMessage(error) {
+  return error instanceof Error ? error.message : String(error);
+}
+
+function responseKey(tabId, requestId) {
+  return `${tabId}:${requestId}`;
+}

extension/lib/observations.js

@@ -0,0 +1,59 @@
+// Observation delivery is independent of the unique-post export filter.
+// Hashes exclude transport fields. Retrying a saved sample preserves its time.
+const SAMPLE_INTERVAL_MS = 5 * 60_000;
+const MAX_SAMPLES = 50_000;
+const OBSERVATION_FIELDS = new Set([
+  'captured_at', 'pooled_at', 'contributed_by', 'metrics', 'source_endpoint',
+  'observation_id', '__xtap_image_backfill',
+]);
+
+export function exactCounter(value) {
+  if (typeof value === 'string' && /^(0|[1-9]\d*)$/.test(value)) {
+    const converted = Number(value);
+    return Number.isSafeInteger(converted) ? converted : null;
+  }
+  return typeof value === 'number' && Number.isSafeInteger(value) && value >= 0 ? value : null;
+}
+
+function canonical(value) {
+  if (value === null || typeof value !== 'object') return JSON.stringify(value);
+  if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
+  return `{${Object.entries(value).filter(([, item]) => item !== undefined)
+    .sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0)
+    .map(([key, item]) => `${JSON.stringify(key)}:${canonical(item)}`).join(',')}}`;
+}
+async function digest(value) {
+  const bytes = new TextEncoder().encode(canonical(value));
+  const hash = await globalThis.crypto.subtle.digest('SHA-256', bytes);
+  return [...new Uint8Array(hash)].map(byte => byte.toString(16).padStart(2, '0')).join('');
+}
+
+/** Pure preparation: callers commit samples and queued records in one write. */
+export async function prepareObservations(tweets, previous = {}) {
+  const samples = new Map(Object.entries(previous));
+  const accepted = [];
+  let sampled = 0;
+  for (const tweet of tweets) {
+    const time = Date.parse(tweet.captured_at);
+    if (typeof tweet.id !== 'string' || !tweet.id || !Number.isFinite(time)) throw new Error('observation requires a post ID and time');
+    const content = Object.fromEntries(Object.entries(tweet).filter(([key]) => !OBSERVATION_FIELDS.has(key)));
+    if (tweet.author && typeof tweet.author === 'object') {
+      content.author = Object.fromEntries(Object.entries(tweet.author).filter(([key]) => key !== 'follower_count'));
+    }
+    const hash = await digest(content);
+    const before = samples.get(tweet.id);
+    // A later visit in another interval is useful even when every count is unchanged.
+    // Deliver late samples too; time ordering must not erase history.
+    if (before?.content === hash && Math.floor(time / SAMPLE_INTERVAL_MS) === Math.floor(before.time / SAMPLE_INTERVAL_MS)) {
+      sampled++;
+      continue;
+    }
+    const captured_at = new Date(time).toISOString();
+    const observation_id = await digest({ id: tweet.id, captured_at, content: hash, metrics: tweet.metrics ?? null });
+    accepted.push({ ...tweet, captured_at, observation_id });
+    if (!before || time >= before.time) samples.set(tweet.id, { time, content: hash });
+  }
+  // Eviction only permits extra sampling; it never deletes queued observations.
+  const retained = [...samples].sort((a, b) => b[1].time - a[1].time).slice(0, MAX_SAMPLES);
+  return { accepted, samples: Object.fromEntries(retained), sampled };
+}

extension/lib/pool-sync.js

@@ -0,0 +1,286 @@
+// xtap-pool — background sync of captured tweets to the shared pool Space.
+//
+// Additive to xTap's local saving: tweets are appended to a persistent queue
+// in chrome.storage.local and flushed in batches to POST <poolUrl>/api/ingest
+// with the user's pool token. Delivery is at-least-once; the Space dedups.
+
+import { prepareObservations } from './observations.js';
+
+const QUEUE_KEY = 'poolQueue';
+const SAMPLES_KEY = 'poolSamples';
+const REJECTIONS_KEY = 'poolRejections';
+const MAX_REJECTIONS = 500;
+const CONFIG_KEYS = ['poolUrl', 'poolToken', 'poolUsername', 'poolPaused', 'poolStats'];
+const MAX_QUEUE = 5000;
+const MAX_BATCH = 500;
+const FLUSH_DEBOUNCE_MS = 20_000;
+const BACKOFF_BASE_MS = 30_000;
+const BACKOFF_MAX_MS = 15 * 60_000;
+
+// Set at build/packaging time so friends get a working default without setup.
+export const DEFAULT_POOL_URL = 'https://osolmaz-xtap-pool.hf.space';
+
+let queue = [];
+let samples = {};
+let rejections = [];
+let queueWrites = Promise.resolve();
+let config = {
+  poolUrl: DEFAULT_POOL_URL,
+  poolToken: '',
+  poolUsername: '',
+  poolPaused: false,
+};
+let stats = { synced: 0, observations: 0, sampled: 0, blocked: 0, lastError: null, lastSyncAt: null };
+let flushTimer = null;
+let backoffMs = 0;
+let flushing = false;
+
+function storage() {
+  return globalThis.chrome.storage.local;
+}
+
+function storageGet(keys) {
+  return new Promise((resolve, reject) => storage().get(keys, result => {
+    const error = globalThis.chrome.runtime?.lastError;
+    if (error) reject(new Error(error.message || 'pool storage read failed'));
+    else resolve(result);
+  }));
+}
+
+function storageSet(items) {
+  return new Promise((resolve, reject) => storage().set(items, () => {
+    const error = globalThis.chrome.runtime?.lastError;
+    if (error) reject(new Error(error.message || 'pool storage write failed'));
+    else resolve();
+  }));
+}
+
+function withQueueWrite(operation) {
+  const result = queueWrites.then(operation, operation);
+  queueWrites = result.catch(() => {});
+  return result;
+}
+
+export async function initPoolSync() {
+  const saved = await storageGet([QUEUE_KEY, SAMPLES_KEY, REJECTIONS_KEY, ...CONFIG_KEYS]);
+  queue = Array.isArray(saved[QUEUE_KEY]) ? saved[QUEUE_KEY] : [];
+  samples = saved[SAMPLES_KEY] && typeof saved[SAMPLES_KEY] === 'object' ? saved[SAMPLES_KEY] : {};
+  rejections = Array.isArray(saved[REJECTIONS_KEY]) ? saved[REJECTIONS_KEY] : [];
+  if (typeof saved.poolUrl === 'string' && saved.poolUrl) config.poolUrl = saved.poolUrl;
+  if (typeof saved.poolToken === 'string') config.poolToken = saved.poolToken;
+  if (typeof saved.poolUsername === 'string') config.poolUsername = saved.poolUsername;
+  config.poolPaused = saved.poolPaused === true;
+  if (saved.poolStats && typeof saved.poolStats === 'object') stats = { ...stats, ...saved.poolStats };
+  if (queue.length > 0) scheduleFlush(0);
+}
+
+async function persistStats() {
+  await storageSet({ poolStats: stats });
+}
+
+/** Persist samples and admission state together. Failure leaves staged work pending. */
+export function poolEnqueue(tweets) {
+  if (!Array.isArray(tweets) || tweets.length === 0) return Promise.resolve();
+  return withQueueWrite(async () => {
+    const prepared = await prepareObservations(tweets, samples);
+    if (queue.length + prepared.accepted.length > MAX_QUEUE) {
+      stats = { ...stats, blocked: stats.blocked + 1, lastError: 'pool queue full — staged observations are waiting' };
+      await persistStats();
+      scheduleFlush(0);
+      throw new Error(stats.lastError);
+    }
+    const nextQueue = [...queue, ...prepared.accepted];
+    const nextStats = { ...stats, observations: stats.observations + prepared.accepted.length, sampled: stats.sampled + prepared.sampled };
+    await storageSet({ [QUEUE_KEY]: nextQueue, [SAMPLES_KEY]: prepared.samples, poolStats: nextStats });
+    queue = nextQueue;
+    samples = prepared.samples;
+    stats = nextStats;
+    if (queue.length > 0) scheduleFlush(queue.length >= MAX_BATCH ? 0 : FLUSH_DEBOUNCE_MS);
+  });
+}
+
+function scheduleFlush(delayMs) {
+  if (flushTimer) {
+    if (delayMs > 0) return;
+    // An immediate flush supersedes a pending debounce timer.
+    clearTimeout(flushTimer);
+  }
+  flushTimer = setTimeout(() => {
+    flushTimer = null;
+    poolFlush().catch(error => {
+      stats.lastError = `pool persistence failed: ${error.message}`;
+      scheduleFlush(BACKOFF_BASE_MS);
+    });
+  }, delayMs);
+}
+
+/** Push queued tweets to the Space, batch by batch. Returns when idle. */
+export async function poolFlush() {
+  if (flushing || config.poolPaused || !config.poolToken || queue.length === 0) return;
+  flushing = true;
+  try {
+    while (queue.length > 0) {
+      const batch = queue.slice(0, MAX_BATCH);
+      const acknowledgment = await sendBatch(batch);
+      if (!acknowledgment) break;
+      await withQueueWrite(async () => {
+        const nextQueue = queue.slice(batch.length);
+        const nextRejections = [...rejections, ...acknowledgment.rejected.map(item => ({ tweet: batch[item.index], reason: item.reason }))];
+        if (nextRejections.length > MAX_REJECTIONS) throw new Error('pool rejection archive full — queued observations retained');
+        const nextStats = { ...stats, synced: stats.synced + batch.length - acknowledgment.rejected.length, lastError: nextRejections.length ? `${nextRejections.length} rejected observations saved for inspection` : null, lastSyncAt: new Date().toISOString() };
+        await storageSet({ [QUEUE_KEY]: nextQueue, [REJECTIONS_KEY]: nextRejections, poolStats: nextStats });
+        queue = nextQueue;
+        rejections = nextRejections;
+        stats = nextStats;
+        backoffMs = 0;
+      });
+    }
+  } finally {
+    flushing = false;
+  }
+}
+
+async function sendBatch(batch) {
+  let response;
+  try {
+    response = await globalThis.fetch(`${config.poolUrl}/api/ingest`, {
+      method: 'POST',
+      headers: {
+        'content-type': 'application/json',
+        authorization: `Bearer ${config.poolToken}`,
+      },
+      body: JSON.stringify({ tweets: batch }),
+    });
+  } catch (e) {
+    return retryLater(`network error: ${e && e.message ? e.message : e}`);
+  }
+  if (response.status === 401) {
+    // Token expired or revoked — needs a manual reconnect, retrying won't help.
+    await withQueueWrite(async () => {
+      stats.lastError = 'pool token rejected — reconnect from the popup';
+      await persistStats();
+    });
+    return false;
+  }
+  if (!response.ok) {
+    return retryLater(`pool responded ${response.status}`);
+  }
+  const body = await response.json().catch(() => null);
+  if (!body || !Number.isSafeInteger(body.added) || body.added < 0 ||
+      !Number.isSafeInteger(body.duplicates) || body.duplicates < 0 ||
+      !Array.isArray(body.rejected) ||
+      !body.rejected.every(item => item && Number.isSafeInteger(item.index) && item.index >= 0 && item.index < batch.length && typeof item.reason === 'string' && item.reason.length > 0) ||
+      new Set(body.rejected.map(item => item.index)).size !== body.rejected.length ||
+      body.added + body.duplicates + body.rejected.length !== batch.length) {
+    return retryLater('pool did not acknowledge the complete observation batch');
+  }
+  return { rejected: body.rejected };
+}
+
+async function retryLater(message) {
+  backoffMs = Math.min(backoffMs > 0 ? backoffMs * 2 : BACKOFF_BASE_MS, BACKOFF_MAX_MS);
+  await withQueueWrite(async () => {
+    stats.lastError = `${message} — retrying in ${Math.round(backoffMs / 1000)}s`;
+    await persistStats();
+  });
+  scheduleFlush(backoffMs);
+  return false;
+}
+
+/**
+ * Handle a token handed off by the /connect page content script.
+ *
+ * The token is only accepted when the sender page's origin matches the
+ * configured pool URL; otherwise any malicious HF Space rendering an
+ * `#xtap-pool-token` element could silently repoint the extension and
+ * receive all future captures. Switching pools requires setting the URL
+ * explicitly (options / POOL_SET_CONFIG) before connecting.
+ */
+export async function poolConnect({ token, username }, senderUrl) {
+  if (typeof token !== 'string' || !token) return { ok: false, error: 'missing token' };
+  if (!senderMatchesPool(senderUrl)) {
+    console.warn(`[xtap-pool] refused pool token from unexpected origin: ${senderUrl}`);
+    return { ok: false, error: `refused token from unexpected origin (pool is ${config.poolUrl})` };
+  }
+  config.poolToken = token;
+  if (typeof username === 'string') config.poolUsername = username;
+  stats.lastError = null;
+  await storageSet({
+    poolToken: config.poolToken,
+    poolUsername: config.poolUsername,
+    poolStats: stats,
+  });
+  scheduleFlush(0);
+  return { ok: true, username: config.poolUsername };
+}
+
+function senderMatchesPool(senderUrl) {
+  if (typeof senderUrl !== 'string' || !senderUrl) return false;
+  try {
+    return new URL(senderUrl).origin === new URL(config.poolUrl).origin;
+  } catch {
+    return false;
+  }
+}
+
+export async function poolSetConfig({ url, token }) {
+  if (typeof url === 'string' && url) {
+    const normalized = url.replace(/\/+$/, '');
+    if (normalized !== config.poolUrl) {
+      config.poolUrl = normalized;
+      // A token is only valid for the pool that minted it. Keeping the old
+      // one would leak it (and queued captures) to whatever Space the new
+      // URL points at on the next flush.
+      config.poolToken = '';
+      config.poolUsername = '';
+    }
+  }
+  if (typeof token === 'string' && token) {
+    config.poolToken = token;
+    config.poolUsername = '';
+  }
+  await storageSet({
+    poolUrl: config.poolUrl,
+    poolToken: config.poolToken,
+    poolUsername: config.poolUsername,
+  });
+  if (config.poolToken) scheduleFlush(0);
+}
+
+export async function poolTogglePause() {
+  config.poolPaused = !config.poolPaused;
+  await storageSet({ poolPaused: config.poolPaused });
+  if (!config.poolPaused) scheduleFlush(0);
+  return config.poolPaused;
+}
+
+export function poolStatus() {
+  return {
+    connected: Boolean(config.poolToken),
+    username: config.poolUsername,
+    url: config.poolUrl,
+    paused: config.poolPaused,
+    queued: queue.length,
+    synced: stats.synced,
+    observations: stats.observations,
+    sampled: stats.sampled,
+    blocked: stats.blocked,
+    rejected: rejections.length,
+    lastError: stats.lastError,
+    lastSyncAt: stats.lastSyncAt,
+  };
+}
+
+// Test-only hook: reset module state between node --test cases.
+export function _resetForTests() {
+  queue = [];
+  samples = {};
+  rejections = [];
+  queueWrites = Promise.resolve();
+  config = { poolUrl: DEFAULT_POOL_URL, poolToken: '', poolUsername: '', poolPaused: false };
+  stats = { synced: 0, observations: 0, sampled: 0, blocked: 0, lastError: null, lastSyncAt: null };
+  if (flushTimer) clearTimeout(flushTimer);
+  flushTimer = null;
+  backoffMs = 0;
+  flushing = false;
+}

extension/lib/scrape-bridge.js

@@ -0,0 +1,235 @@
+import {
+  SCRAPE_PORT_NAME,
+  SCRAPE_PROTOCOL_VERSION,
+  ScrapeReceiptError,
+  ScrapeReceiptStore,
+} from './scrape-receipts.js';
+
+export const SCROLLER_EXTENSION_ID = 'aahdialpkbjlbfjkpamfclbnlbinekal';
+
+const CAPTURE_ATTACH_ATTEMPTS = 5;
+const CAPTURE_ATTACH_RETRY_MS = 200;
+
+export class ScrapeReceiptBridge {
+  constructor({
+    allowedExtensionId = SCROLLER_EXTENSION_ID,
+    ensureSourceCapture,
+    runtime = globalThis.chrome?.runtime,
+    store = new ScrapeReceiptStore(),
+    wait = defaultWait,
+  } = {}) {
+    if (!runtime) throw new Error('Chrome runtime is unavailable');
+    this.allowedExtensionId = allowedExtensionId;
+    this.ensureSourceCapture = ensureSourceCapture;
+    this.runtime = runtime;
+    this.store = store;
+    this.wait = wait;
+    this.connections = new Set();
+    this.cutoverClearPromise = null;
+  }
+
+  attach() {
+    this.runtime.onConnectExternal.addListener((port) => {
+      this.accept(port);
+    });
+    void this.ensureCutoverCleared().catch(() => {
+      // A later handshake retries and reports a persistent IndexedDB failure.
+    });
+  }
+
+  async ensureCutoverCleared() {
+    if (this.cutoverClearPromise) return this.cutoverClearPromise;
+    const clearing = this.store.finishCutover();
+    this.cutoverClearPromise = clearing;
+    try {
+      await clearing;
+    } catch (error) {
+      if (this.cutoverClearPromise === clearing) this.cutoverClearPromise = null;
+      throw error;
+    }
+  }
+
+  async finishSourceTab(sourceTabId, finishedAtMs = Date.now()) {
+    return this.store.finishRunsForSourceTab(sourceTabId, finishedAtMs);
+  }
+
+  async recordGraphqlResponse({ endpoint, requestUrl, sourceTabId, tweets }) {
+    const observations = await this.store.recordTimeline({
+      endpoint,
+      observedAtMs: Date.now(),
+      requestUrl,
+      sourceTabId,
+      tweets,
+    });
+    if (observations.length === 0) return observations;
+
+    const runId = observations[0].runId;
+    const message = {
+      observations,
+      protocolVersion: SCRAPE_PROTOCOL_VERSION,
+      runId,
+      type: 'scrape:observations',
+    };
+    for (const connection of this.connections) {
+      if (connection.runId === runId) post(connection.port, message);
+    }
+    return observations;
+  }
+
+  accept(port) {
+    if (
+      port.name !== SCRAPE_PORT_NAME ||
+      port.sender?.id !== this.allowedExtensionId
+    ) {
+      port.disconnect();
+      return;
+    }
+
+    const connection = { port, runId: undefined };
+    this.connections.add(connection);
+    port.onDisconnect.addListener(() => {
+      this.connections.delete(connection);
+    });
+    port.onMessage.addListener((message) => {
+      void this.handleMessageAfterCutover(connection, message);
+    });
+  }
+
+  async handleMessageAfterCutover(connection, message) {
+    try {
+      await this.ensureCutoverCleared();
+    } catch (error) {
+      post(connection.port, scrapeError(message, error));
+      return;
+    }
+    await this.handleMessage(connection, message);
+  }
+
+  async handleMessage(connection, message) {
+    try {
+      if (message?.protocolVersion !== SCRAPE_PROTOCOL_VERSION) {
+        throw new Error('unsupported scrape protocol version');
+      }
+      if (message.type === 'scrape:open') {
+        if (
+          !Array.isArray(message.capabilities) ||
+          !message.capabilities.includes('run-leases')
+        ) {
+          throw new ScrapeReceiptError(
+            'invalid-request',
+            'scrape client must support run leases',
+          );
+        }
+        connection.runId = message.runId;
+        if (!(await this.ensureCapture(message.sourceTabId))) {
+          throw new ScrapeReceiptError(
+            'internal-error',
+            `xTap passive capture is unavailable for source tab ${message.sourceTabId}`,
+          );
+        }
+        const run = await this.store.beginRun({
+          listId: message.listId,
+          runId: message.runId,
+          sourceTabId: message.sourceTabId,
+          startedAtMs: message.startedAtMs,
+        });
+        const observations = await this.store.readObservations(
+          run.runId,
+          normalizeCursor(message.afterCursor),
+        );
+        post(connection.port, {
+          capabilities: [
+            'search-timeline-observations',
+            'typed-errors',
+            'run-leases',
+          ],
+          observations,
+          protocolVersion: SCRAPE_PROTOCOL_VERSION,
+          run,
+          runId: run.runId,
+          type: 'scrape:opened',
+        });
+        return;
+      }
+
+      if (message.type === 'scrape:heartbeat') {
+        await this.store.renewRun(
+          message.runId,
+          message.sourceTabId,
+          message.renewedAtMs,
+        );
+        return;
+      }
+
+      if (message.type === 'scrape:finish') {
+        const run = await this.store.finishRun(
+          message.runId,
+          message.state,
+          message.finishedAtMs,
+        );
+        post(connection.port, {
+          protocolVersion: SCRAPE_PROTOCOL_VERSION,
+          run,
+          runId: run.runId,
+          type: 'scrape:finished',
+        });
+        return;
+      }
+
+      throw new Error('unknown scrape protocol message');
+    } catch (error) {
+      post(connection.port, scrapeError(message, error));
+    }
+  }
+
+  async ensureCapture(sourceTabId) {
+    if (!this.ensureSourceCapture) return true;
+    for (let attempt = 1; attempt <= CAPTURE_ATTACH_ATTEMPTS; attempt += 1) {
+      if (await this.ensureSourceCapture(sourceTabId)) return true;
+      if (attempt < CAPTURE_ATTACH_ATTEMPTS) {
+        await this.wait(CAPTURE_ATTACH_RETRY_MS);
+      }
+    }
+    return false;
+  }
+}
+
+function defaultWait(delayMs) {
+  return new Promise((resolve) => setTimeout(resolve, delayMs));
+}
+
+function normalizeCursor(value) {
+  return Number.isSafeInteger(value) && value >= 0 ? value : 0;
+}
+
+function scrapeError(message, error) {
+  const failure = protocolFailure(error);
+  return {
+    error: failure.message,
+    errorCode: failure.code,
+    protocolVersion: SCRAPE_PROTOCOL_VERSION,
+    runId: typeof message?.runId === 'string' ? message.runId : '',
+    type: 'scrape:error',
+  };
+}
+
+function protocolFailure(error) {
+  if (error instanceof ScrapeReceiptError) {
+    return { code: error.code, message: error.message };
+  }
+  const message = error instanceof Error ? error.message : String(error);
+  const code = message.startsWith('invalid ') ||
+    message === 'unsupported scrape protocol version' ||
+    message === 'unknown scrape protocol message'
+    ? 'invalid-request'
+    : 'internal-error';
+  return { code, message };
+}
+
+function post(port, message) {
+  try {
+    port.postMessage(message);
+  } catch {
+    // A disconnected client will reconnect and replay from its cursor.
+  }
+}

extension/lib/scrape-receipts.js

@@ -0,0 +1,537 @@
+const DATABASE_NAME = 'xtap-scrape-receipts';
+const DATABASE_VERSION = 1;
+const META_CUTOVER = 'cutoverInProgress';
+const META_SEQUENCE = 'captureSequence';
+const MAX_ACTIVE_RUNS = 4;
+export const SCRAPE_RUN_LEASE_MS = 90_000;
+const TIMELINE_ENDPOINTS = new Set([
+  'ListLatestTweetsTimeline',
+  'SearchTimeline',
+]);
+
+export const SCRAPE_PROTOCOL_VERSION = 1;
+export const SCRAPE_PORT_NAME = 'xtap-scrape-v1';
+
+export class ScrapeReceiptError extends Error {
+  constructor(code, message) {
+    super(message);
+    this.code = code;
+    this.name = 'ScrapeReceiptError';
+  }
+}
+
+export class ScrapeReceiptStore {
+  constructor(indexedDBFactory = globalThis.indexedDB, databaseName = DATABASE_NAME) {
+    if (!indexedDBFactory) throw new Error('IndexedDB is unavailable');
+    this.indexedDB = indexedDBFactory;
+    this.databaseName = databaseName;
+    this.databasePromise = null;
+  }
+
+  async beginRun({ runId, listId, sourceTabId, startedAtMs }, openedAtMs = Date.now()) {
+    validateRunInput({ runId, listId, sourceTabId, startedAtMs });
+    if (!Number.isFinite(openedAtMs)) throw new Error('invalid open time');
+    const database = await this.open();
+    const transaction = database.transaction(
+      ['listReceipts', 'meta', 'runs'],
+      'readwrite',
+    );
+    const listReceipts = transaction.objectStore('listReceipts');
+    const meta = transaction.objectStore('meta');
+    const runs = transaction.objectStore('runs');
+
+    if ((await readMeta(meta, META_CUTOVER)) === true) {
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'cutover-active',
+        'xTap scrape cutover is in progress',
+      );
+    }
+
+    const allRuns = await request(runs.getAll());
+    const activeRuns = [];
+    for (const run of allRuns) {
+      if (run.state !== 'running') continue;
+      if (!hasLiveLease(run, openedAtMs)) {
+        expireRun(run, openedAtMs);
+        runs.put(run);
+        continue;
+      }
+      activeRuns.push(run);
+    }
+
+    const existing = allRuns.find((run) => run.runId === runId);
+    if (existing) {
+      if (existing.listId !== listId || existing.startedAtMs !== startedAtMs) {
+        transaction.abort();
+        throw new ScrapeReceiptError(
+          'run-conflict',
+          'run ID already belongs to different parameters',
+        );
+      }
+      if (existing.state === 'running') {
+        if (existing.sourceTabId !== sourceTabId) {
+          transaction.abort();
+          throw new ScrapeReceiptError(
+            'run-conflict',
+            'run ID already belongs to a different source tab',
+          );
+        }
+        renewLease(existing, openedAtMs);
+        runs.put(existing);
+        await transactionDone(transaction);
+        return existing;
+      }
+      if (isRecoverableStop(existing.stopReason)) {
+        if (activeRuns.some((run) => run.sourceTabId === sourceTabId)) {
+          transaction.abort();
+          throw new ScrapeReceiptError(
+            'source-tab-active',
+            `source tab ${sourceTabId} already belongs to an active scrape run`,
+          );
+        }
+        if (activeRuns.length >= MAX_ACTIVE_RUNS) {
+          transaction.abort();
+          throw new ScrapeReceiptError(
+            'capacity-full',
+            'xTap already has four active scrape runs',
+          );
+        }
+        delete existing.finishedAtMs;
+        delete existing.stopReason;
+        existing.sourceTabId = sourceTabId;
+        existing.state = 'running';
+        renewLease(existing, openedAtMs);
+        runs.put(existing);
+        await transactionDone(transaction);
+        return existing;
+      }
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'run-terminal',
+        `scrape run already ended as ${existing.state}`,
+      );
+    }
+    if (activeRuns.some((run) => run.sourceTabId === sourceTabId)) {
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'source-tab-active',
+        `source tab ${sourceTabId} already belongs to an active scrape run`,
+      );
+    }
+    if (activeRuns.length >= MAX_ACTIVE_RUNS) {
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'capacity-full',
+        'xTap already has four active scrape runs',
+      );
+    }
+
+    const baselineSequence = (await readMeta(meta, META_SEQUENCE)) ?? 0;
+    const knownListCount = await request(
+      listReceipts.index('listId').count(listId),
+    );
+    const run = {
+      baselineSequence,
+      knownListCount,
+      lastCursor: 0,
+      listId,
+      protocolVersion: SCRAPE_PROTOCOL_VERSION,
+      runId,
+      sourceTabId,
+      startedAtMs,
+      state: 'running',
+      leaseExpiresAtMs: openedAtMs + SCRAPE_RUN_LEASE_MS,
+      updatedAtMs: openedAtMs,
+    };
+    runs.put(run);
+    await transactionDone(transaction);
+    return run;
+  }
+
+  async recordTimeline({ endpoint, requestUrl, observedAtMs, sourceTabId, tweets }) {
+    if (!TIMELINE_ENDPOINTS.has(endpoint)) return [];
+    if (!Number.isFinite(observedAtMs) || !isSourceTabId(sourceTabId)) return [];
+    if (!Array.isArray(tweets) || tweets.length === 0) return [];
+
+    const database = await this.open();
+    const transaction = database.transaction(
+      ['listReceipts', 'meta', 'observations', 'receipts', 'runs'],
+      'readwrite',
+    );
+    const listReceipts = transaction.objectStore('listReceipts');
+    const meta = transaction.objectStore('meta');
+    const observations = transaction.objectStore('observations');
+    const receipts = transaction.objectStore('receipts');
+    const runs = transaction.objectStore('runs');
+
+    let sequence = (await readMeta(meta, META_SEQUENCE)) ?? 0;
+    let run = (await request(runs.getAll())).find(
+      (candidate) =>
+        candidate.state === 'running' && candidate.sourceTabId === sourceTabId,
+    );
+    if (run && !hasLiveLease(run, observedAtMs)) {
+      expireRun(run, observedAtMs);
+      runs.put(run);
+      run = undefined;
+    }
+    const listId = run?.listId ?? extractListId(endpoint, requestUrl);
+    if (!isListId(listId)) {
+      await transactionDone(transaction);
+      return [];
+    }
+    const emitted = [];
+    const batchIds = new Set();
+
+    for (const tweet of tweets) {
+      if (!isTweet(tweet) || batchIds.has(tweet.id)) continue;
+      batchIds.add(tweet.id);
+
+      let receipt = await request(receipts.get(tweet.id));
+      let listReceipt = await request(listReceipts.get([listId, tweet.id]));
+      if (!receipt || !listReceipt) sequence += 1;
+
+      if (!receipt) {
+        receipt = {
+          endpoint,
+          firstSeenAt: normalizeTimestamp(tweet.captured_at, observedAtMs),
+          id: tweet.id,
+          listId,
+          postAt: tweet.created_at,
+          sequence,
+        };
+        receipts.put(receipt);
+      }
+
+      if (!listReceipt) {
+        listReceipt = {
+          firstSeenAt: normalizeTimestamp(tweet.captured_at, observedAtMs),
+          listId,
+          postAt: tweet.created_at,
+          sequence,
+          tweetId: tweet.id,
+        };
+        listReceipts.put(listReceipt);
+      }
+
+      if (run) {
+        run.lastCursor += 1;
+        const observation = {
+          captureSequence: listReceipt.sequence,
+          cursor: run.lastCursor,
+          knownBeforeRun: listReceipt.sequence <= run.baselineSequence,
+          observedAtMs,
+          postAt: tweet.created_at,
+          runId: run.runId,
+          sourceEndpoint: endpoint,
+          tweetId: tweet.id,
+        };
+        observations.put(observation);
+        emitted.push(observation);
+      }
+    }
+
+    writeMeta(meta, META_SEQUENCE, sequence);
+    if (run && emitted.length > 0) {
+      run.updatedAtMs = observedAtMs;
+      runs.put(run);
+    }
+    await transactionDone(transaction);
+    return emitted;
+  }
+
+  async renewRun(runId, sourceTabId, renewedAtMs) {
+    if (!isRunId(runId)) throw new Error('invalid run ID');
+    if (!isSourceTabId(sourceTabId)) throw new Error('invalid source tab ID');
+    if (!Number.isFinite(renewedAtMs)) throw new Error('invalid renewal time');
+
+    const database = await this.open();
+    const transaction = database.transaction('runs', 'readwrite');
+    const runs = transaction.objectStore('runs');
+    const run = await request(runs.get(runId));
+    if (!run) {
+      transaction.abort();
+      throw new ScrapeReceiptError('unknown-run', 'unknown scrape run');
+    }
+    if (run.state !== 'running') {
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'run-terminal',
+        `scrape run already ended as ${run.state}`,
+      );
+    }
+    if (run.sourceTabId !== sourceTabId) {
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'run-conflict',
+        'run ID already belongs to a different source tab',
+      );
+    }
+    renewLease(run, renewedAtMs);
+    runs.put(run);
+    await transactionDone(transaction);
+    return run;
+  }
+
+  async finishRunsForSourceTab(sourceTabId, finishedAtMs) {
+    if (!isSourceTabId(sourceTabId)) throw new Error('invalid source tab ID');
+    if (!Number.isFinite(finishedAtMs)) throw new Error('invalid finish time');
+
+    const database = await this.open();
+    const transaction = database.transaction('runs', 'readwrite');
+    const runs = transaction.objectStore('runs');
+    const activeRuns = (await request(runs.getAll())).filter(
+      (run) => run.state === 'running' && run.sourceTabId === sourceTabId,
+    );
+    for (const run of activeRuns) {
+      run.finishedAtMs = finishedAtMs;
+      run.state = 'aborted';
+      run.stopReason = 'source-tab-closed';
+      run.updatedAtMs = finishedAtMs;
+      runs.put(run);
+    }
+    await transactionDone(transaction);
+    return activeRuns.length;
+  }
+
+  async readObservations(runId, afterCursor = 0) {
+    if (!isRunId(runId) || !Number.isSafeInteger(afterCursor) || afterCursor < 0) {
+      throw new Error('invalid observation cursor');
+    }
+    const database = await this.open();
+    const transaction = database.transaction('observations', 'readonly');
+    const values = await request(
+      transaction.objectStore('observations').index('runId').getAll(runId),
+    );
+    await transactionDone(transaction);
+    return values
+      .filter((entry) => entry.cursor > afterCursor)
+      .sort((left, right) => left.cursor - right.cursor);
+  }
+
+  async beginCutover(finishedAtMs) {
+    if (!Number.isFinite(finishedAtMs)) throw new Error('invalid finish time');
+
+    const database = await this.open();
+    const transaction = database.transaction(['meta', 'runs'], 'readwrite');
+    const meta = transaction.objectStore('meta');
+    const runs = transaction.objectStore('runs');
+    writeMeta(meta, META_CUTOVER, true);
+    const activeRuns = (await request(runs.getAll())).filter(
+      (run) => run.state === 'running',
+    );
+    for (const run of activeRuns) {
+      run.finishedAtMs = finishedAtMs;
+      run.state = 'failed';
+      run.updatedAtMs = finishedAtMs;
+      runs.put(run);
+    }
+    await transactionDone(transaction);
+    return activeRuns.length;
+  }
+
+  async finishCutover() {
+    const database = await this.open();
+    const transaction = database.transaction('meta', 'readwrite');
+    writeMeta(transaction.objectStore('meta'), META_CUTOVER, false);
+    await transactionDone(transaction);
+  }
+
+  async finishRun(runId, state, finishedAtMs) {
+    if (!isRunId(runId)) throw new Error('invalid run ID');
+    if (!['aborted', 'completed', 'failed'].includes(state)) {
+      throw new Error('invalid terminal run state');
+    }
+    if (!Number.isFinite(finishedAtMs)) throw new Error('invalid finish time');
+
+    const database = await this.open();
+    const transaction = database.transaction('runs', 'readwrite');
+    const runs = transaction.objectStore('runs');
+    const run = await request(runs.get(runId));
+    if (!run) {
+      transaction.abort();
+      throw new ScrapeReceiptError('unknown-run', 'unknown scrape run');
+    }
+    if (run.state !== 'running' && run.state !== state) {
+      transaction.abort();
+      throw new ScrapeReceiptError(
+        'run-terminal',
+        `scrape run already ended as ${run.state}`,
+      );
+    }
+
+    run.finishedAtMs = finishedAtMs;
+    run.state = state;
+    run.updatedAtMs = finishedAtMs;
+    runs.put(run);
+    await transactionDone(transaction);
+    return run;
+  }
+
+  async getRun(runId) {
+    const database = await this.open();
+    const transaction = database.transaction('runs', 'readonly');
+    const run = await request(transaction.objectStore('runs').get(runId));
+    await transactionDone(transaction);
+    return run;
+  }
+
+  async open() {
+    if (this.databasePromise) return this.databasePromise;
+    const opening = new Promise((resolve, reject) => {
+      const openRequest = this.indexedDB.open(this.databaseName, DATABASE_VERSION);
+      let settled = false;
+      openRequest.onupgradeneeded = () => createSchema(openRequest.result);
+      openRequest.onsuccess = () => {
+        if (settled) {
+          openRequest.result.close();
+          return;
+        }
+        settled = true;
+        resolve(openRequest.result);
+      };
+      openRequest.onerror = () => {
+        settled = true;
+        reject(openRequest.error ?? new Error('failed to open receipt database'));
+      };
+      openRequest.onblocked = () => {
+        settled = true;
+        reject(new Error('receipt database upgrade is blocked'));
+      };
+    });
+    this.databasePromise = opening;
+    opening.catch(() => {
+      if (this.databasePromise === opening) this.databasePromise = null;
+    });
+    return opening;
+  }
+
+  async close() {
+    if (!this.databasePromise) return;
+    const database = await this.databasePromise;
+    database.close();
+    this.databasePromise = null;
+  }
+}
+
+export function extractListId(endpoint, requestUrl) {
+  if (!TIMELINE_ENDPOINTS.has(endpoint) || typeof requestUrl !== 'string') {
+    return undefined;
+  }
+  try {
+    const variables = new URL(requestUrl).searchParams.get('variables');
+    if (!variables) return undefined;
+    const parsed = JSON.parse(variables);
+    if (endpoint === 'ListLatestTweetsTimeline') {
+      return isListId(parsed?.listId) ? parsed.listId : undefined;
+    }
+    const rawQuery = typeof parsed?.rawQuery === 'string' ? parsed.rawQuery : '';
+    return rawQuery.match(/(?:^|\s)list:(\d{4,25})(?:\s|$)/)?.[1];
+  } catch {
+    return undefined;
+  }
+}
+
+function renewLease(run, renewedAtMs) {
+  run.leaseExpiresAtMs = renewedAtMs + SCRAPE_RUN_LEASE_MS;
+  run.updatedAtMs = renewedAtMs;
+}
+
+function hasLiveLease(run, atMs) {
+  return Number.isFinite(run.leaseExpiresAtMs) && run.leaseExpiresAtMs > atMs;
+}
+
+function expireRun(run, finishedAtMs) {
+  run.finishedAtMs = finishedAtMs;
+  run.state = 'failed';
+  run.stopReason = 'lease-expired';
+  run.updatedAtMs = finishedAtMs;
+}
+
+function isRecoverableStop(value) {
+  return value === 'lease-expired' || value === 'source-tab-closed';
+}
+
+function createSchema(database) {
+  if (!database.objectStoreNames.contains('receipts')) {
+    const receipts = database.createObjectStore('receipts', { keyPath: 'id' });
+    receipts.createIndex('sequence', 'sequence', { unique: true });
+  }
+  if (!database.objectStoreNames.contains('listReceipts')) {
+    const listReceipts = database.createObjectStore('listReceipts', {
+      keyPath: ['listId', 'tweetId'],
+    });
+    listReceipts.createIndex('listId', 'listId', { unique: false });
+  }
+  if (!database.objectStoreNames.contains('runs')) {
+    database.createObjectStore('runs', { keyPath: 'runId' });
+  }
+  if (!database.objectStoreNames.contains('observations')) {
+    const observations = database.createObjectStore('observations', {
+      keyPath: ['runId', 'cursor'],
+    });
+    observations.createIndex('runId', 'runId', { unique: false });
+  }
+  if (!database.objectStoreNames.contains('meta')) {
+    database.createObjectStore('meta', { keyPath: 'key' });
+  }
+}
+
+function request(idbRequest) {
+  return new Promise((resolve, reject) => {
+    idbRequest.onsuccess = () => resolve(idbRequest.result);
+    idbRequest.onerror = () => reject(idbRequest.error ?? new Error('IndexedDB request failed'));
+  });
+}
+
+function transactionDone(transaction) {
+  return new Promise((resolve, reject) => {
+    transaction.oncomplete = () => resolve();
+    transaction.onerror = () => reject(transaction.error ?? new Error('IndexedDB transaction failed'));
+    transaction.onabort = () => reject(transaction.error ?? new Error('IndexedDB transaction aborted'));
+  });
+}
+
+async function readMeta(store, key) {
+  const entry = await request(store.get(key));
+  return entry?.value;
+}
+
+function writeMeta(store, key, value) {
+  store.put({ key, value });
+}
+
+function validateRunInput({ runId, listId, sourceTabId, startedAtMs }) {
+  if (!isRunId(runId)) throw new Error('invalid run ID');
+  if (!isListId(listId)) throw new Error('invalid list ID');
+  if (!isSourceTabId(sourceTabId)) throw new Error('invalid source tab ID');
+  if (!Number.isFinite(startedAtMs)) throw new Error('invalid start time');
+}
+
+function isRunId(value) {
+  return typeof value === 'string' && /^[0-9a-f-]{16,64}$/i.test(value);
+}
+
+function isListId(value) {
+  return typeof value === 'string' && /^\d{4,25}$/.test(value);
+}
+
+function isSourceTabId(value) {
+  return Number.isSafeInteger(value) && value >= 0;
+}
+
+function isTweet(value) {
+  return (
+    value &&
+    typeof value.id === 'string' &&
+    value.id.length > 0 &&
+    typeof value.created_at === 'string' &&
+    Number.isFinite(Date.parse(value.created_at))
+  );
+}
+
+function normalizeTimestamp(value, fallbackMs) {
+  return typeof value === 'string' && Number.isFinite(Date.parse(value))
+    ? value
+    : new Date(fallbackMs).toISOString();
+}

extension/lib/tweet-parser.js

@@ -0,0 +1,525 @@
+// xTap — Tweet parser
+// Extracts and normalizes tweets from X/Twitter GraphQL API responses.
+import { exactCounter } from './observations.js';
+
+/**
+ * Main entry point. Given an endpoint name and the raw GraphQL response data,
+ * returns an array of normalized tweet objects.
+ */
+export function extractTweets(endpoint, data, observedAt = new Date().toISOString()) {
+  const capturedAt = new Date(observedAt).toISOString();
+  return parseTweets(endpoint, data).map(tweet => ({ ...tweet, captured_at: capturedAt }));
+}
+
+function parseTweets(endpoint, data) {
+  if (!data) return [];
+
+  // TweetResultByRestId returns a single tweet, not a timeline with instructions.
+  // Always extract it — dedup in enqueueTweets prevents duplicates if the tweet
+  // also appears in TweetDetail. The root tweet on a tweet detail page is often
+  // delivered only via this endpoint and omitted from TweetDetail.
+  if (endpoint === 'TweetResultByRestId') {
+    const result = data?.data?.tweetResult?.result;
+    if (!result) return [];
+    const raw = unwrapTweetResult(result);
+    if (!raw) return [];
+    return normalizeWithQuoted(raw);
+  }
+
+  const instructions = findInstructions(endpoint, data);
+  if (!instructions || !Array.isArray(instructions)) {
+    // Log top-level keys so we can find where instructions moved
+    const topKeys = data?.data ? Object.keys(data.data) : Object.keys(data || {});
+    console.warn(`[xTap:parser] No instructions found for ${endpoint} | top-level keys: [${topKeys.join(', ')}]`);
+    return [];
+  }
+
+  const tweets = [];
+  let skippedEntries = 0;
+  const skippedTypes = {};
+
+  for (const instruction of instructions) {
+    const entries = instruction.entries || instruction.moduleItems || [];
+
+    for (const entry of entries) {
+      const extracted = extractTweetsFromEntry(entry, skippedTypes);
+      tweets.push(...extracted);
+      if (extracted.length === 0 && !entry.content?.cursorType && !entry.content?.entryType?.includes('Cursor')) {
+        skippedEntries++;
+      }
+    }
+
+    if (instruction.entry) {
+      tweets.push(...extractTweetsFromEntry(instruction.entry, skippedTypes));
+    }
+  }
+
+  if (skippedEntries > 0 || Object.keys(skippedTypes).length > 0) {
+    console.log(`[xTap:parser] ${endpoint}: ${tweets.length} tweets extracted, ${skippedEntries} entries skipped | skipped types: ${JSON.stringify(skippedTypes)}`);
+  }
+
+  return tweets;
+}
+
+// --- Known instruction paths per endpoint ---
+const INSTRUCTION_PATHS = {
+  HomeTimeline: ['data', 'home', 'home_timeline_urt', 'instructions'],
+  HomeLatestTimeline: ['data', 'home', 'home_timeline_urt', 'instructions'],
+  UserTweets: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  UserTweetsAndReplies: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  UserMedia: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  UserLikes: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  UserArticlesTweets: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  UserHighlightsTweets: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  TweetDetail: ['data', 'threaded_conversation_with_injections_v2', 'instructions'],
+  SearchTimeline: ['data', 'search_by_raw_query', 'search_timeline', 'timeline', 'instructions'],
+  ListLatestTweetsTimeline: ['data', 'list', 'tweets_timeline', 'timeline', 'instructions'],
+  Bookmarks: ['data', 'bookmark_timeline_v2', 'timeline', 'instructions'],
+  Likes: ['data', 'user', 'result', 'timeline', 'timeline', 'instructions'],
+  CommunityTweetsTimeline: ['data', 'communityResults', 'result', 'ranked_community_timeline', 'timeline', 'instructions'],
+  BookmarkFolderTimeline: ['data', 'bookmark_collection_timeline', 'timeline', 'instructions'],
+};
+
+/**
+ * Navigate to the instructions[] array. Different endpoints nest it at different paths.
+ */
+function findInstructions(endpoint, data) {
+  const path = INSTRUCTION_PATHS[endpoint];
+  if (path) {
+    const result = navigatePath(data, path);
+    if (result) return result;
+    // Known endpoint but path failed — log where it broke and what's there instead
+    let current = data;
+    for (let i = 0; i < path.length; i++) {
+      if (current == null || typeof current !== 'object') {
+        const parent = navigatePath(data, path.slice(0, i));
+        console.warn(`[xTap:parser] Path broke for ${endpoint} at step ${i} ("${path[i]}") | value is: ${typeof current} | parent keys: [${Object.keys(parent || {}).join(', ')}]`);
+        break;
+      }
+      if (current[path[i]] === undefined) {
+        console.warn(`[xTap:parser] Path broke for ${endpoint} at step ${i} ("${path[i]}") | key not found | available keys: [${Object.keys(current).join(', ')}]`);
+        break;
+      }
+      current = current[path[i]];
+    }
+  } else {
+    console.log(`[xTap:parser] Unknown endpoint "${endpoint}", using recursive fallback`);
+  }
+
+  // Generic fallback: recursively search for an instructions array
+  const fallback = findInstructionsRecursive(data, 5);
+  if (fallback) {
+    console.log(`[xTap:parser] Recursive fallback found instructions for ${endpoint}`);
+  }
+  return fallback;
+}
+
+function navigatePath(obj, path) {
+  let current = obj;
+  for (const key of path) {
+    if (current == null || typeof current !== 'object') return null;
+    current = current[key];
+  }
+  return current;
+}
+
+function findInstructionsRecursive(obj, maxDepth) {
+  if (maxDepth <= 0 || obj == null || typeof obj !== 'object') return null;
+
+  if (Array.isArray(obj.instructions)) {
+    const hasEntries = obj.instructions.some(i =>
+      i.type === 'TimelineAddEntries' || i.entries || i.type === 'TimelineAddToModule'
+    );
+    if (hasEntries) return obj.instructions;
+  }
+
+  for (const key of Object.keys(obj)) {
+    if (key === 'instructions') continue;
+    const result = findInstructionsRecursive(obj[key], maxDepth - 1);
+    if (result) return result;
+  }
+
+  return null;
+}
+
+/**
+ * Extract tweets from a single timeline entry.
+ */
+function extractTweetsFromEntry(entry, skippedTypes) {
+  const tweets = [];
+  const content = entry.content || entry;
+
+  // Cursor entries — skip silently
+  if (content.entryType === 'TimelineTimelineCursor' || content.cursorType) {
+    return tweets;
+  }
+
+  // Single tweet item
+  if (content.entryType === 'TimelineTimelineItem' || content.__typename === 'TimelineTimelineItem') {
+    tweets.push(...extractFromItemContent(content.itemContent));
+    return tweets;
+  }
+
+  // Thread / conversation module
+  if (content.entryType === 'TimelineTimelineModule' || content.__typename === 'TimelineTimelineModule') {
+    const items = content.items || [];
+    for (const item of items) {
+      const itemContent = item.item?.itemContent || item.itemContent;
+      tweets.push(...extractFromItemContent(itemContent));
+    }
+    return tweets;
+  }
+
+  // Fallback: try itemContent directly (may be at content.itemContent or content.item.itemContent)
+  const fallbackItemContent = content.itemContent || content.item?.itemContent;
+  if (fallbackItemContent) {
+    tweets.push(...extractFromItemContent(fallbackItemContent));
+    return tweets;
+  }
+
+  // Unknown entry type — log it so we can add support
+  const entryType = content.entryType || content.__typename || 'no_type';
+  skippedTypes[entryType] = (skippedTypes[entryType] || 0) + 1;
+  if (!skippedTypes['_logged_' + entryType]) {
+    skippedTypes['_logged_' + entryType] = true;
+    console.warn(`[xTap:parser] Unknown entry type: "${entryType}" | keys: [${Object.keys(content).join(', ')}]`);
+  }
+
+  return tweets;
+}
+
+function extractFromItemContent(itemContent) {
+  if (!itemContent) return [];
+
+  // Track non-tweet item types we encounter
+  if (itemContent.itemType !== 'TimelineTweet' && itemContent.__typename !== 'TimelineTweet') {
+    const itemType = itemContent.itemType || itemContent.__typename || 'unknown';
+    if (!extractFromItemContent._seenTypes) extractFromItemContent._seenTypes = new Set();
+    if (!extractFromItemContent._seenTypes.has(itemType)) {
+      extractFromItemContent._seenTypes.add(itemType);
+      console.log(`[xTap:parser] Skipping non-tweet itemType: "${itemType}"`);
+    }
+    return [];
+  }
+
+  const tweetResults = itemContent.tweet_results;
+  if (!tweetResults?.result) {
+    console.warn('[xTap:parser] TimelineTweet has no tweet_results.result | keys:', Object.keys(itemContent).join(', '));
+    return [];
+  }
+
+  const raw = unwrapTweetResult(tweetResults.result);
+  if (!raw) return [];
+
+  return normalizeWithQuoted(raw);
+}
+
+/**
+ * Normalize a tweet plus any inline quoted tweet it carries. X delivers the
+ * full quoted payload at quoted_status_result.result — capturing only
+ * legacy.quoted_status_id_str would discard the quoted content entirely.
+ */
+function normalizeWithQuoted(raw) {
+  const out = [];
+  const tweet = normalizeTweet(raw);
+  if (!tweet) return out;
+  out.push(tweet);
+
+  // For retweets the quoted payload sits on the retweeted original.
+  const sources = [raw];
+  const rtResult = raw.legacy?.retweeted_status_result?.result;
+  if (rtResult) {
+    const rtRaw = unwrapTweetResult(rtResult);
+    if (rtRaw) sources.push(rtRaw);
+  }
+  for (const source of sources) {
+    const quotedResult = source.quoted_status_result?.result;
+    if (!quotedResult) continue;
+    const quotedRaw = unwrapTweetResult(quotedResult);
+    if (!quotedRaw) continue;
+    const quoted = normalizeTweet(quotedRaw);
+    // Dedupe by id — a retweet-of-a-quote can carry the same quoted payload
+    // on both the wrapper and the original, and article tweets bypass all
+    // downstream dedup layers.
+    if (quoted && !out.some(t => t.id === quoted.id)) out.push(quoted);
+  }
+  return out;
+}
+
+/**
+ * Unwrap tweet result — handles Tweet, TweetWithVisibilityResults, tombstones.
+ */
+function unwrapTweetResult(result) {
+  if (!result) return null;
+
+  const typename = result.__typename;
+
+  if (typename === 'Tweet') return result;
+
+  if (typename === 'TweetWithVisibilityResults') {
+    const tweet = result.tweet;
+    if (!tweet) return null;
+    tweet._visibilityResults = result.tweetInterstitial || result.visibility_results || null;
+    return tweet;
+  }
+
+  // Tombstones, unavailable tweets — skip silently
+  if (typename === 'TweetTombstone' || typename === 'TweetUnavailable') {
+    return null;
+  }
+
+  // Unknown type but has legacy data — try anyway
+  if (result.legacy && result.core) {
+    console.warn(`[xTap:parser] Unknown tweet __typename "${typename}", attempting extraction (has legacy+core)`);
+    return result;
+  }
+
+  console.warn(`[xTap:parser] Unhandled tweet result __typename: "${typename}" | keys: [${Object.keys(result).join(', ')}]`);
+  return null;
+}
+
+/**
+ * Normalize a raw tweet into our output schema.
+ */
+export function normalizeTweet(raw) {
+  const legacy = raw.legacy;
+  if (!legacy) {
+    console.warn('[xTap:parser] Tweet missing legacy | keys:', Object.keys(raw).join(', '));
+    return null;
+  }
+
+  const tweetId = legacy.id_str || raw.rest_id;
+  if (isArticleStub(raw)) {
+    console.log(`[xTap:parser] Skipping article tweet ${tweetId || '(no id)'} without content_state.blocks`);
+    return null;
+  }
+
+  const userResult = raw.core?.user_results?.result;
+  const userCore = userResult?.core;      // X nests name/screen_name here
+  const userLegacy = userResult?.legacy;  // follower_count, verified, etc.
+
+  // Warn if user data paths have changed
+  if (!userResult) {
+    console.warn(`[xTap:parser] No user data at raw.core.user_results.result | raw.core keys: [${Object.keys(raw.core || {}).join(', ')}]`);
+  } else if (!userCore?.screen_name && !userLegacy?.screen_name) {
+    console.warn(`[xTap:parser] No screen_name found | userResult keys: [${Object.keys(userResult).join(', ')}] | userResult.core keys: [${Object.keys(userCore || {}).join(', ')}] | userResult.legacy keys: [${Object.keys(userLegacy || {}).join(', ')}]`);
+  }
+
+  const text = extractFullText(raw);
+  const media = extractMedia(legacy);
+  const urls = extractUrls(legacy);
+
+  const username = userCore?.screen_name || userLegacy?.screen_name;
+
+  const tweet = {
+    id: tweetId,
+    url: username ? `https://x.com/${username}/status/${tweetId}` : null,
+    created_at: toISO(legacy.created_at),
+    author: {
+      id: userResult?.rest_id || legacy.user_id_str,
+      username: userCore?.screen_name || userLegacy?.screen_name,
+      display_name: userCore?.name || userLegacy?.name,
+      verified: userLegacy?.verified || false,
+      is_blue_verified: userResult?.is_blue_verified || false,
+      follower_count: userLegacy?.followers_count
+    },
+    text,
+    lang: legacy.lang,
+    metrics: {
+      format: 'exact-v1',
+      likes: exactCounter(legacy.favorite_count),
+      retweets: exactCounter(legacy.retweet_count),
+      replies: exactCounter(legacy.reply_count),
+      views: exactCounter(raw.views?.count),
+      bookmarks: exactCounter(legacy.bookmark_count),
+      quotes: exactCounter(legacy.quote_count)
+    },
+    media,
+    urls,
+    hashtags: (legacy.entities?.hashtags || []).map(h => h.text),
+    mentions: (legacy.entities?.user_mentions || []).map(m => ({
+      id: m.id_str,
+      username: m.screen_name
+    })),
+    in_reply_to: legacy.in_reply_to_status_id_str || null,
+    quoted_tweet_id: legacy.quoted_status_id_str || null,
+    conversation_id: legacy.conversation_id_str || null,
+    is_retweet: !!legacy.retweeted_status_result,
+    retweeted_tweet_id: legacy.retweeted_status_result?.result?.legacy?.id_str || null,
+    is_subscriber_only: !!raw.exclusivityInfo,
+    source_endpoint: null, // Set by caller
+    captured_at: new Date().toISOString()
+  };
+
+  // Clean up temporary property from unwrapTweetResult
+  delete raw._visibilityResults;
+
+  // If this is a retweet, also normalize the retweeted tweet
+  // (the text of a retweet is often truncated with "RT @user:")
+  if (legacy.retweeted_status_result?.result) {
+    const rtRaw = unwrapTweetResult(legacy.retweeted_status_result.result);
+    if (rtRaw) {
+      tweet.text = extractFullText(rtRaw);
+    }
+  }
+
+  // Extract article content (long-form posts)
+  const article = extractArticle(raw);
+  if (article) {
+    tweet.text = article.text;
+    tweet.is_article = true;
+    tweet.article = article;
+  }
+
+  return tweet;
+}
+
+/**
+ * Extract article data from a raw tweet result.
+ * Articles use Draft.js block format in content_state.
+ */
+export function extractArticle(raw) {
+  const articleResult = getArticleResult(raw);
+  if (!articleResult) return null;
+
+  // Timeline endpoints return article stubs with only title/preview_text.
+  // Require content_state to be present — full data comes from TweetResultByRestId.
+  if (!hasArticleContent(articleResult)) return null;
+
+  const tweetId = raw.legacy?.id_str || raw.rest_id;
+  const title = articleResult.title || null;
+  const blocks = articleResult.content_state.blocks;
+  const entityMap = articleResult.content_state.entityMap || [];
+
+  // Build mediaId → {url, filename} from media_entities
+  const mediaEntities = articleResult.media_entities || [];
+  const mediaById = new Map();
+  for (const e of mediaEntities) {
+    const url = e.media_info?.original_img_url || null;
+    const filename = url ? url.split('/').pop() : null;
+    mediaById.set(e.media_id, {
+      id: e.media_id,
+      url,
+      filename,
+      local_path: filename ? `media/${tweetId}/${filename}` : null,
+      width: e.media_info?.original_img_width || null,
+      height: e.media_info?.original_img_height || null
+    });
+  }
+
+  // Build entity key → mediaId from entityMap. X emits an array of
+  // {key, value} pairs, but standard Draft.js raw content uses an object
+  // keyed by entity id — accept both shapes.
+  const entityMapEntries = Array.isArray(entityMap)
+    ? entityMap.map((ent) => [ent.key, ent.value])
+    : Object.entries(entityMap);
+  const entityMediaId = new Map();
+  for (const [key, value] of entityMapEntries) {
+    const items = value?.data?.mediaItems || [];
+    if (items.length > 0) {
+      entityMediaId.set(String(key), items[0].mediaId);
+    }
+  }
+
+  // Build plain text from Draft.js blocks, inserting image references for atomic blocks
+  const lines = [];
+  for (const block of blocks) {
+    const blockText = block.text || '';
+    const type = block.type;
+    if (type === 'atomic') {
+      const eKey = String(block.entityRanges?.[0]?.key);
+      const mid = entityMediaId.get(eKey);
+      const m = mid ? mediaById.get(mid) : null;
+      if (m) lines.push(`![${m.filename}](${m.local_path})`);
+      continue;
+    }
+    if (type === 'header-one') { lines.push(`# ${blockText}`); continue; }
+    if (type === 'header-two') { lines.push(`## ${blockText}`); continue; }
+    if (type === 'header-three') { lines.push(`### ${blockText}`); continue; }
+    if (type === 'ordered-list-item') { lines.push(`1. ${blockText}`); continue; }
+    if (type === 'unordered-list-item') { lines.push(`- ${blockText}`); continue; }
+    if (type === 'blockquote') { lines.push(`> ${blockText}`); continue; }
+    lines.push(blockText);
+  }
+  const text = lines.join('\n');
+
+  const media = [...mediaById.values()];
+
+  return { title, text, blocks, media };
+}
+
+function getArticleResult(raw) {
+  return raw.article?.article_results?.result || null;
+}
+
+function hasArticleContent(articleResult) {
+  return Array.isArray(articleResult?.content_state?.blocks) &&
+    articleResult.content_state.blocks.length > 0;
+}
+
+function isArticleStub(raw) {
+  const articleResult = getArticleResult(raw);
+  return !!articleResult && !hasArticleContent(articleResult);
+}
+
+/**
+ * Extract full text, preferring note_tweet (long-form) over legacy.full_text.
+ */
+function extractFullText(raw) {
+  const noteText = raw.note_tweet?.note_tweet_results?.result?.text;
+  if (noteText) return noteText;
+  return raw.legacy?.full_text || '';
+}
+
+/**
+ * Extract media from extended_entities.
+ */
+export function extractMedia(legacy) {
+  const mediaList = legacy.extended_entities?.media || legacy.entities?.media || [];
+  return mediaList.map(m => {
+    const item = {
+      type: m.type, // photo, video, animated_gif
+      url: null,
+      alt_text: m.ext_alt_text || null
+    };
+
+    if (m.type === 'photo') {
+      item.url = m.media_url_https ? m.media_url_https + ':orig' : m.media_url_https;
+    } else if (m.type === 'video' || m.type === 'animated_gif') {
+      const variants = m.video_info?.variants || [];
+      const mp4s = variants
+        .filter(v => v.content_type === 'video/mp4')
+        .sort((a, b) => (b.bitrate || 0) - (a.bitrate || 0));
+      item.url = mp4s[0]?.url || null;
+      if (m.type === 'video') {
+        item.duration_ms = m.video_info?.duration_millis || null;
+      }
+    } else {
+      console.warn(`[xTap:parser] Unknown media type: "${m.type}" | keys: [${Object.keys(m).join(', ')}]`);
+    }
+
+    return item;
+  });
+}
+
+/**
+ * Extract URLs from entities.
+ */
+function extractUrls(legacy) {
+  const urlList = legacy.entities?.urls || [];
+  return urlList.map(u => ({
+    display: u.display_url,
+    expanded: u.expanded_url,
+    shortened: u.url
+  }));
+}
+
+/**
+ * Convert Twitter's legacy date format to ISO 8601.
+ * Falls back to the original string if parsing fails.
+ */
+function toISO(dateStr) {
+  if (!dateStr) return dateStr;
+  const d = new Date(dateStr);
+  return isNaN(d.getTime()) ? dateStr : d.toISOString();
+}

extension/manifest.json

@@ -0,0 +1,48 @@
+{
+  "manifest_version": 3,
+  "name": "xtap-pool",
+  "version": "0.26.0",
+  "description": "Passively captures tweets from X/Twitter as you browse and pools them with friends via a private Hugging Face Space",
+  "permissions": [
+    "alarms",
+    "debugger",
+    "nativeMessaging",
+    "storage",
+    "unlimitedStorage"
+  ],
+  "host_permissions": [
+    "*://*.x.com/*",
+    "*://*.twitter.com/*",
+    "http://127.0.0.1/*",
+    "https://*.hf.space/*"
+  ],
+  "background": {
+    "service_worker": "background.js",
+    "type": "module"
+  },
+  "content_scripts": [
+    {
+      "matches": ["https://*.hf.space/*"],
+      "js": ["pool-connect.js"],
+      "world": "ISOLATED",
+      "run_at": "document_idle"
+    }
+  ],
+  "options_ui": {
+    "page": "options.html",
+    "open_in_tab": false
+  },
+  "action": {
+    "default_popup": "popup.html",
+    "default_icon": {
+      "16": "icons/icon16.png",
+      "48": "icons/icon48.png",
+      "128": "icons/icon128.png"
+    }
+  },
+  "icons": {
+    "16": "icons/icon16.png",
+    "48": "icons/icon48.png",
+    "128": "icons/icon128.png"
+  }
+}

extension/native-host/com.xtap.daemon.plist

@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>Label</key>
+    <string>com.xtap.daemon</string>
+    <key>ProgramArguments</key>
+    <array>
+        <string>__PYTHON_PATH__</string>
+        <string>__DAEMON_PATH__</string>
+    </array>
+    <key>EnvironmentVariables</key>
+    <dict>
+        <key>HOME</key>
+        <string>__HOME_DIR__</string>
+        <key>PATH</key>
+        <string>__PATH__</string>
+        <key>XTAP_LOG_LEVEL</key>
+        <string>__LOG_LEVEL__</string>
+        <key>XTAP_OUTPUT_DIR</key>
+        <string>__OUTPUT_DIR__</string>
+    </dict>
+    <key>RunAtLoad</key>
+    <true/>
+    <key>KeepAlive</key>
+    <true/>
+    <key>StandardOutPath</key>
+    <string>__HOME_DIR__/.xtap/daemon-stdout.log</string>
+    <key>StandardErrorPath</key>
+    <string>__HOME_DIR__/.xtap/daemon-stderr.log</string>
+</dict>
+</plist>

extension/native-host/com.xtap.daemon.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=xTap HTTP daemon
+After=network.target
+
+[Service]
+ExecStart=__PYTHON_PATH__ __DAEMON_PATH__
+Environment=HOME=__HOME_DIR__
+Environment=PATH=__PATH__
+Environment=XTAP_LOG_LEVEL=__LOG_LEVEL__
+Environment="XTAP_OUTPUT_DIR=__OUTPUT_DIR__"
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

extension/native-host/com.xtap.host.json

@@ -0,0 +1,7 @@
+{
+  "name": "com.xtap.host",
+  "description": "xTap native messaging host — passes the HTTP daemon auth token to the extension",
+  "path": "/path/to/xtap_host.py",
+  "type": "stdio",
+  "allowed_origins": ["chrome-extension://EXTENSION_ID/"]
+}

extension/native-host/install.ps1

@@ -0,0 +1,139 @@
+# xTap — Windows installer for the native messaging host (PowerShell).
+# Usage:
+#   .\install.ps1 -ExtensionId <id> [-Browser chrome|firefox]
+#   .\install.ps1 -Browser firefox
+
+param(
+    [Parameter(Mandatory=$false, Position=0)]
+    [string]$ExtensionId,
+
+    [Parameter(Mandatory=$false)]
+    [ValidateSet("chrome", "firefox")]
+    [string]$Browser = "chrome"
+)
+
+$ErrorActionPreference = "Stop"
+
+$HostName = "com.xtap.host"
+$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Definition
+$HostPy = Join-Path $ScriptDir "xtap_host.py"
+$BatPath = Join-Path $ScriptDir "xtap_host.bat"
+# Per-browser file: Chrome needs allowed_origins, Firefox allowed_extensions.
+# A shared file would let the second install break the first browser (both
+# registry keys would point at a manifest missing their key). The generated
+# name also avoids clobbering the repo's template JSON files.
+$ManifestPath = Join-Path $ScriptDir "$HostName.$Browser.generated.json"
+
+if (-not $ExtensionId -and $Browser -eq "firefox") {
+    $ExtensionId = "xtap@mkubicek.dev"
+}
+if (-not $ExtensionId) {
+    Write-Error "ExtensionId is required for Chrome installs (find it at chrome://extensions)."
+    exit 1
+}
+
+$RegKey = if ($Browser -eq "firefox") {
+    "HKCU:\Software\Mozilla\NativeMessagingHosts\$HostName"
+} else {
+    "HKCU:\Software\Google\Chrome\NativeMessagingHosts\$HostName"
+}
+
+# Verify python
+if (-not (Get-Command python -ErrorAction SilentlyContinue)) {
+    Write-Error "python is required but not found in PATH"
+    exit 1
+}
+
+# Write manifest (path must point to the .bat wrapper)
+$manifestData = @{
+    name = $HostName
+    description = "xTap native messaging host -- passes the HTTP daemon auth token to the extension"
+    path = $BatPath
+    type = "stdio"
+}
+if ($Browser -eq "firefox") {
+    $manifestData.allowed_extensions = @($ExtensionId)
+} else {
+    $manifestData.allowed_origins = @("chrome-extension://$ExtensionId/")
+}
+$manifest = $manifestData | ConvertTo-Json -Depth 2
+
+Set-Content -Path $ManifestPath -Value $manifest -Encoding UTF8
+
+# Create registry key pointing to manifest
+if (-not (Test-Path (Split-Path $RegKey))) {
+    New-Item -Path (Split-Path $RegKey) -Force | Out-Null
+}
+New-Item -Path $RegKey -Force | Out-Null
+Set-ItemProperty -Path $RegKey -Name "(Default)" -Value $ManifestPath
+
+Write-Host "Installed native messaging host:"
+Write-Host "  Manifest: $ManifestPath"
+Write-Host "  Registry: $RegKey"
+Write-Host "  Browser: $Browser"
+Write-Host "  Host script: $HostPy"
+Write-Host "  Extension ID: $ExtensionId"
+
+# --- Install HTTP daemon via Scheduled Task ---
+$DaemonPy = Join-Path $ScriptDir "xtap_daemon.py"
+$XtapDir = Join-Path $HOME ".xtap"
+$XtapSecret = Join-Path $XtapDir "secret"
+$TaskName = "xTapDaemon"
+
+# Create ~/.xtap/ directory
+if (-not (Test-Path $XtapDir)) {
+    New-Item -ItemType Directory -Path $XtapDir -Force | Out-Null
+}
+
+# Generate auth token if not exists
+if (-not (Test-Path $XtapSecret)) {
+    $token = python -c "import secrets; print(secrets.token_urlsafe(32))"
+    Set-Content -Path $XtapSecret -Value $token -Encoding ASCII
+    Write-Host "Generated auth token: $XtapSecret"
+} else {
+    Write-Host "Auth token already exists: $XtapSecret"
+}
+
+# Find pythonw (preferred — no console window) or fall back to python
+$PythonW = Get-Command pythonw -ErrorAction SilentlyContinue
+if ($PythonW) {
+    $PythonExe = $PythonW.Source
+} else {
+    $PythonExe = (Get-Command python).Source
+}
+
+# Remove existing scheduled task if present
+if (Get-ScheduledTask -TaskName $TaskName -ErrorAction SilentlyContinue) {
+    Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false
+}
+
+# Create scheduled task: run at logon, restart on failure
+$Action = New-ScheduledTaskAction -Execute $PythonExe -Argument "`"$DaemonPy`""
+$Trigger = New-ScheduledTaskTrigger -AtLogOn -User $env:USERNAME
+$Settings = New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries `
+    -ExecutionTimeLimit ([TimeSpan]::Zero) -RestartCount 3 -RestartInterval ([TimeSpan]::FromMinutes(1))
+Register-ScheduledTask -TaskName $TaskName -Action $Action -Trigger $Trigger `
+    -Settings $Settings -Description "xTap HTTP daemon" | Out-Null
+
+# Start the task now
+Start-ScheduledTask -TaskName $TaskName
+
+Write-Host ""
+Write-Host "HTTP daemon installed:"
+Write-Host "  Scheduled Task: $TaskName"
+Write-Host "  Daemon: $DaemonPy"
+Write-Host "  Python: $PythonExe"
+Write-Host "  Listening on: 127.0.0.1:17381"
+Write-Host ""
+Write-Host "Useful commands:"
+Write-Host "  Get-ScheduledTask -TaskName $TaskName"
+Write-Host "  Start-ScheduledTask -TaskName $TaskName"
+Write-Host "  Stop-ScheduledTask -TaskName $TaskName"
+
+Write-Host ""
+$outputDir = if ($env:XTAP_OUTPUT_DIR) { $env:XTAP_OUTPUT_DIR } else { Join-Path $HOME "Downloads\xtap" }
+Write-Host "Output directory:"
+Write-Host "  $outputDir"
+Write-Host "To change it, set XTAP_OUTPUT_DIR as a *user* environment variable"
+Write-Host "(scheduled tasks do not see shell-session variables), then re-run:"
+Write-Host "  [Environment]::SetEnvironmentVariable('XTAP_OUTPUT_DIR', 'D:\path', 'User')"

extension/native-host/install.sh

@@ -0,0 +1,280 @@
+#!/bin/bash
+# xTap — installer for the native messaging host and HTTP daemon (macOS / Linux).
+# Usage:
+#   ./install.sh <extension-id> [chrome|firefox]
+#   ./install.sh firefox [extension-id]
+
+set -euo pipefail
+
+if [ "$(id -u)" -eq 0 ]; then
+  echo "Error: Do not run this script with sudo. It installs to user-level paths."
+  exit 1
+fi
+
+if [ $# -lt 1 ]; then
+  echo "Usage:"
+  echo "  $0 <extension-id> [chrome|firefox]"
+  echo "  $0 firefox [extension-id]"
+  echo ""
+  echo "Chrome ID: chrome://extensions (Developer mode)"
+  echo "Firefox ID default: xtap@mkubicek.dev (from manifest.firefox.json)"
+  exit 1
+fi
+
+BROWSER="chrome"
+EXT_ID=""
+if [ "$1" = "chrome" ] || [ "$1" = "firefox" ]; then
+  BROWSER="$1"
+  EXT_ID="${2:-}"
+else
+  EXT_ID="$1"
+  if [ $# -ge 2 ]; then
+    BROWSER="$2"
+  fi
+fi
+
+if [ "$BROWSER" != "chrome" ] && [ "$BROWSER" != "firefox" ]; then
+  echo "Error: browser must be 'chrome' or 'firefox'"
+  exit 1
+fi
+
+if [ "$BROWSER" = "firefox" ] && [ -z "$EXT_ID" ]; then
+  EXT_ID="xtap@mkubicek.dev"
+fi
+
+if [ -z "$EXT_ID" ]; then
+  echo "Error: extension ID is required for Chrome installs."
+  echo "Find it at chrome://extensions (enable Developer mode)."
+  exit 1
+fi
+
+HOST_NAME="com.xtap.host"
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+HOST_PATH="${SCRIPT_DIR}/xtap_host.py"
+
+OS="$(uname)"
+case "$OS" in
+  Darwin)
+    if [ "$BROWSER" = "firefox" ]; then
+      TARGET_DIR="$HOME/Library/Application Support/Mozilla/NativeMessagingHosts"
+    else
+      TARGET_DIR="$HOME/Library/Application Support/Google/Chrome/NativeMessagingHosts"
+    fi
+    ;;
+  Linux)
+    if [ "$BROWSER" = "firefox" ]; then
+      TARGET_DIR="$HOME/.mozilla/native-messaging-hosts"
+    else
+      TARGET_DIR="$HOME/.config/google-chrome/NativeMessagingHosts"
+    fi
+    ;;
+  *)
+    echo "Error: Unsupported OS '$OS'. Use install.ps1 on Windows."
+    exit 1
+    ;;
+esac
+MANIFEST_PATH="${TARGET_DIR}/${HOST_NAME}.json"
+
+# Verify python3
+PYTHON_PATH="$(command -v python3 2>/dev/null || true)"
+if [ -z "$PYTHON_PATH" ]; then
+  echo "Error: python3 is required but not found in PATH"
+  exit 1
+fi
+
+# Make host executable
+chmod +x "$HOST_PATH"
+
+# --- Generate wrapper script with absolute Python path ---
+# Chrome and Firefox launch native messaging hosts with a minimal environment
+# (typically PATH=/usr/bin:/bin:/usr/sbin:/sbin).  The shebang
+# #!/usr/bin/env python3 can fail if python3 lives in /usr/local/bin,
+# Homebrew, pyenv, or a framework install.  A thin wrapper with absolute
+# paths avoids this entirely.
+XTAP_DIR_WRAPPER="$HOME/.xtap"
+WRAPPER_PATH="${XTAP_DIR_WRAPPER}/xtap_host_wrapper.sh"
+
+mkdir -p "$XTAP_DIR_WRAPPER"
+
+cat > "$WRAPPER_PATH" <<WRAPPER
+#!/bin/sh
+# xTap native messaging host wrapper — generated by install.sh
+# Launches xtap_host.py with an absolute Python path so the native
+# messaging protocol works even in Chrome/Firefox's minimal environment.
+exec "${PYTHON_PATH}" "${HOST_PATH}" "\$@"
+WRAPPER
+
+chmod +x "$WRAPPER_PATH"
+
+# Create target directory
+mkdir -p "$TARGET_DIR"
+
+# Write native messaging manifest (pointing at the wrapper, not the .py)
+if [ "$BROWSER" = "firefox" ]; then
+  cat > "$MANIFEST_PATH" <<EOF
+{
+  "name": "${HOST_NAME}",
+  "description": "xTap native messaging host — passes the HTTP daemon auth token to the extension",
+  "path": "${WRAPPER_PATH}",
+  "type": "stdio",
+  "allowed_extensions": ["${EXT_ID}"]
+}
+EOF
+else
+  cat > "$MANIFEST_PATH" <<EOF
+{
+  "name": "${HOST_NAME}",
+  "description": "xTap native messaging host — passes the HTTP daemon auth token to the extension",
+  "path": "${WRAPPER_PATH}",
+  "type": "stdio",
+  "allowed_origins": ["chrome-extension://${EXT_ID}/"]
+}
+EOF
+fi
+
+echo "Installed native messaging host manifest to:"
+echo "  $MANIFEST_PATH"
+echo ""
+echo "Browser: $BROWSER"
+echo "Host wrapper: $WRAPPER_PATH"
+echo "Host script: $HOST_PATH"
+echo "Python: $PYTHON_PATH"
+echo "Extension ID: $EXT_ID"
+
+# --- macOS: install HTTP daemon via launchd ---
+if [ "$OS" = "Darwin" ]; then
+  DAEMON_PATH="${SCRIPT_DIR}/xtap_daemon.py"
+  XTAP_DIR="$HOME/.xtap"
+  XTAP_SECRET="${XTAP_DIR}/secret"
+  PLIST_LABEL="com.xtap.daemon"
+  PLIST_DEST="$HOME/Library/LaunchAgents/${PLIST_LABEL}.plist"
+  PLIST_TEMPLATE="${SCRIPT_DIR}/com.xtap.daemon.plist"
+
+  chmod +x "$DAEMON_PATH"
+
+  # Create ~/.xtap/ with restricted permissions
+  mkdir -p "$XTAP_DIR"
+  chmod 700 "$XTAP_DIR"
+
+  # Generate auth token if not exists
+  if [ ! -f "$XTAP_SECRET" ]; then
+    python3 -c "import secrets; print(secrets.token_urlsafe(32))" > "$XTAP_SECRET"
+    chmod 600 "$XTAP_SECRET"
+    echo "Generated auth token: $XTAP_SECRET"
+  else
+    echo "Auth token already exists: $XTAP_SECRET"
+  fi
+
+  # Unload existing daemon if loaded (ignore errors)
+  launchctl bootout "gui/$(id -u)/${PLIST_LABEL}" 2>/dev/null || true
+  # Wait for the service to fully unload before re-bootstrapping
+  for i in 1 2 3 4 5; do
+    launchctl print "gui/$(id -u)/${PLIST_LABEL}" 2>/dev/null || break
+    sleep 0.5
+  done
+
+  # Capture user's PATH so the daemon can find yt-dlp and other tools
+  USER_PATH="$PATH"
+  XTAP_LOG_LEVEL="${XTAP_LOG_LEVEL:-info}"
+  # Bake the output dir into the service environment — launchd does not
+  # inherit shell exports, so this is the only way XTAP_OUTPUT_DIR reaches
+  # the daemon (re-run install.sh after changing it).
+  XTAP_OUTPUT_DIR="${XTAP_OUTPUT_DIR:-$HOME/Downloads/xtap}"
+  # XML-escape for the plist, then sed-escape (&, \, |) so paths containing
+  # those characters can't corrupt the substitution or the plist.
+  _OUT_XML=$(printf '%s' "$XTAP_OUTPUT_DIR" | sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g')
+  _OUT_SED=$(printf '%s' "$_OUT_XML" | sed -e 's/[\\&|]/\\&/g')
+
+  # Substitute plist template
+  mkdir -p "$HOME/Library/LaunchAgents"
+  sed \
+    -e "s|__PYTHON_PATH__|${PYTHON_PATH}|g" \
+    -e "s|__DAEMON_PATH__|${DAEMON_PATH}|g" \
+    -e "s|__HOME_DIR__|${HOME}|g" \
+    -e "s|__PATH__|${USER_PATH}|g" \
+    -e "s|__LOG_LEVEL__|${XTAP_LOG_LEVEL}|g" \
+    -e "s|__OUTPUT_DIR__|${_OUT_SED}|g" \
+    "$PLIST_TEMPLATE" > "$PLIST_DEST"
+
+  # Load daemon
+  launchctl bootstrap "gui/$(id -u)" "$PLIST_DEST"
+
+  echo ""
+  echo "HTTP daemon installed:"
+  echo "  Plist: $PLIST_DEST"
+  echo "  Daemon: $DAEMON_PATH"
+  echo "  Listening on: 127.0.0.1:17381"
+  echo "  Logs: ${XTAP_DIR}/daemon-stderr.log"
+fi
+
+# --- Linux: install HTTP daemon via systemd user service ---
+if [ "$OS" = "Linux" ]; then
+  DAEMON_PATH="${SCRIPT_DIR}/xtap_daemon.py"
+  XTAP_DIR="$HOME/.xtap"
+  XTAP_SECRET="${XTAP_DIR}/secret"
+  SERVICE_NAME="com.xtap.daemon"
+  SERVICE_TEMPLATE="${SCRIPT_DIR}/${SERVICE_NAME}.service"
+  SERVICE_DIR="$HOME/.config/systemd/user"
+  SERVICE_DEST="${SERVICE_DIR}/${SERVICE_NAME}.service"
+
+  chmod +x "$DAEMON_PATH"
+
+  # Create ~/.xtap/ with restricted permissions
+  mkdir -p "$XTAP_DIR"
+  chmod 700 "$XTAP_DIR"
+
+  # Generate auth token if not exists
+  if [ ! -f "$XTAP_SECRET" ]; then
+    python3 -c "import secrets; print(secrets.token_urlsafe(32))" > "$XTAP_SECRET"
+    chmod 600 "$XTAP_SECRET"
+    echo "Generated auth token: $XTAP_SECRET"
+  else
+    echo "Auth token already exists: $XTAP_SECRET"
+  fi
+
+  # Capture user's PATH so the daemon can find yt-dlp and other tools
+  USER_PATH="$PATH"
+  XTAP_LOG_LEVEL="${XTAP_LOG_LEVEL:-info}"
+  # Bake the output dir into the service environment — systemd does not
+  # inherit shell exports, so this is the only way XTAP_OUTPUT_DIR reaches
+  # the daemon (re-run install.sh after changing it). The template quotes the
+  # Environment= value (paths with spaces); escape sed metachars and any
+  # embedded double quotes so the substitution can't corrupt the unit file.
+  XTAP_OUTPUT_DIR="${XTAP_OUTPUT_DIR:-$HOME/Downloads/xtap}"
+  _OUT_SED=$(printf '%s' "$XTAP_OUTPUT_DIR" | sed -e 's/[\\&|]/\\&/g' -e 's/"/\\\\"/g')
+
+  # Substitute service template
+  mkdir -p "$SERVICE_DIR"
+  sed \
+    -e "s|__PYTHON_PATH__|${PYTHON_PATH}|g" \
+    -e "s|__DAEMON_PATH__|${DAEMON_PATH}|g" \
+    -e "s|__HOME_DIR__|${HOME}|g" \
+    -e "s|__PATH__|${USER_PATH}|g" \
+    -e "s|__LOG_LEVEL__|${XTAP_LOG_LEVEL}|g" \
+    -e "s|__OUTPUT_DIR__|${_OUT_SED}|g" \
+    "$SERVICE_TEMPLATE" > "$SERVICE_DEST"
+
+  # Reload and enable
+  systemctl --user daemon-reload
+  systemctl --user enable --now "$SERVICE_NAME"
+  # enable --now does not restart an already-running unit — restart so a
+  # re-install actually applies the freshly baked environment.
+  systemctl --user try-restart "$SERVICE_NAME" 2>/dev/null || true
+
+  echo ""
+  echo "HTTP daemon installed:"
+  echo "  Service: $SERVICE_DEST"
+  echo "  Daemon: $DAEMON_PATH"
+  echo "  Listening on: 127.0.0.1:17381"
+  echo ""
+  echo "Useful commands:"
+  echo "  systemctl --user status $SERVICE_NAME"
+  echo "  journalctl --user -u $SERVICE_NAME -f"
+fi
+
+echo ""
+echo "Output directory (set XTAP_OUTPUT_DIR and re-run install to change):"
+echo "  ${XTAP_OUTPUT_DIR:-$HOME/Downloads/xtap}"
+echo ""
+echo "Debug logging (re-run install after setting):"
+echo "  export XTAP_LOG_LEVEL=debug"

extension/native-host/xtap_core.py

@@ -0,0 +1,836 @@
+"""xTap Core — shared file I/O logic used by both native host and HTTP daemon."""
+
+import glob
+import json
+import os
+import queue
+import re
+import shutil
+import subprocess
+import sys
+import threading
+import time
+import urllib.error
+import urllib.request
+from datetime import date, datetime, timezone
+from urllib.parse import urlparse
+
+
+# realpath so the default dir and an explicit outputDir naming the same
+# physical directory (e.g. via a symlinked home) share one seen-ID set —
+# resolve_output_dir keys per-directory state by this string.
+DEFAULT_OUTPUT_DIR = os.path.realpath(
+    os.environ.get('XTAP_OUTPUT_DIR', os.path.expanduser('~/Downloads/xtap')))
+
+# Allowed roots for outputDir validation: user's home + DEFAULT_OUTPUT_DIR
+# (the latter covers XTAP_OUTPUT_DIR pointing outside home, e.g. /data/xtap)
+_ALLOWED_ROOTS = tuple(dict.fromkeys([
+    os.path.realpath(os.path.expanduser('~')),
+    DEFAULT_OUTPUT_DIR,  # already realpath-resolved at definition
+]))
+
+
+def _article_seen_key(tweet_id):
+    return f'article:{tweet_id}'
+
+
+def validate_output_dir(path):
+    """Validate that a resolved path is under an allowed root directory.
+
+    Args:
+        path: The path to validate (should already be expanduser'd).
+
+    Returns:
+        The realpath-resolved path.
+
+    Raises:
+        ValueError: If the path resolves outside all allowed roots.
+    """
+    resolved = os.path.realpath(path)
+    for root in _ALLOWED_ROOTS:
+        # Append os.sep so '/home/userX' doesn't match root '/home/user'
+        if resolved == root or resolved.startswith(root + os.sep):
+            return resolved
+    raise ValueError(
+        f'outputDir resolves outside allowed directories: {resolved}'
+    )
+
+
+def load_seen_ids(out_dir):
+    """Build a set of tweet IDs from all existing JSONL files in the output directory."""
+    seen = set()
+    escaped_dir = glob.escape(out_dir)
+    # Files evicted by iCloud ("Optimize Mac Storage") become hidden
+    # .name.icloud placeholders the glob below can't read — dedup silently
+    # loses their IDs, so at least make the degradation visible.
+    placeholders = glob.glob(os.path.join(escaped_dir, '.tweets-*.jsonl.icloud'))
+    if placeholders:
+        names = ', '.join(os.path.basename(p) for p in placeholders[:5])
+        print(f'[xtap] WARNING: {len(placeholders)} tweets file(s) evicted by '
+              f'iCloud — their IDs cannot be deduplicated against: {names}',
+              file=sys.stderr)
+    for path in glob.glob(os.path.join(escaped_dir, 'tweets-*.jsonl')):
+        try:
+            # errors='replace': a corrupt file must not crash the daemon at
+            # startup — unreadable lines just fail JSON parsing and are skipped.
+            with open(path, 'r', encoding='utf-8', errors='replace') as f:
+                for line in f:
+                    line = line.strip()
+                    if not line:
+                        continue
+                    try:
+                        tweet = json.loads(line)
+                        tweet_id = tweet.get('id')
+                        if tweet_id:
+                            seen.add(tweet_id)
+                            if tweet.get('is_article'):
+                                seen.add(_article_seen_key(tweet_id))
+                    except (json.JSONDecodeError, KeyError):
+                        continue
+        except OSError as e:
+            print(f'[xtap] WARNING: cannot read {path}: {e}', file=sys.stderr)
+    return seen
+
+
+def resolve_output_dir(msg_dir, default_dir, seen_ids_by_dir):
+    """Resolve output directory from message, scoping seen IDs per directory.
+
+    seen_ids_by_dir maps out_dir -> set of tweet IDs already written there.
+    Per-directory scoping matters: after switching to a new output directory,
+    tweets already captured into the old one must still be written to the new
+    archive (a shared set would silently suppress them).
+
+    Returns (out_dir, seen_ids) for the resolved directory.
+    """
+    if msg_dir:
+        out_dir = validate_output_dir(os.path.expanduser(msg_dir))
+        os.makedirs(out_dir, exist_ok=True)
+    else:
+        out_dir = default_dir
+    if out_dir not in seen_ids_by_dir:
+        seen_ids_by_dir[out_dir] = load_seen_ids(out_dir)
+    return out_dir, seen_ids_by_dir[out_dir]
+
+
+def write_tweets(tweets, out_dir, seen_ids):
+    """Write tweets to JSONL, deduplicating against seen_ids. Returns (count, dupes).
+
+    seen_ids is committed only after the file has been flushed and closed —
+    if the write fails, the extension retries the batch, and pre-committed IDs
+    would make the retry look like all-duplicates (silent data loss). A partial
+    flush before a failure can yield duplicate lines on retry; duplicates are
+    recoverable, dropped tweets are not.
+    """
+    out_file = os.path.join(out_dir, f'tweets-{date.today().isoformat()}.jsonl')
+    count = 0
+    dupes = 0
+    written_ids = set()
+    with open(out_file, 'a', encoding='utf-8') as f:
+        for tweet in tweets:
+            tid = tweet.get('id')
+            article_key = _article_seen_key(tid) if tid and tweet.get('is_article') else None
+            if tid and (tid in seen_ids or tid in written_ids):
+                if not article_key or article_key in seen_ids or article_key in written_ids:
+                    dupes += 1
+                    continue
+            f.write(json.dumps(tweet, ensure_ascii=False) + '\n')
+            if tid:
+                written_ids.add(tid)
+            if article_key:
+                written_ids.add(article_key)
+            count += 1
+    seen_ids |= written_ids
+    return count, dupes
+
+
+def write_log(lines, out_dir):
+    """Append debug log lines to daily log file. Returns logged count."""
+    log_file = os.path.join(out_dir, f'debug-{date.today().isoformat()}.log')
+    with open(log_file, 'a', encoding='utf-8') as f:
+        for line in lines:
+            f.write(line + '\n')
+    return len(lines)
+
+
+def write_dump(filename, content, out_dir):
+    """Write a raw JSON dump file for discovery/debugging."""
+    # Strip path components — only the basename is allowed
+    safe_name = os.path.basename(filename)
+    if not safe_name or safe_name in ('.', '..'):
+        raise ValueError(f'Invalid dump filename: {filename!r}')
+    dump_file = os.path.join(out_dir, safe_name)
+    with open(dump_file, 'w', encoding='utf-8') as f:
+        f.write(content)
+    return dump_file
+
+
+def test_path(out_dir):
+    """Test that we can write to the output directory. Raises on failure."""
+    os.makedirs(out_dir, exist_ok=True)
+    test_file = os.path.join(out_dir, f'.xtap-write-test-{threading.get_ident()}')
+    try:
+        with open(test_file, 'w', encoding='utf-8') as f:
+            f.write('ok')
+    finally:
+        try:
+            os.remove(test_file)
+        except FileNotFoundError:
+            pass
+
+
+# --- Video download ---
+
+VIDEO_REQUEST_TIMEOUT_S = 60
+VIDEO_CHUNK_SIZE = 64 * 1024
+DEFAULT_MAX_VIDEO_MB = 500
+TWEET_STATUS_HOSTS = frozenset({'x.com', 'twitter.com'})
+
+_ytdlp_path = None
+_ytdlp_checked = False
+_downloads = {}
+_downloads_lock = threading.Lock()
+
+
+def check_ytdlp():  # pragma: no cover
+    """Check if yt-dlp is available on PATH. Cached after first call."""
+    global _ytdlp_path, _ytdlp_checked
+    if not _ytdlp_checked:
+        _ytdlp_path = shutil.which('yt-dlp')
+        _ytdlp_checked = True
+    return _ytdlp_path is not None
+
+
+def get_download_status(download_id):
+    """Return current state of a download."""
+    with _downloads_lock:
+        info = _downloads.get(download_id)
+        if not info:
+            return {'status': 'unknown'}
+        return {
+            'status': info['status'],
+            'progress': info.get('progress'),
+            'path': info.get('path'),
+            'error': info.get('error'),
+        }
+
+
+def _date_prefix(post_date):
+    """Convert ISO date string to yyyy.mm.dd prefix, or empty string on failure."""
+    if not isinstance(post_date, str):
+        return ''
+    match = re.match(r'^(\d{4})-(\d{2})-(\d{2})(?:T|$)', post_date)
+    if not match:
+        return ''
+    year, month, day = match.groups()
+    try:
+        datetime.strptime(f'{year}-{month}-{day}', '%Y-%m-%d')
+    except ValueError:
+        return ''
+    return f'{year}.{month}.{day}_'
+
+
+def _tweet_status_id(tweet_url):
+    """Validate an X/Twitter status URL and return its numeric tweet ID."""
+    if not isinstance(tweet_url, str):
+        raise ValueError('tweetUrl must be an X/Twitter status URL')
+    parsed = urlparse(tweet_url)
+    host = (parsed.hostname or '').lower()
+    if parsed.scheme != 'https' or host not in TWEET_STATUS_HOSTS:
+        raise ValueError(f'tweetUrl not allowed: {tweet_url!r}')
+    match = re.search(r'/(?:[^/?#]+/)*status/(\d+)(?:/|$)', parsed.path)
+    if not match:
+        raise ValueError(f'tweetUrl must be a status URL: {tweet_url!r}')
+    return match.group(1)
+
+
+def validate_tweet_url(tweet_url):
+    """Public validation hook for daemon request handling."""
+    _tweet_status_id(tweet_url)
+    return tweet_url
+
+
+def _safe_video_path(video_dir, filename):
+    if (not filename or os.path.basename(filename) != filename or
+            '/' in filename or '\\' in filename):
+        raise ValueError(f'unsafe video filename: {filename!r}')
+    base = os.path.realpath(video_dir)
+    candidate = os.path.realpath(os.path.join(video_dir, filename))
+    try:
+        if os.path.commonpath([base, candidate]) != base:
+            raise ValueError(f'video path escapes output directory: {filename!r}')
+    except ValueError:
+        raise ValueError(f'video path escapes output directory: {filename!r}')
+    return candidate
+
+
+def _video_max_bytes():
+    max_video_mb = _env_float('XTAP_MAX_VIDEO_MB', float(DEFAULT_MAX_VIDEO_MB), label='video')
+    if max_video_mb <= 0:
+        return None
+    return int(max_video_mb * 1024 * 1024)
+
+
+def download_direct(direct_url, tweet_id, video_dir, post_date=''):
+    """Download video via direct CDN URL. Returns the file path."""
+    # Same trust boundary as the image downloader: only fetch from Twitter's
+    # CDN over https — the URL crosses the daemon's HTTP API and must not be
+    # able to point the daemon at file:// or internal hosts.
+    parsed = urlparse(direct_url)
+    host = (parsed.hostname or '').lower()
+    if parsed.scheme != 'https' or not (host == 'twimg.com' or host.endswith('.twimg.com')):
+        raise ValueError(f'direct URL not allowed: {direct_url!r}')
+    os.makedirs(video_dir, exist_ok=True)
+    prefix = _date_prefix(post_date)
+    filename = f'{prefix}{tweet_id}.mp4'
+    filepath = _safe_video_path(video_dir, filename)
+    tmp_path = filepath + '.part'
+    max_bytes = _video_max_bytes()
+    # _NO_REDIRECT_OPENER (not bare urlopen) so a redirect can't bounce the
+    # daemon off the allowlisted host — the host check only validates the
+    # initial URL. Socket-level timeout so a stalled CDN can't hang the
+    # download thread (and the popup's polling) forever.
+    try:
+        with _NO_REDIRECT_OPENER.open(direct_url, timeout=VIDEO_REQUEST_TIMEOUT_S) as resp, \
+                open(tmp_path, 'wb') as f:
+            headers = getattr(resp, 'headers', {}) or {}
+            content_length = headers.get('Content-Length')
+            if max_bytes is not None and content_length and content_length.isdigit():
+                if int(content_length) > max_bytes:
+                    raise ValueError('video exceeds XTAP_MAX_VIDEO_MB')
+            written = 0
+            while True:
+                chunk = resp.read(VIDEO_CHUNK_SIZE)
+                if not chunk:
+                    break
+                written += len(chunk)
+                if max_bytes is not None and written > max_bytes:
+                    raise ValueError('video exceeds XTAP_MAX_VIDEO_MB')
+                f.write(chunk)
+        os.replace(tmp_path, filepath)
+    except Exception:
+        _safe_unlink(tmp_path)
+        raise
+    return filepath
+
+
+def start_download(download_id, tweet_url, direct_url, out_dir, post_date=''):  # pragma: no cover
+    """Start a background download. Returns immediately; poll get_download_status()."""
+    tweet_id = _tweet_status_id(tweet_url)
+    video_dir = os.path.join(out_dir, 'videos')
+    os.makedirs(video_dir, exist_ok=True)
+
+    with _downloads_lock:
+        _downloads[download_id] = {
+            'status': 'downloading',
+            'progress': None,
+            'path': None,
+            'error': None,
+        }
+
+    def run():
+        try:
+            if check_ytdlp():
+                _download_with_ytdlp(download_id, tweet_url, video_dir, post_date)
+            elif direct_url:
+                with _downloads_lock:
+                    _downloads[download_id]['progress'] = 0
+                path = download_direct(direct_url, tweet_id, video_dir, post_date)
+                with _downloads_lock:
+                    _downloads[download_id].update(
+                        progress=100, status='done', path=path)
+            else:
+                with _downloads_lock:
+                    _downloads[download_id].update(
+                        status='error',
+                        error='yt-dlp not found and no direct URL available')
+        except Exception as e:
+            with _downloads_lock:
+                _downloads[download_id].update(
+                    status='error', error=str(e))
+
+    t = threading.Thread(target=run, daemon=True)
+    t.start()
+
+
+class _YtdlpProgress:
+    """Parse yt-dlp stdout and scale progress across multiple HLS streams.
+
+    Call feed(line) for each stripped stdout line.  Read .progress for the
+    current 0-100 value and .final_path for the last detected output path.
+    """
+
+    _progress_re = re.compile(r'(\d+\.?\d*)%')
+    _format_re = re.compile(r'Downloading \d+ format\(s\): (.+)')
+
+    def __init__(self):
+        self.total_streams = 1
+        self.stream_index = 0
+        self.dest_count = 0
+        self.progress = 0.0
+        self.final_path = None
+
+    def feed(self, line: str):
+        # Detect multi-stream downloads by counting '+' in format string
+        # ("Downloading 1 format(s): hls-707+hls-audio" -> 2 streams)
+        if line.startswith('[info]'):
+            fm = self._format_re.search(line)
+            if fm:
+                self.total_streams = max(1, fm.group(1).count('+') + 1)
+
+        # Parse progress percentage, scaled across streams.
+        # Exclude Destination / already-downloaded lines so a tweet title
+        # containing "%" (e.g. "100% agree") doesn't poison progress.
+        if line.startswith('[download]') and 'Destination:' not in line and 'has already been downloaded' not in line:
+            m = self._progress_re.search(line)
+            if m:
+                raw_pct = float(m.group(1))
+                pct = min((self.stream_index * 100 + raw_pct) / self.total_streams, 100.0)
+                # Never report backward progress (safety net if stream
+                # count detection is wrong or yt-dlp format changes)
+                if pct >= self.progress:
+                    self.progress = pct
+
+        # Capture output filename from [download] or [Merger] lines
+        if 'Destination:' in line:
+            self.dest_count += 1
+            if self.dest_count > self.total_streams:
+                # More streams than the format line indicated (or format
+                # line was missing).  Recalibrate so the monotonic guard
+                # doesn't lock progress at 100% for the rest of the download.
+                self.total_streams = self.dest_count
+                self.progress = min(
+                    self.progress,
+                    (self.dest_count - 1) * 100.0 / self.total_streams)
+            self.stream_index = min(self.dest_count - 1, self.total_streams - 1)
+            self.final_path = line.split('Destination:', 1)[1].strip()
+        elif 'has already been downloaded' in line:
+            self.dest_count += 1
+            if self.dest_count > self.total_streams:
+                self.total_streams = self.dest_count
+            self.stream_index = min(self.dest_count - 1, self.total_streams - 1)
+            # Count cached stream as complete for progress
+            pct = min((self.stream_index + 1) * 100.0 / self.total_streams, 100.0)
+            if pct >= self.progress:
+                self.progress = pct
+            # "[download] <path> has already been downloaded"
+            part = line.split(']', 1)[1].strip() if ']' in line else line
+            self.final_path = part.replace(' has already been downloaded', '').strip()
+        elif '[Merger]' in line and 'Merging formats into' in line:
+            self.final_path = line.split('Merging formats into "', 1)[1].rstrip('"').strip() if '"' in line else self.final_path
+
+
+def _download_with_ytdlp(download_id, tweet_url, video_dir, post_date=''):  # pragma: no cover
+    """Download using yt-dlp with progress parsing.
+
+    Downloads into a .downloading/ staging subdirectory so partial files
+    are not visible in video_dir until the download is fully complete.
+    """
+    tweet_id = _tweet_status_id(tweet_url)
+    staging_dir = os.path.join(video_dir, '.downloading')
+    os.makedirs(staging_dir, exist_ok=True)
+    prefix = _date_prefix(post_date)
+    # Pin the tweet status ID in the filename rather than relying on %(id)s,
+    # which some yt-dlp Twitter sub-extractors (amplify/broadcast/card) fill
+    # with a media or broadcast ID instead of the tweet ID.
+    output_template = _safe_video_path(
+        staging_dir,
+        prefix + '%(title)s [' + tweet_id + '].%(ext)s')
+    cmd = [
+        _ytdlp_path,
+        '--newline', '--progress',
+        '--cookies-from-browser', 'chrome',
+        '-o', output_template,
+        tweet_url,
+    ]
+    proc = subprocess.Popen(
+        cmd,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        text=True,
+    )
+    tracker = _YtdlpProgress()
+    last_lines = []
+    for line in proc.stdout:
+        line = line.strip()
+        if line:
+            last_lines.append(line)
+            if len(last_lines) > 20:
+                last_lines.pop(0)
+            print(f'[yt-dlp] {line}', file=sys.stderr)
+        tracker.feed(line)
+        with _downloads_lock:
+            _downloads[download_id]['progress'] = tracker.progress
+    proc.wait()
+    if proc.returncode != 0:
+        # Include yt-dlp's error output in the exception
+        error_lines = [l for l in last_lines if 'ERROR' in l]
+        detail = error_lines[-1] if error_lines else (last_lines[-1] if last_lines else '')
+        raise RuntimeError(f'yt-dlp failed: {detail}' if detail else f'yt-dlp exited with code {proc.returncode}')
+    # Move completed file from staging dir to final video_dir
+    final_path = tracker.final_path
+    if final_path and os.path.isfile(final_path):
+        staging_real = os.path.realpath(staging_dir)
+        final_real = os.path.realpath(final_path)
+        try:
+            if os.path.commonpath([staging_real, final_real]) != staging_real:
+                raise RuntimeError('yt-dlp output escaped staging directory')
+        except ValueError:
+            raise RuntimeError('yt-dlp output escaped staging directory')
+        dest_path = _safe_video_path(video_dir, os.path.basename(final_path))
+        shutil.move(final_real, dest_path)
+        final_path = dest_path
+    with _downloads_lock:
+        _downloads[download_id].update(
+            progress=100, status='done', path=final_path)
+
+
+# --- Image download ---
+
+# Strip a trailing :orig / :large / :medium etc. suffix Twitter appends to media URLs.
+_TWIMG_SIZE_SUFFIX_RE = re.compile(r':(orig|large|medium|small|thumb|tiny)$')
+
+# Twitter snowflake IDs are numeric; reject anything else to block path traversal.
+_TWEET_ID_RE = re.compile(r'^[0-9]+$')
+
+# Only fetch from these CDN hosts. Anything else (including redirects) is rejected.
+ALLOWED_IMAGE_HOSTS = frozenset({'pbs.twimg.com'})
+
+
+def _env_int(name, default):
+    """Parse an int env var, falling back to default on missing/empty/invalid."""
+    raw = (os.environ.get(name) or '').strip()
+    if not raw:
+        return default
+    try:
+        return int(raw)
+    except ValueError:
+        print(f'[xtap:image] invalid {name}={raw!r}, using {default}', file=sys.stderr)
+        return default
+
+
+def _env_float(name, default, label='image'):
+    raw = (os.environ.get(name) or '').strip()
+    if not raw:
+        return default
+    try:
+        return float(raw)
+    except ValueError:
+        print(f'[xtap:{label}] invalid {name}={raw!r}, using {default}', file=sys.stderr)
+        return default
+
+
+def _photo_filename(url):
+    """Extract the CDN filename from a pbs.twimg.com photo URL.
+
+    Returns the basename without any trailing :size suffix, or None if the URL
+    has no usable filename or contains traversal components.
+    """
+    if not url:
+        return None
+    cleaned = _TWIMG_SIZE_SUFFIX_RE.sub('', url)
+    path = urlparse(cleaned).path or cleaned
+    name = os.path.basename(path)
+    if not name or name in ('.', '..'):
+        return None
+    # Reject anything with path separators or that would still resolve to a
+    # parent directory after basename (defense in depth).
+    if '/' in name or '\\' in name or name.startswith('.'):
+        return None
+    return name
+
+
+def _is_safe_rel_path(out_dir, rel_path):
+    """Verify joining rel_path with out_dir stays under out_dir.
+
+    Catches absolute paths (`/etc/...`), traversal (`../foo`), and Windows
+    drive-letter paths.
+    """
+    if not rel_path or os.path.isabs(rel_path):
+        return False
+    out_real = os.path.realpath(out_dir)
+    candidate = os.path.realpath(os.path.join(out_real, rel_path))
+    try:
+        return os.path.commonpath([out_real, candidate]) == out_real
+    except ValueError:  # pragma: no cover — Windows-only: different drives
+        return False
+
+
+def _strip_article_text_local_path(article, rel_path):
+    text = article.get('text')
+    if not isinstance(text, str):
+        return
+    pattern = re.compile(r'!\[[^\]\n]*\]\(' + re.escape(rel_path) + r'\)')
+    article['text'] = pattern.sub('', text)
+
+
+def collect_image_jobs(tweets, out_dir):
+    """Compute the list of image download jobs for a batch of tweets.
+
+    For top-level photo media the rel_path is derived by convention
+    (`media/<tweet_id>/<cdn_filename>`) — the JSONL never carries a
+    redundant `local_path` field for these. Consumers who need the path
+    can reconstruct it from `id` + `basename(url)`.
+
+    Article media items (under `tweet.article.media[]`) already have
+    `local_path` set by the JS parser because that path is also embedded
+    in the article's rendered Markdown text (`![](media/<id>/file.png)`).
+    Those entries are validated here; if the supplied `local_path` would
+    escape `out_dir`, the field and its rendered Markdown image reference
+    are stripped so the unsafe path never lands in the JSONL.
+
+    Path components are validated against `out_dir`: tweet IDs must match
+    [0-9]+, filenames must be plain basenames, and the final resolved
+    path must stay under `out_dir`. Anything that fails validation is
+    skipped.
+
+    Returns: list of {tweet_id, url, rel_path} ready for the downloader.
+    """
+    pending = []
+    for tweet in tweets:
+        tweet_id = tweet.get('id')
+        if not tweet_id or not isinstance(tweet_id, str) or not _TWEET_ID_RE.match(tweet_id):
+            continue
+
+        # Top-level photo media (regular tweets) — derived path, no mutation.
+        for item in tweet.get('media') or []:
+            if not isinstance(item, dict) or item.get('type') != 'photo':
+                continue
+            url = item.get('url')
+            filename = _photo_filename(url)
+            if not filename:
+                continue
+            rel_path = f'media/{tweet_id}/{filename}'
+            if not _is_safe_rel_path(out_dir, rel_path):
+                continue
+            pending.append({'tweet_id': tweet_id, 'url': url, 'rel_path': rel_path})
+
+        # Article media — local_path crosses the trust boundary in the
+        # request body, so re-validate it here.
+        article = tweet.get('article') or {}
+        for item in article.get('media') or []:
+            if not isinstance(item, dict):
+                continue
+            url = item.get('url')
+            rel_path = item.get('local_path')
+            if not url or not isinstance(rel_path, str):
+                continue
+            if not _is_safe_rel_path(out_dir, rel_path):
+                # Strip the unsafe path so it doesn't end up in the JSONL.
+                item.pop('local_path', None)
+                _strip_article_text_local_path(article, rel_path)
+                continue
+            pending.append({'tweet_id': tweet_id, 'url': url, 'rel_path': rel_path})
+
+    return pending
+
+
+_image_downloader = None
+_image_downloader_lock = threading.Lock()
+
+
+def get_image_downloader():
+    """Lazily construct the singleton ImageDownloader."""
+    global _image_downloader
+    with _image_downloader_lock:
+        if _image_downloader is None:
+            _image_downloader = ImageDownloader()
+        return _image_downloader
+
+
+def reset_image_downloader():
+    """Reset the singleton (test-only — leaks the previous worker thread)."""
+    global _image_downloader
+    with _image_downloader_lock:
+        _image_downloader = None
+
+
+class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
+    """Treat any redirect as an error so we never fetch beyond the allowlisted host."""
+
+    def redirect_request(self, req, fp, code, msg, headers, newurl):
+        raise urllib.error.HTTPError(req.full_url, code, f'redirect to {newurl} blocked', headers, fp)
+
+
+_NO_REDIRECT_OPENER = urllib.request.build_opener(_NoRedirectHandler())
+
+
+class ImageDownloader:
+    """Single background-thread image downloader with hardened HTTP fetch.
+
+    - One worker, one queue. Per-job cost is small so a single thread is fine
+      and avoids hammering the CDN.
+    - Idempotent: if the destination file already exists, skip the network call
+      and log status='exists'.
+    - URL hostname is checked against ALLOWED_IMAGE_HOSTS before every request.
+      Redirects are blocked entirely (so an attacker can't 302 us off-allowlist).
+    - Per-file size cap (XTAP_MAX_FILE_MB, default 50) bounds disk impact even
+      for adversarial responses. Optional cumulative cap via XTAP_MAX_MEDIA_MB.
+    - 429 responses trigger exponential backoff (capped at MAX_BACKOFF_S).
+      Other HTTP errors and network failures log status='error'.
+    """
+
+    DEFAULT_DELAY_MS = 100
+    DEFAULT_MAX_FILE_MB = 50
+    USER_AGENT = 'xTap/1.0 (+https://github.com/mkubicek/xTap)'
+    MAX_BACKOFF_S = 30
+    REQUEST_TIMEOUT_S = 30
+    CHUNK_SIZE = 64 * 1024
+
+    def __init__(self):
+        self.queue = queue.Queue()
+        self.delay_s = max(0.0, _env_int('XTAP_IMAGE_DELAY_MS', self.DEFAULT_DELAY_MS) / 1000.0)
+        max_total_mb = _env_float('XTAP_MAX_MEDIA_MB', 0.0)
+        self.max_bytes = int(max_total_mb * 1024 * 1024) if max_total_mb > 0 else None
+        max_file_mb = _env_float('XTAP_MAX_FILE_MB', float(self.DEFAULT_MAX_FILE_MB))
+        self.max_file_bytes = int(max_file_mb * 1024 * 1024) if max_file_mb > 0 else None
+        self._bytes_lock = threading.Lock()
+        self.bytes_downloaded = 0
+        self._last_request_at = 0.0
+        self._thread = threading.Thread(target=self._run, daemon=True, name='xtap-image-downloader')
+        self._thread.start()
+
+    def enqueue(self, jobs, out_dir):
+        """Enqueue a batch of pending downloads against the given output dir."""
+        for job in jobs:
+            self.queue.put((job, out_dir))
+
+    def _run(self):
+        while True:
+            try:
+                job, out_dir = self.queue.get()
+            except Exception:  # pragma: no cover — only fires at interpreter shutdown
+                continue
+            try:
+                self._process(job, out_dir)
+            except Exception as e:  # pragma: no cover — defensive
+                print(f'[xtap:image] worker exception: {e}', file=sys.stderr)
+            finally:
+                self.queue.task_done()
+
+    def _process(self, job, out_dir):
+        tweet_id = job['tweet_id']
+        url = job['url']
+        rel_path = job['rel_path']
+
+        # Re-validate the rel_path defensively: enqueue() is exported and a
+        # future caller could skip inject_image_local_paths.
+        if not _is_safe_rel_path(out_dir, rel_path):
+            self._log(out_dir, tweet_id, url, rel_path, 'error:unsafe_path', 0)
+            return
+
+        dest_path = os.path.join(out_dir, rel_path)
+
+        if os.path.exists(dest_path):
+            self._log(out_dir, tweet_id, url, dest_path, 'exists', os.path.getsize(dest_path))
+            return
+
+        if not _is_allowed_url(url):
+            self._log(out_dir, tweet_id, url, dest_path, 'error:host_not_allowed', 0)
+            return
+
+        if self.max_bytes is not None:
+            with self._bytes_lock:
+                over_quota = self.bytes_downloaded >= self.max_bytes
+            if over_quota:
+                self._log(out_dir, tweet_id, url, dest_path, 'skipped:quota', 0)
+                return
+
+        # Simple rate limiter: enforce delay_s between requests.
+        if self.delay_s > 0:
+            wait = self.delay_s - (time.monotonic() - self._last_request_at)
+            if wait > 0:
+                time.sleep(wait)
+
+        size, err = self._download(url, dest_path)
+        self._last_request_at = time.monotonic()
+
+        if err:
+            self._log(out_dir, tweet_id, url, dest_path, f'error:{err}', 0)
+            return
+
+        with self._bytes_lock:
+            self.bytes_downloaded += size
+        self._log(out_dir, tweet_id, url, dest_path, 'ok', size)
+
+    def _download(self, url, dest_path):
+        """Download to a .part file and rename atomically. Returns (bytes, error)."""
+        os.makedirs(os.path.dirname(dest_path), exist_ok=True)
+        tmp_path = dest_path + '.part'
+        backoff = 1.0
+        attempts = 0
+        while True:
+            attempts += 1
+            req = urllib.request.Request(url, headers={'User-Agent': self.USER_AGENT})
+            try:
+                with _NO_REDIRECT_OPENER.open(req, timeout=self.REQUEST_TIMEOUT_S) as resp:
+                    # Reject up-front when Content-Length advertises an oversize body.
+                    if self.max_file_bytes is not None:
+                        cl = resp.headers.get('Content-Length')
+                        if cl and cl.isdigit() and int(cl) > self.max_file_bytes:
+                            return 0, 'too_large'
+                    written = 0
+                    with open(tmp_path, 'wb') as f:
+                        while True:
+                            chunk = resp.read(self.CHUNK_SIZE)
+                            if not chunk:
+                                break
+                            written += len(chunk)
+                            if self.max_file_bytes is not None and written > self.max_file_bytes:
+                                _safe_unlink(tmp_path)
+                                return 0, 'too_large'
+                            f.write(chunk)
+                size = os.path.getsize(tmp_path)
+                os.replace(tmp_path, dest_path)
+                return size, None
+            except urllib.error.HTTPError as e:
+                _safe_unlink(tmp_path)
+                if e.code == 429 and backoff <= self.MAX_BACKOFF_S and attempts < 4:
+                    time.sleep(backoff)
+                    backoff *= 2
+                    continue
+                return 0, f'http_{e.code}'
+            except (urllib.error.URLError, OSError, TimeoutError) as e:
+                _safe_unlink(tmp_path)
+                return 0, type(e).__name__
+
+    def _log(self, out_dir, tweet_id, url, dest_path, status, size):
+        try:
+            local_path = os.path.relpath(dest_path, out_dir)
+        except ValueError:  # pragma: no cover — Windows-only: different drives
+            local_path = dest_path
+        entry = {
+            'timestamp': datetime.now(timezone.utc).isoformat(),
+            'tweet_id': tweet_id,
+            'url': url,
+            'local_path': local_path,
+            'status': status,
+            'bytes': size,
+        }
+        manifest_path = os.path.join(out_dir, 'media-manifest.jsonl')
+        try:
+            os.makedirs(out_dir, exist_ok=True)
+            with open(manifest_path, 'a', encoding='utf-8') as f:
+                f.write(json.dumps(entry, ensure_ascii=False) + '\n')
+        except OSError as e:
+            print(f'[xtap:image] manifest write failed: {e}', file=sys.stderr)
+
+
+def _is_allowed_url(url):
+    """True iff url is https and the hostname is in ALLOWED_IMAGE_HOSTS."""
+    if not url or not isinstance(url, str):
+        return False
+    try:
+        parsed = urlparse(url)
+    except ValueError:  # pragma: no cover — urlparse rarely raises in practice
+        return False
+    if parsed.scheme != 'https':
+        return False
+    host = (parsed.hostname or '').lower()
+    return host in ALLOWED_IMAGE_HOSTS
+
+
+def _safe_unlink(path):
+    try:
+        os.unlink(path)
+    except OSError:
+        pass

extension/native-host/xtap_daemon.bat

@@ -0,0 +1 @@
+@pythonw "%~dp0xtap_daemon.py" %*

extension/native-host/xtap_daemon.py

@@ -0,0 +1,432 @@
+#!/usr/bin/env python3
+"""xTap HTTP Daemon — runs as a system service (launchd/systemd/Scheduled Task)."""
+
+import hmac
+import json
+import os
+import platform
+import signal
+import socket
+import sys
+import time
+import uuid
+import threading
+from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
+
+from xtap_core import (DEFAULT_OUTPUT_DIR, load_seen_ids, resolve_output_dir,
+                       validate_output_dir, write_tweets, write_log,
+                       write_dump, test_path,
+                       check_ytdlp, start_download, get_download_status,
+                       collect_image_jobs, get_image_downloader,
+                       validate_tweet_url)
+
+VERSION = '0.24.0'
+BIND_HOST = '127.0.0.1'
+BIND_PORT = int(os.environ.get('XTAP_DAEMON_PORT', 17381))
+MAX_BODY_SIZE = 10 * 1024 * 1024  # 10 MB (extension caps POSTs via MAX_TWEETS_PER_POST in background.js)
+MAX_DRAIN_SIZE = 64 * 1024 * 1024  # cap bytes drained from an oversized request before replying 413
+MAX_DRAIN_IDLE_TIMEOUT_S = 0.25  # don't hang on clients that declare a body and send none
+
+
+def _env_timeout_s(name, default):
+    # Always finite — bad/<=0 values fall back to the default rather than
+    # disabling the timeout (disabling would reintroduce the shutdown-hang
+    # these guards exist to prevent).
+    raw = (os.environ.get(name) or '').strip()
+    if not raw:
+        return default
+    try:
+        value = float(raw)
+    except ValueError:
+        return default
+    return value if value > 0 else default
+
+
+# Per-connection socket timeout (see DaemonHandler.timeout). Default 15s:
+# above the extension's 10s client timeout (HTTP_TIMEOUT_MS) so a legit
+# localhost request is never cut off, and below launchd's ~20s SIGTERM->
+# SIGKILL window so an idle connection can't push shutdown past it.
+CONN_TIMEOUT_S = _env_timeout_s('XTAP_CONN_TIMEOUT_S', 15.0)
+
+# Bounded shutdown join. The socket timeout above is per blocking read, not a
+# total request deadline — a trickle peer sending one byte per read window
+# keeps its handler thread alive forever, and server_close() would join it
+# indefinitely. After this grace period the daemon force-exits: legitimate
+# JSONL writes finish in well under a second, so only stuck peers are cut off,
+# and 10s stays below launchd's ~20s SIGTERM->SIGKILL escalation.
+SHUTDOWN_GRACE_S = _env_timeout_s('XTAP_SHUTDOWN_GRACE_S', 10.0)
+XTAP_DIR = os.path.expanduser('~/.xtap')
+XTAP_SECRET = os.path.join(XTAP_DIR, 'secret')
+
+# Log level: 'info' (default) or 'debug'
+LOG_LEVEL = os.environ.get('XTAP_LOG_LEVEL', 'info').lower()
+
+
+def log_info(msg):
+    print(msg, file=sys.stderr)
+
+
+def log_debug(msg):
+    if LOG_LEVEL == 'debug':
+        print(f'[DEBUG] {msg}', file=sys.stderr)
+
+
+def load_token():
+    try:
+        with open(XTAP_SECRET, 'r', encoding='utf-8') as f:
+            return f.read().strip()
+    except FileNotFoundError:
+        log_info(f'FATAL: {XTAP_SECRET} not found. Run install.sh first.')
+        sys.exit(1)
+
+
+# Module-level state shared across requests
+_token = None
+_seen_ids_by_dir = {}
+_state_lock = threading.Lock()
+
+
+class XtapHTTPServer(ThreadingHTTPServer):
+    # daemon_threads defaults to True on ThreadingHTTPServer, which makes
+    # server_close()'s thread-join a no-op (daemon threads aren't tracked) —
+    # an in-flight handler is then killed mid-JSONL-write at interpreter exit.
+    # False + block_on_close (default True) makes server_close() actually wait
+    # for in-flight writes to finish before the process exits.
+    daemon_threads = False
+
+
+class DaemonHandler(BaseHTTPRequestHandler):
+    # Socket timeout for stalled/idle connections. Required by the
+    # daemon_threads = False change above: without it, a peer that opens a
+    # connection to 127.0.0.1 and never finishes sending its request line/body
+    # (a port scanner, a stalled fetch — this is pre-auth, so the token can't
+    # gate it) parks a non-daemon handler thread in a blocking read forever.
+    # server_close() then blocks joining that thread on shutdown until
+    # launchd/systemd escalates to SIGKILL, killing in-flight JSONL writes —
+    # the exact failure daemon_threads = False exists to prevent. The timeout
+    # makes each blocking read raise so idle threads exit. It is per-read, not
+    # a total deadline — a trickle sender can still hold a thread, which is
+    # why shutdown is additionally bounded by SHUTDOWN_GRACE_S in main().
+    timeout = CONN_TIMEOUT_S
+
+    def log_message(self, format, *args):
+        # Log to stderr (captured by launchd/systemd)
+        log_info(f'{self.client_address[0]} - {format % args}')
+
+    def _send_json(self, obj, status=200):
+        body = json.dumps(obj).encode('utf-8')
+        self.send_response(status)
+        self.send_header('Content-Type', 'application/json')
+        self.send_header('Content-Length', len(body))
+        self.end_headers()
+        self.wfile.write(body)
+        log_debug(f'  -> {status} ({len(body)} bytes)')
+
+    def _read_json(self, length):
+        if length == 0:
+            return {}
+        return json.loads(self.rfile.read(length))
+
+    def _check_auth(self):
+        auth = self.headers.get('Authorization', '')
+        if not auth.startswith('Bearer ') or not hmac.compare_digest(auth[7:], _token):
+            log_debug(f'  Auth failed (header {"present" if auth else "missing"})')
+            self._send_json({'ok': False, 'error': 'Unauthorized'}, 401)
+            return False
+        return True
+
+    def do_GET(self):
+        log_debug(f'GET {self.path}')
+        if self.path == '/status':
+            # Validate token when provided (allows probeHttp to detect stale credentials)
+            auth = self.headers.get('Authorization', '')
+            if auth and (not auth.startswith('Bearer ') or not hmac.compare_digest(auth[7:], _token)):
+                self._send_json({'ok': False, 'error': 'Unauthorized'}, 401)
+                return
+            self._send_json({'ok': True, 'version': VERSION})
+            return
+        self._send_json({'ok': False, 'error': 'Not found'}, 404)
+
+    def _validate_content_length(self):
+        """Validate Content-Length header. Returns length or -1 on error (response already sent)."""
+        raw = self.headers.get('Content-Length')
+        if raw is None:
+            self._send_json({'ok': False, 'error': 'Missing Content-Length header'}, 400)
+            return -1
+        try:
+            length = int(raw)
+        except ValueError:
+            self._send_json({'ok': False, 'error': f'Invalid Content-Length: {raw!r}'}, 400)
+            return -1
+        if length < 0:
+            self._send_json({'ok': False, 'error': 'Content-Length must not be negative'}, 400)
+            return -1
+        if length > MAX_BODY_SIZE:
+            # Drain the (bounded) body before replying. An unread request body
+            # on a closed HTTP/1.0 socket triggers a TCP RST, which surfaces to
+            # the extension's fetch() as a network error rather than a 413 — so
+            # the client's batch-splitting recovery never sees the 413 and the
+            # oversized batch wedges the queue. Draining lets the 413 arrive
+            # cleanly. Capped so a bogus huge Content-Length can't tie us up.
+            self._drain_body(min(length, MAX_DRAIN_SIZE))
+            self._send_json({'ok': False, 'error': 'Payload too large'}, 413)
+            return -1
+        return length
+
+    def _drain_body(self, n):
+        old_timeout = self.connection.gettimeout()
+        self.connection.settimeout(MAX_DRAIN_IDLE_TIMEOUT_S)
+        remaining = n
+        try:
+            while remaining > 0:
+                try:
+                    chunk = self.rfile.read(min(65536, remaining))
+                except socket.timeout:
+                    break
+                if not chunk:
+                    break
+                remaining -= len(chunk)
+        finally:
+            self.connection.settimeout(old_timeout)
+
+    def do_POST(self):
+        log_debug(f'POST {self.path} (Content-Length: {self.headers.get("Content-Length", "?")})')
+
+        if not self._check_auth():
+            return
+
+        length = self._validate_content_length()
+        if length < 0:
+            return
+
+        try:
+            body = self._read_json(length)
+        except (json.JSONDecodeError, ValueError) as e:
+            self._send_json({'ok': False, 'error': f'Invalid JSON: {e}'}, 400)
+            return
+
+        t0 = time.monotonic()
+        if self.path == '/tweets':
+            self._handle_tweets(body)
+        elif self.path == '/log':
+            self._handle_log(body)
+        elif self.path == '/dump':
+            self._handle_dump(body)
+        elif self.path == '/test-path':
+            self._handle_test_path(body)
+        elif self.path == '/check-ytdlp':
+            self._handle_check_ytdlp(body)
+        elif self.path == '/download-video':
+            self._handle_download_video(body)
+        elif self.path == '/download-status':
+            self._handle_download_status(body)
+        else:
+            self._send_json({'ok': False, 'error': 'Not found'}, 404)
+        elapsed = (time.monotonic() - t0) * 1000
+        log_debug(f'  Completed in {elapsed:.1f}ms')
+
+    def _handle_tweets(self, body):
+        try:
+            msg_dir = body.get('outputDir', '').strip()
+            # Strict boolean: only `true` enables. Avoids string 'false' / number 1
+            # silently turning the feature on.
+            image_download = body.get('image_download') is True
+            with _state_lock:
+                out_dir, seen_ids = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir)
+                tweets = body.get('tweets', [])
+                # Always collect before write_tweets: collect_image_jobs strips
+                # unsafe article local_paths from the tweets so traversal paths
+                # never land in the JSONL — sanitization must not depend on the
+                # image_download toggle.
+                pending_images = collect_image_jobs(tweets, out_dir)
+                count, dupes = write_tweets(tweets, out_dir, seen_ids)
+            queued = 0
+            if image_download and pending_images:
+                get_image_downloader().enqueue(pending_images, out_dir)
+                queued = len(pending_images)
+            log_debug(f'  Tweets: {count} written, {dupes} dupes, {queued} images queued -> {out_dir}')
+            self._send_json({'ok': True, 'count': count, 'dupes': dupes, 'images_queued': queued})
+        except ValueError as e:
+            self._send_json({'ok': False, 'error': str(e)}, 400)
+        except Exception as e:
+            log_info(f'ERROR /tweets: {e}')
+            log_debug(f'  Traceback: {_format_exc()}')
+            self._send_json({'ok': False, 'error': str(e)}, 500)
+
+    def _handle_log(self, body):
+        try:
+            msg_dir = body.get('outputDir', '').strip()
+            lines = body.get('lines', [])
+            with _state_lock:
+                out_dir, _ = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir)
+                # Write under the lock — concurrent appends through separate
+                # file objects can interleave mid-line at buffer boundaries.
+                logged = write_log(lines, out_dir)
+            self._send_json({'ok': True, 'logged': logged})
+        except ValueError as e:
+            self._send_json({'ok': False, 'error': str(e)}, 400)
+        except Exception as e:
+            log_info(f'ERROR /log: {e}')
+            log_debug(f'  Traceback: {_format_exc()}')
+            self._send_json({'ok': False, 'error': str(e)}, 500)
+
+    def _handle_dump(self, body):
+        try:
+            msg_dir = body.get('outputDir', '').strip()
+            with _state_lock:
+                out_dir, _ = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir)
+                filename = body.get('filename', 'dump.json')
+                content = body.get('content', '')
+                path = write_dump(filename, content, out_dir)
+            self._send_json({'ok': True, 'path': path})
+        except ValueError as e:
+            self._send_json({'ok': False, 'error': str(e)}, 400)
+        except Exception as e:
+            log_info(f'ERROR /dump: {e}')
+            log_debug(f'  Traceback: {_format_exc()}')
+            self._send_json({'ok': False, 'error': str(e)}, 500)
+
+    def _handle_test_path(self, body):
+        try:
+            msg_dir = body.get('outputDir', '').strip()
+            if not msg_dir:
+                self._send_json({'ok': False, 'error': 'outputDir is required'}, 400)
+                return
+            out_dir = validate_output_dir(os.path.expanduser(msg_dir))
+            test_path(out_dir)
+            self._send_json({'ok': True, 'type': 'TEST_PATH'})
+        except ValueError as e:
+            self._send_json({'ok': False, 'error': str(e)}, 400)
+        except Exception as e:
+            log_info(f'ERROR /test-path: {e}')
+            self._send_json({'ok': False, 'error': str(e)}, 500)
+
+    def _handle_check_ytdlp(self, body):
+        available = check_ytdlp()
+        log_debug(f'  yt-dlp available: {available}')
+        self._send_json({'ok': True, 'available': available})
+
+    def _handle_download_video(self, body):
+        try:
+            tweet_url = body.get('tweetUrl', '')
+            direct_url = body.get('directUrl', '')
+            post_date = body.get('postDate', '')
+            msg_dir = body.get('outputDir', '').strip()
+            validate_tweet_url(tweet_url)
+            with _state_lock:
+                out_dir, _ = resolve_output_dir(msg_dir, DEFAULT_OUTPUT_DIR, _seen_ids_by_dir)
+            download_id = str(uuid.uuid4())
+            start_download(download_id, tweet_url, direct_url, out_dir, post_date)
+            log_debug(f'  Download started: {download_id} -> {tweet_url}')
+            self._send_json({'ok': True, 'downloadId': download_id})
+        except ValueError as e:
+            self._send_json({'ok': False, 'error': str(e)}, 400)
+        except Exception as e:
+            log_info(f'ERROR /download-video: {e}')
+            self._send_json({'ok': False, 'error': str(e)}, 500)
+
+    def _handle_download_status(self, body):
+        download_id = body.get('downloadId', '')
+        status = get_download_status(download_id)
+        self._send_json({'ok': True, **status})
+
+
+def _format_exc():
+    import traceback
+    return traceback.format_exc().replace('\n', ' | ')
+
+
+def _setup_stdio():
+    """Redirect stdio to log files when running under pythonw (no console)."""
+    if sys.stderr is not None and sys.stdout is not None:
+        return
+    os.makedirs(XTAP_DIR, exist_ok=True)
+    if sys.stdout is None:
+        sys.stdout = open(os.path.join(XTAP_DIR, 'daemon-stdout.log'), 'a', encoding='utf-8')
+    if sys.stderr is None:
+        sys.stderr = open(os.path.join(XTAP_DIR, 'daemon-stderr.log'), 'a', encoding='utf-8')
+
+
+def _log_startup_diagnostics():
+    """Log system and configuration info on startup."""
+    log_info(f'xTap daemon v{VERSION}')
+    log_info(f'  Python:     {sys.version.split(chr(10))[0]}')
+    log_info(f'  Executable: {sys.executable}')
+    log_info(f'  Script:     {os.path.abspath(__file__)}')
+    log_info(f'  Platform:   {platform.system()} {platform.release()}')
+    log_info(f'  Output dir: {DEFAULT_OUTPUT_DIR}')
+    log_info(f'  Token:      loaded ({len(_token)} chars)')
+    log_info(f'  Log level:  {LOG_LEVEL}')
+
+    # Check output dir writability
+    try:
+        test_path(DEFAULT_OUTPUT_DIR)
+        log_info(f'  Output dir: writable')
+    except Exception as e:
+        log_info(f'  Output dir: NOT writable ({e})')
+
+    # Check yt-dlp
+    ytdlp = check_ytdlp()
+    log_info(f'  yt-dlp:     {"available" if ytdlp else "not found"}')
+
+
+def main():
+    global _token
+
+    _setup_stdio()
+
+    _token = load_token()
+
+    # Initialize output directory and seen IDs
+    os.makedirs(DEFAULT_OUTPUT_DIR, exist_ok=True)
+    _seen_ids_by_dir[DEFAULT_OUTPUT_DIR] = load_seen_ids(DEFAULT_OUTPUT_DIR)
+
+    _log_startup_diagnostics()
+    log_info(f'  Seen IDs:   {len(_seen_ids_by_dir[DEFAULT_OUTPUT_DIR])} loaded')
+
+    try:
+        server = XtapHTTPServer((BIND_HOST, BIND_PORT), DaemonHandler)
+    except OSError as e:
+        log_info(f'FATAL: Cannot bind to {BIND_HOST}:{BIND_PORT} — {e}')
+        log_info(f'  Is another instance already running?')
+        sys.exit(1)
+
+    def shutdown(signum, frame):
+        log_info(f'Received signal {signum}, shutting down...')
+        # shutdown() must run on a different thread than serve_forever(): the
+        # signal handler interrupts the serve_forever thread, and shutdown()
+        # blocks until serve_forever exits — calling it inline deadlocks until
+        # launchd/systemd escalates to SIGKILL (killing in-flight writes).
+        threading.Thread(target=server.shutdown, daemon=True).start()
+
+    signal.signal(signal.SIGINT, shutdown)
+    if platform.system() == 'Windows':
+        signal.signal(signal.SIGBREAK, shutdown)
+    else:
+        signal.signal(signal.SIGTERM, shutdown)
+
+    log_info(f'Listening on {BIND_HOST}:{BIND_PORT}')
+    server.serve_forever()
+    # Join in-flight non-daemon handler threads so any active JSONL write can
+    # finish cleanly — but bound the join: a peer that trickles bytes defeats
+    # the per-read socket timeout and would pin server_close() forever,
+    # holding the SIGTERM window open until launchd/systemd SIGKILLs us
+    # mid-write anyway. Force-exit after the grace period instead: real
+    # writes have long finished; only stuck peers are cut off.
+    closer = threading.Thread(target=server.server_close, daemon=True)
+    closer.start()
+    closer.join(SHUTDOWN_GRACE_S)
+    if closer.is_alive():
+        log_info(f'Shutdown grace ({SHUTDOWN_GRACE_S}s) expired with handler '
+                 f'threads still running — forcing exit')
+        sys.stderr.flush()
+        try:
+            sys.stdout.flush()
+        except (OSError, ValueError):
+            pass
+        os._exit(0)
+    log_info('Shutdown complete')
+
+
+if __name__ == '__main__':
+    main()

extension/native-host/xtap_host.bat

@@ -0,0 +1 @@
+@python "%~dp0xtap_host.py" %*

extension/native-host/xtap_host.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""xTap Native Messaging Host — token bootstrap for the Chrome/Firefox extension.
+
+This minimal host exists solely to pass the HTTP daemon's auth token to the
+browser extension via native messaging.  All tweet/log/dump I/O goes through
+the HTTP daemon (xtap_daemon.py).
+"""
+
+import json
+import os
+import struct
+import sys
+import traceback
+from datetime import datetime
+
+XTAP_PORT = int(os.environ.get('XTAP_DAEMON_PORT', 17381))
+XTAP_DIR = os.path.expanduser('~/.xtap')
+XTAP_SECRET = os.path.join(XTAP_DIR, 'secret')
+XTAP_ERROR_LOG = os.path.join(XTAP_DIR, 'host-error.log')
+
+MAX_MESSAGE_BYTES = 1 * 1024 * 1024  # 1 MiB guard
+
+
+def read_exact(stream, size):
+    """Read exactly *size* bytes from *stream*, handling pipe fragmentation."""
+    buf = b''
+    while len(buf) < size:
+        chunk = stream.read(size - len(buf))
+        if not chunk:
+            raise EOFError('unexpected end of stream')
+        buf += chunk
+    return buf
+
+
+def drain_exact(stream, size):
+    """Discard exactly *size* bytes so the next frame starts at a boundary."""
+    remaining = size
+    while remaining > 0:
+        chunk = stream.read(min(65536, remaining))
+        if not chunk:
+            raise EOFError('unexpected end of stream')
+        remaining -= len(chunk)
+
+
+def read_message():
+    try:
+        header = read_exact(sys.stdin.buffer, 4)
+    except EOFError:
+        raise EOFError('stdin closed')
+    length = struct.unpack('<I', header)[0]
+    if length > MAX_MESSAGE_BYTES:
+        # Drain the payload before raising — leaving it in the pipe would make
+        # the next read interpret payload bytes as a length header, and the
+        # stream would never realign.
+        drain_exact(sys.stdin.buffer, length)
+        raise ValueError(f'message too large: {length} bytes')
+    data = read_exact(sys.stdin.buffer, length)
+    return json.loads(data)
+
+
+def send_message(msg):
+    encoded = json.dumps(msg).encode('utf-8')
+    sys.stdout.buffer.write(struct.pack('<I', len(encoded)))
+    sys.stdout.buffer.write(encoded)
+    sys.stdout.buffer.flush()
+
+
+def _main():
+    while True:
+        try:
+            msg = read_message()
+        except EOFError:
+            break
+        except Exception as e:
+            send_message({'ok': False, 'error': str(e)})
+            continue
+
+        if msg.get('type') == 'GET_TOKEN':
+            try:
+                with open(XTAP_SECRET, 'r', encoding='utf-8') as f:
+                    token = f.read().strip()
+                send_message({'ok': True, 'token': token, 'port': XTAP_PORT})
+            except FileNotFoundError:
+                send_message({'ok': False, 'error': 'Daemon not installed (~/.xtap/secret not found)'})
+            except Exception as e:
+                send_message({'ok': False, 'error': str(e)})
+        else:
+            send_message({
+                'ok': False,
+                'error': f'Unsupported message type: {msg.get("type")}. '
+                         'All data is handled by the HTTP daemon.'
+            })
+
+
+def main():
+    try:
+        _main()
+    except Exception:
+        os.makedirs(XTAP_DIR, exist_ok=True)
+        with open(XTAP_ERROR_LOG, 'a', encoding='utf-8') as f:
+            f.write(f'\n--- {datetime.now().isoformat()} ---\n')
+            f.write(f'Python: {sys.version}\n')
+            f.write(f'Script: {os.path.abspath(__file__)}\n')
+            f.write(f'sys.path: {sys.path}\n')
+            traceback.print_exc(file=f)
+        raise
+
+
+if __name__ == '__main__':
+    main()

extension/options.html

@@ -0,0 +1,45 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="utf-8">
+  <link rel="stylesheet" href="popup.css">
+  <style>
+    body { min-width: 26rem; }
+    .options-note { font-size: 0.85em; opacity: 0.75; margin: 0.3rem 0 0.8rem; }
+  </style>
+</head>
+<body>
+  <div class="container">
+    <h1>xtap-pool options</h1>
+    <div id="pool-status" class="status"></div>
+
+    <div class="settings">
+      <label class="settings-label" for="pool-url">Pool Space URL</label>
+      <p class="options-note">
+        The Space this extension syncs to. Changing it disconnects the current
+        pool token; use Connect in the popup (or paste a token below) after
+        saving.
+      </p>
+      <div class="settings-row">
+        <input type="text" id="pool-url" placeholder="https://your-space.hf.space" spellcheck="false">
+      </div>
+    </div>
+
+    <div class="settings">
+      <label class="settings-label" for="pool-token">Pool token (manual fallback)</label>
+      <p class="options-note">
+        Only needed if the automatic handoff on the Space's /connect page
+        fails. Copy the token shown there and paste it here.
+      </p>
+      <div class="settings-row">
+        <input type="text" id="pool-token" placeholder="xp1.…" spellcheck="false">
+      </div>
+    </div>
+
+    <div class="settings-row">
+      <button id="save" class="save-btn">Save</button>
+    </div>
+  </div>
+  <script src="options.js"></script>
+</body>
+</html>

extension/options.js

@@ -0,0 +1,33 @@
+const statusEl = document.getElementById('pool-status');
+const urlInput = document.getElementById('pool-url');
+const tokenInput = document.getElementById('pool-token');
+const saveBtn = document.getElementById('save');
+
+function render(state) {
+  if (!state) return;
+  urlInput.value = state.url || '';
+  if (state.connected) {
+    const who = state.username ? ` as @${state.username}` : '';
+    statusEl.textContent = `Connected${who} — ${state.queued} queued, ${state.synced} synced`;
+    statusEl.className = 'status connected';
+  } else {
+    statusEl.textContent = 'Not connected to a pool';
+    statusEl.className = 'status disconnected';
+  }
+}
+
+chrome.runtime.sendMessage({ type: 'POOL_STATUS' }, render);
+
+saveBtn.addEventListener('click', () => {
+  const msg = { type: 'POOL_SET_CONFIG' };
+  const url = urlInput.value.trim();
+  const token = tokenInput.value.trim();
+  if (url) msg.url = url;
+  if (token) msg.token = token;
+  saveBtn.disabled = true;
+  chrome.runtime.sendMessage(msg, (state) => {
+    saveBtn.disabled = false;
+    tokenInput.value = '';
+    render(state);
+  });
+});