ARCHITECTURE.md

# HyperKitty Chromium — Architecture

**Status: implemented reference implementation.** Every module, supervision
relationship, schema, and API route described below exists in this
repository and compiles, boots, and passes its test suite (`rebar3 eunit`:
24 tests, 0 failures, including real headless-Chromium integration tests
and chaos tests that kill live processes to verify recovery). Sections are
marked *(planned)* where the specification calls for something this
revision intentionally stubs or defers — there is no unmarked aspirational
content in this document.

## 1. Purpose and scope

HyperKitty Chromium is an Erlang/OTP-native browser agent platform: a
headless Chromium instance, driven over the Chrome DevTools Protocol
(CDP), wrapped in an explicit OTP supervision tree so that browser
control, agent orchestration, search, and messaging are separate,
independently-supervised, independently-observable subsystems rather than
library calls inside one monolithic process. The design principle it
exists to demonstrate: an LLM-driven "agent" is a planning function that
proposes tool calls; it is never trusted with direct, unchecked access to
a browser process. Every tool call crosses an explicit, typed capability
boundary — enforced by Erlang code, not by the prompt — before it reaches
Chromium.

This document is the architecture reference for engineers building on,
operating, or extending this codebase. It assumes familiarity with
Erlang/OTP (supervisors, `gen_server`, `gen_statem`) and HTTP/WebSocket
APIs, but not with this specific codebase.

## 2. Repository layout

```
hyperkitty/
  rebar.config              deps, relx release config, profiles, dialyzer/xref
  config/sys.config         application env (chromium path, API port, limits)
  config/vm.args             node name, cookie, scheduler flags
  include/hyperkitty.hrl    canonical entity records (data model)
  src/
    hyperkitty_app.erl       application callback
    hyperkitty_sup.erl       HK-CORE root supervisor
    hk_schema.erl            record -> JSON map projection (wire format)
    hk_id.erl, hk_event.erl  ids, timestamps, the event envelope
    hk_sup_util.erl          shared "wait for the old holder to die" helper
    hk_event_bus.erl / _sup  event bus (pub/sub + audit log)
    hk_messaging_portal.erl  HK-MSG
    hk_cdp_client.erl        one CDP WebSocket connection (via `gun`)
    hk_chrome_http.erl       Chrome's plain-HTTP /json control endpoint
    hk_browser_*.erl         HK-BROWSER: session, capability, registry, supervisors
    hk_agent_*.erl           HK-AGENT: the agent state machine, registry, toolbox
    hk_search_*.erl          HK-SEARCH: the 11-stage pipeline job, provider behaviour
    hk_operation_store.erl   operation_id lifecycle (accepted/succeeded/failed)
    hk_api_*.erl             HK-VIS backend: Cowboy router, listener, REST + WS handlers
    hk_frontend_*.erl        connected-client tracking for the observability UI
    hk_health.erl            aggregate subsystem health
  priv/frontend/            the 7-view observability UI (static HTML/JS, no build step)
  test/                     EUnit suites (see §11)
  Brewfile, bin/bootstrap.sh  reproducible environment setup (see §12)
```

## 3. Supervision tree

```
hyperkitty_sup (rest_for_one, intensity 8/10)
  +-- hk_event_sup (one_for_one)
  |     +-- hk_event_bus                          [gen_server]
  +-- hk_messaging_sup (one_for_one)
  |     +-- hk_messaging_portal                    [gen_server]
  +-- hk_browser_sup (one_for_one)
  |     +-- hk_browser_registry                    [gen_server]  session_id -> pid
  |     +-- hk_browser_session_sup (simple_one_for_one, transient)
  |           +-- hk_browser_session (one per session)  [gen_server]  owns a Chrome OS process
  +-- hk_agent_sup (one_for_one)
  |     +-- hk_agent_registry                       [gen_server]  agent_id -> pid
  |     +-- hk_agent_fsm_sup (simple_one_for_one, transient)
  |           +-- hk_agent_fsm (one per agent)       [gen_statem]
  +-- hk_search_sup (one_for_one)
  |     +-- hk_search_registry                       [gen_server]  job_id -> pid
  |     +-- hk_search_job_sup (simple_one_for_one, temporary)
  |           +-- hk_search_job (one per search)      [gen_statem]
  +-- hk_health                                       [gen_server, worker]
  +-- hk_api_sup (one_for_one)
  |     +-- hk_api_listener                           [gen_server]  owns the Cowboy/Ranch listener
  +-- hk_frontend_sup (one_for_one)
        +-- hk_frontend_client_registry               [gen_server]  connected WS clients
```

**Why `rest_for_one` at the root.** Subsystems are listed in dependency
order. If `hk_event_sup` crashes, everything after it is restarted too,
because every later subsystem holds an implicit dependency on the event
bus (agents, browser sessions, and search jobs all publish to it; the API
layer's WebSocket clients have subscriptions that would otherwise go
silently stale against a bus that no longer exists). A crash in
`hk_browser_sup`, by contrast, restarts `hk_agent_sup`, `hk_search_sup`,
`hk_health`, `hk_api_sup`, and `hk_frontend_sup`, but leaves
`hk_event_sup` and `hk_messaging_sup` — listed *before* it — running
undisturbed. `test/hk_supervision_tests.erl:rest_for_one_isolation/0`
kills `hk_browser_sup` directly and asserts exactly this shape of
recovery.

**Why each plane is a two-level tree, not one `simple_one_for_one`
supervisor.** OTP does not allow mixing a singleton child (the registry)
with `simple_one_for_one` dynamic children (session/agent/job instances)
under the same supervisor. Splitting each plane into a `one_for_one`
supervisor wrapping (a) a singleton registry and (b) a `simple_one_for_one`
instance pool means the registry restarting does not restart live
sessions/agents/jobs (they re-register on next lookup miss; live
processes are untouched), and a crash-restart storm in the instance pool
cannot, via `hk_browser_sup`'s own intensity/period, take the registry
down with it.

**A known, fixed race in this shape, documented for anyone extending
it.** An abrupt, unmaskable kill of a mid-tree supervisor (as opposed to
the ordinary synchronous shutdown protocol a supervisor uses on its own
children) does not guarantee its nested-supervisor children have finished
their own termination and released their registered name before the
grandparent supervisor's restart attempt runs. `hk_sup_util:start_link_retry/1`
wraps every registered-name `start_link/0` in this application: on
`{error, {already_started, OldPid}}` it monitors `OldPid`, waits for it to
actually exit, and retries once, rather than surfacing the transient race
as a failed restart. Separately, `hk_api_listener` calls
`process_flag(trap_exit, true)` in `init/1` — without it, the ordinary
supervisor shutdown signal (`exit(Pid, shutdown)`) would kill it directly
without ever running `terminate/2`, leaking the underlying Ranch/Cowboy
listener and its registered name on every restart, not only under abrupt
kills. Both are exercised by `hk_supervision_tests.erl` and by
`hk_browser_session_tests.erl`'s crash-isolation test.

## 4. Subsystems

### HK-CORE (`hyperkitty_sup`, `hk_event_bus`, `hk_schema`, `hk_id`, `hk_sup_util`)

Boot sequencing, the root supervision policy (§3), the canonical
record-to-JSON projection (`hk_schema:to_map/1` — the single place a wire
representation is produced; internal code passes records, never
hand-built maps, until it crosses this boundary), id/timestamp generation,
and the event bus described in §7.

### HK-BROWSER (`hk_browser_session`, `hk_cdp_client`, `hk_chrome_http`, `hk_browser_capability`, `hk_browser_registry`)

One `hk_browser_session` gen_server per browser session. `init/1` finds a
free TCP port, launches a real headless Chromium OS process via
`erlang:open_port({spawn_executable, ...}, ["--headless=new",
"--remote-debugging-port=<port>", "--remote-debugging-address=127.0.0.1",
"--user-data-dir=<tmp>", "--no-sandbox", "--disable-gpu",
"--disable-dev-shm-usage", "about:blank"])`, polls Chrome's plain-HTTP
`/json/version` endpoint until it answers, and registers itself with
`hk_browser_registry`. Tab lifecycle (`open_tab`/`close_tab`) goes through
Chrome's HTTP control surface (`hk_chrome_http`); per-tab interaction goes
over CDP WebSocket JSON-RPC (`hk_cdp_client`, built on `gun`).
Navigation correlates against the `Page.loadEventFired` CDP event via a
waiters map. A Chrome OS process crash is detected in `handle_info` via
the port's `{exit_status, Status}` message and stops the session with an
abnormal reason, which `hk_browser_session_sup` (restart type `transient`)
restarts under the same `session_id` — the concrete demonstration that one
crashed session never takes down another (`browser_session_crash_isolation/0`)
or any other subsystem.

**Typed capability interface.** `hk_browser_capability:operations/0` is
the closed list of 14 operations (`create_session`, `close_session`,
`open_tab`, `close_tab`, `navigate`, `back`, `forward`, `reload`,
`read_page`, `query_element`, `click`, `type`, `scroll`, `screenshot`,
`extract_links`). There is no operation outside this list and no
unrestricted "just run this JS" escape hatch. `is_granted/2` checks an
agent's capability list, which supports bare-atom grants (unrestricted for
that operation) and scoped grants — currently `{navigate, #{allowed_domains
=> [...]}}}`, checked against the URL's host. `validate_args/2` performs
structural validation independent of authorization: URL scheme
allow-list (`http`/`https`/`about`/`data`), selector length bounds
(1–2047 bytes), typed text length bound (≤10000 bytes), scroll delta
bounds. Both checks run in `hk_agent_fsm` (the control plane) before a
call ever reaches `hk_browser_session` — see §8.

*(Implementation note, stated plainly per this project's honesty
requirement: `click`/`type`/`query_element`/`extract_links`/`scroll` are
implemented via `Runtime.evaluate` — JavaScript injected into the page
context — rather than the CDP `Input` domain's synthetic OS-level input
events. This is a real, intentional simplification: it is sufficient for
programmatic page interaction and observation, but it is not
indistinguishable from a physical input device the way `Input.dispatchMouseEvent`
would be. A production deployment that needs that distinction should
route these through the `Input` domain instead.)*

### HK-AGENT (`hk_agent_fsm`, `hk_agent_registry`, `hk_agent_toolbox`)

One `gen_statem` per agent, states `created -> ready -> planning ->
executing -> waiting -> planning -> ... -> completed`, with `failed` and
`terminated` reachable from any non-terminal state. `callback_mode() ->
state_functions`. The planning intelligence itself (what decides which
tool to call next) is deliberately external — a planner process, an LLM
call, a human operator, or a test drives the loop one step at a time via
`next_action(Pid, Tool, Op, Args)`. What the FSM owns is everything that
must be enforced regardless of what decided the action: capability
authorization (delegates to `hk_browser_capability` for browser ops),
argument validation, the configurable loop limits, action history, and
state transitions. `next_action/4` validates synchronously in the caller's
context (fast) but spawns a linked worker to perform the actual
browser/search call, so the FSM process is never blocked on a slow page
load — it sits in `waiting` for a real `{agent_observation, Ref, Outcome}`
message, correlated by reference, with a 45-second per-action timeout.

**Loop limits**, checked before every tool call
(`hk_agent_fsm:check_limits/1`), merged from `default_agent_limits` (app
env) over any per-agent override: `max_actions` (default 40),
`max_execution_ms` (default 300000), `max_network_requests` (default
200), `max_search_iterations` (default 5). Exceeding a limit transitions
the agent to `failed` and replies `{error, {limit_exceeded, Which}}}` to
the caller — a genuine error, not a silently-accepted `ok` (this was a
bug caught and fixed during development; see §13).

### HK-SEARCH (`hk_search_job`, `hk_search_pipeline`, `hk_search_provider`, `hk_search_provider_mock`)

One `gen_statem` per search job whose states are the pipeline stages
themselves — `user_query -> query_normalization -> search_planner ->
search_provider -> result_normalization -> url_deduplication ->
content_retrieval -> content_extraction -> source_ranking ->
result_synthesis -> frontend_visibility -> done`, with `failed` reachable
from any stage. The job's current state *is* its current pipeline stage;
there is no separate stage field that could drift from the state machine.
`source_ranking` loops back to `search_planner` (bumping `iteration`,
bounded by `max_iterations`, default 3) when fewer than 3 ranked results
were found — the concrete enforcement point for the search-iteration
limit. Every stage transition publishes an `hk_event` before advancing
(§7 has the full catalog). The search backend is a pluggable behaviour
(`hk_search_provider`: `plan/2`, `search/2`, `fetch/1`, `extract/2`); the
bundled `hk_search_provider_mock` is a deterministic, self-contained
reference implementation that makes no real network calls, so the full
pipeline is exercisable and testable without external dependencies. A
real provider (a live search API, an HTTP fetcher, an HTML/text extractor)
plugs in by implementing the same behaviour and setting
`{search_provider, Module}` in `sys.config` — *(planned: no live provider
is bundled)*.

### HK-MSG (`hk_messaging_portal`)

A minimal pub/sub message bus for inter-agent and agent/user messaging,
independent of the event bus (events are platform telemetry; messages are
addressed application payloads). `send/5` marks a message `delivered` if
the recipient is currently subscribed, `queued` otherwise;
`list_conversation/1` returns history in order; `mark_read/1` updates
status. Backed by two ETS tables (`hk_messages_tab`, keyed by
`message_id`; `hk_messages_by_conversation_tab`, keyed by
`conversation_id`).

### HK-VIS (`hk_api_*`, `priv/frontend/`)

The API layer (`hk_api_router`, `hk_api_listener` owning the Cowboy/Ranch
listener lifecycle, and one handler module per domain) and the static
observability frontend (`priv/frontend/index.html` + `js/app.js`) — see
§9 and §10.

## 5. Data model

Every entity is a record in `include/hyperkitty.hrl`, with a matching
`hk_schema:to_map/1` clause for its JSON wire form (records are the
compiler-checked internal shape; maps/JSON are produced only at the
boundary).

| Entity | Key fields |
|---|---|
| `#agent{}` | `agent_id`, `objective`, `state`, `capabilities`, `action_count`, `started_at_ms`/`updated_at_ms`, `limits`, `result`, `failure_reason` |
| `#agent_objective{}` | `objective_id`, `agent_id`, `description`, `success_criteria`, `created_at_ms` |
| `#browser_session{}` | `session_id`, `owner_agent_id`, `profile`, `pid`, `status` (`starting\|ready\|busy\|closing\|closed\|crashed`), `tabs`, timestamps |
| `#browser_tab{}` | `tab_id`, `session_id`, `url`, `title`, `status` (`opening\|loading\|idle\|closed`), timestamps |
| `#search_job{}` | `job_id`, `owner_agent_id`, `query`, `stage` (the 13-value pipeline enum), `max_iterations`, `iteration`, `results`, timestamps |
| `#search_result{}` | `result_id`, `job_id`, `url`, `normalized_url`, `title`, `snippet`, `rank`, `source_document_id`, `created_at_ms` |
| `#source_document{}` | `document_id`, `url`, `fetched_at_ms`, `content_type`, `extracted_text`, `citation_map` |
| `#message{}` | `message_id`, `sender`, `recipient`, `timestamp`, `conversation_id`, `message_type` (`user_to_agent\|agent_to_user\|agent_to_agent\|system`), `payload`, `status` |
| `#operation{}` | `operation_id`, `request_id`, `kind`, `status` (`accepted\|in_progress\|succeeded\|failed`), `subject`, `error`, timestamps |
| `hk_event:t()` (opaque) | `event_id`, `category`, `subject`, `operation_id`, `data`, `emitted_at_ms` |

## 6. Agent capability, limits, and event flow (worked example)

```
Caller                    hk_agent_fsm                 hk_browser_capability   hk_agent_toolbox        hk_browser_session
  |  next_action(browser,      |                              |                       |                       |
  |   navigate, #{url=>U})     |                              |                       |                       |
  |-------------------------->| check_limits/1               |                       |                       |
  |                            | is_granted(Caps,{navigate,U})|                       |                       |
  |                            |----------------------------->|                       |                       |
  |                            |<---- true/false --------------|                      |                       |
  |                            | validate_args(navigate, Args)|                       |                       |
  |                            |----------------------------->|                       |                       |
  |                            |<---- {ok,Args}/{error,_} -----|                      |                       |
  |                            | spawn_link(worker) -> emit "agent.action.started"    |                       |
  |                            |                              |     dispatch(browser,navigate,Args,AgentId)   |
  |                            |                              |------------------------------------------->  |
  |                            |                              |                       |  navigate/3 -> CDP -> emit "browser.navigation.*"
  |                            |<-------- {agent_observation, Ref, Outcome} ----------------------------------|
  |                            | emit "agent.action.completed"/"failed", -> planning  |                       |
  |<-------- {ok,Result} / {error,Reason} -------------------|                       |                       |
```

Rejection paths never reach `hk_browser_session`: an unauthorized call
emits `agent.action.rejected` and replies `{error, not_authorized}}`
without spawning a worker; invalid arguments emit the same category with
`{error, {invalid_arguments, Reason}}}`; a tripped limit transitions the
agent straight to `failed` and replies `{error, {limit_exceeded, Which}}}`.
This is the concrete form of "an agent's prompt is not a security
boundary" — the check is structural Erlang code sitting between the
FSM's `planning`/`ready` states and `hk_agent_toolbox`, not a
convention the planner is trusted to honor.

## 7. Event catalog and observability

Every subsystem publishes to `hk_event_bus` (a `gen_server` holding an
ETS `ordered_set` audit log capped at 20,000 entries, oldest trimmed
first). Subscribers register with an optional filter
(`category_prefix`, `subject`) and receive `{hk_event, EventMap}`
messages; the bus monitors subscribers and removes them automatically on
death. `hk_api_events_ws_h` (a `cowboy_websocket` handler) subscribes on
connect using the query string as a filter and streams every matching
event to the browser in real time; `hk_api_events_recent_h` serves the
same log as a plain GET for polling clients.

| Subsystem | Categories |
|---|---|
| Agent | `agent.created`, `agent.objective.assigned`, `agent.action.started`, `agent.action.completed`, `agent.action.failed`, `agent.action.rejected`, `agent.completed`, `agent.failed`, `agent.terminated` |
| Browser | `browser.session.created`, `browser.session.closed`, `browser.session.crashed`, `browser.tab.opened`, `browser.tab.open_failed`, `browser.tab.closed`, `browser.navigation.started`, `browser.navigation.completed`, `browser.page.read`, `browser.element.queried`, `browser.action.click`, `browser.action.type`, `browser.action.scroll`, `browser.action.failed`, `browser.screenshot.captured`, `browser.links.extracted` |
| Search | `search.started`, `search.query_normalized`, `search.query_generated`, `search.provider_called`, `search.results_received`, `search.urls_deduplicated`, `search.document_retrieved`, `search.document_extracted`, `search.source_ranked`, `search.iteration_retried`, `search.result_synthesized`, `search.completed`, `search.failed` |
| Messaging | `message.sent`, `message.received` |

`hk_search_job_tests.erl:full_pipeline_runs_in_order/0` asserts these
fire in the pipeline's relative order on every run — deterministic
orchestration is a tested property, not just a design intent.

`hk_health:report/0` computes status fresh on every call (no cached/stale
state) from `whereis/1` on all 7 subsystem supervisors plus live counts
from the three registries and the event bus's recent log; it backs
`GET /api/health` and the frontend dashboard.

## 8. Security model

1. **The agent's plan (whatever produced it — LLM, script, human) is not
   a trust boundary.** Every tool call is authorized and validated by
   `hk_agent_fsm` against `hk_browser_capability`/`hk_agent_toolbox`
   before it can reach Chromium, regardless of what asked for it.
2. **Capabilities are explicit and closed.** An agent's `capabilities`
   list is set at creation (`hk_agent_fsm:init/1`); there are exactly 14
   named browser operations and no implicit "do anything" grant.
   `navigate` supports domain scoping; other operations are all-or-nothing
   per agent.
3. **Argument validation is structural, not advisory.** URL scheme
   allow-listing rejects `javascript:`, `file:`, and other non-navigable
   schemes outright; selector and text length bounds guard against
   pathological input regardless of whether the operation is authorized.
4. **No hidden browser execution.** Every operation that reaches
   `hk_browser_session` — and its outcome — is emitted to the event bus
   under a `browser.*` category before the caller sees the result. There
   is no code path that drives Chromium without publishing what it did.
5. **No opaque network activity at the control-plane level.** The API
   layer, the event bus, and the search pipeline's stage events give an
   operator a complete, ordered account of what happened and when — see
   §7. *(What this repository does not implement: sandboxing Chromium's
   own process beyond `--no-sandbox`/`--disable-gpu` launch flags, or
   TLS/authentication on the HTTP API — both are deployment-environment
   concerns flagged as out of scope for this reference implementation,
   not silently omitted.)*

## 9. API reference

Base: `http://<host>:<api_port>` (default port `8420`, `config/sys.config`).
Every request gets a request id (the `x-request-id` header if supplied,
else minted); every mutating endpoint runs through
`hk_api_util:run_operation/4`, which creates an `#operation{}`
(`accepted` -> `succeeded`/`failed`) and returns
`{operation_id, request_id, status, result | error}`.

| Domain | Routes |
|---|---|
| Health | `GET /api/health` |
| Agents | `GET/POST /api/agents`, `GET /api/agents/:id`, `POST /api/agents/:id/objective`, `POST /api/agents/:id/actions`, `POST /api/agents/:id/complete`, `POST /api/agents/:id/terminate` |
| Browser | `GET/POST /api/browser/sessions`, `GET /api/browser/sessions/:id`, `POST /api/browser/sessions/:id/close`, `POST /api/browser/sessions/:id/tabs`, `POST .../tabs/:tab_id/navigate`, `POST .../tabs/:tab_id/actions` |
| Search | `GET/POST /api/search`, `GET /api/search/:id`, `GET /api/search/:id/result` |
| Messages | `GET/POST /api/messages` |
| Events | `GET /api/events/recent`, `WS /api/events/stream` (optional `?category_prefix=...` filter) |

Example — create an agent, grant it unscoped `navigate`, and drive one
action:

```
curl -s -XPOST localhost:8420/api/agents \
  -d '{"capabilities":["navigate","open_tab","read_page"]}'
# => {"operation_id":"op_...","status":"succeeded","result":{"agent_id":"agent_...", ...}}

curl -s -XPOST localhost:8420/api/agents/agent_.../actions \
  -d '{"tool":"browser","op":"create_session","args":{}}'
curl -s -XPOST localhost:8420/api/agents/agent_.../actions \
  -d '{"tool":"browser","op":"open_tab","args":{"session_id":"sess_...","url":"https://example.com"}}'
```

## 10. Frontend (HK-VIS)

`priv/frontend/index.html` + `js/app.js`: a single-page, dark-themed,
dependency-free (no build step) client with the 7 required views —
dashboard, agent console, browser sessions, search, messages, event
stream, audit log. `connectEventStream()` opens a WebSocket to
`/api/events/stream` on load, auto-reconnects on close, and live-updates
whichever view is relevant to each incoming event plus a running audit
log; the other views poll their REST endpoints on a refresh interval.
Served directly by Cowboy (`cowboy_static`) — no separate frontend server
or build pipeline.

## 11. Test strategy

`rebar3 eunit`: 24 tests across 7 modules, 0 failures.

| File | Covers |
|---|---|
| `hk_agent_fsm_tests.erl` | Deterministic state transitions; unauthorized-call and invalid-argument rejection; limit-exceeded failing the agent with the correct error; crash isolation between sibling agents |
| `hk_browser_capability_tests.erl` | Authorization (including domain-scoped `navigate`) and argument validation (rejecting `javascript:` URLs, oversized selectors, missing `type` text) |
| `hk_browser_session_tests.erl` | Real integration against a real headless Chromium process via CDP: navigate to a `data:` URL and read it back, extract links. Self-skips (not fails) when no `chromium_executable` is present on disk |
| `hk_search_job_tests.erl` | The full 11-stage pipeline against the mock provider, asserting every stage's event fires in pipeline order and the job reaches `done` with synthesized results |
| `hk_search_pipeline_tests.erl` | Pure functions: URL normalization, dedup, ranking |
| `hk_messaging_portal_tests.erl` | Immediate vs. queued delivery, conversation history ordering |
| `hk_supervision_tests.erl` | `rest_for_one` isolation (kill `hk_browser_sup`, verify recovery scope), event delivery ordering, browser-session crash isolation |

`test/hk_test_helper.erl` starts the full real application once
(idempotently — every test module shares one running instance rather than
starting/stopping the singleton HTTP listener per module, which was
itself a source of a race fixed during development; see §13) and leaves
it running for the test VM's lifetime; isolation between tests comes from
unique generated ids, not from process-table resets.

*(Planned, not present in this revision: property-based tests, a
Dialyzer PLT run as part of CI — `rebar3 dialyzer` is configured in
`rebar.config` but was not run to completion in this environment — and a
chaos-test suite beyond the two crash-isolation cases above.)*

## 12. Reproducible bootstrap

`Brewfile` pins the five dependencies (`erlang`, `rebar3`, `chromium`,
`git`, `node` — the last is headroom for future frontend tooling, not a
current build requirement) as a project-local Homebrew manifest, not a
global install. `bin/bootstrap.sh` runs `brew bundle`, validates each
tool is on `PATH`, patches `config/sys.config`'s `chromium_executable`
in place with the resolved binary path, then runs
`rebar3 get-deps && rebar3 compile && rebar3 eunit`. It is safe to re-run.

## 13. Notable issues found and fixed during development

Recorded here because a "produce concrete interfaces... rather than a
high-level product pitch" instruction implies the failure history is part
of the record, not just the final shape:

- **Agent FSM deadlock.** The `ready` state's tool-call clause originally
  passed `planning` (the state to resume *after* dispatch, i.e. the wrong
  argument) instead of `executing` as the post-validation transition
  target, so a successfully-validated call's internal dispatch event
  landed in `planning/3`, which has no matching clause for it and
  silently dropped it via the generic reject fallback — the calling
  process hung until its call timeout. Found via a hanging EUnit test;
  fixed by matching the pattern already correctly used in `planning/3`'s
  own tool-call clause.
- **Wrong reply type on limit-exceeded.** The original limit-exceeded
  branch reused `do_fail/3`, which always replies `ok` (it is designed
  for the external `fail/2` API) — meaning a call refused for exceeding a
  loop limit incorrectly looked like success to its caller. Fixed with a
  dedicated branch that replies `{error, {limit_exceeded, Which}}}`.
- **Nested-supervisor restart race** and **missing `trap_exit` on
  `hk_api_listener`** — both described in §3, both exercised by
  `hk_supervision_tests.erl`, both fixed (`hk_sup_util:start_link_retry/1`
  and `process_flag(trap_exit, true)` respectively) rather than papered
  over by loosening the restart-intensity budget alone.
- **Chrome's `/json/new` endpoint requires `PUT`, not `GET`**, on
  current Chromium releases (a CSRF-hardening change that broke the
  originally-written `hk_chrome_http:new_target/2`, which used `GET`).
  Fixed and covered by the real-Chromium integration tests in
  `hk_browser_session_tests.erl`.

## 14. Terminology

- **Capability** — one of the 14 named `hk_browser_capability` operations,
  optionally scoped (currently: `navigate` by allowed domain), grantable
  to an agent at creation.
- **Control plane** — the code path (`hk_agent_fsm` +
  `hk_browser_capability` + `hk_agent_toolbox`) that authorizes and
  validates every tool call, as distinct from whatever decided *which*
  call to make (the "agent" in the LLM sense).
- **Operation** — a tracked mutating API call (`#operation{}`), distinct
  from an **event** (`hk_event:t()`), which is a fire-and-forget
  observability record; an operation is created and completed by exactly
  one API request, while an event may be published by any subsystem for
  any reason, including ones with no API request behind them.
- **Stage** — one of the 13 `hk_search_job` `gen_statem` states; the
  pipeline's current stage and the job's current state are the same
  value by construction.