Add ARCHITECTURE.md
Browse files- ARCHITECTURE.md +487 -0
ARCHITECTURE.md
ADDED
|
@@ -0,0 +1,487 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# HyperKitty Chromium β Architecture
|
| 2 |
+
|
| 3 |
+
**Status: implemented reference implementation.** Every module, supervision
|
| 4 |
+
relationship, schema, and API route described below exists in this
|
| 5 |
+
repository and compiles, boots, and passes its test suite (`rebar3 eunit`:
|
| 6 |
+
24 tests, 0 failures, including real headless-Chromium integration tests
|
| 7 |
+
and chaos tests that kill live processes to verify recovery). Sections are
|
| 8 |
+
marked *(planned)* where the specification calls for something this
|
| 9 |
+
revision intentionally stubs or defers β there is no unmarked aspirational
|
| 10 |
+
content in this document.
|
| 11 |
+
|
| 12 |
+
## 1. Purpose and scope
|
| 13 |
+
|
| 14 |
+
HyperKitty Chromium is an Erlang/OTP-native browser agent platform: a
|
| 15 |
+
headless Chromium instance, driven over the Chrome DevTools Protocol
|
| 16 |
+
(CDP), wrapped in an explicit OTP supervision tree so that browser
|
| 17 |
+
control, agent orchestration, search, and messaging are separate,
|
| 18 |
+
independently-supervised, independently-observable subsystems rather than
|
| 19 |
+
library calls inside one monolithic process. The design principle it
|
| 20 |
+
exists to demonstrate: an LLM-driven "agent" is a planning function that
|
| 21 |
+
proposes tool calls; it is never trusted with direct, unchecked access to
|
| 22 |
+
a browser process. Every tool call crosses an explicit, typed capability
|
| 23 |
+
boundary β enforced by Erlang code, not by the prompt β before it reaches
|
| 24 |
+
Chromium.
|
| 25 |
+
|
| 26 |
+
This document is the architecture reference for engineers building on,
|
| 27 |
+
operating, or extending this codebase. It assumes familiarity with
|
| 28 |
+
Erlang/OTP (supervisors, `gen_server`, `gen_statem`) and HTTP/WebSocket
|
| 29 |
+
APIs, but not with this specific codebase.
|
| 30 |
+
|
| 31 |
+
## 2. Repository layout
|
| 32 |
+
|
| 33 |
+
```
|
| 34 |
+
hyperkitty/
|
| 35 |
+
rebar.config deps, relx release config, profiles, dialyzer/xref
|
| 36 |
+
config/sys.config application env (chromium path, API port, limits)
|
| 37 |
+
config/vm.args node name, cookie, scheduler flags
|
| 38 |
+
include/hyperkitty.hrl canonical entity records (data model)
|
| 39 |
+
src/
|
| 40 |
+
hyperkitty_app.erl application callback
|
| 41 |
+
hyperkitty_sup.erl HK-CORE root supervisor
|
| 42 |
+
hk_schema.erl record -> JSON map projection (wire format)
|
| 43 |
+
hk_id.erl, hk_event.erl ids, timestamps, the event envelope
|
| 44 |
+
hk_sup_util.erl shared "wait for the old holder to die" helper
|
| 45 |
+
hk_event_bus.erl / _sup event bus (pub/sub + audit log)
|
| 46 |
+
hk_messaging_portal.erl HK-MSG
|
| 47 |
+
hk_cdp_client.erl one CDP WebSocket connection (via `gun`)
|
| 48 |
+
hk_chrome_http.erl Chrome's plain-HTTP /json control endpoint
|
| 49 |
+
hk_browser_*.erl HK-BROWSER: session, capability, registry, supervisors
|
| 50 |
+
hk_agent_*.erl HK-AGENT: the agent state machine, registry, toolbox
|
| 51 |
+
hk_search_*.erl HK-SEARCH: the 11-stage pipeline job, provider behaviour
|
| 52 |
+
hk_operation_store.erl operation_id lifecycle (accepted/succeeded/failed)
|
| 53 |
+
hk_api_*.erl HK-VIS backend: Cowboy router, listener, REST + WS handlers
|
| 54 |
+
hk_frontend_*.erl connected-client tracking for the observability UI
|
| 55 |
+
hk_health.erl aggregate subsystem health
|
| 56 |
+
priv/frontend/ the 7-view observability UI (static HTML/JS, no build step)
|
| 57 |
+
test/ EUnit suites (see Β§11)
|
| 58 |
+
Brewfile, bin/bootstrap.sh reproducible environment setup (see Β§12)
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
## 3. Supervision tree
|
| 62 |
+
|
| 63 |
+
```
|
| 64 |
+
hyperkitty_sup (rest_for_one, intensity 8/10)
|
| 65 |
+
+-- hk_event_sup (one_for_one)
|
| 66 |
+
| +-- hk_event_bus [gen_server]
|
| 67 |
+
+-- hk_messaging_sup (one_for_one)
|
| 68 |
+
| +-- hk_messaging_portal [gen_server]
|
| 69 |
+
+-- hk_browser_sup (one_for_one)
|
| 70 |
+
| +-- hk_browser_registry [gen_server] session_id -> pid
|
| 71 |
+
| +-- hk_browser_session_sup (simple_one_for_one, transient)
|
| 72 |
+
| +-- hk_browser_session (one per session) [gen_server] owns a Chrome OS process
|
| 73 |
+
+-- hk_agent_sup (one_for_one)
|
| 74 |
+
| +-- hk_agent_registry [gen_server] agent_id -> pid
|
| 75 |
+
| +-- hk_agent_fsm_sup (simple_one_for_one, transient)
|
| 76 |
+
| +-- hk_agent_fsm (one per agent) [gen_statem]
|
| 77 |
+
+-- hk_search_sup (one_for_one)
|
| 78 |
+
| +-- hk_search_registry [gen_server] job_id -> pid
|
| 79 |
+
| +-- hk_search_job_sup (simple_one_for_one, temporary)
|
| 80 |
+
| +-- hk_search_job (one per search) [gen_statem]
|
| 81 |
+
+-- hk_health [gen_server, worker]
|
| 82 |
+
+-- hk_api_sup (one_for_one)
|
| 83 |
+
| +-- hk_api_listener [gen_server] owns the Cowboy/Ranch listener
|
| 84 |
+
+-- hk_frontend_sup (one_for_one)
|
| 85 |
+
+-- hk_frontend_client_registry [gen_server] connected WS clients
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
**Why `rest_for_one` at the root.** Subsystems are listed in dependency
|
| 89 |
+
order. If `hk_event_sup` crashes, everything after it is restarted too,
|
| 90 |
+
because every later subsystem holds an implicit dependency on the event
|
| 91 |
+
bus (agents, browser sessions, and search jobs all publish to it; the API
|
| 92 |
+
layer's WebSocket clients have subscriptions that would otherwise go
|
| 93 |
+
silently stale against a bus that no longer exists). A crash in
|
| 94 |
+
`hk_browser_sup`, by contrast, restarts `hk_agent_sup`, `hk_search_sup`,
|
| 95 |
+
`hk_health`, `hk_api_sup`, and `hk_frontend_sup`, but leaves
|
| 96 |
+
`hk_event_sup` and `hk_messaging_sup` β listed *before* it β running
|
| 97 |
+
undisturbed. `test/hk_supervision_tests.erl:rest_for_one_isolation/0`
|
| 98 |
+
kills `hk_browser_sup` directly and asserts exactly this shape of
|
| 99 |
+
recovery.
|
| 100 |
+
|
| 101 |
+
**Why each plane is a two-level tree, not one `simple_one_for_one`
|
| 102 |
+
supervisor.** OTP does not allow mixing a singleton child (the registry)
|
| 103 |
+
with `simple_one_for_one` dynamic children (session/agent/job instances)
|
| 104 |
+
under the same supervisor. Splitting each plane into a `one_for_one`
|
| 105 |
+
supervisor wrapping (a) a singleton registry and (b) a `simple_one_for_one`
|
| 106 |
+
instance pool means the registry restarting does not restart live
|
| 107 |
+
sessions/agents/jobs (they re-register on next lookup miss; live
|
| 108 |
+
processes are untouched), and a crash-restart storm in the instance pool
|
| 109 |
+
cannot, via `hk_browser_sup`'s own intensity/period, take the registry
|
| 110 |
+
down with it.
|
| 111 |
+
|
| 112 |
+
**A known, fixed race in this shape, documented for anyone extending
|
| 113 |
+
it.** An abrupt, unmaskable kill of a mid-tree supervisor (as opposed to
|
| 114 |
+
the ordinary synchronous shutdown protocol a supervisor uses on its own
|
| 115 |
+
children) does not guarantee its nested-supervisor children have finished
|
| 116 |
+
their own termination and released their registered name before the
|
| 117 |
+
grandparent supervisor's restart attempt runs. `hk_sup_util:start_link_retry/1`
|
| 118 |
+
wraps every registered-name `start_link/0` in this application: on
|
| 119 |
+
`{error, {already_started, OldPid}}` it monitors `OldPid`, waits for it to
|
| 120 |
+
actually exit, and retries once, rather than surfacing the transient race
|
| 121 |
+
as a failed restart. Separately, `hk_api_listener` calls
|
| 122 |
+
`process_flag(trap_exit, true)` in `init/1` β without it, the ordinary
|
| 123 |
+
supervisor shutdown signal (`exit(Pid, shutdown)`) would kill it directly
|
| 124 |
+
without ever running `terminate/2`, leaking the underlying Ranch/Cowboy
|
| 125 |
+
listener and its registered name on every restart, not only under abrupt
|
| 126 |
+
kills. Both are exercised by `hk_supervision_tests.erl` and by
|
| 127 |
+
`hk_browser_session_tests.erl`'s crash-isolation test.
|
| 128 |
+
|
| 129 |
+
## 4. Subsystems
|
| 130 |
+
|
| 131 |
+
### HK-CORE (`hyperkitty_sup`, `hk_event_bus`, `hk_schema`, `hk_id`, `hk_sup_util`)
|
| 132 |
+
|
| 133 |
+
Boot sequencing, the root supervision policy (Β§3), the canonical
|
| 134 |
+
record-to-JSON projection (`hk_schema:to_map/1` β the single place a wire
|
| 135 |
+
representation is produced; internal code passes records, never
|
| 136 |
+
hand-built maps, until it crosses this boundary), id/timestamp generation,
|
| 137 |
+
and the event bus described in Β§7.
|
| 138 |
+
|
| 139 |
+
### HK-BROWSER (`hk_browser_session`, `hk_cdp_client`, `hk_chrome_http`, `hk_browser_capability`, `hk_browser_registry`)
|
| 140 |
+
|
| 141 |
+
One `hk_browser_session` gen_server per browser session. `init/1` finds a
|
| 142 |
+
free TCP port, launches a real headless Chromium OS process via
|
| 143 |
+
`erlang:open_port({spawn_executable, ...}, ["--headless=new",
|
| 144 |
+
"--remote-debugging-port=<port>", "--remote-debugging-address=127.0.0.1",
|
| 145 |
+
"--user-data-dir=<tmp>", "--no-sandbox", "--disable-gpu",
|
| 146 |
+
"--disable-dev-shm-usage", "about:blank"])`, polls Chrome's plain-HTTP
|
| 147 |
+
`/json/version` endpoint until it answers, and registers itself with
|
| 148 |
+
`hk_browser_registry`. Tab lifecycle (`open_tab`/`close_tab`) goes through
|
| 149 |
+
Chrome's HTTP control surface (`hk_chrome_http`); per-tab interaction goes
|
| 150 |
+
over CDP WebSocket JSON-RPC (`hk_cdp_client`, built on `gun`).
|
| 151 |
+
Navigation correlates against the `Page.loadEventFired` CDP event via a
|
| 152 |
+
waiters map. A Chrome OS process crash is detected in `handle_info` via
|
| 153 |
+
the port's `{exit_status, Status}` message and stops the session with an
|
| 154 |
+
abnormal reason, which `hk_browser_session_sup` (restart type `transient`)
|
| 155 |
+
restarts under the same `session_id` β the concrete demonstration that one
|
| 156 |
+
crashed session never takes down another (`browser_session_crash_isolation/0`)
|
| 157 |
+
or any other subsystem.
|
| 158 |
+
|
| 159 |
+
**Typed capability interface.** `hk_browser_capability:operations/0` is
|
| 160 |
+
the closed list of 14 operations (`create_session`, `close_session`,
|
| 161 |
+
`open_tab`, `close_tab`, `navigate`, `back`, `forward`, `reload`,
|
| 162 |
+
`read_page`, `query_element`, `click`, `type`, `scroll`, `screenshot`,
|
| 163 |
+
`extract_links`). There is no operation outside this list and no
|
| 164 |
+
unrestricted "just run this JS" escape hatch. `is_granted/2` checks an
|
| 165 |
+
agent's capability list, which supports bare-atom grants (unrestricted for
|
| 166 |
+
that operation) and scoped grants β currently `{navigate, #{allowed_domains
|
| 167 |
+
=> [...]}}}`, checked against the URL's host. `validate_args/2` performs
|
| 168 |
+
structural validation independent of authorization: URL scheme
|
| 169 |
+
allow-list (`http`/`https`/`about`/`data`), selector length bounds
|
| 170 |
+
(1β2047 bytes), typed text length bound (β€10000 bytes), scroll delta
|
| 171 |
+
bounds. Both checks run in `hk_agent_fsm` (the control plane) before a
|
| 172 |
+
call ever reaches `hk_browser_session` β see Β§8.
|
| 173 |
+
|
| 174 |
+
*(Implementation note, stated plainly per this project's honesty
|
| 175 |
+
requirement: `click`/`type`/`query_element`/`extract_links`/`scroll` are
|
| 176 |
+
implemented via `Runtime.evaluate` β JavaScript injected into the page
|
| 177 |
+
context β rather than the CDP `Input` domain's synthetic OS-level input
|
| 178 |
+
events. This is a real, intentional simplification: it is sufficient for
|
| 179 |
+
programmatic page interaction and observation, but it is not
|
| 180 |
+
indistinguishable from a physical input device the way `Input.dispatchMouseEvent`
|
| 181 |
+
would be. A production deployment that needs that distinction should
|
| 182 |
+
route these through the `Input` domain instead.)*
|
| 183 |
+
|
| 184 |
+
### HK-AGENT (`hk_agent_fsm`, `hk_agent_registry`, `hk_agent_toolbox`)
|
| 185 |
+
|
| 186 |
+
One `gen_statem` per agent, states `created -> ready -> planning ->
|
| 187 |
+
executing -> waiting -> planning -> ... -> completed`, with `failed` and
|
| 188 |
+
`terminated` reachable from any non-terminal state. `callback_mode() ->
|
| 189 |
+
state_functions`. The planning intelligence itself (what decides which
|
| 190 |
+
tool to call next) is deliberately external β a planner process, an LLM
|
| 191 |
+
call, a human operator, or a test drives the loop one step at a time via
|
| 192 |
+
`next_action(Pid, Tool, Op, Args)`. What the FSM owns is everything that
|
| 193 |
+
must be enforced regardless of what decided the action: capability
|
| 194 |
+
authorization (delegates to `hk_browser_capability` for browser ops),
|
| 195 |
+
argument validation, the configurable loop limits, action history, and
|
| 196 |
+
state transitions. `next_action/4` validates synchronously in the caller's
|
| 197 |
+
context (fast) but spawns a linked worker to perform the actual
|
| 198 |
+
browser/search call, so the FSM process is never blocked on a slow page
|
| 199 |
+
load β it sits in `waiting` for a real `{agent_observation, Ref, Outcome}`
|
| 200 |
+
message, correlated by reference, with a 45-second per-action timeout.
|
| 201 |
+
|
| 202 |
+
**Loop limits**, checked before every tool call
|
| 203 |
+
(`hk_agent_fsm:check_limits/1`), merged from `default_agent_limits` (app
|
| 204 |
+
env) over any per-agent override: `max_actions` (default 40),
|
| 205 |
+
`max_execution_ms` (default 300000), `max_network_requests` (default
|
| 206 |
+
200), `max_search_iterations` (default 5). Exceeding a limit transitions
|
| 207 |
+
the agent to `failed` and replies `{error, {limit_exceeded, Which}}}` to
|
| 208 |
+
the caller β a genuine error, not a silently-accepted `ok` (this was a
|
| 209 |
+
bug caught and fixed during development; see Β§13).
|
| 210 |
+
|
| 211 |
+
### HK-SEARCH (`hk_search_job`, `hk_search_pipeline`, `hk_search_provider`, `hk_search_provider_mock`)
|
| 212 |
+
|
| 213 |
+
One `gen_statem` per search job whose states are the pipeline stages
|
| 214 |
+
themselves β `user_query -> query_normalization -> search_planner ->
|
| 215 |
+
search_provider -> result_normalization -> url_deduplication ->
|
| 216 |
+
content_retrieval -> content_extraction -> source_ranking ->
|
| 217 |
+
result_synthesis -> frontend_visibility -> done`, with `failed` reachable
|
| 218 |
+
from any stage. The job's current state *is* its current pipeline stage;
|
| 219 |
+
there is no separate stage field that could drift from the state machine.
|
| 220 |
+
`source_ranking` loops back to `search_planner` (bumping `iteration`,
|
| 221 |
+
bounded by `max_iterations`, default 3) when fewer than 3 ranked results
|
| 222 |
+
were found β the concrete enforcement point for the search-iteration
|
| 223 |
+
limit. Every stage transition publishes an `hk_event` before advancing
|
| 224 |
+
(Β§7 has the full catalog). The search backend is a pluggable behaviour
|
| 225 |
+
(`hk_search_provider`: `plan/2`, `search/2`, `fetch/1`, `extract/2`); the
|
| 226 |
+
bundled `hk_search_provider_mock` is a deterministic, self-contained
|
| 227 |
+
reference implementation that makes no real network calls, so the full
|
| 228 |
+
pipeline is exercisable and testable without external dependencies. A
|
| 229 |
+
real provider (a live search API, an HTTP fetcher, an HTML/text extractor)
|
| 230 |
+
plugs in by implementing the same behaviour and setting
|
| 231 |
+
`{search_provider, Module}` in `sys.config` β *(planned: no live provider
|
| 232 |
+
is bundled)*.
|
| 233 |
+
|
| 234 |
+
### HK-MSG (`hk_messaging_portal`)
|
| 235 |
+
|
| 236 |
+
A minimal pub/sub message bus for inter-agent and agent/user messaging,
|
| 237 |
+
independent of the event bus (events are platform telemetry; messages are
|
| 238 |
+
addressed application payloads). `send/5` marks a message `delivered` if
|
| 239 |
+
the recipient is currently subscribed, `queued` otherwise;
|
| 240 |
+
`list_conversation/1` returns history in order; `mark_read/1` updates
|
| 241 |
+
status. Backed by two ETS tables (`hk_messages_tab`, keyed by
|
| 242 |
+
`message_id`; `hk_messages_by_conversation_tab`, keyed by
|
| 243 |
+
`conversation_id`).
|
| 244 |
+
|
| 245 |
+
### HK-VIS (`hk_api_*`, `priv/frontend/`)
|
| 246 |
+
|
| 247 |
+
The API layer (`hk_api_router`, `hk_api_listener` owning the Cowboy/Ranch
|
| 248 |
+
listener lifecycle, and one handler module per domain) and the static
|
| 249 |
+
observability frontend (`priv/frontend/index.html` + `js/app.js`) β see
|
| 250 |
+
Β§9 and Β§10.
|
| 251 |
+
|
| 252 |
+
## 5. Data model
|
| 253 |
+
|
| 254 |
+
Every entity is a record in `include/hyperkitty.hrl`, with a matching
|
| 255 |
+
`hk_schema:to_map/1` clause for its JSON wire form (records are the
|
| 256 |
+
compiler-checked internal shape; maps/JSON are produced only at the
|
| 257 |
+
boundary).
|
| 258 |
+
|
| 259 |
+
| Entity | Key fields |
|
| 260 |
+
|---|---|
|
| 261 |
+
| `#agent{}` | `agent_id`, `objective`, `state`, `capabilities`, `action_count`, `started_at_ms`/`updated_at_ms`, `limits`, `result`, `failure_reason` |
|
| 262 |
+
| `#agent_objective{}` | `objective_id`, `agent_id`, `description`, `success_criteria`, `created_at_ms` |
|
| 263 |
+
| `#browser_session{}` | `session_id`, `owner_agent_id`, `profile`, `pid`, `status` (`starting\|ready\|busy\|closing\|closed\|crashed`), `tabs`, timestamps |
|
| 264 |
+
| `#browser_tab{}` | `tab_id`, `session_id`, `url`, `title`, `status` (`opening\|loading\|idle\|closed`), timestamps |
|
| 265 |
+
| `#search_job{}` | `job_id`, `owner_agent_id`, `query`, `stage` (the 13-value pipeline enum), `max_iterations`, `iteration`, `results`, timestamps |
|
| 266 |
+
| `#search_result{}` | `result_id`, `job_id`, `url`, `normalized_url`, `title`, `snippet`, `rank`, `source_document_id`, `created_at_ms` |
|
| 267 |
+
| `#source_document{}` | `document_id`, `url`, `fetched_at_ms`, `content_type`, `extracted_text`, `citation_map` |
|
| 268 |
+
| `#message{}` | `message_id`, `sender`, `recipient`, `timestamp`, `conversation_id`, `message_type` (`user_to_agent\|agent_to_user\|agent_to_agent\|system`), `payload`, `status` |
|
| 269 |
+
| `#operation{}` | `operation_id`, `request_id`, `kind`, `status` (`accepted\|in_progress\|succeeded\|failed`), `subject`, `error`, timestamps |
|
| 270 |
+
| `hk_event:t()` (opaque) | `event_id`, `category`, `subject`, `operation_id`, `data`, `emitted_at_ms` |
|
| 271 |
+
|
| 272 |
+
## 6. Agent capability, limits, and event flow (worked example)
|
| 273 |
+
|
| 274 |
+
```
|
| 275 |
+
Caller hk_agent_fsm hk_browser_capability hk_agent_toolbox hk_browser_session
|
| 276 |
+
| next_action(browser, | | | |
|
| 277 |
+
| navigate, #{url=>U}) | | | |
|
| 278 |
+
|-------------------------->| check_limits/1 | | |
|
| 279 |
+
| | is_granted(Caps,{navigate,U})| | |
|
| 280 |
+
| |----------------------------->| | |
|
| 281 |
+
| |<---- true/false --------------| | |
|
| 282 |
+
| | validate_args(navigate, Args)| | |
|
| 283 |
+
| |----------------------------->| | |
|
| 284 |
+
| |<---- {ok,Args}/{error,_} -----| | |
|
| 285 |
+
| | spawn_link(worker) -> emit "agent.action.started" | |
|
| 286 |
+
| | | dispatch(browser,navigate,Args,AgentId) |
|
| 287 |
+
| | |-------------------------------------------> |
|
| 288 |
+
| | | | navigate/3 -> CDP -> emit "browser.navigation.*"
|
| 289 |
+
| |<-------- {agent_observation, Ref, Outcome} ----------------------------------|
|
| 290 |
+
| | emit "agent.action.completed"/"failed", -> planning | |
|
| 291 |
+
|<-------- {ok,Result} / {error,Reason} -------------------| | |
|
| 292 |
+
```
|
| 293 |
+
|
| 294 |
+
Rejection paths never reach `hk_browser_session`: an unauthorized call
|
| 295 |
+
emits `agent.action.rejected` and replies `{error, not_authorized}}`
|
| 296 |
+
without spawning a worker; invalid arguments emit the same category with
|
| 297 |
+
`{error, {invalid_arguments, Reason}}}`; a tripped limit transitions the
|
| 298 |
+
agent straight to `failed` and replies `{error, {limit_exceeded, Which}}}`.
|
| 299 |
+
This is the concrete form of "an agent's prompt is not a security
|
| 300 |
+
boundary" β the check is structural Erlang code sitting between the
|
| 301 |
+
FSM's `planning`/`ready` states and `hk_agent_toolbox`, not a
|
| 302 |
+
convention the planner is trusted to honor.
|
| 303 |
+
|
| 304 |
+
## 7. Event catalog and observability
|
| 305 |
+
|
| 306 |
+
Every subsystem publishes to `hk_event_bus` (a `gen_server` holding an
|
| 307 |
+
ETS `ordered_set` audit log capped at 20,000 entries, oldest trimmed
|
| 308 |
+
first). Subscribers register with an optional filter
|
| 309 |
+
(`category_prefix`, `subject`) and receive `{hk_event, EventMap}`
|
| 310 |
+
messages; the bus monitors subscribers and removes them automatically on
|
| 311 |
+
death. `hk_api_events_ws_h` (a `cowboy_websocket` handler) subscribes on
|
| 312 |
+
connect using the query string as a filter and streams every matching
|
| 313 |
+
event to the browser in real time; `hk_api_events_recent_h` serves the
|
| 314 |
+
same log as a plain GET for polling clients.
|
| 315 |
+
|
| 316 |
+
| Subsystem | Categories |
|
| 317 |
+
|---|---|
|
| 318 |
+
| Agent | `agent.created`, `agent.objective.assigned`, `agent.action.started`, `agent.action.completed`, `agent.action.failed`, `agent.action.rejected`, `agent.completed`, `agent.failed`, `agent.terminated` |
|
| 319 |
+
| Browser | `browser.session.created`, `browser.session.closed`, `browser.session.crashed`, `browser.tab.opened`, `browser.tab.open_failed`, `browser.tab.closed`, `browser.navigation.started`, `browser.navigation.completed`, `browser.page.read`, `browser.element.queried`, `browser.action.click`, `browser.action.type`, `browser.action.scroll`, `browser.action.failed`, `browser.screenshot.captured`, `browser.links.extracted` |
|
| 320 |
+
| Search | `search.started`, `search.query_normalized`, `search.query_generated`, `search.provider_called`, `search.results_received`, `search.urls_deduplicated`, `search.document_retrieved`, `search.document_extracted`, `search.source_ranked`, `search.iteration_retried`, `search.result_synthesized`, `search.completed`, `search.failed` |
|
| 321 |
+
| Messaging | `message.sent`, `message.received` |
|
| 322 |
+
|
| 323 |
+
`hk_search_job_tests.erl:full_pipeline_runs_in_order/0` asserts these
|
| 324 |
+
fire in the pipeline's relative order on every run β deterministic
|
| 325 |
+
orchestration is a tested property, not just a design intent.
|
| 326 |
+
|
| 327 |
+
`hk_health:report/0` computes status fresh on every call (no cached/stale
|
| 328 |
+
state) from `whereis/1` on all 7 subsystem supervisors plus live counts
|
| 329 |
+
from the three registries and the event bus's recent log; it backs
|
| 330 |
+
`GET /api/health` and the frontend dashboard.
|
| 331 |
+
|
| 332 |
+
## 8. Security model
|
| 333 |
+
|
| 334 |
+
1. **The agent's plan (whatever produced it β LLM, script, human) is not
|
| 335 |
+
a trust boundary.** Every tool call is authorized and validated by
|
| 336 |
+
`hk_agent_fsm` against `hk_browser_capability`/`hk_agent_toolbox`
|
| 337 |
+
before it can reach Chromium, regardless of what asked for it.
|
| 338 |
+
2. **Capabilities are explicit and closed.** An agent's `capabilities`
|
| 339 |
+
list is set at creation (`hk_agent_fsm:init/1`); there are exactly 14
|
| 340 |
+
named browser operations and no implicit "do anything" grant.
|
| 341 |
+
`navigate` supports domain scoping; other operations are all-or-nothing
|
| 342 |
+
per agent.
|
| 343 |
+
3. **Argument validation is structural, not advisory.** URL scheme
|
| 344 |
+
allow-listing rejects `javascript:`, `file:`, and other non-navigable
|
| 345 |
+
schemes outright; selector and text length bounds guard against
|
| 346 |
+
pathological input regardless of whether the operation is authorized.
|
| 347 |
+
4. **No hidden browser execution.** Every operation that reaches
|
| 348 |
+
`hk_browser_session` β and its outcome β is emitted to the event bus
|
| 349 |
+
under a `browser.*` category before the caller sees the result. There
|
| 350 |
+
is no code path that drives Chromium without publishing what it did.
|
| 351 |
+
5. **No opaque network activity at the control-plane level.** The API
|
| 352 |
+
layer, the event bus, and the search pipeline's stage events give an
|
| 353 |
+
operator a complete, ordered account of what happened and when β see
|
| 354 |
+
Β§7. *(What this repository does not implement: sandboxing Chromium's
|
| 355 |
+
own process beyond `--no-sandbox`/`--disable-gpu` launch flags, or
|
| 356 |
+
TLS/authentication on the HTTP API β both are deployment-environment
|
| 357 |
+
concerns flagged as out of scope for this reference implementation,
|
| 358 |
+
not silently omitted.)*
|
| 359 |
+
|
| 360 |
+
## 9. API reference
|
| 361 |
+
|
| 362 |
+
Base: `http://<host>:<api_port>` (default port `8420`, `config/sys.config`).
|
| 363 |
+
Every request gets a request id (the `x-request-id` header if supplied,
|
| 364 |
+
else minted); every mutating endpoint runs through
|
| 365 |
+
`hk_api_util:run_operation/4`, which creates an `#operation{}`
|
| 366 |
+
(`accepted` -> `succeeded`/`failed`) and returns
|
| 367 |
+
`{operation_id, request_id, status, result | error}`.
|
| 368 |
+
|
| 369 |
+
| Domain | Routes |
|
| 370 |
+
|---|---|
|
| 371 |
+
| Health | `GET /api/health` |
|
| 372 |
+
| Agents | `GET/POST /api/agents`, `GET /api/agents/:id`, `POST /api/agents/:id/objective`, `POST /api/agents/:id/actions`, `POST /api/agents/:id/complete`, `POST /api/agents/:id/terminate` |
|
| 373 |
+
| Browser | `GET/POST /api/browser/sessions`, `GET /api/browser/sessions/:id`, `POST /api/browser/sessions/:id/close`, `POST /api/browser/sessions/:id/tabs`, `POST .../tabs/:tab_id/navigate`, `POST .../tabs/:tab_id/actions` |
|
| 374 |
+
| Search | `GET/POST /api/search`, `GET /api/search/:id`, `GET /api/search/:id/result` |
|
| 375 |
+
| Messages | `GET/POST /api/messages` |
|
| 376 |
+
| Events | `GET /api/events/recent`, `WS /api/events/stream` (optional `?category_prefix=...` filter) |
|
| 377 |
+
|
| 378 |
+
Example β create an agent, grant it unscoped `navigate`, and drive one
|
| 379 |
+
action:
|
| 380 |
+
|
| 381 |
+
```
|
| 382 |
+
curl -s -XPOST localhost:8420/api/agents \
|
| 383 |
+
-d '{"capabilities":["navigate","open_tab","read_page"]}'
|
| 384 |
+
# => {"operation_id":"op_...","status":"succeeded","result":{"agent_id":"agent_...", ...}}
|
| 385 |
+
|
| 386 |
+
curl -s -XPOST localhost:8420/api/agents/agent_.../actions \
|
| 387 |
+
-d '{"tool":"browser","op":"create_session","args":{}}'
|
| 388 |
+
curl -s -XPOST localhost:8420/api/agents/agent_.../actions \
|
| 389 |
+
-d '{"tool":"browser","op":"open_tab","args":{"session_id":"sess_...","url":"https://example.com"}}'
|
| 390 |
+
```
|
| 391 |
+
|
| 392 |
+
## 10. Frontend (HK-VIS)
|
| 393 |
+
|
| 394 |
+
`priv/frontend/index.html` + `js/app.js`: a single-page, dark-themed,
|
| 395 |
+
dependency-free (no build step) client with the 7 required views β
|
| 396 |
+
dashboard, agent console, browser sessions, search, messages, event
|
| 397 |
+
stream, audit log. `connectEventStream()` opens a WebSocket to
|
| 398 |
+
`/api/events/stream` on load, auto-reconnects on close, and live-updates
|
| 399 |
+
whichever view is relevant to each incoming event plus a running audit
|
| 400 |
+
log; the other views poll their REST endpoints on a refresh interval.
|
| 401 |
+
Served directly by Cowboy (`cowboy_static`) β no separate frontend server
|
| 402 |
+
or build pipeline.
|
| 403 |
+
|
| 404 |
+
## 11. Test strategy
|
| 405 |
+
|
| 406 |
+
`rebar3 eunit`: 24 tests across 7 modules, 0 failures.
|
| 407 |
+
|
| 408 |
+
| File | Covers |
|
| 409 |
+
|---|---|
|
| 410 |
+
| `hk_agent_fsm_tests.erl` | Deterministic state transitions; unauthorized-call and invalid-argument rejection; limit-exceeded failing the agent with the correct error; crash isolation between sibling agents |
|
| 411 |
+
| `hk_browser_capability_tests.erl` | Authorization (including domain-scoped `navigate`) and argument validation (rejecting `javascript:` URLs, oversized selectors, missing `type` text) |
|
| 412 |
+
| `hk_browser_session_tests.erl` | Real integration against a real headless Chromium process via CDP: navigate to a `data:` URL and read it back, extract links. Self-skips (not fails) when no `chromium_executable` is present on disk |
|
| 413 |
+
| `hk_search_job_tests.erl` | The full 11-stage pipeline against the mock provider, asserting every stage's event fires in pipeline order and the job reaches `done` with synthesized results |
|
| 414 |
+
| `hk_search_pipeline_tests.erl` | Pure functions: URL normalization, dedup, ranking |
|
| 415 |
+
| `hk_messaging_portal_tests.erl` | Immediate vs. queued delivery, conversation history ordering |
|
| 416 |
+
| `hk_supervision_tests.erl` | `rest_for_one` isolation (kill `hk_browser_sup`, verify recovery scope), event delivery ordering, browser-session crash isolation |
|
| 417 |
+
|
| 418 |
+
`test/hk_test_helper.erl` starts the full real application once
|
| 419 |
+
(idempotently β every test module shares one running instance rather than
|
| 420 |
+
starting/stopping the singleton HTTP listener per module, which was
|
| 421 |
+
itself a source of a race fixed during development; see Β§13) and leaves
|
| 422 |
+
it running for the test VM's lifetime; isolation between tests comes from
|
| 423 |
+
unique generated ids, not from process-table resets.
|
| 424 |
+
|
| 425 |
+
*(Planned, not present in this revision: property-based tests, a
|
| 426 |
+
Dialyzer PLT run as part of CI β `rebar3 dialyzer` is configured in
|
| 427 |
+
`rebar.config` but was not run to completion in this environment β and a
|
| 428 |
+
chaos-test suite beyond the two crash-isolation cases above.)*
|
| 429 |
+
|
| 430 |
+
## 12. Reproducible bootstrap
|
| 431 |
+
|
| 432 |
+
`Brewfile` pins the five dependencies (`erlang`, `rebar3`, `chromium`,
|
| 433 |
+
`git`, `node` β the last is headroom for future frontend tooling, not a
|
| 434 |
+
current build requirement) as a project-local Homebrew manifest, not a
|
| 435 |
+
global install. `bin/bootstrap.sh` runs `brew bundle`, validates each
|
| 436 |
+
tool is on `PATH`, patches `config/sys.config`'s `chromium_executable`
|
| 437 |
+
in place with the resolved binary path, then runs
|
| 438 |
+
`rebar3 get-deps && rebar3 compile && rebar3 eunit`. It is safe to re-run.
|
| 439 |
+
|
| 440 |
+
## 13. Notable issues found and fixed during development
|
| 441 |
+
|
| 442 |
+
Recorded here because a "produce concrete interfaces... rather than a
|
| 443 |
+
high-level product pitch" instruction implies the failure history is part
|
| 444 |
+
of the record, not just the final shape:
|
| 445 |
+
|
| 446 |
+
- **Agent FSM deadlock.** The `ready` state's tool-call clause originally
|
| 447 |
+
passed `planning` (the state to resume *after* dispatch, i.e. the wrong
|
| 448 |
+
argument) instead of `executing` as the post-validation transition
|
| 449 |
+
target, so a successfully-validated call's internal dispatch event
|
| 450 |
+
landed in `planning/3`, which has no matching clause for it and
|
| 451 |
+
silently dropped it via the generic reject fallback β the calling
|
| 452 |
+
process hung until its call timeout. Found via a hanging EUnit test;
|
| 453 |
+
fixed by matching the pattern already correctly used in `planning/3`'s
|
| 454 |
+
own tool-call clause.
|
| 455 |
+
- **Wrong reply type on limit-exceeded.** The original limit-exceeded
|
| 456 |
+
branch reused `do_fail/3`, which always replies `ok` (it is designed
|
| 457 |
+
for the external `fail/2` API) β meaning a call refused for exceeding a
|
| 458 |
+
loop limit incorrectly looked like success to its caller. Fixed with a
|
| 459 |
+
dedicated branch that replies `{error, {limit_exceeded, Which}}}`.
|
| 460 |
+
- **Nested-supervisor restart race** and **missing `trap_exit` on
|
| 461 |
+
`hk_api_listener`** β both described in Β§3, both exercised by
|
| 462 |
+
`hk_supervision_tests.erl`, both fixed (`hk_sup_util:start_link_retry/1`
|
| 463 |
+
and `process_flag(trap_exit, true)` respectively) rather than papered
|
| 464 |
+
over by loosening the restart-intensity budget alone.
|
| 465 |
+
- **Chrome's `/json/new` endpoint requires `PUT`, not `GET`**, on
|
| 466 |
+
current Chromium releases (a CSRF-hardening change that broke the
|
| 467 |
+
originally-written `hk_chrome_http:new_target/2`, which used `GET`).
|
| 468 |
+
Fixed and covered by the real-Chromium integration tests in
|
| 469 |
+
`hk_browser_session_tests.erl`.
|
| 470 |
+
|
| 471 |
+
## 14. Terminology
|
| 472 |
+
|
| 473 |
+
- **Capability** β one of the 14 named `hk_browser_capability` operations,
|
| 474 |
+
optionally scoped (currently: `navigate` by allowed domain), grantable
|
| 475 |
+
to an agent at creation.
|
| 476 |
+
- **Control plane** β the code path (`hk_agent_fsm` +
|
| 477 |
+
`hk_browser_capability` + `hk_agent_toolbox`) that authorizes and
|
| 478 |
+
validates every tool call, as distinct from whatever decided *which*
|
| 479 |
+
call to make (the "agent" in the LLM sense).
|
| 480 |
+
- **Operation** β a tracked mutating API call (`#operation{}`), distinct
|
| 481 |
+
from an **event** (`hk_event:t()`), which is a fire-and-forget
|
| 482 |
+
observability record; an operation is created and completed by exactly
|
| 483 |
+
one API request, while an event may be published by any subsystem for
|
| 484 |
+
any reason, including ones with no API request behind them.
|
| 485 |
+
- **Stage** β one of the 13 `hk_search_job` `gen_statem` states; the
|
| 486 |
+
pipeline's current stage and the job's current state are the same
|
| 487 |
+
value by construction.
|