| # F4 — Memory / Embedder |
|
|
| > Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md) |
| > Status: **planning** (M0+M1+M2 done — 2026-05-22) |
| > Owner: TBD |
| > Reference: `/shared/dev/claude-hooks` (the embedder lifecycle is |
| > mirrored from claude-hooks' Python implementation) |
|
|
| ## Problem |
|
|
| opencoti needs persistent memory on day-one — well before F3 |
| (`opencoti-server`, the Go companion daemon) is ready. Memory |
| needs an embedder, and the embedder needs to work without |
| requiring the user to install a separate runtime. The same |
| constraint that drove F2 (bundle llamafile so opencoti runs out |
| of the box) drives F4: ship a prebuilt embedder llamafile, manage |
| it with a small TypeScript launcher, default to CPU, allow |
| opt-in GPU. |
|
|
| There's a second, sharper problem the user surfaced from |
| real-world claude-hooks operation: the canonical embedder |
| (`qwen3-embedding-0.6b-16k.llamafile`, ~1.5 GB on disk for a |
| 700 MB model) allocates **3.32 GB RSS** at idle. The bottleneck |
| is *per-slot context buffer* allocation, not the autoregressive |
| KV cache. The embedder server runs `--parallel 2 --ctx-size |
| 16384` and pre-allocates the full 16K context for both slots up |
| front. Live workload shows most requests at 7–56 tokens with |
| only 1 slot active most of the time — so the eager allocation |
| pays full price for capacity that is almost never used. |
|
|
| F4 ships the embedder, *and* ships a lazy-slot-context patch |
| against the vendored llamafile/llama.cpp tree so the embedder's |
| RSS scales with actual demand instead of `slot_count × n_ctx_slot`. |
|
|
| ## Goals |
|
|
| - **G1.** TypeScript package `@opencoti/embedder` that owns the |
| embedder lifecycle: spawn, health-probe, idle-reap, GPU/CPU |
| mode, `embed(text)` / `embedBatch(texts)`. |
| - **G2.** First-run setup downloads the prebuilt asset |
| (`qwen3-embedding-0.6b-16k.llamafile`, dim 1024, default port |
| **47092** in opencoti's reserved 47000-48000 range) into the |
| opencoti data dir. Resume + SHA256 verify. |
| - **G2a.** **Daemon-launch policy** (project-wide convention). |
| Any opencoti or opencoti-server daemon that opens a listening |
| port MUST: (1) pick its default port from the **47000-48000 |
| range** — opencoti must never collide with claude-hooks' |
| 38000-39000 / 18790-18811 ranges so the two systems coexist |
| on one host; (2) ask the user at first-run setup where to |
| bind — `127.0.0.1` (default), all interfaces, or a specific |
| IP — and (3) propose the default port but allow the user to |
| override it before persisting the choice. The setup flow |
| validates that the chosen port is free and within range |
| before writing the config. |
| - **G3.** GPU mode = auto-detect with CPU fallback; default = CPU. |
| Reuses F1 M5.7's `gpu-detect` chain (nvidia → amd → vulkan). |
| - **G4.** Composite fallback chain: an `EmbedderClient` |
| interface + a `CompositeEmbedder` that tries primary then |
| fallback on `EmbedderError`, with a dim-match assertion to |
| prevent vector-space corruption. |
| - **G5.** Lazy-slot-context patch (`0001-lazy-slot-context.patch`) |
| against the vendored llamafile so RSS scales with real demand: |
| defer per-slot allocation, grow on demand, **shrink on idle**. |
|
|
| ## Non-goals |
|
|
| - Replacing claude-hooks for users who already run it on the |
| same host. opencoti's embedder is independent and uses its own |
| port + data dir. |
| - Bundling weights. The model GGUF is inside the .llamafile |
| asset; we don't ship a separate GGUF download. |
| - A full vector store + recall pipeline. F4 M5 wires recall + |
| store as a thin seam; the real vector-store ownership lives |
| in F3 (opencoti-server) when it ships. |
|
|
| ## Default asset |
|
|
| - **Repo:** [`mann1x/claude-hooks`](https://github.com/mann1x/claude-hooks) |
| releases — **transitional pin, two-stage migration**: |
| - **Stage 1 (immediate)**: mirror the byte-identical asset to |
| a `mann1x/opencoti` release (proposed tag: |
| `embedder-v1.4.0`) and flip just the `repo` field in |
| `vendors/pin/embedder.txt`. Same SHA, no rebuild — |
| decouples opencoti's release stream from claude-hooks'. |
| - **Stage 2 (after F4 M3 lazy-context patch ships)**: rebuild |
| the embedder composite-llamafile to include the lazy- |
| slot-context patch. The new binary has a different SHA; |
| `tag` + `sha256` in the pin file both bump (proposed tag: |
| `embedder-v1.5.0`). F4 M6 (release bundling) owns the |
| apply-patches → make → zipalign-with-GGUF-and-args |
| pipeline. Without Stage 2, the embedder still benefits |
| from the lazy-context patch only when run against a |
| user-supplied locally-built embedder binary — not the |
| prebuilt asset. |
| - **Tag:** `v1.4.0` |
| - **Asset:** `qwen3-embedding-0.6b-16k.llamafile` |
| - **SHA256:** `414f616689aaba44f6982b474918e8549ad764c03b2f82f6b56a5d6e582ef59b` |
| - **Pinned in:** [`vendors/pin/embedder.txt`](../../vendors/pin/embedder.txt) |
| - **Default port:** **47092** (opencoti's reserved 47000-48000 |
| range — see G2a). |
| - **Endpoint:** `POST /embedding` (llama.cpp server style, not |
| OpenAI-compat). Request: `{"content": "<text>"}` or batch |
| `{"content": ["t1", "t2"]}`. Response: `{"embedding": [...]}` |
| (single) / `[{"embedding": [...]}, ...]` (batch). |
| |
| ## Design sketch |
|
|
| ### Layout |
|
|
| ``` |
| packages/opencoti-embedder/ # NEW |
| ├── package.json |
| ├── tsconfig.json |
| ├── src/ |
| │ ├── index.ts |
| │ ├── client.ts # EmbedderClient interface + types |
| │ ├── llamafile-embedder.ts # POST /embedding HTTP client |
| │ ├── composite.ts # primary → fallback chain |
| │ ├── manager.ts # process spawn / health / idle reap |
| │ ├── gpu-mode.ts # CPU | auto resolution |
| │ ├── download.ts # asset fetch + SHA256 verify |
| │ └── config.ts # Zod schema for embedder config |
| ├── script/ |
| │ └── smoke.ts |
| └── test/ |
| |
| vendors/ |
| ├── pin/ |
| │ └── embedder.txt # NEW (M0) |
| └── patches/ |
| └── llamafile/ |
| └── 0001-lazy-slot-context.patch # NEW (M3) |
| ``` |
|
|
| ### Reused, not rebuilt |
|
|
| - `@opencoti/tiers/registry/gpu-detect` — nvidia/amd/vulkan probe |
| chain (F1 M5.7). |
| - `@opencoti/tiers/registry/hf-download` — resume + SHA256 verify |
| download. To be generalized into a shared |
| `@opencoti/tiers/util/download` helper consumed by both HF |
| downloads and GitHub Release downloads. |
|
|
| ## Milestones |
|
|
| ### M0 — Vendoring decision + asset URL pin *(done — 2026-05-22)* |
|
|
| - `vendors/pin/embedder.txt` records repo / tag / asset / SHA256. |
| - Decision recorded: pin claude-hooks v1.4.0 directly until |
| opencoti has its own release stream. Switch the pin URL to |
| `mann1x/opencoti` when that exists. |
| - This feature plan + F5 plan + MASTER_PLAN.md updated together |
| in the same commit. |
| |
| ### M1 — Embedder package, CPU mode, smoke green *(done — 2026-05-22)* |
| |
| - `@opencoti/embedder` shipped with client interface, |
| llamafile-embedder HTTP impl (POST /embedding, response-shape |
| tolerant for A/B/C/D variants seen in the wild), |
| download.ts (resume + SHA verify, builds the canonical |
| github-release URL from the pin), manager.ts (spawn + |
| `/health` poll + LRU idle reap @ 5 min default + idempotent |
| stop), config.ts (Zod strict schema, defaults claude-hooks- |
| parity), gpu-mode.ts (CPU default + auto-resolution), |
| composite.ts (primary→fallback chain with dim-match guard). |
| - Shared `downloadWithResume` helper extracted to |
| `@opencoti/tiers/util/download`; `hf-download.ts` refactored |
| to consume it. New subpath exports: |
| `@opencoti/tiers/util/download`, |
| `@opencoti/tiers/registry/gpu-detect`, |
| `@opencoti/tiers/registry/hf-download`. |
| - `script/smoke.ts` downloads the asset to `<repo>/.opencoti/embedder/` |
| (skips on SHA match), spawns CPU-mode, embeds "hello world", |
| asserts dim=1024 + L2-norm. Live smoke run is opt-in (asset |
| is 1.5 GB). |
| - 52 unit tests across 7 files. Hook footprint unchanged |
| (still 4 `opencoti-hook:` markers). |
| - **Out of scope (deferred to M5)**: opencode session |
| injection, recall-into-prompt, vector store. |
| |
| ### M2 — Baseline measurement *(done — 2026-05-22)* |
| |
| Ships two scripts and three fixture files: |
| |
| - `script/generate-corpus.ts` — deterministic 48-prompt corpus |
| generator spanning 8-3000 approx_tokens. Buckets: 10 short |
| (8-32 tok), 11 medium (33-128), 19 medium-long (129-512), 6 |
| long (513-2048), 2 very-long (2049-8192). Original draft had |
| 5 in the very-long bucket; 2 mega prompts (60+ LOREM repeats) |
| aborted past the 60s embedder request-timeout during the live |
| baseline run, so they were dropped from the generator. |
| - `script/measure-baseline.ts` — boots the embedder via the M1 |
| manager (or attaches to a running instance with `--use-running |
| <url> --pid <int>`), captures RSS from `/proc/<pid>/status`, |
| extracts allocation lines from the embedder startup log, |
| embeds the corpus, writes the fixture. |
|
|
| Fixtures (in `test/fixtures/`): |
| - `quality-prompts.json` — 48 prompts, deterministic. |
| - `quality-baseline.json` — 48 vectors (dim 1024) M3 must |
| reproduce to cosine ≥ 0.999. |
| - `baseline-meta.json` — RSS + allocation breakdown + |
| startup-log excerpt. |
|
|
| **Live measurement (2026-05-22, attached to claude-hooks |
| embedder, parallel=4 default, ctx=16384, CPU mode)**: |
|
|
| | Component | Size | |
| | --- | --- | |
| | Model weights (CPU_Mapped) | 603.87 MiB | |
| | KV cache (all slots) | **1792.00 MiB** | |
| | Compute buffer | 330.24 MiB | |
| | Output buffer | 2.33 MiB | |
| | RSS idle (process VmRSS) | **2.52 GB** | |
| | RSS peak (after 48-prompt run) | **3.81 GB** | |
| | Avg latency per prompt | 5352 ms | |
| | Total corpus embed time | 256.9 s | |
| |
| Startup-log key lines (captured in baseline-meta.json): |
| `llama_context: n_ctx = 16384`, |
| `llama_kv_cache: CPU KV buffer size = 1792.00 MiB`, |
| `sched_reserve: CPU compute buffer size = 330.24 MiB`, |
| `server_main: embeddings enabled with n_batch (2048) > |
| n_ubatch (512) → setting n_batch = n_ubatch = 512 to avoid |
| assertion failure`, `slot load_model: id 0/1/2/3 | new slot, |
| n_ctx = 16384`. The binary defaults to parallel=4 with no |
| `--parallel` override. |
| |
| The earlier user observation of **3.32 GB RSS** came from a |
| 2-slot config explicitly setting `--parallel 2 --ctx-size |
| 16384`. The M3 patch targets that production-shape config: |
| |
| - Pre-patch per-slot KV allocation = `n_ctx_slot × 2 (K+V) × |
| layers × per-token-dim` ≈ 448 MiB / slot in CPU mode, |
| allocated eagerly for every slot at startup. |
| - M3 target: per-slot allocation deferred to first prompt, |
| sized to `--slot-initial-ctx 4096` (~112 MiB), grown on |
| demand up to 16384, shrunk back to 4096 after |
| `--slot-shrink-idle-ms`. Combined with 2 slots + model + |
| compute, idle RSS target lands near 1.0 GB. |
| |
| **Two corpus prompts intentionally dropped**: `lorem-mega-00` |
| and `lorem-mega-01` (60+ LOREM repeats) aborted past the |
| embedder's 60s default request-timeout on the CPU-mode |
| production instance — n_ubatch is forced to 512 in embedding |
| mode, so 9000+ token requests need ~18 forward passes that |
| exceed 60s. Not an M3 concern (the patch addresses |
| *allocation*, not *throughput*). |
|
|
| ### M3 — Lazy-slot-context patch |
|
|
| > **Architecture reference:** [ADR 0001 — Lazy slot-context |
| > allocation](../decisions/0001-lazy-slot-context.md). The ADR |
| > documents the llama.cpp KV-cache layout, the three+1 orthogonal |
| > axes of "lazy," the four-phase rollout (defer / grow / shrink / |
| > per-stream split), the CLI surface, and the F5 milestone |
| > hand-offs that depend on the allocator hooks landed here. |
| > Implementation is multi-session per user directive B2. |
| > |
| > **Phase status (2026-05-23):** |
| > - **Phase 1 — deferred zero-fill: shipped** as |
| > `vendors/patches/llamafile/0001-lazy-slot-context-defer.patch`. |
| > Patch applies cleanly; build succeeds on x86_64 + aarch64; |
| > smoke validation on Qwen2-Math-1.5B confirms the |
| > `KV cache zero-fill deferred until first batch` startup log |
| > line and the matching `ensure_cleared: ... (deferred)` |
| > first-batch log line. Bench against M2 corpus + RSS table |
| > for the embedder workload pending (Phase 1 alone is a |
| > cold-start win; the headline warm-state RSS savings come |
| > in Phases 2-3). |
| > - **Phase 2 — grow on demand + partial zero-fill: shipped** |
| > as `vendors/patches/llamafile/0002-lazy-slot-context-grow.patch`. |
| > Adds `--slot-initial-ctx N` (default 0 = Phase 1 behavior) that |
| > caps initial KV-cell allocation per sequence; find_slot honors |
| > the soft cap; prepare() grows on overflow up to kv_size_max via |
| > next-pow2 geometric ramp; ensure_cleared partial-memsets only |
| > the [n_cells_cleared, target) range via `ggml_backend_tensor_memset`. |
| > With `--slot-initial-ctx 4096` for the production embedder, |
| > warm-state RSS scales with actual workload instead of the |
| > `slot_count × n_ctx_slot` product. Bench numbers vs M2 corpus |
| > pending. |
| > - **Phase 3 — shrink on idle: shipped** as |
| > `vendors/patches/llamafile/0005-lazy-slot-context-shrink.patch`. |
| > Adds `--slot-shrink-idle-ms N` (default 30000); per-stream |
| > idle timer sweeps at `init_batch` and reclaims buffers back |
| > to `--slot-initial-ctx` after the timeout. Bench delta: |
| > **391.7 MB returned to kernel** on the CPU embedder after 32 s |
| > idle (1.68 GB peak → 1.30 GB post-shrink, cosine 0.999797 vs |
| > M2 baseline). |
| > **CPU-only — the shrink is gated on |
| > `ggml_backend_buffer_is_host()` inside |
| > `opencoti_decommit_layer_range()` because |
| > `madvise(MADV_DONTNEED)` is a no-op against device-backed |
| > pages.** On GPU (`-ngl > 0`) the `shrink_if_idle` log marker |
| > still fires and the soft-cap bookkeeping rewinds, but the |
| > CUDA/ROCm/Vulkan backend buffer is not freed and VRAM stays |
| > at the peak the workload reached. The GPU-shrink path is |
| > tracked at **F5 M2 (HeadInfer)**, not Phase 4. |
| > - **Phase 4 — per-stream tensor split: DEFERRED** (task #109, |
| > target F5 M2). 2026-05-23 user decision after planning-session |
| > exploration showed Phase 4 is a deep refactor (per-stream |
| > tensors + per-stream backend buffers + reworked |
| > `cpy_k`/`cpy_v` scatter ops that currently rely on |
| > `ggml_reshape_2d(k, n_embd, kv_size*n_stream)` to flatten all |
| > streams into a single `ggml_set_rows`) and only adds value in |
| > non-unified mode. The embedder ships unified by default and |
| > already gets all Phase 1-3 wins with the contiguous tensor |
| > intact. Phase 4's real beneficiary is F5 (HeadInfer's per-head |
| > GPU/CPU split + PolyKV's shared pool). Full implementation |
| > map persisted in |
| > [docs/decisions/0001-lazy-slot-context.md §Phase 4 — implementation map](../decisions/0001-lazy-slot-context.md) |
| > for the F5 owner to pick up cold. **Not the GPU-shrink fix — |
| > that's F5 M2 (HeadInfer), independently of Phase 4.** |
|
|
| - `vendors/patches/llamafile/0001-lazy-slot-context.patch`. |
| - Behavior: |
| - Defer per-slot KV/compute-buffer allocation until slot's |
| first real prompt. |
| - Initial allocation rounded up to `--slot-initial-ctx` |
| (default 4096). |
| - Grow on demand (next power-of-two), cap at `n_ctx_slot`. |
| - **Shrink on idle** (`--slot-shrink-idle-ms`, default |
| 30000): free buffers and return to initial-ctx after the |
| timeout, otherwise the first 16K request permanently sticks |
| the slot at 16K and the lazy-allocation win evaporates. |
| - Bench (`perf/llamafile/embedder-rss.bench.ts`): RSS at idle / |
| after 8-token request / after 16K request / after idle |
| timeout + 8-token request. Quality: cosine ≥ 0.999 vs M2's |
| baseline. |
| - Header: `Milestone: F4 M3`, `Upstreaming: TBD — propose to |
| llamafile + llama.cpp upstream once bench numbers are public`. |
| |
| ### M4 — Auto-GPU mode + composite fallback |
|
|
| - `gpu-mode.ts` adds `"auto"`: probe nvidia → amd → vulkan; if |
| VRAM ≥ 1.6 GB, launch with `--gpu auto` + n-gpu-layers fully |
| offloaded; otherwise silent CPU fallback. |
| - `CompositeEmbedder` ships: primary (local llamafile) + |
| fallback (cloud-route delegate, when Tier 1 is configured). |
| Dim-mismatch assertion lives here. |
|
|
| ### M5 — Opencode integration: recall + storage seam *(2026-05-23 — shipped, rescoped mid-milestone)* |
|
|
| **Original scope:** one synthetic tools pair + one surgical session-end hook + |
| setup-flow listen-address question. **Actual scope after the user rescope |
| (verbatim): "memory store and recall is going to be another crucial point, |
| we support sqlite + sqlite_vec as base memory backend, managed by |
| opencoti-server. opencoti-server will manage optional sharing on the |
| network. the pgvector db is an add on memory connector. both works in |
| parallel. memory in opencoti can be managed per-session. user can manage |
| which memories to use and list, delete, connect to the memories via the |
| harness menu. shared memory for all sessions, specific memories can be |
| added and or connected. for each session the user can decide which one |
| to read/write/rw"** |
| |
| Path taken (after user decision): **in-process M5 now, refactor to |
| opencoti-server later**. The opencoti-server piece is queued for F3 |
| once F4 closes. The data model and ACL contract land here so they |
| survive that refactor unchanged. |
| |
| What M5 actually shipped: |
| |
| - **M5-A — `@opencoti/memory` package.** `SqliteVecStore` over |
| bun:sqlite + sqlite-vec 0.1.9 vec0 virtual table. Embedding dim |
| locked at DB creation (default 1024). Schema: `collections`, |
| `memories` (with `content_hash` UNIQUE for idempotent insert), |
| `memory_acl` (per-session override), `vec_memories` (vec0 |
| virtual table with FLOAT[dim]). Default DB path |
| `~/.opencoti/memory/state.db`. ACL model: explicit overrides win; |
| defaults are "rw" for global collections, "rw" for the owning |
| session of a session-private collection, "none" otherwise. The |
| resolver lives in pure code (`resolveAccessMode`, `canRead`, |
| `canWrite`) and is exhaustively tested. |
| - **M5-B — synthetic tools.** `__memory_recall(query, k?, collections?)`, |
| `__memory_store(content, collection?)`, `__memory_list()`. All |
| three are ACL-aware (sessionID is the actor) and best-effort — |
| embedder failure returns `{ok: false, error: "embedder_unavailable"}` |
| rather than throwing. `ensureSessionCollection` is idempotent. |
| - **M5-C — runtime wire-up via dynamic-import bridge.** |
| `@opencoti/tiers/src/memory-bridge.ts` resolves |
| `@opencoti/memory` + `@opencoti/embedder` via dynamic import to |
| break the tiers→memory→embedder→tiers cycle. Wired into |
| `runtime.ts:maybeRoute` via `mergeTools` (no new surgical |
| hook — extends an existing additive seam). The default |
| embedder is `createDefaultEmbedFn` (loopback LlamafileEmbedder |
| on 47092). Workspace dep `@opencoti/memory` added on |
| `packages/opencode` to make the dynamic import resolvable; |
| registered in UPSTREAM_SYNC.md as `memory-tools-resolvability` |
| (dep). |
| - **M5-D2 — `@opencoti/memory-plugin` (opencode SERVER plugin).** |
| Two hooks: `experimental.chat.system.transform` appends a |
| system-prompt suffix advertising the memory tool surface + |
| listing the session's accessible collections; |
| `event` listens for `session.idle` and, when |
| `auto_ingest: true` (default FALSE), ensures the session-private |
| collection. Defaults reflect the rescoped vision: advertise=true |
| (the point of loading the plugin), auto_ingest=false (the agent |
| already has `__memory_store`). |
| - **M5-D1 — `@opencoti/tui-memory` (opencode TUI plugin).** One |
| command `opencoti.memory` opens a DialogSelect listing every |
| collection (global + session-private), with per-session |
| resolved-mode annotations when in a session route. Drill-down |
| per collection offers: ACL toggle r/w/rw/none (session-only), |
| DialogConfirm-guarded delete, Create flow |
| (DialogPrompt name → scope DialogSelect). Lazy store open, |
| closed cleanly on `api.lifecycle.onDispose`. |
| |
| **Default-plugin auto-wiring (2026-05-23 follow-up to user |
| directive: "opencoti will need to wire them by default"):** |
| The opencoti plugins remain opt-in for upstream-opencode users |
| (they stay out of upstream's plugin list unless explicitly |
| declared), but opencoti's distribution auto-wires them whenever a |
| project has any `opencoti.*` config section. Implementation in |
| `@opencoti/tiers/default-plugins`: a pure helper |
| `applyDefaultPlugins(cfg)` gap-fills `@opencoti/memory-plugin`, |
| `@opencoti/tui-memory`, `@opencoti/tui-tiers` into the final |
| `plugin_origins` list — user-declared entries (including their |
| options) are preserved untouched. Wired via two surgical hooks |
| in `packages/opencode/src/config/config.ts` (import + call-site, |
| registered as `opencoti-default-plugins`). Net change vs M5 |
| substance: TWO new opencoti-hook markers (still tagged + in |
| registry). With opencoti disabled (no `cfg.opencoti` block) the |
| helper is a no-op and the byte-output is identical to upstream. |
| |
| What M5 deliberately did NOT ship: |
| |
| - The session-end **surgical** hook (the original plan). Replaced by |
| the optional `event: "session.idle"` listener in `@opencoti/memory-plugin` |
| with `auto_ingest: false` default. Net new `opencoti-hook:` |
| markers from M5 substance + follow-up: TWO (the |
| `opencoti-default-plugins` import + call-site). The |
| `memory-tools-resolvability` dep is a registry-only entry (no |
| source marker — JSON has no comments). |
| - The pgvector add-on connector. Deferred — the in-process backend |
| is sqlite-vec only for now. Pgvector becomes a parallel |
| `MemoryStore` implementation once opencoti-server (F3) lands. |
| - The network-sharing layer. That's opencoti-server's job (F3). |
| - The "Clear ACL override" UI option in the TUI panel. The |
| `MemoryStore` interface does NOT currently expose a clear-ACL |
| method — adding it would require a new schema/interface revision |
| (deferred to a future minor bump). |
| - **M5-E — setup-flow embedder daemon listen address question |
| (shipped 2026-05-23).** `@opencoti/tiers/cli/setup-flow` |
| extended with an embedder listen-address prompt in |
| `finalizePatch`. Steps: (1) DialogSelect over |
| loopback / all-interfaces / specific-IP / skip; (2) if |
| specific-IP, prompt for the IP with an IPv4 validator; |
| (3) text prompt for the port, validated against the opencoti |
| 47000-48000 range; (4) port-free check via injected |
| `isPortAvailable(host, port)` (live: TCP bind probe in |
| setup.ts) — busy port logs a warn and asks the user whether to |
| proceed anyway; "no" recursively re-asks the bind block. The |
| chosen `{host, port}` lands in |
| `~/.config/opencode/opencode.jsonc` under |
| `opencoti.embedder.host` + `.port` via the jsonc-parser |
| modify-then-applyEdits path (comments preserved). A new |
| `embedderConfigured` sentinel (true when BOTH host AND port |
| already present in the user's config) gates the prompt on |
| re-runs. 16 new tests across the flow + the `validatePort` and |
| `isValidIPv4` pure helpers. |
| |
| ### M6 — Production llamafile build pipeline *(2026-05-23 — promoted ahead of M5)* |
| |
| Driver: user directive 2026-05-23 — the post-F4 M4 state (prebuilt |
| claude-hooks asset + manual zipalign of ggml-cuda.so + cross-repo |
| pin dependency) is "patchwork […] not acceptable for production". |
| M6 was originally scoped to the embedder composite alone; the |
| expanded M6 below covers all three backends (CUDA / ROCm / Vulkan) |
| and the local-asset modes the embedder downloader needs to |
| consume an opencoti-built artifact. |
| |
| M6 lands before M5 (opencode session-end hook) because there's no |
| point storing memories on the back of a launcher binary that |
| doesn't engage the M3 lazy-context patch. |
| |
| **What M6 ships:** |
| |
| 1. **Multi-backend DSO builders in |
| `packages/opencoti-llamafile/script/build-pipeline.ts`:** |
| `runCuda` / `runRocm` / `runVulkan` / `runAllBackends`, |
| driven by a shared `buildBackend(spec)` helper. |
| - Each backend has a toolchain probe: `OPENCOTI_CUDA_PATH/bin/nvcc` |
| (default `/usr/local/cuda-12.6` — CUDA 13.x is known-broken |
| with `cuda.sh`), `OPENCOTI_ROCM_PATH/bin/hipcc` (default |
| `/opt/rocm`), `OPENCOTI_VULKAN_SDK` env / system `glslc`. |
| - Soft-fail per backend: a host without ROCm produces a clean |
| CUDA-only build instead of failing the pipeline. |
| - Each built DSO is staged at |
| `vendors/dist/llamafile/<backend>/<ver>/ggml-<backend>.so` |
| AND mirrored to `~/.llamafile/v/<ver>/` so devs running the |
| thin binary directly still get GPU support locally. |
| |
| 2. **`package` subcommand in the same script:** |
| Zipaligns the thin patched binary with every staged DSO via |
| the vendored `vendors/sources/llamafile/o/third_party/zipalign/ |
| zipalign -j0` invocation. Always emits |
| `dist/llamafile/opencoti-llamafile-<ver>-<arch>.llamafile` |
| plus a sibling `MANIFEST.json` (artifact SHA + per-DSO SHA + |
| patches list). |
| - With `--with-model PATH` ALSO emits |
| `dist/llamafile/opencoti-embedder-<ver>-<arch>.llamafile` |
| by copying the bare and zipalign-embedding the GGUF + a |
| `.args` file generated from the F2 M3 canonical embedder |
| args contract (`-m /zip/<basename> --server --embedding |
| --pooling last --ctx-size 16384 --parallel 2 |
| --slot-initial-ctx 4096 --slot-shrink-idle-ms 30000`). |
| - `--with-args PATH` overrides the generated `.args`. |
| - `--cpu-only` produces a binary with NO DSO embedded (for CI |
| smoke on hosts without GPU). |
| |
| 3. **Pure helpers in |
| `packages/opencoti-llamafile/src/build-pipeline.ts`** (kept |
| side-effect-free per F2 M3 convention): `Backend` type + |
| `ALL_BACKENDS`, `dsoFilename`, `stagingDir`, `artifactDir`, |
| `readBackendArtifacts`, `sha256OfFile`, `composeEmbedderArgs`, |
| `parseBuildArgv`, `ArtifactManifest` type. 19 new unit tests |
| under `test/build-pipeline.test.ts`. |
| |
| 4. **`@opencoti/embedder` downloader local-asset modes |
| (`src/download.ts`):** |
| - `file:` prefix on the pin's repo field — bytes come from |
| disk; SHA still verified. |
| - `OPENCOTI_EMBEDDER_LOCAL=<abs-path>` env override — |
| bypasses pin path entirely; SHA still verified. |
| - Both routed through a shared `localCopyPath` helper; the |
| HTTPS+resume path is unchanged. 12 new tests in |
| `test/download.test.ts`. |
| |
| 5. **Hook footprint unchanged.** M6 is pure build orchestration + |
| downloader extension; no surgical hooks in upstream opencode. |
| |
| **Deferred to F4 M7 (release bundling):** |
| |
| - GitHub Releases publish pipeline. Producing artifacts under |
| `dist/llamafile/` is M6; uploading them as `mann1x/opencoti` |
| release assets and flipping `vendors/pin/embedder.txt` to |
| `mann1x/opencoti` (Stage 2 of the migration the existing pin |
| file header describes) is M7. The pin migration is a one-line |
| change once the release exists. |
| - Metal / macOS backend. `vendors/sources/llamafile/llamafile/ |
| metal.c` is macOS-only; M6 ships Linux-only since the build |
| host is solidPC. Needs a Darwin CI runner. |
| - aarch64 / multi-arch DSO orchestration. cosmocc APE is dual- |
| arch by design, but ggml-*.so are arch-specific. The `<arch>` |
| token in the output filename is anticipatory; M6 ships x86_64 |
| only. |
| |
| ### M7 — Release bundling + dev-cut workflow alignment with claude-hooks |
| |
| The build pipeline (M6) produces `dist/llamafile/*.llamafile` artifacts. |
| M7 wraps that with the **commit-the-contract, gitignore-the-bytes** |
| release scheme claude-hooks uses, plus a `release-cut` subcommand |
| that prepares (but does NOT auto-publish) a `mann1x/opencoti` |
| GitHub Release. |
| |
| **What's tracked (committed):** |
| |
| - `vendors/llamafile/README.md` — audit-trail doc describing the |
| pinned upstream + the contract layout. Mirrors claude-hooks's |
| `vendor/llamafile/README.md`. |
| - `vendors/llamafile/LICENSE.upstream` — single-file copy of the |
| four upstream LICENSE files (Mozilla-Ocho/llamafile Apache-2.0 + |
| llama.cpp/whisper.cpp/stable-diffusion.cpp MIT × 3) at the |
| pinned commit. Survives independently of the submodule. |
| - `vendors/llamafile/SHA256SUMS.composite` — the contract. |
| `sha256sum`-compatible lines listing every released composite |
| artifact. Refreshed by `build:llamafile:release-cut`. Verified |
| by the embedder downloader at install time. |
| |
| **What's gitignored (reproducible from source):** |
| |
| - `vendors/dist/llamafile/<backend>/<ver>/ggml-*.so` — per-backend |
| staging DSOs. |
| - `dist/llamafile/*.llamafile` — final composite artifacts. |
| - `dist/llamafile/*.MANIFEST.json` — per-artifact manifests with |
| embedded-backend metadata. |
| |
| **Release-cut workflow** (matches claude-hooks's |
| `make -C vendor/llamafile/dist`): |
| |
| 1. `bun run build:llamafile` — patched binary. |
| 2. `bun run build:llamafile:all-backends` — every available DSO. |
| 3. `bun run build:llamafile:package -- --with-model PATH` — bare |
| + composite artifacts. |
| 4. `bun run build:llamafile:release-cut` — refreshes the SHA |
| contract; prints the suggested `gh release create` invocation |
| (the actual publish is a user-confirmation step, not |
| automated). |
| 5. User executes `gh release create` + `gh release upload`, then |
| commits the refreshed `SHA256SUMS.composite` + the bumped |
| `vendors/pin/embedder.txt`. |
| |
| **Downloader behavior:** when `shaContractText` is provided to |
| `downloadEmbedder`, the pin's SHA is cross-checked against the |
| contract before any download or local-copy. Disagreement is a |
| hard error with a "redownload or rebuild" breadcrumb (matches |
| claude-hooks's `install.py`). An asset not present in the |
| contract is a soft warning, not an error — supports the |
| transitional state where the pin still references claude-hooks |
| pre-flip. |
| |
| **What's NOT in M7:** |
| |
| - Actual first `mann1x/opencoti` GitHub Release publish — that's |
| the user-confirmed external step the scaffolding enables. |
| - Pin flip from `mann1x/claude-hooks` → `mann1x/opencoti` — only |
| meaningful after the first release exists. |
| - macOS / aarch64 — same constraints as M6. |
| |
| **Critical files (M6):** |
| |
| - `packages/opencoti-llamafile/script/build-pipeline.ts` — new |
| subcommands. |
| - `packages/opencoti-llamafile/src/build-pipeline.ts` — pure |
| helpers + types. |
| - `packages/opencoti-llamafile/test/build-pipeline.test.ts` — |
| helper tests (19 new, 46 total). |
| - `packages/opencoti-embedder/src/download.ts` — local-asset |
| modes. |
| - `packages/opencoti-embedder/test/download.test.ts` — local- |
| asset tests (12 new, 13 total for download). |
| - `docs/features/llamafile_build.md` — toolchain + pipeline |
| documentation. |
| - Root `package.json` — new `build:llamafile:{rocm,vulkan, |
| all-backends,package}` scripts. |
| - `.gitignore` — `vendors/dist/`, `/dist/llamafile/`. |
| |
| ## Surgical-hook footprint |
| |
| - M1–M4, M6: zero new hooks. |
| - M5: exactly one new hook (session-end memory ingest). |
| |
| ## Open questions |
| |
| - **Release stream ownership.** Decision (M0): pin claude-hooks |
| v1.4.0. When opencoti's own release cadence starts, switch |
| the pin URL to `mann1x/opencoti`. Same SHA + same asset on |
| disk; just a different `repo` field. |
| - **Composite-llamafile build.** F4 consumes the prebuilt |
| asset. If F4 M3's lazy-context patch lands and we want it in |
| the embedder binary too, we need to build our own |
| composite-llamafile (vendored llamafile + Qwen3 GGUF + |
| `.args`). That's potentially a small new milestone (F4 M6) |
| that overlaps with F2 M6 (release artifact bundling). |
| - **Idle-reap vs always-on for the manager.** 5-min idle reap |
| matches claude-hooks. For opencoti, sessions are typically |
| longer; revisit if cold-start latency turns out to be an |
| issue in production use. |
| - **Sqlite-vec dep choice.** The Bun-native binding vs |
| `better-sqlite3` + the `sqlite-vec` extension. Decide at M5 |
| based on which builds cleanly across Linux/macOS/Windows. |
| |
| ## Risks |
| |
| - **F4 M3 patch surface area.** Slot-init touches code that F5's |
| ReST-KV (M1) and HeadInfer (M2) also modify. Ordering: F4 M3 |
| lands first as patch `0001-`; F5's start at `0010-`. On a |
| later upstream pin bump, M3 needs to rebase first. |
| - **Asset hosting.** While we pin claude-hooks' release, the |
| embedder's lifecycle is tied to claude-hooks' tagging |
| cadence. Mitigation: mirror to opencoti's own release once |
| it's set up. |
| - **`@opencoti/tiers` ↔ `@opencoti/embedder` boundary.** The |
| embedder depends on tiers (for GPU detect + download helper), |
| and tiers dynamic-imports `@opencoti/llamafile`. No cycle |
| introduced (embedder is the new edge of the DAG), but confirm |
| with `bun --cwd packages/opencoti-embedder typecheck` after |
| the shared-download refactor. |
| |
| ## M8 — opencoti-server owns the embedder process (daemon-side lifecycle) |
| |
| > **Status: in progress (2026-07-18).** Local task **#815** / STATE_SUMMARY |
| > "OPEN — Distribution & provisioning rebuild". Opened after the 3-way |
| > distribution audit found the Go daemon's embedder is **client-only**. |
| |
| ### Why |
| |
| M0–M7 built the embedder lifecycle (spawn / `/health` / idle-reap / |
| GPU-auto+CPU-fallback / download) in the **TypeScript** `@opencoti/embedder` |
| package — and even there it is reachable only from scripts; the opencode runtime |
| is itself a bare HTTP client to `:47092` (`default-embed-fn.ts`). The **Go daemon |
| `opencoti-server`** (SP-M1 #768) only ever gained an HTTP *client* |
| (`internal/embed/embed.go`): an empty `--embedder-url` yields a **disabled** |
| embedder with **no fallback and no process management**. That is the gap: the |
| daemon must own the embedder the way `claude-hooks-daemon` does |
| (`embedding_manager.py`) — demand-spawn the bundled `qwen3-embedding-0.6b` |
| llamafile, `/health`-gate, idle-reap + demand-respawn, GPU-auto/CPU-fallback, |
| download-if-missing, and **default to the bundled embedder when no external |
| endpoint is configured**. |
| |
| ### Architecture — reuse the supervisor module, add an embedder controller |
| |
| The `opencoti/llamafile-supervisor/` Go module's package doc **explicitly names |
| "opencoti-server in prod — a future consumer that imports this package and |
| registers its own controller"** (`supervisor/controller.go:1-12`). So: |
| |
| - **`internal/embedsvc` (new).** An `supervisor.EngineController` for the |
| embedder: `Start` spawns the composite `.llamafile` via `sh <bin> <args>` |
| (Cosmopolitan APE), `Stop` SIGTERMs + waits for exit, `HealthURL`/`UpstreamURL` |
| point at the child. **Args ported verbatim from `manager.ts buildEmbedderArgs`** |
| (`--server --embedding --host --port --parallel --ctx-size --pooling` + |
| GPU/CPU flags + lazy-slot flags). **GPU auto→CPU fallback** ported from |
| `gpu-mode.ts` (probe VRAM ≥ min → `--gpu auto -ngl 99`, else `--gpu disable`; |
| on a GPU spawn/health failure, flip a sticky CPU flag and relaunch — silent, |
| matching claude-hooks). |
| - **`supervisor.Supervisor`** provides, unchanged, the single-flight |
| `ensureRunning`, `waitReady` (`/health` 200), idle-reap + `StartReaper`, and the |
| VRAM precheck (reuse `supervisor.VRAMProber`). The daemon runs |
| `Supervisor.Handler()` on a **loopback listener** (a free port in 47000–48000) |
| and points its own `internal/embed` client at that port — so a |
| `/v1/memory/search` query embeds through the supervisor, which **demand-spawns |
| the child on the first request and reaps it on idle**. |
| - **Download/install** (`internal/embedsvc` or `internal/embed`): read |
| `vendors/pin/embedder.txt`, resolve a local path under `~/.opencoti/embedder/`, |
| and **download-if-missing** (resume + SHA256 verify) — porting `download.ts`. |
| Source order: `OPENCOTI_EMBEDDER_LOCAL` / `file:` override → the pin repo. The |
| pin now has a **public HF mirror** at |
| `ManniX-ITA/opencoti-llamafile → embedder/qwen3-embedding-0.6b-16k.llamafile` |
| (uploaded 2026-07-18) so the anonymous path works while `mann1x/opencoti` is |
| private. Download is **lazy** — only on the first embed that finds the binary |
| absent — so daemon boot (and the whole server test suite) stays byte-identical. |
| |
| ### Opt-in / default model |
| |
| - **Default-to-bundled is the point** (user directive): when **no |
| `--embedder-url`** is set and `--embedder-autostart` (default **true**) is on, |
| the daemon stands the bundled embedder up behind the supervisor and embeds |
| server-side. Everything is **lazy** (no spawn, no download at boot) → boot |
| byte-identical; the child only appears on the first real embed. |
| - **External endpoint wins:** a non-empty `--embedder-url` short-circuits |
| autostart (explicit URL → direct client, exactly as SP-M1 today). |
| - **Escape hatch:** `--embedder-autostart=false` + no URL = embedder disabled |
| (today's behaviour, for minimal/test deployments). |
| |
| ### Phases |
| |
| | Phase | Scope | State | |
| |-------|-------|-------| |
| | P0 | M8 plan (this) + `opencoti.server.embedder` config schema + cerebrum | in progress | |
| | P1 | `internal/embedsvc` spawn-child controller (args/gpu ported, spawn/stop/health) + go.mod require/replace the supervisor module + unit tests | **done** — P1a `args.go` (arg/GPU builder ported from `manager.ts`/`gpu-mode.ts`); P1b `controller.go` implements `supervisor.EngineController` (Start/Stop/HealthURL/UpstreamURL/Mode) with `sh <bin>` APE spawn, SIGTERM+wait stop, and **GPU-auto→CPU sticky fallback** on spawn/exit/health-timeout; injectable spawn/VRAM/health seams. 76% cov, `-race`+`vet` clean, `go build ./...` green | |
| | P2 | download/install + pin verify (port `download.ts`; `~/.opencoti/embedder/`; HF-mirror/`file:`/`LOCAL` sources) + tests | **done** — `download.go`: `EnsureBinary` (LocalPath override → cached → download), **HF-native** `DefaultPin`/`Pin.URL()` (claude-hooks GitHub release purged per 2026-07-22 directive), resume (`.part` + Range) + SHA256 verify + atomic rename, `~/.opencoti/embedder/`. 8 tests (pin-is-HF-only, local-override±SHA, cached, fresh-download, SHA-mismatch-cleans-part, resume-from-partial); 75% cov, `-race`+`vet` clean, build green | |
| | P3a | `embedsvc.Service` glue: Controller + `supervisor.Supervisor` + loopback front listener; `EmbeddingsURL()`/`Front()`/`Mode()`/`Close()`; lazy `WithBinResolver` (download deferred to first request). Integration test (real in-process child, demand-spawn+proxy through the front, lazy-resolve-once). 78% cov, `-race`+`vet`+build green | **done** | |
| | P3b | main.go wiring: `buildEmbedder` default-to-bundled path (build+Start the Service, embed URL → `Service.EmbeddingsURL()`, Close on shutdown), `--embedder-autostart`/`--embedder-bin`/`--embedder-gpu`/`--embedder-idle` flags + `opencoti.embedder.*` bridge, `embedder_ok`/`embedder_mode` healthz, Windows baking | **done** — `buildEmbedderStack` (external / bundled-lazy / disabled); `--embedder-autostart`(default true)/`--embedder-bin`/`--embedder-gpu`/`--embedder-idle`/`--embedder-port` flags + Windows install baking; `Options.EmbedderMode` + `embedder_ok`/`embedder_mode` healthz + test. **Live smoke (3 modes):** autostart-off→`disabled` (byte-identical), autostart-on→`bundled-stopped` (lazy, no spawn/download at boot), url-set→`external`. server suite green. TS `opencoti.embedder.*`→flag bridge deferred to the wizard (F1 M11). | |
| | P4 | integration tests (fake embedder child: demand-spawn, reap, GPU→CPU fallback, download SHA), byte-identical-when-disabled, full gate; docs + `.wolf` + memory | **done** — **native suites green** (`embedsvc`: args/controller/download/service tables incl. fake-child demand-spawn+proxy, lazy-resolve-once, GPU→CPU sticky fallback, HF-pin-only, SHA-mismatch-cleans-`.part`, resume-from-partial; `server`: `embedder_ok`/`embedder_mode` disabled/external/bundled). **Cross-compile clean** (windows/amd64, linux/arm64). **Live smoke on solidPC (GPU host, real HF download):** daemon boots `bundled-stopped` (lazy — no spawn/download at boot, byte-identical); first `POST /v1/embeddings` demand-downloads the pin from HF (1.5 GB, SHA `414f6166…2ef59b` verified == `DefaultPin`, `.part` gone, 1 517 042 662 B) → demand-spawns → **HTTP 200, 1024-dim vector**, healthz flips `bundled-stopped`→`bundled-cpu`. **Idle-reap + cached respawn:** after idle (90 s) the reaper flips healthz back to `bundled-stopped` (~40 s); a second embed respawns from the **cached** binary (no re-download, ~1 s) → HTTP 200, 1024-dim, `bundled-cpu`. | |
| |
| ### Constraints |
| |
| Additive: new Go in `opencoti/server/internal/embedsvc` + a cross-module import of |
| the already-present `opencoti/llamafile-supervisor` (require + local `replace`); |
| new `opencoti.server.embedder` config rides the existing |
| `opencoti-server-autostart` schema hook (no new marker). No upstream source edits. |
| Byte-identical when `--embedder-url` is set or `--embedder-autostart=false`. No |
| `meta.schema_version` bump. Go tests `-tags sqlite_fts5`. |
| |
