docs/architecture.org

#+TITLE: Nekomata Architecture
#+SUBTITLE: Five Layers of a Living Container Engine
#+AUTHOR: SnapKitty Systems
#+DATE: 2026
#+OPTIONS: toc:3 num:t
#+STARTUP: content

* Overview

Nekomata is a five-layer system. Each layer has a single responsibility
and a clean interface to the layer above and below it. The layers are
not microservices -- they are modules in a single Lisp image. The
interfaces are function calls, not HTTP requests.

#+BEGIN_EXAMPLE
  Layer 5: LLM -> ASM    (intent compilation)
      |
  Layer 4: Regex Math    (formal policy algebra)
      |
  Layer 3: MoE Router    (sparse expert activation)
      |
  Layer 2: Docker Socket (raw container operations)
      |
  Layer 1: Core Image    (Lisp image-based persistence)
#+END_EXAMPLE

Data flows down (intent -> bytes) and up (telemetry -> fitness signal).
Control flows down (intent -> execution). State is horizontal (every
layer can read the Lisp image).

* Layer 1: Core (Lisp Image-Based Persistence)

** What It Is

The Core layer is a persistent Common Lisp image. All Nekomata state
lives here: running container registry, evolutionary population, hot-swap
log, snapshot store, policy tables.

When Nekomata starts, it restores from the last saved image. When it
stops, it saves the image. The image is not a database dump -- it is
the entire live Lisp process state, including compiled functions,
hash tables, and all in-memory data.

** Why This Matters

Most container engines lose all in-memory state on restart. They
reconstruct state by replaying events from a persistent log (etcd in
Kubernetes). This requires:
- A separate storage system (etcd)
- A reconciliation loop (the control plane rebuilds desired state)
- A startup latency (the control plane must catch up)

Nekomata eliminates all three by persisting the Lisp image directly.
The image is the database. Restart is image restore. There is no
reconciliation loop because there is no gap between desired state and
actual state -- the image already contains the actual state of the
previous session.

** Key Files

| File | Responsibility |
|------|----------------|
| =core/package.lisp= | Package and export declarations |
| =core/sexpr-infra.lisp= | Container spec DSL and =define-container= macro |
| =core/eval-loop.lisp= | REPL and infrastructure eval loop |
| =core/image-persist.lisp= | Lisp image save/restore |
| =core/conditions.lisp= | Condition hierarchy |
| =core/nekomata.lisp= | Daemon entry point and container registry |

** Core Data Model

#+BEGIN_SRC lisp
(defstruct container-spec
  name
  image
  ports          ; list of (host-port . container-port)
  env            ; alist of (KEY . "value")
  volumes        ; list of (host-path . container-path)
  healthcheck    ; lambda or nil
  labels         ; alist
  restart-policy ; :always :on-failure :never
  command)       ; list of strings or nil
#+END_SRC

The =define-container= macro expands to a =defvar= that installs a
=container-spec= in the image and registers it in =*running-containers*=.

** Persistence Protocol

#+BEGIN_SRC lisp
;; Save
(save-nekod-image "/var/lib/nekod/nekod.image")

;; Restore
(restore-nekod-image "/var/lib/nekod/nekod.image")
#+END_SRC

Image saves happen: on clean shutdown, on SIGTERM, on explicit REPL
call, and after every evolutionary generation (configurable).

* Layer 2: Docker (Raw Socket, No Abstraction Layer)

** What It Is

The Docker layer communicates with the Docker daemon over a Unix domain
socket using raw HTTP/1.1 requests. There is no Docker SDK dependency.
There are no generated client bindings. There is a function that writes
bytes to a socket and a function that reads bytes from a socket.

** Why No Abstraction Layer

Every Docker SDK (Python dockerpy, Go docker/client, Java docker-java)
wraps the same HTTP API in a language-specific object model. This adds:
- Dependency chains (transitive dependencies on JSON parsers, HTTP
  clients, TLS libraries)
- Version coupling (the SDK version must match the Docker API version)
- Abstraction leakage (the SDK exposes Docker's internal concepts

  without the option to hide them)



Nekomata talks to Docker the same way curl does: raw TCP, HTTP/1.1

framing, JSON bodies assembled by =format=. This means zero dependencies

beyond a working Lisp and a running Docker daemon.



** Key Files



| File | Responsibility |

|------|----------------|

| =docker/unix-sock.lisp= | Raw Unix socket I/O |

| =docker/protocol.lisp= | HTTP/1.1 request/response framing |

| =docker/container.lisp= | Container lifecycle (create/start/stop/kill) |

| =docker/image.lisp= | Image pull and inspect |

| =docker/network.lisp= | Network create/connect/disconnect |

| =docker/volume.lisp= | Volume create/mount/remove |

| =docker/inspect.lisp= | Container state and stats polling |



** API Surface



#+BEGIN_SRC lisp

;; The full public interface of the Docker layer

(docker-get  "/containers/json")             ; list containers

(docker-post "/containers/create" body-json) ; create container

(start-container  id)                        ; POST /containers/{id}/start

(stop-container   id)                        ; POST /containers/{id}/stop

(kill-container   id)                        ; POST /containers/{id}/kill

(inspect-container id)                       ; GET  /containers/{id}/json

#+END_SRC



All functions return parsed JSON as Lisp association lists.



* Layer 3: MoE Router (Sparse Expert Activation)



** What It Is



The MoE (Mixture of Experts) router matches an infrastructure intent

to the minimal set of expert modules required to handle it. Experts are

Common Lisp functions registered in a routing table with capability

descriptors.



** Architecture



#+BEGIN_EXAMPLE

  Intent string

      |

  [Tokenizer]  -> token list with intent classes

      |

  [Router]     -> score each expert against token list

      |

  [Threshold]  -> keep experts with score > 0.5

      |

  [Dispatch]   -> call each activated expert with the intent AST

      |

  [Aggregate]  -> merge expert outputs into a single action

#+END_EXAMPLE



** Expert Registration



#+BEGIN_SRC lisp

(register-expert :placement

  :capabilities '(:schedule :place :deploy :run)
  :weight 1.0
  :handler #'placement-expert)

(register-expert :routing
  :capabilities '(:route :traffic :proxy :forward)

  :weight 0.9

  :handler #'routing-expert)
#+END_SRC

** Sparse Activation

Given N registered experts and intent I:
1. Compute score(e, I) for all e in E using token overlap + semantic
   similarity against capability keywords
2. Activate A = {e : score(e, I) > threshold}
3. |A| is typically 1-2 for most intents, never all of E

This is the same sparsity property as a sparse MoE language model.
Only the relevant experts burn compute for each token (intent).

** Key Files

| File | Responsibility |
|------|----------------|
| =moe/router.lisp= | Expert registry and routing algorithm |
| =moe/sparse-activate.lisp= | Scoring and threshold gating |
| =moe/experts/= | Directory of expert module implementations |

* Layer 4: Regex Math (Formal Pattern Algebra)

** What It Is

The regex-math layer implements a Kleene algebra over container
identity strings, port patterns, and operation codes. This algebra
is the formal policy language for Nekomata.

** Kleene Algebra Primitives

The algebra has five operations:
- =re-literal= :: match a literal string
- =re-concat= :: sequential composition (A then B)
- =re-union= :: disjunction (A or B)
- =re-star= :: Kleene star (A zero or more times)
- =re-not= :: complement (everything that is not A)

These compose to express any regular policy:

#+BEGIN_SRC lisp
;; Allow traffic from any api-* container to db-* on ports 5432-5435
(define-policy db-access
  :from (re-concat (re-literal "api-") (re-star re-any))
  :to   (re-concat (re-literal "db-")  (re-star re-any))
  :port (re-union (re-literal "5432") (re-literal "5433")
                  (re-literal "5434") (re-literal "5435")))
#+END_SRC

** Formal Properties

Because the policy language is a Kleene algebra:

1. *Equivalence checking*: Two policies P1 and P2 are equivalent iff
   their DFA representations accept the same language. Decidable in
   O(n^2 * m) where n, m are state counts.

2. *Subsumption*: P1 subsumes P2 iff L(P2) ⊆ L(P1). Check by
   computing L(P2) \ L(P1) and verifying it is empty.

3. *Conflict detection*: Policies P_allow and P_deny conflict iff
   L(P_allow) ∩ L(P_deny) ≠ ∅. Compute the intersection NFA and
   check emptiness.

** Key Files

| File | Responsibility |
|------|----------------|
| =regex-math/= | Kleene algebra engine, NFA/DFA, policy compiler |

* Layer 5: LLM to ASM (Intent Compilation)

** What It Is

The llm2asm layer is the compilation pipeline from natural language
intent to raw machine code bytes. It is the outermost interface of
Nekomata: the point at which human language enters the system and
machine instructions exit.

** Compilation Pipeline

#+BEGIN_EXAMPLE
  "deploy the API server"
      |
  [Tokenizer]      tokenize-intent
                   -> list of intent-token structs
      |
  [AST Extractor]  extract-ast
                   -> infra-ast with type :place, target "API"
      |
  [Constraint Builder] extract-constraints-from-tokens
                   -> list of constraint structs
      |
  [Code Generator] codegen
                   -> raw x86-64 or ARM64 bytes
      |
  [Verifier]       verify-bytes
                   -> check: ends with RET, valid prologue, no
                      orphan REX prefixes
      |
  [JIT Linker]     jit-link
                   -> function pointer in executable memory
#+END_EXAMPLE

** Intent Types

Five intent types map to five kernel primitives:

| Intent Type | Trigger Words | Kernel Primitive |
|-------------|---------------|------------------|
| =:place= | PLACE SCHEDULE DEPLOY RUN | ADD (scheduling arithmetic) |
| =:route= | ROUTE TRAFFIC PROXY FORWARD | XOR (packet classification) |
| =:scale= | SCALE REPLICATE EXPAND SHRINK | LOOP (replica iteration) |
| =:heal= | HEAL RESTART RECOVER FIX | MEMCPY (state snapshot) |
| =:secure= | SECURE ISOLATE LOCK PROTECT | MEMSET (zero-fill) |

** Backend Architecture

The codegen layer has two backends:

| Backend | File | Target |
|---------|------|--------|
| x86-64 | =llm2asm/backends/x86-64.lisp= | AMD64 / Intel 64-bit |
| ARM64  | =llm2asm/backends/arm64.lisp=  | Apple Silicon / Graviton |

Backends share the same AST input. The same intent compiles to
native bytes for both architectures. Backend selection is automatic
based on the detected host architecture.

** Formal Verification

Before any generated byte sequence is JIT-linked, it passes through
=verify-bytes=:

#+BEGIN_SRC lisp
;; Checks:
;; 1. Non-empty sequence
;; 2. Terminates with RET (0xC3 on x86-64, 0xd65f03c0 on ARM64)
;; 3. Valid first byte (prologue or known REX prefix)
;; 4. No orphan REX prefixes (each REX must precede a valid instruction)
(verify-bytes generated-bytes) ; => (values t "ok") or (values nil reason)
#+END_SRC

** Key Files

| File | Responsibility |
|------|----------------|
| =llm2asm/tokenizer.lisp= | Intent tokenization and kernel map |
| =llm2asm/ast-extract.lisp= | AST extraction from token stream |
| =llm2asm/codegen.lisp= | Code generation: AST -> bytes |
| =llm2asm/verify.lisp= | Formal byte sequence verification |
| =llm2asm/jit-link.lisp= | JIT linking via mmap/CFFI |
| =llm2asm/constraint-build.lisp= | Constraint algebra and extraction |
| =llm2asm/backends/x86-64.lisp= | Raw x86-64 instruction encoders |
| =llm2asm/backends/arm64.lisp= | Raw AArch64 instruction encoders |

* Self-Modification Layer (selfmod/)

The selfmod layer sits across all five layers. It provides:

** Hot-Swap (hot-swap.lisp)

Redefine any function in the running Nekomata process without
restarting. The old definition is preserved in =*hot-swap-log*= for
rollback. Thread-safe via =bt:with-lock-held=.

** State Capture (state-capture.lisp)

Snapshot the complete Nekomata runtime state: container registry,
evolutionary population, hot-swap log, generation counter. Snapshots
are serializable to S-expressions for WORM-sealed persistence.

** Evolutionary Optimization (gen-loop.lisp, evolve.lisp)

Run a genetic algorithm over the live container spec population.
Each generation:
1. Score all specs against runtime telemetry (fitness function)
2. Select survivors by tournament selection
3. Apply crossover and mutation
4. Replace the live population with the next generation

The scheduler keeps running throughout. Evolution is not a migration.
It is a continuous background optimization.

** Rollback (rollback.lisp)

Multi-level undo: function level (hot-swap log), state level
(snapshot store), generation level. The =with-rollback= macro
provides automatic rollback on error:

#+BEGIN_SRC lisp
(with-rollback ("before-risky-experiment")
  (do-something-that-might-fail))
;; If an error signals, the checkpoint is automatically restored
#+END_SRC

* Data Flow Summary

#+BEGIN_EXAMPLE
  User intent (NL string)
       |
  llm2asm/tokenizer         [tokenize]
       |
  llm2asm/ast-extract       [classify intent type]
       |
  llm2asm/constraint-build  [extract resource constraints]
       |
  moe/router                [activate relevant experts]
       |
  moe/experts/*             [expert domain logic]

       |

  llm2asm/codegen           [emit x86-64 / ARM64 bytes]

       |

  llm2asm/verify            [formal safety check]

       |

  llm2asm/jit-link          [map bytes as executable, get fn ptr]

       |

  docker/container          [create/start via Unix socket]

       |

  selfmod/state-capture     [snapshot new state]

       |

  selfmod/gen-loop          [feed telemetry to evolution]

#+END_EXAMPLE



* System Requirements



- SBCL 2.3+ or CCL 1.12+ (any ANSI Common Lisp with UIOP)

- Docker daemon with Unix socket at =/var/run/docker.sock=

- CFFI (for production JIT mmap integration)

- Bordeaux-Threads (for hot-swap lock)

- Linux kernel 5.10+ or macOS 12+ (for mmap PROT_EXEC)

- x86-64 or ARM64 host (backends are host-architecture-specific)



* Build and Load



#+BEGIN_SRC shell

# Load the system

sbcl --load nekod.asd --eval "(asdf:load-system :nekod)"



# Start the daemon

sbcl --eval "(nekod:nekomata-main)"



# Or use the neko CLI

sbcl --script cli/neko.lisp run myapp.lisp

#+END_SRC