|
|
|
|
|
|
|
|
|
|
|
|
|
|
| * Overview
|
|
|
| Nekomata is a five-layer system. Each layer has a single responsibility
|
| and a clean interface to the layer above and below it. The layers are
|
| not microservices
|
| interfaces are function calls, not HTTP requests.
|
|
|
|
|
| Layer 5: LLM -> ASM (intent compilation)
|
| |
|
| Layer 4: Regex Math (formal policy algebra)
|
| |
|
| Layer 3: MoE Router (sparse expert activation)
|
| |
|
| Layer 2: Docker Socket (raw container operations)
|
| |
|
| Layer 1: Core Image (Lisp image-based persistence)
|
|
|
|
|
| Data flows down (intent -> bytes) and up (telemetry -> fitness signal).
|
| Control flows down (intent -> execution). State is horizontal (every
|
| layer can read the Lisp image).
|
|
|
| * Layer 1: Core (Lisp Image-Based Persistence)
|
|
|
| ** What It Is
|
|
|
| The Core layer is a persistent Common Lisp image. All Nekomata state
|
| lives here: running container registry, evolutionary population, hot-swap
|
| log, snapshot store, policy tables.
|
|
|
| When Nekomata starts, it restores from the last saved image. When it
|
| stops, it saves the image. The image is not a database dump
|
| the entire live Lisp process state, including compiled functions,
|
| hash tables, and all in-memory data.
|
|
|
| ** Why This Matters
|
|
|
| Most container engines lose all in-memory state on restart. They
|
| reconstruct state by replaying events from a persistent log (etcd in
|
| Kubernetes). This requires:
|
| - A separate storage system (etcd)
|
| - A reconciliation loop (the control plane rebuilds desired state)
|
| - A startup latency (the control plane must catch up)
|
|
|
| Nekomata eliminates all three by persisting the Lisp image directly.
|
| The image is the database. Restart is image restore. There is no
|
| reconciliation loop because there is no gap between desired state and
|
| actual state
|
| previous session.
|
|
|
| ** Key Files
|
|
|
| | File | Responsibility |
|
| |
|
| | =core/package.lisp= | Package and export declarations |
|
| | =core/sexpr-infra.lisp= | Container spec DSL and =define-container= macro |
|
| | =core/eval-loop.lisp= | REPL and infrastructure eval loop |
|
| | =core/image-persist.lisp= | Lisp image save/restore |
|
| | =core/conditions.lisp= | Condition hierarchy |
|
| | =core/nekomata.lisp= | Daemon entry point and container registry |
|
|
|
| ** Core Data Model
|
|
|
|
|
| (defstruct container-spec
|
| name
|
| image
|
| ports ; list of (host-port . container-port)
|
| env ; alist of (KEY . "value")
|
| volumes ; list of (host-path . container-path)
|
| healthcheck ; lambda or nil
|
| labels ; alist
|
| restart-policy ; :always :on-failure :never
|
| command) ; list of strings or nil
|
|
|
|
|
| The =define-container= macro expands to a =defvar= that installs a
|
| =container-spec= in the image and registers it in =*running-containers*=.
|
|
|
| ** Persistence Protocol
|
|
|
|
|
| ;; Save
|
| (save-nekod-image "/var/lib/nekod/nekod.image")
|
|
|
| ;; Restore
|
| (restore-nekod-image "/var/lib/nekod/nekod.image")
|
|
|
|
|
| Image saves happen: on clean shutdown, on SIGTERM, on explicit REPL
|
| call, and after every evolutionary generation (configurable).
|
|
|
| * Layer 2: Docker (Raw Socket, No Abstraction Layer)
|
|
|
| ** What It Is
|
|
|
| The Docker layer communicates with the Docker daemon over a Unix domain
|
| socket using raw HTTP/1.1 requests. There is no Docker SDK dependency.
|
| There are no generated client bindings. There is a function that writes
|
| bytes to a socket and a function that reads bytes from a socket.
|
|
|
| ** Why No Abstraction Layer
|
|
|
| Every Docker SDK (Python dockerpy, Go docker/client, Java docker-java)
|
| wraps the same HTTP API in a language-specific object model. This adds:
|
| - Dependency chains (transitive dependencies on JSON parsers, HTTP
|
| clients, TLS libraries)
|
| - Version coupling (the SDK version must match the Docker API version)
|
| - Abstraction leakage (the SDK exposes Docker's internal concepts
|
| without the option to hide them)
|
|
|
| Nekomata talks to Docker the same way curl does: raw TCP, HTTP/1.1
|
| framing, JSON bodies assembled by =format=. This means zero dependencies
|
| beyond a working Lisp and a running Docker daemon.
|
|
|
| ** Key Files
|
|
|
| | File | Responsibility |
|
| |------|----------------|
|
| | =docker/unix-sock.lisp= | Raw Unix socket I/O |
|
| | =docker/protocol.lisp= | HTTP/1.1 request/response framing |
|
| | =docker/container.lisp= | Container lifecycle (create/start/stop/kill) |
|
| | =docker/image.lisp= | Image pull and inspect |
|
| | =docker/network.lisp= | Network create/connect/disconnect |
|
| | =docker/volume.lisp= | Volume create/mount/remove |
|
| | =docker/inspect.lisp= | Container state and stats polling |
|
|
|
| ** API Surface
|
|
|
| #+BEGIN_SRC lisp
|
| ;; The full public interface of the Docker layer
|
| (docker-get "/containers/json") ; list containers
|
| (docker-post "/containers/create" body-json) ; create container
|
| (start-container id) ; POST /containers/{id}/start
|
| (stop-container id) ; POST /containers/{id}/stop
|
| (kill-container id) ; POST /containers/{id}/kill
|
| (inspect-container id) ; GET /containers/{id}/json
|
| #+END_SRC
|
|
|
| All functions return parsed JSON as Lisp association lists.
|
|
|
| * Layer 3: MoE Router (Sparse Expert Activation)
|
|
|
| ** What It Is
|
|
|
| The MoE (Mixture of Experts) router matches an infrastructure intent
|
| to the minimal set of expert modules required to handle it. Experts are
|
| Common Lisp functions registered in a routing table with capability
|
| descriptors.
|
|
|
| ** Architecture
|
|
|
| #+BEGIN_EXAMPLE
|
| Intent string
|
| |
|
| [Tokenizer] -> token list with intent classes
|
| |
|
| [Router] -> score each expert against token list
|
| |
|
| [Threshold] -> keep experts with score > 0.5
|
| |
|
| [Dispatch] -> call each activated expert with the intent AST
|
| |
|
| [Aggregate] -> merge expert outputs into a single action
|
| #+END_EXAMPLE
|
|
|
| ** Expert Registration
|
|
|
| #+BEGIN_SRC lisp
|
| (register-expert :placement
|
| :capabilities '(:schedule :place :deploy :run)
|
| :weight 1.0
|
| :handler
|
|
|
| (register-expert :routing
|
| :capabilities '(:route :traffic :proxy :forward)
|
| :weight 0.9
|
| :handler #'routing-expert)
|
|
|
|
|
| ** Sparse Activation
|
|
|
| Given N registered experts and intent I:
|
| 1. Compute score(e, I) for all e in E using token overlap + semantic
|
| similarity against capability keywords
|
| 2. Activate A = {e : score(e, I) > threshold}
|
| 3. |A| is typically 1-2 for most intents, never all of E
|
|
|
| This is the same sparsity property as a sparse MoE language model.
|
| Only the relevant experts burn compute for each token (intent).
|
|
|
| ** Key Files
|
|
|
| | File | Responsibility |
|
| |
|
| | =moe/router.lisp= | Expert registry and routing algorithm |
|
| | =moe/sparse-activate.lisp= | Scoring and threshold gating |
|
| | =moe/experts/= | Directory of expert module implementations |
|
|
|
| * Layer 4: Regex Math (Formal Pattern Algebra)
|
|
|
| ** What It Is
|
|
|
| The regex-math layer implements a Kleene algebra over container
|
| identity strings, port patterns, and operation codes. This algebra
|
| is the formal policy language for Nekomata.
|
|
|
| ** Kleene Algebra Primitives
|
|
|
| The algebra has five operations:
|
| - =re-literal= :: match a literal string
|
| - =re-concat= :: sequential composition (A then B)
|
| - =re-union= :: disjunction (A or B)
|
| - =re-star= :: Kleene star (A zero or more times)
|
| - =re-not= :: complement (everything that is not A)
|
|
|
| These compose to express any regular policy:
|
|
|
|
|
| ;; Allow traffic from any api-* container to db-* on ports 5432-5435
|
| (define-policy db-access
|
| :from (re-concat (re-literal "api-") (re-star re-any))
|
| :to (re-concat (re-literal "db-") (re-star re-any))
|
| :port (re-union (re-literal "5432") (re-literal "5433")
|
| (re-literal "5434") (re-literal "5435")))
|
|
|
|
|
| ** Formal Properties
|
|
|
| Because the policy language is a Kleene algebra:
|
|
|
| 1. *Equivalence checking*: Two policies P1 and P2 are equivalent iff
|
| their DFA representations accept the same language. Decidable in
|
| O(n^2 * m) where n, m are state counts.
|
|
|
| 2. *Subsumption*: P1 subsumes P2 iff L(P2) ⊆ L(P1). Check by
|
| computing L(P2) \ L(P1) and verifying it is empty.
|
|
|
| 3. *Conflict detection*: Policies P_allow and P_deny conflict iff
|
| L(P_allow) ∩ L(P_deny) ≠ ∅. Compute the intersection NFA and
|
| check emptiness.
|
|
|
| ** Key Files
|
|
|
| | File | Responsibility |
|
| |
|
| | =regex-math/= | Kleene algebra engine, NFA/DFA, policy compiler |
|
|
|
| * Layer 5: LLM to ASM (Intent Compilation)
|
|
|
| ** What It Is
|
|
|
| The llm2asm layer is the compilation pipeline from natural language
|
| intent to raw machine code bytes. It is the outermost interface of
|
| Nekomata: the point at which human language enters the system and
|
| machine instructions exit.
|
|
|
| ** Compilation Pipeline
|
|
|
|
|
| "deploy the API server"
|
| |
|
| [Tokenizer] tokenize-intent
|
| -> list of intent-token structs
|
| |
|
| [AST Extractor] extract-ast
|
| -> infra-ast with type :place, target "API"
|
| |
|
| [Constraint Builder] extract-constraints-from-tokens
|
| -> list of constraint structs
|
| |
|
| [Code Generator] codegen
|
| -> raw x86-64 or ARM64 bytes
|
| |
|
| [Verifier] verify-bytes
|
| -> check: ends with RET, valid prologue, no
|
| orphan REX prefixes
|
| |
|
| [JIT Linker] jit-link
|
| -> function pointer in executable memory
|
|
|
|
|
| ** Intent Types
|
|
|
| Five intent types map to five kernel primitives:
|
|
|
| | Intent Type | Trigger Words | Kernel Primitive |
|
| |
|
| | =:place= | PLACE SCHEDULE DEPLOY RUN | ADD (scheduling arithmetic) |
|
| | =:route= | ROUTE TRAFFIC PROXY FORWARD | XOR (packet classification) |
|
| | =:scale= | SCALE REPLICATE EXPAND SHRINK | LOOP (replica iteration) |
|
| | =:heal= | HEAL RESTART RECOVER FIX | MEMCPY (state snapshot) |
|
| | =:secure= | SECURE ISOLATE LOCK PROTECT | MEMSET (zero-fill) |
|
|
|
| ** Backend Architecture
|
|
|
| The codegen layer has two backends:
|
|
|
| | Backend | File | Target |
|
| |
|
| | x86-64 | =llm2asm/backends/x86-64.lisp= | AMD64 / Intel 64-bit |
|
| | ARM64 | =llm2asm/backends/arm64.lisp= | Apple Silicon / Graviton |
|
|
|
| Backends share the same AST input. The same intent compiles to
|
| native bytes for both architectures. Backend selection is automatic
|
| based on the detected host architecture.
|
|
|
| ** Formal Verification
|
|
|
| Before any generated byte sequence is JIT-linked, it passes through
|
| =verify-bytes=:
|
|
|
|
|
| ;; Checks:
|
| ;; 1. Non-empty sequence
|
| ;; 2. Terminates with RET (0xC3 on x86-64, 0xd65f03c0 on ARM64)
|
| ;; 3. Valid first byte (prologue or known REX prefix)
|
| ;; 4. No orphan REX prefixes (each REX must precede a valid instruction)
|
| (verify-bytes generated-bytes) ; => (values t "ok") or (values nil reason)
|
|
|
|
|
| ** Key Files
|
|
|
| | File | Responsibility |
|
| |
|
| | =llm2asm/tokenizer.lisp= | Intent tokenization and kernel map |
|
| | =llm2asm/ast-extract.lisp= | AST extraction from token stream |
|
| | =llm2asm/codegen.lisp= | Code generation: AST -> bytes |
|
| | =llm2asm/verify.lisp= | Formal byte sequence verification |
|
| | =llm2asm/jit-link.lisp= | JIT linking via mmap/CFFI |
|
| | =llm2asm/constraint-build.lisp= | Constraint algebra and extraction |
|
| | =llm2asm/backends/x86-64.lisp= | Raw x86-64 instruction encoders |
|
| | =llm2asm/backends/arm64.lisp= | Raw AArch64 instruction encoders |
|
|
|
| * Self-Modification Layer (selfmod/)
|
|
|
| The selfmod layer sits across all five layers. It provides:
|
|
|
| ** Hot-Swap (hot-swap.lisp)
|
|
|
| Redefine any function in the running Nekomata process without
|
| restarting. The old definition is preserved in =*hot-swap-log*= for
|
| rollback. Thread-safe via =bt:with-lock-held=.
|
|
|
| ** State Capture (state-capture.lisp)
|
|
|
| Snapshot the complete Nekomata runtime state: container registry,
|
| evolutionary population, hot-swap log, generation counter. Snapshots
|
| are serializable to S-expressions for WORM-sealed persistence.
|
|
|
| ** Evolutionary Optimization (gen-loop.lisp, evolve.lisp)
|
|
|
| Run a genetic algorithm over the live container spec population.
|
| Each generation:
|
| 1. Score all specs against runtime telemetry (fitness function)
|
| 2. Select survivors by tournament selection
|
| 3. Apply crossover and mutation
|
| 4. Replace the live population with the next generation
|
|
|
| The scheduler keeps running throughout. Evolution is not a migration.
|
| It is a continuous background optimization.
|
|
|
| ** Rollback (rollback.lisp)
|
|
|
| Multi-level undo: function level (hot-swap log), state level
|
| (snapshot store), generation level. The =with-rollback= macro
|
| provides automatic rollback on error:
|
|
|
|
|
| (with-rollback ("before-risky-experiment")
|
| (do-something-that-might-fail))
|
| ;; If an error signals, the checkpoint is automatically restored
|
|
|
|
|
| * Data Flow Summary
|
|
|
|
|
| User intent (NL string)
|
| |
|
| llm2asm/tokenizer [tokenize]
|
| |
|
| llm2asm/ast-extract [classify intent type]
|
| |
|
| llm2asm/constraint-build [extract resource constraints]
|
| |
|
| moe/router [activate relevant experts]
|
| |
|
| moe/experts |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
