MnemoCore / README.md
Granis87's picture
Upload README.md with huggingface_hub
ceae510 verified
|
Raw
History Blame Contribute Delete
52.5 kB
---
title: MnemoCore
emoji: 🧠
colorFrom: indigo
colorTo: purple
sdk: docker
pinned: false
license: mit
language:
- en
tags:
- memory
- vector-database
- cognitive-architecture
- agents
- hyperdimensional-computing
- vsa
---
# MnemoCore
### Infrastructure for Persistent Cognitive Memory
> *"Memory is not a container. It is a living process β€” a holographic continuum where every fragment contains the whole."*
<p align="center">
<img src="https://img.shields.io/badge/Status-Beta%203.0.0-blue?style=for-the-badge" />
<img src="https://img.shields.io/badge/Python-3.10%2B-3776AB?style=for-the-badge&logo=python&logoColor=white" />
<img src="https://img.shields.io/badge/FastAPI-Async%20Ready-009688?style=for-the-badge&logo=fastapi&logoColor=white" />
<img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" />
<img src="https://img.shields.io/badge/HDV-16384--dim-purple?style=for-the-badge" />
<img src="https://img.shields.io/badge/Vectors-Binary%20VSA-critical?style=for-the-badge" />
</p>
---
## Quick Install
**Install v3.0.0 after publication:**
```bash
pip install mnemocore==3.0.0
```
**Install from source (development):**
```bash
git clone https://github.com/RobinALG87/MnemoCore-Persistent-Cognitive-Ai-Memory.git
cd MnemoCore-Persistent-Cognitive-Ai-Memory
python -m venv .venv && .\.venv\Scripts\activate # Windows
# source .venv/bin/activate # Linux / macOS
pip install -e . # editable install
pip install -e ".[dev]" # + pytest, mypy, black, etc.
```
> **For the REST server, set your API key before starting:**
> ```bash
> # Windows PowerShell
> $env:HAIM_API_KEY = "your-secure-key"
> # Linux / macOS
> # export HAIM_API_KEY="your-secure-key"
> ```
> Then start the legacy v2 API locally: `mnemocore`.
`mnemocore` starts the legacy v2 REST server; `mnemocore-cli` is the legacy
Click CLI. Neither is the v3 AgentMemory persistence boundary. For v3, compose
`HybridMemoryRuntime` over an explicit scope or deploy `create_v3_app` with a
credential-to-scope authorizer. AgentMemory itself is local-first and does not
require a running Redis or Qdrant service. Full legacy setup, Docker, and
configuration details are in [Installation](#installation) below.
---
## MnemoCore 3.0.0 beta
MnemoCore is persistent memory and context infrastructure for agents. Version
3.0.0 beta releases an **AgentMemory-first** runtime: local-first, exact-scope
SQLite memory with timeline/history support, deterministic context compilation,
and rebuildable projections. `HybridMemoryRuntime` adds deterministic lexical
and BinaryHDV retrieval without adding a second persistence path.
The v3 contract is explicit:
- Every operation requires a complete `MemoryScope`; cross-scope fallback is
forbidden.
- AgentMemory is the only durable source of truth for v3.
- Cognitive effects are validated plans, not autonomous direct writes.
- The v3 HTTP application is composed with a scope authorizer and fails closed
when one is absent.
- v2 HAIM, REST, MCP, and CLI paths remain compatibility-only and are not the
v3 persistence layer.
Version 3.0 retains the established dependency set for installation
compatibility. A dependency-minimal AgentMemory distribution remains a future
major-release migration.
### Legacy cognitive capabilities (v2 compatibility only)
The following HAIM description is retained as a reference for existing v2
deployments. It is not a v3 persistence or deployment contract.
Traditional vector stores retrieve. MnemoCore **thinks**. It is built on the mathematical framework of **Binary Hyperdimensional Computing (HDC)** and **Vector Symbolic Architectures (VSA)**, principles rooted in Pentti Kanerva's landmark 2009 theory of cognitive computing. Every memory is encoded as a **16,384-dimensional binary holographic vector** β€” a format that is simultaneously compact (2,048 bytes), noise-tolerant (Hamming geometry), and algebraically rich (XOR binding, majority bundling, circular permutation).
At its core lives the **Holographic Active Inference Memory (HAIM) Engine** β€” a system that does not merely answer queries, but:
- **Evaluates** the epistemic novelty of every incoming memory before deciding to store it
- **Dreams** β€” strengthening synaptic connections between related memories during idle cycles
- **Reasons by analogy** β€” if `king:man :: ?:woman`, the VSA soul computes `queen`
- **Self-organizes** into tiered storage based on biologically-inspired Long-Term Potentiation (LTP)
- **Scales** from a single process to distributed nodes targeting 1B+ memories
Phase 4.x introduces cognitive enhancements including contextual masking, reliability feedback loops, semantic consolidation, gap detection/filling, temporal recall (episodic chaining + chrono-weighted query), a Subconscious Daemon with LLM-powered dream synthesis, and a full dependency-injection container pattern for clean modularity.
Phase 5 transforms MnemoCore into a true cognitive architecture with Working Memory, Episodic/Semantic/Procedural stores, Meta Memory, and a Self-Improvement Worker β€” all orchestrated by an 11-phase Pulse loop.
Phase 6 adds four research-backed services: **StrategyBank** (reasoning bank with Bayesian confidence), **KnowledgeGraph** (spreading activation + community detection), **MemoryScheduler** (OS-level priority queue with interrupts), and **SAMEP** (multi-agent memory exchange with HMAC integrity).
---
## Table of Contents
- [Architecture](#architecture)
- [Core Technology](#core-technology-binary-hdv--vsa)
- [The Memory Lifecycle](#the-memory-lifecycle)
- [Tiered Storage](#tiered-storage-hotwarmcold)
- [Phase 4.0 Cognitive Enhancements](#phase-40-cognitive-enhancements)
- [Phase 4.4–4.5 Subconscious Daemon & LLM Integration](#phase-4445-subconscious-daemon--llm-integration)
- [API Reference](#api-reference)
- [Python Library Usage](#python-library-usage)
- [Installation](#installation)
- [Configuration](#configuration)
- [MCP Server Integration](#mcp-server-integration)
- [Observability](#observability)
- [Roadmap](#roadmap)
- [Contributing](#contributing)
---
## Architecture
The diagram below documents the legacy v2 architecture. New v3 deployments use
the scoped AgentMemory and HybridMemoryRuntime composition described above.
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ MnemoCore Stack β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ REST API (FastAPI / Async) β”‚ β”‚
β”‚ β”‚ /store /query /feedback /insights/gaps /stats β”‚ β”‚
β”‚ β”‚ Rate Limiting Β· API Key Auth Β· Prometheus Metrics β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ HAIM Engine β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ Text Encoder β”‚ β”‚ EIG / Epist β”‚ β”‚ Subconsc. β”‚ β”‚ β”‚
│ │ │ (token→HDV) │ │ Drive │ │ Dream Loop │ │ │
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ Binary HDV Core (VSA) β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ XOR bind Β· majority_bundle Β· permute Β· Hamming β”‚ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Tier Manager β”‚ β”‚
β”‚ β”‚ β”‚ β”‚
β”‚ β”‚ πŸ”₯ HOT 🌑 WARM ❄️ COLD β”‚ β”‚
β”‚ β”‚ In-Memory Redis / mmap Qdrant / Disk / S3 β”‚ β”‚
β”‚ β”‚ ≀2,000 nodes ≀100,000 nodes ∞ nodes β”‚ β”‚
β”‚ β”‚ <1ms <10ms <100ms β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Conceptual Layer ("The Soul") β”‚ β”‚
β”‚ β”‚ ConceptualMemory Β· Analogy Engine Β· Symbol Algebra β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
### Component Overview
| Component | File | Responsibility |
|-----------|------|----------------|
| **HAIM Engine** | `src/mnemocore/core/engine.py` | Central cognitive coordinator β€” store, query, dream, delete |
| **BinaryHDV** | `src/mnemocore/core/binary_hdv.py` | 16384-dim binary vector math (XOR, Hamming, bundle, permute) |
| **TextEncoder** | `src/mnemocore/core/binary_hdv.py` | Token→HDV pipeline with positional permutation binding |
| **MemoryNode** | `src/mnemocore/core/node.py` | Memory unit with LTP, epistemic values, tier state |
| **TierManager** | `src/mnemocore/core/tier_manager.py` | HOT/WARM/COLD orchestration with LTP-driven eviction |
| **SynapticConnection** | `src/mnemocore/core/synapse.py` | Hebbian synapse with strength, decay, and fire tracking |
| **SynapseIndex** | `src/mnemocore/core/synapse_index.py` | Fast synapse lookup index for associative spreading |
| **ConceptualMemory** | `src/mnemocore/core/holographic.py` | VSA soul for analogy and cross-domain symbolic reasoning |
| **AsyncRedisStorage** | `src/mnemocore/core/async_storage.py` | Async Redis backend (WARM tier + pub/sub) |
| **BayesianLTP** | `src/mnemocore/core/bayesian_ltp.py` | Bayesian reliability scoring on top of LTP strength |
| **SemanticConsolidation** | `src/mnemocore/core/semantic_consolidation.py` | Memory deduplication via majority-bundle prototyping |
| **ConsolidationWorker** | `src/mnemocore/core/consolidation_worker.py` | Async worker scheduling nightly consolidation |
| **GapDetector** | `src/mnemocore/core/gap_detector.py` | Temporal co-occurrence analysis for knowledge gaps |
| **GapFiller** | `src/mnemocore/core/gap_filler.py` | Bridge detected gaps via synapse creation |
| **Immunology** | `src/mnemocore/core/immunology.py` | Auto-associative attractor cleanup for vector drift |
| **Attention** | `src/mnemocore/core/attention.py` | XOR context masking / project isolation |
| **BatchOps** | `src/mnemocore/core/batch_ops.py` | Vectorized bulk store / query operations |
| **HNSWIndex** | `src/mnemocore/core/hnsw_index.py` | In-process HNSW approximate nearest-neighbour index |
| **QdrantStore** | `src/mnemocore/core/qdrant_store.py` | Async Qdrant COLD tier backend |
| **RecursiveSynthesizer** | `src/mnemocore/core/recursive_synthesizer.py` | Deep concept synthesis via iterative VSA composition |
| **RippleContext** | `src/mnemocore/core/ripple_context.py` | Cascading context propagation across synaptic graph |
| **SubconsciousAI** | `src/mnemocore/core/subconscious_ai.py` | LLM-guided dream synthesis worker |
| **SubconsciousDaemon** | `src/mnemocore/subconscious/daemon.py` | Background process orchestrating dream/consolidation cycles |
| **LLMIntegration** | `src/mnemocore/llm_integration.py` | Agent-facing LLM connector (OpenAI / Anthropic compatible) |
| **Container** | `src/mnemocore/core/container.py` | Dependency-injection wiring for all core components |
| **GoalTree** | `src/mnemocore/meta/goal_tree.py` | Hierarchical goal / task tracking for meta-cognition |
| **LearningJournal** | `src/mnemocore/meta/learning_journal.py` | Persistent log of what the agent has learned over time |
| **API** | `src/mnemocore/api/main.py` | Legacy v2 HAIM REST interface; v3 uses `create_v3_app` with scope authorization |
| **MCP Server** | `src/mnemocore/mcp/server.py` | Legacy v2 HAIM REST adapter; v3 uses scoped AgentMemory integration helpers |
---
## Core Technology: Binary HDV & VSA
MnemoCore's mathematical foundation is **Hyperdimensional Computing** β€” a computing paradigm that encodes information in very high-dimensional binary vectors (HDVs), enabling noise-tolerant, distributed, and algebraically composable representations.
### The Vector Space
Every piece of information β€” a word, a sentence, a concept, a goal β€” is encoded as a **16,384-dimensional binary vector**:
```
Dimension D = 16,384 bits = 2,048 bytes per vector
Storage: packed as numpy uint8 arrays
Similarity: Hamming distance (popcount of XOR result)
Random pair: ~50% similarity (orthogonality by probability)
```
At this dimensionality, two random vectors will differ in ~50% of bits. This near-orthogonality is the foundation of the system's expressive power β€” related concepts cluster together while unrelated ones remain maximally distant.
### VSA Algebra
Four primitive operations make the entire system work:
#### Binding β€” XOR `βŠ•`
Creates an association between two concepts. Crucially, the result is **dissimilar to both inputs** (appears as noise), making it a true compositional operation.
```python
# Bind content to its context
bound = content_vec.xor_bind(context_vec) # content βŠ• context
# Self-inverse: unbind by re-binding
recovered = bound.xor_bind(context_vec) # β‰ˆ content (XOR cancels)
```
Key mathematical properties:
- **Self-inverse**: `A βŠ• A = 0` (XOR cancels itself)
- **Commutative**: `A βŠ• B = B βŠ• A`
- **Distance-preserving**: `hamming(AβŠ•C, BβŠ•C) = hamming(A, B)`
#### Bundling β€” Majority Vote
Creates a **prototype** that is similar to all inputs. This is how multiple memories combine into a concept.
```python
from mnemocore.core.binary_hdv import majority_bundle
# Create semantic prototype from related memories
concept = majority_bundle([vec_a, vec_b, vec_c, vec_d]) # similar to all inputs
```
#### Permutation β€” Circular Shift
Encodes **sequence and roles** without separate positional embeddings.
```python
# Positional encoding: token at position i
positioned = token_vec.permute(shift=i) # circular bit-shift
# Encode "hello world" with order information
hello_positioned = encoder.get_token_vector("hello").permute(0)
world_positioned = encoder.get_token_vector("world").permute(1)
sentence_vec = majority_bundle([hello_positioned, world_positioned])
```
#### Similarity β€” Hamming Distance
Fast comparison using vectorized popcount over XOR results:
```python
# Normalized similarity: 1.0 = identical, 0.5 = unrelated
sim = vec_a.similarity(vec_b) # 1.0 - hamming(a, b) / D
# Batch nearest-neighbor search (no Python loops)
distances = batch_hamming_distance(query, database_matrix)
```
### Text Encoding Pipeline
The `TextEncoder` converts natural language to HDVs using a token-position binding scheme:
```
"Python TypeError" β†’
token_hdv("python") βŠ• permute(0) = positioned_0
token_hdv("typeerror") βŠ• permute(1) = positioned_1
majority_bundle([positioned_0, positioned_1]) = final_hdv
```
Token vectors are **deterministic** β€” seeded via SHAKE-256 hash β€” meaning the same word always produces the same base vector, enabling cross-session consistency without a vocabulary file.
---
## The Memory Lifecycle
Every memory passes through a defined lifecycle from ingestion to long-term storage:
```
Incoming Content
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ TextEncoder β”‚ β†’ 16,384-dim binary HDV
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Context Binding β”‚ β†’ XOR bind with goal_context if present
β”‚ (XOR) β”‚ bound_vec = content βŠ• context
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ EIG Evaluation β”‚ β†’ Epistemic Information Gain
β”‚ (Novelty Check) β”‚ eig = normalized_distance(vec, context_vec)
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ tag "epistemic_high" if eig > threshold
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ MemoryNode β”‚ β†’ id, hdv, content, metadata
β”‚ Creation β”‚ ltp_strength = I Γ— log(1+A) Γ— e^(-Ξ»T)
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ HOT Tier β”‚ β†’ In-memory dict (max 2000 nodes)
β”‚ (RAM) β”‚ LTP eviction: low-LTP nodes β†’ WARM
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ (background)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Subconscious β”‚ β†’ Dream cycle fires
β”‚ Dream Loop β”‚ Query similar memories
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ Strengthen synapses (Hebbian)
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ WARM Tier β”‚ β†’ Redis-backed persistence
β”‚ (Redis/mmap) β”‚ async dual-write + pub/sub events
β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ (scheduled, nightly)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ COLD Tier β”‚ β†’ Qdrant / Disk / S3
β”‚ (Archival) β”‚ ANN search, long-term persistence
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
### Long-Term Potentiation (LTP)
Memories are not equal. Importance is computed dynamically using a biologically-inspired LTP formula:
```
S = I Γ— log(1 + A) Γ— e^(-Ξ» Γ— T)
Where:
S = LTP strength (determines tier placement)
I = Importance (derived from epistemic + pragmatic value)
A = Access count (frequency of retrieval)
Ξ» = Decay lambda (configurable, default ~0.01)
T = Age in days
```
Memories with high LTP remain in HOT tier. Those that decay are automatically promoted to WARM, then COLD β€” mirroring how biological memory consolidates from working memory to long-term storage.
### Synaptic Connections
Memories are linked by `SynapticConnection` objects that implement Hebbian learning: *"neurons that fire together, wire together."*
Every time two memories are co-retrieved (via the background dream loop or explicit binding), their synaptic strength increases. During query time, synaptic spreading amplifies scores of connected memories even when they do not directly match the query vector β€” enabling **associative recall**.
```python
# Explicit synapse creation
engine.bind_memories(id_a, id_b, success=True)
# Associative spreading: query top seeds spread activation to neighbors
# neighbor_score += seed_score Γ— synapse_strength Γ— 0.3
```
---
## Tiered Storage: HOT / WARM / COLD
| Tier | Backend | Capacity | Latency | Eviction Trigger |
|------|---------|----------|---------|------------------|
| πŸ”₯ **HOT** | Python dict (RAM) | 2,000 nodes | < 1ms | LTP < threshold |
| 🌑 **WARM** | Redis + mmap | 100,000 nodes | < 10ms | Age + low access |
| ❄️ **COLD** | Qdrant / Disk / S3 | Unlimited | < 100ms | Manual / scheduled |
Promotion is automatic: accessing a WARM or COLD memory re-promotes it to HOT based on recalculated LTP. Eviction is LRU-weighted by LTP strength β€” the most biologically active memories always stay hot.
---
## Phase 4.0 Cognitive Enhancements
MnemoCore Phase 4.0 introduces five architectural enhancements that elevate the system from **data retrieval** to **cognitive reasoning**.
---
### 1. Contextual Query Masking *(XOR Attention)*
**Problem**: Large multi-project deployments suffer from cross-context interference. A query for `"Python error handling"` returns memories from all projects equally, diluting precision.
**Solution**: Bidirectional XOR context binding β€” apply the same context vector at both **storage** and **query** time:
```
Store: bound_vec = content βŠ• context_vec
Query: masked_query = query βŠ• context_vec
Result: (content βŠ• C) Β· (query βŠ• C) β‰ˆ content Β· query
(context cancels, cross-project noise is suppressed)
```
```python
# Store memories in a project context
engine.store("API rate limiting logic", goal_id="ProjectAlpha")
engine.store("Garden watering schedule", goal_id="HomeProject")
# Query with context mask β€” only ProjectAlpha memories surface
results = engine.query("API logic", top_k=5, context="ProjectAlpha")
```
**Expected impact**: +50–80% query precision (P@5) in multi-project deployments.
---
### 2. Reliability Feedback Loop *(Self-Correcting Memory)*
**Problem**: Wrong or outdated memories persist with the same retrieval weight as correct ones. The system has no mechanism to learn from its own mistakes.
**Solution**: Bayesian reliability scoring with real-world outcome feedback:
```
reliability = (successes + 1) / (successes + failures + 2) # Laplace smoothing
LTP_enhanced = I Γ— log(1+A) Γ— e^(-Ξ»T) Γ— reliability
```
```python
# After using a retrieved memory:
engine.provide_feedback(memory_id, outcome=True) # Worked β†’ boost reliability
engine.provide_feedback(memory_id, outcome=False) # Failed β†’ reduce reliability
# System auto-tags consistently wrong memories as "unreliable"
# and verified memories (>5 successes, >0.8 score) as "verified"
```
The system converges toward **high-confidence knowledge** β€” memories that have demonstrably worked in practice rank above theoretically similar but unproven ones.
---
### 3. Semantic Memory Consolidation *(Dream-Phase Synthesis)*
**Problem**: Episodic memory grows without bound. 1,000 memories about `"Python TypeError"` are semantically equivalent but consume 2MB of vector space and slow down linear scan queries.
**Solution**: Nightly `ConsolidationWorker` clusters similar WARM tier memories and replaces them with a **semantic anchor** β€” a majority-bundled prototype:
```
BEFORE consolidation:
mem_001: "Python TypeError in line 45" (2KB vector)
mem_002: "TypeError calling function" (2KB vector)
... Γ—100 similar memories (200KB total)
AFTER consolidation:
anchor_001: "Semantic pattern: python typeerror function"
metadata: {source_count: 100, confidence: 0.94}
hdv: majority_bundle([mem_001.hdv, ..., mem_100.hdv]) (2KB)
```
```python
# Manual trigger (runs automatically at 3 AM)
stats = engine.trigger_consolidation()
# β†’ {"abstractions_created": 12, "memories_consolidated": 847}
# Via API (admin endpoint)
POST /admin/consolidate
```
**Expected impact**: 70–90% memory footprint reduction, 10x query speedup at scale.
---
### 4. Auto-Associative Cleanup Loop *(Vector Immunology)*
**Problem**: Holographic vectors degrade over time through repeated XOR operations, noise accumulation, and long-term storage drift. After months of operation, retrieved vectors become "blurry" and similarity scores fall.
**Solution**: Iterative attractor dynamics β€” when a retrieved vector appears noisy, snap it to the nearest stable concept in a **codebook** of high-confidence prototypes:
```
noisy_vec β†’ find K nearest in codebook
β†’ majority_bundle(K neighbors)
β†’ check convergence (Hamming distance < 5%)
β†’ iterate until converged or max iterations reached
```
```python
# Cleanup runs automatically on retrieval when noise > 15%
node = engine.get_memory(memory_id, auto_cleanup=True)
# node.metadata["cleaned"] = True (if cleanup was triggered)
# node.metadata["cleanup_iterations"] = 3
# Codebook is auto-populated from most-accessed, high-reliability memories
```
**Expected impact**: Maintain >95% similarity fidelity even after years of operation.
---
### 5. Knowledge Gap Detection *(Proactive Curiosity)*
**Problem**: The system is entirely reactive β€” it answers queries but never identifies what it *doesn't know*. True cognitive autonomy requires self-directed learning.
**Solution**: Temporal co-occurrence analysis β€” detect concepts that are frequently accessed **close in time** but have **no synaptic connection**, flagging them as knowledge gaps:
```python
# Automatically runs hourly
gaps = engine.detect_knowledge_gaps(time_window_seconds=300)
# Returns structured insight:
# [
# {
# "concept_a": "Python asyncio event loop",
# "concept_b": "FastAPI dependency injection",
# "suggested_query": "How does asyncio relate to FastAPI dependency injection?",
# "co_occurrence_count": 4
# }
# ]
# Query endpoint
GET /insights/gaps?lookback_hours=24
# Fill gap manually (or via LLM agent)
POST /insights/fill-gap
{"concept_a_id": "mem_xxx", "concept_b_id": "mem_yyy",
"explanation": "FastAPI uses asyncio's event loop internally..."}
```
The system becomes capable of **saying what it doesn't understand** and requesting clarification β€” the first step toward genuine cognitive autonomy.
---
## Phase 4.4–4.5: Subconscious Daemon & LLM Integration
### Subconscious Daemon *(Autonomous Background Mind)*
Phase 4.4 introduced `SubconsciousAI` β€” a worker that fires during idle cycles and calls an external LLM to generate **synthetic dream memories**: structured insights derived by reasoning over existing memory clusters, rather than through direct observation.
Phase 4.5 hardened this into a full `SubconsciousDaemon` β€” an independently managed asyncio process that orchestrates dream cycles, consolidation scheduling, and subconscious queue processing:
```python
# The daemon is started automatically when the API starts up.
# It coordinates:
# - Dream synthesis: SubconsciousAI β†’ LLM β†’ synthetic insights stored back
# - Consolidation scheduling: ConsolidationWorker fired on a configurable interval
# - Subconscious queue: novelty detection from Redis pub/sub stream
```
Configure in `config.yaml`:
```yaml
haim:
subconscious_ai:
enabled: true
api_url: "https://api.openai.com/v1/chat/completions" # or Anthropic
model: "gpt-4o-mini"
# api_key: set via SUBCONSCIOUS_AI_API_KEY env var
dream_interval_seconds: 300
batch_size: 5
```
### Dependency Injection Container
All major services (TierManager, AsyncRedisStorage, QdrantStore, SubconsciousAI, etc.) are now wired through `src/mnemocore/core/container.py`. This eliminates global singleton state and makes every subsystem testable in isolation:
```python
from mnemocore.core.container import build_container
container = build_container(config)
engine = container.engine()
tier_mgr = container.tier_manager()
```
### LLM Agent Integration
`src/mnemocore/llm_integration.py` provides a high-level interface for attaching MnemoCore to any OpenAI/Anthropic-style LLM agent loop:
```python
from mnemocore.llm_integration import MnemoCoreAgent
agent = MnemoCoreAgent(engine)
# Store agent observations
agent.observe("User prefers concise answers over verbose ones")
# Recall relevant context before a response
context = agent.recall("user preference", top_k=3)
```
---
## API Reference (v2 compatibility)
This section documents the legacy HAIM REST service (`mnemocore.api.main:app`).
It remains available for v2 clients but its global JSONL/tiering lifecycle is
not AgentMemory-backed and must not be presented as v3 persistence. For v3,
use `create_v3_app(sqlite_path, scope_authorizer=...)`; the authorizer must
approve the caller's complete requested scope before each operation.
### Authentication
All endpoints require an API key via the `X-API-Key` header:
```bash
export HAIM_API_KEY="your-secure-key"
curl -H "X-API-Key: $HAIM_API_KEY" ...
```
### Endpoints
#### `POST /store`
Store a new memory with optional context binding.
```json
Request:
{
"content": "FastAPI uses Pydantic v2 for request validation.",
"metadata": {"source": "docs", "tags": ["python", "fastapi"]},
"context": "ProjectAlpha",
"agent_id": "agent-001",
"ttl": 3600
}
Response:
{
"ok": true,
"memory_id": "mem_1739821234567",
"message": "Stored memory: mem_1739821234567"
}
```
#### `POST /query`
Query memories by semantic similarity with optional context masking.
```json
Request:
{
"query": "How does FastAPI handle request validation?",
"top_k": 5,
"context": "ProjectAlpha"
}
Response:
{
"ok": true,
"query": "How does FastAPI handle request validation?",
"results": [
{
"id": "mem_1739821234567",
"content": "FastAPI uses Pydantic v2 for request validation.",
"score": 0.8923,
"metadata": {"source": "docs"},
"tier": "hot"
}
]
}
```
#### `POST /feedback`
Report outcome of a retrieved memory (Phase 4.0 reliability loop).
```json
Request:
{
"memory_id": "mem_1739821234567",
"outcome": true,
"comment": "This solution worked perfectly."
}
Response:
{
"ok": true,
"memory_id": "mem_1739821234567",
"reliability_score": 0.714,
"success_count": 4,
"failure_count": 1
}
```
#### `GET /memory/{memory_id}`
Retrieve a specific memory with full metadata.
```json
Response:
{
"id": "mem_1739821234567",
"content": "...",
"metadata": {...},
"created_at": "2026-02-17T20:00:00Z",
"ltp_strength": 1.847,
"epistemic_value": 0.73,
"reliability_score": 0.714,
"tier": "hot"
}
```
#### `DELETE /memory/{memory_id}`
Delete memory from all tiers and clean up synapses.
#### `POST /concept`
Define a symbolic concept for analogical reasoning.
```json
{"name": "king", "attributes": {"gender": "man", "role": "ruler", "domain": "royalty"}}
```
#### `POST /analogy`
Solve analogies using VSA algebra: `source:value :: target:?`
```json
Request: {"source_concept": "king", "source_value": "man", "target_concept": "queen"}
Response: {"results": [{"value": "woman", "score": 0.934}]}
```
#### `GET /insights/gaps`
Detect knowledge gaps from recent temporal co-activity (Phase 4.0).
```json
Response:
{
"gaps_detected": 3,
"knowledge_gaps": [
{
"concept_a": "asyncio event loop",
"concept_b": "FastAPI middleware",
"suggested_query": "How does event loop relate to middleware?",
"co_occurrence_count": 5
}
]
}
```
#### `POST /admin/consolidate`
Trigger manual semantic consolidation (normally runs automatically at 3 AM).
#### `GET /stats`
Engine statistics - tiers, synapse count, consolidation state. Requires `X-API-Key` by default.
#### `GET /health`
Public liveness and dependency diagnostics. A local-only runtime can report
`degraded` when Redis is unavailable while remaining live.
#### `GET /ready`
Readiness of the initialized local runtime. Redis is reported diagnostically
but is not required for the single-node prototype to become ready.
#### `GET /metrics/`
Prometheus metrics endpoint on the API listener (port 8100). The trailing slash
is canonical.
---
## Python Library Usage
For durable SQLite-backed memory with exact agent scopes, async/sync APIs,
bitemporal recall, supersession, and evidence receipts, see the
[Persistent Agent Memory Quickstart](docs/AGENT_MEMORY_QUICKSTART.md). Retrieval
in this foundation is SQLite FTS5 lexical search only; it is not semantic or
embedding retrieval.
### v3 scoped runtime
```python
from mnemocore.agent_memory import AgentMemory, MemoryScope
from mnemocore.hybrid import HybridMemoryRuntime
import asyncio
async def main():
scope = MemoryScope(tenant_id="local", user_id="demo", agent_id="assistant")
async with await AgentMemory.open("memory.db", scope=scope) as memory:
runtime = HybridMemoryRuntime(memory, scope=scope)
await memory.remember("User prefers concise answers")
results = await runtime.recall(scope, "concise", limit=3)
print(results)
asyncio.run(main())
```
In v3, `LiteEngine` and the entire unscoped `Memory(...)` facade are removed and fail
with a migration error; use `AgentMemory` with an explicit `MemoryScope`.
Every runtime operation uses the complete scope tuple; there is no broader
scope fallback. `HAIMEngine` remains available only for v2 compatibility: its
global JSONL/tiering lifecycle is not AgentMemory-backed and must not be used
as v3 persistence. Use `HybridMemoryRuntime` (or `create_v3_app` for HTTP)
for v3.
For a temporary bridge of its legacy `store`/`query`/`delete_memory` shape, use
the deprecated `HAIMEngineAdapter` with an already-open, scope-bound
`AgentMemory` client.
The hybrid runtime also has an explicit synchronous facade for synchronous
applications. Do not call it from an already-running event loop; use the async
runtime there. See the [v3 hybrid-runtime guide](docs/V3_HYBRID_RUNTIME.md) for
retrieval metadata, cognitive-plan safeguards, API composition, and the staged
remaining work.
### Advanced / Low-level (v2 compatibility only)
`HAIMEngine` below is retained for existing v2 integrations. It is deprecated
for new work, does not enforce `MemoryScope`, and is not the v3 persistence
layer. New v3 code must use the scoped runtime shown above.
```python
from mnemocore.core.engine import HAIMEngine
engine = HAIMEngine(persist_path="./data/memory.jsonl")
# Store memories
engine.store("Python generators are lazy iterators", metadata={"topic": "python"})
engine.store("Use 'yield' to create generator functions", metadata={"topic": "python"})
engine.store("Redis XADD appends to a stream", goal_id="infrastructure")
# Query (global)
results = engine.query("How do Python generators work?", top_k=3)
for mem_id, score in results:
mem = engine.get_memory(mem_id)
print(f"[{score:.3f}] {mem.content}")
# Query with context masking
results = engine.query("data streams", top_k=5, context="infrastructure")
engine.close()
```
### Analogical Reasoning
```python
# Define concepts
engine.define_concept("king", {"gender": "man", "role": "ruler"})
engine.define_concept("queen", {"gender": "woman", "role": "ruler"})
engine.define_concept("man", {"gender": "man"})
# VSA analogy: king:man :: ?:woman β†’ queen
result = engine.reason_by_analogy(
src="king", val="man", tgt="woman"
)
print(result) # [("queen", 0.934), ...]
```
### Working with the Binary HDV Layer Directly
```python
from mnemocore.core.binary_hdv import BinaryHDV, TextEncoder, majority_bundle
encoder = TextEncoder(dimension=16384)
# Encode text
python_vec = encoder.encode("Python programming")
fastapi_vec = encoder.encode("FastAPI framework")
error_vec = encoder.encode("error handling")
# Bind concept to role
python_in_fastapi = python_vec.xor_bind(fastapi_vec)
# Bundle multiple concepts into prototype
web_dev_prototype = majority_bundle([python_vec, fastapi_vec, error_vec])
# Similarity
print(python_vec.similarity(web_dev_prototype)) # High (part of bundle)
print(python_vec.similarity(error_vec)) # ~0.5 (unrelated)
# Batch nearest-neighbor search
from mnemocore.core.binary_hdv import batch_hamming_distance
import numpy as np
database = np.stack([v.data for v in [python_vec, fastapi_vec, error_vec]])
distances = batch_hamming_distance(python_vec, database)
```
### Reliability Feedback Loop
```python
mem_id = engine.store("Always use asyncio.Lock() in async code, not threading.Lock()")
results = engine.query("async locking")
# It works β€” report success
engine.provide_feedback(mem_id, outcome=True, comment="Solved deadlock issue")
# Over time, high-reliability memories get 'verified' tag
# and are ranked above unproven ones in future queries
```
### Semantic Consolidation
```python
stats = engine.trigger_consolidation()
print(f"Created {stats['abstractions_created']} semantic anchors")
print(f"Consolidated {stats['memories_consolidated']} episodic memories")
# Automatic: runs every night at 3 AM via background asyncio task
```
---
## Installation
### Prerequisites
- **Python 3.10+**
- Redis and Qdrant are optional runtime services for legacy tiering and vector
integrations. They are not required for local AgentMemory.
- The current 2.x wheel retains its compatibility dependencies. It does not
connect to Redis or Qdrant merely by importing `mnemocore.agent_memory`.
### Quick Start (AgentMemory β€” stable local-first path)
```bash
# 1. Install. No Redis/Qdrant service is needed for AgentMemory.
pip install -e .
```
```python
# 2. Use AgentMemory with an explicit scope and local SQLite store.
from pathlib import Path
from mnemocore.agent_memory import MemoryScope, SyncAgentMemory
scope = MemoryScope(tenant_id="local", user_id="robin", agent_id="assistant")
memory = SyncAgentMemory.open(Path("agent-memory.sqlite3"), scope=scope)
memory.remember("User prefers concise answers")
print(memory.recall("concise"))
```
See "Full server setup (optional)" below for HAIM, Redis, Docker, and the REST API.
### Full server setup (optional, for Redis-backed etc.)
# To include dev tools (pytest, mypy, black, etc.):
pip install -e ".[dev]"
# Start Redis
docker run -d -p 6379:6379 redis:7.2-alpine
# Set API key (never hardcode β€” use env var or .env file)
# Windows PowerShell:
$env:HAIM_API_KEY = "your-secure-key-here"
# Linux / macOS:
# export HAIM_API_KEY="your-secure-key-here"
# Start the API locally
# PowerShell; use export MNEMOCORE_HOST=127.0.0.1 on Unix
$env:MNEMOCORE_HOST = "127.0.0.1"
mnemocore
The API is now live at `http://localhost:8100`. Visit `http://localhost:8100/docs` for the interactive Swagger UI.
> Do not bind MnemoCore directly to a public IP unless it is behind a trusted reverse proxy, TLS, and a strong `HAIM_API_KEY`. Diagnostics (`/stats`, `/rate-limits`, `/metrics`) require `X-API-Key` by default.
### Using the .env file
Copy the provided template and fill in your values β€” the API and docker-compose both pick it up automatically:
```bash
cp .env.example .env
# Edit .env and set HAIM_API_KEY, REDIS_PASSWORD, QDRANT_API_KEY, etc.
```
> **Note:** `.env` is listed in `.gitignore` and must never be committed. Only `.env.example` (with placeholder values) belongs in version control.
### Full Stack with Docker Compose
```bash
# Requires .env with HAIM_API_KEY, REDIS_PASSWORD, and QDRANT_API_KEY set.
docker compose config
docker compose up -d
```
This starts MnemoCore, Redis 7.2, and Qdrant. Only the API is published, on
`127.0.0.1:8100`; metrics are served from `/metrics/` on the same port.
### With Qdrant (Phase 4.x Scale)
```bash
# Start Qdrant alongside Redis
docker run -d -p 6333:6333 qdrant/qdrant
# Enable in config.yaml
qdrant:
enabled: true
host: localhost
port: 6333
```
---
## Configuration
All configuration lives in `config.yaml`. Sensitive values can be overridden with environment variables β€” the config loader looks for `HAIM_`-prefixed vars and also honours per-service overrides like `HAIM_API_KEY`, `REDIS_PASSWORD`, `QDRANT_API_KEY`, `HAIM_CORS_ORIGINS`, and `SUBCONSCIOUS_AI_API_KEY`.
```yaml
haim:
version: "5.0"
dimensionality: 16384 # Binary vector dimensions (must be multiple of 64)
encoding:
mode: "binary" # "binary" (recommended) or "float" (legacy, deprecated)
token_method: "bundle" # "bundle" (XOR+permute) or "hash"
tiers:
hot:
max_memories: 2000 # Max nodes in RAM
ltp_threshold_min: 0.7 # Evict below this LTP strength
eviction_policy: "lru"
warm:
max_memories: 100000 # Max nodes in Redis/mmap
ltp_threshold_min: 0.3
cold:
storage_backend: "filesystem" # "filesystem" or "s3"
compression: "gzip"
ltp:
initial_importance: 0.5
decay_lambda: 0.01 # Higher = faster forgetting
permanence_threshold: 0.95 # LTP above this is immune to decay
half_life_days: 30.0
hysteresis:
promote_delta: 0.15 # LTP must exceed threshold by this much to promote
demote_delta: 0.10
redis:
url: "redis://localhost:6379/0"
stream_key: "haim:subconscious"
max_connections: 10
socket_timeout: 5
# password: set via REDIS_PASSWORD env var
qdrant:
url: "http://localhost:6333"
collection_hot: "haim_hot"
collection_warm: "haim_warm"
enabled: false
# api_key: set via QDRANT_API_KEY env var
security:
# api_key: set via HAIM_API_KEY env var β€” never hardcode here
cors_origins: ["http://localhost:3000"]
public_health_enabled: true
public_stats_enabled: false
public_rate_limits_enabled: false
public_metrics_enabled: false
require_api_key_for_export: true
subconscious_ai:
enabled: false
api_url: "https://api.openai.com/v1/chat/completions"
model: "gpt-4o-mini"
dream_interval_seconds: 300
batch_size: 5
# api_key: set via SUBCONSCIOUS_AI_API_KEY env var
observability:
metrics_port: 8100
log_level: "INFO"
structured_logging: true
paths:
data_dir: "./data"
memory_file: "./data/memory.jsonl"
codebook_file: "./data/codebook.json"
concepts_file: "./data/concepts.json"
synapses_file: "./data/synapses.json"
warm_mmap_dir: "./data/warm_tier"
cold_archive_dir: "./data/cold_archive"
mcp:
enabled: false
transport: "stdio"
host: "127.0.0.1"
port: 8110
api_base_url: "http://localhost:8100"
```
### Security Note
MnemoCore requires an explicit API key. There is no default fallback key in production builds.
```bash
# Generate a cryptographically secure key:
python -c "import secrets; print(secrets.token_urlsafe(32))"
# Set it (never commit this value):
export HAIM_API_KEY="<generated-value>"
```
Persistent webhooks store only an opaque `secret_ref`; the signing secret is
resolved when a delivery is sent. Inline signing secrets, legacy plaintext
records, and all custom headers are rejected for persistent webhook managers.
Writes are serialized and atomically replaced, and malformed persistence loads
fail closed without partial in-memory state. Memory-only webhook managers retain
the legacy inline-secret/header behavior for compatibility.
AgentMemory also exposes a physical SQLite erase operation with exact-scope
ownership checks, optional whole-supersession-component cascade, dependent-row
cleanup, database integrity checks, and a content-free `ErasureReceipt`. All
database users must cooperate with the store's sidecar lock; raw SQLite clients
are outside the safety contract during erasure. Backups, derived external
artifacts, power-loss recovery, and exhaustive failure injection remain separate
production gates.
---
## MCP Server Integration (v2 compatibility)
MnemoCore exposes a legacy **Model Context Protocol (MCP)** server for existing
v2 integrations. It proxies the legacy HAIM REST API and is not backed by v3
AgentMemory truth. New scoped integrations should use the AgentMemory MCP
helpers in `mnemocore.integrations.mcp` (or build on `HybridMemoryRuntime`).
### Setup
```bash
# Start the legacy v2 API first
uvicorn mnemocore.api.main:app --host 0.0.0.0 --port 8100
# Configure MCP in config.yaml
haim:
mcp:
enabled: true
transport: "stdio" # or "sse" for streaming
# Run MCP server
python -m mnemocore.mcp.server
```
### Claude Desktop Configuration
Add to your Claude Desktop `config.json`:
```json
{
"mcpServers": {
"mnemocore": {
"command": "python",
"args": ["-m", "mnemocore.mcp.server"],
"env": {
"HAIM_API_KEY": "your-key",
"HAIM_BASE_URL": "http://localhost:8100"
}
}
}
}
```
Once connected, the agent can:
- `store_memory(content, context)` β€” persist learned information
- `query_memory(query, context, top_k)` β€” recall relevant memories
- `provide_feedback(memory_id, outcome)` β€” signal what worked
- `get_knowledge_gaps()` β€” surface what it doesn't understand
---
## Observability
MnemoCore ships with built-in Prometheus metrics and structured logging.
### Prometheus Metrics
Available at canonical `GET /metrics/` on the API listener, port 8100:
| Metric | Description |
|--------|-------------|
| `haim_api_request_count` | Total requests by endpoint and status |
| `haim_api_request_latency_seconds` | Request latency histogram |
| `haim_storage_operation_count` | Store/query/delete operations |
| `haim_hot_tier_size` | Current HOT tier memory count |
| `haim_synapse_count` | Active synaptic connections |
### Grafana Dashboard
A sample Grafana dashboard config is available at `grafana-dashboard.json` in the repository root. Import it directly into Grafana via **Dashboards β†’ Import β†’ Upload JSON file**.
### Structured Logging
All components use structured Python logging with contextual fields:
```
2026-02-17 20:00:00 INFO Stored memory mem_1739821234567 (EIG: 0.7823)
2026-02-17 20:00:01 INFO Memory mem_1739821234567 reliability updated: 0.714 (4βœ“ / 1βœ—)
2026-02-17 03:00:00 INFO Consolidation complete: abstractions_created=12, consolidated=847
2026-02-17 04:00:00 INFO Knowledge gap detected: asyncio ↔ FastAPI middleware (5 co-occurrences)
```
---
## Testing
```bash
# Run full test suite
pytest
# Run with coverage
pytest --cov=src --cov-report=html
# Run specific feature tests
pytest tests/test_xor_attention.py # Contextual masking
pytest tests/test_stability.py # Reliability/Bayesian stability
pytest tests/test_consolidation.py # Semantic consolidation
pytest tests/test_engine_cleanup.py # Cleanup and decay
pytest tests/test_phase43_regressions.py # Phase 4.3 regression guardrails
pytest tests/test_tier_manager.py # Tier demotion / promotion logic
pytest tests/test_dream_loop.py # Subconscious dream loop
pytest tests/test_subconscious_ai_worker.py # LLM-powered dream worker (if offline: uses mocks)
pytest tests/test_recursive_synthesizer.py # Deep concept synthesis
pytest tests/test_batch_ops.py # Bulk ingestion operations
pytest tests/test_mcp_server.py # MCP server adapter
# End-to-end flow
pytest tests/test_e2e_flow.py -v
```
---
## Roadmap
### v3.0.0 beta: released surface
- [x] Exact-scope AgentMemory persistence, history, timeline, and context
compilation
- [x] Hybrid lexical/BinaryHDV retrieval with versioned deterministic scoring
- [x] Rebuildable tier and graph projections
- [x] Validated, atomic cognitive plan application with provenance
- [x] Scoped HTTP composition that fails closed without authorization
- [x] Explicit migration away from LiteEngine and implicit `Memory(...)`
### v2 compatibility inventory
- [x] Binary HDV core (XOR bind / bundle / permute / Hamming)
- [x] Three-tier HOT/WARM/COLD memory lifecycle
- [x] Async API + MCP integration
- [x] XOR attention masking + Bayesian reliability updates
- [x] Semantic consolidation, immunology cleanup, and gap detection/filling
- [x] Temporal recall: episodic chaining + chrono-weighted query
- [x] Phase 4.4 β€” Subconscious AI Worker (LLM-powered dream synthesis)
- [x] Phase 4.5 β€” Subconscious Daemon, persistence hardening, tier-manager demotion race fix
- [x] Dependency-injection Container pattern (replaces singleton)
- [x] HNSW in-process index for hot-tier ANN search
- [x] Batch operations for bulk ingestion
- [x] Meta-cognition layer: GoalTree + LearningJournal
- [x] Phase 5 β€” Cognitive Architecture: Working Memory, Episodic/Semantic/Procedural stores, Meta Memory, Self-Improvement Worker, 7-phase Pulse loop
- [x] Phase 5 Hardening β€” 136 additional tests, config organization
- [x] Phase 6 β€” Research Services: StrategyBank, KnowledgeGraph, MemoryScheduler, SAMEP, 4 new Pulse phases
- [x] 1291+ tests passing
### Next Steps
- [ ] Production hardening: distributed HOT-tier, CUDA kernels, chaos engineering
- [ ] Self-improvement Phase 1: enable writes with human-in-the-loop approval
- [ ] Multi-modal memory: image/audio embedding via CLIP/Whisper
- [ ] Additional first-party integration adapters
- [ ] Extended observability standardization (`mnemocore_*` metric prefix)
---
## Contributing
MnemoCore is an active research project. Contributions are welcome β€” especially:
- **Performance**: CUDA kernels, FAISS integration, async refactoring
- **Algorithms**: Better clustering for consolidation, improved EIG formulas
- **Integrations**: New storage backends, LLM connectors
- **Tests**: Coverage for edge cases, property-based testing
### Process
```bash
# Fork and clone
git checkout -b feature/your-feature-name
# Make changes, ensure tests pass
pytest
# Commit with semantic message
git commit -m "feat(consolidation): add LLM-powered prototype labeling"
# Open PR β€” describe the what, why, and performance impact
```
Please follow the implementation patterns established in `docs/ARCHITECTURE.md` and `docs/ROADMAP.md` for architectural guidance, and review `CHANGELOG.md` to understand what has already landed.
---
## Documentation
| Document | Description |
|----------|-------------|
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | System architecture with Mermaid diagrams |
| [docs/API.md](docs/API.md) | Complete REST API reference (41 endpoints) |
| [docs/CLI.md](docs/CLI.md) | Command-line interface reference |
| [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | All configuration options (41 dataclasses) |
| [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Docker, Compose, and Kubernetes deployment |
| [docs/EVENTS.md](docs/EVENTS.md) | Event system and webhook delivery |
| [docs/FORGETTING.md](docs/FORGETTING.md) | SM-2 spaced repetition and decay system |
| [docs/LLM_INTEGRATION.md](docs/LLM_INTEGRATION.md) | Multi-provider LLM integration |
| [docs/GLOSSARY.md](docs/GLOSSARY.md) | Terms and abbreviations |
| [docs/SUBCONSCIOUS_AI.md](docs/SUBCONSCIOUS_AI.md) | Subconscious worker and dream pipeline |
| [docs/RECONSTRUCTIVE_MEMORY.md](docs/RECONSTRUCTIVE_MEMORY.md) | Reconstructive recall module |
| [docs/EMBEDDING_REGISTRY.md](docs/EMBEDDING_REGISTRY.md) | Embedding model versioning |
| [CHANGELOG.md](CHANGELOG.md) | Release history |
| [CONTRIBUTING.md](CONTRIBUTING.md) | Contribution guidelines |
---
## License
MIT License β€” see [LICENSE](LICENSE) for details.
---
## Contact
**Robin Granberg**
πŸ“§ robin@veristatesystems.com
---
<p align="center">
<i>Building the cognitive substrate for the next generation of autonomous AI.</i>
</p>