Agent.md

# Agent Deployment Guide for Hugging Face Space

This document serves as the guide and best practices for automated agents deploying and maintaining BrainAPI on Hugging Face Spaces.

## 1. Deployment Configuration

### Target Space
- **Profile:** `Leon4gr45`
- **Space:** `brain`
- **Full Identifier:** `Leon4gr45/brain`
- **Frontend Port:** `7860` (mandatory for Hugging Face Spaces)

### Deployment Method
- **Docker SDK** (`Dockerfile`)

### HF Token
- Always pass/read the HF access token from the environment (`HF_TOKEN` environment variable). Never hardcode secrets or token strings in repository files.

### Required Files
- `Dockerfile` (exposing nginx on port `7860`; uvicorn stays internal on port `8000`)
- `README.md` with Hugging Face YAML frontmatter:
  ```yaml
  ---
  title: BrainAPI
  sdk: docker
  app_port: 7860
  ---
  ```
- `.hfignore` to exclude `.git`, `.venv`, `node_modules`, and cache directories
- `Agent.md` (this file)

---

## 2. API Exposure and Documentation

### Mandatory Endpoints

- **`/health`**
  - Method: GET
  - Purpose: Returns HTTP 200 when the app is ready. Required for Hugging Face Space liveness.
  - Request: None
  - Response:
    ```json
    {
      "status": "ok",
      "service": "brainapi",
      "version": "2.19.0-dev"
    }
    ```

- **`/api-docs`**
  - Method: GET
  - Purpose: Redirects/Exposes full OpenAPI / Swagger documentation for all endpoints. Reachable at `https://Leon4gr45-brain.hf.space/api-docs`.
  - Response: HTTP 307 redirect to `/docs` or Swagger UI HTML page.

- **`/`**
  - Method: GET
  - Purpose: Redirects public root visits to `/console/`.
  - Response: HTTP 307 redirect to `/console/`.

---

## 3. Deployment Workflow & Ongoing Best Practices

1. **Pre-flight Check:**
   Clean up non-project or stale files in the target Hugging Face Space using `huggingface_hub` Python API.

2. **Deploy Command:**
   ```bash
   python deploy/hf_space.py stage --output /tmp/brainapi-hf-source
   # Inspect/build the staged tree before uploading that same Git revision.
   python deploy/hf_space.py upload --revision HEAD
   ```

   This stages the complete committed Git tree, checks every required Docker
   input, and verifies the uploaded Space files against the staged content.
   Run it from the BrainAPI checkout with `huggingface_hub` installed and
   `HF_TOKEN` set through secure environment settings. Do not upload a manually
   selected list of changed files. The readiness wrapper must remain in the
   Space repository alongside `scripts/preload_ollama_models.sh`.

3. **Log Monitoring:**
   - **Build Logs (SSE):**
     ```bash
     curl -N -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/api/spaces/Leon4gr45/brain/logs/build"
     ```
   - **Run Logs (SSE):**
     ```bash
     curl -N -H "Authorization: Bearer $HF_TOKEN" "https://huggingface.co/api/spaces/Leon4gr45/brain/logs/run"
     ```

4. **Iterative Debugging:**
   Inspect build and run logs for missing dependencies, port binding issues, or startup exceptions. Modify codebase accordingly, redeploy, and monitor until the Space reaches `RUNNING` status and responds to `/health` and `/api-docs`.

### Functional Endpoints

### /retrieve/context
- **Method:** POST
- **Purpose:** Retrieve an entity's contextual information based on a given text, generating triples and passages.
- **Authentication:** Requires `BrainPAT` header or Bearer token verified by the central AuthService.
- **Request (JSON):**
  - `text` (str, required): The text to search context for.
  - `brain_id` (str, optional, default: `"default"`): The brain/workspace identifier to query.
  - `historical_limit` (int, optional, default: `10`)
  - `max_facts` (int, optional, default: `40`, min: `0`)
  - `max_passages` (int, optional, default: `8`)
  - `apply_fact_filter` (bool, optional, default: `true`)
  - `use_ppr` (bool, optional, default: `true`)
  - `sufficiency_retry` (bool, optional, default: `false`)
  - `profile_stages` (bool, optional, default: `false`)
  - `cross_event_bridges` (int, optional, default: `3`, min: `0`)
  ```json
  {
    "text": "Identify memory leaks in Python",
    "brain_id": "default",
    "historical_limit": 10,
    "max_facts": 40,
    "max_passages": 8,
    "apply_fact_filter": true,
    "use_ppr": true,
    "sufficiency_retry": false,
    "profile_stages": false,
    "cross_event_bridges": 3
  }
  ```
- **Response (JSON):**
  ```json
  {
    "text_context": "Found 3 passages and 10 triples related to the query...",
    "triples": [
      {
        "identified_entity": "memory_leak",
        "triple": [
          {"id": "python", "label": "Language", "properties": {"name": "Python"}},
          {"id": "causes", "type": "CAUSES", "properties": {}},
          {"id": "memory_leak", "label": "Issue", "properties": {"name": "Memory Leak"}}
        ],
        "source_chunk_ids": ["chunk-123"]
      }
    ],
    "historical_context": ["Previous discussions about memory profiling..."],
    "source_passages": ["Python's garbage collector sometimes misses reference cycles..."],
    "graph_session_ids": ["sess-456"],
    "temporal_conflicts": [],
    "paths": [],
    "topics": [{"topic": "performance", "weight": 0.8}],
    "stage_timings": {"retrieval": 0.15, "formatting": 0.02}
  }
  ```

### /retrieve/search
- **Method:** POST (Canonical) / GET (Compatibility)
- **Purpose:** Execute a ranked search across passages, entities, events, communities, and plugins with optional re-ranking.
- **Authentication:** Requires `BrainPAT` header or Bearer token verified by the central AuthService.
- **Request (POST JSON or GET Query Params):**
  - `query` (str, required): The search query.
  - `brain_id` (str, optional, default: `"default"`): The brain identifier to query.
  - `k` (int, optional, default: `10`, range: `1-200`): Number of hits to return.
  - `channels` (List[str], optional, default: `["passages"]`): Channels to search. Allowed channels include `passages`, `entities`, `events`, `communities`, and/or `plugin:<name>`.
  - `node_labels` (List[str], optional, default: `null`): Node labels to filter the entities channel.
  - `community_labels` (List[str], optional, default: `null`): Hub labels for the communities channel.
  - `expand` (str, optional, default: `"none"`): Allowed values: `"none"`, `"neighbors"`. Optional 1-hop expansion from graph channel seeds.
  - `fusion` (str, optional, default: `null`): Fusion override. Allowed values: `"rrf"`, `"cc"`.
  - `fusion_alpha` (float, optional, default: `null`, range: `0.0-1.0`)
  - `rerank` (str, optional, default: `null`): `none` or `plugin:<name>`. Unknown plugin names return 400.
  - `mode` (str, optional, default: `"default"`): Allowed values: `"default"`, `"catalog"`.
  - `profile_stages` (bool, optional, default: `false`)
  - `extras` (Dict[str, str], optional, default: `null`)
  - `target` (str, optional, default: `null`): Optional USER uuid or id for query-gated rerank of retrieved hits.
  ```json
  {
    "query": "hello world",
    "brain_id": "default",
    "k": 10,
    "channels": ["passages", "entities"],
    "node_labels": ["Document", "Author"],
    "community_labels": ["Tech"],
    "expand": "none",
    "fusion": "rrf",
    "fusion_alpha": 0.5,
    "rerank": "plugin:cohere",
    "mode": "default",
    "profile_stages": false,
    "extras": {},
    "target": "user-123"
  }
  ```
- **Response (JSON):**
  ```json
  {
    "hits": [
      {
        "id": "hit-789",
        "channel": "passages",
        "score": 0.95,
        "scores": {"dense": 0.95, "bm25": 4.2, "rrf": 0.033},
        "snippet": "Hello World example in Python",
        "extras": {"author": "Jane Doe"}
      }
    ],
    "stage_timings": {"search": 0.08, "rerank": 0.12},
    "channel_lists": {"passages": ["hit-789"]},
    "facets": {"author": {"Jane Doe": 1}},
    "node_ids": ["node-456"]
  }
  ```
## HF CPU embedding contract

Use `embeddinggemma2-text-v1`: Ollama `embeddinggemma-2:270m`, 8192-token maximum,
768 native dimensions -> first 512 -> L2 normalization. All five primary pgvector
stores must be `vector(512)`. Never mix model identities even at equal dimensions;
index/profile mismatches must fail without dropping or reusing old vectors.
The baseline image preloads only Gemma; chat and reranking remain optional.
See `deploy/env.hf.cpu.example` and `deploy/README.ollama.md`. Before production,
commit and stage the exact source, build that staged tree, and run
`/app/entrypoint.sh --qualify` inside the local image (it runs the qualification
script with the service configuration). Do not upload a
production deployment until both the full build and real qualification pass.

## 4. Tricks and Ongoing Deployment Best Practices

### Custom Deployment Script
- Deployments to Hugging Face Spaces must use the custom `deploy/hf_space.py` complete-tree helper script rather than a plain `hf upload`. Use `python deploy/hf_space.py stage --revision HEAD --output /tmp/brainapi-hf-source` followed by `python deploy/hf_space.py upload --revision HEAD` to ensure the staged tree correctly matches the committed revision.

### Environment Variable Normalization
- String environment variables mapped into configuration classes (e.g., `provider`, `model`, `profile` in `src/config.py`) must be explicitly normalized early using `.strip()` and `.lower()` at the boundary to prevent invisible whitespace or case mismatch validation errors.
- When logging environment variable diagnostics in shell scripts like `entrypoint.sh`, place `echo` and `printf` statements after the fallback default evaluation variables are evaluated (e.g., `export VAR="${VAR:-default}"`) to ensure the effective runtime configurations are captured.

### Docker and Hub Constraints
- If a Hugging Face Space configured with the Docker SDK reports 'missing an app file', it means the `Dockerfile` is missing from the remote repository and must be explicitly uploaded alongside the `README.md` to trigger the build.
- The `deploy/` directory must not be excluded in `.dockerignore` so that Supervisord and Nginx configurations can be copied during Docker image builds.
- `.dockerignore` must exclude `node_modules/`, `console/node_modules/`, and `tui/node_modules/` to prevent copying local node dependencies into Docker build stages.
- The `README.md` requires Hugging Face YAML frontmatter at the very top of the file including `title: BrainAPI`, `emoji`, `colorFrom`, `colorTo`, `sdk: docker`, `app_port: 7860`, and `pinned` configurations to satisfy constraints.
- Do not hardcode Hugging Face access token strings in repository files like `Agent.md`, as HF Hub security filters reject commits containing raw tokens.

### Docker Startup & Service Coordination
- In Supervisord process definitions, the Celery worker application module is `src.workers.app:ingestion_app` and the MCP ASGI server module is `src.services.mcp.app:app`.
- For Hugging Face Spaces compatibility, the API is configured in `deploy/supervisord.conf` under `[program:brainapi-api]` to run Uvicorn on `0.0.0.0` port `7860` natively (or handled by an Nginx reverse proxy on 7860 to internal 8000).
- Setting `serverSelectionTimeoutMS` (e.g. 2000ms) on `PyMongoClient` prevents 30-second blocking timeouts when MongoDB is offline during startup.
- The PostgreSQL extension `vector` is pre-created in `template1` as superuser `postgres` in `entrypoint.sh` so that newly initialized workspace databases automatically inherit `pgvector` without requiring `SUPERUSER` privileges for user `brainapi`.
- In all-in-one container deployments, Supervisord orchestrates local PostgreSQL (with `pgvector`), Redis, FastAPI API, MCP server, Celery worker, and Nginx, using persistent secrets stored in `/data/brainapi/internal-secrets.env`.
- In standalone deployment container environments, setting `PIPELINE_MODE=lightweight` and `EMBEDDINGS_PROVIDER=mock` or `fastembed` avoids hard runtime dependencies on external LLM services or Ollama instances.

### Security
- `SYSTEM_PAT` must be provided as an environment variable or Hugging Face Space Secret and never hardcoded in repository files or test scripts.
- Public UI and documentation endpoints (`/`, `/console`, `/docs`, `/redoc`, `/health`, `/openapi.json`, `/api-docs`, `/favicon.ico`) must be excluded from `BrainPATMiddleware` and `BrainMiddleware` authentication checks. `GET /` redirects to `/console/`.

### Testing and Documentation Rules
- Functional API endpoints (e.g., `/retrieve/context`, `/retrieve/search`) must be explicitly documented in `Agent.md` under the `### Functional Endpoints` section, including their Method, Path, Purpose, Request, and Response structures.
- API endpoints documented in `Agent.md` should strictly match the actual implementation (route decorators and Pydantic models). The canonical HTTP method (e.g., POST) should be presented as the primary interface, without modifying API behavior just to match docs.
- The local test suite must be executed using `PYTHONPATH=tests poetry run pytest` to resolve import errors for test helper modules.
- `Config` in `src/config.py` conditionally instantiates database sub-configurations (`mongo`, `milvus`, `neo4j`, `postgresql`) based on `DATA_DB`, `VECTOR_DB`, and `GRAPH_DB` environment variables.

## 4. Tricks and Ongoing Deployment Best Practices

### Custom Deployment Script
- Deployments to Hugging Face Spaces must use the custom `deploy/hf_space.py` complete-tree helper script rather than a plain `hf upload`. Use `python deploy/hf_space.py stage --revision HEAD --output /tmp/brainapi-hf-source` followed by `python deploy/hf_space.py upload --revision HEAD` to ensure the staged tree correctly matches the committed revision.

### Environment Variable Normalization
- String environment variables mapped into configuration classes (e.g., `provider`, `model`, `profile` in `src/config.py`) must be explicitly normalized early using `.strip()` and `.lower()` at the boundary to prevent invisible whitespace or case mismatch validation errors.
- When logging environment variable diagnostics in shell scripts like `entrypoint.sh`, place `echo` and `printf` statements after the fallback default evaluation variables are evaluated (e.g., `export VAR="${VAR:-default}"`) to ensure the effective runtime configurations are captured.

### Docker and Hub Constraints
- If a Hugging Face Space configured with the Docker SDK reports 'missing an app file', it means the `Dockerfile` is missing from the remote repository and must be explicitly uploaded alongside the `README.md` to trigger the build.
- The `deploy/` directory must not be excluded in `.dockerignore` so that Supervisord and Nginx configurations can be copied during Docker image builds.
- `.dockerignore` must exclude `node_modules/`, `console/node_modules/`, and `tui/node_modules/` to prevent copying local node dependencies into Docker build stages.
- The `README.md` requires Hugging Face YAML frontmatter at the very top of the file including `title: BrainAPI`, `emoji`, `colorFrom`, `colorTo`, `sdk: docker`, `app_port: 7860`, and `pinned` configurations to satisfy constraints.
- Do not hardcode Hugging Face access token strings in repository files like `Agent.md`, as HF Hub security filters reject commits containing raw tokens.

### Docker Startup & Service Coordination
- In Supervisord process definitions, the Celery worker application module is `src.workers.app:ingestion_app` and the MCP ASGI server module is `src.services.mcp.app:app`.
- For Hugging Face Spaces compatibility, the API is configured in `deploy/supervisord.conf` under `[program:brainapi-api]` to run Uvicorn on `0.0.0.0` port `7860` natively (or handled by an Nginx reverse proxy on 7860 to internal 8000).
- Setting `serverSelectionTimeoutMS` (e.g. 2000ms) on `PyMongoClient` prevents 30-second blocking timeouts when MongoDB is offline during startup.
- The PostgreSQL extension `vector` is pre-created in `template1` as superuser `postgres` in `entrypoint.sh` so that newly initialized workspace databases automatically inherit `pgvector` without requiring `SUPERUSER` privileges for user `brainapi`.
- In all-in-one container deployments, Supervisord orchestrates local PostgreSQL (with `pgvector`), Redis, FastAPI API, MCP server, Celery worker, and Nginx, using persistent secrets stored in `/data/brainapi/internal-secrets.env`.
- In standalone deployment container environments, setting `PIPELINE_MODE=lightweight` and `EMBEDDINGS_PROVIDER=mock` or `fastembed` avoids hard runtime dependencies on external LLM services or Ollama instances.

### Security
- `SYSTEM_PAT` must be provided as an environment variable or Hugging Face Space Secret and never hardcoded in repository files or test scripts.
- Public UI and documentation endpoints (`/`, `/console`, `/docs`, `/redoc`, `/health`, `/openapi.json`, `/api-docs`, `/favicon.ico`) must be excluded from `BrainPATMiddleware` and `BrainMiddleware` authentication checks. `GET /` redirects to `/console/`.

### Testing and Documentation Rules
- Functional API endpoints (e.g., `/retrieve/context`, `/retrieve/search`) must be explicitly documented in `Agent.md` under the `### Functional Endpoints` section, including their Method, Path, Purpose, Request, and Response structures.
- API endpoints documented in `Agent.md` should strictly match the actual implementation (route decorators and Pydantic models). The canonical HTTP method (e.g., POST) should be presented as the primary interface, without modifying API behavior just to match docs.
- The local test suite must be executed using `PYTHONPATH=tests poetry run pytest` to resolve import errors for test helper modules.
- `Config` in `src/config.py` conditionally instantiates database sub-configurations (`mongo`, `milvus`, `neo4j`, `postgresql`) based on `DATA_DB`, `VECTOR_DB`, and `GRAPH_DB` environment variables.

## Credential model

`SYSTEM_PAT` is the single administrator credential across HTTP, console, and MCP.
The deprecated `BRAINPAT_TOKEN` may only alias that same value; a conflict fails
startup. Brains/workspaces do not generate credentials. Issue named access tokens
via `/system/access-tokens`, copy the plaintext once, and revoke independently.
Browser login uses `/auth/session` and `/auth/me`; a system session has no selected
brain (`brain_id=null`). The console stores no submitted access credential.
See `docs/security/credentials.md` for legacy migration and exposed-token rotation.

`/health` reports durability. Ephemeral staging requires explicit
`ALLOW_EPHEMERAL_STORAGE=true` and remains degraded / production NO-GO unless the
operator explicitly accepts the checkpointed-production policy documented below.

## GitHub Brain content recovery

Read [content checkpoint/recovery instructions](docs/content-backup.md) before
working on persistence. HF startup restores validated logical checkpoints from
`JsonLord/nodepad-brain-backup:main` using the separate `GITHUB_PAT` Space secret,
before workspace bootstrap and serving. Never back up auth registries/tokens,
Redis, internal secrets, or PGDATA; never force push or overwrite nonempty local
state of unknown lineage. API/MCP check the current container's restore gate.
Configure the documented `BRAIN_BACKUP_*` variables; keep async checkpoint RPO and
ephemeral health limitations explicit. No automatic deployment. The disposable
round-trip gate is `scripts/qualify_content_backup.py`; never point it at real
Brain databases. Auth tokens are separately reissued after ephemeral recovery.

For planned HF restarts, quiesce writers and verify a completed remote checkpoint
before replacement. HF replacement startup can overlap the old container's final
shutdown push; do not rely on that final push as a restore barrier. Compare the
restored commit with the intended checkpoint before resuming writes. See the
[live qualification and observed limitations](docs/content-backup-live-qualification.md).

The administrator maintenance API now enforces that sequence: quiesce/checkpoint,
record remotely verified SHA, rebuild, verify replacement restored/current SHA and
retrieval while still held, then explicitly resume. See `/system/backup/status`.
A manifest handoff hold survives replacement. Startup remote-head advances fail
closed before serving. Never bypass a hold or lineage failure to force readiness.

`BRAIN_BACKUP_MAX_RPO_SECONDS` defaults to the checkpoint interval (300).
`ALLOW_CHECKPOINTED_PRODUCTION=false` preserves ephemeral production NO-GO. Only
an explicit operator opt-in can accept checkpointed production, and only while
all remote/RPO/auth/database/embedding/access-log checks pass. Report the policy,
observed checkpoint age, and `zero_loss_durable=false` without implying zero loss.