README.md

# Gemma 4 — local, CPU-first LLM stack (llama.cpp + llama-swap + TEI + Open WebUI)

A fully local stack: **Gemma 4** (text + **image** + **native audio**) served by **llama.cpp**
behind **llama-swap**, with **Persian-tuned RAG** (HF **TEI**: `bge-m3` embedder — top-2 on
the Persian FaMTEB benchmark — + `bge-reranker-v2-m3`) and **Open WebUI** as the front-end.
Profiled for CPU servers, with a GPU override.

> Embedder note: the Persian-SOTA **Hakim** (`MCINext/Hakim`) has no weights published in its
> HF repo yet (only a README), so the stack defaults to **`bge-m3`**. Swap `TEI_EMBED_MODEL`
> back to Hakim once its weights land.

```
Open WebUI ──┬─► llama-swap ─► llama-server (Gemma 4 omni + BF16 mmproj)   chat · vision · audio
             ├─► TEI-embed  (BAAI/bge-m3 · Persian)                       RAG embeddings
             └─► TEI-rerank (BAAI/bge-reranker-v2-m3)                      RAG rerank
```

## Architecture

![Architecture diagram: browser talks to Open WebUI, which routes chat/vision to llama-swap (which spawns llama-server on demand), RAG embeddings to tei-embed, and RAG rerank to tei-rerank; the audio Pipe function talks to llama-swap directly for native audio; each service reads its model files from a host-mounted volume.](assets/architecture.png)

Requests flow **User → Open WebUI → {llama-swap, tei-embed, tei-rerank}**; llama-swap in turn
spawns `llama-server` on demand for whichever Gemma 4 profile is selected (only one resident at a
time; the profile's `DEFAULT_MODEL` is preloaded at startup and kept warm for 24 h so the first
message never pays the ~60 s cold load — switching models still swaps on demand). The audio Pipe bypasses Open WebUI's built-in STT
and talks to llama-swap directly so Gemma hears raw audio natively. Dotted arrows are read/write
volume mounts, not network calls.

<details>
<summary>Mermaid source (renders on GitHub/GitLab; Hugging Face model cards don't execute Mermaid, they only syntax-highlight it — hence the PNG above)</summary>

```mermaid
flowchart TD
    User(["User (Browser)"])

    subgraph Net["docker compose network: stack"]
        WebUI["openwebui\nOpen WebUI\nchat UI / RAG orchestrator\nport 3000 -> 8080"]
        Swap["llama-swap\nOpenAI-compatible gateway\nport 8080"]
        Server["llama-server\nspawned on demand\nGemma 4 omni + BF16 mmproj"]
        Embed["tei-embed\nBAAI/bge-m3\nport 8081 -> 80"]
        Rerank["tei-rerank\nBAAI/bge-reranker-v2-m3\nport 8082 -> 80"]
    end

    Pipe["gemma4_audio_pipe.py\nOpen WebUI Function"]
    Models[("./models\ngguf + mmproj")]
    TeiData[("./models/tei\nHF cache")]
    WebUIData[("./openwebui/data\ndb + uploads")]

    User -->|"HTTP :3000"| WebUI
    WebUI -->|"chat / vision"| Swap
    WebUI -->|"RAG embeddings"| Embed
    WebUI -->|"RAG rerank"| Rerank
    WebUI -.->|"imports"| Pipe
    Pipe -->|"native audio"| Swap
    Swap -->|"spawns"| Server

    Server -.-> Models
    Embed -.-> TeiData
    Rerank -.-> TeiData
    WebUI -.-> WebUIData

    classDef svc fill:#1f6feb,color:#fff,stroke:#1f6feb
    classDef vol fill:#57606a,color:#fff,stroke:#57606a
    class WebUI,Swap,Server,Embed,Rerank svc
    class Models,TeiData,WebUIData vol
```
</details>

## What runs where

| Service | Image (CPU) | Internal URL | Host port |
|---|---|---|---|
| Open WebUI | `ghcr.io/open-webui/open-webui:main` | — | `3000` |
| llama-swap | `ghcr.io/mostlygeek/llama-swap:cpu` | `http://llama-swap:8080/v1` | `8080` |
| TEI embed | `…/text-embeddings-inference:cpu-1.9` | `http://tei-embed:80` | `8081` |
| TEI rerank | `…/text-embeddings-inference:cpu-1.9` | `http://tei-rerank:80` | `8082` |

## Prerequisites
- Docker + `docker compose` v2 (`docker compose version`).
- A Hugging Face token with the **Gemma 4 license accepted** (the model is gated):
  ```bash
  pip install -U "huggingface_hub[cli]" && huggingface-cli login
  ```

## Quick start (this CPU dev box)
```bash
cd gemma4-stack
cp profiles/dev-cpu.env .env          # E4B default; edit HF_TOKEN=
./scripts/download-models.sh          # pulls the REQUIRED BF16 mmproj files
docker compose up -d                  # boots TEI + Open WebUI + llama-swap
```
Open <http://localhost:3000>, create the first (admin) account.

**Default Open WebUI login:**
- **Email:** `UI@gmail.com`
- **Password:** `G4H!dg!R!4EdjkR`

> Change this password after first login if the instance is reachable beyond localhost.

**One-time in Admin UI:**
1. **Settings → Models →** enable **Vision** on `gemma-e4b` (lets Open WebUI send images).
2. **Admin → Functions → +** → paste `openwebui/functions/gemma4_audio_pipe.py` → **Save** → enable.
   - In its valves, confirm `MODEL=gemma-e4b` and `LLAMASWAP_URL=http://llama-swap:8080/v1`.
3. **Settings → Documents →** confirm embedding engine = **OpenAI**, model = your `TEI_EMBED_MODEL`,
   **Hybrid Search ON**, and the external reranker URL is reachable.

## Use it
- **Text/Persian chat** — pick `gemma-e4b`, chat.
- **Image** — attach a picture, ask about it (Vision toggle must be on).
- **Native audio** — pick **“Gemma 4 · Omni (audio)”**, attach a short clip (≤ ~30 s),
  ask your question. Gemma *hears* it (no Whisper).
- **RAG** — upload Persian docs to a Knowledge / `#`-reference them; retrieval → rerank → answer.

## Other machines / profiles
```bash
cp profiles/cpu-server.env .env       # 12B default; set THREADS=<physical cores>
docker compose up -d

cp profiles/gpu.env .env              # GPU box
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d
```
Switch the live model anytime from the model dropdown — llama-swap loads it on demand and
unloads the idle one (so RAM never doubles). Profiles: `gemma-e4b`, `gemma-12b`,
`gemma-26b-a4b` (image-only).

## Test it yourself
**One command — runs all 7 checks (text · image · native audio · embeddings · rerank · UI) using the
bundled demo assets, no setup:**
```bash
make test            # expects: RESULT: 7 passed, 0 failed
```
In the **browser** (port 3000): pick `gemma-e4b` and chat in Persian; attach `assets/test_image.png`
and ask about it; for audio, import the Pipe (Admin → Functions) then pick “Gemma 4 · Omni (audio)”
and attach `assets/test_audio_en.wav`.

## Verify from the shell
```bash
curl localhost:8080/v1/models                     # llama-swap lists the profiles
curl localhost:8081/v1/embeddings -d '{"model":"x","input":["سلام"]}'   # embedder
curl localhost:8082/rerank -d '{"query":"پایتخت ایران","texts":["تهران","موز"]}'  # reranker
# native audio (after: ffmpeg -i clip.any -ar 16000 -ac 1 clip.wav):
B64=$(base64 -w0 clip.wav); curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
 -d '{"model":"gemma-e4b","messages":[{"role":"user","content":[{"type":"text","text":"این صدا را بنویس"},{"type":"input_audio","input_audio":{"data":"'"$B64"'","format":"wav"}}]}]}'
```

## Tuning (`scripts/bench.sh`)
`./scripts/download-models.sh --weights` then `./scripts/bench.sh /models/gemma-4-E4B-it-Q4_K_M.gguf`
to sweep `--threads`. Set the winner as `THREADS` in `.env`. Long context? add KV quant
(`-ctk q8_0 -ctv q8_0`) to the macro in `llama-swap/config.tmpl.yaml`.

## Performance & capacity planning (CPU serving)

**Measured** on the 4-core / 16 GB dev box (Xeon 2.6 GHz, E4B Q4_K_M, warm model,
via the real Open WebUI through the public proxy — not just raw llama-bench):

| Metric (E4B, warm) | 4c / 16 GB — measured |
|---|---|
| TTFT in the browser, short prompt | **~3 s** |
| Generation speed | **~3.4–4.0 tok/s** |
| Prompt processing | **~8–15 tok/s** |
| ~150-token chat answer, end to end | **~45–70 s** |
| Follow-up turn (prompt cache hit) | prompt cost ≈ **–70 %** |
| TEI embed / rerank (per request) | **0.7 s / 1.7 s** |

**Back-of-envelope formula** (holds well in practice):

```
TTFT   ≈ prompt_tokens / pp_speed          (+ ~0.5 s overhead)
total  ≈ TTFT + output_tokens / gen_speed
RPS    ≈ parallel_slots / total            (sequential queue otherwise)
```

**Estimates for bigger CPU tiers** (prompt processing is compute-bound → scales ~linearly
with physical cores; token generation is memory-bandwidth-bound → scales sublinearly,
roughly ×1.7 at 8c and ×2.5–3 at 16c vs this box). Assumes a ~300-token prompt and
~150-token answer per request:

| | 4c / 16 GB (this box) | 8c / 16 GB (est.) | 16c / 64 GB (est.) |
|---|---|---|---|
| **E4B** gen / pp (tok/s) | 3.5–4 / 8–15 ✓ | ~6–7 / ~25 | ~9–12 / ~50 |
| **E4B** TTFT / total per chat | ~3 s / ~60 s ✓ | ~1.5 s / ~35 s | ~1 s / ~20 s |
| **E4B** chat throughput | ~1 req/min (RPS ≈ 0.015) | ~1.7 req/min (0.03) | ~3 req/min (0.05) |
| **12B** gen / pp (tok/s) | ~1.2–1.5 / ~4 (RAM-tight) | ~2–2.5 / ~9 (tight) | ~4–5 / ~20 |
| **12B** chat total / throughput | not recommended | ~2 min/req | ~55 s/req, ~1 req/min |
| **26B-A4B** (MoE, ~18 GB) | ✗ no RAM | ✗ no RAM | ~7–9 / ~40; ~30 s/req |
| Comfortable interactive users | 1 | 1–2 | 2–4 |

Practical read of that table:

- **16 GB RAM caps you at E4B** as the daily driver (12B fits but leaves TEI+UI almost no
  headroom). **The 16c / 64 GB tier is the first one where 12B and 26B-A4B are pleasant.**
- **Concurrency**: `llama-server` runs **one inference slot** by default — a second chat
  *queues*, it doesn't run alongside. On the 16c/64 GB tier add `--parallel 2` (and note each
  slot gets `CTX / N` context) to serve 2 streams; aggregate tok/s barely changes
  (bandwidth-bound), it just shares fairly instead of queueing.
- **CPU serving is single-digit-users territory.** For ~10+ concurrent chat users at
  interactive speeds, use the GPU profile — no CPU tier gets you there.
- **RAM budget** (Q4_K_M + BF16 mmproj + 8k ctx): E4B ≈ 5 GB · 12B ≈ 10 GB · 26B-A4B ≈ 18 GB;
  plus TEI embed+rerank ≈ 4 GB, Open WebUI ≈ 0.7 GB, OS ≈ 1 GB.
- Embedding RAG load is cheap next to chat: TEI on this box sustains ≈ 1.4 embeds/s and
  ≈ 0.6 reranks/s per core-set — RAG retrieval adds ~2–3 s per message, not minutes.

**Latency defaults already baked into this stack** (the difference between these numbers and
minutes-long hangs): `--reasoning off` (Gemma 4 otherwise burns the whole budget thinking),
`-n 1024` output cap, `--cache-reuse 256` (multi-turn prompt cache), startup **preload** of
`DEFAULT_MODEL` + 24 h idle-ttl (no cold load on first message), and Open WebUI background
task generation (title/tags/follow-up/query) **disabled** — note those are PersistentConfig:
once in `webui.db`, env vars are ignored; change them in Admin Settings → Interface or via
`POST /api/v1/tasks/config/update`.

## Troubleshooting (the known traps)
- **Audio crashes / asserts** → keep `-b 2048 -ub 2048` (already in the macro). Default 512 crashes.
- **Audio is garbage/repetitive** → the mmproj must be **BF16** (we pin it). Q8_0/F16 break audio.
- **`llama-server` not found in llama-swap** → adjust the binary path in `config.tmpl.yaml` / `entrypoint.sh`.
- **`llama-swap` flags differ** → check `docker run --rm <llamaswap-image> llama-swap --help`; fix `entrypoint.sh`.
- **Audio “No audio found” in the Pipe** → Open WebUI's file-store API differs by version;
  adapt `_resolve_local_path()` in the Pipe.
- **Embeddings fail / wrong model** → set every `RAG_*` env explicitly (Open WebUI doesn't inherit
  `OPENAI_*`); verify in Admin → Documents. Hakim won't load? set `TEI_EMBED_MODEL=BAAI/bge-m3`
  and `TEI_EMBED_POOLING=cls`.
- **Reranker freezes UI / format mismatch** → fall back to Open WebUI's built-in reranker: remove the
  `RAG_RERANKING_ENGINE=external` + URL envs and set `RAG_RERANKING_MODEL=BAAI/bge-reranker-v2-m3`.

## Operational notes (validated live on a 15 GB CPU box)
- **Gemma 4 is a reasoning model** — by default it emits `reasoning_content` before `content`
  and, unbounded, can think for *minutes* before a visible answer. The macro therefore ships
  `--reasoning off` (clean direct answers). If you re-enable it, prefer `--reasoning-budget N`
  over unbounded, give `max_tokens` ≥ 256, and read both fields in raw API callers
  (`--reasoning-budget 0` leaks "Thinking Process:" prose into `content` — use `off`).
- **Send media via a file, not inline** — base64 audio/large images exceed the shell arg limit
  (`argument list too long`). Use `curl -d @payload.json` (the e2e script and Pipe already do).
- **TEI memory on CPU**: the ONNX backend is heavy (bge-m3 ≈ 4.75 GB). Setting `TEI_DTYPE=float16`
  makes ONNX bow out and TEI falls back to the lighter **Candle** backend (≈ 3.4 GB) — ~2.4 GB
  saved, same 1024-dim Persian output. That's why `--dtype float16` is in the compose commands.
- **Tight-RAM tuning (this box)**: dev profile uses `CTX=4096`; E4B is loaded by local path
  (`-m /models/...gguf`) so a studio pause/resume doesn't re-download 5 GB. E4B + bge-m3 +
  reranker + UI ≈ 13–14 GB — fits 15 GB but with little headroom; **16–24 GB is comfortable**.
- **Verified end-to-end here**: Persian text ✓, image→“Red” ✓, native audio→exact transcript ✓,
  Persian embeddings (1024-d) ✓, Persian rerank ordering ✓, Open WebUI ✓.

## Layout
```
docker-compose.yml · docker-compose.gpu.yml · .env.example
profiles/{dev-cpu,cpu-server,cpu-server-xl,gpu}.env
llama-swap/{config.tmpl.yaml,entrypoint.sh}
openwebui/functions/gemma4_audio_pipe.py
scripts/{download-models.sh,bench.sh}
```
Full design rationale: `../.claude/plans/i-want-make-a-ancient-teapot.md`.