| # Gemma 4 — local, CPU-first LLM stack (llama.cpp + llama-swap + TEI + Open WebUI) |
|
|
| A fully local stack: **Gemma 4** (text + **image** + **native audio**) served by **llama.cpp** |
| behind **llama-swap**, with **Persian-tuned RAG** (HF **TEI**: `bge-m3` embedder — top-2 on |
| the Persian FaMTEB benchmark — + `bge-reranker-v2-m3`) and **Open WebUI** as the front-end. |
| Profiled for CPU servers, with a GPU override. |
|
|
| > Embedder note: the Persian-SOTA **Hakim** (`MCINext/Hakim`) has no weights published in its |
| > HF repo yet (only a README), so the stack defaults to **`bge-m3`**. Swap `TEI_EMBED_MODEL` |
| > back to Hakim once its weights land. |
|
|
| ``` |
| Open WebUI ──┬─► llama-swap ─► llama-server (Gemma 4 omni + BF16 mmproj) chat · vision · audio |
| ├─► TEI-embed (BAAI/bge-m3 · Persian) RAG embeddings |
| └─► TEI-rerank (BAAI/bge-reranker-v2-m3) RAG rerank |
| ``` |
|
|
| ## Architecture |
|
|
|  |
|
|
| Requests flow **User → Open WebUI → {llama-swap, tei-embed, tei-rerank}**; llama-swap in turn |
| spawns `llama-server` on demand for whichever Gemma 4 profile is selected (only one resident at a |
| time; the profile's `DEFAULT_MODEL` is preloaded at startup and kept warm for 24 h so the first |
| message never pays the ~60 s cold load — switching models still swaps on demand). The audio Pipe bypasses Open WebUI's built-in STT |
| and talks to llama-swap directly so Gemma hears raw audio natively. Dotted arrows are read/write |
| volume mounts, not network calls. |
|
|
| <details> |
| <summary>Mermaid source (renders on GitHub/GitLab; Hugging Face model cards don't execute Mermaid, they only syntax-highlight it — hence the PNG above)</summary> |
|
|
| ```mermaid |
| flowchart TD |
| User(["User (Browser)"]) |
| |
| subgraph Net["docker compose network: stack"] |
| WebUI["openwebui\nOpen WebUI\nchat UI / RAG orchestrator\nport 3000 -> 8080"] |
| Swap["llama-swap\nOpenAI-compatible gateway\nport 8080"] |
| Server["llama-server\nspawned on demand\nGemma 4 omni + BF16 mmproj"] |
| Embed["tei-embed\nBAAI/bge-m3\nport 8081 -> 80"] |
| Rerank["tei-rerank\nBAAI/bge-reranker-v2-m3\nport 8082 -> 80"] |
| end |
| |
| Pipe["gemma4_audio_pipe.py\nOpen WebUI Function"] |
| Models[("./models\ngguf + mmproj")] |
| TeiData[("./models/tei\nHF cache")] |
| WebUIData[("./openwebui/data\ndb + uploads")] |
| |
| User -->|"HTTP :3000"| WebUI |
| WebUI -->|"chat / vision"| Swap |
| WebUI -->|"RAG embeddings"| Embed |
| WebUI -->|"RAG rerank"| Rerank |
| WebUI -.->|"imports"| Pipe |
| Pipe -->|"native audio"| Swap |
| Swap -->|"spawns"| Server |
| |
| Server -.-> Models |
| Embed -.-> TeiData |
| Rerank -.-> TeiData |
| WebUI -.-> WebUIData |
| |
| classDef svc fill:#1f6feb,color:#fff,stroke:#1f6feb |
| classDef vol fill:#57606a,color:#fff,stroke:#57606a |
| class WebUI,Swap,Server,Embed,Rerank svc |
| class Models,TeiData,WebUIData vol |
| ``` |
| </details> |
|
|
| ## What runs where |
|
|
| | Service | Image (CPU) | Internal URL | Host port | |
| |---|---|---|---| |
| | Open WebUI | `ghcr.io/open-webui/open-webui:main` | — | `3000` | |
| | llama-swap | `ghcr.io/mostlygeek/llama-swap:cpu` | `http://llama-swap:8080/v1` | `8080` | |
| | TEI embed | `…/text-embeddings-inference:cpu-1.9` | `http://tei-embed:80` | `8081` | |
| | TEI rerank | `…/text-embeddings-inference:cpu-1.9` | `http://tei-rerank:80` | `8082` | |
|
|
| ## Prerequisites |
| - Docker + `docker compose` v2 (`docker compose version`). |
| - A Hugging Face token with the **Gemma 4 license accepted** (the model is gated): |
| ```bash |
| pip install -U "huggingface_hub[cli]" && huggingface-cli login |
| ``` |
|
|
| ## Quick start (this CPU dev box) |
| ```bash |
| cd gemma4-stack |
| cp profiles/dev-cpu.env .env # E4B default; edit HF_TOKEN= |
| ./scripts/download-models.sh # pulls the REQUIRED BF16 mmproj files |
| docker compose up -d # boots TEI + Open WebUI + llama-swap |
| ``` |
| Open <http://localhost:3000>, create the first (admin) account. |
|
|
| **Default Open WebUI login:** |
| - **Email:** `UI@gmail.com` |
| - **Password:** `G4H!dg!R!4EdjkR` |
|
|
| > Change this password after first login if the instance is reachable beyond localhost. |
|
|
| **One-time in Admin UI:** |
| 1. **Settings → Models →** enable **Vision** on `gemma-e4b` (lets Open WebUI send images). |
| 2. **Admin → Functions → +** → paste `openwebui/functions/gemma4_audio_pipe.py` → **Save** → enable. |
| - In its valves, confirm `MODEL=gemma-e4b` and `LLAMASWAP_URL=http://llama-swap:8080/v1`. |
| 3. **Settings → Documents →** confirm embedding engine = **OpenAI**, model = your `TEI_EMBED_MODEL`, |
| **Hybrid Search ON**, and the external reranker URL is reachable. |
|
|
| ## Use it |
| - **Text/Persian chat** — pick `gemma-e4b`, chat. |
| - **Image** — attach a picture, ask about it (Vision toggle must be on). |
| - **Native audio** — pick **“Gemma 4 · Omni (audio)”**, attach a short clip (≤ ~30 s), |
| ask your question. Gemma *hears* it (no Whisper). |
| - **RAG** — upload Persian docs to a Knowledge / `#`-reference them; retrieval → rerank → answer. |
|
|
| ## Other machines / profiles |
| ```bash |
| cp profiles/cpu-server.env .env # 12B default; set THREADS=<physical cores> |
| docker compose up -d |
| |
| cp profiles/gpu.env .env # GPU box |
| docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d |
| ``` |
| Switch the live model anytime from the model dropdown — llama-swap loads it on demand and |
| unloads the idle one (so RAM never doubles). Profiles: `gemma-e4b`, `gemma-12b`, |
| `gemma-26b-a4b` (image-only). |
|
|
| ## Test it yourself |
| **One command — runs all 7 checks (text · image · native audio · embeddings · rerank · UI) using the |
| bundled demo assets, no setup:** |
| ```bash |
| make test # expects: RESULT: 7 passed, 0 failed |
| ``` |
| In the **browser** (port 3000): pick `gemma-e4b` and chat in Persian; attach `assets/test_image.png` |
| and ask about it; for audio, import the Pipe (Admin → Functions) then pick “Gemma 4 · Omni (audio)” |
| and attach `assets/test_audio_en.wav`. |
|
|
| ## Verify from the shell |
| ```bash |
| curl localhost:8080/v1/models # llama-swap lists the profiles |
| curl localhost:8081/v1/embeddings -d '{"model":"x","input":["سلام"]}' # embedder |
| curl localhost:8082/rerank -d '{"query":"پایتخت ایران","texts":["تهران","موز"]}' # reranker |
| # native audio (after: ffmpeg -i clip.any -ar 16000 -ac 1 clip.wav): |
| B64=$(base64 -w0 clip.wav); curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \ |
| -d '{"model":"gemma-e4b","messages":[{"role":"user","content":[{"type":"text","text":"این صدا را بنویس"},{"type":"input_audio","input_audio":{"data":"'"$B64"'","format":"wav"}}]}]}' |
| ``` |
|
|
| ## Tuning (`scripts/bench.sh`) |
| `./scripts/download-models.sh --weights` then `./scripts/bench.sh /models/gemma-4-E4B-it-Q4_K_M.gguf` |
| to sweep `--threads`. Set the winner as `THREADS` in `.env`. Long context? add KV quant |
| (`-ctk q8_0 -ctv q8_0`) to the macro in `llama-swap/config.tmpl.yaml`. |
|
|
| ## Performance & capacity planning (CPU serving) |
|
|
| **Measured** on the 4-core / 16 GB dev box (Xeon 2.6 GHz, E4B Q4_K_M, warm model, |
| via the real Open WebUI through the public proxy — not just raw llama-bench): |
|
|
| | Metric (E4B, warm) | 4c / 16 GB — measured | |
| |---|---| |
| | TTFT in the browser, short prompt | **~3 s** | |
| | Generation speed | **~3.4–4.0 tok/s** | |
| | Prompt processing | **~8–15 tok/s** | |
| | ~150-token chat answer, end to end | **~45–70 s** | |
| | Follow-up turn (prompt cache hit) | prompt cost ≈ **–70 %** | |
| | TEI embed / rerank (per request) | **0.7 s / 1.7 s** | |
|
|
| **Back-of-envelope formula** (holds well in practice): |
|
|
| ``` |
| TTFT ≈ prompt_tokens / pp_speed (+ ~0.5 s overhead) |
| total ≈ TTFT + output_tokens / gen_speed |
| RPS ≈ parallel_slots / total (sequential queue otherwise) |
| ``` |
|
|
| **Estimates for bigger CPU tiers** (prompt processing is compute-bound → scales ~linearly |
| with physical cores; token generation is memory-bandwidth-bound → scales sublinearly, |
| roughly ×1.7 at 8c and ×2.5–3 at 16c vs this box). Assumes a ~300-token prompt and |
| ~150-token answer per request: |
|
|
| | | 4c / 16 GB (this box) | 8c / 16 GB (est.) | 16c / 64 GB (est.) | |
| |---|---|---|---| |
| | **E4B** gen / pp (tok/s) | 3.5–4 / 8–15 ✓ | ~6–7 / ~25 | ~9–12 / ~50 | |
| | **E4B** TTFT / total per chat | ~3 s / ~60 s ✓ | ~1.5 s / ~35 s | ~1 s / ~20 s | |
| | **E4B** chat throughput | ~1 req/min (RPS ≈ 0.015) | ~1.7 req/min (0.03) | ~3 req/min (0.05) | |
| | **12B** gen / pp (tok/s) | ~1.2–1.5 / ~4 (RAM-tight) | ~2–2.5 / ~9 (tight) | ~4–5 / ~20 | |
| | **12B** chat total / throughput | not recommended | ~2 min/req | ~55 s/req, ~1 req/min | |
| | **26B-A4B** (MoE, ~18 GB) | ✗ no RAM | ✗ no RAM | ~7–9 / ~40; ~30 s/req | |
| | Comfortable interactive users | 1 | 1–2 | 2–4 | |
|
|
| Practical read of that table: |
|
|
| - **16 GB RAM caps you at E4B** as the daily driver (12B fits but leaves TEI+UI almost no |
| headroom). **The 16c / 64 GB tier is the first one where 12B and 26B-A4B are pleasant.** |
| - **Concurrency**: `llama-server` runs **one inference slot** by default — a second chat |
| *queues*, it doesn't run alongside. On the 16c/64 GB tier add `--parallel 2` (and note each |
| slot gets `CTX / N` context) to serve 2 streams; aggregate tok/s barely changes |
| (bandwidth-bound), it just shares fairly instead of queueing. |
| - **CPU serving is single-digit-users territory.** For ~10+ concurrent chat users at |
| interactive speeds, use the GPU profile — no CPU tier gets you there. |
| - **RAM budget** (Q4_K_M + BF16 mmproj + 8k ctx): E4B ≈ 5 GB · 12B ≈ 10 GB · 26B-A4B ≈ 18 GB; |
| plus TEI embed+rerank ≈ 4 GB, Open WebUI ≈ 0.7 GB, OS ≈ 1 GB. |
| - Embedding RAG load is cheap next to chat: TEI on this box sustains ≈ 1.4 embeds/s and |
| ≈ 0.6 reranks/s per core-set — RAG retrieval adds ~2–3 s per message, not minutes. |
|
|
| **Latency defaults already baked into this stack** (the difference between these numbers and |
| minutes-long hangs): `--reasoning off` (Gemma 4 otherwise burns the whole budget thinking), |
| `-n 1024` output cap, `--cache-reuse 256` (multi-turn prompt cache), startup **preload** of |
| `DEFAULT_MODEL` + 24 h idle-ttl (no cold load on first message), and Open WebUI background |
| task generation (title/tags/follow-up/query) **disabled** — note those are PersistentConfig: |
| once in `webui.db`, env vars are ignored; change them in Admin Settings → Interface or via |
| `POST /api/v1/tasks/config/update`. |
|
|
| ## Troubleshooting (the known traps) |
| - **Audio crashes / asserts** → keep `-b 2048 -ub 2048` (already in the macro). Default 512 crashes. |
| - **Audio is garbage/repetitive** → the mmproj must be **BF16** (we pin it). Q8_0/F16 break audio. |
| - **`llama-server` not found in llama-swap** → adjust the binary path in `config.tmpl.yaml` / `entrypoint.sh`. |
| - **`llama-swap` flags differ** → check `docker run --rm <llamaswap-image> llama-swap --help`; fix `entrypoint.sh`. |
| - **Audio “No audio found” in the Pipe** → Open WebUI's file-store API differs by version; |
| adapt `_resolve_local_path()` in the Pipe. |
| - **Embeddings fail / wrong model** → set every `RAG_*` env explicitly (Open WebUI doesn't inherit |
| `OPENAI_*`); verify in Admin → Documents. Hakim won't load? set `TEI_EMBED_MODEL=BAAI/bge-m3` |
| and `TEI_EMBED_POOLING=cls`. |
| - **Reranker freezes UI / format mismatch** → fall back to Open WebUI's built-in reranker: remove the |
| `RAG_RERANKING_ENGINE=external` + URL envs and set `RAG_RERANKING_MODEL=BAAI/bge-reranker-v2-m3`. |
|
|
| ## Operational notes (validated live on a 15 GB CPU box) |
| - **Gemma 4 is a reasoning model** — by default it emits `reasoning_content` before `content` |
| and, unbounded, can think for *minutes* before a visible answer. The macro therefore ships |
| `--reasoning off` (clean direct answers). If you re-enable it, prefer `--reasoning-budget N` |
| over unbounded, give `max_tokens` ≥ 256, and read both fields in raw API callers |
| (`--reasoning-budget 0` leaks "Thinking Process:" prose into `content` — use `off`). |
| - **Send media via a file, not inline** — base64 audio/large images exceed the shell arg limit |
| (`argument list too long`). Use `curl -d @payload.json` (the e2e script and Pipe already do). |
| - **TEI memory on CPU**: the ONNX backend is heavy (bge-m3 ≈ 4.75 GB). Setting `TEI_DTYPE=float16` |
| makes ONNX bow out and TEI falls back to the lighter **Candle** backend (≈ 3.4 GB) — ~2.4 GB |
| saved, same 1024-dim Persian output. That's why `--dtype float16` is in the compose commands. |
| - **Tight-RAM tuning (this box)**: dev profile uses `CTX=4096`; E4B is loaded by local path |
| (`-m /models/...gguf`) so a studio pause/resume doesn't re-download 5 GB. E4B + bge-m3 + |
| reranker + UI ≈ 13–14 GB — fits 15 GB but with little headroom; **16–24 GB is comfortable**. |
| - **Verified end-to-end here**: Persian text ✓, image→“Red” ✓, native audio→exact transcript ✓, |
| Persian embeddings (1024-d) ✓, Persian rerank ordering ✓, Open WebUI ✓. |
|
|
| ## Layout |
| ``` |
| docker-compose.yml · docker-compose.gpu.yml · .env.example |
| profiles/{dev-cpu,cpu-server,cpu-server-xl,gpu}.env |
| llama-swap/{config.tmpl.yaml,entrypoint.sh} |
| openwebui/functions/gemma4_audio_pipe.py |
| scripts/{download-models.sh,bench.sh} |
| ``` |
| Full design rationale: `../.claude/plans/i-want-make-a-ancient-teapot.md`. |
|
|
