Neohosseinism/gemma4-stack

Gemma 4 — local, CPU-first LLM stack (llama.cpp + llama-swap + TEI + Open WebUI)

A fully local stack: Gemma 4 (text + image + native audio) served by llama.cpp behind llama-swap, with Persian-tuned RAG (HF TEI: bge-m3 embedder — top-2 on the Persian FaMTEB benchmark — + bge-reranker-v2-m3) and Open WebUI as the front-end. Profiled for CPU servers, with a GPU override.

Embedder note: the Persian-SOTA Hakim (MCINext/Hakim) has no weights published in its HF repo yet (only a README), so the stack defaults to bge-m3. Swap TEI_EMBED_MODEL back to Hakim once its weights land.

Open WebUI ──┬─► llama-swap ─► llama-server (Gemma 4 omni + BF16 mmproj)   chat · vision · audio
             ├─► TEI-embed  (BAAI/bge-m3 · Persian)                       RAG embeddings
             └─► TEI-rerank (BAAI/bge-reranker-v2-m3)                      RAG rerank

Architecture

Requests flow User → Open WebUI → {llama-swap, tei-embed, tei-rerank}; llama-swap in turn spawns llama-server on demand for whichever Gemma 4 profile is selected (only one resident at a time; the profile's DEFAULT_MODEL is preloaded at startup and kept warm for 24 h so the first message never pays the ~60 s cold load — switching models still swaps on demand). The audio Pipe bypasses Open WebUI's built-in STT and talks to llama-swap directly so Gemma hears raw audio natively. Dotted arrows are read/write volume mounts, not network calls.

Mermaid source (renders on GitHub/GitLab; Hugging Face model cards don't execute Mermaid, they only syntax-highlight it — hence the PNG above)

flowchart TD
    User(["User (Browser)"])

    subgraph Net["docker compose network: stack"]
        WebUI["openwebui\nOpen WebUI\nchat UI / RAG orchestrator\nport 3000 -> 8080"]
        Swap["llama-swap\nOpenAI-compatible gateway\nport 8080"]
        Server["llama-server\nspawned on demand\nGemma 4 omni + BF16 mmproj"]
        Embed["tei-embed\nBAAI/bge-m3\nport 8081 -> 80"]
        Rerank["tei-rerank\nBAAI/bge-reranker-v2-m3\nport 8082 -> 80"]
    end

    Pipe["gemma4_audio_pipe.py\nOpen WebUI Function"]
    Models[("./models\ngguf + mmproj")]
    TeiData[("./models/tei\nHF cache")]
    WebUIData[("./openwebui/data\ndb + uploads")]

    User -->|"HTTP :3000"| WebUI
    WebUI -->|"chat / vision"| Swap
    WebUI -->|"RAG embeddings"| Embed
    WebUI -->|"RAG rerank"| Rerank
    WebUI -.->|"imports"| Pipe
    Pipe -->|"native audio"| Swap
    Swap -->|"spawns"| Server

    Server -.-> Models
    Embed -.-> TeiData
    Rerank -.-> TeiData
    WebUI -.-> WebUIData

    classDef svc fill:#1f6feb,color:#fff,stroke:#1f6feb
    classDef vol fill:#57606a,color:#fff,stroke:#57606a
    class WebUI,Swap,Server,Embed,Rerank svc
    class Models,TeiData,WebUIData vol

What runs where

Service	Image (CPU)	Internal URL	Host port
Open WebUI	ghcr.io/open-webui/open-webui:main	—	3000
llama-swap	ghcr.io/mostlygeek/llama-swap:cpu	http://llama-swap:8080/v1	8080
TEI embed	…/text-embeddings-inference:cpu-1.9	http://tei-embed:80	8081
TEI rerank	…/text-embeddings-inference:cpu-1.9	http://tei-rerank:80	8082

Prerequisites

• Docker + docker compose v2 (docker compose version).
• A Hugging Face token with the Gemma 4 license accepted (the model is gated):

  pip install -U "huggingface_hub[cli]" && huggingface-cli login
  

Quick start (this CPU dev box)

cd gemma4-stack
cp profiles/dev-cpu.env .env          # E4B default; edit HF_TOKEN=
./scripts/download-models.sh          # pulls the REQUIRED BF16 mmproj files
docker compose up -d                  # boots TEI + Open WebUI + llama-swap

Open http://localhost:3000, create the first (admin) account.

Default Open WebUI login:

• Email: UI@gmail.com
• Password: G4H!dg!R!4EdjkR

Change this password after first login if the instance is reachable beyond localhost.

One-time in Admin UI:

1. Settings → Models → enable Vision on gemma-e4b (lets Open WebUI send images).
2. Admin → Functions → + → paste openwebui/functions/gemma4_audio_pipe.py → Save → enable.
   • In its valves, confirm MODEL=gemma-e4b and LLAMASWAP_URL=http://llama-swap:8080/v1.
3. Settings → Documents → confirm embedding engine = OpenAI, model = your TEI_EMBED_MODEL, Hybrid Search ON, and the external reranker URL is reachable.

Use it

• Text/Persian chat — pick gemma-e4b, chat.
• Image — attach a picture, ask about it (Vision toggle must be on).
• Native audio — pick “Gemma 4 · Omni (audio)”, attach a short clip (≤ ~30 s), ask your question. Gemma hears it (no Whisper).
• RAG — upload Persian docs to a Knowledge / #-reference them; retrieval → rerank → answer.

Other machines / profiles

cp profiles/cpu-server.env .env       # 12B default; set THREADS=<physical cores>
docker compose up -d

cp profiles/gpu.env .env              # GPU box
docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d

Switch the live model anytime from the model dropdown — llama-swap loads it on demand and unloads the idle one (so RAM never doubles). Profiles: gemma-e4b, gemma-12b, gemma-26b-a4b (image-only).

Test it yourself

One command — runs all 7 checks (text · image · native audio · embeddings · rerank · UI) using the bundled demo assets, no setup:

make test            # expects: RESULT: 7 passed, 0 failed

In the browser (port 3000): pick gemma-e4b and chat in Persian; attach assets/test_image.png and ask about it; for audio, import the Pipe (Admin → Functions) then pick “Gemma 4 · Omni (audio)” and attach assets/test_audio_en.wav.

Verify from the shell

curl localhost:8080/v1/models                     # llama-swap lists the profiles
curl localhost:8081/v1/embeddings -d '{"model":"x","input":["سلام"]}'   # embedder
curl localhost:8082/rerank -d '{"query":"پایتخت ایران","texts":["تهران","موز"]}'  # reranker
# native audio (after: ffmpeg -i clip.any -ar 16000 -ac 1 clip.wav):
B64=$(base64 -w0 clip.wav); curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
 -d '{"model":"gemma-e4b","messages":[{"role":"user","content":[{"type":"text","text":"این صدا را بنویس"},{"type":"input_audio","input_audio":{"data":"'"$B64"'","format":"wav"}}]}]}'

Tuning (scripts/bench.sh)

./scripts/download-models.sh --weights then ./scripts/bench.sh /models/gemma-4-E4B-it-Q4_K_M.gguf to sweep --threads. Set the winner as THREADS in .env. Long context? add KV quant (-ctk q8_0 -ctv q8_0) to the macro in llama-swap/config.tmpl.yaml.

Performance & capacity planning (CPU serving)

Measured on the 4-core / 16 GB dev box (Xeon 2.6 GHz, E4B Q4_K_M, warm model, via the real Open WebUI through the public proxy — not just raw llama-bench):

Metric (E4B, warm)	4c / 16 GB — measured
TTFT in the browser, short prompt	~3 s
Generation speed	~3.4–4.0 tok/s
Prompt processing	~8–15 tok/s
~150-token chat answer, end to end	~45–70 s
Follow-up turn (prompt cache hit)	prompt cost ≈ –70 %
TEI embed / rerank (per request)	0.7 s / 1.7 s

Back-of-envelope formula (holds well in practice):

TTFT   ≈ prompt_tokens / pp_speed          (+ ~0.5 s overhead)
total  ≈ TTFT + output_tokens / gen_speed
RPS    ≈ parallel_slots / total            (sequential queue otherwise)

Estimates for bigger CPU tiers (prompt processing is compute-bound → scales ~linearly with physical cores; token generation is memory-bandwidth-bound → scales sublinearly, roughly ×1.7 at 8c and ×2.5–3 at 16c vs this box). Assumes a ~300-token prompt and ~150-token answer per request:

	4c / 16 GB (this box)	8c / 16 GB (est.)	16c / 64 GB (est.)
E4B gen / pp (tok/s)	3.5–4 / 8–15 ✓	~6–7 / ~25	~9–12 / ~50
E4B TTFT / total per chat	~3 s / ~60 s ✓	~1.5 s / ~35 s	~1 s / ~20 s
E4B chat throughput	~1 req/min (RPS ≈ 0.015)	~1.7 req/min (0.03)	~3 req/min (0.05)
12B gen / pp (tok/s)	~1.2–1.5 / ~4 (RAM-tight)	~2–2.5 / ~9 (tight)	~4–5 / ~20
12B chat total / throughput	not recommended	~2 min/req	~55 s/req, ~1 req/min
26B-A4B (MoE, ~18 GB)	✗ no RAM	✗ no RAM	~7–9 / ~40; ~30 s/req
Comfortable interactive users	1	1–2	2–4

Practical read of that table:

• 16 GB RAM caps you at E4B as the daily driver (12B fits but leaves TEI+UI almost no headroom). The 16c / 64 GB tier is the first one where 12B and 26B-A4B are pleasant.
• Concurrency: llama-server runs one inference slot by default — a second chat queues, it doesn't run alongside. On the 16c/64 GB tier add --parallel 2 (and note each slot gets CTX / N context) to serve 2 streams; aggregate tok/s barely changes (bandwidth-bound), it just shares fairly instead of queueing.
• CPU serving is single-digit-users territory. For ~10+ concurrent chat users at interactive speeds, use the GPU profile — no CPU tier gets you there.
• RAM budget (Q4_K_M + BF16 mmproj + 8k ctx): E4B ≈ 5 GB · 12B ≈ 10 GB · 26B-A4B ≈ 18 GB; plus TEI embed+rerank ≈ 4 GB, Open WebUI ≈ 0.7 GB, OS ≈ 1 GB.
• Embedding RAG load is cheap next to chat: TEI on this box sustains ≈ 1.4 embeds/s and ≈ 0.6 reranks/s per core-set — RAG retrieval adds ~2–3 s per message, not minutes.

Latency defaults already baked into this stack (the difference between these numbers and minutes-long hangs): --reasoning off (Gemma 4 otherwise burns the whole budget thinking), -n 1024 output cap, --cache-reuse 256 (multi-turn prompt cache), startup preload of DEFAULT_MODEL + 24 h idle-ttl (no cold load on first message), and Open WebUI background task generation (title/tags/follow-up/query) disabled — note those are PersistentConfig: once in webui.db, env vars are ignored; change them in Admin Settings → Interface or via POST /api/v1/tasks/config/update.

Troubleshooting (the known traps)

• Audio crashes / asserts → keep -b 2048 -ub 2048 (already in the macro). Default 512 crashes.
• Audio is garbage/repetitive → the mmproj must be BF16 (we pin it). Q8_0/F16 break audio.
• llama-server not found in llama-swap → adjust the binary path in config.tmpl.yaml / entrypoint.sh.
• llama-swap flags differ → check docker run --rm <llamaswap-image> llama-swap --help; fix entrypoint.sh.
• Audio “No audio found” in the Pipe → Open WebUI's file-store API differs by version; adapt _resolve_local_path() in the Pipe.
• Embeddings fail / wrong model → set every RAG_* env explicitly (Open WebUI doesn't inherit OPENAI_*); verify in Admin → Documents. Hakim won't load? set TEI_EMBED_MODEL=BAAI/bge-m3 and TEI_EMBED_POOLING=cls.
• Reranker freezes UI / format mismatch → fall back to Open WebUI's built-in reranker: remove the RAG_RERANKING_ENGINE=external + URL envs and set RAG_RERANKING_MODEL=BAAI/bge-reranker-v2-m3.

Operational notes (validated live on a 15 GB CPU box)

• Gemma 4 is a reasoning model — by default it emits reasoning_content before content and, unbounded, can think for minutes before a visible answer. The macro therefore ships --reasoning off (clean direct answers). If you re-enable it, prefer --reasoning-budget N over unbounded, give max_tokens ≥ 256, and read both fields in raw API callers (--reasoning-budget 0 leaks "Thinking Process:" prose into content — use off).
• Send media via a file, not inline — base64 audio/large images exceed the shell arg limit (argument list too long). Use curl -d @payload.json (the e2e script and Pipe already do).
• TEI memory on CPU: the ONNX backend is heavy (bge-m3 ≈ 4.75 GB). Setting TEI_DTYPE=float16 makes ONNX bow out and TEI falls back to the lighter Candle backend (≈ 3.4 GB) — ~2.4 GB saved, same 1024-dim Persian output. That's why --dtype float16 is in the compose commands.
• Tight-RAM tuning (this box): dev profile uses CTX=4096; E4B is loaded by local path (-m /models/...gguf) so a studio pause/resume doesn't re-download 5 GB. E4B + bge-m3 + reranker + UI ≈ 13–14 GB — fits 15 GB but with little headroom; 16–24 GB is comfortable.
• Verified end-to-end here: Persian text ✓, image→“Red” ✓, native audio→exact transcript ✓, Persian embeddings (1024-d) ✓, Persian rerank ordering ✓, Open WebUI ✓.

Layout

docker-compose.yml · docker-compose.gpu.yml · .env.example
profiles/{dev-cpu,cpu-server,cpu-server-xl,gpu}.env
llama-swap/{config.tmpl.yaml,entrypoint.sh}
openwebui/functions/gemma4_audio_pipe.py
scripts/{download-models.sh,bench.sh}

Full design rationale: ../.claude/plans/i-want-make-a-ancient-teapot.md.