jpanasuk/tabby-tavern-stack

Tabby-Tavern AI Stack

Complete containerized local AI infrastructure for a private lab:

Layer	Software
High-perf inference	TabbyAPI + EXL3 / ExLlamaV3
Character chat UI	SillyTavern
General LLM workspace	Open WebUI
GGUF backend	Ollama
Private search	SearXNG (+ Redis cache)

This is a private-lab / portfolio stack — production-shaped, not a multi-tenant SaaS product and not a hosted inference endpoint.

Lab baseline hardware: NVIDIA GeForce RTX 4070 (Linux + Docker Compose + NVIDIA Container Toolkit).

Surface	URL
This HF card	https://huggingface.co/jpanasuk/tabby-tavern-stack
GitHub source of truth	https://github.com/jpanasuk-netizen/tabby-tavern-stack
Sell sheet (Space)	https://huggingface.co/spaces/jpanasuk/tabby-tavern-sell-sheet
Telemetry / benchmarks	https://github.com/jpanasuk-netizen/local_grid_suite
Multi-agent prototype	https://github.com/jpanasuk-netizen/multi-agent-dungeon-crawler
Author	https://huggingface.co/jpanasuk · @jpanasuk-netizen

Weights are not included. You download EXL3 and/or GGUF models yourself.
Secrets are not included. Copy examples and generate your own keys.

---

Service ports (compose defaults)

Compose service	Container name (default)	Host port	In-container	Role
sillytavern	tabby-tavern-sillytavern-1	8000	8000	Chat / character frontend
open-webui	tabby-tavern-open-webui-1	3000	8080	General LLM workspace
tabbyapi	tabby-tavern-tabbyapi-1	5000	5000	EXL3 high-performance inference
ollama	tabby-tavern-ollama-1	11434	11434	GGUF model backend
searxng	tabby-tavern-searxng-1	8080	8080	Private metasearch
searxng-redis	tabby-tavern-searxng-redis-1	(internal)	6379	SearXNG cache

Open WebUI is wired to Ollama via compose env:

OLLAMA_BASE_URL=http://ollama:11434

All services share the bridge network ai-network.

---

System requirements

Host

• OS: Linux native, or Windows WSL2 with Docker Desktop / Docker Engine
• GPU: NVIDIA GPU with recent drivers (lab used RTX 4070)
• Docker: Docker Engine + Compose plugin (docker compose version)
• NVIDIA Container Toolkit (required for GPU passthrough into containers)
• Disk: room for Docker images plus model weights (EXL3 8B class is often tens of GB; GGUF varies by quant)
• RAM / VRAM: 8B-class EXL3 fits a 12 GB class card with headroom when KV/cache is tuned; larger models need more VRAM

One-time NVIDIA Container Toolkit check

# Driver visible on host
nvidia-smi

# Toolkit installed (package name varies by distro)
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

# GPU visible inside a throwaway container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If nvidia-smi works on the host but fails in Docker, fix the toolkit / Docker daemon GPU runtime before bringing the stack up.

Optional host tools

# Hugging Face CLI (model download)
pipx install "huggingface_hub[cli]"
# or: python3 -m venv .venv && . .venv/bin/activate && pip install -U "huggingface_hub[cli]"

---

Cold start (stranger path)

Prefer cloning GitHub for day-to-day work. This HF repo is the public card + sanitized mirror of the same layout.

0) Clone

git clone https://github.com/jpanasuk-netizen/tabby-tavern-stack.git
cd tabby-tavern-stack

Alternate (HF mirror — same public layout, may lag GitHub):

git lfs install   # only if you later pull large assets; weights still not shipped
git clone https://huggingface.co/jpanasuk/tabby-tavern-stack
cd tabby-tavern-stack

1) Build the TabbyAPI image used by compose

Compose expects this exact local tag:

docker build -f Dockerfile.tabby -t local/tabbyapi:exl3-fixed .

Dockerfile.tabby is a thin layer on ghcr.io/theroyallab/tabbyapi:latest (adds build deps used by the lab image path).
There is also a fuller Dockerfile with CUDA/EXL3 env defaults baked in — use it if you prefer image-level defaults:

# optional alternative image build
docker build -f Dockerfile -t local/tabbyapi:exl3-fixed .

2A) Download an EXL3 model for TabbyAPI

Weights go under tabby_models/ (gitignored). Compose mounts:

./tabby_models  →  /app/models   (inside tabbyapi)

TabbyAPI model_name is a directory name under that mount.

mkdir -p tabby_models

# Example EXL3 pack (pick a revision/quant that fits your VRAM)
huggingface-cli download turboderp/Llama-3.1-8B-Instruct-exl3 \
  --revision 6.0bpw \
  --local-dir tabby_models/Llama-3.1-8B-Instruct-6.0bpw-exl3

# Confirm the directory exists and contains model files
ls -la tabby_models/Llama-3.1-8B-Instruct-6.0bpw-exl3 | head

Other EXL3 / ExLlama-family repos work the same way: download into tabby_models/<folder-name>/ and point config at <folder-name>.

2B) Configure TabbyAPI (required before first launch)

# Prefer the sanitized example if present
cp -n tabby_config/config.example.yml tabby_config/config.yml

# If you only have a minimal stub, create a working private-lab config:
cat > tabby_config/config.yml <<'EOF'
network:
  host: 0.0.0.0
  port: 5000
  disable_auth: false

# Generate your own keys — never reuse published historical keys
# python -c "import secrets; print(secrets.token_hex(32))"
security:
  admin_key: "REPLACE_WITH_LONG_RANDOM_ADMIN_KEY"
  api_key: "REPLACE_WITH_LONG_RANDOM_API_KEY"

developer:
  disable_auth: false

model:
  model_dir: models
  # MUST match the directory name under ./tabby_models/
  model_name: Llama-3.1-8B-Instruct-6.0bpw-exl3
  max_seq_len: 8192
  cache_size: 8192
EOF

Generate real keys:

python3 -c "import secrets; print('admin_key:', secrets.token_hex(32)); print('api_key:  ', secrets.token_hex(32))"
# paste into tabby_config/config.yml

Optional API token file (create locally; do not commit):

# compose has an optional mount (commented by default):
# - ./tabby_config/api_tokens.yml:/app/api_tokens.yml

Compose always mounts:

./tabby_config/config.yml  →  /app/config.yml

2C) Ollama / GGUF path (secondary backend)

Ollama data persists in ./ollama_data → /root/.ollama.

After the stack is up:

# Pull a small GGUF-class model through the running container
docker exec -it tabby-tavern-ollama-1 ollama pull qwen2.5:7b

# List models
docker exec -it tabby-tavern-ollama-1 ollama list

# Smoke test generation
docker exec -it tabby-tavern-ollama-1 ollama run qwen2.5:7b "Say hello in one sentence."

Open WebUI (http://localhost:3000) should discover Ollama automatically via OLLAMA_BASE_URL=http://ollama:11434.

You can run TabbyAPI (EXL3) and Ollama (GGUF) together; just remember they both want GPU VRAM — load one heavy model at a time on a single 12 GB class card unless you know your headroom.

3) Frontend / search secrets (placeholders only)

SillyTavern (sillytavern_config/config.yaml):

• Default listen port 8000
• Lab compose sets WHITELIST_MODE=false via environment
• If basic auth is enabled in the YAML, replace:

basicAuthUser:
  username: "YOUR_ST_USERNAME_HERE"
  password: "YOUR_ST_PASSWORD_HERE"

SearXNG (searxng_config/settings.yml):

use_default_settings: true
server:
  secret_key: "YOUR_SEARXNG_SECRET_KEY_HERE"
  image_proxy: true

Replace secret_key with a long random string before any network exposure:

python3 -c "import secrets; print(secrets.token_hex(32))"

4) Launch

docker compose up -d
docker compose ps

Or:

chmod +x start-stack.sh load-model.sh
./start-stack.sh

start-stack.sh runs docker compose down || true then docker compose up -d and prints the service URLs.

5) Health checks (every service)

# Compose view
docker compose ps
docker compose logs --tail=80 tabbyapi
docker compose logs --tail=40 ollama
docker compose logs --tail=40 sillytavern
docker compose logs --tail=40 open-webui
docker compose logs --tail=40 searxng

# HTTP probes (expect non-connection-refused; exact status varies by app)
curl -sS -o /dev/null -w "sillytavern  %{http_code}\n" http://127.0.0.1:8000/ || true
curl -sS -o /dev/null -w "open-webui   %{http_code}\n" http://127.0.0.1:3000/ || true
curl -sS -o /dev/null -w "tabbyapi     %{http_code}\n" http://127.0.0.1:5000/ || true
curl -sS -o /dev/null -w "ollama       %{http_code}\n" http://127.0.0.1:11434/ || true
curl -sS -o /dev/null -w "searxng      %{http_code}\n" http://127.0.0.1:8080/ || true

# Ollama API
curl -sS http://127.0.0.1:11434/api/tags | head -c 400; echo

# GPU inside TabbyAPI container
docker exec -it tabby-tavern-tabbyapi-1 nvidia-smi || \
  docker exec -it tabby-tavern-tabbyapi-1 bash -lc 'nvidia-smi' || true

Browser targets after green probes:

• SillyTavern → http://localhost:8000
• Open WebUI → http://localhost:3000
• TabbyAPI → http://localhost:5000
• Ollama → http://localhost:11434
• SearXNG → http://localhost:8080

6) Point SillyTavern at TabbyAPI

In SillyTavern API settings (UI):

• API type: OpenAI-compatible / TabbyAPI (as offered by your ST build)
• Endpoint: http://tabbyapi:5000 from another container, or http://127.0.0.1:5000 from the host browser depending on how ST is configured
• API key: the api_key you put in tabby_config/config.yml

Cross-container auth/whitelist issues were a real lab pain point — see DEVLOG.md. If ST cannot reach TabbyAPI, check:

1. Both containers on ai-network (docker network inspect / docker compose ps)
2. TabbyAPI keys match what ST sends
3. TabbyAPI finished loading the EXL3 model (docker compose logs -f tabbyapi)

7) Switch / reload TabbyAPI model

# Edit model_name in tabby_config/config.yml to another folder under tabby_models/
docker compose restart tabbyapi
docker compose logs -f tabbyapi

Note: load-model.sh in older trees still hardcodes a host path like ~/tabby-tavern/.... Prefer editing tabby_config/config.yml + docker compose restart tabbyapi from the repo root, or fix the script paths to $PWD before using it.

---

GPU / compose tuning (what the lab actually ships)

From docker-compose.yml on the GPU services (tabbyapi, ollama):

Setting	Value	Why
deploy.resources.reservations.devices	NVIDIA GPU count: all	Passthrough
shm_size	16g	Avoid shared-memory OOM during load/attn
CUDA_VISIBLE_DEVICES	0	Pin first GPU
CUDA_DEVICE_ORDER	PCI_BUS_ID	Stable device ordering
PYTORCH_CUDA_ALLOC_CONF	max_split_size_mb:128	Allocator fragmentation control
EXLLAMA_GPU_LAYERS	999	Prefer full GPU offload
EXLLAMA_KV_CACHE	q8_0	KV cache quant tradeoff
EXLLAMA_FLASH_ATTENTION	1	Flash-attn path when available
OLLAMA_FLASH_ATTENTION	1	Ollama flash-attn
OLLAMA_KV_CACHE_TYPE	q8_0	Ollama KV quant
OLLAMA_NUM_PARALLEL	1	Single parallel slot (VRAM stability)

If you OOM: lower max_seq_len / cache_size, use a smaller bpw EXL3, or stop Ollama models while TabbyAPI holds a large model.

---

Repository structure

tabby-tavern-stack/
├── docker-compose.yml          # full lab orchestration (6 services)
├── Dockerfile                  # TabbyAPI image + CUDA/EXL3 ENV defaults
├── Dockerfile.tabby            # thin TabbyAPI image used by default compose tag
├── start-stack.sh              # down + up + print URLs
├── load-model.sh               # helper (verify paths before use)
├── tabby_config/
│   ├── config.example.yml      # sanitized template (prefer this)
│   └── config.yml              # local runtime config (do not publish real keys)
├── tabby_models/               # EXL3 weights (gitignored contents)
├── sillytavern_config/         # ST server config (placeholders for auth)
├── sillytavern_data/           # ST user data (secrets gitignored)
├── openwebui_data/             # Open WebUI persistence (DB gitignored)
├── ollama_data/                # Ollama models/keys (gitignored)
├── searxng_config/             # settings.yml (placeholder secret_key)
├── searxng/                    # optional extra searxng assets
├── Clean_Sampler.json          # sampler preset helper
├── SECURITY.md
├── DEVLOG.md                   # engineering history
├── LICENSE
└── docs/                       # sell-sheet / extras (when present)

Compose service names (authoritative):

tabbyapi · sillytavern · ollama · open-webui · searxng · searxng-redis

---

Environment & secrets guidance

Secret / file	Where	Rule
TabbyAPI admin_key / api_key	tabby_config/config.yml	Generate yourself; never commit live values
TabbyAPI api_tokens.yml	tabby_config/ (optional mount)	Local only
SillyTavern basic auth	sillytavern_config/config.yaml	Replace YOUR_ST_* placeholders
SillyTavern cookie/secrets	under sillytavern_data/	gitignored
SearXNG secret_key	searxng_config/settings.yml	Replace placeholder
Open WebUI DB	openwebui_data/webui.db*	gitignored
Ollama keys/models	ollama_data/	gitignored
.env	repo root	gitignored if created

Private-lab defaults are intentional. This is not hardened multi-tenant hosting.

Before any LAN/WAN exposure:

1. Replace every placeholder credential
2. Prefer binding host ports to 127.0.0.1 (edit compose ports: to "127.0.0.1:8000:8000", etc.)
3. Put a reverse proxy + TLS in front if you leave the machine
4. Read SECURITY.md

Older public revisions of this mirror may have contained lab convenience keys. Treat any key you ever saw in a public file as burned and rotate it everywhere it might have been reused.

---

Common failure modes

Symptom	Likely cause	Fix
local/tabbyapi:exl3-fixed pull/start fails	Image never built	docker build -f Dockerfile.tabby -t local/tabbyapi:exl3-fixed .
TabbyAPI exits / cannot find model	model_name ≠ folder under tabby_models/	Align names; confirm mount ./tabby_models:/app/models
CUDA / GPU errors in container	Toolkit missing or no --gpus path	Fix NVIDIA Container Toolkit; docker run --gpus all ... nvidia-smi
OOM during load	Model + KV + dual backends too large	Smaller quant; lower max_seq_len/cache_size; unload Ollama models
ST cannot talk to TabbyAPI	Auth/network mismatch	Same compose network; matching API key; wait for model load
Open WebUI shows no models	Ollama empty or URL wrong	ollama pull ... in ollama container; confirm OLLAMA_BASE_URL=http://ollama:11434
SearXNG unhealthy / 500s	Placeholder/missing secret_key or Redis down	Set secret; docker compose logs searxng searxng-redis
Port already allocated	Host process using 3000/5000/8000/8080/11434	Stop conflicting service or remap compose ports
load-model.sh edits wrong file	Hardcoded ~/tabby-tavern path	Edit tabby_config/config.yml in-repo instead
WSL2 GPU not visible	Windows driver / WSL GPU support	Update NVIDIA Windows driver; confirm nvidia-smi inside WSL

---

Measured lab results (defensible only)

Numbers from checked-in sample telemetry in local_grid_suite (benchmarks/sample_hardware_runs.json, methodology in benchmarks/METHODOLOGY.md). Single-box lab runs — not a product SLA.

Stage	Model	Decode tok/s
Baseline	qwen3:8b	1.39
GPU-routed	qwen-gpu:latest	29.7 – 39.3
Stabilized	qwen-gpu:latest	37.47

→ ~27× decode uplift on that run series after GPU routing / tuning.

Warm stream suite on qwen3:8b: ~76 tok/s (400-token runs).

Re-measure on your hardware. Do not advertise these as guaranteed throughput.

---

Engineering notes

See DEVLOG.md for the real build log:

• Compose consolidation (TabbyAPI + ST + Open WebUI + Ollama + SearXNG + Redis)
• EXL3 adoption vs earlier experiments
• GPU env tuning (shm_size, flash-attn / KV cache flags)
• Container-to-TabbyAPI auth / whitelist fixes

---

Related spine

Project	Link
Stack (this card)	https://huggingface.co/jpanasuk/tabby-tavern-stack
Stack (GitHub)	https://github.com/jpanasuk-netizen/tabby-tavern-stack
Sell sheet Space	https://huggingface.co/spaces/jpanasuk/tabby-tavern-sell-sheet
Local Grid Suite	https://github.com/jpanasuk-netizen/local_grid_suite
Multi-agent prototype	https://github.com/jpanasuk-netizen/multi-agent-dungeon-crawler
Collection	https://huggingface.co/collections/jpanasuk/independent-ai-lab-spine-6a727803ed9c6d16164f5130

---

Author

Jeremy Panasuk — enterprise data / platform background; Aug 2024–present independent local-AI systems year (private Docker LLM lab, decode telemetry, multi-agent prototypes).

• GitHub: @jpanasuk-netizen
• Hugging Face: jpanasuk
• LinkedIn: jeremy-p-34203322

License

MIT