File size: 4,494 Bytes
4ef91a7 a34c407 33357aa a34c407 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | # Tabby-Tavern Development Log
Engineering history for the containerized local AI lab.
## Hardware baseline
- GPU: NVIDIA GeForce RTX 4070
- Environment: Linux + Docker Compose with NVIDIA GPU passthrough
- Primary inference path: TabbyAPI + EXL3 / ExLlamaV3
- Secondary path: Ollama (GGUF)
## Week 1 — Core integration
- Consolidated TabbyAPI, SillyTavern, Open WebUI, Ollama, SearXNG (+ Redis) into one compose file
- Adopted EXL3 weights for faster VRAM load vs earlier experiments
- Fixed cross-container TabbyAPI auth/whitelist failures that blocked peer services
- Added persistent volume mounts for configs and data directories
## Week 2 — Model bring-up & GPU tuning
- Loaded Llama-3.1-8B-Instruct EXL3 (6.0 bpw class) into `tabby_models/`
- Tuned container env: flash attention / KV cache flags, `shm_size: 16g`, CUDA device ordering
- Documented start/stop operational loop (`docker compose down && up -d`)
## Week 3 — Public packaging
- Created sanitized publish tree (no weights, no user DBs, no tokens)
- Added SECURITY.md and example TabbyAPI config
- Mirrored narrative to GitHub portfolio rebuild (Aug 2026)
## Week 4 — WSL2 fresh install + MCPO MCP server (Aug 2026)
Full from-scratch rebuild on clean WSL2 Ubuntu 24.04 (same RTX 4070). Documented every fix in the README WSL2 field notes section.
### Issues hit and resolved
- **EXL2 dropped from TabbyAPI**: The `local/tabbyapi:exl3-fixed` image (built on `ghcr.io/theroyallab/tabbyapi:latest`) no longer supports EXL2-quantized models — only EXL3. Switched to `turboderp/Llama-3.1-8B-Instruct-exl3` 6.0bpw branch.
- **NVIDIA Container Toolkit repo URL**: The standard `deb ... noble main` format fails (`Malformed entry (Component)`). NVIDIA uses a flat repo structure — correct format is `deb [arch=amd64 signed-by=...] https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /`.
- **libcuda.so missing in container**: Triton JIT compiler needs `-lcuda` at link time but WSL2 CUDA driver libs aren't in the standard linker path. Fixed by adding `RUN ln -sf /usr/local/cuda-12.8/compat/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so && ldconfig` to the Dockerfile.
- **Port conflict with host Ollama**: Host Ollama already on 11434. Remapped Docker Ollama to 11435.
- **Open WebUI only wired to Ollama**: Added `OPENAI_API_BASE_URL=http://tabbyapi:5000/v1` and `OPENAI_API_KEY` env vars so Open WebUI can use both TabbyAPI EXL3 and Ollama GGUF models.
- **SillyTavern browser launch in Docker**: Disabled `browserLaunch.enabled` (no browser in container). Kept `basicAuthMode: true` (admin/tabby) — ST refuses to start without auth when listening on 0.0.0.0.
- **SearXNG JSON format**: Default config only returns HTML. Added `json` to `search.formats` for API access.
- **TabbyAPI config tuning**: Set `cache_8bit: true`, `max_seq_len: 8192`, `cache_size: 8192` for 12GB VRAM headroom. Measured ~8GB VRAM with 6.0bpw EXL3 + 8-bit cache.
### MCPO MCP server — built and integrated
- Built a proper MCP server (`mcp-servers/server.py`) using `FastMCP` from the `mcp` library (not raw JSON-RPC)
- Server exposes 7 tools via MCPO as OpenAPI endpoints at `/host-master/`:
- `list_tabbyapi_models` — query TabbyAPI for loaded EXL3 models
- `tabbyapi_chat` — send chat to TabbyAPI EXL3 model
- `list_ollama_models` — query Ollama for GGUF models
- `ollama_pull_model` — pull a model into Ollama
- `ollama_chat` — send chat to an Ollama model
- `get_stack_status` — health check all 5 services
- `searxng_search` — web search via SearXNG
- MCPO config (`mcpo/config.json`) uses `/app/.venv/bin/python3` (the MCPO container's venv has `mcp` library) with env vars for inter-container URLs
- All 7 tools verified working: 14/14 checks passed
- Swagger docs available at `http://localhost:8001/docs`
- API key for MCPO: `mcp-secret-key` (set in docker-compose command args)
### Final verified state (24/24 checks passing)
- All 6 containers running, no restart loops
- TabbyAPI: EXL3 model loaded, chat at ~53 tok/s processing
- Ollama: `llama3.1:8b` (4GB) pulled
- Open WebUI: healthy, connected to both TabbyAPI and Ollama
- SillyTavern: running with basic auth
- SearXNG: serving with JSON format
- MCPO: connected, 7 tool endpoints live
- GPU: 8GB/12GB VRAM on RTX 4070
## Open follow-ups
- One-command bootstrap that builds the TabbyAPI image + prints next model download
- Healthcheck targets in compose
- Optional Traefik/Caddy reverse-proxy profile for LAN HTTPS
|