Add lazy two-model router and detached ngrok setup

.gitignore

@@ -7,3 +7,4 @@ results/
 __pycache__/
 .venv/
 .venv_lib/
+/ngrok

README.md

@@ -13,12 +13,12 @@ Reproducible OpenAI-compatible local API for a single NVIDIA A100 40 GB GPU. It
 
 Both files take about 49.02 GB of disk space. The CUDA archives take another 0.76 GB until removed. Only one model can be resident on a 40 GB A100 at a time. Model weights, binaries, logs, benchmark outputs, and API keys are excluded from this repo. The installer uses Hugging Face Hub/Xet for chunked model downloads at pinned revisions and verifies the published SHA-256 digests.
 
-An optional 3.16 GB Qwen3.8 MTP draft can be installed with `WITH_MTP=1 bash scripts/install.sh` and enabled with `LLM_SPEC_MTP=1 scripts/start.sh qwen38`. It is off by default until its speed and output are checked on this GPU.
+An optional 3.16 GB Qwen3.8 MTP draft can be installed with `WITH_MTP=1 bash scripts/install.sh`. The two-model router leaves speculative decoding off by default.
 
 ## Requirements
 
 - Linux x86-64 with an NVIDIA A100 40 GB and a CUDA 12.8-compatible driver.
-- `bash`, `curl`, `tar`, `sha256sum`, and Python 3.
+- `bash`, `curl`, `tar`, `sha256sum`, `setsid`, `nohup`, `taskset`, and Python 3.
 - The pinned prebuilt binary needs `libgomp.so.1`. This machine provides it in `/opt/conda/lib`; adjust `LD_LIBRARY_PATH` in `serve.sh` on other machines if needed.
 - Approximately 55 GB of free storage and internet access for the initial download. Python `venv` and `pip` are needed for the pinned Hub/Xet downloader.
 
@@ -28,15 +28,15 @@ An optional 3.16 GB Qwen3.8 MTP draft can be installed with `WITH_MTP=1 bash scr
 git clone https://huggingface.co/ShoaibSSM/a100-qwen-api-runtime
 cd a100-qwen-api-runtime
 bash scripts/install.sh
-scripts/start.sh qwen38
+scripts/start.sh
 ```
 
-`serve.sh` generates a random API key in `secrets/api_keys.txt` (mode 0600) if needed. It binds to `127.0.0.1:8080` by default. Check readiness with `curl http://127.0.0.1:8080/health` and logs with `tail -f logs/server.log`. Select the other model by running `scripts/stop.sh` and then `scripts/start.sh qwen36`. To keep logs in your terminal, run `scripts/serve.sh qwen38` instead.
+`start.sh` uses `nohup` and `setsid`, so closing Codex or the shell does not stop the API process while the container is running. It generates a random API key in `secrets/api_keys.txt` (mode 0600) if needed and binds to `127.0.0.1:8080`. Both models start unloaded. A request with `"model":"qwen38"` or `"model":"qwen36"` loads it on demand; at most one model runs at once. After 15 minutes without inference work, the model sleeps and releases its memory. The next request wakes it. Switching or waking models incurs a cold-load delay. Check readiness with `curl http://127.0.0.1:8080/health`, logs with `tail -f logs/server.log`, and stop with `scripts/stop.sh`. Change the idle timer with `LLM_IDLE_SECONDS=600 scripts/start.sh`.
 
-The starting configuration uses full GPU offload, flash attention, Q8 KV cache, 65,536 total context tokens, one request slot, and 2048/512 batch sizes. GPU offload is forced so a model that does not fit fails visibly instead of becoming slow through CPU offloading. For larger contexts or concurrent requests, adjust `LLM_CONTEXT` and `LLM_PARALLEL` and measure memory use and latency. The claimed model context lengths are *not* measured throughput guarantees on a 40 GB GPU.
+The starting configuration uses full GPU offload, flash attention, Q8 KV cache, 65,536 total context tokens, one request slot, medium reasoning effort, and 2048/512 batch sizes. GPU offload is forced so a model that does not fit fails visibly instead of becoming slow through CPU offloading. For larger contexts or concurrent requests, adjust `LLM_CONTEXT` and `LLM_PARALLEL` and measure memory use and latency. Set `LLM_REASONING_EFFORT=xhigh` for harder tasks and allow enough `max_tokens` for both reasoning and the final answer. The claimed model context lengths are *not* measured throughput guarantees on a 40 GB GPU.
 
 ```bash
-LLM_CONTEXT=32768 LLM_PARALLEL=2 scripts/serve.sh qwen38
+LLM_CONTEXT=32768 LLM_PARALLEL=2 scripts/start.sh
 ```
 
 ## Use the API
@@ -49,18 +49,43 @@ curl -sS http://127.0.0.1:8080/v1/chat/completions \
   -d '{"model":"qwen38","messages":[{"role":"user","content":"Explain this algorithm."}],"max_tokens":512}'
 ```
 
-With the OpenAI Python client, set `base_url="http://127.0.0.1:8080/v1"`, `api_key` to the generated key, and `model` to `qwen38` or `qwen36` depending on which server is running. For access from another machine, use an SSH tunnel such as `ssh -L 8080:127.0.0.1:8080 user@server`. If you intentionally bind a public interface with `LLM_HOST=0.0.0.0`, put TLS in front of it so API keys are not sent over plain HTTP.
+With the OpenAI Python client, set `base_url="http://127.0.0.1:8080/v1"`, `api_key` to the generated key, and `model` to `qwen38` or `qwen36` per request.
+
+## Reach it through ngrok
+
+From another shell in this directory, install the standalone [ngrok Linux agent](https://ngrok.com/download/linux) and add your own ngrok account authtoken. This token is distinct from the LLM API key.
+
+```bash
+curl -fsSL https://bin.ngrok.com/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz | tar -xz
+read -rsp 'ngrok authtoken: ' NGROK_AUTHTOKEN; echo
+./ngrok config add-authtoken "$NGROK_AUTHTOKEN"
+unset NGROK_AUTHTOKEN
+GOMAXPROCS=2 nohup setsid ./ngrok http 8080 --log=stdout > logs/ngrok.log 2>&1 < /dev/null &
+echo $! > runtime/ngrok.pid
+```
+
+Read the HTTPS URL with `curl -s http://127.0.0.1:4040/api/tunnels | python -c 'import json,sys; print(next(t["public_url"] for t in json.load(sys.stdin)["tunnels"] if t["public_url"].startswith("https://")))'`. Use that URL plus `/v1/chat/completions` from the other machine with the `Authorization: Bearer <LLM API key>` header. The API and tunnel persist after the shell closes, but both stop if the container stops. The tunnel URL may change on restart. Stop the tunnel with `kill "$(cat runtime/ngrok.pid)"`.
+
+On your laptop, set `LLM_URL` to that HTTPS URL and `LLM_API_KEY` to the contents of `secrets/api_keys.txt` from this server:
+
+```bash
+export LLM_URL='https://YOUR-TUNNEL.ngrok-free.app'
+read -rsp 'LLM API key: ' LLM_API_KEY; echo
+export LLM_API_KEY
+curl -sS "$LLM_URL/v1/chat/completions" \
+  -H "Authorization: Bearer $LLM_API_KEY" \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"qwen38","messages":[{"role":"user","content":"Hello"}],"max_tokens":512}'
+```
+
+For an OpenAI-compatible client: `base_url="$LLM_URL/v1"`, `api_key="$LLM_API_KEY"`, and `model="qwen38"` or `model="qwen36"`. The first call after sleep, or a model switch, can take longer while weights load; give the client at least a 180-second timeout.
 
 ## Compare on this hardware
 
-Run each model alone, then run the same API timing prompts:
+Run the same API timing prompts through the router if you want a local comparison:
 
 ```bash
-scripts/start.sh qwen38
 python scripts/bench.py qwen38
-# Stop the server, then:
-scripts/stop.sh
-scripts/start.sh qwen36
 python scripts/bench.py qwen36
 ```
 

scripts/bench.py

@@ -19,7 +19,7 @@ def main():
     p = argparse.ArgumentParser()
     p.add_argument("model", choices=["qwen38", "qwen36"])
     p.add_argument("--base-url", default="http://127.0.0.1:8080")
-    p.add_argument("--max-tokens", type=int, default=1200)
+    p.add_argument("--max-tokens", type=int, default=4096)
     args = p.parse_args()
     key = os.environ.get("LLM_API_KEY") or (ROOT / "secrets/api_keys.txt").read_text().strip()
     records = []
@@ -34,6 +34,7 @@ def main():
                 "temperature": 1.0,
                 "top_p": 0.95,
                 "max_tokens": args.max_tokens,
+                "reasoning_effort": "medium",
                 "stream": False,
             },
             timeout=900,

scripts/router.sh

@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+QWEN38="$ROOT/models/Qwen3.8-27B-Q6_K.gguf"
+QWEN36="$ROOT/models/Qwen3.6-35B-A3B-UD-Q5_K_M.gguf"
+[[ -f "$QWEN38" && "$(stat -c %s "$QWEN38")" == 22563871392 ]] || { echo "Missing or incomplete: $QWEN38" >&2; exit 1; }
+[[ -f "$QWEN36" && "$(stat -c %s "$QWEN36")" == 26456194016 ]] || { echo "Missing or incomplete: $QWEN36" >&2; exit 1; }
+
+BIN="$ROOT/runtime/b11177/llama-server"
+[[ -x "$BIN" ]] || BIN="$ROOT/runtime/b11177/build/bin/llama-server"
+[[ -x "$BIN" ]] || { echo 'llama-server missing. Run bash scripts/install.sh.' >&2; exit 1; }
+
+mkdir -p "$ROOT/secrets" "$ROOT/runtime"
+KEY_FILE="$ROOT/secrets/api_keys.txt"
+if [[ ! -s "$KEY_FILE" ]]; then
+  umask 077
+  python -c 'import secrets; print(secrets.token_urlsafe(32))' > "$KEY_FILE"
+fi
+chmod 600 "$KEY_FILE"
+
+PRESET="$ROOT/runtime/models.ini"
+cat > "$PRESET" <<EOF
+version = 1
+
+[qwen38]
+model = $QWEN38
+
+[qwen36]
+model = $QWEN36
+EOF
+
+export LD_LIBRARY_PATH="$ROOT/runtime/b11177:/opt/conda/lib:${LD_LIBRARY_PATH:-}"
+CPU_SET="$(python -c 'import os; print(",".join(map(str, sorted(os.sched_getaffinity(0))[:int(os.getenv("LLM_CPU_CORES", "32"))])))')"
+exec taskset -c "$CPU_SET" "$BIN" \
+  --models-preset "$PRESET" \
+  --models-max 1 \
+  --models-autoload \
+  --sleep-idle-seconds "${LLM_IDLE_SECONDS:-900}" \
+  --host "${LLM_HOST:-127.0.0.1}" \
+  --port "${LLM_PORT:-8080}" \
+  --api-key-file "$KEY_FILE" \
+  --n-gpu-layers all \
+  --fit off \
+  --flash-attn on \
+  --cache-type-k q8_0 \
+  --cache-type-v q8_0 \
+  --ctx-size "${LLM_CONTEXT:-65536}" \
+  --parallel "${LLM_PARALLEL:-1}" \
+  --threads "${LLM_THREADS:-8}" \
+  --threads-batch "${LLM_THREADS_BATCH:-16}" \
+  --reasoning-effort "${LLM_REASONING_EFFORT:-medium}" \
+  --batch-size 2048 \
+  --ubatch-size 512 \
+  --jinja

scripts/serve.sh

@@ -48,6 +48,7 @@ exec "$BIN" \
   --cache-type-v q8_0 \
   --ctx-size "${LLM_CONTEXT:-65536}" \
   --parallel "${LLM_PARALLEL:-1}" \
+  --reasoning-effort "${LLM_REASONING_EFFORT:-medium}" \
   --batch-size 2048 \
   --ubatch-size 512 \
   --jinja \

scripts/start.sh

@@ -1,14 +1,13 @@
 #!/usr/bin/env bash
 set -euo pipefail
 ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-MODEL="${1:-qwen38}"
 PID_FILE="$ROOT/runtime/server.pid"
 mkdir -p "$ROOT/logs" "$ROOT/runtime"
 if [[ -f "$PID_FILE" ]] && kill -0 "$(cat "$PID_FILE")" 2>/dev/null; then
   echo "Server already running (PID $(cat "$PID_FILE"))." >&2
   exit 1
 fi
-nohup "$ROOT/scripts/serve.sh" "$MODEL" > "$ROOT/logs/server.log" 2>&1 < /dev/null &
+nohup setsid "$ROOT/scripts/router.sh" > "$ROOT/logs/server.log" 2>&1 < /dev/null &
 echo "$!" > "$PID_FILE"
-echo "Started $MODEL (PID $(cat "$PID_FILE")). Log: $ROOT/logs/server.log"
+echo "Started router (PID $(cat "$PID_FILE")). Log: $ROOT/logs/server.log"
 echo "Health: curl http://127.0.0.1:${LLM_PORT:-8080}/health"