Migrate CPU serving to llama.cpp + Q5_K_M GGUF (was bf16 transformers)

.gitignore

@@ -14,6 +14,16 @@ __pycache__/
 models/phi3-text-to-sql/
 models/phi3-text-to-sql-adapter/
 
+# GGUF conversion intermediates (large; GGUF is hosted on HF Hub)
+models/phi3-text-to-sql-merged/
+*.gguf
+llama.cpp/
+
+# llama.cpp prebuilt release binaries (downloaded locally; never commit)
+llama-b*/
+*.dll
+*.exe
+
 # Local database (regenerated by database.py on cold start)
 data/company_sales.db
 

Dockerfile

@@ -1,23 +1,28 @@
-# Use Python 3.11 slim as base
-# HF Spaces injects NVIDIA runtime for GPU; PyTorch ships its own CUDA libs
+# CPU-only Hugging Face Space image.
+# Serving uses llama.cpp + a pre-merged GGUF (Q4_K_M), so there is no torch /
+# transformers / bitsandbytes in the production image. This keeps the image small,
+# cold starts fast, and RAM well under the 16 GB free-tier ceiling.
 FROM python:3.11-slim
 
-# System deps
+# System deps. build-essential + cmake are a safety net in case pip falls back to
+# building llama-cpp-python from source; normally the prebuilt CPU wheel is used.
 RUN apt-get update && apt-get install -y --no-install-recommends \
     git \
     curl \
+    build-essential \
+    cmake \
     && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /app
 
-# Install PyTorch with CUDA 12.1 first (large layer, cache separately)
+# Install llama-cpp-python from the prebuilt CPU wheel index (avoids a slow source build).
 RUN pip install --no-cache-dir \
-    torch==2.4.1 \
-    --index-url https://download.pytorch.org/whl/cu121
+    "llama-cpp-python>=0.3.0,<0.4.0" \
+    --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
 
-# Install remaining dependencies
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
+# Install the slim CPU-serving dependencies.
+COPY requirements-space.txt .
+RUN pip install --no-cache-dir -r requirements-space.txt
 
 # Copy project source
 COPY . .
@@ -32,7 +37,7 @@ RUN mkdir -p data models templates static
 RUN useradd -m -u 1000 appuser && chown -R appuser /app
 USER appuser
 
-# Point HF cache to a writable location inside the container
+# Point HF cache to a writable location inside the container (GGUF is cached here)
 ENV HF_HOME=/app/.cache/huggingface
 
 CMD ["python", "src/server.py"]

requirements-space.txt

@@ -0,0 +1,10 @@
+# Slim dependency set for the Hugging Face CPU Space (used by the Dockerfile).
+# CPU serving uses llama.cpp + a pre-merged GGUF, so torch/transformers/peft/
+# bitsandbytes are NOT needed in the production image.
+#
+# NOTE: llama-cpp-python is installed separately in the Dockerfile from the
+# prebuilt CPU wheel index to avoid a slow source build.
+
+flask==3.0.3
+pandas==2.2.3
+huggingface_hub>=0.24.0

requirements.txt

@@ -15,3 +15,7 @@ pandas==2.2.3
 
 # HF Hub (model + adapter download)
 huggingface_hub>=0.24.0
+
+# CPU/GGUF inference + local GGUF validation harness.
+# Lazy-imported (CPU path only); the CUDA/laptop training path does not require it at runtime.
+llama-cpp-python>=0.3.0,<0.4.0

scripts/CONVERT_GGUF.md

@@ -0,0 +1,101 @@
+# Convert the fine-tuned model to GGUF (Q4_K_M) — run on your laptop
+
+One-time process. Produces `phi3-text-to-sql-Q4_K_M.gguf` (~2.3 GB), which you
+then push to a Hugging Face model repo. The CPU Space downloads it at startup.
+
+Prerequisites: your trained adapter at `models/phi3-text-to-sql-adapter`, Python
+env with `transformers`, `peft`, `torch` (your existing `.venv`), and `git`.
+
+---
+
+## Step 1 — Merge the adapter into the base model
+
+```bash
+python scripts/merge_adapter.py
+```
+
+Output: `models/phi3-text-to-sql-merged/` (fp16, ~7.6 GB).
+
+## Step 2 — Get llama.cpp (for the converter + quantizer)
+
+```bash
+git clone https://github.com/ggerganov/llama.cpp
+pip install -r llama.cpp/requirements.txt
+```
+
+## Step 3 — Convert merged HF model → GGUF (F16)
+
+```bash
+python llama.cpp/convert_hf_to_gguf.py models/phi3-text-to-sql-merged \
+    --outfile models/phi3-text-to-sql-f16.gguf \
+    --outtype f16
+```
+
+## Step 4 — Quantize F16 → Q4_K_M
+
+You need the `llama-quantize` binary. Two options:
+
+**Option A — prebuilt binary (easiest on Windows):**
+Download the latest llama.cpp release for your OS from
+https://github.com/ggerganov/llama.cpp/releases (it includes `llama-quantize`
+/ `llama-quantize.exe`), then:
+
+```bash
+# Linux/macOS
+./llama-quantize models/phi3-text-to-sql-f16.gguf \
+    models/phi3-text-to-sql-Q4_K_M.gguf Q4_K_M
+```
+```powershell
+# Windows
+.\llama-quantize.exe models\phi3-text-to-sql-merged\..\phi3-text-to-sql-f16.gguf `
+    models\phi3-text-to-sql-Q4_K_M.gguf Q4_K_M
+```
+
+**Option B — build from source:**
+```bash
+cd llama.cpp && cmake -B build && cmake --build build --config Release -j
+# binary at: build/bin/llama-quantize
+./build/bin/llama-quantize ../models/phi3-text-to-sql-f16.gguf \
+    ../models/phi3-text-to-sql-Q4_K_M.gguf Q4_K_M
+```
+
+Result: `models/phi3-text-to-sql-Q4_K_M.gguf` (~2.3 GB).
+
+## Step 5 — Validate quality BEFORE deploying
+
+```bash
+python scripts/validate_gguf.py --gguf models/phi3-text-to-sql-Q4_K_M.gguf
+```
+
+Acceptance bar: ≥ 95% execution-equivalent on `data/test_dataset.jsonl`.
+If it fails, redo Step 4 with `Q5_K_M` (larger/slower, higher fidelity) and
+re-validate.
+
+## Step 6 — Push the GGUF to Hugging Face
+
+The Space pulls from repo `Bhuvandesai/phi3-text-to-sql-gguf`, file
+`phi3-text-to-sql-Q4_K_M.gguf` (override via `GGUF_REPO_ID` / `GGUF_FILENAME`
+env vars on the Space). Create the repo and upload:
+
+```bash
+huggingface-cli repo create phi3-text-to-sql-gguf --type model
+huggingface-cli upload Bhuvandesai/phi3-text-to-sql-gguf \
+    models/phi3-text-to-sql-Q4_K_M.gguf phi3-text-to-sql-Q4_K_M.gguf
+```
+
+(If you used Q5_K_M, set `GGUF_FILENAME=phi3-text-to-sql-Q5_K_M.gguf` in the
+Space settings and upload that file instead.)
+
+## Step 7 — Deploy
+
+Commit and push the repo. The Dockerfile installs llama-cpp-python and the slim
+deps; on boot `src/inference.py` downloads the GGUF and serves on CPU.
+
+---
+
+### Notes
+- Do **not** commit the multi-GB `.gguf` into the Space git repo. Host it on the
+  Hub (Step 6) so it is pulled at runtime and cached. If you ever want it bundled,
+  the loader also accepts a local copy at `models/<GGUF_FILENAME>`.
+- `models/phi3-text-to-sql-merged/`, `*.gguf`, and `llama.cpp/` are intermediate
+  artifacts — keep them out of git (see `.gitignore`).

scripts/merge_adapter.py

@@ -0,0 +1,77 @@
+"""
+Merge the trained LoRA adapter into the Phi-3 base model and save a standalone
+fp16 model. This merged model is the input to GGUF conversion.
+
+Run on your laptop (the base model is already cached there):
+
+    python scripts/merge_adapter.py
+
+Env overrides (optional):
+    BASE_MODEL_ID   default: microsoft/Phi-3-mini-4k-instruct
+    ADAPTER_PATH    default: models/phi3-text-to-sql-adapter
+    MERGED_OUT      default: models/phi3-text-to-sql-merged
+
+Notes:
+- Merging is weight arithmetic (W' = W + (alpha/r) * B@A); it is mathematically
+  equivalent to applying the adapter. Under greedy decoding the SQL output is the
+  same as the current fine-tuned model.
+- fp16 keeps the merged checkpoint ~7.6 GB. GGUF conversion + Q4_K_M quantization
+  shrinks it to ~2.3 GB.
+"""
+import os
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+BASE_MODEL_ID = os.environ.get("BASE_MODEL_ID", "microsoft/Phi-3-mini-4k-instruct")
+ADAPTER_PATH = os.environ.get("ADAPTER_PATH", "models/phi3-text-to-sql-adapter")
+MERGED_OUT = os.environ.get("MERGED_OUT", "models/phi3-text-to-sql-merged")
+
+
+def main():
+    print(f"Base model : {BASE_MODEL_ID}")
+    print(f"Adapter    : {ADAPTER_PATH}")
+    print(f"Output     : {MERGED_OUT}")
+
+    print("\nLoading tokenizer...")
+    tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID, trust_remote_code=False)
+
+    print("Loading base model in fp16 (CPU)...")
+    model = AutoModelForCausalLM.from_pretrained(
+        BASE_MODEL_ID,
+        torch_dtype=torch.float16,
+        device_map=None,
+        low_cpu_mem_usage=True,
+        trust_remote_code=False,
+    )
+
+    print("Attaching adapter...")
+    model = PeftModel.from_pretrained(model, ADAPTER_PATH)
+
+    print("Merging adapter into base weights (merge_and_unload)...")
+    model = model.merge_and_unload()
+
+    os.makedirs(MERGED_OUT, exist_ok=True)
+    print(f"Saving merged model to {MERGED_OUT} ...")
+    model.save_pretrained(MERGED_OUT, safe_serialization=True)
+    tokenizer.save_pretrained(MERGED_OUT)
+
+    # Phi-3's GGUF converter needs the original SentencePiece tokenizer.model,
+    # which save_pretrained() does not emit for the fast tokenizer. Copy it over.
+    tok_model_dst = os.path.join(MERGED_OUT, "tokenizer.model")
+    if not os.path.exists(tok_model_dst):
+        try:
+            import shutil
+            from huggingface_hub import hf_hub_download
+            src = hf_hub_download(BASE_MODEL_ID, "tokenizer.model")
+            shutil.copy(src, tok_model_dst)
+            print("Copied tokenizer.model (required by the GGUF converter).")
+        except Exception as e:
+            print(f"WARNING: could not fetch tokenizer.model ({e}). "
+                  "Copy it manually from the base model before converting.")
+
+    print("\nDone. Next: convert to GGUF (see scripts/CONVERT_GGUF.md).")
+
+
+if __name__ == "__main__":
+    main()

scripts/validate_gguf.py

@@ -0,0 +1,242 @@
+"""
+Validate the GGUF model's SQL quality against the held-out test set BEFORE deploying.
+
+For each example in data/test_dataset.jsonl:
+  1. Generate SQL from the GGUF model (greedy / temperature 0).
+  2. Execute both predicted and expected SQL on data/company_sales.db.
+  3. Count as a match if results are EXECUTION-EQUIVALENT (same rows, order-independent).
+
+Acceptance bar: >= 95% match. If below, re-quantize at Q5_K_M and re-run.
+
+Default backend: llama-server (recommended on Windows/local).
+  The harness launches llama-server.exe (from the llama.cpp release you already
+  downloaded), waits for it to load the model ONCE, then hits its OpenAI-compatible
+  /v1/chat/completions endpoint over HTTP -- the same call shape production uses.
+  This avoids:
+    - the prebuilt llama-cpp-python wheel (illegal-instruction 0xc000001d on CPUs
+      without the AVX-512 it was built for), and
+    - llama-cli interactive/conversation-mode hangs.
+
+Usage:
+    python scripts/validate_gguf.py --gguf models/phi3-text-to-sql-Q4_K_M.gguf
+    python scripts/validate_gguf.py --gguf models/...gguf --server .\\llama-b9637-bin-win-cpu-x64\\llama-server.exe
+    python scripts/validate_gguf.py --gguf models/...gguf --use-python   # only if the wheel runs on your CPU
+
+Requires: data/company_sales.db (run `python src/database.py` first if missing).
+"""
+import argparse
+import glob
+import json
+import os
+import re
+import sqlite3
+import subprocess
+import sys
+import time
+import urllib.error
+import urllib.request
+
+DEFAULT_GGUF = "models/phi3-text-to-sql-Q4_K_M.gguf"
+TEST_PATH = "data/test_dataset.jsonl"
+DB_PATH = "data/company_sales.db"
+
+
+def clean_sql(raw_output):
+    """Same extraction logic as src/inference.py so validation matches serving."""
+    if "<|assistant|>" in raw_output:
+        raw_output = raw_output.split("<|assistant|>")[-1]
+    cleaned = re.sub(r"```sql\s*", "", raw_output, flags=re.IGNORECASE)
+    cleaned = re.sub(r"```", "", cleaned)
+    cleaned = re.sub(r"<\|.*?\|>", "", cleaned).strip()
+    m = re.search(r"(SELECT\s+.*?;)", cleaned, re.DOTALL | re.IGNORECASE)
+    if m:
+        return m.group(1).strip()
+    if "select" in cleaned.lower():
+        return cleaned.strip()
+    return cleaned.strip()
+
+
+def run_sql(conn, sql):
+    """Returns (rows_as_sorted_list, error_or_None)."""
+    if not sql or "select" not in sql.lower():
+        return None, "invalid/blank SQL"
+    try:
+        cur = conn.cursor()
+        cur.execute(sql)
+        rows = cur.fetchall()
+        return sorted([tuple(r) for r in rows], key=lambda r: str(r)), None
+    except Exception as e:
+        return None, str(e)
+
+
+def load_test_set(path):
+    items = []
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            obj = json.loads(line)
+            msgs = obj["messages"]
+            user = next(m["content"] for m in msgs if m["role"] == "user")
+            expected = next(m["content"] for m in msgs if m["role"] == "assistant")
+            question = user.split("Question:", 1)[-1].strip() if "Question:" in user else user
+            items.append({"user_prompt": user, "question": question, "expected_sql": expected})
+    return items
+
+
+# ---------- llama-server backend ----------
+
+def find_llama_server():
+    candidates = []
+    candidates += glob.glob(os.path.join("llama-b*", "llama-server.exe"))
+    candidates += glob.glob(os.path.join("llama-b*-bin-win-*", "llama-server.exe"))
+    candidates += glob.glob(os.path.join("llama-bin", "llama-server.exe"))
+    for name in ("llama-server.exe", "llama-server"):
+        if os.path.exists(name):
+            candidates.append(name)
+    return candidates[0] if candidates else None
+
+
+def start_server(server_path, gguf, n_ctx, port, log_path="llama-server.log"):
+    cmd = [server_path, "-m", gguf, "-c", str(n_ctx),
+           "--host", "127.0.0.1", "--port", str(port)]
+    print(f"Starting llama-server (logs -> {log_path}) ...")
+    log = open(log_path, "w", encoding="utf-8")
+    proc = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT)
+
+    health = f"http://127.0.0.1:{port}/health"
+    for _ in range(180):  # up to ~3 min for first model load
+        if proc.poll() is not None:
+            raise RuntimeError(f"llama-server exited early; see {log_path}")
+        try:
+            with urllib.request.urlopen(health, timeout=2) as r:
+                if r.status == 200 and json.loads(r.read()).get("status") == "ok":
+                    print("Server ready.")
+                    return proc
+        except Exception:
+            pass
+        time.sleep(1)
+    proc.terminate()
+    raise RuntimeError(f"llama-server did not become ready in time; see {log_path}")
+
+
+def make_http_generator(port):
+    url = f"http://127.0.0.1:{port}/v1/chat/completions"
+
+    def gen(user_content):
+        body = json.dumps({
+            "messages": [{"role": "user", "content": user_content}],
+            "temperature": 0.0,
+            "max_tokens": 150,
+        }).encode("utf-8")
+        req = urllib.request.Request(url, data=body, headers={"Content-Type": "application/json"})
+        with urllib.request.urlopen(req, timeout=180) as r:
+            out = json.loads(r.read())
+        return out["choices"][0]["message"]["content"]
+    return gen
+
+
+def make_python_generator(gguf, n_ctx):
+    from llama_cpp import Llama
+    llm = Llama(model_path=gguf, n_ctx=n_ctx, n_threads=os.cpu_count() or 2, verbose=False)
+
+    def gen(user_content):
+        out = llm.create_chat_completion(
+            messages=[{"role": "user", "content": user_content}],
+            max_tokens=150, temperature=0.0,
+        )
+        return out["choices"][0]["message"]["content"]
+    return gen
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--gguf", default=DEFAULT_GGUF, help="Path to the GGUF file")
+    ap.add_argument("--threshold", type=float, default=0.95, help="Acceptance match rate")
+    ap.add_argument("--n_ctx", type=int, default=2048)
+    ap.add_argument("--server", default=None, help="Path to llama-server(.exe). Auto-detected if omitted.")
+    ap.add_argument("--port", type=int, default=8080)
+    ap.add_argument("--use-python", action="store_true", help="Use llama-cpp-python bindings instead of llama-server")
+    args = ap.parse_args()
+
+    if not os.path.exists(args.gguf):
+        sys.exit(f"GGUF not found: {args.gguf}")
+    if not os.path.exists(TEST_PATH):
+        sys.exit(f"Test set not found: {TEST_PATH}")
+    if not os.path.exists(DB_PATH):
+        sys.exit(f"Database not found: {DB_PATH}. Run `python src/database.py` first.")
+
+    server_proc = None
+    try:
+        if args.use_python:
+            print("Backend: llama-cpp-python bindings")
+            generate = make_python_generator(args.gguf, args.n_ctx)
+        else:
+            server = args.server or find_llama_server()
+            if not server:
+                sys.exit("Could not find llama-server(.exe). Pass --server <path>, "
+                         "or use --use-python if the wheel runs on your CPU.")
+            print(f"Backend: llama-server -> {server}")
+            server_proc = start_server(server, args.gguf, args.n_ctx, args.port)
+            generate = make_http_generator(args.port)
+
+        tests = load_test_set(TEST_PATH)
+        conn = sqlite3.connect(DB_PATH)
+
+        passed = 0
+        failures = []
+        for i, t in enumerate(tests, 1):
+            raw = generate(t["user_prompt"])
+            pred_sql = clean_sql(raw)
+
+            exp_rows, exp_err = run_sql(conn, t["expected_sql"])
+            pred_rows, pred_err = run_sql(conn, pred_sql)
+
+            if exp_err is not None:
+                ok = pred_sql.strip().lower().rstrip(";") == t["expected_sql"].strip().lower().rstrip(";")
+            else:
+                ok = (pred_err is None) and (pred_rows == exp_rows)
+
+            status = "PASS" if ok else "FAIL"
+            if ok:
+                passed += 1
+            else:
+                failures.append({
+                    "n": i, "question": t["question"],
+                    "expected": t["expected_sql"], "predicted": pred_sql, "pred_err": pred_err,
+                })
+            print(f"[{i:02d}/{len(tests)}] {status}  {t['question'][:70]}")
+
+        conn.close()
+
+        rate = passed / len(tests) if tests else 0.0
+        print("\n" + "=" * 60)
+        print(f"Execution-equivalent: {passed}/{len(tests)} = {rate:.1%}")
+        print(f"Acceptance bar      : {args.threshold:.0%}")
+        print("Result              :", "ACCEPTED" if rate >= args.threshold else "BELOW BAR (try Q5_K_M)")
+        print("=" * 60)
+
+        if failures:
+            print("\nFailures:")
+            for f in failures:
+                print(f"\n  Q{f['n']}: {f['question']}")
+                print(f"    expected : {f['expected']}")
+                print(f"    predicted: {f['predicted']}")
+                if f["pred_err"]:
+                    print(f"    sql error: {f['pred_err']}")
+
+        exit_code = 0 if rate >= args.threshold else 1
+    finally:
+        if server_proc is not None:
+            server_proc.terminate()
+            try:
+                server_proc.wait(timeout=10)
+            except Exception:
+                server_proc.kill()
+
+    sys.exit(exit_code)
+
+
+if __name__ == "__main__":
+    main()

src/inference.py

@@ -15,6 +15,13 @@ class TextToSQLInference:
         self.db_path = "data/company_sales.db"
         self.tokenizer = None
         self.model = None
+        self.llm = None          # llama_cpp.Llama instance (CPU/GGUF backend only)
+        self.backend = None      # "transformers" (CUDA) or "llama_cpp" (CPU)
+
+        # GGUF source for CPU serving (override via env on the Space)
+        self.gguf_repo_id = os.environ.get("GGUF_REPO_ID", "Bhuvandesai/phi3-text-to-sql-gguf")
+        self.gguf_filename = os.environ.get("GGUF_FILENAME", "phi3-text-to-sql-Q5_K_M.gguf")
+
         # Support both local paths and HF Hub repo IDs
         self.has_adapter = self._adapter_available(adapter_path)
 
@@ -33,36 +40,39 @@ class TextToSQLInference:
             return False
 
     def _load_model_and_tokenizer(self):
+        # Two backends, chosen by hardware:
+        #   - CUDA (laptop/dev): transformers + 4-bit bitsandbytes + PEFT adapter
+        #   - CPU  (HF Space):   llama.cpp + pre-merged GGUF (Q4_K_M)
+        # The CPU path never imports torch-side quantization or PEFT, and llama_cpp
+        # is imported lazily so the CUDA path doesn't need it installed.
+        if self.device == "cuda":
+            self.backend = "transformers"
+            self._load_transformers_cuda()
+        else:
+            self.backend = "llama_cpp"
+            self._load_llama_cpp_cpu()
+
+    def _load_transformers_cuda(self):
         print(f"Loading tokenizer: {self.base_model_id}...")
         self.tokenizer = AutoTokenizer.from_pretrained(self.base_model_id, trust_remote_code=False)
         self.tokenizer.pad_token = self.tokenizer.eos_token
-        
-        if self.device == "cuda":
-            print("CUDA detected — configuring 4-bit quantization...")
-            bnb_config = BitsAndBytesConfig(
-                load_in_4bit=True,
-                bnb_4bit_quant_type="nf4",
-                bnb_4bit_use_double_quant=True,
-                bnb_4bit_compute_dtype=torch.float16
-            )
-            print("Loading base model with 4-bit quantization...")
-            self.model = AutoModelForCausalLM.from_pretrained(
-                self.base_model_id,
-                quantization_config=bnb_config,
-                device_map="auto",
-                trust_remote_code=False,
-                attn_implementation="eager"
-            )
-        else:
-            print("CPU detected — loading model in bfloat16 (no quantization)...")
-            self.model = AutoModelForCausalLM.from_pretrained(
-                self.base_model_id,
-                torch_dtype=torch.bfloat16,
-                device_map="cpu",
-                trust_remote_code=False,
-                attn_implementation="eager"
-            )
-        
+
+        print("CUDA detected — configuring 4-bit quantization...")
+        bnb_config = BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_quant_type="nf4",
+            bnb_4bit_use_double_quant=True,
+            bnb_4bit_compute_dtype=torch.float16
+        )
+        print("Loading base model with 4-bit quantization...")
+        self.model = AutoModelForCausalLM.from_pretrained(
+            self.base_model_id,
+            quantization_config=bnb_config,
+            device_map="auto",
+            trust_remote_code=False,
+            attn_implementation="eager"
+        )
+
         if self.has_adapter:
             print(f"Loading fine-tuned adapter from: {self.adapter_path}...")
             self.model = PeftModel.from_pretrained(
@@ -74,6 +84,36 @@ class TextToSQLInference:
         else:
             print("WARNING: No adapter found. Will run base model only.")
 
+    def _load_llama_cpp_cpu(self):
+        # GGUF already contains the merged fine-tuned weights, so there is no
+        # separate adapter to load at runtime.
+        self.has_adapter = True
+        print("CPU detected — using llama.cpp + GGUF (Q4_K_M) backend.")
+
+        from llama_cpp import Llama
+        from huggingface_hub import hf_hub_download
+
+        # Allow a local GGUF (committed under models/) to take precedence; otherwise pull from Hub.
+        local_candidate = os.path.join("models", self.gguf_filename)
+        if os.path.exists(local_candidate):
+            model_path = local_candidate
+            print(f"Loading local GGUF: {model_path}")
+        else:
+            print(f"Downloading GGUF {self.gguf_filename} from {self.gguf_repo_id}...")
+            model_path = hf_hub_download(repo_id=self.gguf_repo_id, filename=self.gguf_filename)
+            print(f"GGUF cached at: {model_path}")
+
+        n_threads = os.cpu_count() or 2
+        self.llm = Llama(
+            model_path=model_path,
+            n_ctx=2048,          # schema prompt (~300 tok) + question + 150 generated
+            n_threads=n_threads,
+            n_batch=512,
+            logits_all=False,
+            verbose=False,
+        )
+        print(f"llama.cpp model loaded (n_threads={n_threads}).")
+
     def clean_sql(self, raw_output):
         """Cleans and extracts SQL query from model output, stripping backticks and markdown formatting."""
         # Print for raw debugging
@@ -119,9 +159,50 @@ class TextToSQLInference:
             return None, str(e)
 
     def generate_sql(self, question, use_adapter=True):
+        """Dispatches to the active backend. Output contract is identical for both."""
+        if self.backend == "llama_cpp":
+            return self._generate_llama(question, use_adapter=use_adapter)
+        return self._generate_transformers(question, use_adapter=use_adapter)
+
+    def _generate_llama(self, question, use_adapter=True):
+        """CPU/GGUF generation via llama.cpp. The GGUF is the merged fine-tuned model."""
+        if not use_adapter:
+            # GGUF has the fine-tuned weights baked in; there is no base toggle on CPU.
+            # The base comparison is served by generate_sql_base_via_api (HF API) instead.
+            # query_pipeline catches this and returns a clean error rather than crashing.
+            raise RuntimeError(
+                "Local base-model comparison is unavailable in CPU/GGUF mode; "
+                "use the HF Inference API path for base comparison."
+            )
+
+        prompt = f"{SCHEMA_PROMPT}\n\nQuestion: {question}"
+        print("--- Generating with Fine-tuned GGUF (llama.cpp) ---")
+
+        start_time = time.perf_counter()
+        result = self.llm.create_chat_completion(
+            messages=[{"role": "user", "content": prompt}],
+            max_tokens=150,
+            temperature=0.0,   # greedy / deterministic, matches do_sample=False
+        )
+        latency_sec = time.perf_counter() - start_time
+
+        raw_output = result["choices"][0]["message"]["content"]
+        usage = result.get("usage", {}) or {}
+        tokens_generated = usage.get("completion_tokens", 0) or 0
+        tokens_per_second = tokens_generated / latency_sec if latency_sec > 0 else 0.0
+
+        sql_query = self.clean_sql(raw_output)
+        metrics = {
+            "latency_sec": latency_sec,
+            "tokens_generated": tokens_generated,
+            "tokens_per_second": tokens_per_second
+        }
+        return sql_query, metrics
+
+    def _generate_transformers(self, question, use_adapter=True):
         """Generates SQL query from a natural language question."""
         prompt = f"{SCHEMA_PROMPT}\n\nQuestion: {question}"
-        
+
         # Standard Phi-3 ChatML template formatting
         messages = [
             {"role": "user", "content": prompt}