Upload 7 files

Dockerfile

@@ -1,9 +1,12 @@
 # CPU + FREE-tier RAG Space (Docker SDK).
-#   Embeddings : BAAI/bge-small-en-v1.5        (fastembed / ONNX, no torch)
+#   Embeddings : BAAI/bge-small-en-v1.5   (fastembed / ONNX, no torch)
 #   Vector DB  : FAISS (in-memory)
-#   LLM        : Qwen2.5-Coder-0.5B-Instruct   (llama.cpp GGUF; code-tuned, fast on CPU)
+#   LLM        : Qwen3.5-0.8B (MoE)        (llama.cpp GGUF; small + fast on CPU)
 #   API        : OpenAI-compatible  ->  /v1/chat/completions  (+ web UI at /)
 #
+# NOTE: Qwen3.5 uses the `qwen35` arch -> requires llama-cpp-python >= 0.3.32
+#       (llama.cpp >= b9616). See requirements.txt.
+#
 # Everything is CPU-only. No GPU, no paid hardware required.
 # Models are baked into the image so the Space cold-starts instantly.
 
@@ -36,10 +39,10 @@ ENV CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DG
 RUN pip install --no-cache-dir --user --no-binary=llama-cpp-python -r requirements.txt
 
 # --- Model config ------------------------------------------------------------
-# Code-tuned 0.5B: same size/speed as the generic 0.5B, but much better at
-# reading & explaining code. Q4_K_M is the CPU sweet spot.
-ENV LLM_REPO=Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF \
-    LLM_FILE=qwen2.5-coder-0.5b-instruct-q4_k_m.gguf \
+# Qwen3.5-0.8B (MoE): tiny active-param count -> fast on CPU, punches above a
+# dense 0.5B. Q4_K_M (~533 MB) is the CPU sweet spot. Text-only GGUF (no vision).
+ENV LLM_REPO=unsloth/Qwen3.5-0.8B-GGUF \
+    LLM_FILE=Qwen3.5-0.8B-Q4_K_M.gguf \
     MODEL_DIR=/home/user/models \
     EMBED_MODEL=BAAI/bge-small-en-v1.5 \
     FASTEMBED_CACHE=/home/user/.cache/fastembed \

README.md

@@ -9,7 +9,7 @@ pinned: false
 license: mit
 ---
 
-# CPU RAG Space — Qwen2.5-Coder-0.5B + FAISS (free tier, no GPU)
+# CPU RAG Space — Qwen3.5-0.8B + FAISS (free tier, no GPU)
 
 A self-contained Retrieval-Augmented Generation service that runs **entirely on
 CPU** and fits the Hugging Face **free tier** (2 vCPU / 16 GB).
@@ -18,16 +18,20 @@ CPU** and fits the Hugging Face **free tier** (2 vCPU / 16 GB).
 |-----------|------|-----|
 | Embeddings | `BAAI/bge-small-en-v1.5` via `fastembed` (ONNX) | fast on CPU, no PyTorch, ~130 MB |
 | Vector DB | **FAISS** (in-memory) | tiny, instant search |
-| LLM | `Qwen2.5-Coder-0.5B-Instruct` GGUF Q4_K_M via llama.cpp | code-tuned, ~350 MB, fastest tier on 2 vCPU |
+| LLM | `Qwen3.5-0.8B` (MoE) GGUF Q4_K_M via llama.cpp | ~533 MB, small MoE — fast on 2 vCPU, smarter than a dense 0.5B |
 | API | OpenAI-compatible `/v1/chat/completions` + web UI | drop-in for any client |
 
-Total footprint ≈ 0.7–1 GB RAM. Retrieval adds ~20 ms; the LLM is the only
+Total footprint ≈ 0.8–1.1 GB RAM. Retrieval adds ~20 ms; the LLM is the only
 real latency.
 
+> **Runtime requirement:** Qwen3.5 uses the `qwen35` architecture, so this needs
+> **`llama-cpp-python >= 0.3.32`** (llama.cpp ≥ b9616). The Dockerfile builds it
+> from source. Older pins fail with *"unknown architecture 'qwen35'"*.
+
 **Tuned for CPU speed:** 2K context (small KV cache), flash-attention, short
-256-token answers, a startup warm-up, and a **streaming** web UI so the first
-word appears in ~1–1.5 s (decode runs ~5–6 tok/s on 2 vCPU). See
-`.github/workflows/keepalive.yml` for the keep-warm ping.
+256-token answers, relevance-gated RAG (greetings/off-topic skip retrieval), a
+startup warm-up, and a **streaming** web UI so the first word appears in ~1 s.
+See `.github/workflows/keepalive.yml` for the keep-warm ping.
 
 ## Deploy (drag & drop)
 
@@ -68,9 +72,17 @@ Extra endpoints: `POST /ingest` (upload a doc), `GET /stats`.
 Change `LLM_REPO` / `LLM_FILE` in the `Dockerfile` (confirm exact filenames on
 the repo's *Files* tab):
 
-- Current (code-tuned, fastest): `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF`
-- Smarter on code, ~½ the speed: `Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF`
-- Best general RAG at this size: `HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF`
+- Current (small MoE, fast on CPU): `unsloth/Qwen3.5-0.8B-GGUF` → `Qwen3.5-0.8B-Q4_K_M.gguf`
+- Fastest tiny dense: `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF`
+- Best general quality (~½ speed): `HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF`
+
+### MTP (Multi-Token Prediction) — later
+
+`unsloth/Qwen3.5-0.8B-MTP-GGUF` adds self-speculative decoding for a ~1.4–1.7×
+decode speedup. It's **not** a drop-in here: the MTP flags (`--spec-type
+draft-mtp`) live in **`llama-server`**, not the `llama_cpp.Llama()` API this app
+uses. To try it, run `llama-server` as the backend and proxy to it. Feasible on
+CPU only because the model is tiny (an MTP 9B on 2 vCPU stays ~1.5–2.5 tok/s).
 
 ## Keep it warm (avoid cold starts)
 

app.py

@@ -1,10 +1,11 @@
 """
-CPU RAG Space — bge-small (fastembed) + FAISS + Qwen2.5-Coder-0.5B (llama.cpp),
+CPU RAG Space — bge-small (fastembed) + FAISS + Qwen3.5-0.8B (llama.cpp),
 served as an OpenAI-compatible API with a small web UI.
 
 Everything runs on CPU and fits the Hugging Face free tier (2 vCPU / 16 GB).
-Tuned for lowest CPU latency: code-tuned 0.5B model, 2K context, flash-attention,
-short answers, and a startup warm-up so the first real query is snappy.
+Tuned for lowest CPU latency: small MoE model, 2K context, flash-attention,
+short answers, relevance-gated RAG, and a startup warm-up so the first real
+query is snappy. Needs llama-cpp-python >= 0.3.32 (qwen35 arch).
 """
 
 import glob
@@ -23,7 +24,7 @@ from pydantic import BaseModel
 # Config (all overridable via Space "Variables")
 # --------------------------------------------------------------------------- #
 MODEL_DIR = os.environ.get("MODEL_DIR", "models")
-LLM_FILE = os.environ.get("LLM_FILE", "qwen2.5-coder-0.5b-instruct-q4_k_m.gguf")
+LLM_FILE = os.environ.get("LLM_FILE", "Qwen3.5-0.8B-Q4_K_M.gguf")
 LLM_PATH = os.path.join(MODEL_DIR, LLM_FILE)
 EMBED_MODEL = os.environ.get("EMBED_MODEL", "BAAI/bge-small-en-v1.5")
 FASTEMBED_CACHE = os.environ.get("FASTEMBED_CACHE")
@@ -33,14 +34,23 @@ N_CTX = int(os.environ.get("N_CTX", "2048"))
 N_THREADS = int(os.environ.get("N_THREADS", str(os.cpu_count() or 2)))
 TOP_K = int(os.environ.get("TOP_K", "4"))
 DOCS_DIR = os.environ.get("DOCS_DIR", "documents")
+# Only inject retrieved context when the best hit is actually relevant (cosine
+# similarity). Below this, treat it as normal chat instead of forcing doc QA —
+# stops greetings/off-topic messages from dredging up irrelevant chunks.
+RAG_MIN_SCORE = float(os.environ.get("RAG_MIN_SCORE", "0.4"))
 
 CHUNK_SIZE = 800        # characters per chunk
 CHUNK_OVERLAP = 120
 
+# Plain assistant persona used when nothing relevant is retrieved.
+CHAT_SYSTEM = "You are a friendly, concise assistant."
+
+# Used when we DO have relevant context. No literal "[filename]" example — a
+# tiny model will just echo it. Sources are returned separately in the JSON.
 RAG_SYSTEM = (
-    "You are a helpful assistant. Answer the user's question using ONLY the "
-    "context below. If the answer is not in the context, say you don't know. "
-    "Cite the source of each fact in square brackets like [filename].\n\n"
+    "You are a helpful assistant. Use the context below to answer the user's "
+    "question. If the context does not contain the answer, say so briefly and "
+    "answer from your own knowledge if you can. Keep answers concise.\n\n"
     "Context:\n{context}"
 )
 
@@ -178,12 +188,20 @@ def _augment(req: ChatRequest):
     msgs = [m.model_dump() for m in req.messages]
     users = [m for m in msgs if m["role"] == "user"]
     query = users[-1]["content"] if users else ""
-    ctxs = retrieve(query) if req.use_rag else []
+
+    # Retrieve, then keep only chunks that clear the relevance bar.
+    hits = retrieve(query) if req.use_rag else []
+    ctxs = [c for c in hits if c["score"] >= RAG_MIN_SCORE]
+
+    non_system = [m for m in msgs if m["role"] != "system"]
     if ctxs:
         context = "\n\n".join(f"[{c['source']}] {c['text']}" for c in ctxs)
         system = {"role": "system", "content": RAG_SYSTEM.format(context=context)}
-        msgs = [system] + [m for m in msgs if m["role"] != "system"]
-    return msgs, ctxs
+    else:
+        # Nothing relevant -> behave like a normal chat assistant, no forced
+        # "answer only from context" (which made the model spit citations).
+        system = {"role": "system", "content": CHAT_SYSTEM}
+    return [system] + non_system, ctxs
 
 
 @app.post("/v1/chat/completions")

documents/sample.txt

@@ -7,8 +7,9 @@ Architecture:
   text into 384-dimensional vectors and needs no GPU or PyTorch.
 - Vector store: FAISS (IndexFlatIP) holds the document vectors in memory and
   returns the most similar chunks for a query using cosine similarity.
-- Language model: Qwen2.5-1.5B-Instruct in GGUF Q4_K_M form, served by
-  llama.cpp. It reads the retrieved chunks and writes a grounded answer.
+- Language model: Qwen3.5-0.8B (a small Mixture-of-Experts model) in GGUF
+  Q4_K_M form, served by llama.cpp. It reads the retrieved chunks and writes a
+  grounded answer.
 
 How retrieval works:
 1. Your question is embedded into a vector.
@@ -19,7 +20,8 @@ How retrieval works:
 Why a small model is fine here:
 RAG moves knowledge out of the model's weights and into the retriever, so the
 model only needs to read and summarise the provided context rather than
-memorise facts. That makes a fast 1.5B model a good fit for CPU serving.
+memorise facts. That makes a fast, small model like Qwen3.5-0.8B a good fit for
+CPU serving.
 
 Replace this file with your own .txt or .md documents, or upload files at
 runtime through the web UI, and the Space will answer questions about them.

index.html

@@ -3,7 +3,7 @@
 <head>
 <meta charset="utf-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1" />
-<title>CPU RAG · Qwen2.5-Coder-0.5B + FAISS</title>
+<title>CPU RAG · Qwen3.5-0.8B + FAISS</title>
 <style>
   :root { color-scheme: light dark; --bg:#0f1220; --panel:#1a1e33; --acc:#7c8cff; --mut:#8b90a8; }
   * { box-sizing: border-box; }
@@ -28,7 +28,7 @@
 <body>
 <header>
   <h1>🦅 CPU RAG</h1>
-  <span class="tag">Qwen2.5-Coder-0.5B · bge-small · FAISS</span>
+  <span class="tag">Qwen3.5-0.8B · bge-small · FAISS</span>
   <span class="tag" id="stat">…</span>
 </header>
 <main>

requirements.txt

@@ -1,8 +1,11 @@
-# Pinned to 0.3.2 and built FROM SOURCE (see Dockerfile --no-binary):
-#   * the prebuilt 0.3.2 wheel is musl-linked -> fails on Debian glibc, and
-#   * newer wheels (0.3.3x) hit the llama.cpp CPU "set_rows" decode crash.
-# Compiling 0.3.2 on this glibc image is the only combo that avoids both.
-llama-cpp-python==0.3.2
+# Bumped 0.3.2 -> 0.3.32 (Jun 2026) and still built FROM SOURCE (Dockerfile
+# --no-binary):
+#   * Qwen3.5 uses the `qwen35` architecture, which needs llama.cpp >= b9616.
+#     The old 0.3.2 pin can't load it ("unknown architecture 'qwen35'").
+#   * 0.3.32 vendors a recent llama.cpp (past the old CPU "set_rows" decode
+#     crash) and works on Debian glibc when compiled here.
+# If a future wheel regresses, pin the last-known-good 0.3.x that loads qwen35.
+llama-cpp-python==0.3.32
 
 # Lightweight CPU embeddings via ONNX (no PyTorch -> small image, fast).
 fastembed