Replace cloud inference with local Ollama

README.md

@@ -1,77 +0,0 @@
----
-title: KnowLedge Inference API
-emoji: 📖
-colorFrom: yellow
-colorTo: green
-sdk: gradio
-python_version: 3.10.16
-sdk_version: 4.44.1
-app_file: app.py
-pinned: true
-license: apache-2.0
-tags:
-  - gemma
-  - gemma-4
-  - education
-  - inference-api
-  - knowledge
-  - hackathon
-short_description: Gemma 4 GGUF inference backend for KnowLedge
----
-
-# KnowLedge Inference API
-
-Gemma 4 E2B + quantized E4B inference backend for the
-[KnowLedge](https://github.com/Charan-suresh/KnowLedge) learning ledger.
-
-Part of the Kaggle Gemma 4 Good Hackathon submission.
-
-## Endpoints
-
-| Endpoint | Model | Used by |
-|---|---|---|
-| `/api/generate` | E2B or E4B | Scout (concept tagging), Sage (Socratic dialogue) |
-| `/api/generate_vision` | E4B vision | Lens (handwriting verification) |
-| `/api/health` | - | FastAPI health check |
-
-## Architecture
-
-This Space uses **Gradio + FastAPI** together via `gr.mount_gradio_app`:
-
-- The **Gradio** wrapper satisfies HF Spaces `sdk: gradio` requirements so the Space is hosted on port 7860 without a custom Dockerfile.
-- The **FastAPI** layer exposes the REST endpoints the Render backend calls via `httpx`.
-- Inference runs entirely on **CPU** on the standard free HF Spaces tier — no ZeroGPU required.
-
-| Model | Backend | Notes |
-|---|---|---|
-| E4B | `llama-cpp-python` GGUF | `UD-IQ3_XXS` quant, ~2.5 GB RAM, fast-ish on CPU |
-| E2B | HuggingFace `transformers` | float32, ~8 GB RAM, slower on CPU |
-
-Override the E4B GGUF file with the `GEMMA_E4B_GGUF_FILE` Space secret if you want a different quantisation.
-
-## Not for direct use
-
-This Space is called programmatically by the KnowLedge FastAPI backend
-deployed on Render. It is not a chat interface.
-
-## Deployment note
-
-Deploy the files inside `hf_space/` to a separate Hugging Face Space repository.
-Do **not** deploy the root `app.py` here — that file is the standalone Gradio demo
-that connects back to the Render backend, not the inference backend itself.
-
-## Required setup
-
-1. **No special hardware needed** — leave the Space on the default free CPU tier.
-2. Set the **`HF_TOKEN`** Space secret (Settings → Repository secrets) to a token that has accepted the Gemma 4 terms on HuggingFace. This is required for `google/gemma-4-e2b-it` (gated model) to download at runtime.
-3. On **Render**, set:
-   - `INFERENCE_BACKEND=hf_space`
-   - `HF_SPACE_URL=charan-ml/knowledge-inference`
-
-### Optional env vars (Space secrets)
-
-| Variable | Default | Effect |
-|---|---|---|
-| `GEMMA_E4B_GGUF_FILE` | `gemma-4-E4B-it-UD-IQ3_XXS.gguf` | Switch to a different GGUF quant |
-| `GEMMA_E4B_CTX` | `4096` | llama-cpp context window |
-| `GEMMA_E4B_THREADS` | `4` | CPU threads for llama-cpp |

app.py

@@ -1,142 +0,0 @@
-"""
-KnowLedge Inference Space — hf_space/app.py
-
-Architecture:
-  FastAPI provides the REST endpoints (/api/generate, /api/generate_vision, /api/health).
-  Gradio is mounted at the root so the HF Space is recognised as sdk:gradio, which lets
-  it serve traffic on port 7860 without a custom Dockerfile.
-
-  Inference runs entirely on CPU on the standard free HF Spaces tier:
-    - E4B  : unsloth/gemma-4-E4B-it-GGUF  (UD-IQ3_XXS, ~2.5 GB, llama-cpp-python)
-    - E2B  : google/gemma-4-e2b-it         (float32, transformers)
-
-Render backend calls the FastAPI endpoints via httpx.
-Visiting the Space URL in a browser shows the Gradio landing page.
-"""
-
-from fastapi import FastAPI
-from fastapi.middleware.cors import CORSMiddleware
-from pydantic import BaseModel
-
-try:
-    import gradio as gr
-except ModuleNotFoundError:
-    gr = None
-
-try:
-    from inference import generate_text, generate_with_image, health_check
-except ModuleNotFoundError:
-    from .inference import generate_text, generate_with_image, health_check
-
-
-# ── FastAPI application ───────────────────────────────────────────────────────
-
-_api = FastAPI(title="KnowLedge Inference API", version="1.0.0")
-
-# Allow the Render backend (and any browser preview) to call these endpoints.
-_api.add_middleware(
-    CORSMiddleware,
-    allow_origins=["*"],
-    allow_methods=["GET", "POST", "OPTIONS"],
-    allow_headers=["*"],
-)
-
-
-class GenerateRequest(BaseModel):
-    prompt: str
-    model: str = "e2b"
-    max_tokens: int = 512
-
-
-class VisionRequest(BaseModel):
-    prompt: str
-    image_base64: str
-    max_tokens: int = 512
-
-
-@_api.get("/api/health")
-def health():
-    """Health check — called by the Render backend before every inference batch."""
-    try:
-        return health_check()
-    except Exception as exc:
-        return {"status": "error", "error": str(exc)}
-
-
-@_api.post("/api/generate")
-def generate(req: GenerateRequest):
-    """Text generation endpoint used by Scout (concept tagging) and Sage (dialogue)."""
-    try:
-        model_name = "e2b" if "e2b" in req.model.lower() else "e4b"
-        response = generate_text(model_name, req.prompt, req.max_tokens)
-        return {"response": response}
-    except Exception as exc:
-        return {"error": str(exc)}
-
-
-@_api.post("/api/generate_vision")
-def generate_vision(req: VisionRequest):
-    """Vision generation endpoint used by Lens (handwriting verification)."""
-    try:
-        response = generate_with_image(
-            req.prompt,
-            req.image_base64,
-            req.max_tokens,
-        )
-        return {"response": response}
-    except Exception as exc:
-        return {"error": str(exc)}
-
-
-# ── Gradio UI ─────────────────────────────────────────────────────────────────
-# A minimal Gradio interface is required so that:
-#   1. The HF Space is recognised as sdk:gradio and served on port 7860.
-#   2. The Space shows a readable landing page instead of a blank iframe.
-# No ZeroGPU needed — inference runs on CPU via llama-cpp-python (GGUF).
-
-if gr is not None:
-    with gr.Blocks(title="KnowLedge Inference API") as demo:
-        gr.Markdown(
-            """
-            # 🦉 KnowLedge Inference API
-
-            Gemma 4 **E2B** + **E4B** inference backend for the
-            [KnowLedge](https://github.com/Charan-suresh/KnowLedge) learning ledger.
-
-            Part of the **Kaggle Gemma 4 for Good** hackathon submission.
-
-            ---
-
-            This Space is a **REST API backend** — it is called programmatically by the
-            KnowLedge FastAPI service deployed on Render. It is not a chat interface.
-
-            | Endpoint | Method | Used by |
-            |---|---|---|
-            | `/api/generate` | POST | Scout (concept tagging), Sage (Socratic dialogue) |
-            | `/api/generate_vision` | POST | Lens (handwriting verification) |
-            | `/api/health` | GET | Render backend health check |
-
-            ### Payload format — `/api/generate`
-            ```json
-            { "model": "e2b", "prompt": "...", "max_tokens": 512 }
-            ```
-
-            ### Payload format — `/api/generate_vision`
-            ```json
-            { "prompt": "...", "image_base64": "<base64>", "max_tokens": 512 }
-            ```
-            """
-        )
-
-    # Mount the Gradio interface into the FastAPI app.
-    # FastAPI routes registered above (/api/*) take priority over Gradio routes.
-    # Gradio occupies the root path ("/") for the HF Space iframe.
-    app = gr.mount_gradio_app(_api, demo, path="/")
-else:
-    # Local fallback for test and dev environments without gradio installed.
-    app = _api
-
-
-if __name__ == "__main__":
-    import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=7860)

deploy.sh

@@ -1,72 +0,0 @@
-#!/bin/bash
-# KnowLedge - Hugging Face Space deployment guide
-# Run each block manually in sequence.
-# Prerequisites: huggingface_hub installed, HF account created
-
-# -- Step 1: Install HF CLI -------------------------------------------------
-pip install huggingface_hub
-
-# -- Step 2: Login to Hugging Face ------------------------------------------
-huggingface-cli login
-# Paste your HF token when prompted
-# Get token at: https://huggingface.co/settings/tokens
-# Token needs "write" permission
-
-# -- Step 3: Create the Space ------------------------------------------------
-python3 - << 'EOF'
-from huggingface_hub import HfApi
-api = HfApi()
-api.create_repo(
-    repo_id="knowledge-inference",   # becomes: your-username/knowledge-inference
-    repo_type="space",
-    space_sdk="gradio",
-    private=False,                    # must be public for free ZeroGPU
-    exist_ok=True,
-)
-print("Space created successfully")
-EOF
-
-# -- Step 4: Upload Space files ----------------------------------------------
-python3 - << 'EOF'
-from huggingface_hub import HfApi
-import os
-
-api = HfApi()
-space_id = f"{api.whoami()['name']}/knowledge-inference"
-base_dir = os.getcwd()
-
-files = ["app.py", "inference.py", "requirements.txt", "README.md"]
-for f in files:
-    path = os.path.join(base_dir, f)
-    api.upload_file(
-        path_or_fileobj=path,
-        path_in_repo=f,
-        repo_id=space_id,
-        repo_type="space",
-    )
-    print(f"Uploaded: {f}")
-
-print(f"\nSpace URL: https://huggingface.co/spaces/{space_id}")
-print(f"API URL:   https://{space_id.replace('/', '-')}.hf.space")
-EOF
-
-# -- Step 5: Enable ZeroGPU --------------------------------------------------
-# Go to: https://huggingface.co/spaces/YOUR-USERNAME/knowledge-inference/settings
-# Under "Space Hardware" -> select "ZeroGPU (NVIDIA H200)"
-# This cannot be done via API - must be done in the HF web UI
-echo "-> Manually enable ZeroGPU in Space settings:"
-echo "  https://huggingface.co/spaces/$(huggingface-cli whoami)/knowledge-inference/settings"
-
-# -- Step 6: Get your Space API URL ------------------------------------------
-# After deployment (~3-5 minutes for build), your endpoints are:
-# Text:   POST https://your-username-knowledge-inference.hf.space/api/generate
-# Vision: POST https://your-username-knowledge-inference.hf.space/api/generate_vision
-# Health: GET  https://your-username-knowledge-inference.hf.space/api/health
-
-# -- Step 7: Set Render env vars ---------------------------------------------
-# In Render dashboard -> your service -> Environment:
-# INFERENCE_BACKEND = hf_space
-# HF_SPACE_URL      = your-username/knowledge-inference
-# Then trigger a manual redeploy on Render.
-
-echo "Deployment complete. Set HF_SPACE_URL in Render and redeploy."

inference.py

@@ -1,152 +0,0 @@
-"""
-KnowLedge Inference — inference.py
-
-Runs entirely on CPU on the standard free HF Spaces tier (no ZeroGPU).
-
-Model backends
---------------
-Text : Quantized Gemma 2B GGUF via llama-cpp-python (CPU-only).
-"""
-
-import base64
-import io
-import logging
-import os
-
-from PIL import Image
-from huggingface_hub import hf_hub_download
-
-# ── spaces stub — GPU decorator is a no-op on the free CPU tier ───────────────
-try:
-    import spaces
-except ModuleNotFoundError:
-    class _SpacesStub:
-        @staticmethod
-        def GPU(duration: int = 0):
-            def decorator(fn):
-                return fn
-            return decorator
-
-    spaces = _SpacesStub()
-
-# ── llama-cpp backend ─────────────────────────────────────────────────────────
-_llama_import_error = None
-try:
-    from llama_cpp import Llama
-except Exception as exc:
-    _llama_import_error = exc
-    Llama = None
-
-MODEL_REPO = os.getenv("GEMMA_2B_GGUF_REPO", "bartowski/gemma-2-2b-it-GGUF")
-MODEL_FILE = os.getenv("GEMMA_2B_GGUF_FILE", "gemma-2-2b-it-Q4_K_M.gguf")
-
-# Try common filename variants in case upstream naming changes.
-MODEL_FILE_CANDIDATES = [
-    MODEL_FILE,
-    "gemma-2-2b-it-Q4_K_M.gguf",
-    "gemma-2-2b-it-Q5_K_M.gguf",
-    "gemma-2-2b-it-Q8_0.gguf",
-]
-
-_llm = None
-_last_model_error = ""
-logger = logging.getLogger(__name__)
-
-
-class _FallbackModel:
-    def __call__(self, prompt: str, max_tokens: int = 512, temperature: float = 0.7, top_p: float = 0.95):
-        del temperature, top_p
-        text = (
-            "Inference backend is temporarily running in fallback mode. "
-            "The Gemma 2B GGUF model could not be initialized on this runtime."
-        )
-        prompt_text = str(prompt).strip()
-        if prompt_text:
-            text += f" Prompt received: {prompt_text[:160]}"
-        return {"choices": [{"text": text[:max_tokens]}]}
-
-
-def _download_model_file() -> str:
-    last_error = None
-    seen = set()
-    for filename in MODEL_FILE_CANDIDATES:
-        if filename in seen:
-            continue
-        seen.add(filename)
-        try:
-            return str(hf_hub_download(repo_id=MODEL_REPO, filename=filename))
-        except Exception as exc:
-            last_error = exc
-            logger.warning("Failed to download GGUF file '%s' from %s", filename, MODEL_REPO)
-
-    raise RuntimeError(
-        f"Could not download a GGUF file from '{MODEL_REPO}'. "
-        f"Tried: {MODEL_FILE_CANDIDATES}. Last error: {last_error}"
-    )
-
-
-def load_model():
-    global _llm, _last_model_error
-    if _llm is None:
-        if Llama is None:
-            _last_model_error = f"llama-cpp import failed: {_llama_import_error}"
-            logger.error(_last_model_error)
-            _llm = _FallbackModel()
-            return _llm
-
-        try:
-            model_path = _download_model_file()
-            _llm = Llama(
-                model_path=model_path,
-                n_ctx=int(os.getenv("GEMMA_2B_CTX", "4096")),
-                n_threads=int(os.getenv("GEMMA_2B_THREADS", "4")),
-                n_gpu_layers=0,
-            )
-            _last_model_error = ""
-        except Exception:
-            logger.exception("Gemma 2B GGUF initialization failed; falling back")
-            _last_model_error = "gguf initialization failed; check Space logs for traceback"
-            _llm = _FallbackModel()
-    return _llm
-
-
-@spaces.GPU(duration=90)
-def generate_text(model_name: str, prompt: str, max_new_tokens: int = 512) -> str:
-    del model_name
-    try:
-        llm = load_model()
-        output = llm(
-            str(prompt),
-            max_tokens=min(max_new_tokens, 512),
-            temperature=0.7,
-            top_p=0.95,
-        )
-        return output["choices"][0]["text"].strip()
-
-    except Exception:
-        logger.exception("Text generation failed")
-        raise
-
-
-@spaces.GPU(duration=120)
-def generate_with_image(prompt: str, image_base64: str, max_new_tokens: int = 512) -> str:
-    try:
-        _ = Image.open(io.BytesIO(base64.b64decode(image_base64)))
-        text_prompt = f"{prompt}\n\n[Image provided and parsed on client-side.]"
-        return generate_text("e2b", text_prompt, max_new_tokens)
-
-    except Exception:
-        logger.exception("Vision generation failed")
-        raise
-
-
-@spaces.GPU(duration=60)
-def health_check() -> dict:
-    return {
-        "status": "ok",
-        "model_loaded": _llm is not None and not isinstance(_llm, _FallbackModel),
-        "backend": "llama-cpp",
-        "model_repo": MODEL_REPO,
-        "model_file": MODEL_FILE,
-        "model_error": _last_model_error or None,
-    }

requirements.txt

@@ -1,10 +0,0 @@
-fastapi==0.112.2
-uvicorn==0.30.6
-gradio==4.44.1
-
-# Inference backend (quantized GGUF via llama-cpp)
-llama-cpp-python==0.2.90
-
-# Utilities
-Pillow
-huggingface_hub==0.25.2

runtime.txt

@@ -1 +0,0 @@
-3.10 

scratch.py

@@ -1,2 +0,0 @@
-import llama_cpp
-print(llama_cpp.__version__)