| --- |
| title: Devseis Chat |
| emoji: 💬 |
| colorFrom: blue |
| colorTo: indigo |
| sdk: static |
| pinned: false |
| license: apache-2.0 |
| --- |
| |
| # Nox — Devseis's free AI assistant |
|
|
| Nox is a general-purpose AI chat assistant built for Devseis to a hard **$0/month** |
| operating constraint. Everything below runs on free tiers — no paid APIs, no rented |
| servers, no billing surprises. This file is meant to fully brief a new session (human |
| or AI) on what exists today and how it works, without needing local machine access. |
|
|
| ## Architecture |
|
|
| - **Frontend**: this HF Space (`Devseis/Nox`), `sdk: static` — a single `index.html` |
| (plain HTML/CSS/vanilla JS, no framework, no build step) plus a small service worker |
| (`sw.js`) for PWA installability. Served at `devseis-nox.static.hf.space`. |
| - **Backend**: Cloudflare Worker `devseis-chat-proxy` |
| (`devseis-chat-proxy.umer-wasim.workers.dev`), source lives outside this repo (deployed |
| via `wrangler`, see **Deploying changes** below — it is not tracked in this HF repo). |
| - **Storage**: one Cloudflare KV namespace (`USAGE_KV`) — usage counters, feedback |
| records, and server-side RAG document chunks (all with TTLs, nothing kept forever). |
| - **Models**: Cloudflare Workers AI (`@cf/meta/llama-3.2-1b-instruct` "Quick", |
| `@cf/meta/llama-3.2-3b-instruct` "Standard", `@cf/meta/llama-3.1-8b-instruct-fp8` |
| "Smart"), `@cf/llava-hf/llava-1.5-7b-hf` for vision, `@cf/baai/bge-m3` for embeddings. |
|
|
| ## Features, and how each is actually built |
|
|
| ### Chat |
| Standard streamed chat against Cloudflare Workers AI (SSE), with three model tiers the |
| visitor can pick from. A daily message cap (`DAILY_MESSAGE_CAP = 1000`) protects the |
| free Neuron allocation; a crisis-keyword regex (`CRISIS_RE`) intercepts self-harm |
| language **before** any model call and returns a fixed, sourced response (SOS Détresse |
| Luxembourg / 112) — a hard code-level guard, not a prompt instruction, because prompt-only |
| guards have been proven to fail under sustained conversational pressure (see Lex's |
| history below). |
|
|
| ### Live data grounding (no paid search API) |
| Regex intent-matchers in the worker route specific question types to free, deterministic |
| APIs instead of letting the model guess or hallucinate: |
| - Weather / air quality / moon phase → Open-Meteo |
| - News → BBC RSS |
| - Currency conversion → Frankfurter.app (ECB rates) |
| - Luxembourg public holidays → date.nager.at |
| - Current time → deterministic timezone math |
| - General facts / population / office-holders → DuckDuckGo Instant Answer + Wikipedia |
| REST summary + Wikidata claims (with entity-description disambiguation so "mayor of |
| Luxembourg" the city doesn't resolve to a head of state of the wrong entity) |
|
|
| Google/Bing search and any paid maps/places API were ruled out by the cost constraint; |
| DuckDuckGo's own HTML search page was tested directly and rejected (anti-bot challenge on |
| automated requests) before landing on the Instant Answer API. |
|
|
| ### Documents — server-side RAG |
| Upload a PDF/text document while in **RAG mode** → the worker chunks it |
| (`RAG_CHUNK_SIZE = 800` chars, 100 overlap), embeds each chunk (`bge-m3`), and stores |
| `{text, vector, name}` in KV, scoped by conversation id, TTL 24h, capped at 300 chunks/ |
| conversation. Every later question in that conversation gets its own question embedded |
| and compared via in-Worker cosine similarity (deliberately not Vectorize — a chat- |
| uploaded document is a handful of pages, a plain JS loop is simpler and fast enough at |
| this scale). Replaces an older design that stuffed up to 12,000 chars of raw document |
| text into every message. |
|
|
| - **Adjustable retrieval breadth**: sidebar "Chunks per answer" (1–8, `ragTopK` in the |
| request body, clamped server-side) — shared by both server and offline RAG. |
| - **Source transparency**: every reply that used retrieved excerpts gets a "📎 N" badge; |
| clicking it shows each chunk's document name, closeness score, and a text preview. |
| Delivered via an `X-Rag-Chunks` response header on the streamed reply (headers arrive |
| before the body finishes, so this doesn't require touching the raw SSE stream) or a |
| `ragChunks` JSON field on the buffered fallback path; CORS |
| `Access-Control-Expose-Headers` had to explicitly list it or the browser hides it. |
| - **Manage indexed documents**: sidebar "📚 View" (only shown once ≥1 document is |
| indexed, hidden again once the last one is removed) lists every document with its |
| chunk count and a running total ("N documents — M chunks / M embeddings" — one |
| embedding per chunk, always, so that's really one number shown two ways), with a |
| Remove button per document. Removing a document is just filtering it out of the flat |
| chunk list and re-saving — there's no separate index to rebuild. |
| - **Multi-file upload**: selecting several files at once indexes each individually with |
| combined progress/summary feedback. |
|
|
| ### Documents — fully offline RAG (local mode) |
| When "Local AI mode" is active (see below), RAG works identically but entirely |
| client-side: chunking mirrors the server's algorithm exactly, embedding uses |
| `Xenova/all-MiniLM-L6-v2` via `@huggingface/transformers` on WebGPU, and chunks are |
| stored in IndexedDB instead of KV. The document never leaves the device. Source |
| transparency and the adjustable top-K setting both work here too, computed from the |
| same `ragTopK` value. |
|
|
| ### Local AI mode (offline fallback) |
| A toggle (or automatic offer, once the free server allocation is exhausted for the day) |
| loads `Llama-3.2-1B-Instruct-q4f16_1-MLC` via WebLLM directly in the browser (requires |
| WebGPU). Chosen deliberately small so it downloads and runs on phones, not just |
| desktops — a real capability gap vs. the server's best model, disclosed to the user in |
| an in-app tooltip, not hidden. |
|
|
| Two real bugs were found and fixed here through actual testing, not by inspection: |
| - **Context-window overflow**: the local model's real context window is 4096 tokens. |
| Chat mode's conversation history was never bounded the way the server path is (capped |
| at the last 20 messages server-side) — a conversation with even one document |
| attachment kept growing every turn until it hit `ContextWindowSizeExceededError`, |
| which then crashed the WebGPU device entirely ("Device was lost"), breaking every |
| later local message until a full page reload. Fixed with `trimHistoryForLocalEngine()` |
| (keeps the system message + as many recent messages as fit a conservative |
| chars/2.5-per-token budget, always keeping at least the message just sent), a lower |
| one-shot-document cap (2,500 chars, down from an initial 4,000), and a catch block |
| that tells context-overflow and device-loss apart from generic errors, clearing the |
| cached engine reference on device loss so the *next* attempt auto-recovers instead of |
| staying broken. |
| - **Double-wrapped attachments**: one-shot document text was being labeled |
| `[Attached file: name]` at file-select time AND wrapped again at send time, on every |
| attachment (not just multi-file) — a real, confusing malformed-prompt bug that had |
| been produacing part of what first looked like pure "the model is too small" garbage |
| output. Fixed by only pre-labeling per-file when multiple files are combined into one |
| message, and skipping the outer wrapper in that case. |
|
|
| Local mode cannot read images (text-only engine — a real capability gap, not a |
| size-tunable limit). |
|
|
| ### Reliability |
| If Cloudflare Workers AI returns a confirmed Neuron-exhaustion error, the worker |
| transparently falls back to Google Gemini's free tier |
| (`gemini-flash-lite-latest` — the "-latest" bigger alias was seen hitting real 503 |
| "high demand" errors in testing, the lite variant didn't), then OpenRouter's free tier |
| (`nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free` — OpenRouter's own auto-router |
| picked models that returned empty content more often than not; this one didn't), each |
| with an 8s timeout so one slow provider can't stall the whole request. Scoped to the |
| text chat path only — the vision endpoint already degrades gracefully on its own and |
| would need real separate per-provider multimodal handling to extend this to it. |
|
|
| **OpenAI, scoped narrowly, unlike Lex**: `OPENAI_API_KEY` is set on this worker too, but |
| tried first **only when the reply is genuinely grounded in retrieved document excerpts** |
| (`ragChunksMeta.length > 0` — a document was actually uploaded and something relevant |
| was found for this question), streamed via `callOpenAIStream` the same way Cloudflare's |
| own reply streams. Plain chat and live-data questions never touch it and stay entirely |
| on the free Cloudflare tiers regardless of which model tier the visitor picked — a |
| deliberate choice to keep OpenAI's real cost scoped to where it was shown to matter most |
| (synthesizing several retrieved excerpts faithfully) rather than all of Nox's traffic |
| the way it ended up scoped for Lex. Capped at `OPENAI_DAILY_CALL_CAP = 200`/day |
| (KV-tracked, separate from `DAILY_MESSAGE_CAP`), a smaller number than Lex's cap since |
| this only fires for document-grounded replies, not every message. |
|
|
| ### Everything else |
| - **PWA**: installable, small app-shell cache via `sw.js` (network-first for |
| `index.html` so a redeploy is visible immediately when online; falls back to cache |
| only when genuinely offline). |
| - **Multi-language**: UI + reply language for English, French, German, Luxembourgish, |
| Portuguese; speech-to-text input and text-to-speech output (Luxembourgish has no |
| browser TTS/STT support anywhere, so it maps to English rather than silently failing). |
| - **Sharing/export**: read-only share links, QR code, text/PDF export. |
| - **Feedback**: 👍/👎 per reply, stored in KV. |
| - **Privacy**: a visitor id (not an account) is used only for daily-cap/unique-user |
| counting; local mode and offline RAG keep data entirely on-device; dedicated |
| `privacy.html` page. |
|
|
| ## Deploying changes |
|
|
| - **Frontend**: edit `index.html` (or other files) in this repo, `git commit`, |
| `git push` — the HF static Space rebuilds and is live within seconds. There is no |
| staging/preview for the frontend; test locally (a plain `python3 -m http.server` plus |
| a headless-browser smoke test against a Cloudflare **preview** worker version has been |
| the working pattern) before pushing. |
| - **Worker**: source is NOT in this repo. Deploy via `wrangler versions upload` first |
| (produces a preview URL, e.g. `https://<version>-devseis-chat-proxy.umer-wasim.workers.dev`) |
| to test against real Cloudflare infra without touching production, then |
| `wrangler versions deploy <id> --yes` to promote to 100% traffic. Never |
| `wrangler deploy` directly for changes to the live service. |
| - Pulling the current live worker source or settings requires a Cloudflare API token, |
| read from `~/Library/Preferences/.wrangler/config/default.toml` via a Python script |
| (not an inline bash one-liner — sandboxed shells tend to block inline commands that |
| read the token and curl with it, even for the account's own resources). |
|
|
| ## Known limitations (disclosed, not hidden) |
|
|
| - The local 1B model's summarization/instruction-following quality is noticeably worse |
| than the server models — expected from a model this size, not a bug. RAG mode |
| generally gives more reliable answers than local Chat mode's one-shot document dump, |
| since it retrieves small named excerpts rather than asking the model to hold and |
| reason over a whole document in one pass. |
| - No automated test suite — changes are verified by real preview-URL/browser testing |
| before promotion, not by a CI pipeline. |
|
|
| ## Future plan: document → fine-tuned model pipeline |
|
|
| Goal: a visitor uploads documents, the app generates synthetic Q&A training examples |
| from them, trains a LoRA adapter, and serves it back for download — still $0/month. |
|
|
| **Chosen approach**: Hugging Face Spaces + ZeroGPU (shared, free, on-demand H200 access, |
| GPU allocated only while a decorated `@spaces.GPU` function runs). **This requires a |
| Gradio Space** — Nox's current frontend is static HTML, not Gradio, so this cannot be |
| bolted onto the existing Nox Space. It would need to be built as a **separate, |
| companion HF Space** (still free to host) rather than a feature added to Nox itself. |
|
|
| **The real open risk, not yet validated**: ZeroGPU's daily quota is small — |
| **120 seconds/day for an unauthenticated visitor, 210 seconds/day for a free signed-in |
| HF account, 1,500 seconds/day for HF PRO ($9/month)**. A single LoRA training call |
| alone could plausibly need most or all of that budget, before accounting for the |
| synthetic-example-generation step (also LLM inference, ideally also GPU-backed to stay |
| free). Paying for HF PRO to make this practically usable would also break the $0 |
| constraint. **Before any further build work**: time a real LoRA training call against |
| the smallest realistic base model and dataset, and check whether it fits inside the |
| 120–210 second window at all. That single number determines whether this pipeline is |
| viable as designed, or needs a fundamentally different approach (e.g., training |
| asynchronously across multiple queued GPU calls with real checkpoint save/resume, which |
| is a meaningfully bigger build than currently sketched). |
|
|
| Stages, at a glance (see prior planning notes for full detail): upload (PDF/DOCX/TXT, |
| chunked like RAG) → synthetic Q&A generation per chunk via an LLM call → dataset |
| format/validation → `@spaces.GPU`-wrapped LoRA training (`peft` + `transformers`) → |
| package `adapter_model.safetensors` + `adapter_config.json` for download. Optional, |
| separate follow-on: serving the resulting adapter via Cloudflare Workers AI has its own |
| constraints (allowed base models, rank ≤ 8, exact filenames, `model_type` field added |
| manually, <100MB) that should be re-verified against current Cloudflare docs before |
| relying on them, since they weren't independently confirmed when first written down. |
|
|
