README.md

metadata

title: Devseis Chat
emoji: 💬
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
license: apache-2.0

Nox — Devseis's free AI assistant

Nox is a general-purpose AI chat assistant built for Devseis to a hard $0/month operating constraint. Everything below runs on free tiers — no paid APIs, no rented servers, no billing surprises. This file is meant to fully brief a new session (human or AI) on what exists today and how it works, without needing local machine access.

Architecture

• Frontend: this HF Space (Devseis/Nox), sdk: static — a single index.html (plain HTML/CSS/vanilla JS, no framework, no build step) plus a small service worker (sw.js) for PWA installability. Served at devseis-nox.static.hf.space.
• Backend: Cloudflare Worker devseis-chat-proxy (devseis-chat-proxy.umer-wasim.workers.dev), source lives outside this repo (deployed via wrangler, see Deploying changes below — it is not tracked in this HF repo).
• Storage: one Cloudflare KV namespace (USAGE_KV) — usage counters, feedback records, and server-side RAG document chunks (all with TTLs, nothing kept forever).
• Models: Cloudflare Workers AI (@cf/meta/llama-3.2-1b-instruct "Quick", @cf/meta/llama-3.2-3b-instruct "Standard", @cf/meta/llama-3.1-8b-instruct-fp8 "Smart"), @cf/llava-hf/llava-1.5-7b-hf for vision, @cf/baai/bge-m3 for embeddings.

Features, and how each is actually built

Chat

Standard streamed chat against Cloudflare Workers AI (SSE), with three model tiers the visitor can pick from. A daily message cap (DAILY_MESSAGE_CAP = 1000) protects the free Neuron allocation; a crisis-keyword regex (CRISIS_RE) intercepts self-harm language before any model call and returns a fixed, sourced response (SOS Détresse Luxembourg / 112) — a hard code-level guard, not a prompt instruction, because prompt-only guards have been proven to fail under sustained conversational pressure (see Lex's history below).

Live data grounding (no paid search API)

Regex intent-matchers in the worker route specific question types to free, deterministic APIs instead of letting the model guess or hallucinate:

• Weather / air quality / moon phase → Open-Meteo
• News → BBC RSS
• Currency conversion → Frankfurter.app (ECB rates)
• Luxembourg public holidays → date.nager.at
• Current time → deterministic timezone math
• General facts / population / office-holders → DuckDuckGo Instant Answer + Wikipedia REST summary + Wikidata claims (with entity-description disambiguation so "mayor of Luxembourg" the city doesn't resolve to a head of state of the wrong entity)

Google/Bing search and any paid maps/places API were ruled out by the cost constraint; DuckDuckGo's own HTML search page was tested directly and rejected (anti-bot challenge on automated requests) before landing on the Instant Answer API.

Documents — server-side RAG

Upload a PDF/text document while in RAG mode → the worker chunks it (RAG_CHUNK_SIZE = 800 chars, 100 overlap), embeds each chunk (bge-m3), and stores {text, vector, name} in KV, scoped by conversation id, TTL 24h, capped at 300 chunks/ conversation. Every later question in that conversation gets its own question embedded and compared via in-Worker cosine similarity (deliberately not Vectorize — a chat- uploaded document is a handful of pages, a plain JS loop is simpler and fast enough at this scale). Replaces an older design that stuffed up to 12,000 chars of raw document text into every message.

• Adjustable retrieval breadth: sidebar "Chunks per answer" (1–8, ragTopK in the request body, clamped server-side) — shared by both server and offline RAG.
• Source transparency: every reply that used retrieved excerpts gets a "📎 N" badge; clicking it shows each chunk's document name, closeness score, and a text preview. Delivered via an X-Rag-Chunks response header on the streamed reply (headers arrive before the body finishes, so this doesn't require touching the raw SSE stream) or a ragChunks JSON field on the buffered fallback path; CORS Access-Control-Expose-Headers had to explicitly list it or the browser hides it.
• Manage indexed documents: sidebar "📚 View" (only shown once ≥1 document is indexed, hidden again once the last one is removed) lists every document with its chunk count and a running total ("N documents — M chunks / M embeddings" — one embedding per chunk, always, so that's really one number shown two ways), with a Remove button per document. Removing a document is just filtering it out of the flat chunk list and re-saving — there's no separate index to rebuild.
• Multi-file upload: selecting several files at once indexes each individually with combined progress/summary feedback.

Documents — fully offline RAG (local mode)

When "Local AI mode" is active (see below), RAG works identically but entirely client-side: chunking mirrors the server's algorithm exactly, embedding uses Xenova/all-MiniLM-L6-v2 via @huggingface/transformers on WebGPU, and chunks are stored in IndexedDB instead of KV. The document never leaves the device. Source transparency and the adjustable top-K setting both work here too, computed from the same ragTopK value.

Local AI mode (offline fallback)

A toggle (or automatic offer, once the free server allocation is exhausted for the day) loads Llama-3.2-1B-Instruct-q4f16_1-MLC via WebLLM directly in the browser (requires WebGPU). Chosen deliberately small so it downloads and runs on phones, not just desktops — a real capability gap vs. the server's best model, disclosed to the user in an in-app tooltip, not hidden.

Two real bugs were found and fixed here through actual testing, not by inspection:

• Context-window overflow: the local model's real context window is 4096 tokens. Chat mode's conversation history was never bounded the way the server path is (capped at the last 20 messages server-side) — a conversation with even one document attachment kept growing every turn until it hit ContextWindowSizeExceededError, which then crashed the WebGPU device entirely ("Device was lost"), breaking every later local message until a full page reload. Fixed with trimHistoryForLocalEngine() (keeps the system message + as many recent messages as fit a conservative chars/2.5-per-token budget, always keeping at least the message just sent), a lower one-shot-document cap (2,500 chars, down from an initial 4,000), and a catch block that tells context-overflow and device-loss apart from generic errors, clearing the cached engine reference on device loss so the next attempt auto-recovers instead of staying broken.
• Double-wrapped attachments: one-shot document text was being labeled [Attached file: name] at file-select time AND wrapped again at send time, on every attachment (not just multi-file) — a real, confusing malformed-prompt bug that had been produacing part of what first looked like pure "the model is too small" garbage output. Fixed by only pre-labeling per-file when multiple files are combined into one message, and skipping the outer wrapper in that case.

Local mode cannot read images (text-only engine — a real capability gap, not a size-tunable limit).

Reliability

If Cloudflare Workers AI returns a confirmed Neuron-exhaustion error, the worker transparently falls back to Google Gemini's free tier (gemini-flash-lite-latest — the "-latest" bigger alias was seen hitting real 503 "high demand" errors in testing, the lite variant didn't), then OpenRouter's free tier (nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free — OpenRouter's own auto-router picked models that returned empty content more often than not; this one didn't), each with an 8s timeout so one slow provider can't stall the whole request. Scoped to the text chat path only — the vision endpoint already degrades gracefully on its own and would need real separate per-provider multimodal handling to extend this to it.

OpenAI, scoped narrowly, unlike Lex: OPENAI_API_KEY is set on this worker too, but tried first only when the reply is genuinely grounded in retrieved document excerpts (ragChunksMeta.length > 0 — a document was actually uploaded and something relevant was found for this question), streamed via callOpenAIStream the same way Cloudflare's own reply streams. Plain chat and live-data questions never touch it and stay entirely on the free Cloudflare tiers regardless of which model tier the visitor picked — a deliberate choice to keep OpenAI's real cost scoped to where it was shown to matter most (synthesizing several retrieved excerpts faithfully) rather than all of Nox's traffic the way it ended up scoped for Lex. Capped at OPENAI_DAILY_CALL_CAP = 200/day (KV-tracked, separate from DAILY_MESSAGE_CAP), a smaller number than Lex's cap since this only fires for document-grounded replies, not every message.

Everything else

• PWA: installable, small app-shell cache via sw.js (network-first for index.html so a redeploy is visible immediately when online; falls back to cache only when genuinely offline).
• Multi-language: UI + reply language for English, French, German, Luxembourgish, Portuguese; speech-to-text input and text-to-speech output (Luxembourgish has no browser TTS/STT support anywhere, so it maps to English rather than silently failing).
• Sharing/export: read-only share links, QR code, text/PDF export.
• Feedback: 👍/👎 per reply, stored in KV.
• Privacy: a visitor id (not an account) is used only for daily-cap/unique-user counting; local mode and offline RAG keep data entirely on-device; dedicated privacy.html page.

Deploying changes

• Frontend: edit index.html (or other files) in this repo, git commit, git push — the HF static Space rebuilds and is live within seconds. There is no staging/preview for the frontend; test locally (a plain python3 -m http.server plus a headless-browser smoke test against a Cloudflare preview worker version has been the working pattern) before pushing.
• Worker: source is NOT in this repo. Deploy via wrangler versions upload first (produces a preview URL, e.g. https://<version>-devseis-chat-proxy.umer-wasim.workers.dev) to test against real Cloudflare infra without touching production, then wrangler versions deploy <id> --yes to promote to 100% traffic. Never wrangler deploy directly for changes to the live service.
• Pulling the current live worker source or settings requires a Cloudflare API token, read from ~/Library/Preferences/.wrangler/config/default.toml via a Python script (not an inline bash one-liner — sandboxed shells tend to block inline commands that read the token and curl with it, even for the account's own resources).

Known limitations (disclosed, not hidden)

• The local 1B model's summarization/instruction-following quality is noticeably worse than the server models — expected from a model this size, not a bug. RAG mode generally gives more reliable answers than local Chat mode's one-shot document dump, since it retrieves small named excerpts rather than asking the model to hold and reason over a whole document in one pass.
• No automated test suite — changes are verified by real preview-URL/browser testing before promotion, not by a CI pipeline.

Future plan: document → fine-tuned model pipeline

Goal: a visitor uploads documents, the app generates synthetic Q&A training examples from them, trains a LoRA adapter, and serves it back for download — still $0/month.

Chosen approach: Hugging Face Spaces + ZeroGPU (shared, free, on-demand H200 access, GPU allocated only while a decorated @spaces.GPU function runs). This requires a Gradio Space — Nox's current frontend is static HTML, not Gradio, so this cannot be bolted onto the existing Nox Space. It would need to be built as a separate, companion HF Space (still free to host) rather than a feature added to Nox itself.

The real open risk, not yet validated: ZeroGPU's daily quota is small — 120 seconds/day for an unauthenticated visitor, 210 seconds/day for a free signed-in HF account, 1,500 seconds/day for HF PRO ($9/month). A single LoRA training call alone could plausibly need most or all of that budget, before accounting for the synthetic-example-generation step (also LLM inference, ideally also GPU-backed to stay free). Paying for HF PRO to make this practically usable would also break the $0 constraint. Before any further build work: time a real LoRA training call against the smallest realistic base model and dataset, and check whether it fits inside the 120–210 second window at all. That single number determines whether this pipeline is viable as designed, or needs a fundamentally different approach (e.g., training asynchronously across multiple queued GPU calls with real checkpoint save/resume, which is a meaningfully bigger build than currently sketched).

Stages, at a glance (see prior planning notes for full detail): upload (PDF/DOCX/TXT, chunked like RAG) → synthetic Q&A generation per chunk via an LLM call → dataset format/validation → @spaces.GPU-wrapped LoRA training (peft + transformers) → package adapter_model.safetensors + adapter_config.json for download. Optional, separate follow-on: serving the resulting adapter via Cloudflare Workers AI has its own constraints (allowed base models, rank ≤ 8, exact filenames, model_type field added manually, <100MB) that should be re-verified against current Cloudflare docs before relying on them, since they weren't independently confirmed when first written down.