DESIGN.md

Design Note

Voice Notes is a FastAPI + SQLite service with self-hosted faster-whisper STT, a provider-agnostic OpenAI-compatible LLM, and a vanilla JS dashboard. README covers setup and API usage; this note covers the important design decisions.

Architecture

Browser: upload / live PCM / typed text
   |           |              |
 ffmpeg     wrap WAV          |
   \__________/               |
        |                     |
  faster-whisper              |
        \_____________________/
                  |
       Note saved before the LLM
                  |
   LLM -> Pydantic validation -> one repair retry
                  |
   task action -> {note, intent, action}

• app/api handles HTTP validation, status codes, pagination, and WebSocket setup.
• app/services contains orchestration, ffmpeg/STT, LLM extraction, task resolution, and streaming.
• app/repositories.py owns every SQLAlchemy query; app/models.py defines notes and tasks.
• STT and LLM sit behind Transcriber and IntentExtractor protocols, so tests use fakes instead of downloading a model or calling the network.

Key invariants

1. A successful transcription becomes a committed note before LLM extraction.
2. A failed LLM or task action never deletes the transcript; the note keeps processing_error.
3. Task mutation and note linking commit together, or roll back together.
4. The final full transcription is authoritative; live partials are provisional.
5. Task lookup is deterministic and never guesses on ambiguity.
6. Timestamps are UTC; due dates are date-only.

Data model

Table	Important fields
notes	id, raw_transcript, created_at, source, task_id, intent, processing_error
tasks	id, title, description, status, priority, due_date, created_at, updated_at

A note optionally links to one task. Deleting a task sets note.task_id to NULL; notes are never cascade-deleted. SQLite foreign-key enforcement is enabled per connection so that behavior actually holds.

Request flows

Voice upload: size-check the upload, convert to 16 kHz mono WAV with ffmpeg, transcribe with faster-whisper, persist the note, extract the intent, execute the action, return {note, intent, action}. If there is no transcript, no note is created. If the LLM or task action fails after transcription, the note remains.

Text: POST /api/notes joins the same pipeline after note creation with source=text. Voice and text share validation, extraction, and task behavior.

Live stream: the browser sends 16 kHz mono int16 PCM over WS /api/voice/stream. The server buffers it, emits provisional partials from rolling 10-second chunks, and on stop transcribes the complete recording before running the normal pipeline. cancel or disconnect discards the buffer.

LLM prompt and validation

The client uses only the common OpenAI-compatible contract: POST {LLM_BASE_URL}/chat/completions, reading choices[0].message.content. It does not require tools, streaming, or provider-specific structured output.

The system prompt includes:

• current UTC datetime plus tomorrow and the current week bounds;
• definitions of CREATE_TASK, UPDATE_TASK_STATUS, QUERY_TASKS, GENERAL_NOTE;
• the exact JSON shape and per-intent rules;
• "use null instead of guessing" instructions;
• a prompt-injection guard treating the transcript as data;
• four short few-shot examples with dynamic dates.

Output handling:

1. Strip optional markdown fences, take the first-to-last brace span, and require a JSON object.
2. Validate with Pydantic: enums, dates, non-blank fields, date ranges, and cross-field payload rules (for example, CREATE_TASK requires task and must not include update/query).
3. On invalid JSON or schema errors, append the bad output and error to the prompt and ask for corrected JSON once. Default is two attempts total.
4. Terminal validation failure returns 422 intent_extraction_failed with the persisted note attached.
5. Network, timeout, HTTP, malformed-envelope, empty-content, and unconfigured LLM failures return 503 llm_unavailable and are not retried.

Using one repair retry keeps latency and cost bounded while fixing the common case: formatting mistakes. Availability failures belong to infrastructure, not prompt repair.

Intent execution

Intent	Behavior
CREATE_TASK	Creates a PENDING task; missing priority defaults to MEDIUM; links the note.
UPDATE_TASK_STATUS	Resolves exact ID first, then normalized title; updates and links the note.
QUERY_TASKS	Applies status/priority/due filters, returns up to 100 tasks plus total; note is not linked.
GENERAL_NOTE	Saves the note only.

Task resolution lowercases and strips punctuation, removes common stop words, and compares the reference to task titles using substring or token overlap. The best score wins; ties return 409 with candidate IDs/titles. No match returns 404. Cancelled tasks are excluded from fuzzy matching but can still be found by exact ID. Guessing would be worse than asking the client to disambiguate.

API and errors

The API exposes health, voice upload, WebSocket streaming, text notes, paginated notes with source/date/task filters, and paginated task reads/updates/deletes with status/priority/due/title filters. Full request/response details are in the README.

REST errors use {"error": {"code", "message", "details"}}; WebSocket errors use {"type": "error", "error": {...}}. Common statuses are 413 (too large), 422 (validation/audio/intent failure), 503 (LLM unavailable), 404 (missing), 409 (ambiguous task), and 500 (unexpected error). Intent-related failures attach the persisted note at the top level of the response.

STT, streaming, and runtime

• faster-whisper is lazy-loaded and cached; inference is serialized with a lock.
• Uploads use ffmpeg normalization, VAD, and the configured beam size.
• Live partials re-transcribe a rolling window with beam_size=1; the final pass uses the normal beam size. This is not word-by-word streaming, but it gives useful feedback without a streaming STT server.
• If AudioWorklet or the WebSocket is unavailable, the dashboard falls back to MediaRecorder/file upload.
• SQLite uses create_all at startup; there is no migration tool. All queries are in the repository layer, so Postgres can replace it later.
• Sessions are per request; expire_on_commit=False keeps response serialization simple. GET /health does not depend on the LLM, so the dashboard and docs are usable before configuration.

Security

Implemented: prompt-injection guard, escaped dashboard output, SQLAlchemy parameter binding, temporary upload files with size/time limits, and ignored .env secrets.

Not implemented and intentionally out of scope: authentication, per-user isolation, rate limiting, CSRF protection (no cookie auth), moderation, and audit logs. Do not expose this service publicly without adding auth and rate limits.

Testing

The suite has 40 offline tests using fake STT/LLM adapters; it never downloads Whisper or calls a network LLM. It covers voice/text pipelines, all intents, filters and pagination, ambiguous task references, invalid LLM output and note preservation, upload limits, deletion behavior, frontend/static serving, and WebSocket streaming. Coverage is about 88% of app/; the gaps are mostly real adapters and error branches. The real adapters were smoke-tested manually with samples/create_task.wav.

Run uv run pytest, uv run ruff check ., and uv run ruff format --check .. See the README for the quick start and end-to-end curl.

Trade-offs

Decision	Benefit	Cost
Note committed before LLM	Transcript durability	One extra write on failure
Bring-your-own OpenAI-compatible LLM	No provider lock-in or bundled key	Reviewer must configure one
Synchronous processing	Simple and testable	Slow model holds the request
SQLite + create_all	Zero setup	Single-node, no migrations
Heuristic task matching	Deterministic and cheap	Paraphrases can miss
Rolling-window partials	Live feedback without streaming STT	Extra CPU; provisional text
No framework dashboard	No build step or Node	Manual DOM/state management
No auth or semantic search	Smaller scope	Not production-ready; no fuzzy note retrieval

What I would probably do with more time

1. System App. Making this a system app that can be activated with a wake word seems like a most obvious thing.
2. Background processing. Move STT/extraction to a queue with status endpoints.
3. Better streaming STT. Current is slow.
4. Semantic search. Embed raw_transcript into FAISS/Qdrant/pgvector; add search.
5. Observability. Structured logs, stage latency, request IDs, and the like.