Spaces:
Sleeping
Sleeping
Aissugur Kurganbekov
Claude Opus 4.8 (1M context)
Add recruiter-facing docs: case study, architecture, interview notes, demo, CV, outreach
b2f4ef1 | # Case Study — FinanceFlow AI | |
| A concise product case study of why this exists, what was built, and what the | |
| results were. | |
| ## Problem | |
| Finance and operations teams answer the same small questions hundreds of times a | |
| week — *"What's the total due? When is it overdue? What's the payment term? What's | |
| the late-fee clause?"* — by opening a PDF and scanning for the number. It's slow, | |
| repetitive, and error-prone. Dropping a general chatbot on top makes it worse: an | |
| LLM will confidently produce a due date or dollar amount that isn't in the | |
| document. In finance, a confidently wrong number is more dangerous than no answer. | |
| ## User pain | |
| - **Time:** minutes per lookup, multiplied across invoices/contracts/notes. | |
| - **Trust:** no way to see *where* an answer came from, so every AI answer has to | |
| be re-verified by hand — which defeats the point. | |
| - **Risk:** hallucinated figures can flow into payments, reconciliations, or | |
| customer comms. | |
| ## Product hypothesis | |
| If every answer is (a) built only from retrieved passages, (b) shown with the | |
| exact source excerpt, and (c) **refused** when the evidence is weak, then a | |
| finance user will trust the tool enough to use it for real lookups — because they | |
| can verify each answer in one glance and know the system won't invent facts. | |
| ## MVP scope | |
| Deliberately small and demoable: | |
| - Upload PDF/TXT → parse → chunk → store. | |
| - Ask a question against one document → retrieve top chunks → answer **only** from | |
| them → return citations (document, page/chunk, excerpt, score). | |
| - Abstain with a fixed message when evidence is weak. | |
| - Dashboard (status, chunk counts, history, delete) + one-click demo mode. | |
| - A golden-dataset **evaluation** that scores grounding, not just accuracy. | |
| Out of scope (on purpose): auth, multi-tenant storage, vector DB, OCR, | |
| cross-document Q&A. | |
| ## Technical approach | |
| - **Backend:** FastAPI + SQLite (SQLAlchemy). A single ingestion pipeline | |
| (parse → chunk → store) shared by uploads and the eval harness. | |
| - **Chunking:** overlapping ~120-word windows (25-word overlap), page numbers | |
| preserved for PDFs. | |
| - **Retrieval:** a custom TF-IDF keyword ranker (no ML dependency, fully offline) | |
| — rarer shared terms weigh more, normalized by chunk length. | |
| - **Answering:** extractive by default — return the densest verbatim span, | |
| re-centered on the matched terms so a value next to its label | |
| (`Total Amount Due $2,413.98`) is preserved. Optional LLM layer (OpenAI/ | |
| Anthropic) behind env vars, with automatic fallback. | |
| - **Anti-hallucination:** two abstention gates (score floor + term coverage), | |
| citations always shown, and `unsupported_answer_count` tracked in evaluation. | |
| ## Tradeoffs | |
| | Decision | Why | Cost | | |
| | -------- | --- | ---- | | |
| | Keyword TF-IDF over embeddings | Zero deps, offline, transparent, fast to demo | Misses pure paraphrases | | |
| | Extractive over generative default | Cannot invent a number not in the source | Phrasing is less fluent than an LLM | | |
| | Discard raw files after parsing | Privacy + no orphaned-file cleanup | Can't re-process without re-upload | | |
| | SQLite | One-file, zero setup for a reviewer | Not for concurrent production load | | |
| | Abstain aggressively | Trust > coverage in finance | Occasionally refuses a borderline-answerable question | | |
| ## Evaluation method | |
| A **golden dataset** (`backend/app/golden_dataset.json`) pairs each question with | |
| its target document, expected behaviour (`answerable` / `should_abstain`), and the | |
| evidence string that should be cited. The harness runs every question through the | |
| real answer pipeline and reports: `answered_with_citation`, `correct_abstentions`, | |
| `unsupported_answer_count`, `citation_coverage`, and `evidence_match_rate`. | |
| Runnable from the UI **and** the CLI (`python -m app.eval_cli`); CI fails on any | |
| unsupported answer. | |
| ## Results | |
| - **7/7 golden questions pass**, `citation_coverage = 1.0`, `evidence_match_rate | |
| = 1.0`, **`unsupported_answer_count = 0`**, 2 correct abstentions. | |
| - **20 backend tests** pass (unit + API via TestClient), green in CI. | |
| - Runs fully offline with no API key; one-click demo seeds three sample documents. | |
| ## What I would build next | |
| 1. Swap the keyword ranker for sentence-embedding retrieval (FAISS / sqlite-vec) | |
| to handle paraphrased questions, and A/B it against the current ranker using | |
| the same golden dataset. | |
| 2. Exact answer-span highlighting inside the cited excerpt. | |
| 3. Cross-document Q&A and document collections. | |
| 4. Auth + per-tenant isolation; background processing + OCR for scanned PDFs. | |
| 5. Expand the golden set and add precision/recall on citations. | |
| ## Why this matters for AI workflow automation | |
| This is the core loop behind most useful document AI: **ingest → retrieve → | |
| answer → cite → evaluate**, with guardrails that make it safe to put in front of | |
| a real team. The interesting work isn't calling a model — it's the retrieval | |
| quality, the abstention policy, and the evaluation that proves the system stays | |
| grounded. That's exactly the work of shipping trustworthy AI features in fintech | |
| and operations. | |