# Case Study — FinanceFlow AI

A concise product case study of why this exists, what was built, and what the
results were.

## Problem

Finance and operations teams answer the same small questions hundreds of times a
week — *"What's the total due? When is it overdue? What's the payment term? What's
the late-fee clause?"* — by opening a PDF and scanning for the number. It's slow,
repetitive, and error-prone. Dropping a general chatbot on top makes it worse: an
LLM will confidently produce a due date or dollar amount that isn't in the
document. In finance, a confidently wrong number is more dangerous than no answer.

## User pain

- **Time:** minutes per lookup, multiplied across invoices/contracts/notes.
- **Trust:** no way to see *where* an answer came from, so every AI answer has to
  be re-verified by hand — which defeats the point.
- **Risk:** hallucinated figures can flow into payments, reconciliations, or
  customer comms.

## Product hypothesis

If every answer is (a) built only from retrieved passages, (b) shown with the
exact source excerpt, and (c) **refused** when the evidence is weak, then a
finance user will trust the tool enough to use it for real lookups — because they
can verify each answer in one glance and know the system won't invent facts.

## MVP scope

Deliberately small and demoable:

- Upload PDF/TXT → parse → chunk → store.
- Ask a question against one document → retrieve top chunks → answer **only** from
  them → return citations (document, page/chunk, excerpt, score).
- Abstain with a fixed message when evidence is weak.
- Dashboard (status, chunk counts, history, delete) + one-click demo mode.
- A golden-dataset **evaluation** that scores grounding, not just accuracy.

Out of scope (on purpose): auth, multi-tenant storage, vector DB, OCR,
cross-document Q&A.

## Technical approach

- **Backend:** FastAPI + SQLite (SQLAlchemy). A single ingestion pipeline
  (parse → chunk → store) shared by uploads and the eval harness.
- **Chunking:** overlapping ~120-word windows (25-word overlap), page numbers
  preserved for PDFs.
- **Retrieval:** a custom TF-IDF keyword ranker (no ML dependency, fully offline)
  — rarer shared terms weigh more, normalized by chunk length.
- **Answering:** extractive by default — return the densest verbatim span,
  re-centered on the matched terms so a value next to its label
  (`Total Amount Due  $2,413.98`) is preserved. Optional LLM layer (OpenAI/
  Anthropic) behind env vars, with automatic fallback.
- **Anti-hallucination:** two abstention gates (score floor + term coverage),
  citations always shown, and `unsupported_answer_count` tracked in evaluation.

## Tradeoffs

| Decision | Why | Cost |
| -------- | --- | ---- |
| Keyword TF-IDF over embeddings | Zero deps, offline, transparent, fast to demo | Misses pure paraphrases |
| Extractive over generative default | Cannot invent a number not in the source | Phrasing is less fluent than an LLM |
| Discard raw files after parsing | Privacy + no orphaned-file cleanup | Can't re-process without re-upload |
| SQLite | One-file, zero setup for a reviewer | Not for concurrent production load |
| Abstain aggressively | Trust > coverage in finance | Occasionally refuses a borderline-answerable question |

## Evaluation method

A **golden dataset** (`backend/app/golden_dataset.json`) pairs each question with
its target document, expected behaviour (`answerable` / `should_abstain`), and the
evidence string that should be cited. The harness runs every question through the
real answer pipeline and reports: `answered_with_citation`, `correct_abstentions`,
`unsupported_answer_count`, `citation_coverage`, and `evidence_match_rate`.
Runnable from the UI **and** the CLI (`python -m app.eval_cli`); CI fails on any
unsupported answer.

## Results

- **7/7 golden questions pass**, `citation_coverage = 1.0`, `evidence_match_rate
  = 1.0`, **`unsupported_answer_count = 0`**, 2 correct abstentions.
- **20 backend tests** pass (unit + API via TestClient), green in CI.
- Runs fully offline with no API key; one-click demo seeds three sample documents.

## What I would build next

1. Swap the keyword ranker for sentence-embedding retrieval (FAISS / sqlite-vec)
   to handle paraphrased questions, and A/B it against the current ranker using
   the same golden dataset.
2. Exact answer-span highlighting inside the cited excerpt.
3. Cross-document Q&A and document collections.
4. Auth + per-tenant isolation; background processing + OCR for scanned PDFs.
5. Expand the golden set and add precision/recall on citations.

## Why this matters for AI workflow automation

This is the core loop behind most useful document AI: **ingest → retrieve →
answer → cite → evaluate**, with guardrails that make it safe to put in front of
a real team. The interesting work isn't calling a model — it's the retrieval
quality, the abstention policy, and the evaluation that proves the system stays
grounded. That's exactly the work of shipping trustworthy AI features in fintech
and operations.