Deploy de83ed8cb4e1bbfa6f888d334f247492ce1b5bbf

README.md

@@ -7,102 +7,4 @@ sdk: docker
 pinned: false
 app_port: 7860
 license: mit
----
-
-# Document Data Extraction
-
-Extract arbitrary user-defined keys from invoices, receipts and scanned
-documents. Upload a PDF or image, list the keys you want, get one structured
-result per key.
-
-```bash
-curl -X POST https://<your-space>.hf.space/v1/extract \
-  -F "file=@invoice.pdf" \
-  -F "keys=INVOICE NO,Vendor Id,Invoice Date,Total Amount,GSTIN"
-```
-
-Keys are free-form. Nothing is hard-coded — `INVOICE NO` and `Total Amount` are
-just strings. Duplicates are returned separately, index-aligned with the
-request.
-
-## Read `status`, not `value`
-
-| status | meaning |
-|---|---|
-| `extracted` | confidence >= threshold. Use it. |
-| `low_confidence` | a span was found but fell below threshold. Route to a human. |
-| `not_found` | nothing usable on the page. |
-| `error` | the key could not be processed. |
-
-`impira/layoutlm-invoices` is an extractive model with **no null-answer head**:
-it returns *some* span for every input, including questions the document
-cannot answer. Measured on this project's benchmark it gave a confident wrong
-value for **16.7%** of deliberately unanswerable invoice questions, and
-abstained **0%** of the time. The confidence threshold exists specifically to
-catch this. Treating `value` without checking `status` will surface invented
-data.
-
-## Model
-
-`impira/layoutlm-invoices`, loaded from the Hugging Face cache at start. That
-model is **CC-BY-NC-SA-4.0, non-commercial only**. For commercial use set
-`DOCX_MODEL_ID=impira/layoutlm-document-qa` (MIT) — no code change needed.
-
-LayoutLM reads 2D position embeddings from bounding boxes, so it needs the
-document's geometry. Digital PDFs use the text layer; scans and images are
-rasterised and OCR'd with poppler-utils and tesseract-ocr, both in the image.
-
-## Configuration
-
-| variable | default | meaning |
-|---|---|---|
-| `DOCX_MODEL_ID` | `impira/layoutlm-invoices` | any LayoutLM QA checkpoint |
-| `DOCX_TORCH_THREADS` | `4` | intra-op threads |
-| `DOCX_BATCH_SIZE` | `8` | keys per forward pass |
-| `DOCX_MAX_PAGES` | `3` | pages processed per document |
-| `DOCX_MAX_UPLOAD_MB` | `25` | body size limit |
-| `DOCX_CONFIDENCE_THRESHOLD` | `0.5` | `extracted` vs `low_confidence` |
-| `DOCX_REQUEST_TIMEOUT_S` | `60` | per-request timeout |
-
-Check what is actually in effect with `GET /v1/config`.
-
-## Endpoints
-
-| method | path |
-|---|---|
-| `GET` | `/health` |
-| `GET` | `/v1/config` |
-| `POST` | `/v1/extract` (multipart `file`, `keys`) |
-| `POST` | `/v1/extract/json` (base64 + keys array) |
-
-`/docs` serves the OpenAPI UI.
-
-## Measured performance
-
-Single key, 148-word page, CPU:
-
-| runtime | ms/key |
-|---|---|
-| PyTorch fp32 | 578 |
-| ONNX fp32 | 407 |
-
-Batching gives no CPU speedup (measured exactly linear) — the GEMMs are already
-compute-bound. Thread count is the real lever.
-
-Accuracy on the 12-invoice corpus with pdfplumber word extraction:
-
-| metric | invoices | general docs |
-|---|---|---|
-| exact match | 0.7604 | 0.5349 |
-| F1 | 0.8091 | 0.5416 |
-| thresholded | 0.7188 | — |
-
-## Notes
-
-`requirements.txt` pins `numpy==1.26.4`; `onnx` pulls numpy 2.x, which breaks
-scipy and sklearn in the benchmark scripts. `torch` comes from the CPU wheel
-index — the default PyPI torch pulls multi-GB CUDA libraries.
-
-The previous llama-server (Qwen3-0.6B) service on this same port is commented
-out in the `Dockerfile`, not deleted, so the OpenAI-compatible chat endpoint can
-be restored.
+---

docqa/bench/accuracy.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Accuracy check for the docqa service against corpus ground truth.
+
+    python bench/accuracy.py --corpus <dir> [--limit N] [--base URL]
+
+Every number printed is measured from a real run against a running server.
+Nothing here is estimated.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import statistics
+import subprocess
+import time
+from pathlib import Path
+
+# Requested key -> ground-truth field in corpus/ground_truth.json
+KEY_MAP = [
+    ("INVOICE NO", "invoice_number"),
+    ("Invoice Date", "invoice_date"),
+    ("Due Date", "due_date"),
+    ("PO Number", "po_number"),
+    ("Payment Term", "payment_terms"),
+    ("Currency", "currency"),
+    ("Vendor Name", "vendor_name"),
+    ("Vendor Address", "vendor_address"),
+    ("Vendor Id", "vendor_tax_id"),
+    ("GSTIN", "gstin"),
+    ("Bill To", "bill_to"),
+    ("Subtotal", "subtotal"),
+    ("Tax Amount", "tax_amount"),
+    ("Total Amount", "total"),
+    ("Tax Label", "tax_label"),
+]
+
+
+def extract(base: str, pdf: Path, keys: list[str]) -> tuple[dict, float]:
+    started = time.perf_counter()
+    proc = subprocess.run(
+        ["curl.exe", "-sS", "-X", "POST", f"{base}/v1/extract",
+         "-F", f"file=@{pdf}", "-F", "keys=" + ",".join(keys)],
+        capture_output=True, text=True,
+    )
+    wall = (time.perf_counter() - started) * 1000
+    if proc.returncode != 0:
+        raise RuntimeError(proc.stderr.strip() or "curl failed")
+    return json.loads(proc.stdout), wall
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--base", default="http://127.0.0.1:7860")
+    ap.add_argument("--corpus", type=Path, required=True)
+    ap.add_argument("--limit", type=int, default=1)
+    args = ap.parse_args()
+
+    manifest = json.loads((args.corpus / "ground_truth.json").read_text())
+    records = manifest[: args.limit] if args.limit else manifest
+    keys = [k for k, _ in KEY_MAP]
+
+    print(f"target  {args.base}")
+    print(f"model   impira/layoutlm-invoices   keys={len(keys)}   "
+          f"documents={len(records)}\n")
+
+    per_key: dict[str, list[int]] = {k: [] for k in keys}
+    latencies: list[float] = []
+
+    for record in records:
+        body, wall = extract(args.base, Path(record["path"]), keys)
+        latencies.append(body.get("latency_ms", 0.0))
+        print(f"{record['document_id']}   variant={record['variant']}   "
+              f"source={body.get('source')}   words={body.get('word_count')}   "
+              f"pages={body.get('pages_processed')}")
+        print(f"server={body.get('latency_ms', 0):.0f} ms   "
+              f"wall={wall:.0f} ms (includes model load on first call)\n")
+
+        by_key = {f["key"]: f for f in body.get("fields", [])}
+        print(f"  {'KEY':<14} {'EXPECTED':<26} {'STATUS':<15} "
+              f"{'GOT':<28} CONF")
+        print("  " + "-" * 92)
+        for key, truth in KEY_MAP:
+            expected = str(record[truth])
+            field = by_key.get(key, {})
+            got = (field.get("value") or "").strip()
+            ok = got == expected.strip()
+            per_key[key].append(1 if ok else 0)
+            print(f"  {key:<14} {expected[:26]:<26} "
+                  f"{str(field.get('status')):<15} "
+                  f"{(got[:26] + ('  OK' if ok else '  BAD')):<28} "
+                  f"{field.get('confidence', 0):.4f}")
+        print("  " + "-" * 92 + "\n")
+
+    total_hit = sum(sum(v) for v in per_key.values())
+    total_n = sum(len(v) for v in per_key.values())
+
+    print("PER-KEY EXACT MATCH")
+    print("-" * 46)
+    for key, _ in KEY_MAP:
+        scores = per_key[key]
+        if scores:
+            acc = 100 * sum(scores) / len(scores)
+            print(f"  {key:<16} {acc:6.1f}%  ({sum(scores)}/{len(scores)})")
+    if total_n:
+        print(f"\n  {'OVERALL':<16} {100*total_hit/total_n:6.1f}%  "
+              f"({total_hit}/{total_n})")
+
+    if latencies:
+        print("\nLATENCY (whole document, all keys)")
+        print("-" * 46)
+        print(f"  mean  {statistics.mean(latencies):8.0f} ms")
+        print(f"  min   {min(latencies):8.0f} ms")
+        print(f"  max   {max(latencies):8.0f} ms")
+        print(f"  per key {statistics.mean(latencies)/len(keys):6.0f} ms")
+
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docqa/bench/check_deps.py

@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+"""Verify requirements.txt covers every third-party import in docqa/.
+
+Run in CI or after adding a module:
+
+    python bench/check_deps.py
+
+Exits non-zero if an import is unlisted, so a missing dependency is caught at
+build time rather than as an ImportError on the first request that happens to
+hit that code path.
+"""
+
+from __future__ import annotations
+
+import ast
+import re
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+REPO = ROOT.parent
+
+# import name -> pip distribution name, where they differ
+DISTRIBUTION = {
+    "PIL": "pillow",
+    "fitz": "pymupdf",
+    "sklearn": "scikit-learn",
+    "yaml": "pyyaml",
+    "cv2": "opencv-python",
+    "bs4": "beautifulsoup4",
+    "dotenv": "python-dotenv",
+}
+
+# Imported only inside functions guarded by try/except ImportError, or only by
+# optional benchmark paths. Still listed in requirements.txt, but a missing
+# one is a warning rather than a hard failure.
+OPTIONAL = {"onnx", "onnxruntime", "sklearn"}
+
+# Runtime plugin dependencies that no AST scan of our own code can discover,
+# because the import happens inside the framework. FastAPI resolves these
+# lazily: the app imports fine and the endpoint 500s on first use. Keep this in
+# step with any File()/Form()/UploadFile declaration in api.py.
+FRAMEWORK_PLUGINS = {
+    "python-multipart": "FastAPI File()/Form(); app starts but /v1/extract 500s",
+}
+
+
+def parse_requirements() -> dict[str, str]:
+    """Distribution name -> pinned version, from requirements.txt."""
+    out: dict[str, str] = {}
+    for line in (REPO / "requirements.txt").read_text(encoding="utf-8").splitlines():
+        line = line.split("#", 1)[0].strip()
+        if not line or line.startswith("-"):
+            continue
+        match = re.match(r"^([A-Za-z0-9._-]+)(?:\[[^\]]*\])?==(.+)$", line)
+        if match:
+            out[match.group(1).lower().replace("_", "-")] = match.group(2)
+    return out
+
+
+def collect_imports() -> set[str]:
+    found: set[str] = set()
+    for path in ROOT.rglob("*.py"):
+        tree = ast.parse(path.read_text(encoding="utf-8"))
+        for node in ast.walk(tree):
+            if isinstance(node, ast.Import):
+                for alias in node.names:
+                    found.add(alias.name.split(".")[0])
+            elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
+                found.add(node.module.split(".")[0])
+    return found
+
+
+def verify_installed(declared: dict[str, str]) -> list[str]:
+    """Actually import each declared distribution.
+
+    An AST scan only sees what docqa/ imports directly. It cannot see what a
+    framework pulls in at request time, which is precisely how python-multipart
+    was missed: fastapi imports cleanly without it and only fails when
+    File()/Form() is exercised. Importing the real modules closes that gap.
+    """
+    import importlib
+
+    problems: list[str] = []
+    for dist in sorted(declared):
+        module = MODULE_FOR_DIST.get(dist)
+        if module is None:
+            continue
+        try:
+            importlib.import_module(module)
+        except ImportError as exc:
+            problems.append(f"{dist} (import {module}): {exc}")
+    return problems
+
+
+# distribution name -> importable module, where they differ
+MODULE_FOR_DIST = {
+    "pillow": "PIL",
+    "pymupdf": "fitz",
+    "scikit-learn": "sklearn",
+    "python-multipart": "multipart",
+    "python-dotenv": "dotenv",
+    "pyyaml": "yaml",
+}
+
+
+def main() -> int:
+    declared = parse_requirements()
+    imported = collect_imports()
+
+    stdlib = set(sys.stdlib_module_names)
+    local = {"docxextract"}
+    missing: list[str] = []
+    optional_missing: list[str] = []
+
+    for name in sorted(imported):
+        if name in stdlib or name in local:
+            continue
+        dist = DISTRIBUTION.get(name, name).lower().replace("_", "-")
+        if dist in declared:
+            continue
+        (optional_missing if name in OPTIONAL else missing).append(f"{name} -> {dist}")
+
+    # Plugins the framework needs at request time, invisible to the scan.
+    for dist, why in FRAMEWORK_PLUGINS.items():
+        if dist not in declared:
+            missing.append(f"{dist}  <- {why}")
+
+    print(f"requirements.txt declares {len(declared)} distributions")
+    print(f"docqa/ imports {len(imported)} modules "
+          f"({len([m for m in imported if m not in stdlib])} non-stdlib)\n")
+
+    if optional_missing:
+        print("OPTIONAL, not declared (guarded imports / optional paths):")
+        for item in optional_missing:
+            print(f"  {item}")
+        print()
+
+    broken = verify_installed(declared)
+    if broken:
+        print("DECLARED BUT NOT IMPORTABLE in this environment:")
+        for item in broken:
+            print(f"  {item}")
+        print()
+
+    if missing or broken:
+        if missing:
+            print("MISSING, must be added to requirements.txt:")
+            for item in missing:
+                print(f"  {item}")
+        return 1
+
+    print("OK: every non-stdlib import is declared and importable.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

requirements.txt

@@ -1,30 +1,49 @@
-# Pinned exactly. Loose pins here have broken this build more than once:
-#   - onnx pulls numpy 2.x, which breaks scipy and sklearn used by the
-#     benchmark scripts.
-#   - torch must come from the CPU wheel index; the default PyPI torch pulls
-#     multi-GB CUDA libraries and blows the Space image size limit.
+# Every third-party module imported anywhere under docqa/ appears here.
+# Verified by AST-scanning all imports, not by guessing -- see
+# bench/check_deps.py, which fails if an import is unlisted.
+#
+# Pinned exactly. Loose pins broke this build more than once:
+#   - torch must come from the CPU wheel index. The default PyPI torch pulls
+#     multi-GB CUDA libraries and exceeds the Space image limit.
+#   - numpy must stay on 1.x. onnxruntime/onnx pull numpy 2.x, which breaks
+#     scipy and scikit-learn used by the corpus generator.
 --extra-index-url https://download.pytorch.org/whl/cpu
 
+# --- web service ---
 fastapi==0.115.5
 uvicorn[standard]==0.34.0
 pydantic==2.13.4
 pydantic-settings==2.15.0
 orjson==3.11.9
+httpx==0.28.1
+# REQUIRED by FastAPI at runtime for any endpoint declaring File() or Form().
+# It is not imported by our own code, so an AST scan of docqa/ cannot find it:
+# FastAPI raises at request time ("Form data requires python-multipart"), not
+# at import time, which means the app starts cleanly and then 500s on
+# /v1/extract. Added explicitly after that exact failure.
+python-multipart==0.0.20
 
-# model
+# --- model ---
 torch==2.5.1+cpu
 transformers==4.50.2
-huggingface_hub>=0.26
+huggingface-hub==0.28.1
+onnxruntime==1.20.1
+numpy==1.26.4
+scipy==1.10.1
 
-# parsing / OCR
+# --- document parsing / OCR ---
 pdfplumber==0.11.9
 pytesseract==0.3.13
-Pillow==10.3.0
+pillow==10.3.0
+# fitz: rasterises scanned pages for the OCR fallback
+pymupdf==1.25.1
 
-# numpy must stay on 1.x; see note above
-numpy==1.26.4
+# --- test corpus generation ---
+# reportlab draws the synthetic invoices in tests/corpus.py; scikit-learn is
+# used by bench/compare_backends.py for scoring.
+reportlab==3.6.13
+scikit-learn==1.6.1
 
-# tests
+# --- tests ---
 pytest==8.3.4
-httpx==0.28.1
-pytest-asyncio==0.24.0
+pytest-asyncio==1.3.0