| # keys-automatic-receipt-doc-scanner |
|
|
| Local receipt/document scanner for **Autonomous OS + Autonomous Lamp**. |
| Lamp camera captures; a GPU box runs the model. Nothing cloud. |
|
|
| ## Hardware split (do not blur this) |
|
|
| | Where | Fits | Does not fit | |
| |---|---|---| |
| | **Lamp** (ARM64, **6 GB RAM**) | This skill (`SKILL.md` + `scripts/`), HAL snapshot, SQLite | Gemma 4 12B Unified, Qwen3.8-27B, Nemotron 3.5 Lightning | |
| | **GPU box** (DGX Spark / Omen) | Gemma 4 12B Unified (omni: vision + embed) or Qwen3.8-27B VLM + Embed-1B | — | |
|
|
| Gemma 4 12B Unified is ~12B dense (`hidden_size` **3840**, `Gemma4UnifiedForConditionalGeneration`). Weights alone exceed the Lamp's 6 GB even in NVFP4. Treat the Lamp as eyes/hands; treat the GPU box as the brain. |
|
|
| ## Default brain: Gemma 4 12B Unified (omni) |
|
|
| Encoder-free VLM. One OpenAI-compatible server does: |
|
|
| 1. **Vision extract** — `POST /v1/chat/completions` with `image_url` data URI (receipt/doc JPEG) |
| 2. **Embed** — `POST /v1/embeddings` against the **same** server (mean-pool / convert-embed). Dim **3840**. |
|
|
| Never mix 3840 (Gemma) and 2048 (Nemotron-3-Embed-1B) in one sqlite-vec index. |
|
|
| ## Fallback brains (same skill, env only) |
|
|
| - **Qwen3.8-27B ADay777** VLM at `:8078` (`qwen38-nvfp4`) for extract; Nemotron-3-Embed-1B 2048-d for embed |
| - **Lightning** is text-only. Never send images to it. |
|
|
| Fleet GB10: `--gpu-memory-utilization` **0.85** hard cap. |
|
|
| ## Pipeline |
|
|
| ``` |
| voice / phone / drop → Lamp camera or inbox/ |
| → optional OCR assist |
| → Gemma4 (or Qwen) vision JSON extract + category |
| → omni embed (or Nemotron-3-Embed-1B) |
| → sqlite-vec vendor/SKU/category match |
| → review / speak summary |
| ``` |
|
|
| ## Rules |
|
|
| - No cloud APIs. Backends behind `OCRBackend` / `LLMBackend` / `EmbedBackend`. |
| - Lightning: `accepts_images=False`. Never attach image parts. |
| - Python 3.12, typed, pytest. No notebooks. Don't vendor weights. |
| - Idle-batch inbox 30s. Dedup sha256. |
|
|
| ## Extract JSON |
|
|
| ```json |
| { |
| "doc_kind": "receipt|invoice|document", |
| "category": "groceries|dining|transport|household|health|entertainment|utilities|office|travel|other", |
| "vendor": "string|null", |
| "date": "YYYY-MM-DD|null", |
| "tax": "number|null", |
| "total": "number|null", |
| "currency": "string|null", |
| "line_items": [ |
| {"description": "string", "qty": "number|null", "unit_price": "number|null", |
| "amount": "number|null", "sku": "string|null"} |
| ] |
| } |
| ``` |
|
|
| Money stored as integer cents. |
|
|
| ## Match (cosine similarity = 1 - sqlite-vec distance) |
|
|
| | | Auto | Review | Unmatched | |
| |---|---|---|---| |
| | SKU / line | ≥ 0.88 | 0.72–0.88 | < 0.72 | |
| | Vendor | ≥ 0.82 | 0.65–0.82 | < 0.65 | |
|
|
| Exact catalog SKU wins first. |
|
|
| ## Layout |
|
|
| ``` |
| app/ config, schemas, media, camera, extract, embed, db, match, |
| pipeline, watcher, cli, ui |
| backends/ base, openai_compat, gemma, nvidia, ollama, apple, cpu |
| skills/keys-receipt-scanner/ Autonomous OS built-in skill (Lamp) |
| inbox/ processing/ processed/ failed/ exports/ |
| ``` |
|
|
| ## Autonomous OS skill |
|
|
| `skills/keys-receipt-scanner/` is a **built-in skill** in Autonomous OS format: |
|
|
| - `SKILL.md` + `skill.json` (`capabilities: ["vision"]`) |
| - Installs on any body that declares vision (Lamp, Reachy Mini — not Intern) |
| - Acts via HAL `GET :5001/camera/snapshot` then `python -m app.cli scan --image PATH` |
| - Does not load 12B weights on the robot |
|
|
