docs/client-integration-guide.md

# Client CRM Integration Guide

How a SaaS CRM consumes the Delivery Note OCR API and turns a raw extraction into
resolved, database-linked, human-verifiable records.

This guide is the contract between **our stateless OCR backend** and **the
client-side CRM**. It exists because the two systems have deliberately different
responsibilities: the server never touches the client database, and the client
never re-implements OCR or accounting logic.

---

## 1. Separation of Concerns

The system is split along a clean architecture boundary. Everything that is
*document-intrinsic* (what the paper says, whether the numbers are internally
consistent) lives on the server. Everything that is *tenant-specific* (which of
your suppliers/products this refers to) lives on the client.

```mermaid
flowchart LR
    subgraph Server["Server Side — our API (stateless)"]
        OCR["OCR extraction"]
        SCHEMA["JSON Schema enforcement"]
        REPAIR["Deterministic accounting repair"]
        WARN["Consistency warnings"]
    end
    subgraph Client["Client Side — SaaS CRM (stateful, per-tenant)"]
        NORM["Text normalization"]
        CAND["Candidate generation (pg_trgm)"]
        RANK["Attribute reranking"]
        CONF["Confidence fusion"]
        UI["Human-in-the-loop review UI"]
        DB[("PostgreSQL<br/>(tenant-scoped)")]
    end

    Image[/"BL image"/] --> OCR --> SCHEMA --> REPAIR --> WARN
    WARN -->|"JSON response"| NORM
    NORM --> CAND --> RANK --> CONF --> UI
    CAND <--> DB
    UI -->|"confirmed links"| DB
```

### Server side (our API) — what you receive, not what you resolve

| Responsibility | Detail |
| --- | --- |
| **OCR extraction** | Single-pass structured extraction from the BL image. |
| **Schema enforcement** | Output validated against `/v1/schema`; `schema_valid` + `schema_errors` reported. |
| **Deterministic repair** | Price-column confusions corrected by accounting identities (e.g. `pharmacist_price_ttc = line_total_ttc / quantity`). Provenance in `repairs[]`. |
| **Consistency warnings** | Row-count, line arithmetic, and total reconciliation surfaced in `validation_warnings[]`. |

The server is **stateless** and **tenant-agnostic**. It has no notion of *which*
supplier or product a value refers to. It never connects to your database.

Consumed via `POST /v1/ocr` (multipart) or `POST /v1/ocr/base64` (JSON). See
`docs/api-spec.md` for the full response shape.

### Client side (SaaS CRM) — what you own

| Responsibility | Detail |
| --- | --- |
| **PostgreSQL** | Tenant catalogs: products, suppliers, pharmacies, plus public directories. |
| **Tenant scoping** | Every query is filtered by `tenant_id`; candidates never cross tenants. |
| **Candidate generation** | `pg_trgm` GIN-indexed fuzzy lookup + exact-identifier lookup. |
| **Attribute reranking** | Deterministic re-scoring of candidates by dosage, form, pack size. |
| **Confidence fusion** | Combine OCR confidence, DB match score, and top-2 margin. |
| **Review UI** | Highlight low-confidence fields; capture the human's final choice. |

> **Golden rule:** entity identity is a client decision. The server gives you
> *values and how sure it is of reading them*; you decide *what they map to* in
> your world.

---

## 2. Entity Resolution Cascade (Client Side)

Run a tiered cascade over the OCR data. Each entity to resolve — supplier,
client/pharmacy, and every product line — descends the tiers until it matches or
falls through to manual review.

```
Tier 1  Exact identifier   ──match──▶ auto-link (highest confidence)
   │ no match
Tier 2  Fuzzy DB match     ──match──▶ rank candidates, auto-link if clear
   │ no confident match           else ▶ suggest candidates in UI
Tier 3  Unresolved         ─────────▶ manual human selection in the CRM UI
```

### 2.0 One-time database setup

```sql
-- Extensions (run once per database, superuser).
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;

-- A normalized, generated column keeps the GIN index and the query in sync.
-- IMMUTABLE wrapper around unaccent() so it can be used in a generated column.
CREATE OR REPLACE FUNCTION normalize_label(txt text)
RETURNS text
LANGUAGE sql IMMUTABLE STRICT AS $$
  SELECT regexp_replace(
           lower(public.unaccent('public.unaccent', txt)),
           '[^a-z0-9]+', ' ', 'g')
$$;

ALTER TABLE products
  ADD COLUMN IF NOT EXISTS designation_norm text
  GENERATED ALWAYS AS (normalize_label(designation)) STORED;

-- GIN trigram index on the *normalized* column — the lookups below rely on it.
CREATE INDEX IF NOT EXISTS idx_products_designation_norm_trgm
  ON products USING gin (designation_norm gin_trgm_ops);

-- Exact-identifier lookups stay on a plain btree.
CREATE INDEX IF NOT EXISTS idx_products_code   ON products (tenant_id, product_code);
CREATE INDEX IF NOT EXISTS idx_suppliers_taxid ON suppliers (tenant_id, tax_id);
```

> **Why normalize before indexing?** The GIN index is built on
> `designation_norm`. If the query normalizes text a *different* way than the
> index was built, Postgres cannot use the index and silently falls back to a
> sequential scan. Normalize identically on both sides — always via
> `normalize_label()`.

### 2.1 Client-side text normalization (Python, before querying)

Normalize the OCR value with the **same** rules as `normalize_label()` before it
reaches SQL. This guarantees consistent GIN lookups and stable trigram scores.

```python
import re
import unicodedata


def normalize_label(text: str | None) -> str:
    """Mirror the SQL normalize_label(): unaccent, lowercase, collapse non-alnum."""
    if not text:
        return ""
    # Strip accents (é -> e) the same way SQL unaccent does.
    decomposed = unicodedata.normalize("NFKD", text)
    ascii_text = decomposed.encode("ascii", "ignore").decode("ascii")
    ascii_text = ascii_text.lower()
    return re.sub(r"[^a-z0-9]+", " ", ascii_text).strip()


# Structured attribute extraction from a pharmaceutical designation.
_DOSAGE_RE = re.compile(r"(\d+(?:[.,]\d+)?)\s?(mg|g|ml|mcg|ui|%)", re.IGNORECASE)
_PACK_RE = re.compile(r"(?:bte|b|boite|t|tube|fl|flacon)\s?/?\s?(\d+)", re.IGNORECASE)
_FORM_KEYWORDS = {
    "gel": "gel", "cp": "comprime", "comprime": "comprime", "gelule": "gelule",
    "sirop": "sirop", "sol": "solution", "susp": "suspension", "pom": "pommade",
    "cr": "creme", "creme": "creme", "inj": "injectable", "sach": "sachet",
}


def parse_attributes(designation: str | None) -> dict:
    """Pull dosage, galenic form, and pack size out of a free-text designation."""
    norm = normalize_label(designation)
    dosage = None
    if m := _DOSAGE_RE.search(designation or ""):
        dosage = f"{m.group(1).replace(',', '.')}{m.group(2).lower()}"
    pack_size = int(m.group(1)) if (m := _PACK_RE.search(designation or "")) else None
    form = next((v for k, v in _FORM_KEYWORDS.items() if k in norm.split()), None)
    return {"norm": norm, "dosage": dosage, "form": form, "pack_size": pack_size}
```

### 2.2 Tier 1 — Exact identifier match

Use identifiers that are globally stable: supplier `tax_id`, product `product_code`.

```sql
-- Product by exact code (tenant-scoped).
SELECT id, product_code, designation, dosage, galenic_form, pack_size
FROM products
WHERE tenant_id = %(tenant_id)s
  AND product_code = %(product_code)s
LIMIT 1;

-- Supplier by exact tax id (digits only, to survive OCR spacing/punctuation).
SELECT id, name, tax_id
FROM suppliers
WHERE tenant_id = %(tenant_id)s
  AND regexp_replace(tax_id, '\D', '', 'g') = regexp_replace(%(tax_id)s, '\D', '', 'g')
LIMIT 1;
```

A Tier-1 hit is an auto-link: identity is certain, and DB match score = `1.0`.

### 2.3 Tier 2 — Fuzzy database match (`pg_trgm`)

When there is no exact identifier (or the code is low-confidence), fall back to
trigram similarity on the normalized designation. Return the top-N candidates —
**do not** auto-pick here; hand them to reranking (§3).

```sql
-- :q_norm is normalize_label(ocr_designation) computed in Python (§2.1).
SELECT
    id,
    product_code,
    designation,
    dosage,
    galenic_form,
    pack_size,
    similarity(designation_norm, %(q_norm)s) AS name_sim
FROM products
WHERE tenant_id = %(tenant_id)s
  AND designation_norm %% %(q_norm)s          -- %% = trigram "similar" operator, uses the GIN index
ORDER BY name_sim DESC
LIMIT 10;
```

```sql
-- Set the similarity floor for the %% operator per session/transaction.
SET pg_trgm.similarity_threshold = 0.30;
```

Directory lookups (suppliers, pharmacies) follow the same pattern against their
own normalized, GIN-indexed name columns.

### 2.4 Tier 3 — Unresolved

If Tier 1 finds nothing and Tier 2 returns no candidate above the acceptance
threshold (§4), the entity is **unresolved**. Present the raw OCR value plus the
best fuzzy suggestions and require an explicit human selection. Persist that
choice as tenant training data so future matches improve.

---

## 3. Product Attribute Reranking (Client Side)

Trigram name similarity alone confuses same-name products of different strengths
or pack sizes (`DOLIPRANE 1000MG BTE/8` vs `DOLIPRANE 500MG BTE/16`). Rerank the
Tier-2 candidates by fusing name similarity with structured attributes.

### Scoring formula

For each candidate, all components are normalized to `[0, 1]`:

```
match_score = 0.45 · name_sim
            + 0.30 · dosage_match
            + 0.15 · pack_match
            + 0.10 · form_match
```

| Component | Weight | How to compute (each in [0, 1]) |
| --- | ---: | --- |
| `name_sim` | 0.45 | Trigram `similarity(designation_norm, q_norm)` from the SQL. |
| `dosage_match` | 0.30 | `1.0` if normalized dosages are equal (`1000mg == 1000mg`); `0.0` otherwise. Treat missing OCR dosage as `0.5` (unknown, not contradictory). |
| `pack_match` | 0.15 | `1.0` if pack sizes equal; else `max(0, 1 − |a−b| / max(a, b))`; `0.5` if unknown. |
| `form_match` | 0.10 | `1.0` if galenic forms equal after normalization; `0.0` if both known and different; `0.5` if unknown. |

The weights sum to `1.0`, so `match_score` is directly comparable across
candidates and interpretable as a DB match score in §4.

### Reference implementation

```python
def rerank_candidates(ocr_attrs: dict, candidates: list[dict]) -> list[dict]:
    """Return candidates sorted by fused match_score (desc)."""
    for c in candidates:
        cand = parse_attributes(c["designation"])
        c["dosage_match"] = _cmp_exact(ocr_attrs["dosage"], cand["dosage"] or c.get("dosage"))
        c["pack_match"] = _cmp_pack(ocr_attrs["pack_size"], cand["pack_size"] or c.get("pack_size"))
        c["form_match"] = _cmp_exact(ocr_attrs["form"], cand["form"] or c.get("galenic_form"))
        c["match_score"] = (
            0.45 * c["name_sim"]
            + 0.30 * c["dosage_match"]
            + 0.15 * c["pack_match"]
            + 0.10 * c["form_match"]
        )
    return sorted(candidates, key=lambda c: c["match_score"], reverse=True)


def _cmp_exact(a, b) -> float:
    if a is None or b is None:
        return 0.5                      # unknown — neither confirms nor contradicts
    return 1.0 if str(a).lower() == str(b).lower() else 0.0


def _cmp_pack(a, b) -> float:
    if a is None or b is None:
        return 0.5
    if a == b:
        return 1.0
    return max(0.0, 1.0 - abs(a - b) / max(a, b))
```

---

## 4. Confidence Fusion & UI Highlighting (Client Side)

### 4.1 Final field confidence

The final confidence a reviewer sees blends three independent signals:

1. **OCR confidence** — how clearly the value was *read* (from the API,
   `field.confidence ∈ [0, 1]`).
2. **DB match score** — how well it *resolved* to a catalog entry (`match_score`
   from §3, or `1.0` for a Tier-1 exact hit).
3. **Margin factor** — how *decisive* the win was over the runner-up. A 0.91 vs
   0.90 near-tie is risky even if both scores are high.

```
margin        = match_score(top1) − match_score(top2)      # 0 if only one candidate
margin_factor = 0.5 + 0.5 · min(margin / MARGIN_FULL, 1)    # MARGIN_FULL ≈ 0.15
                                                            # → 0.5 (tie) … 1.0 (clear win)

final_confidence = ocr_confidence · db_match_score · margin_factor
```

Multiplicative fusion is deliberately conservative: any weak signal (a blurry
read, a poor match, or a near-tie) drags the result down, which is the correct
bias for a human-in-the-loop workflow. For an unresolved (Tier 3) field, set
`db_match_score = 0` so `final_confidence = 0` and the field always demands
review.

```python
MARGIN_FULL = 0.15

def final_confidence(ocr_conf: float, ranked: list[dict]) -> float:
    if not ranked:
        return 0.0
    top1 = ranked[0]["match_score"]
    top2 = ranked[1]["match_score"] if len(ranked) > 1 else 0.0
    margin_factor = 0.5 + 0.5 * min((top1 - top2) / MARGIN_FULL, 1.0)
    return round(ocr_conf * top1 * margin_factor, 4)
```

### 4.2 UI highlighting for human verification

Map `final_confidence` to a review state. Anything **below 80%** is boxed for
verification; the color signals urgency.

| `final_confidence` | State | UI treatment |
| --- | --- | --- |
| `≥ 0.80` | Trusted | No box (or subtle green). Auto-accepted. |
| `0.60 – 0.80` | Review | **Orange** highlight box; pre-filled with the top candidate, editable. |
| `< 0.60` | Uncertain | **Red** highlight box; requires explicit confirmation before save. |

```python
def review_state(conf: float) -> str:
    if conf >= 0.80:
        return "trusted"      # green / no box
    if conf >= 0.60:
        return "review"       # orange box
    return "uncertain"        # red box
```

**Where to draw the box.** Highlight the corresponding input in the CRM form
(product row, supplier field, total). If you later enable provider-side
geometry (bounding boxes / source spans — see the report's provenance section),
overlay the box directly on the rendered BL image at the field's coordinates so
the reviewer's eye goes straight to the source text. Until then, field-level
highlighting on the form is sufficient and keeps the client fully decoupled from
provider geometry support.

### 4.3 Worked example

```
OCR:  "DOLIPRAN 1000 BTE/8", ocr_confidence = 0.82  ->  attrs: dosage 1000mg, pack 8, form unknown

Candidate A: DOLIPRANE 1000MG BTE/8
    name_sim 0.86  dosage match 1.0  pack match 1.0  form unknown 0.5
    match = 0.45*0.86 + 0.30*1.0 + 0.15*1.0 + 0.10*0.5 = 0.887

Candidate B: DOLIPRANE 500MG BTE/16
    name_sim 0.71  dosage mismatch 0.0  pack |8-16|/16 -> 0.5  form unknown 0.5
    match = 0.45*0.71 + 0.30*0.0 + 0.15*0.5 + 0.10*0.5 = 0.445

margin        = 0.887 - 0.445 = 0.442   ->  margin_factor = 1.0  (clear win)
final_conf    = 0.82 * 0.887 * 1.0      =  0.727            ->  "review" (orange)
```

Even though the match is unambiguous, the mediocre OCR read (`0.82`) pulls the
final confidence into the orange band — so the reviewer confirms the strength
before it is committed. That is the intended, safety-first behavior.

---

## 5. Integration Checklist

- [ ] Call `/v1/ocr` (or `/v1/ocr/base64`); keep `repair=true`, `validate=true`.
- [ ] Reject/queue the document if `schema_valid` is `false` or `parse_error` is set.
- [ ] Surface `validation_warnings` to the reviewer (they are non-blocking).
- [ ] Normalize every text value with `normalize_label()` before querying.
- [ ] Run the Tier 1 → 2 → 3 cascade per entity (supplier, pharmacy, each product).
- [ ] Rerank Tier-2 candidates with the weighted attribute formula (§3).
- [ ] Compute `final_confidence` and set the UI review state (§4).
- [ ] Persist confirmed human selections as tenant-scoped resolution history.