docs/client-integration-guide.md

Client CRM Integration Guide

How a SaaS CRM consumes the Delivery Note OCR API and turns a raw extraction into resolved, database-linked, human-verifiable records.

This guide is the contract between our stateless OCR backend and the client-side CRM. It exists because the two systems have deliberately different responsibilities: the server never touches the client database, and the client never re-implements OCR or accounting logic.

---

1. Separation of Concerns

The system is split along a clean architecture boundary. Everything that is document-intrinsic (what the paper says, whether the numbers are internally consistent) lives on the server. Everything that is tenant-specific (which of your suppliers/products this refers to) lives on the client.

flowchart LR
    subgraph Server["Server Side — our API (stateless)"]
        OCR["OCR extraction"]
        SCHEMA["JSON Schema enforcement"]
        REPAIR["Deterministic accounting repair"]
        WARN["Consistency warnings"]
    end
    subgraph Client["Client Side — SaaS CRM (stateful, per-tenant)"]
        NORM["Text normalization"]
        CAND["Candidate generation (pg_trgm)"]
        RANK["Attribute reranking"]
        CONF["Confidence fusion"]
        UI["Human-in-the-loop review UI"]
        DB[("PostgreSQL<br/>(tenant-scoped)")]
    end

    Image[/"BL image"/] --> OCR --> SCHEMA --> REPAIR --> WARN
    WARN -->|"JSON response"| NORM
    NORM --> CAND --> RANK --> CONF --> UI
    CAND <--> DB
    UI -->|"confirmed links"| DB

Server side (our API) — what you receive, not what you resolve

Responsibility	Detail
OCR extraction	Single-pass structured extraction from the BL image.
Schema enforcement	Output validated against /v1/schema; schema_valid + schema_errors reported.
Deterministic repair	Price-column confusions corrected by accounting identities (e.g. pharmacist_price_ttc = line_total_ttc / quantity). Provenance in repairs[].
Consistency warnings	Row-count, line arithmetic, and total reconciliation surfaced in validation_warnings[].

The server is stateless and tenant-agnostic. It has no notion of which supplier or product a value refers to. It never connects to your database.

Consumed via POST /v1/ocr (multipart) or POST /v1/ocr/base64 (JSON). See docs/api-spec.md for the full response shape.

Client side (SaaS CRM) — what you own

Responsibility	Detail
PostgreSQL	Tenant catalogs: products, suppliers, pharmacies, plus public directories.
Tenant scoping	Every query is filtered by tenant_id; candidates never cross tenants.
Candidate generation	pg_trgm GIN-indexed fuzzy lookup + exact-identifier lookup.
Attribute reranking	Deterministic re-scoring of candidates by dosage, form, pack size.
Confidence fusion	Combine OCR confidence, DB match score, and top-2 margin.
Review UI	Highlight low-confidence fields; capture the human's final choice.

Golden rule: entity identity is a client decision. The server gives you values and how sure it is of reading them; you decide what they map to in your world.

---

2. Entity Resolution Cascade (Client Side)

Run a tiered cascade over the OCR data. Each entity to resolve — supplier, client/pharmacy, and every product line — descends the tiers until it matches or falls through to manual review.

Tier 1  Exact identifier   ──match──▶ auto-link (highest confidence)
   │ no match
Tier 2  Fuzzy DB match     ──match──▶ rank candidates, auto-link if clear
   │ no confident match           else ▶ suggest candidates in UI
Tier 3  Unresolved         ─────────▶ manual human selection in the CRM UI

2.0 One-time database setup

-- Extensions (run once per database, superuser).
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS unaccent;

-- A normalized, generated column keeps the GIN index and the query in sync.
-- IMMUTABLE wrapper around unaccent() so it can be used in a generated column.
CREATE OR REPLACE FUNCTION normalize_label(txt text)
RETURNS text
LANGUAGE sql IMMUTABLE STRICT AS $$
  SELECT regexp_replace(
           lower(public.unaccent('public.unaccent', txt)),
           '[^a-z0-9]+', ' ', 'g')
$$;

ALTER TABLE products
  ADD COLUMN IF NOT EXISTS designation_norm text
  GENERATED ALWAYS AS (normalize_label(designation)) STORED;

-- GIN trigram index on the *normalized* column — the lookups below rely on it.
CREATE INDEX IF NOT EXISTS idx_products_designation_norm_trgm
  ON products USING gin (designation_norm gin_trgm_ops);

-- Exact-identifier lookups stay on a plain btree.
CREATE INDEX IF NOT EXISTS idx_products_code   ON products (tenant_id, product_code);
CREATE INDEX IF NOT EXISTS idx_suppliers_taxid ON suppliers (tenant_id, tax_id);

Why normalize before indexing? The GIN index is built on designation_norm. If the query normalizes text a different way than the index was built, Postgres cannot use the index and silently falls back to a sequential scan. Normalize identically on both sides — always via normalize_label().

2.1 Client-side text normalization (Python, before querying)

Normalize the OCR value with the same rules as normalize_label() before it reaches SQL. This guarantees consistent GIN lookups and stable trigram scores.

import re
import unicodedata


def normalize_label(text: str | None) -> str:
    """Mirror the SQL normalize_label(): unaccent, lowercase, collapse non-alnum."""
    if not text:
        return ""
    # Strip accents (é -> e) the same way SQL unaccent does.
    decomposed = unicodedata.normalize("NFKD", text)
    ascii_text = decomposed.encode("ascii", "ignore").decode("ascii")
    ascii_text = ascii_text.lower()
    return re.sub(r"[^a-z0-9]+", " ", ascii_text).strip()


# Structured attribute extraction from a pharmaceutical designation.
_DOSAGE_RE = re.compile(r"(\d+(?:[.,]\d+)?)\s?(mg|g|ml|mcg|ui|%)", re.IGNORECASE)
_PACK_RE = re.compile(r"(?:bte|b|boite|t|tube|fl|flacon)\s?/?\s?(\d+)", re.IGNORECASE)
_FORM_KEYWORDS = {
    "gel": "gel", "cp": "comprime", "comprime": "comprime", "gelule": "gelule",
    "sirop": "sirop", "sol": "solution", "susp": "suspension", "pom": "pommade",
    "cr": "creme", "creme": "creme", "inj": "injectable", "sach": "sachet",
}


def parse_attributes(designation: str | None) -> dict:
    """Pull dosage, galenic form, and pack size out of a free-text designation."""
    norm = normalize_label(designation)
    dosage = None
    if m := _DOSAGE_RE.search(designation or ""):
        dosage = f"{m.group(1).replace(',', '.')}{m.group(2).lower()}"
    pack_size = int(m.group(1)) if (m := _PACK_RE.search(designation or "")) else None
    form = next((v for k, v in _FORM_KEYWORDS.items() if k in norm.split()), None)
    return {"norm": norm, "dosage": dosage, "form": form, "pack_size": pack_size}

2.2 Tier 1 — Exact identifier match

Use identifiers that are globally stable: supplier tax_id, product product_code.

-- Product by exact code (tenant-scoped).
SELECT id, product_code, designation, dosage, galenic_form, pack_size
FROM products
WHERE tenant_id = %(tenant_id)s
  AND product_code = %(product_code)s
LIMIT 1;

-- Supplier by exact tax id (digits only, to survive OCR spacing/punctuation).
SELECT id, name, tax_id
FROM suppliers
WHERE tenant_id = %(tenant_id)s
  AND regexp_replace(tax_id, '\D', '', 'g') = regexp_replace(%(tax_id)s, '\D', '', 'g')
LIMIT 1;

A Tier-1 hit is an auto-link: identity is certain, and DB match score = 1.0.

2.3 Tier 2 — Fuzzy database match (pg_trgm)

When there is no exact identifier (or the code is low-confidence), fall back to trigram similarity on the normalized designation. Return the top-N candidates — do not auto-pick here; hand them to reranking (§3).

-- :q_norm is normalize_label(ocr_designation) computed in Python (§2.1).
SELECT
    id,
    product_code,
    designation,
    dosage,
    galenic_form,
    pack_size,
    similarity(designation_norm, %(q_norm)s) AS name_sim
FROM products
WHERE tenant_id = %(tenant_id)s
  AND designation_norm %% %(q_norm)s          -- %% = trigram "similar" operator, uses the GIN index
ORDER BY name_sim DESC
LIMIT 10;

-- Set the similarity floor for the %% operator per session/transaction.
SET pg_trgm.similarity_threshold = 0.30;

Directory lookups (suppliers, pharmacies) follow the same pattern against their own normalized, GIN-indexed name columns.

2.4 Tier 3 — Unresolved

If Tier 1 finds nothing and Tier 2 returns no candidate above the acceptance threshold (§4), the entity is unresolved. Present the raw OCR value plus the best fuzzy suggestions and require an explicit human selection. Persist that choice as tenant training data so future matches improve.

---

3. Product Attribute Reranking (Client Side)

Trigram name similarity alone confuses same-name products of different strengths or pack sizes (DOLIPRANE 1000MG BTE/8 vs DOLIPRANE 500MG BTE/16). Rerank the Tier-2 candidates by fusing name similarity with structured attributes.

Scoring formula

For each candidate, all components are normalized to [0, 1]:

match_score = 0.45 · name_sim
            + 0.30 · dosage_match
            + 0.15 · pack_match
            + 0.10 · form_match

Component	Weight	How to compute (each in [0, 1])
name_sim	0.45	Trigram similarity(designation_norm, q_norm) from the SQL.
dosage_match	0.30	1.0 if normalized dosages are equal (1000mg == 1000mg); 0.0 otherwise. Treat missing OCR dosage as 0.5 (unknown, not contradictory).
pack_match	0.15	1.0 if pack sizes equal; else `max(0, 1 −
form_match	0.10	1.0 if galenic forms equal after normalization; 0.0 if both known and different; 0.5 if unknown.

The weights sum to 1.0, so match_score is directly comparable across candidates and interpretable as a DB match score in §4.

Reference implementation

def rerank_candidates(ocr_attrs: dict, candidates: list[dict]) -> list[dict]:
    """Return candidates sorted by fused match_score (desc)."""
    for c in candidates:
        cand = parse_attributes(c["designation"])
        c["dosage_match"] = _cmp_exact(ocr_attrs["dosage"], cand["dosage"] or c.get("dosage"))
        c["pack_match"] = _cmp_pack(ocr_attrs["pack_size"], cand["pack_size"] or c.get("pack_size"))
        c["form_match"] = _cmp_exact(ocr_attrs["form"], cand["form"] or c.get("galenic_form"))
        c["match_score"] = (
            0.45 * c["name_sim"]
            + 0.30 * c["dosage_match"]
            + 0.15 * c["pack_match"]
            + 0.10 * c["form_match"]
        )
    return sorted(candidates, key=lambda c: c["match_score"], reverse=True)


def _cmp_exact(a, b) -> float:
    if a is None or b is None:
        return 0.5                      # unknown — neither confirms nor contradicts
    return 1.0 if str(a).lower() == str(b).lower() else 0.0


def _cmp_pack(a, b) -> float:
    if a is None or b is None:
        return 0.5
    if a == b:
        return 1.0
    return max(0.0, 1.0 - abs(a - b) / max(a, b))

---

4. Confidence Fusion & UI Highlighting (Client Side)

4.1 Final field confidence

The final confidence a reviewer sees blends three independent signals:

1. OCR confidence — how clearly the value was read (from the API, field.confidence ∈ [0, 1]).
2. DB match score — how well it resolved to a catalog entry (match_score from §3, or 1.0 for a Tier-1 exact hit).
3. Margin factor — how decisive the win was over the runner-up. A 0.91 vs 0.90 near-tie is risky even if both scores are high.

margin        = match_score(top1) − match_score(top2)      # 0 if only one candidate
margin_factor = 0.5 + 0.5 · min(margin / MARGIN_FULL, 1)    # MARGIN_FULL ≈ 0.15
                                                            # → 0.5 (tie) … 1.0 (clear win)

final_confidence = ocr_confidence · db_match_score · margin_factor

Multiplicative fusion is deliberately conservative: any weak signal (a blurry read, a poor match, or a near-tie) drags the result down, which is the correct bias for a human-in-the-loop workflow. For an unresolved (Tier 3) field, set db_match_score = 0 so final_confidence = 0 and the field always demands review.

MARGIN_FULL = 0.15

def final_confidence(ocr_conf: float, ranked: list[dict]) -> float:
    if not ranked:
        return 0.0
    top1 = ranked[0]["match_score"]
    top2 = ranked[1]["match_score"] if len(ranked) > 1 else 0.0
    margin_factor = 0.5 + 0.5 * min((top1 - top2) / MARGIN_FULL, 1.0)
    return round(ocr_conf * top1 * margin_factor, 4)

4.2 UI highlighting for human verification

Map final_confidence to a review state. Anything below 80% is boxed for verification; the color signals urgency.

final_confidence	State	UI treatment
≥ 0.80	Trusted	No box (or subtle green). Auto-accepted.
0.60 – 0.80	Review	Orange highlight box; pre-filled with the top candidate, editable.
< 0.60	Uncertain	Red highlight box; requires explicit confirmation before save.

def review_state(conf: float) -> str:
    if conf >= 0.80:
        return "trusted"      # green / no box
    if conf >= 0.60:
        return "review"       # orange box
    return "uncertain"        # red box

Where to draw the box. Highlight the corresponding input in the CRM form (product row, supplier field, total). If you later enable provider-side geometry (bounding boxes / source spans — see the report's provenance section), overlay the box directly on the rendered BL image at the field's coordinates so the reviewer's eye goes straight to the source text. Until then, field-level highlighting on the form is sufficient and keeps the client fully decoupled from provider geometry support.

4.3 Worked example

OCR:  "DOLIPRAN 1000 BTE/8", ocr_confidence = 0.82  ->  attrs: dosage 1000mg, pack 8, form unknown

Candidate A: DOLIPRANE 1000MG BTE/8
    name_sim 0.86  dosage match 1.0  pack match 1.0  form unknown 0.5
    match = 0.45*0.86 + 0.30*1.0 + 0.15*1.0 + 0.10*0.5 = 0.887

Candidate B: DOLIPRANE 500MG BTE/16
    name_sim 0.71  dosage mismatch 0.0  pack |8-16|/16 -> 0.5  form unknown 0.5
    match = 0.45*0.71 + 0.30*0.0 + 0.15*0.5 + 0.10*0.5 = 0.445

margin        = 0.887 - 0.445 = 0.442   ->  margin_factor = 1.0  (clear win)
final_conf    = 0.82 * 0.887 * 1.0      =  0.727            ->  "review" (orange)

Even though the match is unambiguous, the mediocre OCR read (0.82) pulls the final confidence into the orange band — so the reviewer confirms the strength before it is committed. That is the intended, safety-first behavior.

---

5. Integration Checklist

• Call /v1/ocr (or /v1/ocr/base64); keep repair=true, validate=true.
• Reject/queue the document if schema_valid is false or parse_error is set.
• Surface validation_warnings to the reviewer (they are non-blocking).
• Normalize every text value with normalize_label() before querying.
• Run the Tier 1 → 2 → 3 cascade per entity (supplier, pharmacy, each product).
• Rerank Tier-2 candidates with the weighted attribute formula (§3).
• Compute final_confidence and set the UI review state (§4).
• Persist confirmed human selections as tenant-scoped resolution history.