Refactor project structure and enhance batch processing capabilities

- Updated `.gitignore` to exclude sensitive client material and system files.
- Removed unnecessary comments and documentation from `main.py` and `cbh/__init__.py`.
- Revised `README.md` to clarify batch processing routes and export formats.
- Added new manual shipment and row addition endpoints in `cbh/api/batch/views.py`.
- Enhanced batch model and schema definitions to support additional fields and metrics.
- Updated batch processing logic to handle new parameters for destination country and customs procedures.
- Introduced new utility functions for file handling and improved error management in batch processing.
- Removed deprecated account module files to streamline the codebase.

.gitignore

@@ -8,3 +8,9 @@ __pycache__/
 
 # Uploaded packs, produced result files and reference-data files supplied by hand.
 storage/
+
+# Client material: real declarations, invoices and email threads. The parcel
+# manifests carry consignee names, home addresses, telephone numbers and email
+# addresses, so none of it belongs in the repository.
+Attachments-*/
+.DS_Store

README.md

@@ -24,7 +24,8 @@ Storage is MongoDB, not Postgres. No queues, no vector database.
 | `cbh/api/linecheck` | The engine. Code resolution, document determination, warnings. The single entry point. |
 | `cbh/api/importer` | Loads the reference files. Idempotent, restartable, logged in `synclogs`. |
 | `cbh/api/client` | Per-client column mapping and export profile. |
-| `cbh/api/batch` | The Excel processor: a thin client over the line-check API. |
+| `cbh/api/batch` | Shipments: both input routes, the review table and the exports. |
+| `cbh/api/rules` | The broker's own logic, written down once and replayed. |
 | `cbh/api/common` | Shared DTOs, schemas, generic Mongo access and the TTL cache. |
 
 Each module follows the same layout: `__init__.py` (router) · `dto.py` (enums) ·
@@ -86,24 +87,94 @@ Two rules the code holds to:
   reported as a warning, because "unregulated" and "the data does not cover
   this" look identical from the inside.
 
-## Processing a pack
+## Two ways in, one review table, one export
+
+Both input routes build the same shipment, and the review table is read the
+same way whichever route filled it.
+
+**A — the form, one line at a time.**
+
+```
+POST   /api/batch/manual                     # open an empty shipment
+POST   /api/batch/{id}/rows                  # add one line; it is checked on arrival
+DELETE /api/batch/{id}/rows/{rowIndex}
+```
+
+**B — upload, the daily route.**
 
 ```
 POST /api/client/suggest-mapping   # guess the column mapping from a file, for confirmation
 POST /api/batch/upload             # upload + process (clientProfileId, file)
-GET  /api/batch/{id}/download      # the same file back, with the added columns
-GET  /api/batch/{id}/rows?needsReviewOnly=true
 ```
 
-The client's own file comes back: same sheet, same columns, same row order,
-with `resolved_code`, `documents_required`, `documents_exemption`, `warnings`
-and `status` appended in the order the export profile fixes, a totals row at the
-bottom (packages, gross weight, value) and `ambiguous`/`invalid` rows
-highlighted.
+**The table, either way.**
+
+```
+GET   /api/batch/{id}/review?needsReviewOnly=true&confidence=…
+PATCH /api/batch/{id}/rows/{rowIndex}        # accept, correct, CBAM/EUDR/licence, note
+POST  /api/batch/{id}/rows/{rowIndex}/make-rule
+POST  /api/batch/{id}/recheck                # replay every row against today's rules and data
+```
+
+`GET /review` returns the header the screen is built from — reference, route,
+countries, the totals (packages, weight, value) and the two scores, extraction
+and document-codes, reported separately so neither can hide the other — plus
+one row per line carrying its confidence, its reason, its document codes with
+source and validity, its warnings and the rules that fired.
 
 Identical (code, country, direction, date) tuples are resolved once and reused
 across the pack — a 580-line pack typically collapses to a few dozen lookups.
 
+## Export, per tenant
+
+```
+GET /api/batch/export-formats?clientProfileId=…   # fills the dropdown
+GET /api/batch/{id}/export?format=…
+```
+
+| Format | What it is |
+| --- | --- |
+| `review` | The tool's own table: every column the reviewer saw, reasons included. |
+| `source` | The uploaded file returned unchanged, with the checked columns appended. Upload route only. |
+| `softpak` | The Goenee line list, rebuilt from the stored rows. |
+| `scope`, `fiton` | Declared, listed, and refused with 501. |
+
+`scope` and `fiton` are not implemented because their import specifications
+have not been received. They are listed as unavailable with the reason rather
+than quietly approximated, so the gap shows up in the UI instead of in a
+rejected declaration. Adding one is a profile change, not a code change.
+
+Totals are counted per physical package, not per row. A parcel manifest
+carries one row per item and repeats the parcel's weight on each of them;
+`TotalsConfig.parcelKeyColumn` names the column that identifies the parcel and
+`parcelLevelFields` the fields to count once per parcel. For the G-Star
+manifest of 04-09-2026 this is what reconciles the header with the Master
+Invoice: 167 packages and GBP 11,376.42 across 252 rows.
+
+## Rules — the broker's own logic
+
+```
+GET|POST /api/rules        PUT|DELETE /api/rules/{id}
+POST /api/batch/{id}/rows/{rowIndex}/make-rule
+```
+
+A rule matches narrowly — any combination of code prefix, origin, destination,
+direction and a description substring — and does one thing: pin a code, add
+document codes, suppress document codes, or record a note. A rule with no
+conditions is refused, because it would touch every line in every pack.
+
+Two guarantees hold whatever a rule does:
+
+- **It is never silent.** Every rule that fires is named on the line in
+  `appliedRules` and as an INFO warning saying what it changed.
+- **It is never green.** A rule is the broker's logic, not TARIC's, so the
+  confidence it touched drops to amber and the source becomes `MANUAL`.
+
+Pinning a code re-runs the check under the pinned code, so the documents belong
+to the code actually being declared rather than to the one the reference
+proposed. Rules are scoped to one client profile or to the tenant, and each
+counts its own hits — which is the input the learning layer will need.
+
 ## Acceptance metrics
 
 ```
@@ -150,5 +221,12 @@ the first real files:
   document-code prefix (C/N/X = obligation, Y = exemption) instead. Verify
   against the published Measure-action list.
 - **The Softpak column order** — `default_export_columns()` in
-  `client/models.py`, pending Judith's template specification.
-- **The Goenee column mapping** — `goenee_profile()` in `client/models.py`.
+  `client/models.py`, pending Judith's template specification. The `softpak`
+  export currently reproduces the layout of the packs Goenee actually sends
+  (`PF 04-09-2026.xlsx`), which is evidence, not a specification.
+- **The Goenee column mapping** — `goenee_softpak_profile()` in
+  `client/models.py`.
+- **The G-Star gross weight.** Counting the manifest once per parcel gives
+  175.86 kg; the Master Invoice for the same shipment says 148.69 kg. Packages
+  and value reconcile exactly, weight does not. Ask which basis the declared
+  weight uses before trusting that figure.

cargoflow-codecontrole.html

@@ -0,0 +1,23 @@
+<!doctype html>
+<html lang="nl"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>CargoFlow — Minder zoeken. Sneller klaar.</title><style>
+*{box-sizing:border-box}body{margin:0;background:#f6f8fb;color:#192d40;font:15px/1.55 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}button,input,select,textarea{font:inherit}button{cursor:pointer;border:0}header{height:80px;padding:0 6%;display:flex;align-items:center;justify-content:space-between;border-bottom:1px solid #e4e9ef;background:white}.logo{font-size:24px;font-weight:750;letter-spacing:-1px}.logo span{color:#346fe3}.demo{font-size:12px;color:#64748b;background:#f1f4f8;padding:5px 11px;border-radius:20px}main{max-width:740px;margin:52px auto;padding:0 22px}h1{font-size:36px;line-height:1.2;letter-spacing:-1.3px;margin:0 0 12px}h2{font-size:22px;line-height:1.3;letter-spacing:-.5px;margin:0 0 9px}p{margin:0;color:#748094}.intro{text-align:center;margin-bottom:30px}.steps{display:flex;justify-content:center;gap:24px;font-size:12px;color:#8893a3;margin-bottom:25px}.steps span.active{color:#2865d9;font-weight:650}.card{background:white;border:1px solid #e1e7ef;box-shadow:0 8px 35px #19304d05;border-radius:16px;padding:30px}.row{display:grid;grid-template-columns:1fr 1fr;gap:18px}.field{margin-bottom:20px}label{display:block;font-size:13px;font-weight:600;margin-bottom:7px}input,select,textarea{width:100%;border:1px solid #dce3ec;border-radius:9px;padding:12px;background:#fff;color:#24394e}textarea{height:105px;resize:vertical}input:focus,select:focus,textarea:focus{outline:2px solid #b7d2ff;border-color:#6f9fee}button:focus-visible,summary:focus-visible{outline:3px solid #a4caff;outline-offset:3px}.primary{background:#2867e3;color:white;border-radius:9px;padding:14px 22px;font-weight:650;width:100%;margin-top:12px}.primary:hover{background:#1d57c7}.primary:disabled{background:#a9b8cf;cursor:not-allowed}.textbutton{background:none;color:#3970cb;padding:7px 0;font-size:12px}.under{text-align:center;font-size:11px;color:#8b96a6;margin-top:12px}summary{cursor:pointer;color:#728096;font-size:12px}details .content{padding-top:18px}details{margin:4px 0 15px}.context{font-size:12px;color:#748094;margin-bottom:12px}.context b{color:#42566d;font-weight:500}.roi{margin:23px 0 0;padding:21px 24px;background:#ebf3f0;border:1px solid #dce9e3;border-radius:13px}.roihead{display:flex;justify-content:space-between;gap:16px;align-items:center}.roi label{font-size:11px;color:#637b73;font-weight:500}.roi strong{font-size:25px;letter-spacing:-.8px;color:#236953}.roi p{font-size:11px;color:#71877e}.roi summary{color:#527969;margin-top:15px}.metrics{display:grid;grid-template-columns:repeat(3,1fr);gap:15px}.roi .inputs{display:grid;grid-template-columns:repeat(2,1fr);gap:12px}.roi input{padding:8px;font-size:13px}.roi .formula{margin-top:12px;font-size:11px;color:#587567}.question{margin:23px 0}.choice{display:flex;gap:10px;flex-wrap:wrap}.choice button{flex:1;border:1px solid #dce3ec;border-radius:9px;padding:12px;background:white;color:#4a5c70;white-space:nowrap}.choice button.selected{border-color:#2d6ce5;color:#2867df;background:#eff5ff}.tag{display:inline-block;background:#edf4ff;color:#4974b1;border-radius:6px;padding:5px 9px;font-size:11px;margin-bottom:14px}.resultrow{border-bottom:1px solid #e9edf3;padding:16px 0;display:flex;gap:20px;justify-content:space-between;font-size:13px}.resultrow span{color:#718198}.resultrow b{text-align:right;max-width:65%;overflow-wrap:anywhere}.note{background:#f7f9fc;padding:14px;border-radius:8px;color:#778498;font-size:12px;margin:18px 0}.back{background:none;color:#76869a;font-size:12px;padding:0;margin-bottom:18px}.hidden{display:none!important}.error{color:#b45535;font-size:12px;margin-top:10px}.done{text-align:center}.check{margin:12px auto 18px;background:#e8f4ee;color:#2b8060;width:50px;height:50px;border-radius:50%;display:grid;place-items:center;font-size:26px}footer{text-align:center;font-size:11px;color:#94a0af;margin:24px 0}.small{font-size:12px}.secondary{border:1px solid #dce3ec;border-radius:9px;padding:12px 18px;background:white;color:#415972;width:100%;margin-top:10px}@media(max-width:560px){header{height:64px}main{margin:30px auto;padding:0 16px}h1{font-size:29px}.card{padding:23px 20px}.row{grid-template-columns:1fr;gap:0}.steps{gap:18px}.roi{padding:18px}.metrics{gap:9px}.roi strong{font-size:21px}.roihead{align-items:start}.choice button{padding:10px;font-size:12px}}
+</style></head><body><header><div class="logo">Cargo<span>Flow</span></div><span class="demo">Interactieve demo</span></header><main><div class="intro"><h1>Minder zoeken. Sneller klaar.</h1><p>Van product naar een controleerbaar codevoorstel.</p></div><div class="steps" aria-label="Voortgang"><span id="step1" class="active">1. Product</span><span id="step2">2. Aanvullen</span><span id="step3">3. Resultaat</span></div><div class="card">
+<section id="input"><form id="form"><div class="field"><label for="product">Wat wil je controleren?</label><textarea id="product" required placeholder="Omschrijving van de goederen, bijvoorbeeld: katoenen T-shirts, gebreid, 100% katoen."></textarea></div><div class="row"><div class="field"><label for="code">HS- of goederencode <span style="font-weight:400;color:#939ead">· indien bekend</span></label><input id="code" inputmode="numeric" placeholder="Bijv. 610910" pattern="([0-9]{6}|[0-9]{8}|[0-9]{10})" title="Gebruik 6, 8 of 10 cijfers"></div><div class="field"><label for="origin">Land van oorsprong</label><input id="origin" required placeholder="Bijv. India"></div></div><div class="context" id="contextSummary"></div><details><summary>Context aanpassen</summary><div class="content row"><div class="field"><label for="direction">Goederenstroom</label><select id="direction"><option>Invoer</option><option>Uitvoer</option></select></div><div class="field"><label for="country">Land van aangifte</label><select id="country"><option>Nederland</option><option>België</option><option>Duitsland</option></select></div><div class="field"><label for="date">Peildatum</label><input id="date" type="date" required></div><div class="field"><label for="procedure">Regeling</label><select id="procedure"><option>Vrij verkeer</option><option>Douane-entrepot</option><option>Tijdelijke invoer</option><option>Uitvoer</option></select></div><div class="field"><label for="dispatch">Verzending / bestemming</label><input id="dispatch" placeholder="Indien bekend"></div></div></details><button type="submit" class="primary">Controle starten →</button><p class="under">Alleen aanvullende vragen waar nodig.</p><button type="button" id="example" class="textbutton">Probeer een voorbeeld</button></form></section>
+<section id="questions" class="hidden"><button class="back" data-goto="input">← Invoer aanpassen</button><br><span class="tag">Nog 1 vraag · voorbeeldflow</span><h2>Is de productsamenstelling bekend?</h2><p class="small">In productie bepaalt de broninformatie welke vraag nodig is. Deze demo laat zien hoe je alleen het ontbrekende aanvult.</p><div class="question"><label>Heb je de samenstelling gecontroleerd?</label><div class="choice"><button data-answer="Ja" aria-pressed="false">Ja, gecontroleerd</button><button data-answer="Nee" aria-pressed="false">Nee</button><button data-answer="Onbekend" aria-pressed="false">Weet ik niet</button></div></div><button id="toResult" class="primary" disabled>Bekijk resultaat →</button></section>
+<section id="result" class="hidden"><button class="back" data-goto="questions">← Antwoord aanpassen</button><br><span class="tag">Concept · demo</span><h2 id="resultTitle">Dit is je controleoverzicht.</h2><p class="small" id="resultIntro"></p><div class="resultrow"><span>Product</span><b id="outProduct"></b></div><div class="resultrow"><span>Aangeleverde code</span><b id="outCode"></b></div><div class="resultrow"><span>Bescheidcodes</span><b>Broncontrole nodig</b></div><details style="margin-top:18px"><summary>Bekijk invoer en onderbouwing</summary><div class="content small" id="evidence"></div></details><div class="note">Live DTV/TARIC is niet aangesloten. Daarom worden hier geen definitieve codes of verplichtingen vastgesteld.</div><button class="primary" id="finish">Sla concept op</button><p class="under">Geen bevestiging van een douanebeoordeling.</p></section>
+<section id="done" class="hidden done"><div class="check">✓</div><h2>Concept gereed.</h2><p class="small" id="savedMessage">Download je invoer en open punten om verder te beoordelen.</p><button class="primary" id="download">↓ Download controleoverzicht</button><button class="secondary" id="restart">Volgend product controleren</button></section></div>
+<aside class="roi" aria-label="ROI calculator"><div class="metrics" aria-live="polite"><div><label>Potentiële tijdwinst / maand</label><strong id="hours">—</strong></div><div><label>Netto waarde / maand</label><strong id="net">—</strong></div><div><label>ROI per maand</label><strong id="roi">—</strong></div></div><details><summary>Jouw ROI berekenen · aannames aanpassen</summary><div class="content"><div class="inputs"><div><label for="volume">Controles per maand</label><input type="number" id="volume" value="400" min="0" step="1"></div><div><label for="rate">Uurkosten medewerker (€)</label><input type="number" id="rate" value="55" min="0"></div><div><label for="before">Minuten per controle, nu</label><input type="number" id="before" value="8" min="0" step="0.5"></div><div><label for="after">Minuten met CargoFlow, incl. review</label><input type="number" id="after" value="3" min="0" step="0.5"></div><div><label for="fee">Softwarekosten per maand (€)</label><input type="number" id="fee" value="299" min="0"></div><div><label for="other">Overige maandkosten (€)</label><input type="number" id="other" value="0" min="0"></div></div><div class="formula" id="formula"></div></div></details><p id="assumption">Rekenvoorbeeld, geen gemeten besparing of prijsaanbod.</p></aside><footer>Prototype · Geen live douaneadvies · De declarant houdt de controle.</footer></main>
+<script>
+const $=s=>document.querySelector(s);let answer='',record=null;const money=n=>new Intl.NumberFormat('nl-NL',{style:'currency',currency:'EUR',maximumFractionDigits:0}).format(n);const fmt=n=>new Intl.NumberFormat('nl-NL',{maximumFractionDigits:1}).format(n);
+function calculate(v,b,a,r,f,o){const hours=v*(b-a)/60,gross=hours*r,cost=f+o,net=gross-cost;return{hours,gross,cost,net,roi:cost>0?net/cost*100:null}}
+function roi(){const ids=['volume','before','after','rate','fee','other'],values=ids.map(id=>Number($('#'+id).value));if(ids.some(id=>$('#'+id).value==='')||values.some(v=>!Number.isFinite(v)||v<0)){$('#hours').textContent=$('#net').textContent=$('#roi').textContent='—';$('#formula').textContent='Vul voor elke aanname een getal van minimaal 0 in.';return}const x=calculate(...values);$('#hours').textContent=fmt(x.hours)+' uur';$('#net').textContent=money(x.net);$('#roi').textContent=x.roi===null?'n.v.t.':fmt(x.roi)+'%';$('#formula').textContent=money(x.gross)+' capaciteitswaarde − '+money(x.cost)+' kosten = '+money(x.net)+' netto per maand. ROI = netto waarde ÷ kosten × 100%. Eenmalige implementatiekosten zijn niet meegerekend.';$('#assumption').textContent='Aannames, geen gemeten resultaat. Vrijgekomen capaciteit is niet automatisch een besparing op uitgaven. €299 is een rekenvoorbeeld, geen prijsaanbod.'}
+function show(id){['input','questions','result','done'].forEach(s=>$('#'+s).classList.toggle('hidden',s!==id));[1,2,3].forEach(n=>$('#step'+n).classList.toggle('active',n===(id==='input'?1:id==='questions'?2:3)));window.scrollTo({top:0,behavior:'smooth'})}
+function context(){$('#contextSummary').textContent=[$('#direction').value,$('#country').value,$('#procedure').value,$('#date').value].join(' · ')}
+$('#date').value=new Date().toISOString().slice(0,10);['direction','country','procedure','date'].forEach(id=>$('#'+id).addEventListener('change',context));context();
+$('#example').onclick=()=>{$('#product').value='Gebreide T-shirts, 100% katoen';$('#code').value='610910';$('#origin').value='India';$('#product').focus()};
+$('#form').onsubmit=e=>{e.preventDefault();if(!$('#product').value.trim()||!$('#origin').value.trim())return;answer='';document.querySelectorAll('[data-answer]').forEach(b=>{b.classList.remove('selected');b.setAttribute('aria-pressed','false')});$('#toResult').disabled=true;show('questions')};
+document.querySelectorAll('[data-answer]').forEach(b=>b.onclick=()=>{answer=b.dataset.answer;document.querySelectorAll('[data-answer]').forEach(c=>{c.classList.toggle('selected',c===b);c.setAttribute('aria-pressed',String(c===b))});$('#toResult').disabled=false});document.querySelectorAll('[data-goto]').forEach(b=>b.onclick=()=>show(b.dataset.goto));
+$('#toResult').onclick=()=>{record={type:'CargoFlow concept — DEMO, geen douanebeoordeling',product:$('#product').value.trim(),aangeleverdeCode:$('#code').value||'Niet ingevuld',oorsprong:$('#origin').value.trim(),goederenstroom:$('#direction').value,landVanAangifte:$('#country').value,regeling:$('#procedure').value,peildatum:$('#date').value,verzendingOfBestemming:$('#dispatch').value||'Onbekend',samenstellingGecontroleerd:answer,bescheidcodes:'Niet vastgesteld — live broncontrole nodig',status:'Concept; beoordeling door declarant nodig'};$('#outProduct').textContent=record.product;$('#outCode').textContent=record.aangeleverdeCode;$('#resultTitle').textContent=answer==='Ja'?'Je invoer staat klaar.':'Er staat nog een vraag open.';$('#resultIntro').textContent=answer==='Ja'?'De volgende stap is inhoudelijke broncontrole.':'Vraag de productspecificatie op. Je kunt dit concept alvast bewaren.';$('#evidence').textContent=record.oorsprong+' · '+record.goederenstroom+' · '+record.landVanAangifte+' · '+record.regeling+' · '+record.peildatum+'. Samenstelling gecontroleerd: '+answer+'. Aangeleverde code niet gevalideerd. Geen bronmaatregelen geladen.';show('result')};
+$('#finish').onclick=()=>{try{localStorage.setItem('cargoflow-simple-concept',JSON.stringify(record));$('#savedMessage').textContent='Lokaal opgeslagen in deze browser. Download het overzicht voor verdere beoordeling.'}catch{$('#savedMessage').textContent='Download het overzicht om dit concept te bewaren. Lokale opslag is hier niet beschikbaar.'}show('done')};
+$('#download').onclick=()=>{const blob=new Blob([JSON.stringify(record,null,2)],{type:'application/json'}),url=URL.createObjectURL(blob),a=document.createElement('a');a.href=url;a.download='cargoflow-controleconcept.json';a.click();setTimeout(()=>URL.revokeObjectURL(url),1000)};$('#restart').onclick=()=>{$('#product').value='';$('#code').value='';answer='';record=null;show('input');$('#product').focus()};['volume','before','after','rate','fee','other'].forEach(id=>$('#'+id).addEventListener('input',roi));roi();
+</script></body></html>

cbh/__init__.py

@@ -1,8 +1,3 @@
-# pylint: disable=C0415
-"""
-CargoFlow application package.
-"""
-
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
 from starlette.exceptions import HTTPException as StarletteHTTPException
@@ -11,14 +6,6 @@ from cbh.core.wrappers import CbhResponseWrapper, ErrorCbhResponse
 
 
 def register_routers(app: FastAPI) -> None:
-    """
-    Attach every module's router to the application.
-
-    Each module defines its router in its ``__init__`` and its endpoints in
-    ``views``; importing the view module is what binds the two together.
-    """
-    from cbh.api.account import account_router
-    from cbh.api.account import views as account_views  # noqa: F401
     from cbh.api.batch import batch_router
     from cbh.api.batch import views as batch_views  # noqa: F401
     from cbh.api.client import client_router
@@ -27,24 +14,23 @@ def register_routers(app: FastAPI) -> None:
     from cbh.api.importer import views as importer_views  # noqa: F401
     from cbh.api.linecheck import linecheck_router
     from cbh.api.linecheck import views as linecheck_views  # noqa: F401
+    from cbh.api.rules import rules_router
+    from cbh.api.rules import views as rules_views  # noqa: F401
     from cbh.api.taric import taric_router
     from cbh.api.taric import views as taric_views  # noqa: F401
 
     for router in (
-        account_router,
         taric_router,
         linecheck_router,
         importer_router,
         client_router,
         batch_router,
+        rules_router,
     ):
         app.include_router(router)
 
 
 def create_app() -> FastAPI:
-    """
-    Create and configure the FastAPI application.
-    """
     app = FastAPI(
         title="CargoFlow Regelcheck",
         description=(
@@ -70,18 +56,12 @@ def create_app() -> FastAPI:
 
     @app.exception_handler(StarletteHTTPException)
     async def http_exception_handler(_, exc):
-        """
-        Handle HTTP exceptions and convert them to standardized error responses.
-        """
         return CbhResponseWrapper(
             data=None, successful=False, error=ErrorCbhResponse(message=str(exc.detail))
         ).response(exc.status_code)
 
     @app.on_event("startup")
     async def startup_event():
-        """
-        Execute startup tasks when the application starts.
-        """
         from cbh.api.client.db_requests import ensure_default_profiles
         from cbh.api.common.db_requests import ensure_indexes
         from cbh.core.config import settings
@@ -96,9 +76,6 @@ def create_app() -> FastAPI:
 
     @app.on_event("shutdown")
     async def shutdown_event():
-        """
-        Stop background jobs when the application stops.
-        """
         from cbh.core.config import settings
 
         if settings.SCHEDULER.running:
@@ -106,16 +83,10 @@ def create_app() -> FastAPI:
 
     @app.get("/api/health")
     async def health():
-        """
-        Health check endpoint for container orchestration and monitoring.
-        """
         return {"status": "healthy"}
 
     @app.get("/")
     async def root():
-        """
-        Root endpoint for the application.
-        """
         return {"service": "CargoFlow Regelcheck", "version": "0.3.0"}
 
     return app

cbh/api/account/__init__.py

@@ -1,12 +0,0 @@
-"""
-Account module initialization.
-
-This module defines the FastAPI router for account API endpoints
-and imports related views for account management.
-"""
-
-from fastapi import APIRouter
-
-account_router = APIRouter(
-    prefix="/api/account",
-)

cbh/api/account/dto.py

@@ -1,38 +0,0 @@
-"""
-Account DTOs.
-"""
-
-from enum import Enum
-
-
-class AccountType(Enum):
-    """
-    Enum for account types.
-    """
-
-    USER = 1
-    ADMIN = 2
-    OWNER = 3
-    SUPER_ADMIN = 4
-
-
-class AccountStatus(Enum):
-    """
-    Enum for account statuses.
-    """
-
-    ACTIVE = 1
-    BLOCKED = 2
-    PENDING_INVITATION = 3
-    PENDING_COMPLETION = 4
-
-
-class RegistrationType(Enum):
-    """
-    Enum for registration types.
-    """
-
-    ORGANIC = 1
-    GOOGLE = 2
-    GITHUB = 3
-    APPLE = 4

cbh/api/account/models.py

@@ -1,124 +0,0 @@
-"""
-Account models.
-"""
-
-from datetime import date, datetime
-
-from passlib.context import CryptContext
-from pydantic import Field, field_validator
-
-from cbh.api.account.dto import (
-    AccountStatus,
-    AccountType,
-    RegistrationType
-)
-from cbh.core.config import settings
-from cbh.core.database import MongoBaseModel, MongoBaseShortenModel
-
-
-class OrganizationShorten(MongoBaseShortenModel):
-    """
-    Organization shorten model.
-    """
-
-    name: str
-    slug: str | None = None
-    pictureUrl: str | None = None
-
-    @field_validator("pictureUrl", mode="before")
-    @classmethod
-    def serialize_picture_url(cls, v: str | None) -> str | None:
-        """
-        Serialize the scenario picture URL.
-        """
-        if v:
-            if not v.startswith("https://"):
-                return settings.S3_CLIENT.generate_presigned_url(v, expiration=3600)
-            return v
-        return None
-
-
-class AccountModel(MongoBaseModel):
-    """
-    Account model class.
-    """
-
-    name: str | None = None
-    email: str
-    pictureUrl: str | None = None
-    password: str | None = Field(exclude=True, default=None)
-
-    status: AccountStatus = AccountStatus.ACTIVE
-    accountType: AccountType = Field(default=AccountType.USER)
-    registrationType: RegistrationType | None = Field(
-        default=RegistrationType.ORGANIC, exclude=True
-    )
-
-    organization: OrganizationShorten | None = None
-    freeMinutesUsed: int = Field(default=0, exclude=True)
-
-    lastLogin: date | None = Field(exclude=True, default=None)
-    datetimeInserted: datetime = Field(default_factory=datetime.now)
-    datetimeUpdated: datetime = Field(default_factory=datetime.now)
-
-    @field_validator("password", mode="before", check_fields=False)
-    @classmethod
-    def set_password_hash(cls, v: str | None) -> str | None:
-        """
-        Set the password hash.
-
-        Args:
-            v (str): The password to hash.
-
-        Returns:
-            str: The hashed password.
-        """
-        if isinstance(v, str) and not v.startswith("$2b$"):
-            return CryptContext(schemes=["bcrypt"], deprecated="auto").hash(v)
-        return v
-
-    @field_validator("pictureUrl", mode="before")
-    @classmethod
-    def serialize_picture_url(cls, v: str | None) -> str | None:
-        """ """
-        if v:
-            if not v.startswith("https://"):
-                return settings.S3_CLIENT.generate_presigned_url(v, expiration=3600)
-            return v
-        return None
-
-    class Config:  # pylint: disable=R0903
-        """
-        Config for the AccountModel class.
-        """
-
-        arbitrary_types_allowed = True
-        populate_by_name = True
-        json_encoders = {datetime: lambda dt: dt.isoformat()}
-
-
-class AccountShorten(MongoBaseShortenModel):
-    """
-    Account shorten model.
-    """
-
-    id: str
-    name: str | None = None
-    email: str
-    pictureUrl: str | None = None
-
-    accountType: AccountType
-
-    organization: OrganizationShorten | None = None
-
-    @field_validator("pictureUrl", mode="before")
-    @classmethod
-    def serialize_picture_url(cls, v: str | None) -> str | None:
-        """
-        Serialize the account picture URL
-        """
-        if v:
-            if not v.startswith("https://"):
-                return settings.S3_CLIENT.generate_presigned_url(v, expiration=3600)
-            return v
-        return None

cbh/api/account/schemas.py

@@ -1,11 +0,0 @@
-from pydantic import BaseModel
-
-
-class AccountFilter(BaseModel):
-    """
-    Account filter.
-    """
-
-    accountTypes: list[int] | None = None
-    statuses: list[int] | None = None
-    searchTerm: str | None = None

cbh/api/account/utils.py

@@ -1,36 +0,0 @@
-from io import BytesIO
-
-from PIL import Image, ImageOps, UnidentifiedImageError
-from fastapi import HTTPException, UploadFile
-
-from cbh.api.account.models import AccountModel
-
-
-def compress_image(file: UploadFile) -> bytes:
-    if not file.content_type or not file.content_type.startswith("image/"):
-        raise HTTPException(status_code=400, detail="File must be an image")
-    try:
-        file.file.seek(0)
-        original_bytes = file.file.read()
-        image = Image.open(BytesIO(original_bytes))
-        image = ImageOps.exif_transpose(image)
-        max_size = 512
-        image.thumbnail((max_size, max_size), Image.Resampling.LANCZOS)
-        if image.mode not in ("RGB", "RGBA"):
-            image = image.convert("RGBA")
-        buffer = BytesIO()
-        image.save(buffer, format="PNG", optimize=True, compress_level=9)
-        return buffer.getvalue()
-    except (UnidentifiedImageError, OSError):
-        raise HTTPException(status_code=400, detail="Invalid image file")
-    finally:
-        try:
-            file.file.close()
-        except Exception:
-            pass
-
-
-def prepare_additional_filter(account: AccountModel) -> dict:
-    return {
-        "organization.id": account.organization.id,
-    }

cbh/api/account/views.py

@@ -1,44 +0,0 @@
-"""
-Account views module.
-"""
-
-from fastapi import Depends
-
-from cbh.api.account import account_router
-from cbh.api.account.dto import AccountType
-from cbh.api.account.models import AccountModel
-from cbh.api.common.db_requests import delete_obj
-from cbh.core.security import PermissionDependency
-from cbh.core.wrappers import CbhResponseWrapper
-
-
-@account_router.delete("/{accountId}")
-async def delete_account(
-    accountId: str,
-    account: AccountModel = Depends(
-        PermissionDependency(
-            [AccountType.OWNER],
-        )
-    ),
-) -> CbhResponseWrapper[AccountModel]:
-    """
-    Delete an account.
-    """
-    removed_account = await delete_obj(
-        AccountModel,
-        accountId,
-        additional_filter={
-            "organization.id": account.organization.id,
-        },
-    )
-    return CbhResponseWrapper()
-
-
-@account_router.get("")
-async def get_own_account(
-    account: AccountModel = Depends(PermissionDependency()),
-) -> CbhResponseWrapper[AccountModel]:
-    """
-    Get own account.
-    """
-    return CbhResponseWrapper(data=account)

cbh/api/batch/__init__.py

@@ -1,11 +1,3 @@
-"""
-Batch module initialization.
-
-This module defines the FastAPI router for the Excel processor: uploading a
-client's pack, running it through the line-check API and handing back the same
-file with the checker's columns added.
-"""
-
 from fastapi import APIRouter
 
 batch_router = APIRouter(

cbh/api/batch/db_requests.py

@@ -1,31 +1,20 @@
-"""
-Batch database requests.
-"""
-
 from cbh.api.batch.models import BatchModel, BatchRowModel
 from cbh.api.common.db_requests import find_objs, get_all_objs, get_obj_by_id
 
 
 async def get_batch(batch_id: str) -> BatchModel:
-    """
-    Fetch a pack, 404 when it does not exist.
-    """
     return await get_obj_by_id(BatchModel, batch_id)
 
 
 async def get_batches(
     page_size: int, page_index: int, status: int | None = None
 ) -> tuple[list[BatchModel], int]:
-    """
-    Page through the packs, newest first.
-    """
-    filter_ = {"status": status} if status is not None else {}
     return await get_all_objs(
         BatchModel,
         page_size=page_size,
         page_index=page_index,
         sort=("datetimeInserted", -1),
-        additional_filter=filter_,
+        additional_filter={"status": status} if status is not None else {},
     )
 
 
@@ -35,12 +24,6 @@ async def get_batch_rows(
     page_index: int = 0,
     needs_review_only: bool = False,
 ) -> tuple[list[BatchRowModel], int]:
-    """
-    Read the rows of a pack in spreadsheet order.
-
-    ``needs_review_only`` is the working view for a broker: the lines the
-    machine refused to answer for.
-    """
     filter_: dict = {"batchId": batch_id}
     if needs_review_only:
         filter_["needsReview"] = True

cbh/api/batch/dto.py

@@ -1,18 +1,7 @@
-"""
-Batch (Excel processing) DTOs.
-"""
+from enum import IntEnum
 
-from enum import Enum
-
-
-class BatchStatus(Enum):
-    """
-    Lifecycle of an uploaded pack.
-
-    ``MAPPING_REQUIRED`` means the file was read but the profile does not map
-    everything needed — the pack waits rather than being processed on guesses.
-    """
 
+class BatchStatus(IntEnum):
     UPLOADED = 1
     MAPPING_REQUIRED = 2
     PROCESSING = 3
@@ -20,30 +9,48 @@ class BatchStatus(Enum):
     FAILED = 5
 
 
-class FileFormat(Enum):
-    """
-    Format a pack arrived in.
-    """
-
+class FileFormat(IntEnum):
     XLSX = 1
     CSV = 2
 
     @classmethod
     def from_name(cls, file_name: str) -> "FileFormat":
-        """
-        Derive the format from a file name.
-        """
         return cls.CSV if file_name.lower().endswith((".csv", ".txt")) else cls.XLSX
 
 
-class ReviewStage(Enum):
+class ReviewStage(IntEnum):
+    NOT_STARTED = 1
+    IN_REVIEW = 2
+    REVIEWED = 3
+
+
+class BatchSource(IntEnum):
+    """How the lines got into the shipment.
+
+    Both routes end in the same review table; this only records which one was
+    taken, so a manually built shipment is not mistaken for a parsed file.
     """
-    Where a pack stands in human review.
 
-    Review time is the metric that decides whether the product is worth having,
-    so it is measured rather than estimated.
+    UPLOAD = 1
+    MANUAL = 2
+
+
+class ExportFormat(IntEnum):
+    """The client system the reviewed table is exported for.
+
+    REVIEW is the tool's own full table. SOURCE returns the client's uploaded
+    file with the checked columns appended, which is what the upload route has
+    always produced. SOFTPAK is the Goenee line list. SCOPE and FITON are
+    declared but not implemented: their import specifications have not been
+    received, and guessing at a layout is worse than refusing.
     """
 
-    NOT_STARTED = 1
-    IN_REVIEW = 2
-    REVIEWED = 3
+    REVIEW = 1
+    SOURCE = 2
+    SOFTPAK = 3
+    SCOPE = 4
+    FITON = 5
+
+    @property
+    def implemented(self) -> bool:
+        return self in (ExportFormat.REVIEW, ExportFormat.SOURCE, ExportFormat.SOFTPAK)

cbh/api/batch/models.py

@@ -1,67 +1,58 @@
-"""
-Batch models.
-"""
-
 from datetime import date, datetime
 
 from pydantic import Field
 
-from cbh.api.batch.dto import BatchStatus, FileFormat, ReviewStage
-from cbh.api.common.dto import TradeDirection
+from cbh.api.batch.dto import BatchSource, BatchStatus, FileFormat, ReviewStage
+from cbh.api.common.dto import CustomsProcedure, TradeDirection
 from cbh.api.common.models import FileRef
-from cbh.core.database import MongoBaseModel, MongoBaseShortenModel, MongoSerializableModel
+from cbh.api.linecheck.dto import RowDecision
+from cbh.core.database import MongoBaseModel, MongoSerializableModel
 
 
 class ClientProfileRef(MongoSerializableModel):
-    """
-    The profile a pack was processed under.
-
-    Stored by value as well as by id: a pack processed last month was processed
-    under the mapping as it stood then, and re-reading it must not silently
-    adopt a mapping edited since.
-    """
-
     id: str
     name: str
 
 
 class BatchMetrics(MongoSerializableModel):
-    """
-    The numbers the acceptance criteria are stated in.
-
-    ``reviewSeconds`` is the one that matters commercially: it is the time a
-    human spent on the pack after the machine had finished.
-    """
-
     totalLines: int = 0
     uniqueLookups: int = 0
     machineSeconds: float = 0.0
     reviewSeconds: float | None = None
     statusCounts: dict[str, int] = Field(default_factory=dict)
 
+    extractionScore: float = 0.0
+    documentScore: float = 0.0
+    codeConfidenceCounts: dict[str, int] = Field(default_factory=dict)
+    documentConfidenceCounts: dict[str, int] = Field(default_factory=dict)
+
+    totalPackages: float = 0.0
+    totalGrossWeight: float = 0.0
+    totalNetWeight: float = 0.0
+    totalValue: float = 0.0
+
     @property
     def lines_needing_review(self) -> int:
-        """
-        Lines a broker has to look at before filing.
-        """
         return self.statusCounts.get("AMBIGUOUS", 0) + self.statusCounts.get("INVALID", 0)
 
 
 class BatchModel(MongoBaseModel):
-    """
-    One uploaded pack of declaration lines.
-    """
-
     name: str
     clientProfile: ClientProfileRef
     status: BatchStatus = BatchStatus.UPLOADED
+    source: BatchSource = BatchSource.UPLOAD
     fileFormat: FileFormat = FileFormat.XLSX
 
-    sourceFile: FileRef
+    # A shipment built line by line through the form has no source file.
+    sourceFile: FileRef | None = None
     resultFile: FileRef | None = None
 
     checkDate: date | None = None
     defaultDirection: TradeDirection = TradeDirection.IMPORT
+    defaultProcedure: CustomsProcedure = CustomsProcedure.RELEASE_FOR_FREE_CIRCULATION
+    destinationCountry: str | None = None
+    shipmentReference: str | None = None
+    currency: str = "EUR"
 
     metrics: BatchMetrics = Field(default_factory=BatchMetrics)
     errors: list[str] = Field(default_factory=list)
@@ -76,24 +67,7 @@ class BatchModel(MongoBaseModel):
     datetimeUpdated: datetime = Field(default_factory=datetime.now)
 
 
-class BatchShorten(MongoBaseShortenModel):
-    """
-    Batch projection for list endpoints.
-    """
-
-    name: str
-    status: BatchStatus
-    datetimeInserted: datetime
-
-
 class BatchRowModel(MongoBaseModel):
-    """
-    One row of a pack, with the answer it got.
-
-    The raw input is kept alongside the result so a disputed line can be
-    replayed exactly as it was submitted.
-    """
-
     batchId: str
     rowIndex: int
 
@@ -104,4 +78,12 @@ class BatchRowModel(MongoBaseModel):
     status: str = "INVALID"
     needsReview: bool = False
 
+    decision: RowDecision = RowDecision.PENDING
+    correctedCode: str | None = None
+    cbamPermitNumber: str | None = None
+    eudrReference: str | None = None
+    licenceNumbers: list[str] = Field(default_factory=list)
+    reviewNote: str | None = None
+    datetimeReviewed: datetime | None = None
+
     datetimeInserted: datetime = Field(default_factory=datetime.now)

cbh/api/batch/schemas.py

@@ -1,32 +1,36 @@
-"""
-Batch schemas.
-"""
-
 from datetime import date, datetime
 from typing import Any
 
 from pydantic import BaseModel
 
-from cbh.api.batch.dto import BatchStatus, ReviewStage
+from cbh.api.batch.dto import BatchSource, BatchStatus, ExportFormat, ReviewStage
 from cbh.api.batch.models import BatchMetrics, ClientProfileRef
-from cbh.api.common.dto import TradeDirection
+from cbh.api.common.dto import Paging
+from cbh.api.common.dto import CustomsProcedure, TradeDirection
+from cbh.api.linecheck.dto import ConfidenceLevel, ReferenceSource, RowDecision
+from cbh.api.linecheck.schemas import (
+    AppliedRule,
+    DeclarationDate,
+    DocumentRequirement,
+    LineWarning,
+)
 
 
 class BatchResponse(BaseModel):
-    """
-    A pack as returned by the API.
-    """
-
     id: str
     name: str
     clientProfile: ClientProfileRef
     status: BatchStatus
+    source: BatchSource = BatchSource.UPLOAD
 
-    sourceFileName: str
+    sourceFileName: str | None = None
     resultFileName: str | None = None
 
     checkDate: date | None = None
     defaultDirection: TradeDirection
+    destinationCountry: str | None = None
+    shipmentReference: str | None = None
+    currency: str = "EUR"
 
     metrics: BatchMetrics
     errors: list[str] = []
@@ -40,10 +44,6 @@ class BatchResponse(BaseModel):
 
 
 class BatchRowResponse(BaseModel):
-    """
-    One row of a pack, with the answer it received.
-    """
-
     rowIndex: int
     status: str
     needsReview: bool
@@ -52,13 +52,6 @@ class BatchRowResponse(BaseModel):
 
 
 class GoldSetScoreResponse(BaseModel):
-    """
-    Accuracy of a pack against a confirmed gold file, plus its timings.
-
-    Review time is the metric that sells the product, so it is reported
-    alongside accuracy rather than separately.
-    """
-
     goldRows: int
     matchedRows: int
     unmatchedGoldRows: list[int] = []
@@ -75,3 +68,150 @@ class GoldSetScoreResponse(BaseModel):
     reviewSeconds: float | None = None
     totalLines: int | None = None
     linesNeedingReview: int | None = None
+
+
+class ReviewDocuments(BaseModel):
+    required: list[DocumentRequirement] = []
+    alternatives: list[list[DocumentRequirement]] = []
+    exemption: list[DocumentRequirement] = []
+    suspended: list[DocumentRequirement] = []
+    noneRequired: bool = False
+    label: str
+
+
+class ReviewRow(BaseModel):
+    rowIndex: int
+    description: str | None = None
+
+    inputCode: str | None = None
+    proposedCode: str | None = None
+    formattedCode: str | None = None
+    codeDescription: str | None = None
+    candidates: list[str] = []
+    replacementCodes: list[str] = []
+
+    originCountry: str | None = None
+    destinationCountry: str | None = None
+    measureCountry: str | None = None
+    direction: TradeDirection | None = None
+    procedure: CustomsProcedure | None = None
+
+    codeConfidence: ConfidenceLevel
+    codeConfidenceReason: str = ""
+    documentConfidence: ConfidenceLevel
+    documentConfidenceReason: str = ""
+    source: ReferenceSource = ReferenceSource.NONE
+
+    documents: ReviewDocuments
+    warnings: list[LineWarning] = []
+    appliedRules: list[AppliedRule] = []
+
+    packages: float | None = None
+    grossWeight: float | None = None
+    netWeight: float | None = None
+    value: float | None = None
+
+    decision: RowDecision = RowDecision.PENDING
+    correctedCode: str | None = None
+    cbamPermitNumber: str | None = None
+    eudrReference: str | None = None
+    licenceNumbers: list[str] = []
+    reviewNote: str | None = None
+
+    status: str
+    needsReview: bool
+
+
+class ShipmentTotals(BaseModel):
+    packages: float = 0.0
+    grossWeight: float = 0.0
+    netWeight: float = 0.0
+    value: float = 0.0
+    currency: str = "EUR"
+
+
+class ReviewHeader(BaseModel):
+    batchId: str
+    shipmentReference: str | None = None
+    name: str
+    clientProfile: ClientProfileRef
+    status: BatchStatus
+    source: BatchSource = BatchSource.UPLOAD
+    direction: TradeDirection
+    procedure: CustomsProcedure = CustomsProcedure.RELEASE_FOR_FREE_CIRCULATION
+    originCountries: list[str] = []
+    destinationCountry: str | None = None
+    checkDate: date | None = None
+    referenceAsOf: str | None = None
+
+    totals: ShipmentTotals
+    totalLines: int = 0
+    extractionScore: float = 0.0
+    documentScore: float = 0.0
+    codeConfidenceCounts: dict[str, int] = {}
+    documentConfidenceCounts: dict[str, int] = {}
+    linesNeedingReview: int = 0
+
+
+class ReviewTableResponse(BaseModel):
+    header: ReviewHeader
+    paging: Paging
+    rows: list[ReviewRow]
+
+
+class RowReviewRequest(BaseModel):
+    correctedCode: str | None = None
+    decision: RowDecision | None = None
+    cbamPermitNumber: str | None = None
+    eudrReference: str | None = None
+    licenceNumbers: list[str] | None = None
+    reviewNote: str | None = None
+
+
+class ManualShipmentRequest(BaseModel):
+    """Open an empty shipment that the form will fill line by line."""
+
+    clientProfileId: str
+    name: str | None = None
+    checkDate: date | None = None
+    direction: TradeDirection | None = None
+    procedure: CustomsProcedure | None = None
+    destinationCountry: str | None = None
+    shipmentReference: str | None = None
+    currency: str = "EUR"
+
+
+class ManualLineRequest(BaseModel):
+    """One line from the single-item form, added to the review table.
+
+    These are the fields the UI specification lists for the form, plus the
+    quantities the totals row needs. A partial code is allowed: it resolves,
+    or it comes back ambiguous with its candidates, exactly as an uploaded
+    line does.
+    """
+
+    code: str
+    originCountry: str | None = None
+    destinationCountry: str | None = None
+    direction: TradeDirection | None = None
+    procedure: CustomsProcedure | None = None
+    date: DeclarationDate | None = None
+    additionalCodes: list[str] = []
+
+    description: str | None = None
+    quantity: float | None = None
+    packages: float | None = None
+    grossWeight: float | None = None
+    netWeight: float | None = None
+    value: float | None = None
+
+
+class ExportFormatResponse(BaseModel):
+    """One entry of the export dropdown."""
+
+    format: ExportFormat
+    name: str
+    description: str
+    available: bool
+    unavailableReason: str | None = None
+    isProfileDefault: bool = False

cbh/api/batch/services/excel/__init__.py

@@ -1,12 +1,24 @@
-from .reader import ParsedRow, ParsedSheet, parse_pack, read_headers
-from .writer import build_output_workbook, render_output_value, workbook_to_bytes
+from .export import (
+    build_export_workbook,
+    build_review_workbook,
+    build_softpak_workbook,
+    export_file_name,
+    unavailable_reason,
+)
+from .reader import ParsedRow, ParsedSheet, load_pack_workbook, parse_pack, read_headers
+from .writer import build_output_workbook, workbook_to_bytes
 
 __all__ = [
     "ParsedRow",
     "ParsedSheet",
+    "build_export_workbook",
     "build_output_workbook",
+    "build_review_workbook",
+    "build_softpak_workbook",
+    "export_file_name",
+    "load_pack_workbook",
     "parse_pack",
     "read_headers",
-    "render_output_value",
+    "unavailable_reason",
     "workbook_to_bytes",
 ]

cbh/api/batch/services/excel/export.py

@@ -0,0 +1,258 @@
+from openpyxl.styles import Alignment, Font, PatternFill
+from openpyxl.utils import get_column_letter
+from openpyxl.workbook import Workbook
+
+from cbh.api.batch.dto import ExportFormat
+from cbh.api.batch.models import BatchModel, BatchRowModel
+from cbh.api.batch.services.excel.writer import SOFTPAK_ORDER, render_output_value
+from cbh.api.batch.services.metrics.totals import TotalsInput, input_value, sheet_totals
+from cbh.api.client.dto import InputField, OutputField
+from cbh.api.client.models import ClientProfileModel
+from cbh.api.linecheck.dto import LineStatus, RowDecision
+from cbh.api.linecheck.schemas import LineCheckRequest, LineCheckResponse
+from cbh.api.linecheck.utils import render_required_documents, render_warnings
+
+STATUS_FILLS = {
+    LineStatus.INVALID: PatternFill("solid", fgColor="FFF8CBCB"),
+    LineStatus.AMBIGUOUS: PatternFill("solid", fgColor="FFFDE9A9"),
+    LineStatus.EXPIRING: PatternFill("solid", fgColor="FFE2F0D9"),
+}
+
+REVIEW_COLUMNS: list[tuple[str, int]] = [
+    ("#", 6),
+    ("Description", 38),
+    ("Input code", 14),
+    ("Proposed code", 14),
+    ("Code description", 38),
+    ("Origin", 8),
+    ("Destination", 11),
+    ("Code confidence", 15),
+    ("Code confidence reason", 40),
+    ("Documents required", 26),
+    ("Document codes exempt", 22),
+    ("No document required", 20),
+    ("Document confidence", 18),
+    ("Document confidence reason", 40),
+    ("Warnings", 46),
+    ("Rules applied", 26),
+    ("CBAM permit", 18),
+    ("EUDR reference", 18),
+    ("Licences", 18),
+    ("Decision", 12),
+    ("Review note", 30),
+    ("Status", 12),
+    ("Packages", 10),
+    ("Gross weight", 12),
+    ("Net weight", 12),
+    ("Value", 12),
+]
+
+
+def _unpack(row: BatchRowModel) -> tuple[LineCheckRequest | None, LineCheckResponse | None]:
+    request = LineCheckRequest(**row.request) if row.request else None
+    result = LineCheckResponse(**row.result) if row.result else None
+    return request, result
+
+
+def _totals_lines(rows: list[BatchRowModel]) -> list[TotalsInput]:
+    return [
+        TotalsInput(raw=row.inputRow, request=_unpack(row)[0]) for row in rows
+    ]
+
+
+def _review_values(row: BatchRowModel) -> list:
+    request, result = _unpack(row)
+    return [
+        row.rowIndex,
+        request.description if request else "",
+        result.inputCode if result else "",
+        row.correctedCode or (result.resolvedCode if result else "") or "",
+        result.codeDescription if result else "",
+        result.originCountry if result else "",
+        result.destinationCountry if result else "",
+        result.codeConfidence.name.lower() if result else "",
+        result.codeConfidenceReason if result else "",
+        render_required_documents(result) if result else "",
+        ", ".join(document.code for document in result.documentsExemption) if result else "",
+        "yes" if result and result.noneRequired else "",
+        result.documentConfidence.name.lower() if result else "",
+        result.documentConfidenceReason if result else "",
+        render_warnings(result) if result else "",
+        ", ".join(rule.name for rule in result.appliedRules) if result else "",
+        row.cbamPermitNumber or "",
+        row.eudrReference or "",
+        ", ".join(row.licenceNumbers),
+        RowDecision(row.decision).name.lower(),
+        row.reviewNote or "",
+        row.status.lower(),
+        request.packages if request else None,
+        request.grossWeight if request else None,
+        request.netWeight if request else None,
+        request.value if request else None,
+    ]
+
+
+def _write_header(worksheet, headers: list[str], row_index: int) -> None:
+    for column_index, header in enumerate(headers, start=1):
+        cell = worksheet.cell(row=row_index, column=column_index)
+        cell.value = header
+        cell.font = Font(bold=True)
+        cell.fill = PatternFill("solid", fgColor="FFE8EEF7")
+        cell.alignment = Alignment(vertical="top")
+
+
+def build_review_workbook(
+    batch: BatchModel, rows: list[BatchRowModel], highlight: bool = True
+) -> Workbook:
+    """The tool's own table, every column the reviewer saw, one row per line."""
+    workbook = Workbook()
+    worksheet = workbook.active
+    worksheet.title = "Review"
+
+    headers = [header for header, _ in REVIEW_COLUMNS]
+    _write_header(worksheet, headers, 1)
+    for index, (_, width) in enumerate(REVIEW_COLUMNS, start=1):
+        worksheet.column_dimensions[get_column_letter(index)].width = width
+
+    for offset, row in enumerate(rows, start=2):
+        _, result = _unpack(row)
+        fill = STATUS_FILLS.get(result.status) if result else None
+        for column_index, value in enumerate(_review_values(row), start=1):
+            cell = worksheet.cell(row=offset, column=column_index)
+            cell.value = value
+            cell.alignment = Alignment(vertical="top", wrap_text=True)
+            if fill is not None and highlight:
+                cell.fill = fill
+
+    totals_row = len(rows) + 2
+    worksheet.cell(row=totals_row, column=1).value = "TOTAL"
+    worksheet.cell(row=totals_row, column=1).font = Font(bold=True)
+    for offset, total in (
+        (23, batch.metrics.totalPackages),
+        (24, batch.metrics.totalGrossWeight),
+        (25, batch.metrics.totalNetWeight),
+        (26, batch.metrics.totalValue),
+    ):
+        cell = worksheet.cell(row=totals_row, column=offset)
+        cell.value = total
+        cell.font = Font(bold=True)
+
+    worksheet.freeze_panes = "A2"
+    return workbook
+
+
+def build_softpak_workbook(
+    batch: BatchModel,
+    profile: ClientProfileModel,
+    rows: list[BatchRowModel],
+) -> Workbook:
+    """The Goenee line list: the client's own columns plus the codes we added.
+
+    Values come from the stored request rather than the source sheet, so a
+    shipment typed into the form exports exactly like an uploaded one.
+    """
+    workbook = Workbook()
+    worksheet = workbook.active
+    worksheet.title = "Blad1"
+
+    header_rows = profile.softpakHeaderRows or [
+        ["Government procedure", "10", " "],
+        ["Previous ", "00"],
+        ["Invoice N935", " "],
+        ["Currency", batch.currency],
+        ["Container"],
+    ]
+    for offset, header_row in enumerate(header_rows, start=1):
+        for column_index, value in enumerate(header_row, start=1):
+            worksheet.cell(row=offset, column=column_index).value = value
+
+    fields = [field for field in SOFTPAK_ORDER if field.value in profile.columnMapping]
+    if not fields:
+        fields = list(SOFTPAK_ORDER)
+
+    export_columns = profile.exportColumns or []
+    headers = [
+        profile.columnMapping.get(field.value, field.value) for field in fields
+    ] + [column.header for column in export_columns]
+
+    header_row_index = len(header_rows) + 2
+    _write_header(worksheet, headers, header_row_index)
+
+    data_row_index = header_row_index
+    for row in rows:
+        data_row_index += 1
+        request, result = _unpack(row)
+        values = [input_value(field, row.inputRow, request, profile) for field in fields]
+        if request is not None and InputField.CODE in fields:
+            # The code column carries what is actually being declared, not what
+            # the client sent: that is the whole point of the check.
+            values[fields.index(InputField.CODE)] = (
+                row.correctedCode or (result.resolvedCode if result else None) or request.code
+            )
+        values.extend(render_output_value(column.field, result) for column in export_columns)
+
+        fill = STATUS_FILLS.get(result.status) if result else None
+        for column_index, value in enumerate(values, start=1):
+            cell = worksheet.cell(row=data_row_index, column=column_index)
+            cell.value = value
+            if fill is not None and profile.highlightForReview:
+                cell.fill = fill
+
+    for index, header in enumerate(headers, start=1):
+        worksheet.column_dimensions[get_column_letter(index)].width = max(
+            12, min(38, len(str(header)) + 6)
+        )
+
+    if profile.totals.enabled:
+        totals = sheet_totals(_totals_lines(rows), profile)
+        if totals:
+            totals_row = data_row_index + 1
+            label = worksheet.cell(row=totals_row, column=1)
+            label.value = profile.totals.label
+            label.font = Font(bold=True)
+            for canonical, total in totals.items():
+                try:
+                    column_index = fields.index(InputField(canonical)) + 1
+                except ValueError:
+                    continue
+                cell = worksheet.cell(row=totals_row, column=column_index)
+                cell.value = total
+                cell.font = Font(bold=True)
+
+    return workbook
+
+
+def build_export_workbook(
+    export_format: ExportFormat,
+    batch: BatchModel,
+    profile: ClientProfileModel,
+    rows: list[BatchRowModel],
+) -> Workbook:
+    if export_format is ExportFormat.REVIEW:
+        return build_review_workbook(batch, rows, profile.highlightForReview)
+    if export_format is ExportFormat.SOFTPAK:
+        return build_softpak_workbook(batch, profile, rows)
+    raise ValueError(f"{export_format.name} cannot be built from stored rows.")
+
+
+def export_file_name(batch: BatchModel, export_format: ExportFormat) -> str:
+    stem = "".join(
+        character if character.isalnum() or character in "-_" else "_"
+        for character in batch.name
+    ) or "shipment"
+    return f"{stem}_{export_format.name.lower()}.xlsx"
+
+
+def unavailable_reason(export_format: ExportFormat) -> str | None:
+    """Why a declared format cannot be produced yet. None means it can."""
+    if export_format.implemented:
+        return None
+    return (
+        f"The {export_format.name.title()} import specification has not been received. "
+        "The layout is not guessed at: ask the client for their import template "
+        "and it becomes a profile change, not a code change."
+    )
+
+
+def output_field_choices() -> list[str]:
+    return [field.value for field in OutputField]

cbh/api/batch/services/excel/reader.py

@@ -1,11 +1,3 @@
-"""
-Reading a client's pack.
-
-Client files are not a fixed schema: each broker sends their own layout, so the
-reader works off the profile's column mapping and carries every column it does
-not recognise straight through to the output.
-"""
-
 import csv
 import io
 from dataclasses import dataclass, field
@@ -17,7 +9,7 @@ from openpyxl import Workbook, load_workbook
 from cbh.api.batch.dto import FileFormat
 from cbh.api.client.dto import InputField
 from cbh.api.client.models import ClientProfileModel
-from cbh.api.common.dto import TradeDirection
+from cbh.api.common.dto import CustomsProcedure, TradeDirection
 from cbh.api.common.utils import normalize_ws, parse_date, parse_number
 from cbh.api.importer.utils import normalize_header
 from cbh.api.linecheck.schemas import LineCheckRequest
@@ -25,13 +17,6 @@ from cbh.api.linecheck.schemas import LineCheckRequest
 
 @dataclass
 class ParsedRow:
-    """
-    One data row of a pack.
-
-    ``rowIndex`` is the worksheet row number, so results can be written back
-    exactly where they came from and the client's row order is preserved.
-    """
-
     rowIndex: int
     raw: dict[str, Any]
     request: LineCheckRequest | None = None
@@ -40,20 +25,48 @@ class ParsedRow:
 
 @dataclass
 class ParsedSheet:
-    """
-    A pack read into memory.
-    """
-
     headers: list[str]
     rows: list[ParsedRow]
     headerRowIndex: int
     sheetName: str | None = None
 
 
-def workbook_from_csv(payload: bytes) -> Workbook:
-    """
-    Load a CSV pack into a workbook so one writer serves both formats.
+def _workbook_from_legacy_xls(payload: bytes) -> Workbook:
+    """Read a BIFF `.xls` into the same Workbook the rest of the code expects.
+
+    Goenee's own line lists come out of JasperReports as legacy `.xls`, which
+    openpyxl cannot open at all.
     """
+    import xlrd
+
+    book = xlrd.open_workbook(file_contents=payload)
+    workbook = Workbook()
+    workbook.remove(workbook.active)
+
+    for sheet in book.sheets():
+        worksheet = workbook.create_sheet(title=(sheet.name or "Sheet1")[:31])
+        for index in range(sheet.nrows):
+            values = []
+            for position in range(sheet.ncols):
+                cell = sheet.cell(index, position)
+                if cell.ctype == xlrd.XL_CELL_DATE:
+                    values.append(xlrd.xldate_as_datetime(cell.value, book.datemode))
+                elif cell.ctype == xlrd.XL_CELL_EMPTY:
+                    values.append(None)
+                else:
+                    values.append(cell.value)
+            worksheet.append(values)
+
+    return workbook
+
+
+def load_pack_workbook(payload: bytes, file_name: str) -> Workbook:
+    if file_name.lower().endswith(".xls"):
+        return _workbook_from_legacy_xls(payload)
+
+    if FileFormat.from_name(file_name) is not FileFormat.CSV:
+        return load_workbook(io.BytesIO(payload), data_only=True)
+
     text = payload.decode("utf-8-sig", errors="replace")
     try:
         dialect = csv.Sniffer().sniff(text[:4096], delimiters=",;\t|")
@@ -67,44 +80,15 @@ def workbook_from_csv(payload: bytes) -> Workbook:
     return workbook
 
 
-def load_pack_workbook(payload: bytes, file_name: str) -> Workbook:
-    """
-    Load a pack, whichever of the two formats it arrived in.
-    """
-    if FileFormat.from_name(file_name) is FileFormat.CSV:
-        return workbook_from_csv(payload)
-    return load_workbook(io.BytesIO(payload), data_only=True)
-
-
 def read_headers(payload: bytes, file_name: str, header_row: int = 1) -> list[Any]:
-    """
-    Read the header row of a pack, for mapping suggestions.
-    """
-    workbook = load_pack_workbook(payload, file_name)
-    worksheet = workbook.active
+    worksheet = load_pack_workbook(payload, file_name).active
     for index, row in enumerate(worksheet.iter_rows(values_only=True), start=1):
         if index == header_row:
             return list(row)
     return []
 
 
-def _header_positions(headers: list[Any], profile: ClientProfileModel) -> dict[str, int]:
-    """
-    Map canonical fields onto column positions using the profile's mapping.
-    """
-    normalized = {normalize_header(header): index for index, header in enumerate(headers)}
-    positions: dict[str, int] = {}
-    for canonical, client_header in profile.columnMapping.items():
-        position = normalized.get(normalize_header(client_header))
-        if position is not None:
-            positions[canonical] = position
-    return positions
-
-
 def _cell(row: tuple, position: int | None) -> Any:
-    """
-    Read a cell, tolerating short rows.
-    """
     if position is None or position >= len(row):
         return None
     return row[position]
@@ -115,14 +99,10 @@ def parse_pack(
     file_name: str,
     profile: ClientProfileModel,
     check_date: date | None = None,
+    destination_country: str | None = None,
+    direction_default: TradeDirection | None = None,
+    procedure_default: CustomsProcedure | None = None,
 ) -> ParsedSheet:
-    """
-    Read a pack into line-check requests.
-
-    A row that cannot be read is kept with its errors rather than dropped: the
-    output file must come back with the same rows, in the same order, as the
-    one that went in.
-    """
     workbook = load_pack_workbook(payload, file_name)
     worksheet = (
         workbook[profile.sheetName]
@@ -136,7 +116,22 @@ def parse_pack(
         return ParsedSheet(headers=[], rows=[], headerRowIndex=profile.headerRow)
 
     headers = [normalize_ws(header) or "" for header in all_rows[header_index]]
-    positions = _header_positions(list(all_rows[header_index]), profile)
+
+    normalized = {
+        normalize_header(header): index
+        for index, header in enumerate(all_rows[header_index])
+    }
+    positions: dict[str, int] = {}
+    for canonical, client_header in profile.columnMapping.items():
+        # One client's packs are not spelled identically from month to month —
+        # the same column is "Cn" in one export and "Case No." in the next — so
+        # a mapping may list the spellings it accepts, best first.
+        for spelling in str(client_header).split("|"):
+            position = normalized.get(normalize_header(spelling))
+            if position is not None:
+                positions[canonical] = position
+                break
+
     defaults = profile.defaults
 
     rows: list[ParsedRow] = []
@@ -145,19 +140,34 @@ def parse_pack(
         if raw_row is None or all(cell is None or cell == "" for cell in raw_row):
             continue
 
-        raw = {
-            header: _cell(raw_row, index)
-            for index, header in enumerate(headers)
-            if header
-        }
-        parsed = ParsedRow(rowIndex=row_index, raw=raw)
+        parsed = ParsedRow(
+            rowIndex=row_index,
+            raw={
+                header: _cell(raw_row, index)
+                for index, header in enumerate(headers)
+                if header
+            },
+        )
 
         code = _cell(raw_row, positions.get(InputField.CODE.value))
         country = _cell(raw_row, positions.get(InputField.COUNTRY.value)) or defaults.country
+        destination = (
+            _cell(raw_row, positions.get(InputField.DESTINATION_COUNTRY.value))
+            or destination_country
+            or defaults.destinationCountry
+        )
         direction = (
             TradeDirection.parse(_cell(raw_row, positions.get(InputField.DIRECTION.value)))
+            or direction_default
             or defaults.direction
         )
+        procedure = (
+            CustomsProcedure.parse(
+                _cell(raw_row, positions.get(InputField.PROCEDURE.value))
+            )
+            or procedure_default
+            or defaults.procedure
+        )
         line_date = parse_date(_cell(raw_row, positions.get(InputField.DATE.value)))
         if line_date is None:
             line_date = check_date or (
@@ -173,8 +183,10 @@ def parse_pack(
 
         parsed.request = LineCheckRequest(
             code=str(code or "").strip(),
-            country=str(country or "").strip(),
+            originCountry=str(country or "").strip() or None,
+            destinationCountry=str(destination or "").strip() or None,
             direction=direction,
+            procedure=procedure,
             date=line_date,
             additionalCodes=[str(additional).strip()] if additional else [],
             description=normalize_ws(

cbh/api/batch/services/excel/writer.py

@@ -1,13 +1,3 @@
-"""
-Writing the output pack.
-
-The client gets their own file back: same sheet, same columns, same row order,
-with the checker's columns appended in the order their export profile fixes,
-and a totals row at the bottom — the standard check Judith runs on every pack.
-
-Rows that need a human are highlighted. They are never corrected.
-"""
-
 import io
 from typing import Any
 
@@ -16,56 +6,45 @@ from openpyxl.utils import get_column_letter
 from openpyxl.workbook import Workbook
 
 from cbh.api.batch.services.excel.reader import ParsedSheet, load_pack_workbook
-from cbh.api.client.dto import InputField, OutputField
+from cbh.api.batch.services.metrics.totals import TotalsInput, sheet_totals
+from cbh.api.client.dto import ExportLayout, InputField, OutputField
 from cbh.api.client.models import ClientProfileModel
+from cbh.api.importer.utils import normalize_header
 from cbh.api.linecheck.dto import LineStatus
 from cbh.api.linecheck.schemas import LineCheckResponse
-from cbh.api.linecheck.utils import (
-    render_candidates,
-    render_exemptions,
-    render_required_documents,
-    render_warnings,
-    status_label,
-)
-
-HEADER_FILL = PatternFill("solid", fgColor="FFE8EEF7")
-INVALID_FILL = PatternFill("solid", fgColor="FFF8CBCB")
-AMBIGUOUS_FILL = PatternFill("solid", fgColor="FFFDE9A9")
-EXPIRING_FILL = PatternFill("solid", fgColor="FFE2F0D9")
+from cbh.api.linecheck.utils import render_required_documents, render_warnings
 
-STATUS_FILLS = {
-    LineStatus.INVALID: INVALID_FILL,
-    LineStatus.AMBIGUOUS: AMBIGUOUS_FILL,
-    LineStatus.EXPIRING: EXPIRING_FILL,
-}
 
-MAX_CELL_LENGTH = 32000
-
-
-def _truncate(value: str) -> str:
-    """
-    Keep a cell inside Excel's length limit.
-    """
-    if len(value) <= MAX_CELL_LENGTH:
-        return value
-    return value[: MAX_CELL_LENGTH - 3] + "..."
+SOFTPAK_ORDER = [
+    InputField.CODE,
+    InputField.DESCRIPTION,
+    InputField.GROSS_WEIGHT,
+    InputField.NET_WEIGHT,
+    InputField.COUNTRY,
+    InputField.QUANTITY,
+    InputField.COLLI_TYPE,
+    InputField.PACKAGES,
+    InputField.REMARKS,
+    InputField.VALUE,
+]
 
 
 def render_output_value(field: OutputField, result: LineCheckResponse | None) -> Any:
-    """
-    Render one appended column for one row.
-    """
     if result is None:
         return ""
 
     renderers = {
         OutputField.RESOLVED_CODE: lambda: result.resolvedCode or "",
         OutputField.DOCUMENTS_REQUIRED: lambda: render_required_documents(result),
-        OutputField.DOCUMENTS_EXEMPTION: lambda: render_exemptions(result),
+        OutputField.DOCUMENTS_EXEMPTION: lambda: "; ".join(
+            document.code for document in result.documentsExemption
+        ),
         OutputField.WARNINGS: lambda: render_warnings(result),
-        OutputField.STATUS: lambda: status_label(result.status),
+        OutputField.STATUS: lambda: result.status.name.lower(),
         OutputField.CODE_DESCRIPTION: lambda: result.codeDescription or "",
-        OutputField.CANDIDATES: lambda: render_candidates(result),
+        OutputField.CANDIDATES: lambda: ", ".join(
+            candidate.code for candidate in result.candidates
+        ),
         OutputField.REPLACEMENT_CODES: lambda: ", ".join(result.replacementCodes),
         OutputField.EXPIRING_ON: lambda: (
             result.expiringOn.isoformat() if result.expiringOn else ""
@@ -76,34 +55,15 @@ def render_output_value(field: OutputField, result: LineCheckResponse | None) ->
             for measure in result.appliedMeasures
         ),
         OutputField.REFERENCE_AS_OF: lambda: result.referenceAsOf or "",
+        OutputField.YCODES: lambda: ",".join(
+            document.code for document in result.documentsExemption
+        ),
+        OutputField.CODE_CONFIDENCE: lambda: result.codeConfidence.name.lower(),
+        OutputField.DOCUMENT_CONFIDENCE: lambda: result.documentConfidence.name.lower(),
     }
-    return _truncate(str(renderers[field]()))
-
-
-def _totals(
-    sheet: ParsedSheet, profile: ClientProfileModel
-) -> dict[str, float]:
-    """
-    Sum the quantity columns the profile asks to total.
-    """
-    totals: dict[str, float] = {}
-    for canonical in profile.totals.sumFields:
-        try:
-            field = InputField(canonical)
-        except ValueError:
-            continue
-        total = 0.0
-        found = False
-        for row in sheet.rows:
-            if row.request is None:
-                continue
-            value = getattr(row.request, field.value, None)
-            if isinstance(value, (int, float)):
-                total += float(value)
-                found = True
-        if found:
-            totals[canonical] = round(total, 2)
-    return totals
+
+    value = str(renderers[field]())
+    return value if len(value) <= 32000 else value[:31997] + "..."
 
 
 def build_output_workbook(
@@ -113,12 +73,9 @@ def build_output_workbook(
     sheet: ParsedSheet,
     results: dict[int, LineCheckResponse],
 ) -> Workbook:
-    """
-    Append the checker's columns to the client's own workbook.
+    if profile.exportLayout is ExportLayout.SOFTPAK_SHEET:
+        return _build_softpak_workbook(profile, sheet, results)
 
-    The original file is reopened and written into rather than rebuilt, so the
-    client's formatting, extra columns and row order survive untouched.
-    """
     workbook = load_pack_workbook(payload, file_name)
     worksheet = (
         workbook[sheet.sheetName]
@@ -133,12 +90,18 @@ def build_output_workbook(
         cell = worksheet.cell(row=header_row, column=first_new_column + offset)
         cell.value = column.header
         cell.font = Font(bold=True)
-        cell.fill = HEADER_FILL
+        cell.fill = PatternFill("solid", fgColor="FFE8EEF7")
         cell.alignment = Alignment(vertical="top")
         worksheet.column_dimensions[
             get_column_letter(first_new_column + offset)
         ].width = 28
 
+    status_fills = {
+        LineStatus.INVALID: PatternFill("solid", fgColor="FFF8CBCB"),
+        LineStatus.AMBIGUOUS: PatternFill("solid", fgColor="FFFDE9A9"),
+        LineStatus.EXPIRING: PatternFill("solid", fgColor="FFE2F0D9"),
+    }
+
     last_data_row = header_row
     for row in sheet.rows:
         result = results.get(row.rowIndex)
@@ -151,14 +114,10 @@ def build_output_workbook(
 
         if not profile.highlightForReview or result is None:
             continue
-        fill = STATUS_FILLS.get(result.status)
+        fill = status_fills.get(result.status)
         if fill is None:
             continue
-        # Highlight the whole row: a broker scanning the pack should see the
-        # lines needing attention without reading the status column.
-        for column_index in range(
-            1, first_new_column + len(profile.exportColumns)
-        ):
+        for column_index in range(1, first_new_column + len(profile.exportColumns)):
             worksheet.cell(row=row.rowIndex, column=column_index).fill = fill
 
     if profile.totals.enabled:
@@ -167,22 +126,86 @@ def build_output_workbook(
     return workbook
 
 
+def _build_softpak_workbook(
+    profile: ClientProfileModel,
+    sheet: ParsedSheet,
+    results: dict[int, LineCheckResponse],
+) -> Workbook:
+    workbook = Workbook()
+    worksheet = workbook.active
+    worksheet.title = "Blad1"
+
+    for offset, header_row in enumerate(profile.softpakHeaderRows, start=1):
+        for column_index, value in enumerate(header_row, start=1):
+            worksheet.cell(row=offset, column=column_index).value = value
+
+    source_headers = [
+        profile.columnMapping[field.value]
+        for field in SOFTPAK_ORDER
+        if field.value in profile.columnMapping
+    ]
+    headers = source_headers + [column.header for column in profile.exportColumns]
+
+    header_row_index = max(profile.headerRow, len(profile.softpakHeaderRows) + 1)
+    for column_index, header in enumerate(headers, start=1):
+        cell = worksheet.cell(row=header_row_index, column=column_index)
+        cell.value = header
+        cell.font = Font(bold=True)
+        cell.fill = PatternFill("solid", fgColor="FFE8EEF7")
+
+    status_fills = {
+        LineStatus.INVALID: PatternFill("solid", fgColor="FFF8CBCB"),
+        LineStatus.AMBIGUOUS: PatternFill("solid", fgColor="FFFDE9A9"),
+        LineStatus.EXPIRING: PatternFill("solid", fgColor="FFE2F0D9"),
+    }
+
+    data_row_index = header_row_index
+    for row in sheet.rows:
+        data_row_index += 1
+        result = results.get(row.rowIndex)
+        by_header = {normalize_header(key): value for key, value in row.raw.items()}
+        values = [by_header.get(normalize_header(header)) for header in source_headers]
+        values.extend(
+            render_output_value(column.field, result) for column in profile.exportColumns
+        )
+        fill = status_fills.get(result.status) if result else None
+        for column_index, value in enumerate(values, start=1):
+            cell = worksheet.cell(row=data_row_index, column=column_index)
+            cell.value = value
+            if fill is not None and profile.highlightForReview:
+                cell.fill = fill
+
+    for index, header in enumerate(headers, start=1):
+        worksheet.column_dimensions[get_column_letter(index)].width = max(
+            12, min(38, len(str(header)) + 6)
+        )
+
+    if profile.totals.enabled:
+        _append_totals_row(worksheet, profile, sheet, data_row_index, source_headers)
+
+    return workbook
+
+
 def _append_totals_row(
     worksheet: Any,
     profile: ClientProfileModel,
     sheet: ParsedSheet,
     last_data_row: int,
+    sheet_headers: list[str] | None = None,
 ) -> None:
-    """
-    Write the totals row under the last data row.
-    """
-    totals = _totals(sheet, profile)
+    totals = sheet_totals(
+        [TotalsInput(raw=row.raw, request=row.request) for row in sheet.rows], profile
+    )
+
     if not totals:
         return
 
     totals_row = last_data_row + 1
-    normalized_headers = {
-        header: index + 1 for index, header in enumerate(sheet.headers)
+    header_columns = {
+        header: index + 1
+        for index, header in enumerate(
+            sheet.headers if sheet_headers is None else sheet_headers
+        )
     }
 
     label_cell = worksheet.cell(row=totals_row, column=1)
@@ -191,7 +214,7 @@ def _append_totals_row(
 
     for canonical, total in totals.items():
         client_header = profile.columnMapping.get(canonical)
-        column_index = normalized_headers.get(client_header) if client_header else None
+        column_index = header_columns.get(client_header) if client_header else None
         if column_index is None:
             continue
         cell = worksheet.cell(row=totals_row, column=column_index)
@@ -200,9 +223,6 @@ def _append_totals_row(
 
 
 def workbook_to_bytes(workbook: Workbook) -> bytes:
-    """
-    Serialize a workbook for storage or download.
-    """
     buffer = io.BytesIO()
     workbook.save(buffer)
     return buffer.getvalue()

cbh/api/batch/services/metrics/__init__.py

@@ -1,3 +1,14 @@
-from .metrics import GoldSetScore, read_gold_rows, score_against_gold, split_codes
+from .metrics import GoldSetScore, read_gold_rows, score_against_gold
+from .recompute import recompute_metrics
+from .totals import TotalsInput, input_value, sheet_totals, shipment_totals
 
-__all__ = ["GoldSetScore", "read_gold_rows", "score_against_gold", "split_codes"]
+__all__ = [
+    "GoldSetScore",
+    "TotalsInput",
+    "input_value",
+    "read_gold_rows",
+    "recompute_metrics",
+    "score_against_gold",
+    "sheet_totals",
+    "shipment_totals",
+]

cbh/api/batch/services/metrics/metrics.py

@@ -1,17 +1,3 @@
-"""
-Gold-set scoring.
-
-The acceptance criteria are stated per pack: percentage of correct codes,
-percentage of correct document codes (required and exemption scored
-separately), machine time, and human review time. This module computes the
-first three by comparing a processed pack against a gold file whose answers
-Judith has confirmed; review time is measured on the pack itself.
-
-Document codes are compared as sets, because the order they appear in carries
-no meaning. Codes are compared after normalisation so that formatting
-differences between the two files do not read as errors.
-"""
-
 from dataclasses import dataclass, field
 from typing import Any
 
@@ -21,10 +7,19 @@ from cbh.api.common.utils import normalize_ws
 from cbh.api.importer.utils import normalize_header
 from cbh.api.taric.utils import normalize_code
 
-# Header spellings accepted in a gold file for each column being scored.
 GOLD_COLUMNS: dict[str, tuple[str, ...]] = {
     "rowIndex": ("row", "rowindex", "regel", "line"),
-    "resolvedCode": ("resolved_code", "resolvedcode", "code", "goederencode", "taric"),
+    # `hs code` and `y-code` are the spellings Goenee's own line list uses, so a
+    # finished pack can be handed back as the gold file without being retyped.
+    "resolvedCode": (
+        "resolved_code",
+        "resolvedcode",
+        "code",
+        "hs code",
+        "hscode",
+        "goederencode",
+        "taric",
+    ),
     "documentsRequired": (
         "documents_required",
         "documentsrequired",
@@ -35,52 +30,37 @@ GOLD_COLUMNS: dict[str, tuple[str, ...]] = {
     "documentsExemption": (
         "documents_exemption",
         "documentsexemption",
+        "y-code",
+        "ycode",
+        "ycodes",
         "vrijstellingen",
         "exemption",
     ),
 }
 
-_SEPARATORS = (";", ",", "|", " ")
-
 
 def split_codes(value: Any) -> set[str]:
-    """
-    Split a document-code cell into a comparable set.
-
-    Either/or groups are written ``C400|Y900`` in the output; for scoring the
-    alternatives are individual codes, since presenting either is correct.
-    """
     text = normalize_ws(value)
     if not text:
         return set()
-    for separator in _SEPARATORS:
+    for separator in (";", ",", "|", " "):
         text = text.replace(separator, ";")
     return {part.strip().upper() for part in text.split(";") if part.strip()}
 
 
 @dataclass
 class FieldScore:
-    """
-    Score for one compared field.
-    """
-
     compared: int = 0
     correct: int = 0
     mismatches: list[dict[str, Any]] = field(default_factory=list)
 
     @property
     def accuracy(self) -> float | None:
-        """
-        Share of compared rows that matched, as a percentage.
-        """
         if not self.compared:
             return None
         return round(100.0 * self.correct / self.compared, 2)
 
     def record(self, row_index: int, expected: Any, actual: Any, matched: bool) -> None:
-        """
-        Record one comparison, keeping the first mismatches for inspection.
-        """
         self.compared += 1
         if matched:
             self.correct += 1
@@ -96,10 +76,6 @@ class FieldScore:
 
 @dataclass
 class GoldSetScore:
-    """
-    The accuracy half of the acceptance metrics for one pack.
-    """
-
     goldRows: int = 0
     matchedRows: int = 0
     unmatchedGoldRows: list[int] = field(default_factory=list)
@@ -109,9 +85,6 @@ class GoldSetScore:
     documentsExemption: FieldScore = field(default_factory=FieldScore)
 
     def to_dict(self) -> dict[str, Any]:
-        """
-        Render the score for the API.
-        """
         return {
             "goldRows": self.goldRows,
             "matchedRows": self.matchedRows,
@@ -126,15 +99,7 @@ class GoldSetScore:
 
 
 def read_gold_rows(payload: bytes, file_name: str) -> list[dict[str, Any]]:
-    """
-    Read a gold file into comparable rows.
-
-    Rows are matched on an explicit row number when the gold file carries one,
-    which is the only reliable key when the same code appears many times in a
-    pack.
-    """
-    workbook = load_pack_workbook(payload, file_name)
-    worksheet = workbook.active
+    worksheet = load_pack_workbook(payload, file_name).active
 
     rows = list(worksheet.iter_rows(values_only=True))
     if not rows:
@@ -160,7 +125,7 @@ def read_gold_rows(payload: bytes, file_name: str) -> list[dict[str, Any]]:
         if row is None or all(cell is None or cell == "" for cell in row):
             continue
 
-        def cell(name: str) -> Any:
+        def cell(name: str, row=row) -> Any:
             position = positions.get(name)
             return row[position] if position is not None and position < len(row) else None
 
@@ -183,13 +148,6 @@ def read_gold_rows(payload: bytes, file_name: str) -> list[dict[str, Any]]:
 def score_against_gold(
     rows: list[BatchRowModel], gold_rows: list[dict[str, Any]]
 ) -> GoldSetScore:
-    """
-    Compare a processed pack against a gold file.
-
-    Only rows present in both are scored; gold rows with no counterpart are
-    reported rather than counted as failures, since a row-numbering mismatch is
-    a problem with the comparison, not with the engine.
-    """
     score = GoldSetScore(goldRows=len(gold_rows))
     by_index = {row.rowIndex: row for row in rows}
 
@@ -202,6 +160,12 @@ def score_against_gold(
 
         result = actual_row.result or {}
         actual_code = normalize_code(result.get("resolvedCode"))
+        # A gold file written at 8 digits is confirming the subheading, not
+        # disputing the last two digits the engine resolved. Compare at the
+        # precision the gold file was written to, and no further.
+        gold_code = gold["resolvedCode"]
+        if gold_code and actual_code and len(gold_code) < len(actual_code):
+            actual_code = actual_code[: len(gold_code)]
         score.codes.record(
             gold["rowIndex"],
             gold["resolvedCode"],
@@ -210,7 +174,17 @@ def score_against_gold(
         )
 
         if gold["hasRequiredColumn"]:
-            actual_required = _required_codes(result)
+            actual_required = {
+                str(document.get("code", "")).upper()
+                for document in result.get("documentsRequired", [])
+                if document.get("code")
+            }
+            for group in result.get("documentsAlternatives", []):
+                actual_required.update(
+                    str(document.get("code", "")).upper()
+                    for document in group
+                    if document.get("code")
+                )
             score.documentsRequired.record(
                 gold["rowIndex"],
                 gold["documentsRequired"],
@@ -232,21 +206,3 @@ def score_against_gold(
             )
 
     return score
-
-
-def _required_codes(result: dict[str, Any]) -> set[str]:
-    """
-    Every obligation code on a result, alternatives flattened.
-    """
-    codes = {
-        str(document.get("code", "")).upper()
-        for document in result.get("documentsRequired", [])
-        if document.get("code")
-    }
-    for group in result.get("documentsAlternatives", []):
-        codes.update(
-            str(document.get("code", "")).upper()
-            for document in group
-            if document.get("code")
-        )
-    return codes

cbh/api/batch/services/metrics/recompute.py

@@ -0,0 +1,66 @@
+from datetime import datetime
+
+from cbh.api.batch.db_requests import get_batch_rows
+from cbh.api.batch.models import BatchMetrics, BatchModel
+from cbh.api.batch.services.metrics.totals import TotalsInput, shipment_totals
+from cbh.api.client.db_requests import get_profile
+from cbh.api.client.models import ClientProfileModel
+from cbh.api.common.db_requests import update_obj
+from cbh.api.linecheck.dto import ConfidenceLevel
+from cbh.api.linecheck.schemas import LineCheckRequest, LineCheckResponse
+
+
+async def recompute_metrics(
+    batch: BatchModel, profile: ClientProfileModel | None = None
+) -> BatchModel:
+    """Rebuild a shipment's header figures from the rows currently stored.
+
+    The upload route computes these once while it processes the pack. A
+    shipment built through the form changes one row at a time, so the same
+    figures have to be derived again from whatever is in the table now.
+    """
+    rows, _ = await get_batch_rows(batch.id)
+    profile = profile or await get_profile(batch.clientProfile.id)
+
+    status_counts: dict[str, int] = {}
+    code_counts: dict[str, int] = {}
+    document_counts: dict[str, int] = {}
+    lines: list[TotalsInput] = []
+
+    for row in rows:
+        status_counts[row.status] = status_counts.get(row.status, 0) + 1
+
+        if row.result:
+            result = LineCheckResponse(**row.result)
+            code_counts[result.codeConfidence.name] = (
+                code_counts.get(result.codeConfidence.name, 0) + 1
+            )
+            document_counts[result.documentConfidence.name] = (
+                document_counts.get(result.documentConfidence.name, 0) + 1
+            )
+
+        lines.append(
+            TotalsInput(
+                raw=row.inputRow,
+                request=LineCheckRequest(**row.request) if row.request else None,
+            )
+        )
+
+    total = len(rows) or 1
+    batch.metrics = BatchMetrics(
+        totalLines=len(rows),
+        uniqueLookups=batch.metrics.uniqueLookups,
+        machineSeconds=batch.metrics.machineSeconds,
+        reviewSeconds=batch.metrics.reviewSeconds,
+        statusCounts=status_counts,
+        extractionScore=round(100.0 * code_counts.get(ConfidenceLevel.GREEN.name, 0) / total, 1),
+        documentScore=round(
+            100.0 * document_counts.get(ConfidenceLevel.GREEN.name, 0) / total, 1
+        ),
+        codeConfidenceCounts=code_counts,
+        documentConfidenceCounts=document_counts,
+        **shipment_totals(lines, profile),
+    )
+    batch.datetimeUpdated = datetime.now()
+    await update_obj(BatchModel, batch.id, batch.to_mongo())
+    return batch

cbh/api/batch/services/metrics/totals.py

@@ -0,0 +1,151 @@
+from dataclasses import dataclass
+
+from cbh.api.client.dto import InputField
+from cbh.api.client.models import ClientProfileModel
+from cbh.api.common.utils import parse_number
+from cbh.api.importer.utils import normalize_header
+from cbh.api.linecheck.schemas import LineCheckRequest
+
+REQUEST_ATTRIBUTES: dict[InputField, str] = {
+    InputField.CODE: "code",
+    InputField.COUNTRY: "originCountry",
+    InputField.DESTINATION_COUNTRY: "destinationCountry",
+    InputField.DESCRIPTION: "description",
+    InputField.QUANTITY: "quantity",
+    InputField.PACKAGES: "packages",
+    InputField.GROSS_WEIGHT: "grossWeight",
+    InputField.NET_WEIGHT: "netWeight",
+    InputField.VALUE: "value",
+}
+
+
+@dataclass
+class TotalsInput:
+    """One line, as both the client wrote it and as the checker understood it."""
+
+    raw: dict
+    request: LineCheckRequest | None
+
+
+def input_value(
+    field: InputField,
+    raw: dict,
+    request: LineCheckRequest | None,
+    profile: ClientProfileModel,
+):
+    """The client's own value for a canonical field.
+
+    The source row wins, because it is what the client actually wrote; the
+    parsed request is the fallback for a line that was typed into the form and
+    so has no source row.
+    """
+    header = profile.columnMapping.get(field.value)
+    if header and raw:
+        by_header = {normalize_header(key): value for key, value in raw.items()}
+        value = by_header.get(normalize_header(header))
+        if value not in (None, ""):
+            return value
+
+    attribute = REQUEST_ATTRIBUTES.get(field)
+    if attribute and request is not None:
+        return getattr(request, attribute, None)
+    return None
+
+
+def _number(value) -> float | None:
+    """A source cell may be a number or the text of one; a CSV only ever has text."""
+    if isinstance(value, bool):
+        return None
+    if isinstance(value, (int, float)):
+        return float(value)
+    return parse_number(value)
+
+
+def _parcel_key(line: TotalsInput, profile: ClientProfileModel) -> str | None:
+    column = profile.totals.parcelKeyColumn
+    if not column or not line.raw:
+        return None
+    by_header = {normalize_header(key): value for key, value in line.raw.items()}
+    value = by_header.get(normalize_header(column))
+    return str(value) if value not in (None, "") else None
+
+
+def sum_field(
+    field: InputField, lines: list[TotalsInput], profile: ClientProfileModel
+) -> float | None:
+    """Add a field up over a pack, counting each physical package once.
+
+    A parcel manifest states weight per parcel and repeats it on every item
+    line of that parcel. Adding those rows up inflates the total, and the
+    total is exactly what the broker checks the shipment against, so a
+    parcel-level field is counted once per parcel key.
+    """
+    parcel_level = field.value in profile.totals.parcelLevelFields
+    total = 0.0
+    found = False
+    seen: set[str] = set()
+
+    for line in lines:
+        number = _number(input_value(field, line.raw, line.request, profile))
+        if number is None:
+            continue
+        if parcel_level:
+            key = _parcel_key(line, profile)
+            if key is not None:
+                if key in seen:
+                    continue
+                seen.add(key)
+        total += number
+        found = True
+
+    return round(total, 2) if found else None
+
+
+def count_parcels(lines: list[TotalsInput], profile: ClientProfileModel) -> int | None:
+    keys = {key for line in lines if (key := _parcel_key(line, profile)) is not None}
+    return len(keys) or None
+
+
+def total_packages(lines: list[TotalsInput], profile: ClientProfileModel) -> float:
+    """How many physical packages the shipment holds.
+
+    For a parcel manifest that is the number of distinct parcels, not the sum
+    of the per-item piece counts, which counts a two-item parcel twice.
+    """
+    if profile.totals.packagesFromParcelCount:
+        parcels = count_parcels(lines, profile)
+        if parcels is not None:
+            return float(parcels)
+    return sum_field(InputField.PACKAGES, lines, profile) or 0.0
+
+
+def shipment_totals(
+    lines: list[TotalsInput], profile: ClientProfileModel
+) -> dict[str, float]:
+    return {
+        "totalPackages": total_packages(lines, profile),
+        "totalGrossWeight": sum_field(InputField.GROSS_WEIGHT, lines, profile) or 0.0,
+        "totalNetWeight": sum_field(InputField.NET_WEIGHT, lines, profile) or 0.0,
+        "totalValue": sum_field(InputField.VALUE, lines, profile) or 0.0,
+    }
+
+
+def sheet_totals(
+    lines: list[TotalsInput], profile: ClientProfileModel
+) -> dict[str, float]:
+    """The figures for the TOTAL row written into an exported sheet."""
+    totals: dict[str, float] = {}
+    for canonical in profile.totals.sumFields:
+        try:
+            field = InputField(canonical)
+        except ValueError:
+            continue
+        if field is InputField.PACKAGES and profile.totals.packagesFromParcelCount:
+            parcels = count_parcels(lines, profile)
+            if parcels is not None:
+                totals[canonical] = float(parcels)
+                continue
+        total = sum_field(field, lines, profile)
+        if total is not None:
+            totals[canonical] = total
+    return totals

cbh/api/batch/services/processor.py

@@ -1,31 +1,32 @@
-"""
-The pack processor: read the client's file, run it through the line-check API,
-write the file back out.
-
-This layer is deliberately thin. It owns no customs logic — every decision
-comes from the line-check engine, so the Excel path and any future declaration
-system cannot drift apart.
-"""
-
 from datetime import date, datetime
+from pathlib import Path
 
-from cbh.api.batch.dto import BatchStatus, FileFormat
+from cbh.api.batch.dto import BatchSource, BatchStatus, FileFormat
 from cbh.api.batch.models import BatchMetrics, BatchModel, BatchRowModel, ClientProfileRef
 from cbh.api.batch.services.excel import (
+    ParsedSheet,
     build_output_workbook,
     parse_pack,
     workbook_to_bytes,
 )
-from cbh.api.batch.utils import read_file, result_file_name, store_file
+from cbh.api.batch.services.metrics.totals import TotalsInput, shipment_totals
+from cbh.api.batch.utils import result_file_name, store_file
 from cbh.api.client.models import ClientProfileModel
-from cbh.api.common.db_requests import delete_objs, insert_obj, insert_objs, update_obj
+from cbh.api.common.db_requests import (
+    delete_objs,
+    find_objs,
+    insert_obj,
+    insert_objs,
+    update_obj,
+)
+from cbh.api.common.dto import CustomsProcedure, TradeDirection
 from cbh.api.linecheck.dto import LineStatus
 from cbh.api.linecheck.schemas import LineCheckRequest, LineCheckResponse
-from cbh.api.linecheck.services.engine import check_lines
+from cbh.api.linecheck.services.engine import check_line_cached, check_lines, recount_confidence
+from cbh.api.rules.db_requests import get_active_rules, record_rule_hits
+from cbh.api.rules.services.apply import apply_rules, apply_rules_to_lines
 from cbh.api.taric.db_requests import get_reference_as_of
 
-DB_WRITE_CHUNK = 500
-
 
 async def create_batch(
     name: str,
@@ -33,22 +34,23 @@ async def create_batch(
     file_name: str,
     profile: ClientProfileModel,
     check_date: date | None = None,
+    destination_country: str | None = None,
+    direction: TradeDirection | None = None,
+    procedure: CustomsProcedure | None = None,
+    shipment_reference: str | None = None,
 ) -> BatchModel:
-    """
-    Store an uploaded pack and record it.
-
-    The source file is kept: a disputed line is settled by replaying the file
-    that was actually submitted.
-    """
-    stored = store_file(payload, file_name, subdirectory="source")
-
     batch = BatchModel(
         name=name,
         clientProfile=ClientProfileRef(id=profile.id, name=profile.name),
         fileFormat=FileFormat.from_name(file_name),
-        sourceFile=stored,
+        sourceFile=store_file(payload, file_name, subdirectory="source"),
         checkDate=check_date,
-        defaultDirection=profile.defaults.direction,
+        defaultDirection=direction or profile.defaults.direction,
+        defaultProcedure=procedure or profile.defaults.procedure,
+        destinationCountry=(
+            (destination_country or profile.defaults.destinationCountry or "").upper() or None
+        ),
+        shipmentReference=shipment_reference,
         status=BatchStatus.UPLOADED,
     )
 
@@ -65,14 +67,16 @@ async def create_batch(
 
 
 async def process_batch(batch: BatchModel, profile: ClientProfileModel) -> BatchModel:
-    """
-    Run a pack end to end and produce the output file.
-
-    Safe to repeat: the rows of a previous attempt are cleared first, so a
-    reprocessed pack is replaced rather than doubled.
-    """
-    payload = read_file(batch.sourceFile)
-    sheet = parse_pack(payload, batch.sourceFile.fileName, profile, batch.checkDate)
+    payload = Path(batch.sourceFile.storagePath).read_bytes()
+    sheet = parse_pack(
+        payload,
+        batch.sourceFile.fileName,
+        profile,
+        batch.checkDate,
+        batch.destinationCountry,
+        batch.defaultDirection,
+        batch.defaultProcedure,
+    )
 
     await delete_objs(BatchRowModel, {"batchId": batch.id})
 
@@ -81,6 +85,12 @@ async def process_batch(batch: BatchModel, profile: ClientProfileModel) -> Batch
     ]
     batch_result = await check_lines(requests, batch.checkDate)
 
+    reference_as_of = await get_reference_as_of()
+    batch_result.results = await apply_rules_to_lines(
+        requests, batch_result.results, batch.clientProfile.id, reference_as_of
+    )
+    recount_confidence(batch_result)
+
     results: dict[int, LineCheckResponse] = {
         result.rowIndex: result
         for result in batch_result.results
@@ -100,13 +110,21 @@ async def process_batch(batch: BatchModel, profile: ClientProfileModel) -> Batch
 
     batch.resultFile = result_file
     batch.status = BatchStatus.COMPLETED
-    batch.referenceAsOf = await get_reference_as_of()
+    batch.referenceAsOf = reference_as_of
     batch.metrics = BatchMetrics(
         totalLines=batch_result.totalLines,
         uniqueLookups=batch_result.uniqueLookups,
         machineSeconds=batch_result.machineSeconds,
         statusCounts=batch_result.statusCounts,
         reviewSeconds=batch.metrics.reviewSeconds,
+        extractionScore=batch_result.extractionScore,
+        documentScore=batch_result.documentScore,
+        codeConfidenceCounts=batch_result.codeConfidenceCounts,
+        documentConfidenceCounts=batch_result.documentConfidenceCounts,
+        **shipment_totals(
+            [TotalsInput(raw=row.raw, request=row.request) for row in sheet.rows],
+            profile,
+        ),
     )
     batch.datetimeUpdated = datetime.now()
 
@@ -114,14 +132,95 @@ async def process_batch(batch: BatchModel, profile: ClientProfileModel) -> Batch
     return batch
 
 
+async def create_manual_batch(
+    name: str,
+    profile: ClientProfileModel,
+    check_date: date | None = None,
+    destination_country: str | None = None,
+    direction: TradeDirection | None = None,
+    procedure: CustomsProcedure | None = None,
+    shipment_reference: str | None = None,
+    currency: str = "EUR",
+) -> BatchModel:
+    """Open an empty shipment for the form to fill.
+
+    It is COMPLETED from the start: there is no file to parse, and every line
+    is checked as it arrives, so the table is always current.
+    """
+    batch = BatchModel(
+        name=name,
+        clientProfile=ClientProfileRef(id=profile.id, name=profile.name),
+        source=BatchSource.MANUAL,
+        status=BatchStatus.COMPLETED,
+        sourceFile=None,
+        checkDate=check_date,
+        defaultDirection=direction or profile.defaults.direction,
+        defaultProcedure=procedure or profile.defaults.procedure,
+        destinationCountry=(
+            (destination_country or profile.defaults.destinationCountry or "").upper() or None
+        ),
+        shipmentReference=shipment_reference,
+        currency=currency,
+        referenceAsOf=await get_reference_as_of(),
+    )
+    await insert_obj(batch)
+    return batch
+
+
+async def check_and_store_row(
+    batch: BatchModel,
+    request: LineCheckRequest,
+    row_index: int,
+    existing: BatchRowModel | None = None,
+) -> BatchRowModel:
+    """Check one line and write it into the shipment's table.
+
+    The same path serves a new form line, a corrected code and a re-check, so
+    a manually added row is indistinguishable from an uploaded one downstream.
+    """
+    reference_as_of = await get_reference_as_of()
+    request.rowIndex = row_index
+    if request.date is None:
+        request.date = batch.checkDate
+
+    result = await check_line_cached(request, reference_as_of)
+    result.rowIndex = row_index
+
+    rules = await get_active_rules(batch.clientProfile.id)
+    if rules:
+        result = await apply_rules(request, result, rules, reference_as_of)
+        await record_rule_hits([rule.ruleId for rule in result.appliedRules])
+
+    row = existing or BatchRowModel(batchId=batch.id, rowIndex=row_index)
+    row.inputRow = {
+        key: value
+        for key, value in request.model_dump(mode="json").items()
+        if value is not None
+    }
+    row.request = request.model_dump(mode="json")
+    row.result = result.model_dump(mode="json")
+    row.status = result.status.name
+    row.needsReview = result.status.needs_review
+
+    if existing is None:
+        await insert_obj(row)
+    else:
+        await update_obj(BatchRowModel, row.id, row.to_mongo())
+    return row
+
+
+async def next_row_index(batch_id: str) -> int:
+    rows = await find_objs(
+        BatchRowModel, {"batchId": batch_id}, limit=1, sort=("rowIndex", -1)
+    )
+    return (rows[0].rowIndex + 1) if rows else 1
+
+
 async def _persist_rows(
     batch: BatchModel,
-    sheet,
+    sheet: ParsedSheet,
     results: dict[int, LineCheckResponse],
 ) -> None:
-    """
-    Store every row with the answer it received.
-    """
     rows: list[BatchRowModel] = []
     for row in sheet.rows:
         result = results.get(row.rowIndex)
@@ -130,7 +229,12 @@ async def _persist_rows(
             BatchRowModel(
                 batchId=batch.id,
                 rowIndex=row.rowIndex,
-                inputRow={key: _serializable(value) for key, value in row.raw.items()},
+                inputRow={
+                    key: value
+                    if value is None or isinstance(value, (str, int, float, bool))
+                    else str(value)
+                    for key, value in row.raw.items()
+                },
                 request=row.request.model_dump(mode="json") if row.request else {},
                 result=result.model_dump(mode="json") if result else {},
                 status=status.name,
@@ -138,14 +242,5 @@ async def _persist_rows(
             )
         )
 
-    for start in range(0, len(rows), DB_WRITE_CHUNK):
-        await insert_objs(rows[start : start + DB_WRITE_CHUNK])
-
-
-def _serializable(value):
-    """
-    Render a raw spreadsheet cell as something BSON can store.
-    """
-    if value is None or isinstance(value, (str, int, float, bool)):
-        return value
-    return str(value)
+    for start in range(0, len(rows), 500):
+        await insert_objs(rows[start : start + 500])

cbh/api/batch/utils.py

@@ -1,7 +1,3 @@
-"""
-Batch helpers: file storage and naming.
-"""
-
 import re
 import uuid
 from datetime import datetime
@@ -10,35 +6,13 @@ from pathlib import Path
 from cbh.api.common.models import FileRef
 from cbh.core.config import settings
 
-CONTENT_TYPES = {
-    ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
-    ".xlsm": "application/vnd.ms-excel.sheet.macroEnabled.12",
-    ".csv": "text/csv",
-    ".txt": "text/plain",
-}
-
 
 def safe_file_name(file_name: str) -> str:
-    """
-    Strip anything that could escape the storage directory.
-    """
     cleaned = re.sub(r"[^A-Za-z0-9._-]", "_", Path(file_name).name)
     return cleaned or "upload.xlsx"
 
 
-def content_type_for(file_name: str) -> str:
-    """
-    Content type for a stored pack.
-    """
-    return CONTENT_TYPES.get(Path(file_name).suffix.lower(), "application/octet-stream")
-
-
 def store_file(payload: bytes, file_name: str, subdirectory: str = "source") -> FileRef:
-    """
-    Write a pack to local storage and describe where it went.
-
-    Files are prefixed with a UUID so two packs of the same name never collide.
-    """
     directory = settings.BATCH_STORAGE_DIR / subdirectory
     directory.mkdir(parents=True, exist_ok=True)
 
@@ -49,25 +23,16 @@ def store_file(payload: bytes, file_name: str, subdirectory: str = "source") ->
     return FileRef(
         fileName=clean_name,
         storagePath=str(path),
-        contentType=content_type_for(clean_name),
+        contentType={
+            ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
+            ".xlsm": "application/vnd.ms-excel.sheet.macroEnabled.12",
+            ".csv": "text/csv",
+            ".txt": "text/plain",
+        }.get(path.suffix.lower(), "application/octet-stream"),
         sizeBytes=len(payload),
     )
 
 
-def read_file(file_ref: FileRef) -> bytes:
-    """
-    Read a stored pack back off disk.
-    """
-    return Path(file_ref.storagePath).read_bytes()
-
-
 def result_file_name(source_name: str) -> str:
-    """
-    Name of the output file for a pack.
-
-    Always .xlsx: the output carries highlighting and a totals row, which a CSV
-    cannot represent.
-    """
     stem = Path(safe_file_name(source_name)).stem
-    stamp = datetime.now().strftime("%Y%m%d")
-    return f"{stem}_checked_{stamp}.xlsx"
+    return f"{stem}_checked_{datetime.now().strftime('%Y%m%d')}.xlsx"

cbh/api/batch/views.py

@@ -1,43 +1,78 @@
-"""
-Batch views module.
-"""
-
 from datetime import date, datetime
 from pathlib import Path
 
-from fastapi import Depends, File, Form, HTTPException, Query, UploadFile
+from fastapi import File, Form, HTTPException, Query, UploadFile
 from fastapi.responses import FileResponse
+from pydantic import ValidationError
 
-from cbh.api.account.dto import AccountType
-from cbh.api.account.models import AccountModel
 from cbh.api.batch import batch_router
 from cbh.api.batch.db_requests import get_batch, get_batch_rows, get_batches
-from cbh.api.batch.dto import BatchStatus, ReviewStage
-from cbh.api.batch.models import BatchModel
-from cbh.api.batch.schemas import BatchResponse, BatchRowResponse, GoldSetScoreResponse
-from cbh.api.batch.services.metrics import read_gold_rows, score_against_gold
-from cbh.api.batch.services.processor import create_batch, process_batch
+from cbh.api.batch.dto import BatchSource, BatchStatus, ExportFormat, ReviewStage
+from cbh.api.batch.models import BatchModel, BatchRowModel
+from cbh.api.batch.schemas import (
+    BatchResponse,
+    BatchRowResponse,
+    ExportFormatResponse,
+    GoldSetScoreResponse,
+    ManualLineRequest,
+    ManualShipmentRequest,
+    ReviewDocuments,
+    ReviewHeader,
+    ReviewRow,
+    ReviewTableResponse,
+    RowReviewRequest,
+    ShipmentTotals,
+)
+from cbh.api.batch.services.excel import (
+    build_export_workbook,
+    export_file_name,
+    unavailable_reason,
+    workbook_to_bytes,
+)
+from cbh.api.batch.services.metrics import (
+    read_gold_rows,
+    recompute_metrics,
+    score_against_gold,
+)
+from cbh.api.batch.services.processor import (
+    check_and_store_row,
+    create_batch,
+    create_manual_batch,
+    next_row_index,
+    process_batch,
+)
+from cbh.api.batch.utils import store_file
 from cbh.api.client.db_requests import get_profile
-from cbh.api.common.db_requests import delete_obj, delete_objs, update_obj
+from cbh.api.client.dto import ExportLayout
+from cbh.api.client.models import ClientProfileModel
+from cbh.api.common.db_requests import (
+    delete_obj,
+    delete_objs,
+    find_objs,
+    insert_obj,
+    update_obj,
+)
+from cbh.api.common.dto import CustomsProcedure, Paging, TradeDirection
+from cbh.api.linecheck.dto import ConfidenceLevel, ReferenceSource, RowDecision
+from cbh.api.linecheck.schemas import LineCheckRequest, LineCheckResponse
+from cbh.api.rules.models import RuleModel
+from cbh.api.rules.schemas import MakeRuleRequest, RuleRequest, RuleResponse
+from cbh.api.rules.views import to_rule_response
+from cbh.api.taric.db_requests import get_reference_as_of
+from cbh.api.taric.utils import normalize_code
 from cbh.api.common.schemas import AllObjectsResponse
-from cbh.api.common.utils import to_paging
-from cbh.core.security import PermissionDependency
+from cbh.core.config import settings
 from cbh.core.wrappers import CbhResponseWrapper
 
-ANY_ACCOUNT = [AccountType.USER, AccountType.ADMIN, AccountType.OWNER, AccountType.SUPER_ADMIN]
-ADMIN_ONLY = [AccountType.ADMIN, AccountType.OWNER, AccountType.SUPER_ADMIN]
-
 
 def _to_response(batch: BatchModel) -> BatchResponse:
-    """
-    Map a pack onto its API representation.
-    """
     return BatchResponse(
         id=batch.id,
         name=batch.name,
         clientProfile=batch.clientProfile,
         status=batch.status,
-        sourceFileName=batch.sourceFile.fileName,
+        source=batch.source,
+        sourceFileName=batch.sourceFile.fileName if batch.sourceFile else None,
         resultFileName=batch.resultFile.fileName if batch.resultFile else None,
         checkDate=batch.checkDate,
         defaultDirection=batch.defaultDirection,
@@ -51,18 +86,31 @@ def _to_response(batch: BatchModel) -> BatchResponse:
     )
 
 
+async def _run(batch: BatchModel, profile: ClientProfileModel) -> BatchModel:
+    try:
+        return await process_batch(batch, profile)
+    except Exception as error:
+        batch.status = BatchStatus.FAILED
+        batch.errors.append(f"{type(error).__name__}: {error}")
+        batch.datetimeUpdated = datetime.now()
+        await update_obj(BatchModel, batch.id, batch.to_mongo())
+        raise HTTPException(
+            status_code=400, detail=f"Could not process the pack: {error}"
+        ) from error
+
+
 @batch_router.post("/upload")
 async def upload_batch(
     clientProfileId: str = Form(...),
     name: str | None = Form(default=None),
     checkDate: date | None = Form(default=None),
+    destinationCountry: str | None = Form(default=None),
+    direction: TradeDirection | None = Form(default=None),
+    procedure: CustomsProcedure | None = Form(default=None),
+    shipmentReference: str | None = Form(default=None),
     process: bool = Form(default=True),
     file: UploadFile = File(...),
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
 ) -> CbhResponseWrapper[BatchResponse]:
-    """
-    Upload a client's pack and, by default, process it straight away.
-    """
     payload = await file.read()
     if not payload:
         raise HTTPException(status_code=400, detail="Empty file.")
@@ -76,6 +124,10 @@ async def upload_batch(
         file_name=file_name,
         profile=profile,
         check_date=checkDate,
+        destination_country=destinationCountry,
+        direction=direction,
+        procedure=procedure,
+        shipment_reference=shipmentReference,
     )
 
     if process and batch.status is not BatchStatus.MAPPING_REQUIRED:
@@ -84,17 +136,95 @@ async def upload_batch(
     return CbhResponseWrapper(data=_to_response(batch))
 
 
-@batch_router.post("/{batchId}/process")
-async def process_existing_batch(
-    batchId: str,
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
+@batch_router.post("/manual")
+async def create_manual_shipment(
+    data: ManualShipmentRequest,
 ) -> CbhResponseWrapper[BatchResponse]:
+    """Route A of the specification: start a shipment the form will fill.
+
+    It produces the same review table as an upload, so the two input routes
+    converge on one product rather than two.
     """
-    Process, or reprocess, an uploaded pack.
+    profile = await get_profile(data.clientProfileId)
+    batch = await create_manual_batch(
+        name=data.name or f"Manual shipment {datetime.now():%Y-%m-%d %H:%M}",
+        profile=profile,
+        check_date=data.checkDate,
+        destination_country=data.destinationCountry,
+        direction=data.direction,
+        procedure=data.procedure,
+        shipment_reference=data.shipmentReference,
+        currency=data.currency,
+    )
+    return CbhResponseWrapper(data=_to_response(batch))
+
+
+@batch_router.post("/{batchId}/rows")
+async def add_manual_row(
+    batchId: str,
+    data: ManualLineRequest,
+) -> CbhResponseWrapper[ReviewRow]:
+    """Add one form line to the table, checked on arrival."""
+    batch = await get_batch(batchId)
+
+    request = LineCheckRequest(
+        code=data.code,
+        originCountry=data.originCountry,
+        destinationCountry=data.destinationCountry or batch.destinationCountry,
+        direction=data.direction or batch.defaultDirection,
+        procedure=data.procedure or batch.defaultProcedure,
+        date=data.date or batch.checkDate,
+        additionalCodes=data.additionalCodes,
+        description=data.description,
+        quantity=data.quantity,
+        packages=data.packages,
+        grossWeight=data.grossWeight,
+        netWeight=data.netWeight,
+        value=data.value,
+        clientProfileId=batch.clientProfile.id,
+    )
 
-    Reprocessing is how a pack is brought up to date after a reference-data
-    import: the answers are recomputed against the rules now loaded.
+    row = await check_and_store_row(batch, request, await next_row_index(batchId))
+    await recompute_metrics(batch)
+    return CbhResponseWrapper(data=_to_review_row(row))
+
+
+@batch_router.delete("/{batchId}/rows/{rowIndex}")
+async def delete_manual_row(
+    batchId: str,
+    rowIndex: int,
+) -> CbhResponseWrapper[BatchResponse]:
+    batch = await get_batch(batchId)
+    removed = await delete_objs(BatchRowModel, {"batchId": batchId, "rowIndex": rowIndex})
+    if not removed:
+        raise HTTPException(status_code=404, detail="Object not found.")
+    return CbhResponseWrapper(data=_to_response(await recompute_metrics(batch)))
+
+
+@batch_router.post("/{batchId}/recheck")
+async def recheck_shipment(batchId: str) -> CbhResponseWrapper[BatchResponse]:
+    """Re-run every row against the reference data and rules in force now.
+
+    This is what makes a rule retroactive: write the rule, press re-check, and
+    the lines it covers are answered again rather than edited by hand.
     """
+    batch = await get_batch(batchId)
+    rows, _ = await get_batch_rows(batchId)
+
+    for row in rows:
+        if not row.request:
+            continue
+        request = LineCheckRequest(**row.request)
+        if row.correctedCode:
+            request.code = row.correctedCode
+        await check_and_store_row(batch, request, row.rowIndex, existing=row)
+
+    batch.referenceAsOf = await get_reference_as_of()
+    return CbhResponseWrapper(data=_to_response(await recompute_metrics(batch)))
+
+
+@batch_router.post("/{batchId}/process")
+async def process_existing_batch(batchId: str) -> CbhResponseWrapper[BatchResponse]:
     batch = await get_batch(batchId)
     profile = await get_profile(batch.clientProfile.id)
 
@@ -108,48 +238,71 @@ async def process_existing_batch(
     return CbhResponseWrapper(data=_to_response(await _run(batch, profile)))
 
 
-async def _run(batch: BatchModel, profile) -> BatchModel:
-    """
-    Run a pack, recording a failure on the pack rather than losing it.
-    """
-    try:
-        return await process_batch(batch, profile)
-    except Exception as error:  # pylint: disable=W0718
-        batch.status = BatchStatus.FAILED
-        batch.errors.append(f"{type(error).__name__}: {error}")
-        batch.datetimeUpdated = datetime.now()
-        await update_obj(BatchModel, batch.id, batch.to_mongo())
-        raise HTTPException(
-            status_code=400, detail=f"Could not process the pack: {error}"
-        ) from error
-
-
 @batch_router.get("")
 async def list_batches(
     pageSize: int = Query(default=25, le=100),
     pageIndex: int = Query(default=0, ge=0),
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
 ) -> CbhResponseWrapper[AllObjectsResponse[BatchResponse]]:
-    """
-    List the processed packs, newest first.
-    """
     batches, total = await get_batches(pageSize, pageIndex)
     return CbhResponseWrapper(
         data=AllObjectsResponse(
-            paging=to_paging(pageSize, pageIndex, total),
+            paging=Paging(pageSize=pageSize, pageIndex=pageIndex, totalCount=total),
             data=[_to_response(batch) for batch in batches],
         )
     )
 
 
-@batch_router.get("/{batchId}")
-async def get_batch_details(
-    batchId: str,
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
-) -> CbhResponseWrapper[BatchResponse]:
-    """
-    Fetch one pack.
+EXPORT_FORMAT_DESCRIPTIONS: dict[ExportFormat, str] = {
+    ExportFormat.REVIEW: (
+        "The tool's own table: every column the reviewer saw, including "
+        "confidence reasons, warnings and the rules that fired."
+    ),
+    ExportFormat.SOURCE: (
+        "The client's uploaded file returned unchanged, with the checked "
+        "columns appended in the order the profile fixes."
+    ),
+    ExportFormat.SOFTPAK: "Goenee line list: the client's own columns plus the document codes.",
+    ExportFormat.SCOPE: "Scope line list (Fregat, Spare Cargo).",
+    ExportFormat.FITON: "Fiton line list (IP Handlers type).",
+}
+
+
+@batch_router.get("/export-formats")
+async def list_export_formats(
+    clientProfileId: str | None = Query(default=None),
+) -> CbhResponseWrapper[list[ExportFormatResponse]]:
+    """Fills the `Export: … ▾` dropdown.
+
+    Formats whose import specification has not been received are listed but
+    marked unavailable, so the gap is visible in the UI instead of being
+    papered over with a guessed layout.
     """
+    default_format = None
+    if clientProfileId:
+        profile = await get_profile(clientProfileId)
+        default_format = (
+            ExportFormat.SOFTPAK
+            if profile.exportLayout is ExportLayout.SOFTPAK_SHEET
+            else ExportFormat.SOURCE
+        )
+
+    return CbhResponseWrapper(
+        data=[
+            ExportFormatResponse(
+                format=export_format,
+                name=export_format.name.title(),
+                description=EXPORT_FORMAT_DESCRIPTIONS[export_format],
+                available=export_format.implemented,
+                unavailableReason=unavailable_reason(export_format),
+                isProfileDefault=export_format is default_format,
+            )
+            for export_format in ExportFormat
+        ]
+    )
+
+
+@batch_router.get("/{batchId}")
+async def get_batch_details(batchId: str) -> CbhResponseWrapper[BatchResponse]:
     return CbhResponseWrapper(data=_to_response(await get_batch(batchId)))
 
 
@@ -159,18 +312,11 @@ async def list_batch_rows(
     needsReviewOnly: bool = Query(default=False),
     pageSize: int = Query(default=100, le=1000),
     pageIndex: int = Query(default=0, ge=0),
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
 ) -> CbhResponseWrapper[AllObjectsResponse[BatchRowResponse]]:
-    """
-    Read the rows of a pack.
-
-    ``needsReviewOnly`` gives the broker's working list: the lines the machine
-    would not answer for.
-    """
     rows, total = await get_batch_rows(batchId, pageSize, pageIndex, needsReviewOnly)
     return CbhResponseWrapper(
         data=AllObjectsResponse(
-            paging=to_paging(pageSize, pageIndex, total),
+            paging=Paging(pageSize=pageSize, pageIndex=pageIndex, totalCount=total),
             data=[
                 BatchRowResponse(
                     rowIndex=row.rowIndex,
@@ -185,17 +331,306 @@ async def list_batch_rows(
     )
 
 
-@batch_router.get("/{batchId}/download")
-async def download_batch_result(
+def _to_review_row(row: BatchRowModel) -> ReviewRow:
+    result = LineCheckResponse(**row.result) if row.result else None
+    request = LineCheckRequest(**row.request) if row.request else None
+
+    if result is None:
+        documents = ReviewDocuments(label="Not checked")
+    elif result.noneRequired:
+        documents = ReviewDocuments(noneRequired=True, label="No document required")
+    elif result.documentsRequired or result.documentsAlternatives:
+        documents = ReviewDocuments(
+            required=result.documentsRequired,
+            alternatives=result.documentsAlternatives,
+            exemption=result.documentsExemption,
+            suspended=result.documentsSuspended,
+            label="; ".join(
+                [document.code for document in result.documentsRequired]
+                + [
+                    "|".join(document.code for document in group)
+                    for group in result.documentsAlternatives
+                ]
+            ),
+        )
+    elif result.documentsExemption:
+        documents = ReviewDocuments(
+            exemption=result.documentsExemption,
+            suspended=result.documentsSuspended,
+            label=", ".join(document.code for document in result.documentsExemption),
+        )
+    elif result.documentsSuspended:
+        # Nothing to write in Box 44 here, but the line is not settled: the
+        # label says which obligations the procedure carried forward, so the
+        # empty cell cannot be read as "nothing was owed".
+        documents = ReviewDocuments(
+            suspended=result.documentsSuspended,
+            label=(
+                "Suspended: "
+                + ", ".join(document.code for document in result.documentsSuspended)
+            ),
+        )
+    else:
+        documents = ReviewDocuments(label="Not established")
+
+    return ReviewRow(
+        rowIndex=row.rowIndex,
+        description=request.description if request else None,
+        inputCode=result.inputCode if result else None,
+        proposedCode=row.correctedCode or (result.resolvedCode if result else None),
+        formattedCode=result.formattedCode if result else None,
+        codeDescription=result.codeDescription if result else None,
+        candidates=[candidate.code for candidate in result.candidates] if result else [],
+        replacementCodes=result.replacementCodes if result else [],
+        originCountry=result.originCountry if result else None,
+        destinationCountry=result.destinationCountry if result else None,
+        measureCountry=result.measureCountry if result else None,
+        direction=result.direction if result else None,
+        procedure=result.procedure if result else None,
+        codeConfidence=result.codeConfidence if result else ConfidenceLevel.RED,
+        codeConfidenceReason=result.codeConfidenceReason if result else "Row was not checked.",
+        documentConfidence=result.documentConfidence if result else ConfidenceLevel.RED,
+        documentConfidenceReason=(
+            result.documentConfidenceReason if result else "Row was not checked."
+        ),
+        source=result.source if result else ReferenceSource.NONE,
+        documents=documents,
+        warnings=result.warnings if result else [],
+        appliedRules=result.appliedRules if result else [],
+        packages=request.packages if request else None,
+        grossWeight=request.grossWeight if request else None,
+        netWeight=request.netWeight if request else None,
+        value=request.value if request else None,
+        decision=row.decision,
+        correctedCode=row.correctedCode,
+        cbamPermitNumber=row.cbamPermitNumber,
+        eudrReference=row.eudrReference,
+        licenceNumbers=row.licenceNumbers,
+        reviewNote=row.reviewNote,
+        status=row.status,
+        needsReview=row.needsReview,
+    )
+
+
+@batch_router.get("/{batchId}/review")
+async def get_review_table(
     batchId: str,
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
-) -> FileResponse:
+    needsReviewOnly: bool = Query(default=False),
+    confidence: ConfidenceLevel | None = Query(default=None),
+    pageSize: int = Query(default=100, le=1000),
+    pageIndex: int = Query(default=0, ge=0),
+) -> CbhResponseWrapper[ReviewTableResponse]:
+    batch = await get_batch(batchId)
+    rows, total = await get_batch_rows(batchId, pageSize, pageIndex, needsReviewOnly)
+    review_rows = [_to_review_row(row) for row in rows]
+    if confidence is not None:
+        review_rows = [
+            row
+            for row in review_rows
+            if confidence in (row.codeConfidence, row.documentConfidence)
+        ]
+
+    origins = await settings.DB_CLIENT.batchrows.distinct(
+        "request.originCountry", {"batchId": batchId}
+    )
+
+    return CbhResponseWrapper(
+        data=ReviewTableResponse(
+            header=ReviewHeader(
+                batchId=batch.id,
+                shipmentReference=batch.shipmentReference,
+                name=batch.name,
+                clientProfile=batch.clientProfile,
+                status=batch.status,
+                source=batch.source,
+                direction=batch.defaultDirection,
+                procedure=batch.defaultProcedure,
+                originCountries=sorted(origin for origin in origins if origin),
+                destinationCountry=batch.destinationCountry,
+                checkDate=batch.checkDate,
+                referenceAsOf=batch.referenceAsOf,
+                totals=ShipmentTotals(
+                    packages=batch.metrics.totalPackages,
+                    grossWeight=batch.metrics.totalGrossWeight,
+                    netWeight=batch.metrics.totalNetWeight,
+                    value=batch.metrics.totalValue,
+                    currency=batch.currency,
+                ),
+                totalLines=batch.metrics.totalLines,
+                extractionScore=batch.metrics.extractionScore,
+                documentScore=batch.metrics.documentScore,
+                codeConfidenceCounts=batch.metrics.codeConfidenceCounts,
+                documentConfidenceCounts=batch.metrics.documentConfidenceCounts,
+                linesNeedingReview=batch.metrics.lines_needing_review,
+            ),
+            paging=Paging(pageSize=pageSize, pageIndex=pageIndex, totalCount=total),
+            rows=review_rows,
+        )
+    )
+
+
+@batch_router.patch("/{batchId}/rows/{rowIndex}")
+async def review_row(
+    batchId: str,
+    rowIndex: int,
+    data: RowReviewRequest,
+) -> CbhResponseWrapper[ReviewRow]:
+    batch = await get_batch(batchId)
+    matches = await find_objs(
+        BatchRowModel, {"batchId": batchId, "rowIndex": rowIndex}, limit=1
+    )
+    if not matches:
+        raise HTTPException(status_code=404, detail="Object not found.")
+    row = matches[0]
+
+    if data.correctedCode and data.correctedCode != row.correctedCode:
+        request = LineCheckRequest(**row.request) if row.request else None
+        if request is None:
+            raise HTTPException(
+                status_code=400, detail="This row has no request to re-check."
+            )
+        request.code = data.correctedCode
+        row = await check_and_store_row(batch, request, rowIndex, existing=row)
+        row.correctedCode = data.correctedCode
+        row.decision = RowDecision.CORRECTED
+
+    if data.decision is not None:
+        row.decision = data.decision
+    if data.cbamPermitNumber is not None:
+        row.cbamPermitNumber = data.cbamPermitNumber
+    if data.eudrReference is not None:
+        row.eudrReference = data.eudrReference
+    if data.licenceNumbers is not None:
+        row.licenceNumbers = data.licenceNumbers
+    if data.reviewNote is not None:
+        row.reviewNote = data.reviewNote
+    row.datetimeReviewed = datetime.now()
+
+    await update_obj(BatchRowModel, row.id, row.to_mongo())
+    await recompute_metrics(batch)
+    return CbhResponseWrapper(data=_to_review_row(row))
+
+
+@batch_router.post("/{batchId}/rows/{rowIndex}/make-rule")
+async def make_rule_from_row(
+    batchId: str,
+    rowIndex: int,
+    data: MakeRuleRequest,
+) -> CbhResponseWrapper[RuleResponse]:
+    """The `Make a rule` action on a review row.
+
+    Conditions the broker does not spell out are taken from the row in front
+    of them, so pressing the button on a line produces a rule that matches
+    that line and the ones like it.
     """
-    Download the checked pack.
+    batch = await get_batch(batchId)
+    matches = await find_objs(
+        BatchRowModel, {"batchId": batchId, "rowIndex": rowIndex}, limit=1
+    )
+    if not matches:
+        raise HTTPException(status_code=404, detail="Object not found.")
+
+    row = matches[0]
+    if not row.request:
+        raise HTTPException(
+            status_code=400, detail="This row has no request to build a rule from."
+        )
+
+    request = LineCheckRequest(**row.request)
+    result = LineCheckResponse(**row.result) if row.result else None
+    code = row.correctedCode or (result.resolvedCode if result else None) or str(request.code)
+    normalized = normalize_code(code) or code
+
+    prefix = None
+    if data.useCodePrefix:
+        length = max(2, min(data.codePrefixLength, len(normalized)))
+        prefix = normalized[:length]
+
+    try:
+        rule_request = RuleRequest(
+            name=data.name,
+            description=data.description,
+            scope=data.scope,
+            clientProfileId=batch.clientProfile.id,
+            codePrefix=prefix,
+            originCountry=request.originCountry if data.useOriginCountry else None,
+            destinationCountry=(
+                request.destinationCountry if data.useDestinationCountry else None
+            ),
+            direction=request.direction if data.useDirection else None,
+            descriptionContains=data.descriptionContains,
+            action=data.action,
+            pinnedCode=data.pinnedCode,
+            documentCodes=data.documentCodes,
+            note=data.note,
+            createdFromBatchId=batchId,
+            createdFromRowIndex=rowIndex,
+            createdBy=data.createdBy,
+        )
+    except ValidationError as error:
+        raise HTTPException(status_code=400, detail=str(error)) from error
+
+    rule = RuleModel(**rule_request.model_dump())
+    await insert_obj(rule)
+    return CbhResponseWrapper(data=to_rule_response(rule))
+
+
+@batch_router.get("/{batchId}/export")
+async def export_batch(
+    batchId: str,
+    format: ExportFormat | None = Query(default=None),
+) -> FileResponse:
+    """One export, per tenant, chosen at download time.
+
+    Without a format the client profile's own layout is used. REVIEW and
+    SOFTPAK are built from the stored rows, so a shipment typed into the form
+    exports exactly like an uploaded one.
     """
+    batch = await get_batch(batchId)
+    profile = await get_profile(batch.clientProfile.id)
+
+    if format is None:
+        format = (
+            ExportFormat.SOFTPAK
+            if profile.exportLayout is ExportLayout.SOFTPAK_SHEET
+            else ExportFormat.SOURCE
+        )
+
+    reason = unavailable_reason(format)
+    if reason:
+        raise HTTPException(status_code=501, detail=reason)
+
+    if format is ExportFormat.SOURCE:
+        return await download_batch_result(batchId)
+
+    rows, _ = await get_batch_rows(batchId)
+    if not rows:
+        raise HTTPException(status_code=400, detail="This shipment has no lines yet.")
+
+    workbook = build_export_workbook(format, batch, profile, rows)
+    file_ref = store_file(
+        workbook_to_bytes(workbook), export_file_name(batch, format), subdirectory="export"
+    )
+    return FileResponse(
+        path=file_ref.storagePath,
+        filename=file_ref.fileName,
+        media_type=file_ref.contentType,
+    )
+
+
+@batch_router.get("/{batchId}/download")
+async def download_batch_result(batchId: str) -> FileResponse:
     batch = await get_batch(batchId)
     if not batch.resultFile:
-        raise HTTPException(status_code=404, detail="This pack has not been processed yet.")
+        raise HTTPException(
+            status_code=404,
+            detail=(
+                "This shipment has no source-file result. It was built through the "
+                "form, or has not been processed yet. Use /export?format=review."
+            )
+            if batch.source is BatchSource.MANUAL
+            else "This pack has not been processed yet.",
+        )
 
     path = Path(batch.resultFile.storagePath)
     if not path.exists():
@@ -209,16 +644,7 @@ async def download_batch_result(
 
 
 @batch_router.post("/{batchId}/review/start")
-async def start_review(
-    batchId: str,
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
-) -> CbhResponseWrapper[BatchResponse]:
-    """
-    Mark the start of human review.
-
-    Review time is the metric the pilot is judged on, so it is measured here
-    rather than estimated afterwards.
-    """
+async def start_review(batchId: str) -> CbhResponseWrapper[BatchResponse]:
     batch = await get_batch(batchId)
     batch.reviewStage = ReviewStage.IN_REVIEW
     batch.reviewStartedAt = datetime.now()
@@ -230,13 +656,7 @@ async def start_review(
 
 
 @batch_router.post("/{batchId}/review/finish")
-async def finish_review(
-    batchId: str,
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
-) -> CbhResponseWrapper[BatchResponse]:
-    """
-    Mark the end of human review and record how long it took.
-    """
+async def finish_review(batchId: str) -> CbhResponseWrapper[BatchResponse]:
     batch = await get_batch(batchId)
     if not batch.reviewStartedAt:
         raise HTTPException(status_code=400, detail="Review was never started.")
@@ -255,15 +675,7 @@ async def finish_review(
 async def score_batch_against_gold(
     batchId: str,
     file: UploadFile = File(...),
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
 ) -> CbhResponseWrapper[GoldSetScoreResponse]:
-    """
-    Score a processed pack against a gold file confirmed by the client.
-
-    Returns the acceptance metrics for that pack: correct codes, correct
-    document codes (required and exemption scored separately), machine time and
-    review time.
-    """
     batch = await get_batch(batchId)
     if batch.status is not BatchStatus.COMPLETED:
         raise HTTPException(status_code=400, detail="This pack has not been processed yet.")
@@ -291,18 +703,7 @@ async def score_batch_against_gold(
 
 
 @batch_router.delete("/{batchId}")
-async def delete_batch(
-    batchId: str,
-    account: AccountModel = Depends(PermissionDependency(ADMIN_ONLY)),
-) -> CbhResponseWrapper[BatchResponse]:
-    """
-    Delete a pack and its rows.
-
-    The stored files are left on disk: they are the evidence behind answers
-    already given to the client.
-    """
-    from cbh.api.batch.models import BatchRowModel
-
+async def delete_batch(batchId: str) -> CbhResponseWrapper[BatchResponse]:
     batch = await delete_obj(BatchModel, batchId)
     await delete_objs(BatchRowModel, {"batchId": batchId})
     return CbhResponseWrapper(data=_to_response(batch))

cbh/api/client/__init__.py

@@ -1,10 +1,3 @@
-"""
-Client module initialization.
-
-This module defines the FastAPI router for client profiles: the per-client
-column mapping and export profile used to read and write their files.
-"""
-
 from fastapi import APIRouter
 
 client_router = APIRouter(

cbh/api/client/db_requests.py

@@ -1,33 +1,35 @@
-"""
-Client profile database requests.
-"""
-
-from cbh.api.client.models import ClientProfileModel, goenee_profile
-from cbh.api.common.db_requests import count_objs, find_objs, get_obj_by_id, insert_obj
+from cbh.api.client.models import (
+    ClientProfileModel,
+    efg_henx_transit_profile,
+    goenee_export_line_list_profile,
+    goenee_softpak_profile,
+    gstar_ecom_profile,
+)
+from cbh.api.common.db_requests import find_objs, get_obj_by_id, insert_objs
 
 
 async def get_profile(profile_id: str) -> ClientProfileModel:
-    """
-    Fetch a client profile, 404 when it does not exist.
-    """
     return await get_obj_by_id(ClientProfileModel, profile_id)
 
 
-async def get_profile_by_name(name: str) -> ClientProfileModel | None:
-    """
-    Fetch a client profile by name.
-    """
-    profiles = await find_objs(ClientProfileModel, {"name": name}, limit=1)
-    return profiles[0] if profiles else None
-
-
 async def ensure_default_profiles() -> None:
-    """
-    Seed the Goenee profile on an empty database.
+    """Add any built-in profile that is not there yet, by name.
 
-    Goenee is the first client, so their profile ships with the application
-    rather than having to be typed in before the first pack can be processed.
+    Bailing out as soon as one profile exists meant a profile added later never
+    reached a database that had already been started once.
     """
-    if await count_objs(ClientProfileModel) > 0:
-        return
-    await insert_obj(goenee_profile())
+    defaults = [
+        goenee_softpak_profile(),
+        gstar_ecom_profile(),
+        goenee_export_line_list_profile(),
+        efg_henx_transit_profile(),
+    ]
+    existing = {
+        profile.name
+        for profile in await find_objs(
+            ClientProfileModel, {"name": {"$in": [default.name for default in defaults]}}
+        )
+    }
+    missing = [default for default in defaults if default.name not in existing]
+    if missing:
+        await insert_objs(missing)

cbh/api/client/dto.py

@@ -1,58 +1,34 @@
-"""
-Client profile DTOs.
-"""
+from enum import Enum, IntEnum
 
-from enum import Enum
-
-
-class ExportProfileType(Enum):
-    """
-    Which declaration system the output file is shaped for.
-    """
 
+class ExportProfileType(IntEnum):
     SOFTPAK = 1
     GENERIC = 2
 
 
 class InputField(Enum):
-    """
-    The canonical fields a client column can be mapped onto.
-
-    Client files arrive in the client's own layout, so the mapping is
-    configured once per client and reused for every pack.
-    """
-
     CODE = "code"
     COUNTRY = "country"
+    DESTINATION_COUNTRY = "destinationCountry"
     DIRECTION = "direction"
+    PROCEDURE = "procedure"
     DATE = "date"
     ADDITIONAL_CODE = "additionalCode"
     DESCRIPTION = "description"
     QUANTITY = "quantity"
     PACKAGES = "packages"
+    COLLI_TYPE = "colliType"
     GROSS_WEIGHT = "grossWeight"
     NET_WEIGHT = "netWeight"
     VALUE = "value"
+    REMARKS = "remarks"
 
     @property
     def required(self) -> bool:
-        """
-        Whether a pack cannot be processed without this field.
-
-        Direction and date have profile-level defaults, so only the code and
-        the country have to come from the file.
-        """
         return self in (InputField.CODE, InputField.COUNTRY)
 
 
 class OutputField(Enum):
-    """
-    The columns the checker appends to the client's file.
-
-    The first five are the contract agreed for the Softpak template; the rest
-    are optional detail a reviewer can switch on.
-    """
-
     RESOLVED_CODE = "resolved_code"
     DOCUMENTS_REQUIRED = "documents_required"
     DOCUMENTS_EXEMPTION = "documents_exemption"
@@ -66,3 +42,12 @@ class OutputField(Enum):
     NONE_REQUIRED = "none_required"
     APPLIED_MEASURES = "applied_measures"
     REFERENCE_AS_OF = "reference_as_of"
+
+    YCODES = "ycodes"
+    CODE_CONFIDENCE = "code_confidence"
+    DOCUMENT_CONFIDENCE = "document_confidence"
+
+
+class ExportLayout(Enum):
+    APPEND_COLUMNS = 1
+    SOFTPAK_SHEET = 2

cbh/api/client/models.py

@@ -1,43 +1,26 @@
-"""
-Client profile models.
-"""
-
 from datetime import datetime
 
 from pydantic import Field
 
-from cbh.api.client.dto import ExportProfileType, InputField, OutputField
-from cbh.api.common.dto import TradeDirection
-from cbh.core.database import MongoBaseModel, MongoBaseShortenModel, MongoSerializableModel
+from cbh.api.client.dto import ExportLayout, ExportProfileType, InputField, OutputField
+from cbh.api.common.dto import CustomsProcedure, TradeDirection
+from cbh.core.database import MongoBaseModel, MongoSerializableModel
 
 
 class ExportColumn(MongoSerializableModel):
-    """
-    One appended output column: what to write and what to call it.
-    """
-
     field: OutputField
     header: str
 
 
 class ProfileDefaults(MongoSerializableModel):
-    """
-    What to assume when the client's file does not say.
-
-    A pack with no date column is checked against today, which is what a broker
-    filing today means by an undated line.
-    """
-
     direction: TradeDirection = TradeDirection.IMPORT
+    procedure: CustomsProcedure = CustomsProcedure.RELEASE_FOR_FREE_CIRCULATION
     country: str | None = None
+    destinationCountry: str | None = None
     useTodayWhenDateMissing: bool = True
 
 
 class TotalsConfig(MongoSerializableModel):
-    """
-    The totals row Judith checks at the bottom of every pack.
-    """
-
     enabled: bool = True
     label: str = "TOTAL"
     sumFields: list[str] = Field(
@@ -48,11 +31,16 @@ class TotalsConfig(MongoSerializableModel):
         ]
     )
 
+    # A parcel manifest carries one row per item but states weight and piece
+    # count per parcel, repeated on every row of that parcel. Adding the rows
+    # up inflates the shipment totals, which is the figure the broker checks
+    # the consignment against, so those fields are counted once per parcel.
+    parcelKeyColumn: str | None = None
+    parcelLevelFields: list[str] = Field(default_factory=list)
+    packagesFromParcelCount: bool = False
+
 
 def default_export_columns() -> list[ExportColumn]:
-    """
-    The agreed set of appended columns, in order.
-    """
     return [
         ExportColumn(field=OutputField.RESOLVED_CODE, header="resolved_code"),
         ExportColumn(field=OutputField.DOCUMENTS_REQUIRED, header="documents_required"),
@@ -63,14 +51,6 @@ def default_export_columns() -> list[ExportColumn]:
 
 
 class ClientProfileModel(MongoBaseModel):
-    """
-    How one client's files are read and written.
-
-    ``columnMapping`` maps a canonical field onto the header used in that
-    client's spreadsheets; ``exportColumns`` fixes the order of the columns
-    appended for their declaration system.
-    """
-
     name: str
     description: str | None = None
     exportProfile: ExportProfileType = ExportProfileType.SOFTPAK
@@ -83,20 +63,13 @@ class ClientProfileModel(MongoBaseModel):
     headerRow: int = 1
     sheetName: str | None = None
     highlightForReview: bool = True
+    exportLayout: ExportLayout = ExportLayout.APPEND_COLUMNS
+    softpakHeaderRows: list[list[str]] = Field(default_factory=list)
 
     datetimeInserted: datetime = Field(default_factory=datetime.now)
     datetimeUpdated: datetime = Field(default_factory=datetime.now)
 
-    def header_for(self, field: InputField) -> str | None:
-        """
-        The client's header for a canonical field, if mapped.
-        """
-        return self.columnMapping.get(field.value)
-
     def missing_required_fields(self) -> list[str]:
-        """
-        Required canonical fields this profile does not map.
-        """
         return [
             field.value
             for field in InputField
@@ -104,38 +77,171 @@ class ClientProfileModel(MongoBaseModel):
         ]
 
 
-class ClientProfileShorten(MongoBaseShortenModel):
-    """
-    Client profile projection for list endpoints.
-    """
+def goenee_softpak_profile() -> ClientProfileModel:
+    return ClientProfileModel(
+        name="Goenee Softpak",
+        description="Jac. Goenee B.V., Vianen. Softpak line list, Excel in and out.",
+        exportProfile=ExportProfileType.SOFTPAK,
+        exportLayout=ExportLayout.SOFTPAK_SHEET,
+        headerRow=7,
+        columnMapping={
+            InputField.CODE.value: "hs code",
+            InputField.DESCRIPTION.value: "description",
+            InputField.GROSS_WEIGHT.value: "WEIGHT GROSS",
+            InputField.NET_WEIGHT.value: "WEIGHT nett",
+            InputField.COUNTRY.value: "origin",
+            InputField.QUANTITY.value: "sup_quantity",
+            InputField.COLLI_TYPE.value: "type colli",
+            InputField.PACKAGES.value: "nr colli",
+            InputField.REMARKS.value: "remarks",
+            InputField.VALUE.value: "value",
+        },
+        exportColumns=[ExportColumn(field=OutputField.YCODES, header="Ycodes")],
+        softpakHeaderRows=[
+            ["Government procedure", "10", " "],
+            ["Previous ", "00"],
+            ["Invoice N935", " "],
+            ["Currency", "EUR"],
+            ["Container"],
+        ],
+        defaults=ProfileDefaults(
+            direction=TradeDirection.EXPORT, useTodayWhenDateMissing=True
+        ),
+        totals=TotalsConfig(
+            sumFields=[
+                InputField.GROSS_WEIGHT.value,
+                InputField.NET_WEIGHT.value,
+                InputField.PACKAGES.value,
+                InputField.VALUE.value,
+            ]
+        ),
+    )
 
-    name: str
-    exportProfile: ExportProfileType
 
+def goenee_export_line_list_profile() -> ClientProfileModel:
+    """The line list Goenee produces from a G-Star manifest — `Export Goenee.xls`.
 
-def goenee_profile() -> ClientProfileModel:
+    It is the same shipment as the ecom manifest but already worked: the Y-code
+    column holds the exemptions a declarant assigned by hand, which is what the
+    gold-set scoring is measured against.
     """
-    The starting profile for Goenee (Softpak).
+    return ClientProfileModel(
+        name="Goenee export line list",
+        description="Jac. Goenee B.V. line list for a G-Star export, legacy .xls.",
+        exportProfile=ExportProfileType.SOFTPAK,
+        columnMapping={
+            InputField.CODE.value: "HS Code",
+            InputField.DESCRIPTION.value: "Item Descr",
+            InputField.GROSS_WEIGHT.value: "Total Item Weight",
+            InputField.COUNTRY.value: "Item COO",
+            InputField.PACKAGES.value: "Pieces",
+            InputField.COLLI_TYPE.value: "Colli type",
+            # The header is truncated in the file itself; it has no closing
+            # bracket, and the mapping matches what is actually written.
+            InputField.REMARKS.value: "Remarks (EAN",
+            InputField.VALUE.value: "Total Value (GBP)",
+        },
+        exportColumns=[ExportColumn(field=OutputField.YCODES, header="Y-code")],
+        defaults=ProfileDefaults(
+            direction=TradeDirection.EXPORT, destinationCountry="GB"
+        ),
+        totals=TotalsConfig(
+            sumFields=[
+                InputField.PACKAGES.value,
+                InputField.GROSS_WEIGHT.value,
+                InputField.VALUE.value,
+            ],
+            parcelKeyColumn="Tracking no",
+            parcelLevelFields=[
+                InputField.PACKAGES.value,
+                InputField.GROSS_WEIGHT.value,
+            ],
+            packagesFromParcelCount=True,
+        ),
+    )
 
-    The column names are the ones their files have used so far; the export
-    order follows the five agreed columns until Judith's Softpak template
-    specification lands, at which point only this function changes.
+
+def efg_henx_transit_profile() -> ClientProfileModel:
+    """EFG Arabia to Henx Europe — the packs Goenee files under external transit.
+
+    The goods are not released on arrival: they move under T1 to Henx and are
+    declared there, so duty, VAT and excise are secured rather than paid. The
+    prohibitions and restrictions still bind, which is what this profile is for.
+
+    The codes in these packs are eight-digit CN. An import declaration is filed
+    on ten, so most lines come back with their declarable candidates rather than
+    an answer — that is the pack's gap, not the engine's, and it is left visible.
     """
     return ClientProfileModel(
-        name="Goenee",
-        description="Customs broker, Vianen. Files Softpak declarations.",
+        name="EFG Arabia to Henx (T1)",
+        description="EFG Arabia commercial invoice / packing list, external transit to Henx Europe BV.",
+        exportProfile=ExportProfileType.SOFTPAK,
+        # The pack opens with the invoice head — shipper, receiver, totals — and
+        # the line table starts at row 20. The workbook's other sheet is a pivot
+        # of the same lines and holds no declarable detail.
+        sheetName="1. SUMMARY (EQ LIST FOR SHIPPIN",
+        headerRow=20,
+        columnMapping={
+            InputField.CODE.value: "HS CODE",
+            InputField.COUNTRY.value: "(country)",
+            InputField.DESCRIPTION.value: "Description",
+            InputField.QUANTITY.value: "Quantity",
+            InputField.GROSS_WEIGHT.value: "Case weight",
+            InputField.VALUE.value: "Total Value (EUR)|Total Value",
+            InputField.REMARKS.value: "Serial No.",
+        },
+        exportColumns=[ExportColumn(field=OutputField.YCODES, header="Y-code")],
+        defaults=ProfileDefaults(
+            direction=TradeDirection.IMPORT,
+            procedure=CustomsProcedure.EXTERNAL_TRANSIT_T1,
+            destinationCountry="NL",
+            useTodayWhenDateMissing=True,
+        ),
+        totals=TotalsConfig(
+            sumFields=[
+                InputField.GROSS_WEIGHT.value,
+                InputField.QUANTITY.value,
+                InputField.VALUE.value,
+            ]
+        ),
+    )
+
+
+def gstar_ecom_profile() -> ClientProfileModel:
+    return ClientProfileModel(
+        name="G-Star ecom GB",
+        description="Group eStore BV parcel manifest, NL to GB. One row per item.",
         exportProfile=ExportProfileType.SOFTPAK,
         columnMapping={
-            InputField.CODE.value: "Goederencode",
-            InputField.COUNTRY.value: "Land",
-            InputField.DIRECTION.value: "Richting",
-            InputField.DATE.value: "Datum",
-            InputField.DESCRIPTION.value: "Omschrijving",
-            InputField.QUANTITY.value: "Aantal",
-            InputField.PACKAGES.value: "Colli",
-            InputField.GROSS_WEIGHT.value: "Brutogewicht",
-            InputField.NET_WEIGHT.value: "Nettogewicht",
-            InputField.VALUE.value: "Waarde",
+            InputField.CODE.value: "HS Code",
+            InputField.DESCRIPTION.value: "Item Descr",
+            InputField.GROSS_WEIGHT.value: "Total Weight",
+            InputField.COUNTRY.value: "Item COO",
+            InputField.DESTINATION_COUNTRY.value: "Cust Country",
+            InputField.PACKAGES.value: "Pieces",
+            InputField.QUANTITY.value: "Item Qty",
+            InputField.VALUE.value: "Item Value",
+            InputField.REMARKS.value: "SKU",
         },
-        defaults=ProfileDefaults(direction=TradeDirection.EXPORT),
+        exportColumns=[ExportColumn(field=OutputField.YCODES, header="Y-code")],
+        defaults=ProfileDefaults(
+            direction=TradeDirection.EXPORT, destinationCountry="GB"
+        ),
+        # One row per item, many rows per parcel. "Total Weight" and the piece
+        # count belong to the parcel, so both are counted once per tracking
+        # number. Checked against the Master Invoice for 04-09-2026: 167
+        # packages and GBP 11,376.42, which only reconcile this way.
+        totals=TotalsConfig(
+            sumFields=[
+                InputField.PACKAGES.value,
+                InputField.GROSS_WEIGHT.value,
+                InputField.VALUE.value,
+            ],
+            parcelKeyColumn="Tracking no",
+            parcelLevelFields=[
+                InputField.PACKAGES.value,
+                InputField.GROSS_WEIGHT.value,
+            ],
+            packagesFromParcelCount=True,
+        ),
     )

cbh/api/client/schemas.py

@@ -1,7 +1,3 @@
-"""
-Client profile schemas.
-"""
-
 from pydantic import BaseModel
 
 from cbh.api.client.dto import ExportProfileType
@@ -9,10 +5,6 @@ from cbh.api.client.models import ExportColumn, ProfileDefaults, TotalsConfig
 
 
 class ClientProfileRequest(BaseModel):
-    """
-    Create or replace a client profile.
-    """
-
     name: str
     description: str | None = None
     exportProfile: ExportProfileType = ExportProfileType.SOFTPAK
@@ -28,10 +20,6 @@ class ClientProfileRequest(BaseModel):
 
 
 class ClientProfileResponse(BaseModel):
-    """
-    A client profile as returned by the API.
-    """
-
     id: str
     name: str
     description: str | None = None
@@ -49,10 +37,6 @@ class ClientProfileResponse(BaseModel):
 
 
 class ColumnMappingSuggestion(BaseModel):
-    """
-    A guessed mapping for a newly uploaded file, for confirmation.
-    """
-
     headers: list[str]
     suggestedMapping: dict[str, str]
     unmappedHeaders: list[str]

cbh/api/client/utils.py

@@ -1,14 +1,8 @@
-"""
-Client profile helpers: guessing a column mapping from a file's headers.
-"""
-
 from typing import Any
 
 from cbh.api.client.dto import InputField
 from cbh.api.importer.utils import normalize_header
 
-# Header spellings seen in broker files, per canonical field. Dutch first:
-# that is what the files in scope are written in.
 FIELD_SYNONYMS: dict[InputField, tuple[str, ...]] = {
     InputField.CODE: (
         "goederencode",
@@ -86,14 +80,6 @@ FIELD_SYNONYMS: dict[InputField, tuple[str, ...]] = {
 
 
 def suggest_column_mapping(headers: list[Any]) -> dict[str, str]:
-    """
-    Guess a column mapping from a file's header row.
-
-    Exact matches on a known spelling win; otherwise a header that contains a
-    known spelling is accepted. A guess is a starting point for the one-time
-    per-client configuration, not a substitute for it — which is why the API
-    returns it for confirmation rather than saving it.
-    """
     normalized = [(normalize_header(header), str(header)) for header in headers if header]
     mapping: dict[str, str] = {}
     taken: set[str] = set()
@@ -135,11 +121,5 @@ def suggest_column_mapping(headers: list[Any]) -> dict[str, str]:
 
 
 def unmapped_headers(headers: list[Any], mapping: dict[str, str]) -> list[str]:
-    """
-    Headers the mapping does not claim.
-
-    These are carried through to the output untouched, so the client's file
-    comes back with everything it arrived with.
-    """
     claimed = set(mapping.values())
     return [str(header) for header in headers if header and str(header) not in claimed]

cbh/api/client/views.py

@@ -1,15 +1,11 @@
-"""
-Client profile views module.
-"""
-
 from datetime import datetime
 
-from fastapi import Depends, File, Query, UploadFile
+from fastapi import File, Query, UploadFile
 
-from cbh.api.account.dto import AccountType
-from cbh.api.account.models import AccountModel
+from cbh.api.batch.services.excel import read_headers
 from cbh.api.client import client_router
 from cbh.api.client.db_requests import get_profile
+from cbh.api.client.dto import InputField
 from cbh.api.client.models import ClientProfileModel, default_export_columns
 from cbh.api.client.schemas import (
     ClientProfileRequest,
@@ -18,19 +14,12 @@ from cbh.api.client.schemas import (
 )
 from cbh.api.client.utils import suggest_column_mapping, unmapped_headers
 from cbh.api.common.db_requests import delete_obj, get_all_objs, insert_obj, update_obj
+from cbh.api.common.dto import Paging
 from cbh.api.common.schemas import AllObjectsResponse
-from cbh.api.common.utils import to_paging
-from cbh.core.security import PermissionDependency
 from cbh.core.wrappers import CbhResponseWrapper
 
-ANY_ACCOUNT = [AccountType.USER, AccountType.ADMIN, AccountType.OWNER, AccountType.SUPER_ADMIN]
-ADMIN_ONLY = [AccountType.ADMIN, AccountType.OWNER, AccountType.SUPER_ADMIN]
-
 
 def _to_response(profile: ClientProfileModel) -> ClientProfileResponse:
-    """
-    Map a client profile onto its API representation.
-    """
     return ClientProfileResponse(
         id=profile.id,
         name=profile.name,
@@ -51,17 +40,13 @@ def _to_response(profile: ClientProfileModel) -> ClientProfileResponse:
 async def list_profiles(
     pageSize: int = Query(default=25, le=100),
     pageIndex: int = Query(default=0, ge=0),
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
 ) -> CbhResponseWrapper[AllObjectsResponse[ClientProfileResponse]]:
-    """
-    List the configured client profiles.
-    """
     profiles, total = await get_all_objs(
         ClientProfileModel, page_size=pageSize, page_index=pageIndex, sort=("name", 1)
     )
     return CbhResponseWrapper(
         data=AllObjectsResponse(
-            paging=to_paging(pageSize, pageIndex, total),
+            paging=Paging(pageSize=pageSize, pageIndex=pageIndex, totalCount=total),
             data=[_to_response(profile) for profile in profiles],
         )
     )
@@ -70,22 +55,14 @@ async def list_profiles(
 @client_router.get("/{profileId}")
 async def get_client_profile(
     profileId: str,
-    account: AccountModel = Depends(PermissionDependency(ANY_ACCOUNT)),
 ) -> CbhResponseWrapper[ClientProfileResponse]:
-    """
-    Fetch one client profile.
-    """
     return CbhResponseWrapper(data=_to_response(await get_profile(profileId)))
 
 
 @client_router.post("")
 async def create_profile(
     data: ClientProfileRequest,
-    account: AccountModel = Depends(PermissionDependency(ADMIN_ONLY)),
 ) -> CbhResponseWrapper[ClientProfileResponse]:
-    """
-    Create a client profile.
-    """
     profile = ClientProfileModel(
         **data.model_dump(exclude_none=True),
     )
@@ -99,11 +76,7 @@ async def create_profile(
 async def replace_profile(
     profileId: str,
     data: ClientProfileRequest,
-    account: AccountModel = Depends(PermissionDependency(ADMIN_ONLY)),
 ) -> CbhResponseWrapper[ClientProfileResponse]:
-    """
-    Replace a client profile.
-    """
     existing = await get_profile(profileId)
     updated = ClientProfileModel(
         **{
@@ -120,11 +93,7 @@ async def replace_profile(
 @client_router.delete("/{profileId}")
 async def delete_profile(
     profileId: str,
-    account: AccountModel = Depends(PermissionDependency(ADMIN_ONLY)),
 ) -> CbhResponseWrapper[ClientProfileResponse]:
-    """
-    Delete a client profile.
-    """
     removed = await delete_obj(ClientProfileModel, profileId)
     return CbhResponseWrapper(data=_to_response(removed))
 
@@ -133,18 +102,7 @@ async def delete_profile(
 async def suggest_mapping(
     file: UploadFile = File(...),
     headerRow: int = Query(default=1, ge=1),
-    account: AccountModel = Depends(PermissionDependency(ADMIN_ONLY)),
 ) -> CbhResponseWrapper[ColumnMappingSuggestion]:
-    """
-    Guess a column mapping from an uploaded client file.
-
-    The guess is returned for confirmation, never saved: the mapping is
-    configured once per client and a wrong guess would quietly mis-read every
-    pack that followed.
-    """
-    from cbh.api.batch.services.excel import read_headers
-    from cbh.api.client.dto import InputField
-
     payload = await file.read()
     headers = read_headers(payload, file.filename or "upload.xlsx", headerRow)
     mapping = suggest_column_mapping(headers)

cbh/api/common/db_requests.py

@@ -1,27 +1,16 @@
-"""
-Common database requests.
-"""
-
-import re
-from datetime import datetime, timedelta
 from typing import Any, TypeVar
 
 from fastapi import HTTPException
 from pydantic import BaseModel
 from pymongo import ASCENDING, DESCENDING, UpdateOne
 
-from cbh.api.common.schemas import SearchRequest
 from cbh.core.config import settings
 
 T = TypeVar("T", bound=BaseModel)
 
 collection_map = {
-    "AccountModel": "accounts",
-    "AccountShorten": "accounts",
     "CacheEntryModel": "caches",
-    # --- TARIC / national reference data ---
     "NomenclatureModel": "nomenclature",
-    "NomenclatureShorten": "nomenclature",
     "MeasureModel": "measures",
     "MeasureTypeModel": "measuretypes",
     "GeoAreaModel": "geoareas",
@@ -29,19 +18,14 @@ collection_map = {
     "DocumentCodeModel": "documentcodes",
     "CodeChangeModel": "codechanges",
     "CodeboekEntryModel": "codeboekentries",
-    # --- importer ---
     "SyncLogModel": "synclogs",
-    # --- client configuration ---
     "ClientProfileModel": "clientprofiles",
-    "ClientProfileShorten": "clientprofiles",
-    # --- excel batches ---
     "BatchModel": "batches",
-    "BatchShorten": "batches",
     "BatchRowModel": "batchrows",
+    "RuleModel": "rules",
 }
 
 INDEXES: dict[str, list[tuple[list[tuple[str, int]], dict]]] = {
-    "accounts": [([("id", ASCENDING)], {"unique": True})],
     "caches": [
         ([("namespace", ASCENDING), ("key", ASCENDING)], {"unique": True}),
         ([("deletedAt", ASCENDING)], {"expireAfterSeconds": 0}),
@@ -77,25 +61,20 @@ INDEXES: dict[str, list[tuple[list[tuple[str, int]], dict]]] = {
         ([("datetimeInserted", DESCENDING)], {}),
     ],
     "batchrows": [([("batchId", ASCENDING), ("rowIndex", ASCENDING)], {"unique": True})],
+    "rules": [
+        ([("id", ASCENDING)], {"unique": True}),
+        ([("active", ASCENDING), ("scope", ASCENDING), ("clientProfileId", ASCENDING)], {}),
+    ],
 }
 
 
 async def ensure_indexes() -> None:
-    """
-    Create the indexes the application relies on.
-
-    Idempotent: MongoDB ignores a create_index call for an index that already
-    exists with the same definition.
-    """
     for collection, definitions in INDEXES.items():
         for keys, options in definitions:
             await settings.DB_CLIENT[collection].create_index(keys, **options)
 
 
 def resolve_collection(model: T, collection: str | None = None) -> str:
-    """
-    Resolve the MongoDB collection backing a model.
-    """
     if collection:
         return collection
     name = model if isinstance(model, str) else model.__name__
@@ -112,9 +91,6 @@ async def get_obj_by_id(
     exception: bool = True,
     collection: str | None = None,
 ) -> T | None:
-    """
-    Get an object by ID.
-    """
     filter_ = {"id": obj_id} if obj_id else {}
     if additional_filter:
         filter_.update(additional_filter)
@@ -128,18 +104,15 @@ async def get_obj_by_id(
     return model.from_mongo(obj)
 
 
-async def _paginate_objs(
+async def get_all_objs(
     model: T,
-    filter_: dict,
     page_size: int,
     page_index: int,
-    sort: tuple[str, int],
-    projection: dict | None,
+    sort: tuple[str, int] = ("id", -1),
+    additional_filter: dict | None = None,
+    projection: dict | None = None,
     collection: str | None = None,
 ) -> tuple[list[T], int]:
-    """
-    Fetch one page plus the total count in a single $facet aggregate.
-    """
     if page_size > 0:
         page_stages = [{"$skip": page_index * page_size}, {"$limit": page_size}]
     else:
@@ -147,7 +120,7 @@ async def _paginate_objs(
     if projection:
         page_stages.append({"$project": projection})
     pipeline = [
-        {"$match": filter_},
+        {"$match": additional_filter or {}},
         {"$sort": {sort[0]: sort[1]}},
         {
             "$facet": {
@@ -163,22 +136,6 @@ async def _paginate_objs(
     return objs, total_count
 
 
-async def get_all_objs(
-    model: T,
-    page_size: int,
-    page_index: int,
-    sort: tuple[str, int] = ("id", -1),
-    additional_filter: dict | None = None,
-    projection: dict | None = None,
-    collection: str | None = None,
-) -> tuple[list[T], int]:
-    """
-    Get all objects.
-    """
-    filter_ = additional_filter if additional_filter else {}
-    return await _paginate_objs(model, filter_, page_size, page_index, sort, projection, collection)
-
-
 async def find_objs(
     model: T,
     filter_: dict,
@@ -187,9 +144,6 @@ async def find_objs(
     projection: dict | None = None,
     collection: str | None = None,
 ) -> list[T]:
-    """
-    Find objects without pagination metadata.
-    """
     cursor = settings.DB_CLIENT[resolve_collection(model, collection)].find(filter_, projection)
     if sort:
         cursor = cursor.sort(sort[0], sort[1])
@@ -201,27 +155,18 @@ async def find_objs(
 async def count_objs(
     model: T, additional_filter: dict | None = None, collection: str | None = None
 ) -> int:
-    """
-    Count objects.
-    """
     return await settings.DB_CLIENT[resolve_collection(model, collection)].count_documents(
         additional_filter or {}
     )
 
 
 async def insert_obj(model_instance: BaseModel, collection: str | None = None) -> BaseModel:
-    """
-    Insert a single object.
-    """
     collection = resolve_collection(type(model_instance), collection)
     await settings.DB_CLIENT[collection].insert_one(model_instance.to_mongo())
     return model_instance
 
 
 async def insert_objs(model_instances: list[BaseModel], collection: str | None = None) -> int:
-    """
-    Insert many objects at once.
-    """
     if not model_instances:
         return 0
     collection = resolve_collection(type(model_instances[0]), collection)
@@ -239,9 +184,6 @@ async def update_obj(
     collection: str | None = None,
     exception: bool = True,
 ) -> T | None:
-    """
-    Update an object by ID and return the updated document.
-    """
     filter_ = {"id": obj_id}
     if additional_filter:
         filter_.update(additional_filter)
@@ -261,16 +203,6 @@ async def upsert_objs(
     key_fields: list[str],
     collection: str | None = None,
 ) -> tuple[int, int]:
-    """
-    Idempotently upsert reference-data documents.
-
-    The natural key (``key_fields``) decides identity, not the generated ``id``:
-    re-running an import must update in place rather than duplicate. ``id`` and
-    ``datetimeInserted`` are only written when the document is first created, so
-    existing references stay stable across re-imports.
-
-    Returns a (matched, upserted) tuple.
-    """
     if not documents:
         return 0, 0
     collection = resolve_collection(model, collection)
@@ -293,9 +225,6 @@ async def delete_obj(
     additional_filter: dict | None = None,
     collection: str | None = None,
 ) -> T:
-    """
-    Delete an object.
-    """
     filter_ = {"id": obj_id} if obj_id else {}
     if additional_filter:
         filter_.update(additional_filter)
@@ -310,89 +239,7 @@ async def delete_obj(
 async def delete_objs(
     model: T, additional_filter: dict | None = None, collection: str | None = None
 ) -> int:
-    """
-    Delete every object matching a filter. Used by full-replacement imports.
-    """
     result = await settings.DB_CLIENT[resolve_collection(model, collection)].delete_many(
         additional_filter or {}
     )
     return result.deleted_count
-
-
-async def search_objs(
-    model: T,
-    data: SearchRequest,
-    additional_filter: dict | None = None,
-    projection: dict | None = None,
-    collection: str | None = None,
-) -> tuple[list[T], int]:
-    """
-    Search for objects in a specified collection based on search filters.
-    """
-    filters = []
-    date_filters: dict[str, list[dict]] = {}
-
-    for search_filter in data.filter:
-        if isinstance(search_filter.value, str):
-            date_match = re.fullmatch(r"^(\d{4}-\d{2}-\d{2});([+-]\d{1,2})$", search_filter.value)
-
-            if date_match:
-                date_filters.setdefault(search_filter.name, []).append(
-                    {
-                        "date": datetime.strptime(date_match.group(1), "%Y-%m-%d"),
-                        "timezone_offset": int(date_match.group(2)),
-                    }
-                )
-            else:
-                filters.append(
-                    {
-                        search_filter.name: {
-                            "$regex": re.escape(search_filter.value),
-                            "$options": "i",
-                        }
-                    }
-                )
-        else:
-            filters.append({search_filter.name: search_filter.value})
-
-    for field_name, dates in date_filters.items():
-        if len(dates) == 1:
-            date_info = dates[0]
-            user_local_day_start = date_info["date"]
-            user_local_day_end = user_local_day_start + timedelta(days=1)
-            filters.append(
-                {
-                    field_name: {
-                        "$gte": (
-                            user_local_day_start - timedelta(hours=date_info["timezone_offset"])
-                        ).isoformat(),
-                        "$lt": (
-                            user_local_day_end - timedelta(hours=date_info["timezone_offset"])
-                        ).isoformat(),
-                    }
-                }
-            )
-        elif len(dates) >= 2:
-            start_date = min(dates, key=lambda x: x["date"])
-            end_date = max(dates, key=lambda x: x["date"])
-
-            start_datetime = start_date["date"] - timedelta(hours=start_date["timezone_offset"])
-            end_datetime = (
-                end_date["date"] + timedelta(days=1) - timedelta(hours=end_date["timezone_offset"])
-            )
-
-            filters.append(
-                {
-                    field_name: {
-                        "$gte": start_datetime.isoformat(),
-                        "$lt": end_datetime.isoformat(),
-                    }
-                }
-            )
-
-    if additional_filter:
-        filters.append(additional_filter)
-    regex_filter = {"$and": filters} if filters else {}
-    return await _paginate_objs(
-        model, regex_filter, data.pageSize, data.pageIndex, ("id", -1), projection, collection
-    )

cbh/api/common/dto.py

@@ -1,113 +1,105 @@
-"""
-Common DTOs shared across CargoFlow modules.
-"""
-
-from datetime import datetime
-from enum import Enum
+from enum import IntEnum
 
 from pydantic import BaseModel
 
 
-class OrderType(Enum):
-    """
-    Sort order.
-    """
-
-    ASC = 1
-    DESC = 2
-
-
-class SortBy(BaseModel):
-    """
-    Sorting instruction for list endpoints.
-    """
-
-    name: str
-    order: OrderType = OrderType.DESC
-
-    def to_mongo_sort(self) -> tuple[str, int]:
-        """
-        Convert to a MongoDB sort tuple.
-        """
-        return self.name, 1 if self.order == OrderType.ASC else -1
-
-
 class Paging(BaseModel):
-    """
-    Pagination model for API responses.
-    """
-
     pageSize: int
     pageIndex: int
     totalCount: int
 
 
-class SearchFilter(BaseModel):
-    """
-    Search filter model for constructing database queries.
-    """
+class CustomsProcedure(IntEnum):
+    """The customs procedure the declaration is filed under.
 
-    name: str
-    value: str | int
-
-
-class IDName(BaseModel):
-    """
-    Generic id/name pair.
-    """
-
-    id: str | int
-    name: str
-
-
-class DateValue(BaseModel):
-    """
-    Date/value pair for time series responses.
+    Direction says which way the goods move; this says what is being asked of
+    customs when they move. The difference matters because under transit the
+    goods are not released, so duty and tax are secured rather than paid, while
+    prohibitions and restrictions bind exactly as they would on release.
     """
 
-    date: datetime
-    value: int
+    RELEASE_FOR_FREE_CIRCULATION = 1
+    EXTERNAL_TRANSIT_T1 = 2
+    INTERNAL_TRANSIT_T2 = 3
+    CUSTOMS_WAREHOUSE = 4
 
+    @property
+    def is_transit(self) -> bool:
+        return self in (
+            CustomsProcedure.EXTERNAL_TRANSIT_T1,
+            CustomsProcedure.INTERNAL_TRANSIT_T2,
+        )
 
-class TradeDirection(Enum):
-    """
-    Direction of the declaration line.
-
-    A measure may apply to imports only, exports only, or both.
-    """
+    @property
+    def suspends_duty(self) -> bool:
+        """Whether duty, tax and quota are secured now and settled elsewhere."""
+        return self is not CustomsProcedure.RELEASE_FOR_FREE_CIRCULATION
 
+    @classmethod
+    def parse(
+        cls, value: "str | int | CustomsProcedure | None"
+    ) -> "CustomsProcedure | None":
+        if value is None or isinstance(value, CustomsProcedure):
+            return value
+        if isinstance(value, int):
+            return cls(value)
+        return {
+            "t1": cls.EXTERNAL_TRANSIT_T1,
+            "t-1": cls.EXTERNAL_TRANSIT_T1,
+            "extern douanevervoer": cls.EXTERNAL_TRANSIT_T1,
+            "external transit": cls.EXTERNAL_TRANSIT_T1,
+            "t2": cls.INTERNAL_TRANSIT_T2,
+            "t-2": cls.INTERNAL_TRANSIT_T2,
+            "intern douanevervoer": cls.INTERNAL_TRANSIT_T2,
+            "internal transit": cls.INTERNAL_TRANSIT_T2,
+            "free circulation": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "release for free circulation": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "in het vrije verkeer brengen": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "40": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "4000": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "42": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "4200": cls.RELEASE_FOR_FREE_CIRCULATION,
+            "warehouse": cls.CUSTOMS_WAREHOUSE,
+            "entrepot": cls.CUSTOMS_WAREHOUSE,
+            "douane-entrepot": cls.CUSTOMS_WAREHOUSE,
+            "71": cls.CUSTOMS_WAREHOUSE,
+            "7100": cls.CUSTOMS_WAREHOUSE,
+        }.get(str(value).strip().lower())
+
+
+class TradeDirection(IntEnum):
     IMPORT = 1
     EXPORT = 2
     BOTH = 3
+    WAREHOUSE = 4
 
     @classmethod
     def parse(cls, value: "str | int | TradeDirection | None") -> "TradeDirection | None":
-        """
-        Parse a direction from the loose representations found in client files.
-        """
         if value is None or isinstance(value, TradeDirection):
             return value
         if isinstance(value, int):
             return cls(value)
-        normalized = str(value).strip().lower()
-        mapping = {
+        return {
             "import": cls.IMPORT,
             "invoer": cls.IMPORT,
             "in": cls.IMPORT,
             "i": cls.IMPORT,
             "1": cls.IMPORT,
+            "40": cls.IMPORT,
+            "4000": cls.IMPORT,
             "export": cls.EXPORT,
             "uitvoer": cls.EXPORT,
             "out": cls.EXPORT,
             "e": cls.EXPORT,
             "2": cls.EXPORT,
+            "10": cls.EXPORT,
+            "1000": cls.EXPORT,
             "both": cls.BOTH,
             "3": cls.BOTH,
-        }
-        return mapping.get(normalized)
-
-    def matches(self, other: "TradeDirection") -> bool:
-        """
-        Check whether a measure direction applies to a requested direction.
-        """
-        return self is TradeDirection.BOTH or other is TradeDirection.BOTH or self is other
+            "warehouse": cls.WAREHOUSE,
+            "entrepot": cls.WAREHOUSE,
+            "douane-entrepot": cls.WAREHOUSE,
+            "douane entrepot": cls.WAREHOUSE,
+            "71": cls.WAREHOUSE,
+            "7100": cls.WAREHOUSE,
+        }.get(str(value).strip().lower())

cbh/api/common/models.py

@@ -1,7 +1,3 @@
-"""
-Common models shared across CargoFlow modules.
-"""
-
 from datetime import date, datetime, timedelta
 
 from pydantic import BaseModel, Field
@@ -10,41 +6,16 @@ from cbh.core.database import MongoBaseModel, MongoSerializableModel
 
 
 class ValidityMixin(BaseModel):
-    """
-    Validity period carried by every piece of reference data.
-
-    History is never deleted: each record keeps the window it was in force for,
-    so a check can always be replayed with the rules of the day a declaration
-    was filed. ``validTo = None`` means "open ended".
-    """
-
     validFrom: date | None = None
     validTo: date | None = None
 
-    def is_valid_on(self, on_date: date) -> bool:
-        """
-        Check whether this record is in force on a given date.
-        """
-        if self.validFrom and on_date < self.validFrom:
-            return False
-        if self.validTo and on_date > self.validTo:
-            return False
-        return True
-
     def days_until_expiry(self, on_date: date) -> int | None:
-        """
-        Days left until this record expires, or None when open ended.
-        """
         if self.validTo is None:
             return None
         return (self.validTo - on_date).days
 
 
 class FileRef(MongoSerializableModel):
-    """
-    Reference to a file kept in local storage.
-    """
-
     fileName: str
     storagePath: str
     contentType: str | None = None
@@ -53,13 +24,6 @@ class FileRef(MongoSerializableModel):
 
 
 class CacheEntryModel(MongoBaseModel):
-    """
-    Generic TTL cache entry.
-
-    Used by the line-check engine so that an identical (code, country,
-    direction, date) tuple is resolved once and reused across a batch.
-    """
-
     namespace: str
     key: str
     value: dict | list | str | int | float | bool | None = None

cbh/api/common/schemas.py

@@ -1,81 +1,12 @@
-"""
-Common schemas.
-"""
-
 from typing import Generic, TypeVar
 
 from pydantic import BaseModel
 
-from cbh.api.common.dto import Paging, SearchFilter, SortBy
+from cbh.api.common.dto import Paging
 
 T = TypeVar("T", bound=BaseModel)
 
 
 class AllObjectsResponse(BaseModel, Generic[T]):
-    """
-    Response model for paginated collections.
-    """
-
     paging: Paging
     data: list[T]
-
-
-class SearchRequest(BaseModel):
-    """
-    Request schema for searching objects.
-    """
-
-    filter: list[SearchFilter]
-    pageSize: int = 10
-    pageIndex: int = 0
-
-
-class FilterRequest(BaseModel, Generic[T]):
-    """
-    Filter request.
-    """
-
-    filter: T
-    sortBy: SortBy | None = None
-    pageSize: int = 10
-    pageIndex: int = 0
-
-
-class PlainTextResponse(BaseModel):
-    """
-    Response model for plain text.
-    """
-
-    text: str
-
-
-class ContentRequest(BaseModel):
-    """
-    Content request.
-    """
-
-    content: str | None = None
-
-
-class BatchIdsRequest(BaseModel):
-    """
-    Batch ids request.
-    """
-
-    ids: list[str]
-
-
-class EmailRequest(BaseModel):
-    """
-    Email request.
-    """
-
-    email: str
-
-
-class CountResponse(BaseModel):
-    """
-    Generic count response, used by importer and reference data endpoints.
-    """
-
-    count: int

cbh/api/common/services/cache/__init__.py

@@ -1,19 +1,3 @@
-from .cache import (
-    LINE_CHECK_NAMESPACE,
-    build_cache_key,
-    cached_payload,
-    get_cache,
-    invalidate_cache,
-    invalidate_line_check_cache,
-    set_cache,
-)
+from .cache import build_cache_key, get_cache, invalidate_line_check_cache, set_cache
 
-__all__ = [
-    "LINE_CHECK_NAMESPACE",
-    "build_cache_key",
-    "cached_payload",
-    "get_cache",
-    "invalidate_cache",
-    "invalidate_line_check_cache",
-    "set_cache",
-]
+__all__ = ["build_cache_key", "get_cache", "invalidate_line_check_cache", "set_cache"]

cbh/api/common/services/cache/cache.py

@@ -1,16 +1,4 @@
-"""
-Generic TTL cache backed by MongoDB.
-
-The line-check engine resolves each unique (code, country, direction, date)
-tuple once; a pack of ~580 declaration lines typically collapses to a few dozen
-distinct tuples, which is where most of the runtime saving comes from.
-
-The cache is keyed on the reference-data state as well, so an importer run
-invalidates it rather than serving answers built on superseded rules.
-"""
-
 import hashlib
-import json
 from datetime import datetime, timedelta
 from typing import Any
 
@@ -20,21 +8,13 @@ from cbh.api.common.models import CacheEntryModel
 from cbh.core.config import settings
 from cbh.core.wrappers import background_task
 
-LINE_CHECK_NAMESPACE = "linecheck"
-
 
 def build_cache_key(*parts: Any) -> str:
-    """
-    Build a stable cache key from its parts.
-    """
     raw = "|".join("" if part is None else str(part) for part in parts)
     return hashlib.sha256(raw.encode("utf-8")).hexdigest()
 
 
 def _serialize(value: Any) -> Any:
-    """
-    Serialize a value into something BSON can store.
-    """
     if isinstance(value, BaseModel):
         return value.model_dump(mode="json")
     if isinstance(value, list):
@@ -45,9 +25,6 @@ def _serialize(value: Any) -> Any:
 
 
 async def get_cache(namespace: str, key: str) -> Any | None:
-    """
-    Read a cache entry, or None when it is absent or expired.
-    """
     entry = await settings.DB_CLIENT.caches.find_one({"namespace": namespace, "key": key})
     if not entry:
         return None
@@ -63,16 +40,12 @@ async def set_cache(
     value: Any,
     ttl_seconds: int | None = None,
 ) -> None:
-    """
-    Write a cache entry, replacing any previous value for the same key.
-    """
     ttl = ttl_seconds if ttl_seconds is not None else settings.LINE_CHECK_CACHE_TTL_SECONDS
-    entry = CacheEntryModel(
+    document = CacheEntryModel(
         namespace=namespace,
         key=key,
         value=_serialize(value),
-    )
-    document = entry.to_mongo()
+    ).to_mongo()
     document["deletedAt"] = datetime.now() + timedelta(seconds=ttl)
     await settings.DB_CLIENT.caches.update_one(
         {"namespace": namespace, "key": key},
@@ -81,30 +54,6 @@ async def set_cache(
     )
 
 
-@background_task()
-async def invalidate_cache(namespace: str, key: str | None = None) -> None:
-    """
-    Drop a single cache entry, or an entire namespace.
-
-    Called after every successful reference-data import: answers computed from
-    superseded rules must never be replayed.
-    """
-    filter_: dict[str, Any] = {"namespace": namespace}
-    if key:
-        filter_["key"] = key
-    await settings.DB_CLIENT.caches.delete_many(filter_)
-
-
 @background_task()
 async def invalidate_line_check_cache() -> None:
-    """
-    Invalidate every cached line-check result.
-    """
-    await settings.DB_CLIENT.caches.delete_many({"namespace": LINE_CHECK_NAMESPACE})
-
-
-def cached_payload(value: Any) -> str:
-    """
-    Compact JSON rendering used for cache-key fingerprints.
-    """
-    return json.dumps(value, sort_keys=True, default=str, separators=(",", ":"))
+    await settings.DB_CLIENT.caches.delete_many({"namespace": "linecheck"})

cbh/api/common/utils.py

@@ -1,56 +1,11 @@
-"""
-Common utilities.
-"""
-
 import re
 from datetime import date, datetime
-from typing import Any, Iterable, Iterator, TypeVar
-
-from cbh.api.common.dto import Paging
+from typing import Any, Iterable, TypeVar
 
 T = TypeVar("T")
 
-_DATE_FORMATS = (
-    "%Y-%m-%d",
-    "%d-%m-%Y",
-    "%d/%m/%Y",
-    "%Y/%m/%d",
-    "%d.%m.%Y",
-    "%Y%m%d",
-)
-
-
-def chunked(items: list[T], size: int) -> Iterator[list[T]]:
-    """
-    Split a list into chunks of at most ``size`` elements.
-    """
-    for start in range(0, len(items), size):
-        yield items[start : start + size]
-
-
-def to_paging(page_size: int, page_index: int, total_count: int) -> Paging:
-    """
-    Build a Paging object for a list response.
-    """
-    return Paging(pageSize=page_size, pageIndex=page_index, totalCount=total_count)
-
-
-def to_iso(value: date | datetime | None) -> str | None:
-    """
-    Serialize a date for MongoDB comparisons.
-
-    Dates are stored as ISO strings by ``MongoSerializableModel.to_mongo``;
-    ISO-8601 sorts lexicographically, so range queries stay correct.
-    """
-    if value is None:
-        return None
-    return value.isoformat()
-
 
 def parse_date(value: Any) -> date | None:
-    """
-    Parse a date coming from a client spreadsheet or a reference-data file.
-    """
     if value is None or value == "":
         return None
     if isinstance(value, datetime):
@@ -60,7 +15,7 @@ def parse_date(value: Any) -> date | None:
     text = str(value).strip()
     if not text:
         return None
-    for date_format in _DATE_FORMATS:
+    for date_format in ("%Y-%m-%d", "%d-%m-%Y", "%d/%m/%Y", "%Y/%m/%d", "%d.%m.%Y", "%Y%m%d"):
         try:
             return datetime.strptime(text, date_format).date()
         except ValueError:
@@ -72,9 +27,6 @@ def parse_date(value: Any) -> date | None:
 
 
 def normalize_ws(value: Any) -> str | None:
-    """
-    Collapse whitespace in a free-text cell.
-    """
     if value is None:
         return None
     text = re.sub(r"\s+", " ", str(value)).strip()
@@ -82,14 +34,11 @@ def normalize_ws(value: Any) -> str | None:
 
 
 def parse_number(value: Any) -> float | None:
-    """
-    Parse a numeric cell tolerating both comma and dot decimal separators.
-    """
     if value is None or value == "":
         return None
     if isinstance(value, (int, float)):
         return float(value)
-    text = str(value).strip().replace(" ", "").replace(" ", "")
+    text = str(value).strip().replace(" ", "").replace(" ", "")
     if not text:
         return None
     if "," in text and "." in text:
@@ -104,9 +53,6 @@ def parse_number(value: Any) -> float | None:
 
 
 def normalize_country(value: Any) -> str | None:
-    """
-    Normalize an ISO country code to its uppercase two-letter form.
-    """
     if value is None:
         return None
     text = re.sub(r"[^A-Za-z]", "", str(value)).upper()
@@ -116,9 +62,6 @@ def normalize_country(value: Any) -> str | None:
 
 
 def dedupe(items: Iterable[T]) -> list[T]:
-    """
-    Remove duplicates while preserving order.
-    """
     seen: set[T] = set()
     result: list[T] = []
     for item in items:

cbh/api/importer/__init__.py

@@ -1,10 +1,3 @@
-"""
-Importer module initialization.
-
-This module defines the FastAPI router for the reference-data importer:
-triggering loads, inspecting the sync log and supplying files by hand.
-"""
-
 from fastapi import APIRouter
 
 importer_router = APIRouter(

cbh/api/importer/db_requests.py

@@ -1,7 +1,3 @@
-"""
-Importer database requests.
-"""
-
 from cbh.api.common.db_requests import find_objs, get_all_objs
 from cbh.api.importer.dto import SyncSource, SyncStatus
 from cbh.api.importer.models import SyncLogModel
@@ -13,9 +9,6 @@ async def get_sync_logs(
     source: SyncSource | None = None,
     status: SyncStatus | None = None,
 ) -> tuple[list[SyncLogModel], int]:
-    """
-    Page through the import history, newest first.
-    """
     filter_: dict = {}
     if source:
         filter_["source"] = source.value
@@ -31,12 +24,6 @@ async def get_sync_logs(
 
 
 async def get_latest_sync_per_source() -> dict[str, SyncLogModel]:
-    """
-    The most recent run for each source.
-
-    This is the operational view: which reference files are loaded, how fresh
-    they are, and which ones last failed.
-    """
     latest: dict[str, SyncLogModel] = {}
     for source in SyncSource:
         runs = await find_objs(
@@ -48,16 +35,3 @@ async def get_latest_sync_per_source() -> dict[str, SyncLogModel]:
         if runs:
             latest[source.name] = runs[0]
     return latest
-
-
-async def get_last_successful_sync(source: SyncSource) -> SyncLogModel | None:
-    """
-    The most recent successful run for one source.
-    """
-    runs = await find_objs(
-        SyncLogModel,
-        {"source": source.value, "status": SyncStatus.SUCCESS.value},
-        limit=1,
-        sort=("finishedAt", -1),
-    )
-    return runs[0] if runs else None

cbh/api/importer/dto.py

@@ -1,18 +1,7 @@
-"""
-Importer DTOs.
-"""
+from enum import IntEnum
 
-from enum import Enum
-
-
-class SyncSource(Enum):
-    """
-    A reference-data file the importer knows how to load.
-
-    TARIC is the source of truth for measures and document requirements;
-    Codeboek Douane supplies the Dutch national code lists.
-    """
 
+class SyncSource(IntEnum):
     TARIC_NOMENCLATURE = 1
     TARIC_DECLARABLE_CODES = 2
     TARIC_DUTIES_IMPORT = 3
@@ -23,38 +12,16 @@ class SyncSource(Enum):
     TARIC_BOX44_CODES = 8
     TARIC_CODE_CHANGES = 9
     CODEBOEK_DOUANE = 10
-
-    @property
-    def is_taric(self) -> bool:
-        """
-        Whether this source comes from the TARIC/CIRCABC distribution.
-        """
-        return self is not SyncSource.CODEBOEK_DOUANE
+    DTV_NATIONAL = 11
 
 
-class SyncMode(Enum):
-    """
-    How a source is loaded.
-
-    TARIC nomenclature arrives as a monthly full load plus daily deltas;
-    Codeboek Douane is republished whole on every change, so it is replaced
-    rather than merged.
-    """
-
+class SyncMode(IntEnum):
     FULL = 1
     DELTA = 2
     REPLACE = 3
 
 
-class SyncStatus(Enum):
-    """
-    Lifecycle of a single import run.
-
-    A run left in ``RUNNING`` by a crash is safe to start again: imports are
-    keyed on the natural key of each record, so re-running only rewrites what
-    it would have written the first time.
-    """
-
+class SyncStatus(IntEnum):
     PENDING = 1
     RUNNING = 2
     SUCCESS = 3

cbh/api/importer/models.py

@@ -1,7 +1,3 @@
-"""
-Importer models.
-"""
-
 from datetime import datetime
 
 from pydantic import Field
@@ -11,14 +7,6 @@ from cbh.core.database import MongoBaseModel
 
 
 class SyncLogModel(MongoBaseModel):
-    """
-    One import run.
-
-    The log is the audit trail behind every answer the engine gives: it records
-    which file was loaded, when, and what it changed. ``fileHash`` makes a
-    repeated load of an unchanged file cheap to detect and skip.
-    """
-
     source: SyncSource
     mode: SyncMode
     status: SyncStatus = SyncStatus.PENDING
@@ -41,10 +29,3 @@ class SyncLogModel(MongoBaseModel):
     durationSeconds: float | None = None
 
     datetimeInserted: datetime = Field(default_factory=datetime.now)
-
-    @property
-    def succeeded(self) -> bool:
-        """
-        Whether this run can be relied on as a reference-data version.
-        """
-        return self.status is SyncStatus.SUCCESS

cbh/api/importer/schemas.py

@@ -1,7 +1,3 @@
-"""
-Importer schemas.
-"""
-
 from datetime import date, datetime
 
 from pydantic import BaseModel
@@ -10,10 +6,6 @@ from cbh.api.importer.dto import SyncMode, SyncSource, SyncStatus
 
 
 class SyncLogResponse(BaseModel):
-    """
-    One import run as returned by the API.
-    """
-
     id: str
     source: SyncSource
     mode: SyncMode
@@ -38,32 +30,34 @@ class SyncLogResponse(BaseModel):
 
 
 class SyncRequest(BaseModel):
-    """
-    Request to import one source.
-
-    With no ``url``, the importer reads the most recent file supplied for the
-    source in local storage.
-    """
-
     source: SyncSource
     mode: SyncMode = SyncMode.FULL
     url: str | None = None
 
 
 class CodebookSyncRequest(BaseModel):
-    """
-    Request to refresh one or all Dutch codebooks.
-    """
-
     fileName: str | None = None
     publishedOn: date | None = None
 
 
-class ImporterStatusResponse(BaseModel):
-    """
-    Operational view of the reference data.
-    """
+class PublicationSyncRequest(BaseModel):
+    mode: SyncMode = SyncMode.FULL
+    period: str | None = None
+    keepArchive: bool = False
 
+
+class PublicationSyncResponse(BaseModel):
+    period: str | None = None
+    url: str | None = None
+    runs: list[SyncLogResponse] = []
+    missingSources: list[SyncSource] = []
+    note: str | None = None
+
+
+class ImporterStatusResponse(BaseModel):
     latestPerSource: dict[str, SyncLogResponse]
     counts: dict[str, int]
     referenceAsOf: str | None = None
+    publicationPeriods: list[str] = []
+    latestPublication: str | None = None
+    scheduledJobs: dict[str, datetime | None] = {}

cbh/api/importer/services/base.py

@@ -1,14 +1,6 @@
-"""
-Import run scaffolding: the sync log, and the idempotent write path.
-
-Every import runs inside a :class:`SyncRun`, which opens a log entry, records
-what the run touched and closes it as success or failure. Runs are safe to
-repeat: records are written under their natural key, so a re-run rewrites
-exactly what it wrote before instead of duplicating it.
-"""
-
 import time
 from datetime import datetime
+from pathlib import Path
 from typing import Any
 
 import httpx
@@ -22,10 +14,6 @@ from cbh.core.config import settings
 
 
 class SyncRun:
-    """
-    Async context manager wrapping one import run.
-    """
-
     def __init__(
         self,
         source: SyncSource,
@@ -64,22 +52,14 @@ class SyncRun:
         )
 
         if self.log.status in (SyncStatus.SUCCESS, SyncStatus.PARTIAL):
-            # Cached answers were computed against the previous reference data.
             await invalidate_line_check_cache()
 
-        # Never swallow the exception: a failed import must surface.
         return False
 
     def record_error(self, message: str) -> None:
-        """
-        Note a recoverable problem; the run finishes as PARTIAL.
-        """
         self.log.errors.append(message)
 
     def set_hash(self, payload: bytes) -> None:
-        """
-        Store the content hash of the file this run loaded.
-        """
         self.log.fileHash = file_hash(payload)
 
     async def write(
@@ -88,9 +68,6 @@ class SyncRun:
         documents: list[dict],
         key_fields: list[str],
     ) -> None:
-        """
-        Upsert a batch of parsed records and account for them in the log.
-        """
         self.log.recordsRead += len(documents)
         matched, upserted = await upsert_objs(model, documents, key_fields)
         self.log.recordsMatched += matched
@@ -98,9 +75,6 @@ class SyncRun:
 
 
 async def download(url: str) -> bytes:
-    """
-    Fetch a reference file over HTTP.
-    """
     async with httpx.AsyncClient(
         timeout=settings.IMPORTER_HTTP_TIMEOUT, follow_redirects=True
     ) as client:
@@ -109,16 +83,43 @@ async def download(url: str) -> bytes:
         return response.content
 
 
-async def already_loaded(source: SyncSource, payload: bytes) -> bool:
-    """
-    Whether this exact file has already been loaded successfully.
+async def download_to(url: str, path: Path) -> Path:
+    """Stream a download to disk. The monthly extraction is ~80 MB zipped."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    async with httpx.AsyncClient(
+        timeout=settings.IMPORTER_HTTP_TIMEOUT, follow_redirects=True
+    ) as client:
+        async with client.stream("GET", url) as response:
+            response.raise_for_status()
+            with path.open("wb") as handle:
+                async for chunk in response.aiter_bytes(1 << 20):
+                    handle.write(chunk)
+    return path
+
 
-    Lets a daily job run unconditionally and do nothing when upstream has not
-    republished.
-    """
+async def url_exists(url: str) -> bool:
+    async with httpx.AsyncClient(
+        timeout=settings.IMPORTER_HTTP_TIMEOUT, follow_redirects=True
+    ) as client:
+        try:
+            response = await client.head(url)
+        except httpx.HTTPError:
+            return False
+        return response.status_code == 200
+
+
+async def latest_loaded_hash(source: SyncSource) -> str | None:
     latest = await settings.DB_CLIENT.synclogs.find_one(
-        {"source": source.value, "status": SyncStatus.SUCCESS.value},
+        {
+            "source": source.value,
+            "status": SyncStatus.SUCCESS.value,
+            "fileHash": {"$ne": None},
+        },
         sort=[("finishedAt", -1)],
         projection={"_id": 0, "fileHash": 1},
     )
-    return bool(latest) and latest.get("fileHash") == file_hash(payload)
+    return latest.get("fileHash") if latest else None
+
+
+async def already_loaded(source: SyncSource, payload: bytes) -> bool:
+    return await latest_loaded_hash(source) == file_hash(payload)

cbh/api/importer/services/codeboek/codeboek.py

@@ -1,22 +1,10 @@
-"""
-The Codeboek Douane importer.
-
-The Dutch national code lists are republished whole on every change, at
-predictable URLs of the form ``.../codebooks/[yyyyMMdd]/[filename]``, as either
-a zipped XML file or JSON. They are therefore replaced rather than merged: the
-previous contents of a codebook are dropped and the new file takes its place.
-
-These files carry no code x country -> document mapping. That lives in the
-TARIC measure conditions; the codebooks explain what a national code means.
-"""
-
 from datetime import date
 from typing import Any
 
 import httpx
 
 from cbh.api.common.db_requests import delete_objs, upsert_objs
-from cbh.api.common.utils import normalize_ws
+from cbh.api.common.utils import normalize_ws, parse_date
 from cbh.api.importer.dto import SyncMode, SyncSource
 from cbh.api.importer.services.base import SyncRun, already_loaded, download
 from cbh.api.importer.utils import (
@@ -27,8 +15,6 @@ from cbh.api.importer.utils import (
 )
 from cbh.api.taric.models import CodeboekEntryModel
 
-# Codebooks worth loading for the chapters in scope. The key is the file name
-# as published; the value is the codebook name records are filed under.
 CODEBOOKS: dict[str, str] = {
     "Bescheiden.zip": "documents",
     "Landen.zip": "countries",
@@ -38,7 +24,7 @@ CODEBOOKS: dict[str, str] = {
     "Maatstaven.zip": "supplementary_units",
 }
 
-_XML_FIELDS: dict[str, tuple[str, ...]] = {
+XML_FIELDS: dict[str, tuple[str, ...]] = {
     "code": ("code", "Code", "waarde", "id"),
     "description": ("omschrijving", "Omschrijving", "description", "naam"),
     "descriptionEn": ("descriptionEN", "omschrijvingEN", "descriptionEn"),
@@ -48,15 +34,10 @@ _XML_FIELDS: dict[str, tuple[str, ...]] = {
 
 
 def _parse_entry(codebook: str, record: dict[str, Any]) -> dict[str, Any] | None:
-    """
-    Parse one codebook record.
-    """
     code = normalize_ws(record.get("code"))
     if not code:
         return None
 
-    from cbh.api.common.utils import parse_date
-
     known = {"code", "description", "descriptionEn", "validFrom", "validTo"}
     return {
         "codebook": codebook,
@@ -66,22 +47,19 @@ def _parse_entry(codebook: str, record: dict[str, Any]) -> dict[str, Any] | None
         "validFrom": parse_date(record.get("validFrom")),
         "validTo": parse_date(record.get("validTo")),
         "extra": {
-            key: value for key, value in record.items() if key not in known and value not in (None, "")
+            key: value
+            for key, value in record.items()
+            if key not in known and value not in (None, "")
         },
     }
 
 
 def _read_records(payload: bytes, file_name: str) -> list[dict[str, Any]]:
-    """
-    Read a codebook in whichever format it was published in.
-    """
     unwrapped = load_payload(payload, file_name)
     if file_name.lower().endswith(".json") or unwrapped.lstrip()[:1] in (b"{", b"["):
         return read_json_records(unwrapped)
-    # The record tag varies per codebook; every leaf element carrying a code is
-    # treated as a record.
     for tag in ("record", "row", "item", "codeItem", "regel"):
-        records = list(read_xml_records(unwrapped, tag, _XML_FIELDS))
+        records = list(read_xml_records(unwrapped, tag, XML_FIELDS))
         if records:
             return records
     return []
@@ -93,9 +71,6 @@ async def import_codebook(
     file_name: str,
     source_url: str | None = None,
 ) -> SyncRun:
-    """
-    Replace one codebook with the contents of a published file.
-    """
     async with SyncRun(
         SyncSource.CODEBOEK_DOUANE,
         SyncMode.REPLACE,
@@ -120,8 +95,6 @@ async def import_codebook(
                 continue
             documents.append(CodeboekEntryModel(**parsed).to_mongo())
 
-        # Full replacement: the published file is the whole truth for this
-        # codebook, so anything not in it is gone.
         run.log.recordsDeleted = await delete_objs(
             CodeboekEntryModel, {"codebook": codebook}
         )
@@ -135,12 +108,7 @@ async def import_codebook(
         return run
 
 
-async def sync_codebook(
-    file_name: str, on_date: date | None = None
-) -> SyncRun:
-    """
-    Download and replace one codebook from its published URL.
-    """
+async def sync_codebook(file_name: str, on_date: date | None = None) -> SyncRun:
     codebook = CODEBOOKS.get(file_name, file_name.rsplit(".", 1)[0].lower())
     url = codeboek_url(file_name, on_date)
     payload = await download(url)
@@ -148,12 +116,6 @@ async def sync_codebook(
 
 
 async def sync_all_codebooks(on_date: date | None = None) -> list[SyncRun]:
-    """
-    Refresh every configured codebook.
-
-    A codebook that is not published for the requested date is skipped rather
-    than failing the whole run: the Douane does not republish them all daily.
-    """
     runs: list[SyncRun] = []
     for file_name in CODEBOOKS:
         try:

cbh/api/importer/services/demo/demo.py

@@ -1,24 +1,3 @@
-"""
-A small, realistic reference dataset for the chapters in scope.
-
-This exists so the whole pipeline — resolution, documents, warnings, Excel in
-and out — can be exercised before the real TARIC extracts and the Goenee gold
-set arrive. The shapes match what the importer produces from the published
-files; only the volume is different.
-
-It deliberately includes the awkward cases the engine has to get right:
-
-* an 8-digit code with two declarable children (ambiguous);
-* a code that expired last year, with a recorded successor (invalid + hint);
-* a code expiring inside the 60-day horizon (expiring);
-* an either/or document pair and a Y-code exemption;
-* an anti-dumping measure tied to an additional code;
-* a dual-use code that must be handed to a human.
-
-Everything here is illustrative and must be replaced by a real import before
-any acceptance run.
-"""
-
 from datetime import date, timedelta
 
 from cbh.api.common.db_requests import upsert_objs
@@ -41,13 +20,8 @@ from cbh.api.taric.models import (
     NomenclatureModel,
 )
 
-EPOCH = date(2020, 1, 1)
-
 
 def _geo_areas() -> list[GeoAreaModel]:
-    """
-    Countries and the groups used by the demo measures.
-    """
     countries = {
         "NL": "Netherlands",
         "DE": "Germany",
@@ -60,71 +34,56 @@ def _geo_areas() -> list[GeoAreaModel]:
         "ZA": "South Africa",
         "GB": "United Kingdom",
     }
-    areas = [
+    return [
         GeoAreaModel(
-            code=code, name=name, areaType=GeoAreaType.COUNTRY, validFrom=EPOCH
+            code=code, name=name, areaType=GeoAreaType.COUNTRY, validFrom=date(2020, 1, 1)
         )
         for code, name in countries.items()
-    ]
-    areas.append(
+    ] + [
         GeoAreaModel(
             code="1011",
             name="Erga Omnes (all third countries)",
             areaType=GeoAreaType.GROUP,
             memberCodes=list(countries),
-            validFrom=EPOCH,
-        )
-    )
-    areas.append(
+            validFrom=date(2020, 1, 1),
+        ),
         GeoAreaModel(
             code="1013",
             name="European Union",
             areaType=GeoAreaType.GROUP,
             memberCodes=["NL", "DE"],
-            validFrom=EPOCH,
-        )
-    )
-    areas.append(
+            validFrom=date(2020, 1, 1),
+        ),
         GeoAreaModel(
             code="2005",
             name="GSP beneficiary countries",
             areaType=GeoAreaType.GROUP,
             memberCodes=["BD", "IN"],
-            validFrom=EPOCH,
-        )
-    )
-    return areas
+            validFrom=date(2020, 1, 1),
+        ),
+    ]
 
 
 def _nomenclature(today: date) -> list[NomenclatureModel]:
-    """
-    A slice of the nomenclature across the chapters in scope.
-    """
-    soon = today + timedelta(days=30)
-
     entries: list[tuple[str, str, str, bool, date, date | None]] = [
-        # --- chapter 61/62: clothing (G-Star export) ---
-        ("6109", "T-shirts, singlets and other vests, knitted", "T-shirts en onderhemden", False, EPOCH, None),
-        ("61091000", "Of cotton", "Van katoen", False, EPOCH, None),
-        ("6109100010", "T-shirts of cotton, men's or boys'", "T-shirts van katoen, heren", True, EPOCH, None),
-        ("6109100090", "T-shirts of cotton, other", "T-shirts van katoen, overige", True, EPOCH, None),
-        ("62034231", "Men's trousers of denim", "Herenbroeken van denim", False, EPOCH, None),
-        ("6203423110", "Men's trousers of denim, workwear", "Herenbroeken van denim, werkkleding", True, EPOCH, None),
-        # --- chapter 84/85: IT/AV (EFG Arabia -> Henx, T1) ---
-        ("8471", "Automatic data-processing machines", "Automatische gegevensverwerkende machines", False, EPOCH, None),
-        ("84713000", "Portable computers weighing <= 10 kg", "Draagbare computers", False, EPOCH, None),
-        ("8471300000", "Portable automatic data-processing machines", "Draagbare computers", True, EPOCH, None),
-        ("8517130000", "Smartphones", "Smartphones", True, EPOCH, None),
-        ("8532241000", "Ceramic multilayer capacitors", "Keramische meerlaagse condensatoren", True, EPOCH, None),
-        ("8543709099", "Other electrical machines, n.e.s.", "Andere elektrische machines", True, EPOCH, None),
-        # Expires inside the warning horizon: the 1 January switch, seen early.
-        ("8528721010", "Colour television sets, legacy subheading", "Kleurentelevisies, oude onderverdeling", True, EPOCH, soon),
-        # --- chapter 22: wine ---
-        ("22042110", "Wine in bottles of 2 litres or less", "Wijn in flessen van 2 liter of minder", False, EPOCH, None),
-        ("2204211010", "Quality wine in bottles of 2 litres or less", "Kwaliteitswijn in flessen", True, EPOCH, None),
+        ("6109", "T-shirts, singlets and other vests, knitted", "T-shirts en onderhemden", False, date(2020, 1, 1), None),
+        ("61091000", "Of cotton", "Van katoen", False, date(2020, 1, 1), None),
+        ("6109100010", "T-shirts of cotton, men's or boys'", "T-shirts van katoen, heren", True, date(2020, 1, 1), None),
+        ("6109100090", "T-shirts of cotton, other", "T-shirts van katoen, overige", True, date(2020, 1, 1), None),
+        ("62034231", "Men's trousers of denim", "Herenbroeken van denim", False, date(2020, 1, 1), None),
+        ("6203423110", "Men's trousers of denim, workwear", "Herenbroeken van denim, werkkleding", True, date(2020, 1, 1), None),
+        ("8471", "Automatic data-processing machines", "Automatische gegevensverwerkende machines", False, date(2020, 1, 1), None),
+        ("84713000", "Portable computers weighing <= 10 kg", "Draagbare computers", False, date(2020, 1, 1), None),
+        ("8471300000", "Portable automatic data-processing machines", "Draagbare computers", True, date(2020, 1, 1), None),
+        ("8517130000", "Smartphones", "Smartphones", True, date(2020, 1, 1), None),
+        ("8532241000", "Ceramic multilayer capacitors", "Keramische meerlaagse condensatoren", True, date(2020, 1, 1), None),
+        ("8543709099", "Other electrical machines, n.e.s.", "Andere elektrische machines", True, date(2020, 1, 1), None),
+        ("8528721010", "Colour television sets, legacy subheading", "Kleurentelevisies, oude onderverdeling", True, date(2020, 1, 1), today + timedelta(days=30)),
+        ("22042110", "Wine in bottles of 2 litres or less", "Wijn in flessen van 2 liter of minder", False, date(2020, 1, 1), None),
+        ("2204211010", "Quality wine in bottles of 2 litres or less", "Kwaliteitswijn in flessen", True, date(2020, 1, 1), None),
     ]
 
-    models = [
+    return [
         NomenclatureModel(
             code=code if len(code) == 10 else code.ljust(10, "0"),
             descriptionEn=description_en,
@@ -135,27 +94,19 @@ def _nomenclature(today: date) -> list[NomenclatureModel]:
             validTo=valid_to,
         )
         for code, description_en, description_nl, declarable, valid_from, valid_to in entries
-    ]
-
-    # A code withdrawn at a previous 1 January, kept so the engine can tell
-    # "never existed" from "no longer valid".
-    models.append(
+    ] + [
         NomenclatureModel(
             code="8528721090",
             descriptionEn="Colour television sets, withdrawn subheading",
             descriptionNl="Kleurentelevisies, vervallen onderverdeling",
             declarable=True,
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
             validTo=date(today.year - 1, 12, 31),
         )
-    )
-    return models
+    ]
 
 
 def _document_codes() -> list[DocumentCodeModel]:
-    """
-    The Box 44 codes the demo measures point at.
-    """
     entries = [
         ("C015", "V I 1 document (wine import)", "V I 1-document (wijninvoer)"),
         ("C400", "CITES certificate", "CITES-certificaat"),
@@ -172,40 +123,36 @@ def _document_codes() -> list[DocumentCodeModel]:
     ]
     return [
         DocumentCodeModel(
-            code=code, description=description, descriptionNl=description_nl, validFrom=EPOCH
+            code=code,
+            description=description,
+            descriptionNl=description_nl,
+            validFrom=date(2020, 1, 1),
         )
         for code, description, description_nl in entries
     ]
 
 
 def _additional_codes() -> list[AdditionalCodeModel]:
-    """
-    Additional codes used by the demo anti-dumping measure.
-    """
     return [
         AdditionalCodeModel(
             codeType="8",
             code="B999",
             fullCode="8B999",
             description="All other producers (residual anti-dumping rate)",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
         ),
         AdditionalCodeModel(
             codeType="8",
             code="B001",
             fullCode="8B001",
             description="Named cooperating producer (reduced rate)",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
         ),
     ]
 
 
 def _measures() -> list[MeasureModel]:
-    """
-    Measures covering the interesting paths through the engine.
-    """
     return [
-        # Wine import: either a V I 1 document or the exemption code.
         MeasureModel(
             measureSid="DEMO-2204-IMP",
             goodsCode="22042110",
@@ -215,7 +162,7 @@ def _measures() -> list[MeasureModel]:
             direction=TradeDirection.IMPORT,
             geoAreaCode="1011",
             geoAreaDescription="Erga Omnes",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
             conditions=[
                 MeasureConditionEmbedded(
                     conditionCode="B",
@@ -235,7 +182,6 @@ def _measures() -> list[MeasureModel]:
                 ),
             ],
         ),
-        # Wine is excise goods: information only, no document.
         MeasureModel(
             measureSid="DEMO-2204-EXC",
             goodsCode="22042110",
@@ -244,10 +190,8 @@ def _measures() -> list[MeasureModel]:
             category=MeasureCategory.EXCISE,
             direction=TradeDirection.IMPORT,
             geoAreaCode="1011",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
         ),
-        # Anti-dumping tied to an additional code: the rate depends on the
-        # producer, so the line has to carry one.
         MeasureModel(
             measureSid="DEMO-8532-ADD",
             goodsCode="8532241000",
@@ -259,7 +203,7 @@ def _measures() -> list[MeasureModel]:
             geoAreaDescription="China",
             additionalCode="8B999",
             dutyExpression="32.4 %",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
             conditions=[
                 MeasureConditionEmbedded(
                     conditionCode="A",
@@ -271,7 +215,6 @@ def _measures() -> list[MeasureModel]:
                 ),
             ],
         ),
-        # Dual-use: flagged as out of scope rather than answered.
         MeasureModel(
             measureSid="DEMO-8543-DU",
             goodsCode="8543709099",
@@ -280,7 +223,7 @@ def _measures() -> list[MeasureModel]:
             category=MeasureCategory.DUAL_USE,
             direction=TradeDirection.EXPORT,
             geoAreaCode="1011",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
             conditions=[
                 MeasureConditionEmbedded(
                     conditionCode="E",
@@ -300,8 +243,6 @@ def _measures() -> list[MeasureModel]:
                 ),
             ],
         ),
-        # Clothing export: a surveillance measure with no document obligation,
-        # so the line legitimately comes back with none required.
         MeasureModel(
             measureSid="DEMO-6109-EXP",
             goodsCode="6109",
@@ -310,9 +251,8 @@ def _measures() -> list[MeasureModel]:
             category=MeasureCategory.SURVEILLANCE,
             direction=TradeDirection.EXPORT,
             geoAreaCode="1011",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
         ),
-        # Preferential duty for GSP countries: fiscal only.
         MeasureModel(
             measureSid="DEMO-6109-GSP",
             goodsCode="61091000",
@@ -323,33 +263,12 @@ def _measures() -> list[MeasureModel]:
             geoAreaCode="2005",
             geoAreaDescription="GSP beneficiaries",
             dutyExpression="9.6 %",
-            validFrom=EPOCH,
+            validFrom=date(2020, 1, 1),
         ),
     ]
 
 
-def _code_changes(today: date) -> list[CodeChangeModel]:
-    """
-    The correlation entry behind the withdrawn-code demo case.
-    """
-    return [
-        CodeChangeModel(
-            oldCode="8528721090",
-            newCodes=["8528721010"],
-            changeType=CodeChangeType.REPLACED,
-            changedOn=date(today.year, 1, 1).isoformat(),
-            note="Subheading merged at the 1 January nomenclature update.",
-        )
-    ]
-
-
 async def seed_demo_reference_data() -> SyncRun:
-    """
-    Load the demo reference dataset.
-
-    Written through the same idempotent upsert path as a real import, under a
-    sync log that says plainly where the data came from.
-    """
     today = date.today()
 
     async with SyncRun(
@@ -366,7 +285,19 @@ async def seed_demo_reference_data() -> SyncRun:
             (DocumentCodeModel, _document_codes(), ["code", "validFrom"]),
             (AdditionalCodeModel, _additional_codes(), ["fullCode", "validFrom"]),
             (MeasureModel, _measures(), ["measureSid"]),
-            (CodeChangeModel, _code_changes(today), ["oldCode", "changedOn"]),
+            (
+                CodeChangeModel,
+                [
+                    CodeChangeModel(
+                        oldCode="8528721090",
+                        newCodes=["8528721010"],
+                        changeType=CodeChangeType.REPLACED,
+                        changedOn=date(today.year, 1, 1).isoformat(),
+                        note="Subheading merged at the 1 January nomenclature update.",
+                    )
+                ],
+                ["oldCode", "changedOn"],
+            ),
         ]
 
         for model, instances, key_fields in batches:

cbh/api/importer/services/dtv/__init__.py

@@ -0,0 +1,15 @@
+from .dtv import (
+    DtvRefresh,
+    import_national_measures,
+    list_publications,
+    refresh_national_measures,
+    resolve_latest_snapshot,
+)
+
+__all__ = [
+    "DtvRefresh",
+    "import_national_measures",
+    "list_publications",
+    "refresh_national_measures",
+    "resolve_latest_snapshot",
+]

cbh/api/importer/services/dtv/dtv.py

@@ -0,0 +1,368 @@
+"""The Dutch Gebruikstarief — national measures on top of the Union ones.
+
+TARIC states what the Union requires. What the Netherlands adds on its own
+authority — omzetbelasting, accijns, verbruiksbelasting, the invoer- and
+uitvoerbeperkingen written in Dutch law — is published only here, by the
+Belastingdienst, as an Arctic Tariff export.
+
+The feed carries the whole tariff, Union rules included. Only the records
+flagged `national="1"` are taken: the Union side is already loaded from the
+published TARIC extraction and reloading it from a second source would make it
+impossible to say which publication an answer came from.
+"""
+
+import zipfile
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Iterator
+from xml.etree import ElementTree
+
+import httpx
+
+from cbh.api.common.dto import TradeDirection
+from cbh.api.common.utils import parse_date
+from cbh.api.importer.dto import SyncMode, SyncSource
+from cbh.api.importer.services.base import SyncRun, download_to, latest_loaded_hash
+from cbh.api.importer.utils import file_hash_path
+from cbh.api.taric.dto import MeasureCategory
+from cbh.api.taric.models import (
+    DocumentCodeModel,
+    MeasureConditionEmbedded,
+    MeasureModel,
+    MeasureTypeModel,
+)
+from cbh.core.config import settings
+
+NS = "{http://www.arcticgroup.se/tariff/arctictariff/export}"
+
+# Arctic's `tradeMovementCode`, read off the published MeasureType table: the
+# NLINV* (invoer) types are 0, the NLUIT* (uitvoer) types are 1.
+TRADE_MOVEMENT = {
+    "0": TradeDirection.IMPORT,
+    "1": TradeDirection.EXPORT,
+    "2": TradeDirection.BOTH,
+}
+
+# The series a national measure type belongs to already says what kind of rule
+# it is, so the fallback is data rather than a guess: A prohibits, B restricts,
+# P is turnover tax, Q is one of the consumption taxes.
+SERIES_CATEGORY = {
+    "A": MeasureCategory.PROHIBITION,
+    "B": MeasureCategory.RESTRICTION,
+    "P": MeasureCategory.VAT,
+    "Q": MeasureCategory.EXCISE,
+}
+
+# Where the series is too coarse to be useful, the type is named outright.
+NATIONAL_TYPE_CATEGORY = {
+    "NLBTW": MeasureCategory.VAT,
+    "NLACC": MeasureCategory.EXCISE,
+    "NLACVH": MeasureCategory.EXCISE,
+    "NLVBB": MeasureCategory.EXCISE,
+    "NLKOBE": MeasureCategory.EXCISE,
+    "NLINVF": MeasureCategory.PHYTOSANITARY_CONTROL,
+    "NLUSGM": MeasureCategory.DUAL_USE,
+    "NLUITT": MeasureCategory.DUAL_USE,
+    "NLINVE": MeasureCategory.DUAL_USE,
+    "NLUITE": MeasureCategory.DUAL_USE,
+}
+
+
+@dataclass
+class NationalMeasureType:
+    typeId: str
+    direction: TradeDirection
+    category: MeasureCategory
+    description: str | None = None
+    descriptionNl: str | None = None
+
+
+@dataclass
+class DtvRefresh:
+    fileName: str | None = None
+    url: str | None = None
+    runs: list[SyncRun] = field(default_factory=list)
+    note: str | None = None
+
+
+def _attrs(element: ElementTree.Element) -> dict[str, str]:
+    return {key.split("}")[-1]: value for key, value in element.attrib.items()}
+
+
+def _descriptions(element: ElementTree.Element, tag: str) -> dict[str, str]:
+    found: dict[str, str] = {}
+    for child in element.iter(NS + tag):
+        attributes = _attrs(child)
+        language = (attributes.get("languageId") or "").upper()
+        if language and attributes.get("description"):
+            found[language] = attributes["description"]
+    return found
+
+
+def document_code(condition: dict[str, str]) -> str | None:
+    """Box 44 writes the certificate type and its number as one code.
+
+    An EU certificate comes out as `N851`, a Dutch national one as `1550`; the
+    type is the first character either way.
+    """
+    code = (condition.get("certificateCode") or "").strip()
+    if not code:
+        return None
+    return f"{(condition.get('certificateType') or '').strip()}{code}"
+
+
+def _member(archive: zipfile.ZipFile, marker: str) -> str | None:
+    return next((name for name in archive.namelist() if marker in name), None)
+
+
+def parse_measure_types(payload: bytes) -> dict[str, NationalMeasureType]:
+    root = ElementTree.fromstring(payload)
+    types: dict[str, NationalMeasureType] = {}
+    for element in root.iter(NS + "measureType"):
+        attributes = _attrs(element)
+        if attributes.get("national") != "1":
+            continue
+        type_id = attributes.get("measureType")
+        if not type_id:
+            continue
+        descriptions = _descriptions(element, "measureTypeDescription")
+        category = NATIONAL_TYPE_CATEGORY.get(
+            type_id,
+            SERIES_CATEGORY.get(
+                attributes.get("measureTypeSeriesId") or "", MeasureCategory.OTHER
+            ),
+        )
+        types[type_id] = NationalMeasureType(
+            typeId=type_id,
+            direction=TRADE_MOVEMENT.get(
+                attributes.get("tradeMovementCode") or "", TradeDirection.BOTH
+            ),
+            category=category,
+            description=descriptions.get("EN") or descriptions.get("NL"),
+            descriptionNl=descriptions.get("NL"),
+        )
+    return types
+
+
+def parse_certificates(payload: bytes) -> list[dict[str, Any]]:
+    root = ElementTree.fromstring(payload)
+    documents: list[dict[str, Any]] = []
+    for element in root.iter(NS + "certificate"):
+        attributes = _attrs(element)
+        if attributes.get("national") != "1":
+            continue
+        code = document_code(
+            {
+                "certificateType": attributes.get("certificateType"),
+                "certificateCode": attributes.get("certificateCode"),
+            }
+        )
+        if not code:
+            continue
+        descriptions = _descriptions(element, "certificateDescription")
+        documents.append(
+            DocumentCodeModel(
+                code=code,
+                description=descriptions.get("EN") or descriptions.get("NL"),
+                descriptionNl=descriptions.get("NL"),
+                national=True,
+                validFrom=parse_date(attributes.get("dateStart")),
+                validTo=parse_date(attributes.get("dateEnd")),
+            ).to_mongo()
+        )
+    return documents
+
+
+def iter_national_measures(
+    archive_path: Path, member: str, types: dict[str, NationalMeasureType]
+) -> Iterator[dict[str, Any]]:
+    """Stream the measure table, keeping only the national records.
+
+    The full snapshot's measure file is about three gigabytes once unpacked, so
+    it is never held in memory or written out: it is parsed straight off the
+    archive and each element is dropped as soon as it has been read.
+    """
+    with zipfile.ZipFile(archive_path) as archive, archive.open(member) as stream:
+        for _, element in ElementTree.iterparse(stream, events=("end",)):
+            if element.tag != NS + "measure":
+                continue
+            attributes = _attrs(element)
+            if attributes.get("national") != "1" or attributes.get("stoppedFlag") == "1":
+                element.clear()
+                continue
+
+            parsed = _parse_measure(element, attributes, types)
+            element.clear()
+            if parsed:
+                yield parsed
+
+
+def _parse_measure(
+    element: ElementTree.Element,
+    attributes: dict[str, str],
+    types: dict[str, NationalMeasureType],
+) -> dict[str, Any] | None:
+    goods_code = (attributes.get("goodsNomenclatureCode") or "").strip()
+    type_id = (attributes.get("measureType") or "").strip()
+    sid = (attributes.get("SID") or "").strip()
+    if not goods_code or not type_id or not sid:
+        return None
+
+    measure_type = types.get(type_id)
+    conditions: list[dict[str, Any]] = []
+    for child in element.iter(NS + "measureCondition"):
+        condition = _attrs(child)
+        code = condition.get("conditionCodeId")
+        if not code:
+            continue
+        conditions.append(
+            MeasureConditionEmbedded(
+                conditionCode=code,
+                sequenceNumber=int(condition.get("sequenceNumber") or 1),
+                documentCode=document_code(condition),
+                actionCode=condition.get("actionCode"),
+            ).to_mongo()
+        )
+
+    return MeasureModel(
+        # Prefixed because the Belastingdienst numbers its own records and those
+        # numbers mean nothing in the TARIC publication we load beside them.
+        measureSid=f"dtv:{sid}",
+        goodsCode=goods_code,
+        additionalCode=(attributes.get("additionalCodeId") or "").strip() or None,
+        measureTypeId=type_id,
+        measureTypeDescription=measure_type.description if measure_type else None,
+        category=measure_type.category if measure_type else MeasureCategory.OTHER,
+        direction=measure_type.direction if measure_type else TradeDirection.BOTH,
+        geoAreaCode=(attributes.get("geographicalAreaId") or "").strip(),
+        dutyExpression=(attributes.get("expression") or "").strip() or None,
+        regulationId=(attributes.get("regulationId") or "").strip() or None,
+        national=True,
+        conditions=conditions,
+        validFrom=parse_date(attributes.get("dateStart")),
+        validTo=parse_date(attributes.get("dateEnd")),
+    ).to_mongo()
+
+
+async def list_publications() -> list[tuple[str, str]]:
+    async with httpx.AsyncClient(
+        timeout=settings.IMPORTER_HTTP_TIMEOUT, follow_redirects=True
+    ) as client:
+        response = await client.get(settings.DTV_MANIFEST_URL)
+        response.raise_for_status()
+    root = ElementTree.fromstring(response.content)
+    return [
+        (entry.findtext("bestand") or "", entry.findtext("url") or "")
+        for entry in root.iter("download")
+        if entry.findtext("url")
+    ]
+
+
+async def resolve_latest_snapshot() -> tuple[str, str] | None:
+    """Pick the newest full snapshot without downloading any of them.
+
+    The manifest lists nightly increments and full snapshots side by side with
+    nothing to tell them apart but their size, which the server reports.
+    """
+    publications = await list_publications()
+    async with httpx.AsyncClient(
+        timeout=settings.IMPORTER_HTTP_TIMEOUT, follow_redirects=True
+    ) as client:
+        for name, url in reversed(publications):
+            try:
+                head = await client.head(url)
+            except httpx.HTTPError:
+                continue
+            size = int(head.headers.get("content-length") or 0)
+            if size >= settings.DTV_FULL_SNAPSHOT_MIN_BYTES:
+                return name, url
+    return None
+
+
+async def import_national_measures(
+    archive_path: Path, file_name: str, mode: SyncMode = SyncMode.FULL
+) -> SyncRun:
+    async with SyncRun(SyncSource.DTV_NATIONAL, mode, file_name=file_name) as run:
+        run.log.fileHash = file_hash_path(archive_path)
+        with zipfile.ZipFile(archive_path) as archive:
+            type_member = _member(archive, "/MeasureType_")
+            certificate_member = _member(archive, "/Certificate_")
+            measure_member = _member(archive, "/Measure_")
+
+            if not measure_member or not type_member:
+                raise ValueError(
+                    "This archive holds no full measure table — it is a nightly "
+                    "increment, not a snapshot."
+                )
+
+            types = parse_measure_types(archive.read(type_member))
+            certificates = (
+                parse_certificates(archive.read(certificate_member))
+                if certificate_member
+                else []
+            )
+
+        run.log.note = f"{len(types)} national measure types."
+
+        await run.write(
+            MeasureTypeModel,
+            [
+                MeasureTypeModel(
+                    typeId=item.typeId,
+                    description=item.description,
+                    descriptionNl=item.descriptionNl,
+                    category=item.category,
+                    direction=item.direction,
+                ).to_mongo()
+                for item in types.values()
+            ],
+            ["typeId"],
+        )
+
+        if certificates:
+            await run.write(DocumentCodeModel, certificates, ["code", "validFrom"])
+
+        batch: list[dict[str, Any]] = []
+        for measure in iter_national_measures(archive_path, measure_member, types):
+            batch.append(measure)
+            if len(batch) >= 2000:
+                await run.write(MeasureModel, batch, ["measureSid"])
+                batch = []
+        if batch:
+            await run.write(MeasureModel, batch, ["measureSid"])
+
+        return run
+
+
+async def refresh_national_measures(
+    mode: SyncMode = SyncMode.FULL, keep_archive: bool = False
+) -> DtvRefresh:
+    refresh = DtvRefresh()
+
+    resolved = await resolve_latest_snapshot()
+    if resolved is None:
+        refresh.note = (
+            "No full snapshot found in the Gebruikstarief feed; only increments "
+            f"are listed at {settings.DTV_MANIFEST_URL}."
+        )
+        return refresh
+
+    refresh.fileName, refresh.url = resolved
+    archive_path = settings.REFERENCE_STORAGE_DIR / "dtv_national" / refresh.fileName
+    await download_to(refresh.url, archive_path)
+
+    try:
+        if await latest_loaded_hash(SyncSource.DTV_NATIONAL) == file_hash_path(
+            archive_path
+        ):
+            refresh.note = "Unchanged since the last successful import; nothing loaded."
+            return refresh
+
+        refresh.runs.append(
+            await import_national_measures(archive_path, refresh.fileName, mode)
+        )
+    finally:
+        if not keep_archive:
+            archive_path.unlink(missing_ok=True)
+
+    return refresh