Squash history to reclaim LFS storage (1 GB cap); tree unchanged

Co-authored-by: Beemer <Beemer@users.noreply.huggingface.co>
Co-authored-by: claude fable 5 <claude fable 5@users.noreply.huggingface.co>

.dockerignore

@@ -0,0 +1,18 @@
+# Keep the build context small and keep secrets / regenerated files out.
+.git/
+.venv/
+venv/
+__pycache__/
+**/__pycache__/
+*.pyc
+
+# Raw downloads are not needed -- only data/processed/ is.
+data/raw/
+
+# Local-only: the CanLII key (injected as a runtime secret) and disk caches.
+canlii_key.txt
+data/citator_dbmap.json
+data/citator_cache.json
+
+# Local MCP client config; not used by the remote server.
+.mcp.json

.gitattributes

@@ -0,0 +1,2 @@
+data/processed/caselaw.json filter=lfs diff=lfs merge=lfs -text
+data/processed/embeddings.npz filter=lfs diff=lfs merge=lfs -text

.github/workflows/ci.yml

@@ -0,0 +1,26 @@
+# CI for the GitHub mirror (create the mirror per RUNBOOK.md; this workflow
+# activates the moment the repo exists). Runs the offline test suite and the
+# curated-dataset schema validation on every push -- the cheap failures that
+# should never reach the production Space. The full retrieval eval needs the
+# ONNX models and LFS embeddings, so it stays a manual/scheduled job.
+name: tests
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  offline-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          lfs: false          # tests are offline; embeddings not needed
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install minimal deps (no ONNX/model downloads)
+        run: pip install snowballstemmer pydantic "mcp>=1.0" beautifulsoup4 pypdf
+      - name: Offline test suite
+        run: python -m unittest discover -s tests -v

.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+.venv/
+venv/
+data/raw/
+canlii_key.txt
+data/citator_*.json
+US-Dispositions-Reference.pdf
+*.bundle

.mcp.json

@@ -0,0 +1,11 @@
+{
+  "mcpServers": {
+    "canlex": {
+      "command": "C:\\projects\\CanLex\\.venv\\Scripts\\python.exe",
+      "args": ["C:\\projects\\CanLex\\canlex\\server.py"],
+      "env": {
+        "CANLEX_LOG_DIR": "C:\\projects\\CanLex\\data\\logs"
+      }
+    }
+  }
+}

Dockerfile

@@ -0,0 +1,58 @@
+# CanLex MCP server -- remote (streamable-HTTP) image.
+#
+# Builds anywhere: Hugging Face Spaces, Google Cloud Run, Fly.io, plain Docker.
+# Retrieval is fully local and key-free; the optional CanLII case citator reads
+# its key from the CANLII_API_KEY environment variable (supplied as a host
+# secret -- the key is never copied into the image).
+FROM python:3.12-slim
+
+# libgomp1 is the OpenMP runtime that onnxruntime (the reranker) links against.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends libgomp1 \
+ && rm -rf /var/lib/apt/lists/*
+
+# Run as a non-root user (UID 1000) -- required by Hugging Face Spaces.
+RUN useradd --create-home --home-dir /app --uid 1000 app
+WORKDIR /app
+
+# Python dependencies first, so this layer caches across code changes.
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Application code, the processed corpus (section-chunk JSON), and the prebuilt
+# semantic embeddings (embeddings.npz, committed via git-LFS). Shipping the
+# embeddings as an artifact keeps the build fast: embedding ~20k chunks on the
+# build host took ~33 min and was tripping Hugging Face's job timeout as the
+# corpus grew. Regenerate with 'python -m canlex.embed' after any corpus change
+# and commit the result.
+COPY --chown=app:app canlex/ ./canlex/
+COPY --chown=app:app data/processed/*.json ./data/processed/
+COPY --chown=app:app data/processed/embeddings.npz ./data/processed/
+# Curated commentary datasets (the canlex_us_disposition tool reads the
+# structured source file directly, not just its processed chunks).
+COPY --chown=app:app data/curated/ ./data/curated/
+
+USER app
+# CANLEX_LOG_DIR turns on the opt-in query log (query text + outcome only;
+# loganalyze.py mines it for corpus gaps). On the free tier this path is
+# ephemeral -- lost on rebuild/sleep -- so treat it as best-effort telemetry;
+# the durable log is the local server's (see .mcp.json).
+ENV HOME=/app \
+    HF_HOME=/app/.hf_cache \
+    CANLEX_HTTP=1 \
+    CANLEX_LOG_DIR=/tmp/canlex-logs \
+    PORT=7860 \
+    PYTHONUNBUFFERED=1
+
+# Pre-fetch the embedder and cross-encoder models so the cache is baked into the
+# image and the first request needs no network. The corpus is NOT re-embedded
+# here -- the committed embeddings.npz is used as-is; only the bge query model
+# (for embedding live queries) and the reranker are downloaded.
+RUN python -c "from canlex.embed import Embedder; Embedder()" \
+ && python -c "from canlex.rerank import Reranker; Reranker()"
+
+# From here on, model files are served from the baked cache, never fetched.
+ENV HF_HUB_OFFLINE=1
+
+EXPOSE 7860
+CMD ["python", "-m", "canlex.server"]

README.md

@@ -0,0 +1,56 @@
+---
+title: CanLex MCP
+sdk: docker
+app_port: 7860
+pinned: false
+---
+
+# CanLex MCP
+
+A remote [Model Context Protocol](https://modelcontextprotocol.io) server for
+Canadian legal research, built for front-line border-services work. The
+legislation corpus is **bilingual**: every Act and regulation also carries its
+parallel French text, and a French-language query searches both languages
+automatically. Ten read-only tools over streamable HTTP (or local stdio):
+
+- **canlex_search_legislation** — hybrid (BM25 + semantic) search with a
+  cross-encoder reranker over the whole corpus: federal border, customs,
+  criminal, firearms, drug, food/health, labour and privacy legislation with
+  their Schedules; the ENF operational manuals; CBSA D-Memoranda and the AMPS
+  Master Penalty Document; the Charter; Treasury Board collective agreements;
+  NJC directives; ~110 leading decisions; delegation instruments;
+  benefit-plan booklets; and CanLex's curated commentary.
+- **canlex_get_section** — fetch one provision verbatim by Act and section
+  (accepts 's. 34', '34(1)(c)', 'Schedule I').
+- **canlex_us_disposition** — step 1 of the IRPA s. 36 screening chain: is a
+  US disposition a "conviction"? 51-jurisdiction state-by-state dataset.
+- **canlex_us_equivalency** — step 2: the Canadian equivalent offence, its
+  verified maximum penalty, and the inadmissibility branch.
+- **canlex_rehabilitation** — step 3: deemed rehabilitation / eligibility to
+  apply, under IRPA s. 36(3)(c) and IRPR ss. 17–18.1.
+- **canlex_list_acts** — the corpus inventory and its currency.
+- **canlex_whats_new** — provisions amended or newly in force since a date.
+- **canlex_citing** — reverse cross-references: what cites a given section.
+- **canlex_toc** — an Act's table of contents (Parts and marginal notes).
+- **canlex_case** — live CanLII case citator (citation graph by neutral
+  citation or URL).
+
+The server also publishes an MCP prompt (`us_record_screening`, the guided
+three-step screening flow) and resources (the corpus inventory and the
+conviction/equivalency methodology texts) that any MCP client can use.
+
+The labour side now carries the full union stack alongside the FB collective
+agreement: the CIU National By-Laws (EN/FR), CIU Regulations, Branch Rules
+and local modifications, the Branch Elections Kit, the PSAC Constitution,
+the union's member guides, and every published site VSSA — all bannered as
+bargaining-agent documents, distinct from employer policy.
+
+Retrieval runs fully locally inside the container and needs no API key. The
+optional case citator reads a CanLII key from the `CANLII_API_KEY` environment
+variable. A weekly scheduled staleness check watches upstream sources for
+drift; see RUNBOOK.md for the maintainer's refresh choreography.
+
+Add this Space's `/mcp` URL as a custom connector in any Claude client, or
+point any MCP-capable agent (Gemini in Antigravity, etc.) at it. A companion
+Gradio webapp (`webapp/`, deployed as its own Space) gives the same corpus a
+Gemini-powered agentic chat front-end.

RUNBOOK.md

@@ -0,0 +1,230 @@
+# CanLex maintainer runbook
+
+Everything a future maintainer needs that the code alone does not say.
+
+## The refresh choreography
+
+Order matters — chunks feed embeddings feed the eval:
+
+```
+py -m canlex.ingest [--force] [CODES]   # Justice Laws XML -> per-Act JSON (incl. Schedules)
+py -m canlex.ingest --fr                # parallel French corpus (same schema; SOR->DORS URLs)
+py -m canlex.pdi                        # IRCC PDI trees (TR / refugee / standard /
+                                        #   identity / PR-card; capped BFS, drops logged)
+py -m canlex.ciu                        # CIU/PSAC union documents + site VSSAs
+py -m canlex.dmemo [--fr]               # CBSA D-Memoranda (EN; --fr = Mémorandums D)
+py -m canlex.enf                        # ENF operational manuals (canada.ca PDFs)
+py -m canlex.amps                       # AMPS Master Penalty Document
+py -m canlex.charter                    # Charter / Constitution HTML
+py -m canlex.caselaw                    # Lexum/Decisia courts + cached CanLII texts
+py -m canlex.delegation                 # CBSA/IRCC delegation instruments
+py -m canlex.directive [--fr]           # NJC directives (EN; --fr = éditions françaises)
+py -m canlex.agreement                  # FB collective agreement
+py -m canlex.benefits                   # benefit-plan booklets
+py -m canlex.tariff_schedule            # Customs Tariff ch. 98/99 (CBSA edition)
+py -m canlex.commentary                 # curated datasets -> commentary chunks
+py -m canlex.embed                      # re-embed EVERYTHING (run after ANY of the above)
+py -m unittest discover -s tests        # all offline
+py -m canlex.eval                       # gate: no metric below the last committed run
+```
+
+`canlex.embed` is mandatory after any chunk change: the index zero-fills
+vectors for chunks missing from embeddings.npz and quietly degrades them to
+BM25-only (a stderr warning is the only signal).
+
+### The corpus-write guard
+
+Every ingester above rebuilds its whole processed file from a fresh scrape, so
+an upstream page that changes shape turns into data loss at the write. All of
+them now go through `_common.write_corpus`, which **refuses to write a corpus
+below 80% of the stored one** and makes the process exit non-zero:
+
+```
+  REFUSING to write: 0 chunks would replace 1870 already in dmemos.json. ...
+```
+
+That is not a transient error — it means the scrape broke. Diagnose it, don't
+retry. Once the drop is understood and intended (a source genuinely shrank),
+re-run the same module with `--allow-shrink`. Because the runbook chains these
+as `py -m canlex.X && py -m canlex.embed`, a refusal now correctly stops the
+embed instead of propagating a gutted corpus.
+
+Most ingesters also **preserve on failure**: an item that errors keeps its
+last-good chunks (`_common.preserve_dropped`), restricted to items that
+actually failed, so anything removed on purpose stays gone. Deliberate
+exception: `enf.py` treats an HTTP 404 as IRCC retiring a chapter — its
+documented policy — so a 404 drops the chapter while any other error preserves
+it. Writes are atomic (tmp + `os.replace`): a killed run cannot leave truncated
+JSON that would read as an empty corpus and disable the guard next time.
+
+## Deploys
+
+- **MCP Space** (`Beemer0/CanLex`): commit, then `git push space main`.
+  The Space rebuilds automatically. `embeddings.npz` ships in-repo via
+  git-LFS — a fresh clone needs `git lfs install`.
+- **Webapp Space** (`Beemer0/Canlex-web`): separate repo. Clone it, copy
+  `webapp/app.py` + `webapp/requirements.txt` + `webapp/Dockerfile` +
+  `webapp/README.md` over, commit, push. Do NOT use `hf upload` — HF gates
+  it behind PRO for Docker/Gradio Spaces; git pushes work on the free tier.
+
+## Secrets and variables (set in each Space's settings)
+
+| Where | Name | Purpose |
+|---|---|---|
+| MCP Space | `CANLII_API_KEY` | optional, enables canlex_case |
+| Webapp | `GEMINI_API_KEY` | required |
+| Webapp | `CANLEX_WEB_AUTH` | required (`user:pass` per line); the app refuses to start without it |
+| Webapp | `CANLEX_GEMINI_MODEL` | optional; the deployed Space sets `gemini-2.5-pro` (code default is flash) |
+
+Locally, the CanLII key lives in `canlii_key.txt` (gitignored — never
+commit it).
+
+## The staleness watch
+
+`py -m canlex.refresh` (alias `--check`) diffs upstream: Justice Laws
+consolidation dates, CBSA/IRCC delegation instruments, NJC directive dates,
+the newest per-memo `LAST_UPDATED` in the D-memo datasource, curated-dataset
+age (>180 days → AGING) and curated-vs-chunk regen drift. Non-zero exit =
+something drifted.
+A scheduled task runs it Mondays 03:01 local — if this machine is off, the
+check silently doesn't happen that week. Note: network fetch errors on the
+legislation rows DO fail the check (exit 1) so a broken watch can't pass
+silently.
+
+`_PENDING_BILLS` in canlex/refresh.py names bills heading into force that
+affect corpus Acts — edit that list when a new one appears (source-code edit,
+deliberate). A bill stays listed after its in-force date until Justice Laws
+consolidates it: C-16 came into force 2026-07-18 but the 2026-06-14
+consolidation predates it, so the coercive-control provisions are NOT in the
+corpus yet and the notice correctly keeps printing.
+
+## The CBSA D-memo index (rebuilt 2026-07-10)
+
+CBSA replaced the static index with a WET DataTable: the page ships **no memo
+links at all**, the rows come from `dmemoranda-datasource-eng.json` beside it.
+`dmemo.memo_urls` reads that datasource and falls back to scraping the page
+only if it disappears. Two consequences worth knowing:
+
+- Cancelled memos (`STATUS: "Cancelled"`, 34 of 398) are skipped — retired
+  guidance must not retrieve as current. D10-14-48 left the corpus this way.
+- Eight memos are listed Active but their pages 404 (D8-2-2, D8-9-3, D10-2-3,
+  D10-14-3, D10-14-18, D10-14-65, D10-15-16, D10-15-25) — CBSA's own broken
+  links. `dmemo.preserve_failed` re-attaches the last-good chunks of any memo
+  that fails to fetch, so a 404 (or a dropped connection) can no longer delete
+  still-in-force guidance. The two D8s were never ingestable and are absent.
+
+This is the incident the write guard above exists for: the first run after the
+rebuild scraped zero memos and wrote an empty dmemos.json straight over 1,870
+good chunks, recovered from git. Every ingester now carries the guard.
+
+## Curated-dataset review protocol
+
+`data/curated/us_dispositions.json` and `us_equivalency.json` are the only
+hand-maintained legal content. To change them: edit → bump `"reviewed"` →
+`py -m canlex.commentary` → `py -m canlex.embed` → eval → deploy. The refresh
+check flags any drift between the curated `reviewed` date and the rendered
+chunks. Provenance and the verbatim-verification log live inside the JSON
+files themselves; keep them current.
+
+## Backups
+
+The HF Space remote is also production. A full-history bundle lives at
+`C:\projects\CanLex-backup.bundle` (recreate: `git bundle create
+../CanLex-backup.bundle --all`). For a real offsite mirror, create a private
+GitHub repo and:
+
+```
+git remote add mirror https://github.com/<you>/CanLex.git
+git push mirror --all
+```
+
+(GitHub needs git-lfs enabled for embeddings.npz, or add it to a
+`.lfsconfig`-excluded push.)
+
+## Known gaps / parked
+
+- ENF coverage (researched and URL-verified 2026-07-22): ENF 20 Detention
+  and the current ENF 3 edition live under non-standard filenames — handled
+  by `_URL_OVERRIDES` in canlex/enf.py (the old enf03-eng.pdf still serves a
+  STALE 2015-era copy; never remove the override). ENF 7 was never publicly
+  released; ENF 13/16/25 never existed; ENF 9 (Judicial Reviews) and ENF 11
+  (Verifying Departure) are retired with no successor (Wayback copies exist
+  but retired guidance is deliberately not ingested). ENF 27 (PR Card) was
+  replaced by the PDI section at
+  operational-bulletins-manuals/permanent-residence/card.html — ingested by
+  `canlex.pdi` (the `pdi-prcard` set). ENF 2 (Evaluating Inadmissibility)
+  and ENF 14/OP 19 (Criminal Rehabilitation) exist publicly only as
+  ATIP-released copies hosted by a private law firm — NOT ingested pending
+  the maintainer's call on unofficial sources; the Tran PDI page now comes
+  in with the whole `standard-requirements` tree and partially covers
+  ENF 2's ground.
+- PDI scope: `canlex.pdi` crawls six category trees breadth-first
+  (`_MAX_DEPTH` 2, `_MAX_PAGES` 160 per tree). Deliberately out of scope:
+  citizenship, fees, office service delivery, and the year-by-year
+  operational-bulletin archives (mostly expired instructions). If a tree
+  ever hits the page cap the run PRINTS how many in-scope pages it dropped
+  — raise `_MAX_PAGES` rather than letting the silence read as coverage.
+- CanLII capture workflow (used for the Hill/Brannson/Li trilogy,
+  2026-07-22): open the URL a missing-file stub prints in a real browser,
+  copy the page text to the named `data/raw/caselaw/*.txt`, re-run
+  `py -m canlex.caselaw && py -m canlex.embed`.
+- Dag (FCA) parses to 2 chunks because it genuinely is a 9-paragraph bench
+  judgment — correct, not a bug.
+- A weekly citator sweep over the ~110 corpus decisions (negative-treatment
+  detection) was considered and deliberately skipped: ~6 minutes of
+  rate-limited CanLII API per run for marginal signal. Run `canlex_case` on
+  a decision before relying on it instead.
+- French support is now phase 3: the full parallel French legislation corpus
+  is ingested (`--fr`; lang='fr' chunks, '-fr-' id infix, 'art.'/'annexe'
+  citations). Retrieval is language-routed: English queries never see French
+  chunks; French queries (detected by accents or French function words)
+  search both languages. get_section/xref/list_acts stay English-canonical.
+  Phase 2 (a multilingual embedder) was tried and REJECTED 2026-07-22 —
+  regressed every slice and broke confidence calibration; don't retry
+  without re-calibrating HEDGE_THRESHOLD. French guidance (D-memos are
+  published bilingually) is the remaining bilingual frontier.
+- ENF 2 and ENF 14/OP 19 are both ingested from their ATIP releases
+  (authorized 2026-07-23, bannered UNOFFICIAL on every chunk). ENF 14's
+  image-scan PDF was recovered by OCR (pymupdf rasterization + Windows OCR;
+  cached at data/raw/enf/enf14-ocr.txt — enf.py auto-falls-back to
+  `*-ocr.txt` when a PDF extracts under 500 chars).
+- Typo robustness, two layers: unknown STEMS get the trigram bridge in
+  `index._fuzzy_term` (BM25 recall only), and misspelled WORDS are repaired
+  by `index._correct_query` (trigram candidates, Damerau-Levenshtein <= 2,
+  shared 2-char prefix, frequency floor 5 so OCR debris can't win) -- the
+  repaired text feeds the semantic embedder and the reranker. Ultra-rare
+  stems (df <= 2) get the correction APPENDED, never substituted, so rare
+  legitimate terms keep their recall. The `typo` eval slice tracks it.
+- Bilingual guidance: the D-memos and NJC directives now carry their French
+  editions (`--fr` modes; dmemos-fr.json / directives-fr.json) alongside
+  the French legislation, all lang-routed. Remaining English-only: case
+  law, ENF/AMPS/PDI, benefits booklets, and the curated commentary.
+
+## Two retrieval invariants worth not re-breaking (2026-07-29)
+
+Ingesting the French guidance and the wider PDI trees exposed both of these.
+Each was measured question-by-question against the previous corpus, not
+guessed at — the harness is a per-question rank dump over two corpus
+variants, which is how a 5-question shuffle gets attributed instead of
+argued about.
+
+- **BM25 collection statistics are language-scoped.** `_build_bm25` keeps
+  `idf`/`avgdl` (whole corpus) AND `idf_en`/`avgdl_en` (English scope);
+  every English query scores with the English pair. An English query can
+  never retrieve the French twins, but a single corpus-wide idf let those
+  unreachable chunks set the rarity of every shared token (numbers, act
+  codes, folded cognates) and drag the average length: adding the French
+  D-memos and directives alone moved legacy Hit@1 0.767 → 0.750 without a
+  single French chunk being retrievable. French queries still use the
+  whole-corpus pair — they search both languages.
+- **The diversity cap keys memoranda on the parent DOCUMENT.** `_source_key`
+  used `section` for every memorandum, which is the document identity only
+  for the flat families (`_FLAT_MEMO_CODES`: one act_code across ~360
+  D-memos / ~1,000 AMPS contraventions). ENF chapters, PDI pages and IRB
+  guidelines carry a per-document act_code and use `section` to subdivide
+  it, so each ENF section and each PDI page was its own "source" and
+  neither RECALL_CAP nor SOURCE_CAP ever bound: the ~150-page PDI refugee
+  tree filled the candidate pool and pushed IRPA s. 112 out of the top 20
+  of "what is a pre-removal risk assessment" entirely. If a new corpus
+  family lands, check which field carries its document identity before
+  trusting the caps.

canlex/__init__.py

@@ -0,0 +1 @@
+"""CanLex - Canadian legal retrieval for Claude."""

canlex/_common.py

@@ -0,0 +1,198 @@
+"""Shared helpers for the ingestion modules.
+
+The ingesters grew organically and each carried private copies of the same
+plumbing; the copies were AST-diffed and consolidated here (the audit found the
+download helper in 8 near-identical copies, the line splitter in 5, and the
+whitespace collapser in 13). Deliberate per-source differences -- a politeness
+UA, a longer timeout, a different split target -- stay at the call sites as
+arguments. caselaw.py keeps its own fetch (retry/throttle for Lexum's CAPTCHA)
+and its own splitter (sentence-boundary windowing, a different algorithm).
+"""
+import json
+import os
+import re
+import subprocess
+import time
+import urllib.request
+
+# The browser User-Agent used for government sites whose CDN denylists obvious
+# non-browser clients (the pages are public; politeness comes from caching and
+# the pauses below). The polite UA is for hosts that accept ordinary clients.
+BROWSER_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+              "(KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")
+POLITE_UA = "CanLex/0.1"
+
+# Month-name lookup for parsing prose dates ("June 21, 2026").
+MONTHS = {m: i for i, m in enumerate(
+    ["january", "february", "march", "april", "may", "june", "july", "august",
+     "september", "october", "november", "december"], start=1)}
+
+
+def norm_ws(text):
+    """Collapse all whitespace to single spaces and trim.
+
+    \\s in a Python 3 str pattern already matches NBSP, en/em/thin spaces and
+    friends, so no pre-replacement is needed (several old copies carried a
+    redundant .replace("\\xa0", " ")).
+    """
+    return re.sub(r"\s+", " ", text or "").strip()
+
+
+def fetch_cached(url, dest, *, ua=BROWSER_UA, timeout=60, pause=0.5,
+                 force=False, powershell=False):
+    """Download a URL to `dest` once and reuse the cached bytes thereafter.
+
+    powershell=True routes the request through PowerShell's (.NET) HTTP stack,
+    which canada.ca accepts while blocking Python's client at the TLS layer.
+    `pause` is a politeness sleep after an actual download (0 to skip);
+    `timeout` applies to the urllib path (the PowerShell path keeps the fixed
+    180s subprocess budget every prior copy used).
+    """
+    if dest.exists() and not force:
+        return dest.read_bytes()
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    if powershell:
+        command = (f"Invoke-WebRequest -Uri '{url}' -OutFile '{dest}' "
+                   f"-UseBasicParsing -UserAgent '{ua}'")
+        subprocess.run(
+            ["powershell", "-NoProfile", "-NonInteractive", "-Command", command],
+            check=True, capture_output=True, timeout=180)
+    else:
+        req = urllib.request.Request(url, headers={"User-Agent": ua})
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            dest.write_bytes(resp.read())
+    if pause:
+        time.sleep(pause)
+    return dest.read_bytes()
+
+
+def split_lines(text, target=1800, fold_stubs=False):
+    """Split text into ~target-sized pieces at line boundaries.
+
+    fold_stubs=True additionally folds a small (<200-char) leading or trailing
+    remnant into its neighbour so a section never yields a stub piece -- the
+    behaviour ingest.py and benefits.py always had. The other callers
+    deliberately keep fold_stubs=False: enabling it would change their chunk
+    boundaries, which is an eval-gated change, not a refactor.
+    """
+    if len(text) <= target:
+        return [text]
+    pieces, buf, size = [], [], 0
+    for line in text.split("\n"):
+        if size + len(line) > target and buf:
+            pieces.append("\n".join(buf))
+            buf, size = [], 0
+        buf.append(line)
+        size += len(line) + 1
+    if buf:
+        pieces.append("\n".join(buf))
+    if fold_stubs:
+        if len(pieces) > 1 and len(pieces[-1]) < 200:
+            pieces[-2] = pieces[-2] + "\n" + pieces[-1]
+            pieces.pop()
+        if len(pieces) > 1 and len(pieces[0]) < 200:
+            pieces[1] = pieces[0] + "\n" + pieces[1]
+            pieces.pop(0)
+    return pieces
+
+
+# Every ingester rebuilds its whole processed file from a fresh scrape, so an
+# upstream page that changes shape -- or a run that loses its network halfway --
+# shows up as a collapse in chunk count, and the write turns that into data
+# loss. On 2026-07-27 CBSA's rebuilt D-memo index scraped to zero memos and the
+# write replaced 1,870 good chunks with an empty list; one line of stdout was
+# the only warning. Refuse to write below this share of what is already stored.
+CORPUS_SHRINK_GUARD = 0.8
+
+
+def stored_chunks(path):
+    """The chunks already written to `path`, or [] if absent or unreadable."""
+    if not path.exists():
+        return []
+    try:
+        return json.loads(path.read_text(encoding="utf-8"))
+    except (ValueError, OSError):
+        return []
+
+
+def safe_to_write(new_count, old_count, ratio=CORPUS_SHRINK_GUARD):
+    """False when writing `new_count` chunks would gut an existing corpus.
+
+    Pure, for tests. The write is the irreversible step, so it is the one to
+    gate -- a first-ever run (old_count 0) is always allowed.
+    """
+    return old_count == 0 or new_count >= old_count * ratio
+
+
+def preserve_dropped(chunks, stored, *, key=lambda c: c["section"], only=None):
+    """Re-attach the last-good chunks of items this run failed to produce.
+
+    `key` maps a chunk to the item it belongs to; `only` restricts preservation
+    to a set of item identities (pass the ones that actually errored, so items
+    dropped ON PURPOSE -- cancelled memos, retired manuals -- are not
+    resurrected). Returns (chunks, preserved identities).
+    """
+    def identity(chunk):
+        # A chunk written by an older schema may not carry the identity field
+        # at all. It simply is not preservable -- but crashing here would take
+        # down the rebuild at the exact moment the guard exists to protect it.
+        try:
+            return key(chunk)
+        except (KeyError, IndexError, TypeError):
+            return None
+
+    fresh = {identity(c) for c in chunks}
+    by_item = {}
+    for chunk in stored:
+        item = identity(chunk)
+        if item is not None:
+            by_item.setdefault(item, []).append(chunk)
+    dropped = sorted(item for item in by_item
+                     if item not in fresh and (only is None or item in only))
+    for item in dropped:
+        chunks.extend(by_item[item])
+    return chunks, dropped
+
+
+def write_corpus(path, chunks, *, indent=2, allow_shrink=False,
+                 ratio=CORPUS_SHRINK_GUARD):
+    """Write `chunks` to `path`, unless that would gut the stored corpus.
+
+    Returns True if written, False if refused -- callers exit non-zero on False
+    so a broken scrape fails loudly instead of quietly emptying a file.
+    """
+    stored = stored_chunks(path)
+    if not allow_shrink and not safe_to_write(len(chunks), len(stored), ratio):
+        print(f"  REFUSING to write: {len(chunks)} chunks would replace "
+              f"{len(stored)} already in {path.name}. The upstream source "
+              f"likely changed shape -- check the scrape before writing "
+              f"(re-run with --allow-shrink once the drop is understood).")
+        return False
+    path.parent.mkdir(parents=True, exist_ok=True)
+    # Write-then-rename, not write-in-place: a run killed mid-write (Ctrl-C on
+    # the scheduled refresh, a full disk) would otherwise leave truncated JSON
+    # behind, and stored_chunks reads unparseable as empty -- which disables
+    # the guard above on the very next run, when it is needed most.
+    tmp = path.with_name(path.name + ".tmp")
+    tmp.write_text(json.dumps(chunks, ensure_ascii=False, indent=indent),
+                   encoding="utf-8")
+    os.replace(tmp, path)
+    return True
+
+
+def uniquify_ids(chunks):
+    """Rename duplicate chunk ids in place (second occurrence -> id-b, -c...).
+
+    Duplicate ids silently orphan chunks downstream: the embeddings loader
+    keys rows by id, so one twin gets vectors and the other degrades to
+    BM25-only with nothing but a stderr note. Generators whose source
+    numbering can repeat (a manual's TOC line and body heading both reading
+    '5.1'; delegation items re-using an instrument number) run their output
+    through this before writing."""
+    seen = {}
+    for c in chunks:
+        n = seen.get(c["id"], 0)
+        seen[c["id"]] = n + 1
+        if n:
+            c["id"] = f"{c['id']}-{chr(ord('a') + n)}"
+    return chunks

canlex/agreement.py

@@ -0,0 +1,196 @@
+"""Ingest Treasury Board collective agreements (HTML) into Article-level chunks.
+
+A collective agreement is a binding contract between the Treasury Board and a
+bargaining agent for one occupational group. Chunks are tagged
+doc_type="agreement" so CanLex keeps them distinct from legislation and guidance.
+"""
+import re
+import sys
+
+from bs4 import BeautifulSoup
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      split_lines, stored_chunks, write_corpus)
+from .config import RAW_DIR, PROCESSED_DIR
+
+AGREEMENT_DIR = RAW_DIR / "agreements"
+OUT_FILE = PROCESSED_DIR / "agreements.json"
+
+# Treasury Board collective agreements to ingest. Add an entry to ingest more.
+AGREEMENTS = {
+    "FB": {
+        "short": "FB Agreement",
+        "name": "FB Group Collective Agreement (Border Services)",
+        "cite": "FB Collective Agreement",
+        "url": "https://www.canada.ca/en/treasury-board-secretariat/topics/pay/"
+               "collective-agreements/fb.html",
+    },
+}
+
+_SKIP_HEADINGS = {"table of contents", "note to readers", "page details",
+                  "on this page"}
+_CONTENT_TAGS = {"p", "dl", "table", "h4", "h5", "h6", "blockquote"}
+_ARTICLE = re.compile(r"Article\s+(\S+?)\s*[:–-]\s*(.+)", re.I)
+
+
+def _fetch(url, dest, force=False):
+    """Download a page. canada.ca blocks Python's HTTP client at the TLS layer,
+    so fetch via PowerShell's (.NET) HTTP stack, which the site accepts."""
+    return fetch_cached(url, dest, force=force, powershell=True)
+
+
+_PIECE_TARGET = 1800   # keep pieces under the embedder's ~2000-char window
+
+
+def _split(text, target=_PIECE_TARGET):
+    """Split long text into ~target-sized pieces at line boundaries."""
+    return split_lines(text, target)
+
+
+def _block_text(heading):
+    """Readable text from a heading up to the next h2/h3 (sections unwrapped)."""
+    lines = []
+    for sib in heading.find_next_siblings():
+        if sib.name in ("h2", "h3"):
+            break
+        if sib.name in ("ul", "ol"):
+            for li in sib.find_all("li", recursive=False):
+                item = _norm(li.get_text(" ", strip=True))
+                if item:
+                    lines.append(f"- {item}")
+        elif sib.name in _CONTENT_TAGS:
+            text = _norm(sib.get_text(" ", strip=True))
+            if text:
+                lines.append(text)
+    return "\n".join(lines)
+
+
+def parse_agreement(html, code):
+    """Parse a collective agreement page into one chunk per Article / Appendix."""
+    meta = AGREEMENTS[code]
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+
+    # The first <time datetime> is the agreement's expiry; the dateModified one
+    # is the page's date.
+    expiry = ""
+    for t in main.find_all("time"):
+        if t.get("datetime") and t.get("property") != "dateModified":
+            expiry = _norm(t.get("datetime"))
+            break
+
+    for tag in main.find_all(["section", "div"]):
+        tag.unwrap()  # flatten so each heading and its content become siblings
+
+    chunks = []
+    current_part = ""
+    sec_idx = 0
+    for h in main.find_all(["h2", "h3"]):
+        if "wb-inv" in (h.get("class") or []):
+            continue
+        text = _norm(h.get_text(" ", strip=True)).lstrip("*").strip()
+        if not text or text.lower() in _SKIP_HEADINGS:
+            continue
+        if h.name == "h2":
+            current_part = text
+        body = _block_text(h)
+        if not body:
+            continue
+
+        article = _ARTICLE.match(text)
+        if article:
+            number = article.group(1).rstrip(":.")
+            note = article.group(2).strip()
+            citation = f"{meta['cite']}, Article {number}"
+        elif re.match(r"Appendix\b", text, re.I):
+            number, note = text, ""
+            citation = f"{meta['cite']}, {text}"
+        else:
+            # Memoranda of Agreement/Understanding have long titles: keep them in
+            # the note only, so the rendered header does not repeat them.
+            number, note = "", text
+            citation = meta["cite"]
+
+        sec_idx += 1
+        # Split a long article so each piece fits the embed window; pieces share
+        # the base id and take a '-p<k>' suffix so the diversity cap groups them.
+        pieces = _split(body)
+        for k, piece in enumerate(pieces, start=1):
+            base = f"agreement-{code}-{sec_idx}"
+            chunk_id = base if len(pieces) == 1 else f"{base}-p{k}"
+            pnote = note if len(pieces) == 1 else f"{note} (part {k})".strip()
+            chunks.append({
+                "id": chunk_id,
+                "doc_type": "agreement",
+                "act_code": code,
+                "act_short": meta["short"],
+                "act_name": meta["name"],
+                "section": number,
+                "marginal_note": pnote,
+                "part": current_part if h.name == "h3" else "",
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": "",
+                "current_to": expiry,
+                "citation": citation,
+                "source_url": meta["url"],
+            })
+    return chunks
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of agreements this run could not rebuild.
+
+    Ingestion is a full rebuild, so an agreement whose page fails to fetch --
+    or whose canada.ca layout changes so that nothing parses out of it --
+    disappears from the corpus. An agreement that is still in force is better
+    served from the copy we already had than not at all. Restricted to the
+    codes that actually failed, so one removed from AGREEMENTS on purpose is
+    not resurrected. Returns (chunks, preserved codes).
+    """
+    failed = {code for code, _why in failures}
+    return preserve_dropped(chunks, stored, key=lambda c: c["act_code"],
+                            only=failed)
+
+
+def ingest(force=False, allow_shrink=False):
+    all_chunks, failures = [], []
+    for code, meta in AGREEMENTS.items():
+        print(f"Ingesting {code} ({meta['short']})...")
+        try:
+            html = _fetch(meta["url"], AGREEMENT_DIR / f"{code}.html", force=force)
+            chunks = parse_agreement(html, code)
+            print(f"  {len(chunks)} chunks")
+            all_chunks.extend(chunks)
+            if not chunks:
+                # A page that parses to nothing is a redesign upstream, not an
+                # empty agreement -- same class of failure as a dead fetch.
+                failures.append((code, "no content parsed"))
+        except Exception as exc:
+            failures.append((code, f"{type(exc).__name__}: {exc}"))
+            print(f"  FAILED {code}: {type(exc).__name__}: {exc}")
+
+    all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                            stored_chunks(OUT_FILE))
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} agreement(s): "
+              f"{', '.join(preserved)}")
+    if not write_corpus(OUT_FILE, all_chunks, indent=2,
+                        allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} chunks from {len(AGREEMENTS)} agreement(s) -> {OUT_FILE.name}")
+    return True
+
+
+def main():
+    ok = ingest(force="--force" in sys.argv,
+                allow_shrink="--allow-shrink" in sys.argv)
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/amps.py

@@ -0,0 +1,154 @@
+"""Ingest the CBSA Master Penalty Document (AMPS contraventions).
+
+The Master Penalty Document is the public per-contravention reference for the
+Administrative Monetary Penalty System: each page (C001 onward) gives the
+contravention text, the penalty amounts by occurrence level, and the
+designated provision it enforces. Together with the Designated Provisions
+(Customs) Regulations schedule, it answers the trade-compliance staple
+"what does contravention X cost on a second occurrence?" -- which the corpus
+previously could not.
+
+Same fetch/cache/parse pattern as dmemo.py: index page -> per-contravention
+pages, cached under data/raw/amps, one chunk per contravention (they are
+short), doc_type='memorandum' (administrative guidance, not law).
+
+    python -m canlex.amps
+"""
+import re
+import sys
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      stored_chunks, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+INDEX_URL = "https://www.cbsa-asfc.gc.ca/trade-commerce/amps/mpd-dmi-eng.html"
+BASE = "https://www.cbsa-asfc.gc.ca"
+RAW = RAW_DIR / "amps"
+
+# Hrefs on the index may be absolute or relative; key on the c-number page
+# name alone.
+_LINK = re.compile(r'href="([^"]*?contraventions-infractions/'
+                   r'([a-z]\d{3})-eng\.html)"')
+_MODIFIED = re.compile(r'<time[^>]*>(\d{4}-\d{2}-\d{2})</time>')
+_TITLE = re.compile(r"<h1[^>]*>(.*?)</h1>", re.S)
+
+
+def _strip(html):
+    """Readable text: table cells joined with ' | ', block ends -> newlines."""
+    html = re.sub(r"</t[dh]>", " | ", html)
+    html = re.sub(r"</(?:p|li|tr|h\d|caption)>", "\n", html)
+    html = re.sub(r"<[^>]+>", " ", html)
+    lines = [_norm(l).strip("| ").strip() for l in html.split("\n")]
+    return "\n".join(l for l in lines if l)
+
+
+def _main_content(html):
+    """The page's main block, shorn of nav/footer boilerplate."""
+    m = re.search(r"<main[^>]*>(.*?)</main>", html, re.S)
+    return m.group(1) if m else html
+
+
+def parse_page(html, code, url):
+    content = _main_content(html)
+    title_m = _TITLE.search(content)
+    title = _norm(re.sub(r"<[^>]+>", " ", title_m.group(1))) if title_m else code.upper()
+    body = _strip(content[title_m.end():] if title_m else content)
+    mod = _MODIFIED.search(html)
+    modified = mod.group(1) if mod else ""
+    # The first non-empty line after the title is the contravention text
+    # itself -- promote it to the marginal note so retrieval titles carry it.
+    first = next((l for l in body.split("\n") if len(l) > 30), "")
+    return {
+        "id": f"amps-{code}",
+        "doc_type": "memorandum",
+        "act_code": "AMPS",
+        "act_short": "AMPS",
+        "act_name": "AMPS Master Penalty Document",
+        "section": code.upper(),
+        "marginal_note": first[:160],
+        "part": "Master Penalty Document",
+        "division": "",
+        "heading": title,
+        "text": body,
+        "history": "",
+        "last_amended": modified,
+        "current_to": modified,
+        "citation": f"AMPS contravention {code.upper()}",
+        "source_url": url,
+    }
+
+
+def _fetch(url, dest):
+    """cbsa-asfc.gc.ca serves this section empty to Python's HTTP client;
+    route through PowerShell (like canada.ca fetches) and treat an empty
+    cache as a miss."""
+    if dest.exists() and dest.stat().st_size == 0:
+        dest.unlink()
+    data = fetch_cached(url, dest, powershell=True, pause=0.4)
+    if not data.strip():
+        raise ValueError(f"empty response for {url}")
+    return data
+
+
+def preserve_failed(chunks, failed, stored):
+    """Re-attach the last-good chunks of contraventions this run could not fetch.
+
+    Ingestion is a full rebuild, so a page that 404s or times out silently
+    leaves the corpus -- and a penalty still in force is worse missing than
+    slightly stale. `failed` carries the (code, why) pairs build() collects;
+    codes are lower-case on the index but upper-case in a chunk's `section`,
+    so match on the upper-cased form. Restricted to codes that actually
+    errored, so a contravention CBSA has genuinely retired stays gone.
+    Returns (chunks, preserved codes).
+    """
+    codes = {code.upper() for code, _why in failed}
+    return preserve_dropped(chunks, stored, key=lambda c: c["section"].upper(),
+                            only=codes)
+
+
+def build(limit=None, allow_shrink=False):
+    html = _fetch(INDEX_URL, RAW / "index.html").decode("utf-8", "replace")
+    links = sorted({(m.group(1), m.group(2)) for m in _LINK.finditer(html)},
+                   key=lambda t: t[1])
+    if limit:
+        links = links[:limit]
+    print(f"amps: {len(links)} contravention pages on the index")
+    chunks, failed = [], []
+    for path, code in links:
+        if path.startswith("http"):
+            url = path
+        elif path.startswith("/"):
+            url = BASE + path
+        else:
+            url = INDEX_URL.rsplit("/", 1)[0] + "/" + path
+        try:
+            page = _fetch(url, RAW / f"{code}.html").decode("utf-8", "replace")
+            chunks.append(parse_page(page, code, url))
+        except Exception as exc:
+            failed.append((code, f"{type(exc).__name__}: {exc}"))
+    out = PROCESSED_DIR / "amps.json"
+    # Printed before the write so a refusal still shows what went wrong.
+    if failed:
+        print(f"amps: {len(failed)} failed: {failed[:5]}")
+    if not limit:
+        chunks, preserved = preserve_failed(chunks, failed, stored_chunks(out))
+        if preserved:
+            print(f"amps: kept the last-good copy of {len(preserved)} "
+                  f"unfetchable contravention(s): {', '.join(preserved)}")
+    # The guard applies to --limit runs too: a smoke test over five
+    # contraventions must not be able to write a five-page corpus over the real
+    # one. indent=2 matches what is already on disk -- changing it would
+    # rewrite every line of the file.
+    if not write_corpus(out, chunks, indent=2, allow_shrink=allow_shrink):
+        return False
+    print(f"amps: {len(chunks)} contraventions -> {out.name}")
+    return True
+
+
+def main():
+    ok = build(allow_shrink="--allow-shrink" in sys.argv)
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/benefits.py

@@ -0,0 +1,346 @@
+"""Ingest the member booklets of the federal public-service benefit plans.
+
+Four plans cover federal employees (and a Border Services Officer over a
+career), by population:
+
+  - PSHCP -- Public Service Health Care Plan (health; active + pensioners)
+  - PSDCP -- Public Service Dental Care Plan (dental; active employees)
+  - PDSP  -- Pensioners' Dental Services Plan (dental; pensioners)
+  - PSMIP -- Public Service Management Insurance Plan (life / AD&D / disability
+             insurance; management-category and excluded employees -- so EX and
+             excluded CBSA staff rather than the represented FB group, who carry
+             the separate Disability Insurance Plan)
+
+Each plan's *member booklet* is the plain-language guide the plan administrator
+or Treasury Board publishes -- what is covered, who is eligible, how to claim,
+the appeals process. The booklets are guidance: the governing terms live in the
+PSHCP Directive (already ingested as doc_type='directive') and the respective
+plan rules, which prevail in any discrepancy. They are tagged doc_type=
+'benefits' and labelled as such in results.
+
+Sources: the Canada Life PDFs (welcome.canadalife.com / canadalife.com), which
+-- unlike canada.ca -- serve Python's urllib fine; and the PSMIP booklet, an
+HTML page on canada.ca (which blocks urllib at the TLS layer, so it is fetched
+via PowerShell, as agreement.py does). A PDF booklet carries a table of
+contents whose headings are located in the body to segment it; the HTML booklet
+is segmented at its h2/h3 headings. Either way a retrieved chunk is a coherent
+benefit topic.
+
+    py -m canlex.benefits
+"""
+import io
+import re
+import sys
+
+from bs4 import BeautifulSoup
+from pypdf import PdfReader
+
+from ._common import (fetch_cached, norm_ws, preserve_dropped, split_lines,
+                      stored_chunks, write_corpus)
+
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "benefits"
+OUT = PROCESSED_DIR / "benefits.json"
+
+_CHUNK_CHARS = 1800     # target characters per chunk
+
+# Each plan: its booklet, and how to read it. PDF booklets give the page
+# indices of the table of contents and the first body page (everything before
+# it is cover + TOC). The HTML booklet is parsed at its h2/h3 headings.
+# 'edition' is the booklet's own date (for HTML it is read from the page's
+# "Date modified"); bump a PDF edition and re-run when a new booklet is posted.
+SOURCES = {
+    "PSHCP": {
+        "code": "PSHCP", "format": "pdf",
+        "name": "Public Service Health Care Plan",
+        "url": ("https://www.welcome.canadalife.com/content/dam/rfp/"
+                "welcome-sites/pshcp/PSHCP-member-booklet.pdf"),
+        "label": "Member Booklet",
+        "toc_pages": [2, 3, 4],
+        "body_start": 5,
+        "edition": "June 2024",
+    },
+    "PSDCP": {
+        "code": "PSDCP", "format": "pdf",
+        "name": "Public Service Dental Care Plan",
+        "url": ("https://www.welcome.canadalife.com/content/dam/rfp/"
+                "welcome-sites/psdcp/psdcp-member-booklet.pdf"),
+        "label": "Member Booklet",
+        "toc_pages": [2, 3],
+        "body_start": 4,
+        "edition": "May 2026",
+    },
+    "PDSP": {
+        "code": "PDSP", "format": "pdf",
+        "name": "Pensioners' Dental Services Plan",
+        "url": ("https://www.canadalife.com/content/dam/rfp/welcome-sites/"
+                "pdsp/pdsp-member-booklet.pdf"),
+        "label": "Member Booklet",
+        "toc_pages": [2, 3],
+        "body_start": 4,
+        "edition": "November 2024",
+    },
+    "PSMIP": {
+        "code": "PSMIP", "format": "html",
+        "name": "Public Service Management Insurance Plan",
+        "url": ("https://www.canada.ca/en/treasury-board-secretariat/services/"
+                "benefit-plans/management-insurance-plan/main-plan-booklet.html"),
+        "label": "Main Plan Booklet",
+        # The canada.ca page exposes no machine-readable date (its <time> tags are
+        # content dates, not a page revision). Rather than render currency as
+        # 'n/a', state plainly that the source is undated so a reader knows to
+        # verify rather than assume staleness.
+        "edition": "undated (TBS web edition)",
+    },
+}
+
+# canada.ca section labels that are navigation/footer, not plan content.
+_SKIP_HEADINGS = {"on this page", "page details", "from:", "table of contents"}
+
+# A table-of-contents line: optional dot-leaders, the heading text, then the
+# printed page number. The heading must end in a letter or ')' so a bare page
+# number or a dot run alone is not mistaken for a heading.
+_TOC_LINE = re.compile(r"^[\s.]*(.*?[A-Za-z)])\s*\.*\s*(\d{1,3})\s*$")
+
+
+def _fetch(url, dest):
+    return fetch_cached(url, dest, timeout=90, pause=0)
+
+
+def _clean(text):
+    """Tidy extracted PDF text: turn bullets into dashes, collapse runs of
+    spaces, and trim excess blank lines. Apostrophes, accents and en-dashes
+    extract as proper Unicode, so no transliteration is needed."""
+    text = text.replace("•", "\n- ").replace("\xa0", " ")
+    text = re.sub(r"[ \t]+", " ", text)
+    text = re.sub(r"\n[ \t]+", "\n", text)
+    return re.sub(r"\n{3,}", "\n\n", text).strip()
+
+
+def _page_texts(pdf_bytes):
+    reader = PdfReader(io.BytesIO(pdf_bytes))
+    return [p.extract_text() or "" for p in reader.pages]
+
+
+def parse_toc(page_texts, toc_pages):
+    """Return the ordered, de-duplicated list of section headings from the
+    table of contents."""
+    seen, headings = set(), []
+    for i in toc_pages:
+        if i >= len(page_texts):
+            continue
+        for line in page_texts[i].split("\n"):
+            line = line.strip()
+            if not line or "Table of Contents" in line:
+                continue
+            m = _TOC_LINE.match(line)
+            if not m:
+                continue
+            title = re.sub(r"\s+", " ", m.group(1)).strip(" .")
+            if len(title) > 2 and title.lower() not in seen:
+                seen.add(title.lower())
+                headings.append(title)
+    return headings
+
+
+def segment(body, headings):
+    """Split the body text at each heading, in order, returning a list of
+    (heading, section_text). A heading the extractor split across lines is
+    matched with flexible whitespace; a heading not found in the body is
+    skipped (its text folds into the preceding section)."""
+    cuts = []
+    pos = 0
+    for h in headings:
+        pattern = re.compile(r"\s*".join(re.escape(w) for w in h.split()), re.I)
+        m = pattern.search(body, pos)
+        if m:
+            cuts.append((h, m.start()))
+            pos = m.end()
+    sections = []
+    for j, (h, start) in enumerate(cuts):
+        end = cuts[j + 1][1] if j + 1 < len(cuts) else len(body)
+        sections.append((h, body[start:end].strip()))
+    return sections
+
+
+def _split(text, target=_CHUNK_CHARS):
+    """Split a long section into ~target-sized pieces at line boundaries,
+    folding small leading/trailing remnants into a neighbour so a section
+    never yields a stub chunk."""
+    return split_lines(text, target, fold_stubs=True)
+
+
+def _slug(heading, used):
+    """A short, unique, citable section locator from a heading."""
+    base = re.sub(r"[^a-z0-9]+", "-", heading.lower()).strip("-")[:40] or "section"
+    slug, n = base, 2
+    while slug in used:
+        slug = f"{base}-{n}"
+        n += 1
+    used.add(slug)
+    return slug
+
+
+def _emit(sections, src):
+    """Build CanLex chunk dicts from a list of (heading, section_text), shared
+    by the PDF and HTML paths. Skips pure section-divider headings and
+    sub-splits any section over the chunk budget."""
+    chunks = []
+    used_slugs = set()
+    label = src["label"]
+    for heading, text in sections:
+        # Skip pure section dividers -- a heading whose only content is the
+        # heading itself (its substance lives in the child sections below it).
+        # Compare with whitespace collapsed, since a PDF extractor often breaks
+        # a heading across lines ("...Plan \n(PSDCP)").
+        norm_text = norm_ws(text)
+        norm_head = norm_ws(heading)
+        remainder = (norm_text[len(norm_head):]
+                     if norm_text.lower().startswith(norm_head.lower())
+                     else norm_text)
+        if len(remainder.strip()) < 40:
+            continue
+        slug = _slug(heading, used_slugs)
+        pieces = _split(text)
+        for k, piece in enumerate(pieces, start=1):
+            section = slug if len(pieces) == 1 else f"{slug}-{k}"
+            note = heading if len(pieces) == 1 else f"{heading} (part {k})"
+            chunks.append({
+                "id": f"benefits-{src['code']}-{section}",
+                "doc_type": "benefits",
+                "act_code": src["code"],
+                "act_short": src["code"],
+                "act_name": f"{src['name']} — {label}",
+                "section": section,
+                "marginal_note": note,
+                "part": f"{src['name']} {label}",
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": "",
+                "current_to": src["edition"],
+                "citation": (f"{src['code']} {label}"
+                             + (f" ({src['edition']})" if src["edition"] else "")
+                             + f", “{heading}”"),
+                "source_url": src["url"],
+            })
+    return chunks
+
+
+def parse_pdf_booklet(pdf_bytes, src):
+    """Segment a PDF booklet by its table-of-contents headings."""
+    pages = _page_texts(pdf_bytes)
+    headings = parse_toc(pages, src["toc_pages"])
+    body = _clean("\n".join(pages[src["body_start"]:]))
+    return _emit(segment(body, headings), src)
+
+
+def _html_block_text(heading):
+    """Readable text from an h2/h3 up to the next heading of the same or higher
+    level; lists become dash items."""
+    lines = []
+    for sib in heading.find_next_siblings():
+        if sib.name in ("h2", "h3"):
+            break
+        if sib.name in ("ul", "ol"):
+            for li in sib.find_all("li", recursive=False):
+                item = norm_ws(li.get_text(" ", strip=True))
+                if item:
+                    lines.append(f"- {item}")
+        elif sib.name in ("p", "div", "table", "section", "h4", "h5"):
+            text = norm_ws(sib.get_text(" ", strip=True))
+            if text:
+                lines.append(text)
+    return "\n".join(lines)
+
+
+def parse_html_booklet(html, src):
+    """Segment the canada.ca HTML booklet at its h2/h3 headings."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main") or soup
+    # Read the page's "Date modified" so the booklet's currency is recorded.
+    if not src["edition"]:
+        t = main.find("time", property="dateModified") or soup.find(
+            "time", property="dateModified")
+        if t:
+            src = {**src, "edition": t.get_text(strip=True)}
+    sections = []
+    for h in main.find_all(["h2", "h3"]):
+        title = norm_ws(h.get_text(" ", strip=True))
+        if not title or title.lower() in _SKIP_HEADINGS:
+            continue
+        body = _html_block_text(h)
+        if body:
+            sections.append((title, f"{title}\n{body}"))
+    return _emit(sections, src)
+
+
+def _fetch_html(url, dest):
+    """Fetch a canada.ca page via PowerShell -- the site blocks Python's HTTP
+    client at the TLS layer, but accepts PowerShell's .NET stack."""
+    return fetch_cached(url, dest, powershell=True)
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of plans this run could not rebuild.
+
+    Ingestion is a full rebuild of all four booklets into one file, so one plan
+    failing -- Canada Life moves a PDF, canada.ca times out, a re-typeset
+    booklet no longer segments -- silently deletes that plan's coverage while
+    the other three look healthy. A plan's chunks all carry its code in
+    act_code, so the stored copy can be restored intact. Restricted to plans
+    that actually failed, so a plan removed from SOURCES on purpose (a wound-up
+    plan) is not resurrected. Returns (chunks, preserved plan codes).
+    """
+    failed = {code for code, _why in failures}
+    return preserve_dropped(chunks, stored, key=lambda c: c["act_code"],
+                            only=failed)
+
+
+def build(allow_shrink=False):
+    all_chunks, failures = [], []
+    for src in SOURCES.values():
+        print(f"Ingesting {src['code']} {src['label'].lower()} ...")
+        try:
+            if src["format"] == "pdf":
+                pdf = _fetch(src["url"], RAW / f"{src['code']}.pdf")
+                chunks = parse_pdf_booklet(pdf, src)
+            else:
+                html = _fetch_html(src["url"], RAW / f"{src['code']}.html"
+                                   ).decode("utf-8", "replace")
+                chunks = parse_html_booklet(html, src)
+        except Exception as exc:
+            print(f"  !! {src['code']}: {type(exc).__name__}: {exc}")
+            failures.append((src["code"], f"{type(exc).__name__}: {exc}"))
+            continue
+        if not chunks:
+            # A booklet that downloads fine but segments to nothing has failed
+            # just as surely as one that 404s -- that is how the CBSA index
+            # loss looked -- so it counts as a failure rather than a plan that
+            # legitimately has no content.
+            print(f"  !! {src['code']}: no sections parsed")
+            failures.append((src["code"], "no sections parsed"))
+            continue
+        all_chunks.extend(chunks)
+        print(f"  {len(chunks)} chunks")
+
+    all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                            stored_chunks(OUT))
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} booklet(s): "
+              f"{', '.join(preserved)}")
+    if not write_corpus(OUT, all_chunks, indent=1, allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} benefit-plan chunks from {len(SOURCES)} "
+          f"booklet(s) -> {OUT}")
+    return True
+
+
+def main():
+    sys.exit(0 if build(allow_shrink="--allow-shrink" in sys.argv) else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/caselaw.py

@@ -0,0 +1,1279 @@
+"""Ingest leading Canadian court and tribunal decisions as section-level chunks.
+
+Sources: the official Lexum decisions databases of the Supreme Court of Canada
+(decisions.scc-csc.ca), the Federal Court of Appeal (decisions.fca-caf.gc.ca)
+and the Federal Court (decisions.fct-cf.gc.ca) -- a decision's text sits inside
+an iframe, fetched by appending ?iframe=true to its URL -- plus the in-force
+jurisprudential guides the Immigration and Refugee Board publishes on its own
+site (irb-cisr.gc.ca). This ingests a *curated* set of leading decisions; it is
+deliberately not a comprehensive scrape.
+
+    py -m canlex.caselaw
+"""
+import re
+import sys
+import time
+import urllib.error
+import urllib.request
+
+from bs4 import BeautifulSoup
+
+from ._common import (MONTHS as _MONTHS, norm_ws as _norm, preserve_dropped,
+                      stored_chunks, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+# Each court or tribunal's Lexum decisions database: (display name, item-URL
+# template). All five run the same Lexum platform -- same iframe trick, metadata
+# block and bracketed paragraph numbers -- so one parser serves them all.
+COURTS = {
+    "scc": ("Supreme Court of Canada",
+            "https://decisions.scc-csc.ca/scc-csc/scc-csc/en/item/{id}/index.do"),
+    "fca": ("Federal Court of Appeal",
+            "https://decisions.fca-caf.gc.ca/fca-caf/decisions/en/item/{id}/index.do"),
+    "fc": ("Federal Court",
+           "https://decisions.fct-cf.gc.ca/fc-cf/decisions/en/item/{id}/index.do"),
+    "fpslreb": ("Federal Public Sector Labour Relations and Employment Board",
+                "https://decisions.fpslreb-crtespf.gc.ca/fpslreb-crtespf/d/en/"
+                "item/{id}/index.do"),
+    "onca": ("Court of Appeal for Ontario",
+             "https://coadecisions.ontariocourts.ca/coa/coa/en/item/{id}/"
+             "index.do"),
+    "cirb": ("Canada Industrial Relations Board",
+             "https://decisia.lexum.com/cirb-ccri/cirb-ccri/en/item/{id}/index.do"),
+}
+_RAW = RAW_DIR / "caselaw"
+OUT = PROCESSED_DIR / "caselaw.json"
+
+# A normal browser User-Agent: the courts' Lexum sites denylist a few crawler
+# UAs, while robots.txt otherwise permits the decision pages. Politeness comes
+# from the throttle below and from caching every fetched page on disk.
+_UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+       "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
+_THROTTLE = 30.0      # seconds between fetches -- Lexum Decisia CAPTCHA-gates high-volume access
+_CHUNK_CHARS = 1800   # target characters per chunk
+
+# Marks the post-reasons apparatus (appended legislation, solicitors list),
+# which is not part of the judgment's reasons.
+_APPARATUS = re.compile(
+    r"^\s*(APPENDIX\b|Solicitors?\s+for\b|NAMES\s+OF\s+COUNSEL\b)", re.I)
+
+# Curated leading cases on Canadian border / immigration / customs / Charter
+# law. 'court' selects a database in COURTS; 'id' is the verified Lexum item ID;
+# 'short' and 'topic' are curated. The case name, citation and date are parsed
+# from the page. FCA decisions later reversed by the SCC are deliberately
+# omitted -- CanLex serves good law, and the SCC versions are already listed.
+CASES = [
+    # --- Supreme Court of Canada ---
+    {"court": "scc", "id": 18078, "short": "Vavilov",
+     "topic": "Standard of review on judicial review; the reasonableness "
+              "standard for administrative decisions"},
+    {"court": "scc", "id": 20081, "short": "Mason",
+     "topic": "Inadmissibility under IRPA s. 34(1)(e) for acts of violence "
+              "endangering safety in Canada; reasonableness review"},
+    {"court": "scc", "id": 18510, "short": "Fraser",
+     "topic": "Adverse-effect discrimination under Charter s. 15(1); the "
+              "RCMP job-sharing pension rules' disproportionate impact on "
+              "women; substantive equality methodology"},
+    {"court": "scc", "id": 16146, "short": "Igloo Vikski",
+     "topic": "Customs tariff classification methodology; how the General "
+              "Rules for the Interpretation of the Harmonized System apply "
+              "(Rules 1 and 2(b)); the SCC anchor for tariff-classification "
+              "disputes"},
+    {"court": "scc", "id": 16803, "short": "Tran",
+     "topic": "Serious criminality and inadmissibility under IRPA s. 36; the "
+              "meaning of a term of imprisonment and an offence punishable by"},
+    {"court": "scc", "id": 15647, "short": "B010",
+     "topic": "Inadmissibility for people smuggling under IRPA s. 37(1)(b); "
+              "organized criminality"},
+    {"court": "scc", "id": 15648, "short": "Appulonappa",
+     "topic": "The human smuggling offence in IRPA s. 117; constitutional "
+              "overbreadth and humanitarian aid to asylum seekers"},
+    {"court": "scc", "id": 14419, "short": "Febles",
+     "topic": "Exclusion from refugee protection for a serious non-political "
+              "crime under Article 1F(b) of the Refugee Convention"},
+    {"court": "scc", "id": 13184, "short": "Ezokola",
+     "topic": "Complicity and exclusion from refugee protection for "
+              "international crimes under Article 1F(a)"},
+    {"court": "scc", "id": 15665, "short": "Kanthasamy",
+     "topic": "Humanitarian and compassionate relief under IRPA s. 25; the "
+              "best interests of a child"},
+    {"court": "scc", "id": 13137, "short": "Agraira",
+     "topic": "Ministerial relief from inadmissibility on security grounds; "
+              "the national interest under IRPA"},
+    {"court": "scc", "id": 6901, "short": "Khosa",
+     "topic": "Standard of review of immigration decisions; judicial review "
+              "of a removal order"},
+    {"court": "scc", "id": 2345, "short": "Charkaoui",
+     "topic": "Security certificates and immigration detention; the Charter "
+              "and procedural fairness"},
+    {"court": "scc", "id": 1937, "short": "Suresh",
+     "topic": "Deportation to a risk of torture; Charter s. 7 and removal on "
+              "security grounds"},
+    {"court": "scc", "id": 17759, "short": "Chhina",
+     "topic": "Habeas corpus as a remedy for immigration detention; review of "
+              "lengthy detention"},
+    {"court": "scc", "id": 1717, "short": "Baker",
+     "topic": "Procedural fairness in administrative decisions; humanitarian "
+              "and compassionate review; the duty to give reasons"},
+    {"court": "scc", "id": 39, "short": "Singh",
+     "topic": "Charter s. 7 rights of refugee claimants; the right to an oral "
+              "hearing"},
+    {"court": "scc", "id": 377, "short": "Simmons",
+     "topic": "Customs searches at the border; Charter s. 8 and the reasonable "
+              "expectation of privacy on entry to Canada"},
+    {"court": "scc", "id": 1694, "short": "Monney",
+     "topic": "Border detention for a customs search; reasonable suspicion "
+              "and the Customs Act"},
+    {"court": "scc", "id": 986, "short": "Dehghani",
+     "topic": "Charter rights at a port of entry; secondary examination and "
+              "the right to counsel"},
+    {"court": "scc", "id": 1627, "short": "Pushpanathan",
+     "topic": "Exclusion from refugee protection under Article 1F(c) for acts "
+              "contrary to the purposes of the United Nations"},
+    {"court": "scc", "id": 1023, "short": "Ward",
+     "topic": "The refugee definition; a particular social group; the "
+              "availability of state protection"},
+    {"court": "scc", "id": 855, "short": "Chiarelli",
+     "topic": "Charter s. 7 fundamental justice and the deportation of "
+              "permanent residents for serious criminality"},
+    {"court": "scc", "id": 1940, "short": "Chieu",
+     "topic": "Immigration Appeal Division appeals from a removal order; "
+              "foreign hardship is a relevant consideration"},
+    {"court": "scc", "id": 2284, "short": "Medovarski",
+     "topic": "The objectives of IRPA; a non-citizen has no Charter s. 7 "
+              "right to remain in Canada"},
+    {"court": "scc", "id": 2273, "short": "Mugesera",
+     "topic": "Inadmissibility for crimes against humanity; incitement to "
+              "genocide and the elements of the offence"},
+    {"court": "scc", "id": 13643, "short": "Harkat",
+     "topic": "Constitutionality of the IRPA security-certificate regime and "
+              "the special advocate scheme"},
+    {"court": "scc", "id": 7899, "short": "Németh",
+     "topic": "Extradition and the refugee principle of non-refoulement; the "
+              "Minister's duty when surrendering a protected person"},
+    {"court": "scc", "id": 7944, "short": "Mavi",
+     "topic": "Sponsorship-undertaking debt; the procedural fairness owed "
+              "before the Crown enforces a sponsorship debt"},
+    {"court": "scc", "id": 12904, "short": "Pham",
+     "topic": "Collateral immigration consequences as a factor in fixing a "
+              "criminal sentence"},
+    {"court": "scc", "id": 1299, "short": "Chan",
+     "topic": "Refugee protection; membership in a particular social group "
+              "and a well-founded fear of persecution"},
+    {"court": "scc", "id": 1425, "short": "Jacques",
+     "topic": "Border searches; reasonable suspicion for a vehicle stop near "
+              "the border under the Customs Act"},
+    {"court": "scc", "id": 2198, "short": "Martineau",
+     "topic": "Whether an ascertained-forfeiture proceeding under the Customs "
+              "Act is penal and engages the right against self-incrimination"},
+    {"court": "scc", "id": 21317, "short": "Carignan",
+     "topic": "Powers of arrest without warrant under Criminal Code s. 495; an "
+              "arrest that contravenes the s. 495(2) limits can ground a breach "
+              "of the Charter s. 9 right against arbitrary arrest, despite "
+              "s. 495(3)"},
+    {"court": "scc", "id": 14502, "short": "Fearon",
+     "topic": "Searches of cell phones incident to arrest; the tailored "
+              "common-law framework under Charter s. 8 -- the doctrinal "
+              "companion to the border device-search cases (Canfield, Pike)"},
+    {"court": "scc", "id": 19957, "short": "CCRf",
+     "topic": "Constitutionality of the Safe Third Country Agreement; Charter "
+              "ss. 7 and 15 review of designating the United States as a safe "
+              "country for refugee claimants"},
+    {"court": "scc", "id": 2507, "short": "Hydro-Quebec",
+     "topic": "Duty to accommodate and undue hardship; the employer must "
+              "arrange the workplace and duties to allow the employee to work, "
+              "but is not required to fundamentally alter the conditions of work"},
+    {"court": "scc", "id": 16679, "short": "Elk Valley",
+     "topic": "Duty to accommodate addiction in safety-sensitive workplaces; "
+              "the prima facie discrimination test and dismissal for breach of "
+              "a no-free-accident drug and alcohol policy"},
+    {"court": "scc", "id": 910, "short": "Renaud",
+     "topic": "Tripartite duty to accommodate in unionized workplaces; the "
+              "shared obligations of the employer, the union and the employee "
+              "to the point of undue hardship"},
+    {"court": "scc", "id": 13106, "short": "Irving",
+     "topic": "Random alcohol testing in safety-sensitive workplaces; balancing "
+              "the employer's safety interests against employee privacy under a "
+              "collective agreement"},
+    {"court": "scc", "id": 14577, "short": "MPAO",
+     "topic": "Charter s. 2(d) freedom of association and collective bargaining; "
+              "the exclusion of RCMP members from a meaningful labour relations "
+              "regime is unconstitutional"},
+    {"court": "scc", "id": 14610, "short": "SFL",
+     "topic": "Constitutional right to strike under Charter s. 2(d); essential "
+              "services legislation that prohibits strike action without a "
+              "meaningful alternative dispute-resolution mechanism"},
+    {"court": "scc", "id": 2306, "short": "Ferras",
+     "topic": "Extradition under the record-of-the-case procedure; Charter s. 7 "
+              "requires the extradition judge to weigh the sufficiency and "
+              "reliability of the evidence before committal"},
+
+    # --- Federal Court of Appeal ---
+    {"court": "fca", "id": 345651, "short": "Jane Doe",
+     "topic": "Judicial review of 2017 PSLREB 55; compensation for a "
+              "victim of workplace sexual assault in the federal public "
+              "service; remedial authority of the Board"},
+    {"court": "fca", "id": 127201, "short": "Patanguli",
+     "topic": "A de novo hearing before the Board cures procedural defects "
+              "in the employer's investigation (applying Tipple); dismissal "
+              "of a CIC employee for cheating in a staffing process"},
+    {"court": "fc", "id": 525537, "short": "Marentette",
+     "topic": "Workplace violence and harassment investigations under the "
+              "Canada Labour Code; procedural fairness owed to the "
+              "complainant in the investigation process"},
+    {"court": "fca", "id": 33278, "short": "Cha",
+     "topic": "Scope of s. 44(1) inadmissibility reports and Minister's "
+              "delegate removal orders at the port of entry; officers and "
+              "delegates have no discretion to weigh humanitarian factors "
+              "for a foreign national's ordinary criminality; the content of "
+              "procedural fairness at the report stage"},
+    {"court": "fca", "id": 143136, "short": "Huruglica",
+     "topic": "The Refugee Appeal Division reviews the Refugee Protection "
+              "Division for correctness and makes its own independent assessment"},
+    {"court": "fca", "id": 143152, "short": "Singh (new evidence)",
+     "topic": "Admitting new evidence at the Refugee Appeal Division under "
+              "IRPA s. 110(4)"},
+    {"court": "fca", "id": 37663, "short": "Hernandez Febles",
+     "topic": "Exclusion from refugee protection for a serious non-political "
+              "crime, Refugee Convention Article 1F(b)"},
+    {"court": "fca", "id": 36253, "short": "Jayasekara",
+     "topic": "The test for a serious crime under Article 1F(b) exclusion "
+              "from refugee protection"},
+    {"court": "fca", "id": 35786, "short": "Raza",
+     "topic": "Pre-removal risk assessment; the test for new evidence under "
+              "IRPA s. 113(a)"},
+    {"court": "fca", "id": 99694, "short": "Najafi",
+     "topic": "Security inadmissibility under IRPA s. 34(1)(b), subversion "
+              "against a government"},
+    {"court": "fca", "id": 108889, "short": "Kanagendren",
+     "topic": "Security inadmissibility for membership under IRPA s. 34(1)(f); "
+              "membership requires no complicity analysis"},
+    {"court": "fca", "id": 35313, "short": "Sittampalam",
+     "topic": "Organized criminality under IRPA s. 37(1)(a); the broad meaning "
+              "of a criminal organization"},
+    {"court": "fca", "id": 31607, "short": "Thanabalasingham",
+     "topic": "Immigration detention review; the Minister's onus and the "
+              "weight owed to prior detention decisions"},
+    {"court": "fca", "id": 305100, "short": "Lunyamila",
+     "topic": "Immigration detention where a detainee will not cooperate with "
+              "their own removal"},
+    {"court": "fca", "id": 483607, "short": "Brown",
+     "topic": "The constitutionality of immigration detention; the Charter "
+              "and lengthy detention"},
+    {"court": "fca", "id": 36347, "short": "Baron",
+     "topic": "Deferral of removal by an enforcement officer; the narrow "
+              "scope of the discretion to defer"},
+    {"court": "fca", "id": 520921, "short": "Galindo Camayo",
+     "topic": "Reasonableness review of an immigration decision and the role "
+              "of certified questions on appeal"},
+    {"court": "fca", "id": 501244, "short": "Best Buy",
+     "topic": "Customs tariff classification; methodology and appeals from "
+              "the Canadian International Trade Tribunal"},
+    {"court": "fca", "id": 466027, "short": "Honey Fashions",
+     "topic": "Customs; judicial review of the Canada Border Services "
+              "Agency's exercise of discretion"},
+    {"court": "fca", "id": 419470, "short": "Hociung",
+     "topic": "Whether gold coins are currency or monetary instruments that "
+              "must be reported on import under the PCMLTFA cross-border regime"},
+    {"court": "fca", "id": 32333, "short": "Poshteh",
+     "topic": "Inadmissibility for membership in a terrorist organization "
+              "under IRPA s. 34(1)(f); relevance of the claimant's age"},
+    {"court": "fca", "id": 35576, "short": "Hinzman",
+     "topic": "Refugee and humanitarian and compassionate claims by United "
+              "States military deserters"},
+    {"court": "fca", "id": 35611, "short": "Thamotharem",
+     "topic": "Refugee Protection Division hearing procedure; order of "
+              "questioning and fettering of discretion by guidelines"},
+    {"court": "fca", "id": 37227, "short": "Toussaint",
+     "topic": "Access to the Interim Federal Health Program for an indigent "
+              "applicant without immigration status"},
+    {"court": "fca", "id": 31447, "short": "Rahaman",
+     "topic": "Refugee claims and the meaning of a 'no credible basis' "
+              "finding under IRPA"},
+    {"court": "fca", "id": 521840, "short": "Rodriguez Anzola",
+     "topic": "Criminal inadmissibility under IRPA s. 36(1)(b) for a "
+              "conviction abroad; whether an immigration decision-maker may "
+              "consider that the defence of duress was effectively unavailable "
+              "in the foreign jurisdiction"},
+
+    # --- Federal Court ---
+    {"court": "fc", "id": 64594, "short": "Goburdhun",
+     "topic": "Inadmissibility for misrepresentation under IRPA s. 40; a "
+              "frequently-cited summary of the governing principles"},
+    {"court": "fc", "id": 492842, "short": "Garcia",
+     "topic": "Criminal inadmissibility; the effect of withdrawn charges and "
+              "a claim of self-defence"},
+    {"court": "fc", "id": 483303, "short": "Kaur",
+     "topic": "Procedural fairness; the right to a meaningful opportunity to "
+              "answer a visa officer's concerns"},
+    {"court": "fc", "id": 56900, "short": "Nguyen",
+     "topic": "Customs Act; judicial review of the amount set for a penalty "
+              "or ascertained forfeiture"},
+    {"court": "fc", "id": 57473, "short": "Almrei",
+     "topic": "Reasonableness of an IRPA security certificate after "
+              "Charkaoui; the test on re-determination"},
+    {"court": "fc", "id": 48199, "short": "Wang",
+     "topic": "Innocent-misrepresentation exception to IRPA s. 40(1)(a); the "
+              "section reaches indirect misrepresentations made by another "
+              "party, subject to a narrow carve-out where the applicant had no "
+              "knowledge of and no control over the misrepresentation"},
+    {"court": "fc", "id": 47039, "short": "Bellido",
+     "topic": "Materiality of a misrepresentation under IRPA s. 40(1)(a); a "
+              "misrepresentation need not be intentional, deliberate or "
+              "negligent so long as it could have induced an error in the "
+              "administration of the Act"},
+
+    # --- Federal courts: PCMLTFA cross-border currency forfeiture ---
+    {"court": "fca", "id": 36092, "short": "Sellathurai",
+     "topic": "PCMLTFA cross-border currency forfeiture; on a s. 30 appeal the "
+              "issue is whether the s. 12 reporting duty was contravened, and "
+              "the claimant must satisfy the Minister that the seized funds "
+              "are of legitimate origin"},
+    {"court": "fca", "id": 35921, "short": "Dag",
+     "topic": "PCMLTFA currency forfeiture; the limited scope of review of "
+              "the Minister's decision and the privative clause"},
+    {"court": "fca", "id": 35595, "short": "Tourki",
+     "topic": "PCMLTFA; the structure of the s. 30 appeal to the Federal "
+              "Court against the forfeiture of seized currency"},
+    # Williams, 2017 FCA 252 (item 304112) interpreted PCMLTFA s. 13, the
+    # 'election not to proceed' exception to the reporting duty. Parliament
+    # repealed s. 13 in 2018 (S.C. 2018, c. 27, s. 174), so Williams is no
+    # longer good law on a live provision and is excluded from the corpus.
+    {"court": "fc", "id": 37898, "short": "Dokaj",
+     "topic": "PCMLTFA; the Federal Court's jurisdiction on a s. 30 appeal is "
+              "limited to whether the s. 12 reporting duty was contravened"},
+    {"court": "fc", "id": 62413, "short": "Da Huang",
+     "topic": "PCMLTFA currency forfeiture; partial return of seized funds "
+              "where only part is shown to be of legitimate origin"},
+
+    # --- Federal Public Sector Labour Relations and Employment Board ---
+    # 2026-07 gap audit: the governing authorities the corpus's own
+    # decisions apply, identified per doctrinal domain and verified
+    # against the issuing courts' records before ingestion.
+    {"court": "scc", "id": 7799, "short": "Grant",
+     "topic": "The governing s. 24(2) exclusion framework and the "
+              "controlling definition of physical and psychological "
+              "detention"},
+    {"court": "scc", "id": 2015, "short": "Jarvis",
+     "topic": "The predominant-purpose test: when a regulatory audit "
+              "becomes a penal investigation engaging full Charter "
+              "protections"},
+    {"court": "scc", "id": 1924, "short": "Golden",
+     "topic": "Strip searches are never routine: independent reasonable "
+              "grounds and a reasonable manner required"},
+    {"court": "scc", "id": 13264, "short": "Chehil",
+     "topic": "The reasonable-suspicion standard: an objectively "
+              "discernible constellation of factors assessed in their "
+              "totality"},
+    {"court": "scc", "id": 5274, "short": "Hunter",
+     "topic": "The s. 8 foundation: reasonable expectation of privacy and "
+              "the presumptive warrant requirement"},
+    {"court": "scc", "id": 7800, "short": "Suberu",
+     "topic": "Section 10(b) attaches immediately on detention; when an "
+              "investigative encounter crystallizes into detention"},
+    {"court": "scc", "id": 4627, "short": "Kang-Brown",
+     "topic": "Sniffer-dog deployment is a s. 8 search, grounded on "
+              "reasonable suspicion"},
+    {"court": "scc", "id": 1492, "short": "Collins",
+     "topic": "The tripartite s. 8 reasonableness test: authorized by "
+              "law, a reasonable law, conducted reasonably"},
+    {"court": "scc", "id": 15478, "short": "Guindon",
+     "topic": "When an administrative monetary penalty engages Charter s. "
+              "11 protections (it ordinarily does not)"},
+    {"court": "scc", "id": 13327, "short": "Vu",
+     "topic": "Computers attract a distinct, heightened privacy interest "
+              "requiring specific search authorization"},
+    {"court": "scc", "id": 13265, "short": "MacKenzie",
+     "topic": "Reasonable suspicion applied to a vehicle sniffer-dog "
+              "search on traveller-profile indicators"},
+    {"court": "scc", "id": 21114, "short": "Pepa",
+     "topic": "IAD appeal rights under s. 63(2) for a foreign national "
+              "whose permanent resident visa expired after entry"},
+    {"court": "scc", "id": 1874, "short": "Mattel",
+     "topic": "Customs valuation under s. 48: royalties and licence fees "
+              "are dutiable only as a condition of the sale"},
+    {"court": "scc", "id": 266, "short": "Wigglesworth",
+     "topic": "The two-part test for Charter s. 11: proceedings criminal "
+              "by nature or carrying a true penal consequence"},
+    {"court": "scc", "id": 1276, "short": "Weber",
+     "topic": "Exclusive jurisdiction: disputes arising from the "
+              "collective agreement go to grievance arbitration, not the "
+              "courts"},
+    {"court": "scc", "id": 2211, "short": "Vaughan",
+     "topic": "Courts decline jurisdiction over federal public-service "
+              "workplace disputes covered by the statutory grievance "
+              "scheme"},
+    {"court": "scc", "id": 1876, "short": "McKinley",
+     "topic": "Dishonesty is not automatically just cause for dismissal: "
+              "the contextual, proportional approach"},
+    {"court": "scc", "id": 5271, "short": "Gagnon",
+     "topic": "The five governing principles of the union's duty of fair "
+              "representation"},
+    {"court": "scc", "id": 1877, "short": "Noel (DFR)",
+     "topic": "The duty of fair representation restated: arbitrary, "
+              "discriminatory, bad-faith or seriously negligent conduct"},
+    {"court": "scc", "id": 1724, "short": "Meiorin",
+     "topic": "The unified three-step bona fide occupational requirement "
+              "test for workplace standards and accommodation"},
+    {"court": "scc", "id": 96, "short": "Fraser (loyalty)",
+     "topic": "The federal public servant's duty of loyalty balanced "
+              "against freedom of expression"},
+    {"court": "scc", "id": 106, "short": "Cardinal",
+     "topic": "Procedural fairness binds every public authority; a breach "
+              "voids the decision regardless of the likely outcome"},
+    {"court": "scc", "id": 2574, "short": "CJL v NEB",
+     "topic": "The reasonable-apprehension-of-bias test: what an informed "
+              "person, viewing the matter realistically, would conclude"},
+    {"court": "scc", "id": 2660, "short": "Harrison",
+     "topic": "The Carltona doctrine: ministerial powers are "
+              "presumptively exercisable by responsible departmental "
+              "officials"},
+    {"court": "scc", "id": 2446, "short": "Maple Lodge",
+     "topic": "Fettering of discretion: policy guidelines cannot be "
+              "applied as binding rules excluding relevant considerations"},
+    {"court": "scc", "id": 1879, "short": "Mount Sinai",
+     "topic": "Legitimate expectations are procedural only; public-law "
+              "estoppel against a minister is narrow and yields to "
+              "statute"},
+    {"court": "scc", "id": 1107, "short": "Mobil Oil",
+     "topic": "The futility exception: relief for a fairness breach may "
+              "be withheld where the outcome was legally inevitable"},
+    {"court": "scc", "id": 15438, "short": "Strickland",
+     "topic": "Discretionary refusal of judicial review and the adequate- "
+              "alternative-remedy framework"},
+    {"court": "scc", "id": 1808, "short": "Blencoe",
+     "topic": "Administrative delay is an abuse of process only on "
+              "significant prejudice from unacceptable delay"},
+    {"court": "scc", "id": 19439, "short": "Abrametz",
+     "topic": "The modern three-part test for inordinate administrative "
+              "delay, with proportionate remedies"},
+    {"court": "scc", "id": 20730, "short": "Auer",
+     "topic": "Vavilov reasonableness, with a presumption of validity, "
+              "governs vires challenges to subordinate legislation"},
+    {"court": "scc", "id": 431, "short": "Prassad",
+     "topic": "Administrative tribunals are masters of their own "
+              "procedure, subject only to fairness"},
+    {"court": "fca", "id": 214260, "short": "Sharma",
+     "topic": "Section 44 reports and referrals for permanent residents: "
+              "very limited discretion, no duty to weigh H&C factors"},
+    {"court": "fca", "id": 407446, "short": "Sidhu",
+     "topic": "Misrepresentation under s. 40: the duty of candour and the "
+              "narrow innocent-misrepresentation exception"},
+    {"court": "fca", "id": 33323, "short": "Ragupathy",
+     "topic": "The s. 115(2)(a) danger-opinion framework: danger "
+              "assessment, then risk, then balancing"},
+    {"court": "fca", "id": 521220, "short": "Obazughanmwen",
+     "topic": "Section 44 officers and delegates have no mandate for "
+              "detailed H&C or abuse-of-process inquiries"},
+    {"court": "fca", "id": 35977, "short": "Nagalingam",
+     "topic": "The s. 115(2)(b) refoulement branch: assessing the nature "
+              "and severity of the acts committed"},
+    {"court": "fca", "id": 423316, "short": "Revell",
+     "topic": "An inadmissibility determination and s. 44 referral do not "
+              "engage Charter s. 7, even for long-term permanent "
+              "residents"},
+    {"court": "fca", "id": 143934, "short": "Bermudez",
+     "topic": "Cessation applications: the hearings officer has no "
+              "discretion to weigh H&C factors or a child's best "
+              "interests"},
+    {"court": "fca", "id": 31594, "short": "Coomaraswamy",
+     "topic": "Vacation of refugee protection under s. 109: establishing "
+              "misrepresentation; only untainted evidence on the second "
+              "branch"},
+    {"court": "fca", "id": 36788, "short": "Zeng",
+     "topic": "The three-question test for Article 1E exclusion: status "
+              "substantially similar to nationals of a third country"},
+    {"court": "fca", "id": 32011, "short": "Li (s. 97)",
+     "topic": "The s. 97 protected-person standard is balance of "
+              "probabilities, unlike the lower s. 96 threshold"},
+    {"court": "fca", "id": 36349, "short": "Li (detention)",
+     "topic": "Anticipated future length of detention is assessed on "
+              "proceedings conducted expeditiously"},
+    {"court": "fca", "id": 36729, "short": "C.B. Powell",
+     "topic": "Prematurity: the Customs Act re-determination and CITT "
+              "route must be exhausted before judicial review"},
+    {"court": "fca", "id": 181427, "short": "Bri-Chem",
+     "topic": "CBSA must follow CITT precedent when processing "
+              "corrections; relitigating settled rulings is an abuse of "
+              "process"},
+    {"court": "fca", "id": 407467, "short": "Chen",
+     "topic": "A s. 135 Customs Act action tests only whether the "
+              "contravention occurred; officer conduct belongs to other "
+              "recourse"},
+    {"court": "fca", "id": 36351, "short": "Fritz Marketing",
+     "topic": "Challenges to Detailed Adjustment Statements proceed "
+              "through the Customs Act appeal route, not judicial review"},
+    {"court": "fca", "id": 108541, "short": "Skechers",
+     "topic": "Transaction value: research cost-sharing payments to the "
+              "vendor form part of the price paid or payable"},
+    {"court": "fca", "id": 231365, "short": "Heyser",
+     "topic": "Termination on revocation of reliability status must rest "
+              "on legitimate grounds, reviewable by the Board"},
+    {"court": "fca", "id": 308603, "short": "CPR (fairness)",
+     "topic": "Procedural fairness on judicial review: whether the party "
+              "knew the case to meet and had a full chance to respond"},
+    {"court": "fc", "id": 308992, "short": "McAlpin",
+     "topic": "The s. 44 discretion synthesis: any H&C discretion in "
+              "criminality cases is very limited, if it exists at all"},
+    {"court": "fc", "id": 54573, "short": "Frazee",
+     "topic": "Whether an employer measure labelled administrative is in "
+              "substance discipline: corrective intent and impact"},
+    {"court": "onca", "id": 14196, "short": "Chaudhary",
+     "topic": "Habeas corpus is available for lengthy immigration "
+              "detention of uncertain duration"},
+    # Second tier from the 2026-07 gap audit, added at the maintainer's
+    # direction: the strongest of each reviewer's deliberate exclusions.
+    {"court": "scc", "id": 7865, "short": "Nolet",
+     "topic": "Dual-purpose vehicle searches: a valid regulatory search may "
+              "shift to criminal investigation; the Charter consequences of "
+              "each step assessed on its own footing"},
+    {"court": "scc", "id": 19043, "short": "Horrocks",
+     "topic": "Exclusive labour-arbitration jurisdiction over human-rights "
+              "disputes arising from the collective agreement; the Weber "
+              "framework applied to competing statutory tribunals"},
+    {"court": "scc", "id": 2075, "short": "Parry Sound",
+     "topic": "Statutory employment and human-rights protections are read "
+              "into every collective agreement and enforceable at "
+              "arbitration"},
+    {"court": "scc", "id": 16827, "short": "AJC v Canada",
+     "topic": "Management rights are exercised subject to reasonableness "
+              "and consistency with the collective agreement (standby "
+              "scheduling of counsel)"},
+    {"court": "scc", "id": 2291, "short": "Hilewitz",
+     "topic": "Medical inadmissibility for excessive demand requires "
+              "individualized assessment of the applicant's actual "
+              "circumstances, including family support"},
+    {"court": "fca", "id": 363658, "short": "Tapambwa",
+     "topic": "Restricted PRRA under s. 112(3): no revisiting of the "
+              "exclusion finding; the scope of protection for excluded "
+              "persons"},
+    # The CIU's published "Important Decisions" list (ciu-sdi.ca), ingested
+    # 2026-07-23; topics adapted from the union's own subject lines.
+    {"court": "fpslreb", "id": 358837, "short": "Bartlett",
+     "topic": "Hours of work and variable shift scheduling under the FB "
+              "collective agreement (Border Services officers)"},
+    {"court": "fpslreb", "id": 359582, "short": "Legros",
+     "topic": "Age discrimination grievance against the CBSA"},
+    {"court": "fpslreb", "id": 359605, "short": "Bétournay",
+     "topic": "Breach of trust, disguised disciplinary action and "
+              "termination (CRA)"},
+    {"court": "fpslreb", "id": 359635, "short": "Frezza",
+     "topic": "Bad faith and disguised discipline in the federal public "
+              "service (DND)"},
+    {"court": "fpslreb", "id": 360491, "short": "PSAC v TB (CBSA) 2018-66",
+     "topic": "Policy grievance: interpretation of the FB collective "
+              "agreement (PSAC v Treasury Board / CBSA)"},
+    {"court": "fpslreb", "id": 361700, "short": "PSAC v TB (CBSA) 2018-82",
+     "topic": "Policy grievance: interpretation of the FB collective "
+              "agreement (PSAC v Treasury Board / CBSA)"},
+    {"court": "fpslreb", "id": 361701, "short": "Michaud",
+     "topic": "Suspension grievance in the federal public service (CRA)"},
+    {"court": "fpslreb", "id": 407654, "short": "Hyslop",
+     "topic": "Suspension grievance of a CBSA employee"},
+    {"court": "fpslreb", "id": 407686, "short": "Girard",
+     "topic": "Suspension and the effective date of termination (CRA)"},
+    {"court": "fpslreb", "id": 420718, "short": "Edwards",
+     "topic": "Sick leave, personal-needs leave and disability "
+              "discrimination under the FB collective agreement (CBSA)"},
+    {"court": "fpslreb", "id": 420308, "short": "Touchette",
+     "topic": "Proportionality of discipline; suspension of a CBSA "
+              "employee reduced"},
+    {"court": "fpslreb", "id": 521306, "short": "Guay-Bassett",
+     "topic": "Scheduling, designated paid holidays and sick leave under "
+              "the FB collective agreement (CBSA)"},
+    {"court": "fpslreb", "id": 520990, "short": "Menzies",
+     "topic": "Progressive discipline of a CBSA border services officer; the "
+              "lock-step approach to discipline rejected"},
+    {"court": "fpslreb", "id": 521231, "short": "Kline",
+     "topic": "Bad-faith termination of a CBSA employee; reinstatement and "
+              "damages"},
+    {"court": "fpslreb", "id": 521195, "short": "Sousa Dias",
+     "topic": "Discipline and termination grievance of a CBSA employee"},
+    {"court": "fpslreb", "id": 521082, "short": "Anderson",
+     "topic": "CBSA grievance; interpretation of the FB-group collective "
+              "agreement"},
+    {"court": "fpslreb", "id": 520948, "short": "Burlacu",
+     "topic": "CBSA; occupational health and safety and staffing"},
+    {"court": "fpslreb", "id": 483604, "short": "Malik",
+     "topic": "Discipline and termination grievance of a CBSA employee"},
+    {"court": "fpslreb", "id": 500554, "short": "Andruszkiewicz",
+     "topic": "Unfair labour practice complaint involving the CBSA"},
+    {"court": "fpslreb", "id": 359013, "short": "PSAC v TB (CBSA)",
+     "topic": "Policy grievance; collective agreement interpretation at the "
+              "CBSA"},
+    {"court": "fpslreb", "id": 359065, "short": "Martin-Ivie",
+     "topic": "Occupational health and safety; the arming and safety of CBSA "
+              "border officers"},
+    {"court": "fpslreb", "id": 358886, "short": "Basra (2012)",
+     "topic": "Discipline and termination grievance; a later proceeding in "
+              "the leading Basra line"},
+    {"court": "fpslreb", "id": 358025, "short": "Basra (2007)",
+     "topic": "The foundational Basra decision on discipline and the burden "
+              "of proof in a grievance"},
+    {"court": "fpslreb", "id": 358150, "short": "Quadrini",
+     "topic": "Unfair labour practice and freedom of expression in the "
+              "federal public service"},
+    {"court": "fpslreb", "id": 358180, "short": "Pepper",
+     "topic": "Discipline and termination; frequently-cited principles on "
+              "just cause"},
+    {"court": "fpslreb", "id": 358097, "short": "Richmond",
+     "topic": "Classification grievance in the federal public service"},
+    {"court": "fpslreb", "id": 358890, "short": "Baldasaro and Thiessen",
+     "topic": "Hours of work and overtime under a collective agreement"},
+    {"court": "fpslreb", "id": 358203, "short": "PSAC v TB (pay)",
+     "topic": "Collective agreement and pay administration policy grievance"},
+    {"court": "fpslreb", "id": 360456, "short": "Kinhnicki",
+     "topic": "Occupational health and safety; a refusal to work in a customs "
+              "context"},
+
+    # --- Court of Appeal for Ontario (own Decisia database) ---
+    {"court": "onca", "id": 22592, "short": "Pike",
+     "topic": "Border searches of personal electronic devices; following and "
+              "extending Canfield, Customs Act s. 99(1)(a) is unconstitutional "
+              "under Charter s. 8 insofar as it permits suspicionless device "
+              "searches -- a reasonable-suspicion requirement is read in "
+              "(suspended declaration of invalidity)"},
+
+    # --- Canada Industrial Relations Board ---
+    {"court": "cirb", "id": 519772, "short": "Watson",
+     "topic": "Duty of fair representation and a mandatory vaccination policy "
+              "under the Canada Labour Code"},
+    {"court": "cirb", "id": 5478, "short": "McRaeJackson",
+     "topic": "The leading test for the duty of fair representation under "
+              "s. 37 of the Canada Labour Code"},
+    {"court": "cirb", "id": 5491, "short": "Securicor",
+     "topic": "Certification and bargaining-unit determination under the "
+              "Canada Labour Code"},
+    {"court": "cirb", "id": 5593, "short": "Dover Industries",
+     "topic": "Successor rights on the sale of a business under the Canada "
+              "Labour Code"},
+    {"court": "cirb", "id": 301063, "short": "Swissport",
+     "topic": "Unfair labour practice complaint under the Canada Labour Code"},
+    {"court": "cirb", "id": 5599, "short": "Cooney Transport",
+     "topic": "Related-employer (common-employer) declaration under the "
+              "Canada Labour Code"},
+]
+
+# Provincial appellate decisions, sourced from CanLII as cached plain text.
+# CanLII fronts its pages with DataDome bot protection, so these CANNOT be
+# fetched automatically: open the CanLII page in a real browser, select-all,
+# and save the page text to data/raw/caselaw/<file>. Because there is no
+# parseable metadata block, the name / citation / date / court are curated
+# here rather than parsed. The paragraph text still parses the same way --
+# CanLII renders the same bracketed [N] paragraph numbers as Lexum.
+CANLII_CASES = [
+    {"file": "canlii-2020abca383.txt",
+     "canlii_path": "en/ab/abca/doc/2020/2020abca383/2020abca383.html",
+     "short": "Canfield",
+     "name": "R v Canfield",
+     "cite": "2020 ABCA 383",
+     "report": "395 CCC (3d) 483",
+     "date": "2020-10-29",
+     "court_name": "Court of Appeal of Alberta",
+     "topic": "Border searches of personal electronic devices; Customs Act "
+              "s. 99(1)(a) held unconstitutional under Charter s. 8 insofar as "
+              "it imposes no limits on device searches at the border (leave to "
+              "appeal to the SCC refused; binding in Alberta and persuasive "
+              "elsewhere)"},
+    {"file": "canlii-2021canlii95825.txt",
+     "canlii_path": "en/ca/irb/doc/2021/2021canlii95825/2021canlii95825.html",
+     "short": "TC1-05038",
+     "name": "IRB Jurisprudential Guide TC1-05038 — X (Re)",
+     "cite": "2021 CanLII 95825 (CA IRB)",
+     "report": "",
+     "date": "2021-08-30",
+     "court_name": "Immigration and Refugee Board — Refugee Appeal Division",
+     "topic": "RAD jurisprudential guide on the 'generalized risk' exception "
+              "to protection under IRPA s. 97(1)(b)(ii) in claims involving "
+              "gang targeting, extortion or other criminality (in force; "
+              "identified as a guide July 2022)"},
+    # The foreign-conviction equivalency line -- the FCA authorities the
+    # curated us_equivalency dataset applies. Pre-Lexum era, so CanLII-only:
+    # the raw text arrives via the manual browser-capture workflow and
+    # ingestion skips gracefully until each file exists.
+    {"file": "canlii-1981-brannson.txt",
+     "canlii_path": "en/ca/fca/doc/1980/1980canlii4197/1980canlii4197.html",
+     "short": "Brannson",
+     "name": "Brannson v Canada (Minister of Employment and Immigration)",
+     "cite": "[1981] 2 FC 141 (CA)",
+     "report": "1980 CanLII 4197",
+     "date": "1980-06-05",
+     "court_name": "Federal Court of Appeal",
+     "topic": "Foreign-conviction equivalency: compare the essential elements "
+              "of the foreign and Canadian offences; the foreign offence's "
+              "wording and the evidence may both be examined (first of the "
+              "equivalency trilogy)"},
+    {"file": "canlii-1987-hill.txt",
+     "canlii_path": "en/ca/fca/doc/1987/1987canlii9881/1987canlii9881.html",
+     "short": "Hill",
+     "name": "Hill v Canada (Minister of Employment and Immigration)",
+     "cite": "(1987), 73 NR 315 (FCA)",
+     "report": "1987 CanLII 9881; 1 Imm LR (2d) 1",
+     "date": "1987-01-29",
+     "court_name": "Federal Court of Appeal",
+     "topic": "The governing three-branch equivalency test: (1) compare the "
+              "precise wording and essential elements of the two provisions; "
+              "(2) examine the evidence of what was actually done against the "
+              "Canadian offence's elements; or (3) both combined"},
+    # 2026-07 gap audit, pre-Decisia authorities: text arrives via the
+    # browser-capture workflow (or a direct PDF fetch for Wm. Scott);
+    # ingestion skips gracefully until each file exists.
+    {"file": "canlii-1991-rasaratnam.txt",
+     "canlii_path": "en/ca/fca/doc/1991/1991canlii13517/1991canlii13517.html",
+     "short": "Rasaratnam", "name": "Rasaratnam v Canada (MEI)",
+     "cite": "[1992] 1 FC 706 (CA)", "report": "1991 CanLII 13517",
+     "date": "1991-12-05", "court_name": "Federal Court of Appeal",
+     "topic": "The two-prong internal flight alternative test: no serious "
+              "possibility of persecution in the IFA, and reasonableness of "
+              "relocation"},
+    {"file": "canlii-1993-thirunavukkarasu.txt",
+     "canlii_path": "en/ca/fca/doc/1993/1993canlii3011/1993canlii3011.html",
+     "short": "Thirunavukkarasu", "name": "Thirunavukkarasu v Canada (MEI)",
+     "cite": "[1994] 1 FC 589 (CA)", "report": "1993 CanLII 3011",
+     "date": "1993-11-10", "court_name": "Federal Court of Appeal",
+     "topic": "The reasonableness prong of the IFA test: an objective "
+              "standard, with notice required before an IFA is raised"},
+    {"file": "canlii-2000-ranganathan.txt",
+     "canlii_path": "en/ca/fca/doc/2000/2000canlii16789/2000canlii16789.html",
+     "short": "Ranganathan", "name": "Ranganathan v Canada (MCI)",
+     "cite": "[2001] 2 FC 164 (CA)", "report": "2000 CanLII 16789",
+     "date": "2000-12-21", "court_name": "Federal Court of Appeal",
+     "topic": "The very high threshold for an unreasonable IFA: conditions "
+              "jeopardizing life and safety in travelling or relocating"},
+    {"file": "canlii-1994-sahin.txt",
+     "canlii_path": "en/ca/fct/doc/1994/1994canlii3521/1994canlii3521.html",
+     "short": "Sahin", "name": "Sahin v Canada (MCI)",
+     "cite": "[1995] 1 FC 214 (TD)", "report": "1994 CanLII 3521",
+     "date": "1994-10-14", "court_name": "Federal Court (Trial Division)",
+     "topic": "The s. 7 factors for continued immigration detention -- the "
+              "source of IRPR s. 248: reasons, length, future duration, "
+              "delay attribution, alternatives"},
+    {"file": "canlii-1979-maldonado.txt",
+     "canlii_path": "en/ca/fca/doc/1979/1979canlii4098/1979canlii4098.html",
+     "short": "Maldonado", "name": "Maldonado v Canada (MEI)",
+     "cite": "[1980] 2 FC 302 (CA)", "report": "1979 CanLII 4098",
+     "date": "1979-11-19", "court_name": "Federal Court of Appeal",
+     "topic": "The presumption that a claimant's sworn testimony is true "
+              "unless there is reason to doubt it"},
+    {"file": "canlii-1989-adjei.txt",
+     "canlii_path": "en/ca/fca/doc/1989/1989canlii5184/1989canlii5184.html",
+     "short": "Adjei", "name": "Adjei v Canada (MEI)",
+     "cite": "[1989] 2 FC 680 (CA)", "report": "1989 CanLII 9466",
+     "date": "1989-01-27", "court_name": "Federal Court of Appeal",
+     "topic": "The s. 96 standard of proof: a reasonable chance or serious "
+              "possibility of persecution, less than balance of probabilities"},
+    {"file": "canlii-1988-toth.txt",
+     "canlii_path": "en/ca/fca/doc/1988/1988canlii1420/1988canlii1420.html",
+     "short": "Toth", "name": "Toth v Canada (MEI)",
+     "cite": "(1988), 86 NR 302 (FCA)", "report": "1988 CanLII 1420",
+     "date": "1988-06-21", "court_name": "Federal Court of Appeal",
+     "topic": "The tripartite stay test on motions to stay removal: serious "
+              "issue, irreparable harm, balance of convenience"},
+    {"file": "canlii-1980-burchill.txt",
+     "canlii_path": "en/ca/fca/doc/1980/1980canlii4207/1980canlii4207.html",
+     "short": "Burchill", "name": "Burchill v Canada (Attorney General)",
+     "cite": "[1981] 1 FC 109 (CA)", "report": "1980 CanLII 4207",
+     "date": "1980-05-20", "court_name": "Federal Court of Appeal",
+     "topic": "A grievor cannot change the nature of a grievance when it "
+              "reaches adjudication"},
+    {"file": "canlii-1992-lidder.txt",
+     "canlii_path": "en/ca/fca/doc/1992/1992canlii14712/1992canlii14712.html",
+     "short": "Lidder", "name": "Canada (MEI) v Lidder",
+     "cite": "[1992] 2 FC 621 (CA)", "report": "1992 CanLII 14712",
+     "date": "1992-03-10", "court_name": "Federal Court of Appeal",
+     "topic": "Estoppel and legitimate expectations cannot defeat mandatory "
+              "statutory requirements: an officer's representation cannot "
+              "confer a status or bypass conditions the Act imposes"},
+    # Tipple ([1985] FCJ No 818) is QuickLaw-only -- no CanLII record
+    # exists. Its de-novo-cures-defects rule enters the corpus through
+    # Patanguli, 2015 FCA 291 (CASES entry), which quotes and applies it.
+    {"file": "canlii-1976-wmscott.txt",
+     "canlii_path": "en/bc/bclrb/doc/1976/1976canlii1017/1976canlii1017.html",
+     "short": "Wm. Scott", "name": "Wm. Scott & Company Ltd v CFAWU Local P-162",
+     "cite": "[1977] 1 Can LRBR 1 (BCLRB)", "report": "",
+     "date": "1976-10-26",
+     "court_name": "British Columbia Labour Relations Board",
+     "topic": "The three-question discipline framework: was there cause; was "
+              "the penalty excessive; what lesser penalty should substitute"},
+    {"file": "canlii-1996-li.txt",
+     "canlii_path": "en/ca/fca/doc/1996/1996canlii4086/1996canlii4086.html",
+     "short": "Li",
+     "name": "Li v Canada (Minister of Citizenship and Immigration)",
+     "cite": "[1997] 1 FC 235 (CA)",
+     "report": "1996 CanLII 4086",
+     "date": "1996-08-07",
+     "court_name": "Federal Court of Appeal",
+     "topic": "Equivalency compares the essential elements of the offences, "
+              "not procedural or evidentiary rules; defences available under "
+              "the foreign law are part of the comparison"},
+]
+
+# In-force jurisprudential guides of the Immigration and Refugee Board's Refugee
+# Appeal Division -- decisions the Board designates as models that members apply
+# to similar cases or explain why not. Full text is hosted on the IRB's own
+# site; 'topic' is curated, while the date and paragraphs are parsed from the
+# page. (A fourth in-force guide, TC1-05038, is hosted only on CanLII.)
+IRB_GUIDES = [
+    {"file": "MB8-00025",
+     "url": "https://www.irb-cisr.gc.ca/en/decisions/Pages/mb8-00025.aspx",
+     "topic": "Exclusion from refugee protection under Article 1E where the "
+              "claimant has protected status in a third country (Haiti / Brazil)"},
+    {"file": "TB7-01837",
+     "url": "https://www.irb-cisr.gc.ca/en/decisions/Pages/TB7-01837.aspx",
+     "topic": "Persecution of Ahmadis in Pakistan; state protection and the "
+              "availability of an internal flight alternative"},
+    {"file": "TB4-05778",
+     "url": "https://www.irb-cisr.gc.ca/en/decisions/Pages/TB4-05778.aspx",
+     "topic": "Whether a North Korean refugee claimant has deemed citizenship "
+              "of South Korea and protection available there"},
+]
+
+
+def _get(url, cache_name):
+    """Fetch a page, caching the raw HTML under data/raw/caselaw.
+
+    Retries once on HTTP 403/429 -- the Lexum hosts rate-limit by IP.
+    """
+    cache = _RAW / cache_name
+    if cache.exists():
+        return cache.read_text(encoding="utf-8")
+    req = urllib.request.Request(url, headers={"User-Agent": _UA})
+    text = None
+    for attempt in range(2):
+        time.sleep(_THROTTLE if attempt == 0 else 25.0)
+        try:
+            with urllib.request.urlopen(req, timeout=60) as resp:
+                text = resp.read().decode("utf-8", errors="replace")
+            break
+        except urllib.error.HTTPError as exc:
+            if exc.code in (403, 429) and attempt == 0:
+                continue
+            raise
+    _RAW.mkdir(parents=True, exist_ok=True)
+    cache.write_text(text, encoding="utf-8")
+    return text
+
+
+def _fetch(court, item_id):
+    """Return a Lexum court decision's iframe HTML."""
+    url = COURTS[court][1].format(id=item_id) + "?iframe=true"
+    return _get(url, f"{court}-{item_id}.html")
+
+
+def _irb_date(texts):
+    """Pull the ISO 'Date of decision' from an IRB decision's front matter.
+
+    The label and the date sometimes sit in separate elements, so the search
+    runs over the joined text rather than block by block.
+    """
+    m = re.search(r"Date of decision:?\s*([A-Za-z]+)\s+(\d{1,2}),?\s*(\d{4})",
+                  " ".join(texts))
+    if m and m.group(1).lower() in _MONTHS:
+        return (f"{m.group(3)}-{_MONTHS[m.group(1).lower()]:02d}-"
+                f"{int(m.group(2)):02d}")
+    return ""
+
+
+def _metadata(soup):
+    """Return (case_name, {label: value}) from the decision's metadata block."""
+    box = soup.find("div", class_="metadata")
+    if not box:
+        return "", {}
+    title = box.find("h3", class_="title")
+    name = _norm(title.get_text()) if title else ""
+    fields = {}
+    for row in box.find_all("tr"):
+        label = row.find("td", class_="label")
+        value = row.find("td", class_="metadata")
+        if label and value:
+            fields[_norm(label.get_text()).lower()] = _norm(value.get_text())
+    return name, fields
+
+
+def _body(soup):
+    """Locate the container holding the decision text.
+
+    Handles the Lexum court pages and the IRB's SharePoint pages, whose text
+    sits in a 'RichHtmlField' rich-text div.
+    """
+    return (soup.find(id="document-content")
+            or soup.find("div", class_="documentcontent")
+            or soup.find("div", class_="WordSection1")
+            or soup.find("div", id=lambda v: v and "RichHtmlField" in v)
+            or soup.find("div", class_="ms-rtestate-field")
+            or soup.body or soup)
+
+
+def _paragraphs(soup):
+    """Return (is_numbered, [(label, text), ...]) for the judgment body.
+
+    Modern judgments open each paragraph with a bracketed number "[N]". They are
+    detected by content -- a run of sequentially numbered <p> blocks -- so the
+    parser does not depend on Word style names, which vary by court and era.
+    Every <p> between one numbered opener and the next belongs to that paragraph.
+    Older, unnumbered decisions fall back to taking every <p> in document order.
+    """
+    body = _body(soup)
+    for aside in body.find_all("aside", class_="wb-fnote"):
+        aside.decompose()                        # drop IRB/WET footnote blocks
+    blocks = [p for p in body.find_all("p")
+              if "MsoFootnoteText" not in (p.get("class") or [])]
+    texts = [p.get_text() for p in blocks]
+
+    # Some pre-2009 board pages (e.g. PSSRB 2003) carry the reasons in bare
+    # <div> blocks with no <p> markup at all, so the <p> strategy collects
+    # only header furniture. When it captured almost none of the body's
+    # text, hand the whole body to the plain-text parser instead -- the same
+    # one the CanLII paste captures use, with its own numbering detection
+    # and apparatus cutoff.
+    if sum(len(t) for t in texts) < 0.2 * len(body.get_text()):
+        return _text_paragraphs(body.get_text("\n"))
+
+    # Drop the post-reasons apparatus (appended legislation, solicitors list);
+    # it is not part of the reasons and would otherwise swell the last paragraph.
+    for i, raw in enumerate(texts):
+        if _APPARATUS.match(raw):
+            texts = texts[:i]
+            break
+
+    # A paragraph opens with its number: "[12]" (most decisions) or a bare "12"
+    # followed by wide tab spacing (pre-2009 decisions). The brackets are
+    # self-identifying; a bare number must have 2+ trailing spaces, which
+    # rejects quoted enumerations ("2. ..."). The sequential check rejects
+    # stray bracketed years like "[1998]".
+    openers = {}          # block index -> paragraph number
+    expected = 1
+    for i, raw in enumerate(texts):
+        match = (re.match(r"\s*\[\s*(\d+)\s*\]", raw)
+                 or re.match(r"\s*(\d+)\s{2,}\S", raw))
+        if match:
+            n = int(match.group(1))
+            if expected <= n <= expected + 2:    # sequential, small-gap tolerant
+                openers[i] = n
+                expected = n + 1
+
+    if len(openers) < 5:                         # unnumbered: take every <p>
+        paras = [(str(j), _norm(t)) for j, t in enumerate(texts, start=1)]
+        return False, [(n, t) for n, t in paras if len(t) > 1]
+
+    paras, num, buf = [], None, []
+    for i, raw in enumerate(texts):
+        if i in openers:
+            if num is not None:
+                paras.append((str(num), _norm(" ".join(buf))))
+            num = openers[i]
+            buf = [re.sub(r"^\s*\[?\s*\d+\s*\]?\s*", "", raw)]
+        elif num is not None:
+            buf.append(raw)
+    if num is not None:
+        paras.append((str(num), _norm(" ".join(buf))))
+    return True, [(n, t) for n, t in paras if t]
+
+
+def _split_text(text, limit):
+    """Split text longer than `limit` into pieces, breaking on a sentence or
+    word boundary so no single chunk blows the retrieval/reranker budget."""
+    if len(text) <= limit:
+        return [text]
+    pieces, start = [], 0
+    while start < len(text):
+        if len(text) - start <= limit:
+            pieces.append(text[start:])
+            break
+        window = text[start:start + limit]
+        cut = window.rfind(". ")
+        cut = cut + 1 if cut > limit // 2 else window.rfind(" ")
+        if cut <= 0:
+            cut = limit
+        pieces.append(text[start:start + cut])
+        start += cut
+    return [p.strip() for p in pieces if p.strip()]
+
+
+def _chunk(paras):
+    """Group consecutive paragraphs into ~_CHUNK_CHARS-sized chunks, first
+    splitting any single paragraph that on its own exceeds the budget."""
+    units = []
+    for label, text in paras:
+        for piece in _split_text(text, _CHUNK_CHARS):
+            units.append((label, piece))
+    chunks, buf, size = [], [], 0
+    for label, text in units:
+        if buf and size + len(text) > _CHUNK_CHARS:
+            chunks.append(buf)
+            buf, size = [], 0
+        buf.append((label, text))
+        size += len(text)
+    if buf:
+        chunks.append(buf)
+    return chunks
+
+
+# The post-reasons apparatus in plain-text (CanLII) decisions: the hearing
+# date, filing formula, counsel list, or signature block -- and everything
+# after them (endnotes, the CanLII page footer). The "(signed)" marker is how
+# IRB decisions end; their endnotes reuse bracketed [N] numbering and would
+# otherwise collide with the paragraph sequence once N catches up.
+_TEXT_APPARATUS = re.compile(
+    r"^\s*(Appeal heard on\b|Memorandum filed\b|Appearances:|\(signed\)|"
+    r"APPENDIX\b|Solicitors?\s+for\b|NAMES\s+OF\s+COUNSEL\b)", re.I)
+
+
+def _text_paragraphs(text):
+    """Return (is_numbered, [(label, text), ...]) for a plain-text judgment.
+
+    The text analogue of _paragraphs() for decisions cached as page text
+    rather than HTML: physical lines replace <p> blocks. A line opening with a
+    sequential bracketed number "[N]" starts paragraph N; lines before [1]
+    (style of cause, coram) and everything from the first apparatus marker on
+    (counsel, endnotes, site footer) are dropped."""
+    lines = [ln.strip() for ln in text.splitlines()]
+
+    for i, ln in enumerate(lines):
+        if _TEXT_APPARATUS.match(ln):
+            lines = lines[:i]
+            break
+
+    openers = {}          # line index -> paragraph number
+    expected = 1
+    for i, ln in enumerate(lines):
+        match = re.match(r"\[\s*(\d+)\s*\]", ln)
+        if match:
+            n = int(match.group(1))
+            if expected <= n <= expected + 2:
+                openers[i] = n
+                expected = n + 1
+
+    def _unnumbered():
+        paras = [(str(j), _norm(ln)) for j, ln in enumerate(lines, start=1)]
+        return False, [(n, t) for n, t in paras if len(t) > 1]
+
+    if len(openers) < 5:
+        return _unnumbered()
+
+    paras, num, buf = [], None, []
+    for i, ln in enumerate(lines):
+        if i in openers:
+            if num is not None:
+                paras.append((str(num), _norm(" ".join(buf))))
+            num = openers[i]
+            buf = [re.sub(r"^\[\s*\d+\s*\]\s*", "", ln)]
+        elif num is not None and ln:
+            buf.append(ln)
+    if num is not None:
+        paras.append((str(num), _norm(" ".join(buf))))
+    paras = [(n, t) for n, t in paras if t]
+
+    # A pre-2001 report-style decision has unnumbered prose reasons but a
+    # trailing footnote block whose own [1]..[N] sequence the detector will
+    # happily latch onto (Li: 26 footnotes, 4% of the text). If the
+    # "numbered paragraphs" cover only a sliver of the document, the real
+    # reasons are the unnumbered prose -- fall back to it.
+    if sum(len(t) for _, t in paras) < 0.2 * len(text):
+        return _unnumbered()
+    return True, paras
+
+
+def _canlii_chunks(case, text):
+    """Build CanLex chunk dicts for one CanLII cached-text decision."""
+    citation = f"{case['name']}, {case['cite']}"
+    source_url = f"https://www.canlii.org/{case['canlii_path']}"
+    modern, paras = _text_paragraphs(text)
+    chunks = []
+    for i, group in enumerate(_chunk(paras), start=1):
+        if modern:
+            first, last = group[0][0], group[-1][0]
+            locator = (f"para {first}" if first == last
+                       else f"paras {first}–{last}")
+        else:
+            locator = f"excerpt {i}"
+        chunks.append({
+            "id": f"canlii-{case['cite'].replace(' ', '').lower()}-{i}",
+            "doc_type": "caselaw",
+            "act_code": case["cite"],
+            "act_short": case["short"],
+            "act_name": case["name"],
+            "section": "",
+            "citation": citation,
+            "marginal_note": locator,
+            "heading": case["topic"],
+            "part": case["court_name"],
+            "division": "",
+            "text": "\n\n".join(t for _, t in group),
+            "current_to": case["date"],
+            "last_amended": "",
+            "history": case.get("report", ""),
+            "source_url": source_url,
+        })
+    return chunks, citation, len(paras)
+
+
+def _decision_chunks(case, soup):
+    """Build CanLex chunk dicts for one decision."""
+    court_name, item_tmpl = COURTS[case["court"]]
+    name, fields = _metadata(soup)
+    name = name or case["short"]
+    cite = (fields.get("neutral citation") or fields.get("citation")
+            or fields.get("report") or "")
+    report = fields.get("report", "")
+    date = fields.get("date") or fields.get("decision rendered") or ""
+    citation = f"{name}, {cite}" if cite else name
+    item_url = item_tmpl.format(id=case["id"])
+    modern, paras = _paragraphs(soup)
+    chunks = []
+    for i, group in enumerate(_chunk(paras), start=1):
+        if modern:
+            first, last = group[0][0], group[-1][0]
+            locator = (f"para {first}" if first == last
+                       else f"paras {first}–{last}")
+        else:
+            locator = f"excerpt {i}"
+        chunks.append({
+            "id": f"{case['court']}-{case['id']}-{i}",
+            "doc_type": "caselaw",
+            "act_code": cite or f"{case['court'].upper()} item {case['id']}",
+            "act_short": case["short"],
+            "act_name": name,
+            "section": "",
+            "citation": citation,
+            "marginal_note": locator,
+            "heading": case["topic"],
+            "part": court_name,
+            "division": "",
+            "text": "\n\n".join(t for _, t in group),
+            "current_to": date,
+            "last_amended": "",
+            "history": report if report and report != cite else "",
+            "source_url": item_url,
+        })
+    return chunks, citation, len(paras)
+
+
+def _irb_chunks(guide, soup):
+    """Build CanLex chunk dicts for one IRB jurisprudential guide."""
+    cite = f"IRB Jurisprudential Guide {guide['file']}"
+    date = _irb_date(p.get_text() for p in _body(soup).find_all("p"))
+    modern, paras = _paragraphs(soup)
+    chunks = []
+    for i, group in enumerate(_chunk(paras), start=1):
+        if modern:
+            first, last = group[0][0], group[-1][0]
+            locator = (f"para {first}" if first == last
+                       else f"paras {first}–{last}")
+        else:
+            locator = f"excerpt {i}"
+        chunks.append({
+            "id": f"irb-{guide['file']}-{i}",
+            "doc_type": "caselaw",
+            "act_code": guide["file"],
+            "act_short": guide["file"],
+            "act_name": cite,
+            "section": "",
+            "citation": cite,
+            "marginal_note": locator,
+            "heading": guide["topic"],
+            "part": "Immigration and Refugee Board — Refugee Appeal Division",
+            "division": "",
+            "text": "\n\n".join(t for _, t in group),
+            "current_to": date,
+            "last_amended": "",
+            "history": "",
+            "source_url": guide["url"],
+        })
+    return chunks, cite, len(paras)
+
+
+def build(allow_shrink=False):
+    """Fetch, parse and chunk every curated decision into caselaw.json.
+
+    Returns True when the corpus was written, False when the write was refused
+    -- a Lexum CAPTCHA wall or an upstream re-platforming makes every fetch
+    fail at once, and the rebuild-then-write shape would turn that into an
+    emptied caselaw.json.
+    """
+    # Every path that drops a decision records it here, keyed the way the
+    # chunks are (act_short): Lexum throttles, CanLII captures go missing and
+    # court templates shift, so on any given run some decisions fail while the
+    # rest succeed -- and the rebuild-then-write shape would quietly delete
+    # exactly those. `failed` is what preserve_dropped is allowed to restore,
+    # so decisions removed on purpose (deleted from CASES) stay gone.
+    all_chunks, failed = [], set()
+    for case in CASES:
+        try:
+            soup = BeautifulSoup(_fetch(case["court"], case["id"]),
+                                 "html.parser")
+        except Exception as exc:
+            print(f"  !! {case['short']}: fetch failed -- "
+                  f"{type(exc).__name__}: {exc}")
+            failed.add(case["short"])
+            continue
+        chunks, citation, n_paras = _decision_chunks(case, soup)
+        if not chunks:
+            print(f"  !! {case['short']} ({case['court']} item "
+                  f"{case['id']}): 0 chunks -- check parsing")
+            failed.add(case["short"])
+            continue
+        all_chunks.extend(chunks)
+        print(f"  {case['court']:4s} {case['short']:20s} {n_paras:4d} paras -> "
+              f"{len(chunks):3d} chunks   {citation}")
+    for case in CANLII_CASES:
+        cache = _RAW / case["file"]
+        if not cache.exists():
+            print(f"  !! {case['short']}: {cache} missing -- CanLII blocks "
+                  f"automated fetches (DataDome). Open https://www.canlii.org/"
+                  f"{case['canlii_path']} in a browser and save the page text "
+                  f"to that path, then re-run.")
+            failed.add(case["short"])
+            continue
+        text = cache.read_text(encoding="utf-8-sig")
+        chunks, citation, n_paras = _canlii_chunks(case, text)
+        if not chunks:
+            print(f"  !! {case['short']} ({case['file']}): 0 chunks -- "
+                  f"check parsing")
+            failed.add(case["short"])
+            continue
+        all_chunks.extend(chunks)
+        print(f"  cnli {case['short']:20s} {n_paras:4d} paras -> "
+              f"{len(chunks):3d} chunks   {citation}")
+    for guide in IRB_GUIDES:
+        try:
+            soup = BeautifulSoup(_get(guide["url"], f"irb-{guide['file']}.html"),
+                                 "html.parser")
+        except Exception as exc:
+            print(f"  !! {guide['file']}: fetch failed -- "
+                  f"{type(exc).__name__}: {exc}")
+            failed.add(guide["file"])
+            continue
+        chunks, citation, n_paras = _irb_chunks(guide, soup)
+        if not chunks:
+            print(f"  !! {guide['file']}: 0 chunks -- check parsing")
+            failed.add(guide["file"])
+            continue
+        all_chunks.extend(chunks)
+        print(f"  irb  {guide['file']:20s} {n_paras:4d} paras -> "
+              f"{len(chunks):3d} chunks   {citation}")
+    # act_short is the per-decision identity in this corpus (section is "" for
+    # every case-law chunk), and it is unique across all three lists.
+    all_chunks, preserved = preserve_dropped(all_chunks, stored_chunks(OUT),
+                                             key=lambda c: c["act_short"],
+                                             only=failed)
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} decision(s) "
+              f"that failed this run: {', '.join(preserved)}")
+    # indent=1 is what this corpus was written with; changing it would rewrite
+    # every line of caselaw.json for no content change.
+    if not write_corpus(OUT, all_chunks, indent=1, allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} case-law chunks from "
+          f"{len(CASES) + len(CANLII_CASES) + len(IRB_GUIDES)} decisions "
+          f"-> {OUT}")
+    return True
+
+
+def main():
+    ok = build(allow_shrink="--allow-shrink" in sys.argv)
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/charter.py

@@ -0,0 +1,119 @@
+"""Ingest the Charter and border-relevant Constitution Act, 1867 provisions.
+
+The Constitution has no Justice Laws XML, so the corpus long had a hole a
+large fraction of its own case law turns on: Simmons, Canfield, Pike and the
+detention line all apply Charter ss. 7-10 and 24, yet "what does s. 10(b)
+actually say?" returned nothing. Justice Laws publishes the Constitution Acts
+as stable public HTML (laws-lois.justice.gc.ca/eng/const/FullText.html) with
+a regular MarginalNote/Section markup, parsed here into legislation-shaped
+chunks. Scope: the whole Charter (1982 ss. 1-34), Aboriginal rights s. 35,
+the supremacy clause s. 52, and the 1867 heads-of-power ss. 91-92.
+
+    python -m canlex.charter
+"""
+import re
+import sys
+
+from ._common import fetch_cached, norm_ws as _norm, write_corpus
+from .config import PROCESSED_DIR, RAW_DIR
+
+URL = "https://laws-lois.justice.gc.ca/eng/const/FullText.html"
+RAW = RAW_DIR / "charter"
+OUT_FILE = PROCESSED_DIR / "charter.json"
+
+# (act filter, sections wanted). The 1982 Act's Charter is ss. 1-34; s. 35 is
+# Part II (Aboriginal rights) and s. 52 the supremacy clause -- both cited
+# constantly alongside Charter arguments. From 1867, only the ss. 91-92
+# division of powers earns its place in a border-law corpus.
+_WANT_1982 = {str(n) for n in range(1, 36)} | {"52"}
+_WANT_1867 = {"91", "92"}
+
+_MN = re.compile(
+    r'<p class="MarginalNote">(?:<span[^>]*>.*?</span>)?(.*?)</p>', re.S)
+_SEC_ID = re.compile(r'<span class="sectionLabel">([0-9.]+)</span>')
+
+
+def _strip(html):
+    """Readable text from a fragment: block ends become newlines, tags go."""
+    html = re.sub(r"</(?:p|li|h\d)>", "\n", html)
+    html = re.sub(r"<[^>]+>", " ", html)
+    return "\n".join(p for p in (_norm(line) for line in html.split("\n")) if p)
+
+
+def _sections(segment_html):
+    """Yield (number, marginal_note, text) for each section in a segment."""
+    notes = list(_MN.finditer(segment_html))
+    for i, m in enumerate(notes):
+        end = notes[i + 1].start() if i + 1 < len(notes) else len(segment_html)
+        body_html = segment_html[m.end():end]
+        num_m = _SEC_ID.search(body_html)
+        if not num_m:
+            continue
+        yield num_m.group(1), _norm(_strip(m.group(1))), _strip(body_html)
+
+
+def parse(html):
+    """Chunks for every in-scope section of both Acts. Pure, for tests.
+
+    Justice Laws could re-shape this page the way CBSA re-shaped its D-memo
+    index, in which case the MarginalNote/Section markup stops matching and
+    this returns []; the write guard in build() is what keeps that from
+    emptying the corpus.
+    """
+    # The page is both Acts in sequence; split at the 1982 Act's title so
+    # section numbers (which restart) land in the right instrument.
+    cut = html.find("CONSTITUTION ACT, 1982")
+    if cut < 0:
+        raise ValueError("could not locate the Constitution Act, 1982 heading")
+    parts = [
+        ("CONST-1867", "Constitution Act, 1867", "Constitution Act, 1867",
+         html[:cut], _WANT_1867),
+        ("CONST-1982", "Charter", "Constitution Act, 1982 (incl. the Canadian "
+         "Charter of Rights and Freedoms)", html[cut:], _WANT_1982),
+    ]
+    chunks = []
+    for code, short, name, segment, want in parts:
+        for num, note, text in _sections(segment):
+            if num not in want or not text:
+                continue
+            label = f"s. {num}"
+            chunks.append({
+                "id": f"{code}-s{num}",
+                "doc_type": "legislation",
+                "act_code": code,
+                "act_short": short,
+                "act_name": name,
+                "section": num,
+                "marginal_note": note,
+                "part": "",
+                "division": "",
+                "heading": "",
+                "text": text,
+                "history": "",
+                "last_amended": "",
+                "in_force": "",
+                "status": "in force",
+                "current_to": "",
+                "citation": f"{short}, {label}",
+                "source_url": URL,
+            })
+    return chunks
+
+
+def build(allow_shrink=False):
+    html = fetch_cached(URL, RAW / "FullText.html").decode("utf-8", "replace")
+    chunks = parse(html)
+    # indent=2 is what charter.json already carries -- changing it would
+    # rewrite every line of the file for nothing.
+    if not write_corpus(OUT_FILE, chunks, indent=2, allow_shrink=allow_shrink):
+        return False
+    print(f"charter: {len(chunks)} sections -> {OUT_FILE.name}")
+    return True
+
+
+def main():
+    sys.exit(0 if build(allow_shrink="--allow-shrink" in sys.argv) else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/citator.py

@@ -0,0 +1,175 @@
+"""CanLII case citator -- live lookup of a Canadian case's citation graph.
+
+Uses the CanLII API (key in canlii_key.txt). The API has no name/topic search,
+so a case is identified by its full canlii.org URL. Responses are cached on disk
+and calls are throttled, because the API rate-limits aggressively.
+"""
+import json
+import os
+import re
+import sys
+import time
+import urllib.error
+import urllib.request
+
+from .config import ROOT, DATA_DIR
+
+API = "https://api.canlii.org/v1"
+KEY_FILE = ROOT / "canlii_key.txt"
+_PLACEHOLDER = "PASTE-YOUR-CANLII-API-KEY-ON-THIS-LINE"
+_DBMAP_FILE = DATA_DIR / "citator_dbmap.json"
+_CACHE_FILE = DATA_DIR / "citator_cache.json"
+_THROTTLE = 3.0     # seconds between CanLII API calls (the API rate-limits hard)
+_MAX_LIST = 20      # items shown per citator list (lists can run to thousands)
+_CASE_URL = re.compile(r"canlii\.org/[a-z]{2}/[a-z]+/([a-z0-9-]+)/doc/\d+/([a-z0-9-]+)")
+_DB_URL = re.compile(r"canlii\.org/[a-z]{2}/[a-z]+/([a-z0-9-]+)/?$")
+
+# A neutral citation, e.g. "2019 SCC 65", and the CanLII URL segment per
+# court/tribunal. The tribunal entries cover the boards whose decisions the
+# corpus itself carries, so the natural loop (search result -> canlex_case)
+# works for every citation a result can display.
+_NEUTRAL = re.compile(
+    r"\b(\d{4})\s+(SCC|FCA|FC|FPSLREB|CIRB|ONCA|ABCA)\s+(\d+)\b",
+    re.IGNORECASE)
+_CANLII_SEG = {"scc": "scc", "fca": "fca", "fc": "fct",
+               "fpslreb": "pslreb", "cirb": "cirb",
+               "onca": "onca", "abca": "abca"}
+_PROVINCE_SEG = {"onca": "on", "abca": "ab"}  # others are federal ('ca')
+
+
+def canlii_url_from_citation(text):
+    """Build a canlii.org case URL from a neutral citation, or '' if none found.
+
+    Works for the Supreme Court, Federal Court of Appeal, Federal Court, the
+    FPSLREB and CIRB labour boards, and the Ontario and Alberta Courts of
+    Appeal -- e.g. "2019 SCC 65" -> .../en/ca/scc/doc/2019/2019scc65/...
+    """
+    m = _NEUTRAL.search(text)
+    if not m:
+        return ""
+    year, court, num = m.group(1), m.group(2).lower(), m.group(3)
+    doc = f"{year}{court}{num}"
+    juris = _PROVINCE_SEG.get(court, "ca")
+    return (f"https://www.canlii.org/en/{juris}/{_CANLII_SEG[court]}/doc/"
+            f"{year}/{doc}/{doc}.html")
+
+
+def api_key():
+    """Return the configured CanLII API key, or '' if not set.
+
+    The CANLII_API_KEY environment variable is checked first -- the remote
+    deployment injects the key as a secret rather than shipping the file.
+    A local run falls back to canlii_key.txt.
+    """
+    env_key = os.environ.get("CANLII_API_KEY", "").strip()
+    if env_key and env_key != _PLACEHOLDER:
+        return env_key
+    if not KEY_FILE.exists():
+        return ""
+    key = KEY_FILE.read_text(encoding="utf-8").strip()
+    return "" if not key or key == _PLACEHOLDER else key
+
+
+def _load_json(path):
+    try:
+        return json.loads(path.read_text(encoding="utf-8"))
+    except Exception:
+        return {}
+
+
+class Citator:
+    """Live CanLII citator with on-disk caching and rate-limit throttling."""
+
+    def __init__(self):
+        self.key = api_key()
+        if not self.key:
+            raise RuntimeError(f"No CanLII API key -- put your key in {KEY_FILE}.")
+        self._dbmap = _load_json(_DBMAP_FILE)   # URL court segment -> databaseId
+        self._cache = _load_json(_CACHE_FILE)   # case URL -> report
+
+    def _get(self, path):
+        """Throttled GET against the CanLII API, retrying on HTTP 429."""
+        for attempt in range(3):
+            time.sleep(_THROTTLE if attempt == 0 else 15.0)
+            sep = "&" if "?" in path else "?"
+            url = f"{API}/{path}{sep}api_key={self.key}"
+            try:
+                with urllib.request.urlopen(url, timeout=45) as resp:
+                    return json.loads(resp.read().decode("utf-8"))
+            except urllib.error.HTTPError as exc:
+                if exc.code == 429 and attempt < 2:
+                    continue
+                raise
+        raise RuntimeError("CanLII API rate limit reached; retry shortly.")
+
+    def _ensure_dbmap(self):
+        if self._dbmap:
+            return
+        data = self._get("caseBrowse/en/")
+        dbmap = {}
+        for db in data.get("caseDatabases", []):
+            match = _DB_URL.search(db.get("url") or "")
+            if match and db.get("databaseId"):
+                dbmap[match.group(1)] = db["databaseId"]
+        self._dbmap = dbmap
+        DATA_DIR.mkdir(parents=True, exist_ok=True)
+        _DBMAP_FILE.write_text(json.dumps(dbmap), encoding="utf-8")
+
+    def _citator(self, db, case_id, kind):
+        data = self._get(f"caseCitator/en/{db}/{case_id}/{kind}")
+        items = data.get(kind, [])
+        return {"total": len(items), "items": items[:_MAX_LIST]}
+
+    def case_report(self, case_url):
+        """Return a citation-graph report for a case.
+
+        Accepts a full canlii.org case URL, or a neutral citation (e.g.
+        "2019 SCC 65") for a Supreme Court / Federal Court of Appeal / Federal
+        Court decision.
+        """
+        if not _CASE_URL.search(case_url):
+            case_url = canlii_url_from_citation(case_url) or case_url
+        if case_url in self._cache:
+            return self._cache[case_url]
+        match = _CASE_URL.search(case_url)
+        if not match:
+            return {"error": "Provide a full canlii.org case URL, or a neutral "
+                    "citation such as '2019 SCC 65' (Supreme Court, Federal "
+                    "Court of Appeal, or Federal Court)."}
+        self._ensure_dbmap()
+        segment, case_id = match.group(1), match.group(2)
+        db = self._dbmap.get(segment)
+        if not db:
+            return {"error": f"Unrecognized CanLII court segment '{segment}'."}
+        report = {
+            "meta": self._get(f"caseBrowse/en/{db}/{case_id}/"),
+            "citingCases": self._citator(db, case_id, "citingCases"),
+            "citedCases": self._citator(db, case_id, "citedCases"),
+            "citedLegislations": self._citator(db, case_id, "citedLegislations"),
+        }
+        self._cache[case_url] = report
+        DATA_DIR.mkdir(parents=True, exist_ok=True)
+        _CACHE_FILE.write_text(json.dumps(self._cache, ensure_ascii=False),
+                               encoding="utf-8")
+        return report
+
+
+def main():
+    if len(sys.argv) < 2:
+        print('usage: python -m canlex.citator "<canlii-case-url>"')
+        return
+    report = Citator().case_report(sys.argv[1])
+    if "error" in report:
+        print("ERROR:", report["error"])
+        return
+    meta = report["meta"]
+    print(f"{meta.get('title')} -- {meta.get('citation')}  ({meta.get('decisionDate')})")
+    print(f"  cited by:  {report['citingCases']['total']}")
+    print(f"  cites:     {report['citedCases']['total']}")
+    print(f"  legislation cited: {report['citedLegislations']['total']}")
+    for item in report["citedLegislations"]["items"][:6]:
+        print(f"    - {item.get('title')} ({item.get('citation')})")
+
+
+if __name__ == "__main__":
+    main()

canlex/ciu.py

@@ -0,0 +1,239 @@
+"""Ingest CIU and PSAC union documents (doc_type='union').
+
+The Customs and Immigration Union is the PSAC component representing the FB
+bargaining unit; its governance instruments and published reference material
+are the union-side counterpart to the FB collective agreement already in the
+corpus. Ingested here, all as public documents from ciu-sdi.ca / psacunion.ca:
+
+- the CIU National By-Laws (EN + FR), CIU Regulations, Branch Rules and the
+  Local Modifications to Branch Rules -- the union's internal law;
+- the PSAC Constitution and Regulations (the parent instrument the by-laws
+  operate under: dues, discipline and appeals bottom out there);
+- the Branch Elections Kit and the members' guides (grievance handling,
+  employment equity, workers in transition);
+- every site-specific Variable Shift Schedule Agreement (VSSA) the union
+  publishes -- the locally negotiated schedule grids for ports of entry.
+
+None of this is employer policy or part of the collective agreement itself;
+_format_section banners doc_type='union' accordingly.
+
+    python -m canlex.ciu
+"""
+import json
+import re
+import urllib.request
+
+from ._common import fetch_cached, norm_ws as _norm, split_lines, uniquify_ids
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "ciu"
+VSSA_PAGE = "https://ciu-sdi.ca/en/vssa/"
+_DOCS_BASE = "https://www.ciu-sdi.ca/wp-content/uploads/documents/"
+
+# (code, act_short, act_name, url, lang)
+DOCS = [
+    ("ciu-bylaws", "CIU By-Laws",
+     "CIU National By-Laws (adopted by CIU National Conventions)",
+     _DOCS_BASE + "bylaws_en.pdf", "en"),
+    ("ciu-bylaws", "CIU By-Laws",
+     "Règlements nationaux du SDI (adoptés par les congrès)",
+     _DOCS_BASE + "bylaws_fr.pdf", "fr"),
+    ("ciu-regs", "CIU Regulations",
+     "CIU Regulations (adopted by the CIU National Board of Directors)",
+     _DOCS_BASE + "ciu-regs-en.pdf", "en"),
+    ("ciu-branch-rules", "CIU Branch Rules",
+     "CIU Branch Rules (adopted by the CIU National Board of Directors)",
+     _DOCS_BASE + "branch-rules-en.pdf", "en"),
+    ("ciu-local-mods", "CIU Local Modifications",
+     "Local Modifications to CIU Branch Rules",
+     _DOCS_BASE + "local_mod_branch_rules_en.pdf", "en"),
+    ("ciu-elections-kit", "CIU Elections Kit",
+     "CIU Branch Elections Kit",
+     "https://www.ciu-sdi.ca/wp-content/uploads/elections/2023/EN/"
+     "elections-kit.pdf", "en"),
+    ("psac-constitution", "PSAC Constitution",
+     "PSAC Constitution and Regulations (November 2024 edition)",
+     "https://psacunion.ca/sites/psac/files/"
+     "psac.constitution.november_2024_edition.eng_v002.pdf", "en"),
+    ("ciu-guide-grievance", "CIU Grievance Guide",
+     "CIU Guide to Grievance Handling",
+     "https://ciu-sdi.ca/wp-content/uploads/2012/09/"
+     "guide_to_grievance_handling-e.pdf", "en"),
+    ("ciu-guide-ee", "CIU EE Toolkit",
+     "CIU Employment Equity Toolkit",
+     "https://ciu-sdi.ca/wp-content/uploads/2012/09/"
+     "EMPLOYMENT-EQUITY-Tool-Kit-e.pdf", "en"),
+    ("ciu-guide-transition", "CIU Transition Guide",
+     "Workers in Transition: a practical guide about gender transition",
+     "https://ciu-sdi.ca/wp-content/uploads/2012/09/"
+     "workers_in_transition_guide_-_en_-_web.pdf", "en"),
+]
+
+# Governance headings: "BY-LAW 5", "REGULATION 12", "RULE 3", "ARTICLE 7",
+# "SECTION 25", "PART B" -- at line start, optionally followed by a title.
+_HEADING = re.compile(
+    r"^(BY-?LAW|R[EÈ]GLEMENT|REGULATION|RULE|ARTICLE|SECTION|PART|ANNEX(?:E)?"
+    r"|APPENDIX)\s+([0-9]+[0-9A-Za-z.\-]*|[A-Z])\b\s*[-–—:]?\s*(.{0,80})",
+    re.IGNORECASE)
+_DATE = re.compile(r"\b(20\d{2}|19\d{2})\b")
+
+
+def _pdf_pages(path):
+    from pypdf import PdfReader
+    pages = []
+    for page in PdfReader(str(path)).pages:
+        try:
+            pages.append(page.extract_text() or "")
+        except Exception:
+            pages.append("")
+    return pages
+
+
+def _currency(pages):
+    """Newest adoption/amendment year on the cover pages."""
+    years = _DATE.findall("\n".join(pages[:3]))
+    return max(years) if years else ""
+
+
+def _doc_chunks(code, short, name, url, lang, pages):
+    text = "\n".join(pages)
+    lines = text.split("\n")
+    currency = _currency(pages)
+    marks = []
+    for i, l in enumerate(lines):
+        m = _HEADING.match(l.strip())
+        if m:
+            marks.append((i, f"{m.group(1).title()} {m.group(2)}",
+                          _norm(m.group(3)) or ""))
+    segments = []
+    if len(marks) >= 6:
+        for j, (i, label, title) in enumerate(marks):
+            end = marks[j + 1][0] if j + 1 < len(marks) else len(lines)
+            body = "\n".join(p for p in (_norm(x) for x in lines[i:end]) if p)
+            if len(body) > 120:
+                segments.append((label, title, body))
+    if not segments:
+        for k, piece in enumerate(split_lines(text, 1800), start=1):
+            body = "\n".join(p for p in (_norm(x) for x in piece.split("\n")) if p)
+            if len(body) > 120:
+                segments.append((f"part {k}", "", body))
+    chunks = []
+    fr = "-fr" if lang == "fr" else ""
+    for n, (label, title, body) in enumerate(segments, start=1):
+        pieces = split_lines(body, 1800) if len(body) > 2400 else [body]
+        total = len(pieces)
+        for k, piece in enumerate(pieces, start=1):
+            suffix = "" if total == 1 else f"-p{k}"
+            note = "" if total == 1 else f" (part {k} of {total})"
+            chunk = {
+                "id": f"{code}{fr}-{n}{suffix}",
+                "doc_type": "union",
+                "act_code": code.upper(),
+                "act_short": short,
+                "act_name": name,
+                "section": label,
+                "marginal_note": title,
+                "part": short,
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": currency,
+                "in_force": "",
+                "status": "in force",
+                "current_to": currency,
+                "citation": f"{short}, {label}"
+                            + (f" ({title})" if title else "") + note,
+                "source_url": url,
+            }
+            if lang == "fr":
+                chunk["lang"] = "fr"
+            chunks.append(chunk)
+    return chunks
+
+
+def _vssa_links():
+    req = urllib.request.Request(VSSA_PAGE, headers={"User-Agent": "Mozilla/5.0"})
+    with urllib.request.urlopen(req, timeout=60) as r:
+        html = r.read().decode("utf-8", "replace")
+    links = sorted(set(re.findall(
+        r'href="(https?://[^"]*?/uploads/vssa/([^"/]+)\.pdf)"', html)))
+    return links
+
+
+def _vssa_chunks(url, stem, pages):
+    site = _norm(stem.replace("-", " "))
+    text = "\n".join(p for p in (_norm(x) for x in "\n".join(pages).split("\n")) if p)
+    if len(text) < 60:
+        return []
+    out = []
+    pieces = split_lines(text, 1800)
+    total = len(pieces)
+    for k, piece in enumerate(pieces, start=1):
+        suffix = "" if total == 1 else f"-p{k}"
+        note = "" if total == 1 else f" (part {k} of {total})"
+        out.append({
+            "id": f"vssa-{stem.lower()}{suffix}",
+            "doc_type": "union",
+            "act_code": "VSSA",
+            "act_short": "VSSA",
+            "act_name": "Variable Shift Schedule Agreements (by work site)",
+            "section": site,
+            "marginal_note": f"VSSA — {site}",
+            "part": "Variable Shift Schedule Agreements",
+            "division": "",
+            "heading": "",
+            "text": piece,
+            "history": "",
+            "last_amended": "",
+            "in_force": "",
+            "status": "in force",
+            "current_to": "",
+            "citation": f"VSSA — {site}{note}",
+            "source_url": url,
+        })
+    return out
+
+
+def build():
+    RAW.mkdir(parents=True, exist_ok=True)
+    all_chunks = []
+    for code, short, name, url, lang in DOCS:
+        fr = "-fr" if lang == "fr" else ""
+        dest = RAW / f"{code}{fr}.pdf"
+        try:
+            fetch_cached(url, dest, pause=0.8)
+            chunks = _doc_chunks(code, short, name, url, lang,
+                                 _pdf_pages(dest))
+        except Exception as exc:
+            print(f"  !! {short} ({lang}): {type(exc).__name__}: {exc}")
+            continue
+        print(f"  {short:24} ({lang}): {len(chunks)} chunks")
+        all_chunks.extend(chunks)
+    try:
+        links = _vssa_links()
+    except Exception as exc:
+        links = []
+        print(f"  !! VSSA index: {type(exc).__name__}: {exc}")
+    n_vssa = 0
+    for url, stem in links:
+        dest = RAW / f"vssa-{stem.lower()}.pdf"
+        try:
+            fetch_cached(url, dest, pause=0.6)
+            chunks = _vssa_chunks(url, stem, _pdf_pages(dest))
+        except Exception as exc:
+            print(f"  !! VSSA {stem}: {type(exc).__name__}: {exc}")
+            continue
+        all_chunks.extend(chunks)
+        n_vssa += 1
+    print(f"  VSSA sites: {n_vssa}")
+    uniquify_ids(all_chunks)
+    out = PROCESSED_DIR / "ciu.json"
+    out.write_text(json.dumps(all_chunks, ensure_ascii=False, indent=2),
+                   encoding="utf-8")
+    print(f"ciu: {len(all_chunks)} chunks -> {out.name}")
+    return all_chunks
+
+
+if __name__ == "__main__":
+    build()

canlex/commentary.py

@@ -0,0 +1,272 @@
+"""Ingest CanLex's curated commentary datasets into searchable chunks.
+
+Commentary is the one doc_type CanLex AUTHORS rather than mirrors: structured
+legal-analysis datasets (currently the US-dispositions helper -- whether each
+kind of US state criminal disposition is a "conviction" for IRPA s. 36
+purposes). Every chunk is banner-labelled as commentary, every proposition
+carries its authorities, and entries with no authority say so explicitly and
+flag their reasoning as interpretation. The source of truth is
+data/curated/us_dispositions.json, which is reviewed by the user before it
+ships; this module just renders it into corpus chunks.
+
+    py -m canlex.commentary [--allow-shrink]
+"""
+import json
+import sys
+
+from ._common import write_corpus
+from .config import DATA_DIR, PROCESSED_DIR
+
+CURATED = DATA_DIR / "curated" / "us_dispositions.json"
+EQUIV = DATA_DIR / "curated" / "us_equivalency.json"
+OUT = PROCESSED_DIR / "commentary.json"
+
+ACT_CODE = "US-DISP"
+ACT_SHORT = "US Dispositions Helper"
+ACT_NAME = ("US criminal dispositions and the IRPA 'conviction' concept "
+            "(curated CanLex commentary)")
+
+BANNER = ("CURATED ANALYSIS -- commentary compiled for CanLex, not a source "
+          "of law. Verify against the cited authorities before relying on it.")
+
+_STATUS_LABEL = {
+    "settled": "Settled by binding authority",
+    "judicially-considered": "Judicially considered (persuasive authority)",
+    "guidance-only": "IRCC guidance only -- no judicial authority located",
+    "no-authority": "NO AUTHORITY LOCATED -- reasoned interpretation only",
+}
+
+# Verdicts assess the COMPLETED disposition; the methodology chunk explains
+# the rule each one derives from. 'depends' is the legacy spelling of
+# fact-specific, kept for backwards compatibility.
+_VERDICT_LABEL = {
+    "yes": "YES", "likely-yes": "LIKELY YES", "likely-no": "LIKELY NO",
+    "no": "NO", "fact-specific": "FACT-SPECIFIC",
+    "depends": "FACT-SPECIFIC",
+}
+
+
+def _vlabel(v):
+    return _VERDICT_LABEL.get(v, str(v).upper())
+
+
+def _entry_text(e, max_states=None):
+    """Render one disposition entry as a readable, retrieval-friendly block.
+
+    With the 51-jurisdiction survey an entry can carry dozens of state rows;
+    max_states caps how many render inline (the corpus keeps full per-state
+    detail in separate per-state chunks so retrieval still reaches every
+    row). None renders everything -- the tool uses that for direct lookups."""
+    lines = [BANNER, ""]
+    lines.append(f"Disposition: {'; '.join(e['names'])}")
+    lines.append(f"Is the COMPLETED disposition a conviction for IRPA s. 36: "
+                 f"{_vlabel(e['is_conviction'])}")
+    lines.append(f"Authority status: {_STATUS_LABEL[e['status']]}")
+    if e.get("bottom_line"):
+        lines.append("")
+        lines.append(f"BOTTOM LINE: {e['bottom_line']}")
+    lines.append("")
+    lines.append(e["analysis"])
+    if e.get("state_variations"):
+        named = [v for v in e["state_variations"]
+                 if v["state"].lower() != "general"]
+        shown = named if max_states is None else named[:max_states]
+        lines.append("")
+        if max_states is not None and len(named) > len(shown):
+            from collections import Counter
+            counts = Counter(v.get("is_conviction", "depends") for v in named)
+            lines.append(f"State-by-state coverage: {len(named)} jurisdictions "
+                         f"({', '.join(f'{k}: {n}' for k, n in counts.most_common())}) "
+                         f"-- full per-state detail in the per-state entries.")
+        else:
+            lines.append("State variations:")
+        for v in shown:
+            flag = (f" (conviction: {_vlabel(v['is_conviction'])})"
+                    if v.get("is_conviction") else "")
+            lines.append(f"- {v['state']}{flag}: {v['note']}")
+        for v in e["state_variations"]:
+            if v["state"].lower() == "general":
+                lines.append(f"- General: {v['note']}")
+    if e.get("authorities"):
+        lines.append("")
+        lines.append("Authorities:")
+        for a in e["authorities"]:
+            pin = f", {a['pin']}" if a.get("pin") else ""
+            lines.append(f"- {a['cite']} ({a['court']}{pin}): {a['holding']}")
+    if e.get("guidance"):
+        lines.append("")
+        lines.append("IRCC guidance (cited by reference; not reproduced here):")
+        for g in e["guidance"]:
+            lines.append(f"- {g['ref']}: {g['note']}")
+    if e.get("interpretation"):
+        lines.append("")
+        lines.append("INTERPRETATION (no direct authority -- this is CanLex's "
+                     "reasoned view from the governing principles; treat it as "
+                     "a starting point, not an answer): "
+                     + e["interpretation"])
+    return "\n".join(lines)
+
+
+def build(allow_shrink=False):
+    data = json.loads(CURATED.read_text(encoding="utf-8"))
+    chunks = []
+    for e in data.get("methodology", []):
+        chunks.append({
+            "id": f"commentary-method-{e['id']}",
+            "doc_type": "commentary",
+            "act_code": ACT_CODE,
+            "act_short": ACT_SHORT,
+            "act_name": ACT_NAME,
+            "section": e["id"],
+            "marginal_note": e["title"],
+            "part": "Methodology",
+            "division": "",
+            "heading": e["title"],
+            "text": BANNER + "\n\n" + e["text"],
+            "history": "",
+            "last_amended": "",
+            "current_to": data.get("reviewed", ""),
+            "citation": f"{ACT_SHORT} — {e['title']}",
+            "source_url": "",
+        })
+    for e in data.get("dispositions", []):
+        chunks.append({
+            "id": f"commentary-disp-{e['id']}",
+            "doc_type": "commentary",
+            "act_code": ACT_CODE,
+            "act_short": ACT_SHORT,
+            "act_name": ACT_NAME,
+            "section": e["id"],
+            "marginal_note": e["names"][0],
+            "part": "US dispositions",
+            "division": "",
+            "heading": (f"Is a US {e['names'][0]} a conviction for IRPA "
+                        f"s. 36? ({_vlabel(e['is_conviction'])})"),
+            "text": _entry_text(e, max_states=6),
+            "history": "",
+            "last_amended": "",
+            "current_to": data.get("reviewed", ""),
+            "citation": f"{ACT_SHORT} — {e['names'][0]}",
+            "source_url": "",
+        })
+
+    # One chunk per jurisdiction: every disposition row for that state, so a
+    # query naming a state ("Georgia first offender act", "Missouri SIS")
+    # retrieves that state's page directly.
+    by_state = {}
+    for e in data.get("dispositions", []):
+        for v in e.get("state_variations", []):
+            st = v["state"]
+            if st.lower() == "general":
+                continue
+            flag = _vlabel(v.get("is_conviction") or e["is_conviction"])
+            by_state.setdefault(st, []).append(
+                f"- {e['names'][0]} (conviction: {flag}): {v['note']}")
+    for st in sorted(by_state):
+        body = (BANNER + "\n\n"
+                + f"US dispositions — {st}: whether each disposition type is a "
+                  f"conviction for IRPA s. 36, under {st} law.\n\n"
+                + "\n".join(by_state[st])
+                + "\n\nThe act branch (IRPA s. 36(1)(c)/(2)(c)) can apply even "
+                  "where a disposition is not a conviction. See the "
+                  "per-disposition entries for the governing analysis and "
+                  "authorities.")
+        slug = st.lower().replace(" ", "-")
+        chunks.append({
+            "id": f"commentary-state-{slug}",
+            "doc_type": "commentary",
+            "act_code": ACT_CODE,
+            "act_short": ACT_SHORT,
+            "act_name": ACT_NAME,
+            "section": f"state-{slug}",
+            "marginal_note": f"US dispositions — {st}",
+            "part": "US dispositions by state",
+            "division": "",
+            "heading": (f"{st}: criminal dispositions vs the IRPA "
+                        f"'conviction' concept"),
+            "text": body,
+            "history": "",
+            "last_amended": "",
+            "current_to": data.get("reviewed", ""),
+            "citation": f"{ACT_SHORT} — {st}",
+            "source_url": "",
+        })
+    # --- equivalency pairings (step 2), same banner discipline
+    n_pairings = 0
+    if EQUIV.exists():
+        eq = json.loads(EQUIV.read_text(encoding="utf-8"))
+        for m in eq.get("methodology", []):
+            chunks.append({
+                "id": f"commentary-method-{m['id']}",
+                "doc_type": "commentary",
+                "act_code": ACT_CODE,
+                "act_short": ACT_SHORT,
+                "act_name": ACT_NAME,
+                "section": m["id"],
+                "marginal_note": m["title"],
+                "part": "Methodology",
+                "division": "",
+                "heading": m["title"],
+                "text": BANNER + "\n\n" + m["text"],
+                "history": "",
+                "last_amended": "",
+                "current_to": eq.get("reviewed", ""),
+                "citation": f"{ACT_SHORT} — {m['title']}",
+                "source_url": "",
+            })
+        for p in eq.get("pairings", []):
+            n_pairings += 1
+            lines = [BANNER, "",
+                     f"US offence: {'; '.join(p['us_terms'][:5])}",
+                     f"Canadian equivalent: {p['canadian_offence']}",
+                     f"Maximum penalty (verified): {p['penalty']}",
+                     f"Inadmissibility branch: {p['branch']}", "",
+                     p["analysis"]]
+            if p.get("caveats"):
+                lines.append("")
+                lines.append("Caveats:")
+                lines += [f"- {c}" for c in p["caveats"]]
+            if p.get("authorities"):
+                lines.append("")
+                lines.append("Authorities:")
+                lines += [f"- {a['cite']} ({a['court']}): {a['holding']}"
+                          for a in p["authorities"]]
+            chunks.append({
+                "id": f"commentary-equiv-{p['id']}",
+                "doc_type": "commentary",
+                "act_code": ACT_CODE,
+                "act_short": ACT_SHORT,
+                "act_name": ACT_NAME,
+                "section": f"equiv-{p['id']}",
+                "marginal_note": f"Equivalency: {p['us_terms'][0]}",
+                "part": "US offence equivalency",
+                "division": "",
+                "heading": (f"What does a US {p['us_terms'][0]} conviction "
+                            f"equate to in Canada?"),
+                "text": "\n".join(lines),
+                "history": "",
+                "last_amended": "",
+                "current_to": eq.get("reviewed", ""),
+                "citation": f"{ACT_SHORT} — equivalency: {p['us_terms'][0]}",
+                "source_url": "",
+            })
+
+    # A full re-render of the curated files, so a truncated or half-edited
+    # us_dispositions.json (or an equivalency file that moved) collapses the
+    # chunk count and the write would replace good analysis with the remnant.
+    # indent=1 is the stored file's format -- changing it rewrites every line.
+    if not write_corpus(OUT, chunks, indent=1, allow_shrink=allow_shrink):
+        return False
+    print(f"{len(chunks)} commentary chunks "
+          f"({len(data.get('dispositions', []))} dispositions, "
+          f"{len(by_state)} state pages, {n_pairings} equivalency pairings, "
+          f"{len(data.get('methodology', []))}+ methodology) -> {OUT}")
+    return True
+
+
+def main():
+    sys.exit(0 if build(allow_shrink="--allow-shrink" in sys.argv) else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/config.py

@@ -0,0 +1,336 @@
+"""Paths and legislation sources for CanLex."""
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+DATA_DIR = ROOT / "data"
+RAW_DIR = DATA_DIR / "raw"
+PROCESSED_DIR = DATA_DIR / "processed"
+
+# Justice Laws Website XML sources. Each key is the Justice Laws consolidated
+# identifier, used as the XML filename and the local raw/processed filename.
+# To add an instrument, add an entry with a verified xml_url.
+#
+# The Constitution Acts, 1867 to 1982 are intentionally absent: Justice Laws
+# publishes them only as HTML/PDF, with no XML version available.
+SOURCES = {
+    # --- Immigration ---
+    "I-2.5": {
+        "short": "IRPA", "name": "Immigration and Refugee Protection Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/I-2.5.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/i-2.5/"},
+    "SOR-2002-227": {
+        "short": "IRPR", "name": "Immigration and Refugee Protection Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2002-227.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2002-227/"},
+    "C-29": {
+        "short": "Citizenship Act", "name": "Citizenship Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-29.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-29/"},
+    # IRB tribunal procedural rules. (SOR-2002-230 was repealed because the
+    # Immigration Appeal Division Rules, 2022 -- SOR-2022-277, below --
+    # replaced it in January 2023.)
+    "SOR-2022-277": {
+        "short": "IAD Rules", "name": "Immigration Appeal Division Rules, 2022",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2022-277.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2022-277/"},
+    "SOR-2002-229": {
+        "short": "Immigration Division Rules", "name": "Immigration Division Rules",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2002-229.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2002-229/"},
+    "SOR-2012-256": {
+        "short": "RPD Rules", "name": "Refugee Protection Division Rules",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2012-256.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2012-256/"},
+    "SOR-2012-257": {
+        "short": "RAD Rules", "name": "Refugee Appeal Division Rules",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2012-257.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2012-257/"},
+
+    # --- Customs & trade ---
+    "C-52.6": {
+        "short": "Customs Act", "name": "Customs Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-52.6.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-52.6/"},
+    # skip_schedules: the Customs Tariff's schedule is the entire tariff
+    # nomenclature -- tariff_schedule.py serves chapters 98/99 from the CBSA
+    # edition instead, so the XML schedule would only duplicate it at scale.
+    "C-54.011": {
+        "short": "Customs Tariff", "name": "Customs Tariff", "kind": "act",
+        "skip_schedules": True,
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-54.011.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-54.011/"},
+    # Traveller/trade instruments under the Customs Act and EIPA. The two
+    # control lists are almost entirely Schedules, so they depend on the
+    # schedule parser; Presentation of Persons is the legal basis for NEXUS
+    # and every other alternative-presentation program.
+    "E-14.1": {
+        "short": "Excise Act, 2001", "name": "Excise Act, 2001", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/E-14.1.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/e-14.1/"},
+    "CRC-604": {
+        "short": "Import Control List", "name": "Import Control List",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C.R.C.,_c._604.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/C.R.C.,_c._604/"},
+    "SOR-89-202": {
+        "short": "Export Control List", "name": "Export Control List",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-89-202.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-89-202/"},
+    "SOR-2003-323": {
+        "short": "Presentation of Persons Regs",
+        "name": "Presentation of Persons (2003) Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2003-323.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2003-323/"},
+    "SOR-86-1072": {
+        "short": "Duty Free Shop Regs", "name": "Duty Free Shop Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-86-1072.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-86-1072/"},
+    "SOR-86-1065": {
+        "short": "Sufferance Warehouses Regs",
+        "name": "Customs Sufferance Warehouses Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-86-1065.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-86-1065/"},
+    "S-8.35": {
+        "short": "Luxury Tax Act", "name": "Select Luxury Items Tax Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/S-8.35.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/s-8.35/"},
+    "C-1.4": {
+        "short": "CBSA Act", "name": "Canada Border Services Agency Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-1.4.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"},
+    "S-15": {
+        "short": "SIMA", "name": "Special Import Measures Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/S-15.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/s-15/"},
+    "E-19": {
+        "short": "EIPA", "name": "Export and Import Permits Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/E-19.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/e-19/"},
+    "P-19.32": {
+        "short": "Preclearance Act", "name": "Preclearance Act, 2016", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-19.32.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-19.32/"},
+    # Customs Act operational regulations -- the procedural detail behind the
+    # bare Act (reporting, accounting, exemptions, and the designated-provisions
+    # regulation that is the legal hook for every AMPS monetary penalty).
+    "SOR-86-873": {
+        "short": "Reporting of Imported Goods Regs",
+        "name": "Reporting of Imported Goods Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-86-873.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-86-873/"},
+    "SOR-86-1062": {
+        "short": "Accounting for Imported Goods Regs",
+        "name": "Accounting for Imported Goods and Payment of Duties Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-86-1062.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-86-1062/"},
+    "SOR-2002-336": {
+        "short": "Designated Provisions (Customs) Regs",
+        "name": "Designated Provisions (Customs) Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2002-336.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2002-336/"},
+    "SOR-96-46": {
+        "short": "Customs Bonded Warehouses Regs",
+        "name": "Customs Bonded Warehouses Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-96-46.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-96-46/"},
+    "SOR-98-58": {
+        "short": "Returning Persons Exemption Regs",
+        "name": "Returning Persons Exemption Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-98-58.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-98-58/"},
+
+    # --- Criminal & financial crime ---
+    "C-46": {
+        "short": "Criminal Code", "name": "Criminal Code", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-46.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-46/"},
+    "F-11.6": {
+        "short": "Firearms Act", "name": "Firearms Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/F-11.6.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/f-11.6/"},
+    # Made under Criminal Code s. 117.15: the model-by-model prohibited/
+    # restricted/non-restricted classifications (incl. the May 2020
+    # SOR/2020-96 prohibitions). Nearly all substance is in its Schedules --
+    # this entry is why the schedule parser exists.
+    "SOR-98-462": {
+        "short": "Firearms Classification Regs",
+        "name": ("Regulations Prescribing Certain Firearms and Other Weapons, "
+                 "Components and Parts of Weapons, Accessories, Cartridge "
+                 "Magazines, Ammunition and Projectiles as Prohibited, "
+                 "Restricted or Non-Restricted"), "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-98-462.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-98-462/"},
+    "P-24.501": {
+        "short": "PCMLTFA",
+        "name": "Proceeds of Crime (Money Laundering) and Terrorist Financing Act",
+        "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-24.501.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-24.501/"},
+    "SOR-2002-412": {
+        "short": "Cross-border Currency Reporting Regs",
+        "name": "Cross-border Currency and Monetary Instruments Reporting Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2002-412.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2002-412/"},
+    "S-6.9": {
+        "short": "SCIDA", "name": "Security of Canada Information Disclosure Act",
+        "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/S-6.9.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/s-6.9/"},
+
+    # --- Controlled substances & cannabis ---
+    "C-38.8": {
+        "short": "CDSA", "name": "Controlled Drugs and Substances Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-38.8.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-38.8/"},
+    "SOR-97-234": {
+        "short": "CDSA Police Enforcement Regs",
+        "name": "Controlled Drugs and Substances Act (Police Enforcement) Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-97-234.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-97-234/"},
+    "SOR-2000-217": {
+        "short": "Benzodiazepines Regs",
+        "name": "Benzodiazepines and Other Targeted Substances Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2000-217.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2000-217/"},
+    "C.R.C.,_c._1041": {
+        "short": "Narcotic Control Regs", "name": "Narcotic Control Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C.R.C.%2C_c._1041.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/C.R.C.%2C_c._1041/"},
+    "SOR-2012-230": {
+        "short": "New Classes of Practitioners Regs",
+        "name": "New Classes of Practitioners Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2012-230.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2012-230/"},
+    "SOR-2002-359": {
+        "short": "Precursor Control Regs", "name": "Precursor Control Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2002-359.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2002-359/"},
+    "SOR-97-229": {
+        "short": "Precursor & Substance Exemption Regs",
+        "name": "Regulations Exempting Certain Precursors and Controlled Substances "
+                "from the Application of the Controlled Drugs and Substances Act",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-97-229.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-97-229/"},
+    "C-24.5": {
+        "short": "Cannabis Act", "name": "Cannabis Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C-24.5.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/c-24.5/"},
+    "SOR-2018-144": {
+        "short": "Cannabis Regs", "name": "Cannabis Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2018-144.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2018-144/"},
+
+    # --- Food, health, agriculture ---
+    "F-27": {
+        "short": "Food and Drugs Act", "name": "Food and Drugs Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/F-27.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/f-27/"},
+    "C.R.C.,_c._870": {
+        "short": "Food and Drug Regs", "name": "Food and Drug Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C.R.C.%2C_c._870.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/C.R.C.%2C_c._870/"},
+    "H-3.3": {
+        "short": "Health of Animals Act", "name": "Health of Animals Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/H-3.3.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/h-3.3/"},
+    "C.R.C.,_c._296": {
+        "short": "Health of Animals Regs", "name": "Health of Animals Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/C.R.C.%2C_c._296.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/C.R.C.%2C_c._296/"},
+    "P-14.8": {
+        "short": "Plant Protection Act", "name": "Plant Protection Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-14.8.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-14.8/"},
+    "SOR-95-212": {
+        "short": "Plant Protection Regs", "name": "Plant Protection Regulations",
+        "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-95-212.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-95-212/"},
+    "SOR-2018-108": {
+        "short": "Safe Food for Canadians Regs",
+        "name": "Safe Food for Canadians Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-2018-108.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-2018-108/"},
+    "A-8.8": {
+        "short": "AAAMPA",
+        "name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+        "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/A-8.8.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/"},
+    "Q-1.1": {
+        "short": "Quarantine Act", "name": "Quarantine Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/Q-1.1.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/q-1.1/"},
+    # CITES enforcement at the border -- the basis for seizures of protected
+    # wildlife, ivory and exotic-species goods.
+    "W-8.5": {
+        "short": "WAPPRIITA",
+        "name": "Wild Animal and Plant Protection and Regulation of "
+                "International and Interprovincial Trade Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/W-8.5.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/w-8.5/"},
+    "SOR-96-263": {
+        "short": "WAPTR",
+        "name": "Wild Animal and Plant Trade Regulations", "kind": "regulation",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/SOR-96-263.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/regulations/SOR-96-263/"},
+
+    # --- Labour & public service ---
+    # CHRA grounds the accommodation/discrimination case law already in the
+    # corpus (Renaud, Hydro-Québec, Elk Valley); GECA governs injury-on-duty
+    # compensation interlocking with the FB agreement's IOD leave.
+    "H-6": {
+        "short": "CHRA", "name": "Canadian Human Rights Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/H-6.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/h-6/"},
+    "G-5": {
+        "short": "GECA", "name": "Government Employees Compensation Act",
+        "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/G-5.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/g-5/"},
+    "L-2": {
+        "short": "Canada Labour Code", "name": "Canada Labour Code", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/L-2.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/l-2/"},
+    "P-33.3": {
+        "short": "FPSLRA", "name": "Federal Public Sector Labour Relations Act",
+        "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-33.3.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-33.3/"},
+    "P-33.35": {
+        "short": "FPSLREB Act",
+        "name": "Federal Public Sector Labour Relations and Employment Board Act",
+        "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-33.35.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-33.35/"},
+    "P-33.01": {
+        "short": "PSEA", "name": "Public Service Employment Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-33.01.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-33.01/"},
+    "P-36": {
+        "short": "PSSA", "name": "Public Service Superannuation Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-36.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-36/"},
+
+    # --- Privacy ---
+    "P-21": {
+        "short": "Privacy Act", "name": "Privacy Act", "kind": "act",
+        "xml_url": "https://laws-lois.justice.gc.ca/eng/XML/P-21.xml",
+        "web_base": "https://laws-lois.justice.gc.ca/eng/acts/p-21/"},
+}

canlex/delegation.py

@@ -0,0 +1,506 @@
+"""Ingest instruments of delegation and designation under IRPA / IRPR.
+
+These instruments record which officer positions the Minister has delegated
+powers to, or designated for functions, under the Immigration and Refugee
+Protection Act and its Regulations. They are administrative instruments -- not
+enacted law, and not guidance -- so every chunk is tagged doc_type="delegation".
+
+Sources:
+  - the CBSA "Delegation of Authority and Designations of Officers ..."
+    instrument (HTML, cbsa-asfc.gc.ca);
+  - the IRCC "IL3 -- Instrument of Designation and Delegation" (PDF, canada.ca).
+
+    py -m canlex.delegation
+"""
+import io
+import re
+import sys
+
+from bs4 import BeautifulSoup
+from pypdf import PdfReader
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      stored_chunks, uniquify_ids, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "delegation"
+OUT = PROCESSED_DIR / "delegation.json"
+
+# The CBSA IRPA delegation has been a moving target: the November 28, 2017
+# instrument was expressly superseded by a full restatement on May 8, 2023
+# (signed by Mendicino), which has itself been amended five times since. CBSA
+# does not publish a consolidated version, so the current effective state is the
+# 2023 restatement read together with its later amendments; we ingest each one
+# as a separate "act" so a user (or the LLM) sees the base item and any
+# amendments that touch it side-by-side in retrieval.
+_CBSA_DELEG_URL = ("https://www.cbsa-asfc.gc.ca/agency-agence/actreg-loireg/"
+                   "delegation/")
+_CBSA_DELEG_NAME = ("Delegation of Authority and Designations of Officers by "
+                    "the Minister of Public Safety and Emergency Preparedness "
+                    "under the Immigration and Refugee Protection Act and the "
+                    "Immigration and Refugee Protection Regulations")
+
+SOURCES = {
+    # The 2023 restatement -- the current base instrument.
+    "cbsa-2023-05": {
+        "code": "cbsa-2023-05",
+        "kind": "html-cbsa",
+        "act_code": "CBSA-IRPA-DELEG-2023-05",
+        "act_short": "CBSA Deleg 2023-05-08",
+        "act_name": _CBSA_DELEG_NAME,
+        "url": _CBSA_DELEG_URL + "irpa-lipr-2023-05-08-eng.html",
+        "effective": "2023-05-08",
+    },
+    # Amendments to the 2023 restatement, in chronological order.
+    "cbsa-2023-09": {
+        "code": "cbsa-2023-09",
+        "kind": "html-cbsa",
+        "act_code": "CBSA-IRPA-DELEG-AMEND-2023-09-08",
+        "act_short": "CBSA Deleg Amend 2023-09-08",
+        "act_name": "Amendment to the " + _CBSA_DELEG_NAME,
+        "url": _CBSA_DELEG_URL + "irpa-lipr-2023-09-08-eng.html",
+        "effective": "2023-09-08",
+    },
+    "cbsa-2023-11": {
+        "code": "cbsa-2023-11",
+        "kind": "html-cbsa",
+        "act_code": "CBSA-IRPA-DELEG-AMEND-2023-11-17",
+        "act_short": "CBSA Deleg Amend 2023-11-17",
+        "act_name": "Amendment to the " + _CBSA_DELEG_NAME,
+        "url": _CBSA_DELEG_URL + "irpa-lipr-2023-11-17-eng.html",
+        "effective": "2023-11-17",
+    },
+    "cbsa-2024-03-05": {
+        "code": "cbsa-2024-03-05",
+        "kind": "html-cbsa",
+        "act_code": "CBSA-IRPA-DELEG-AMEND-2024-03-05",
+        "act_short": "CBSA Deleg Amend 2024-03-05",
+        "act_name": "Amendment to the " + _CBSA_DELEG_NAME,
+        "url": _CBSA_DELEG_URL + "irpa-lipr-2024-03-05-eng.html",
+        "effective": "2024-03-05",
+    },
+    "cbsa-2024-03-15": {
+        "code": "cbsa-2024-03-15",
+        "kind": "html-cbsa",
+        "act_code": "CBSA-IRPA-DELEG-AMEND-2024-03-15",
+        "act_short": "CBSA Deleg Amend 2024-03-15",
+        "act_name": "Amendment to the " + _CBSA_DELEG_NAME,
+        "url": _CBSA_DELEG_URL + "irpa-lipr-2024-03-15-eng.html",
+        "effective": "2024-03-15",
+    },
+    "cbsa-2025-07": {
+        "code": "cbsa-2025-07",
+        "kind": "html-cbsa",
+        "act_code": "CBSA-IRPA-DELEG-AMEND-2025-07-10",
+        "act_short": "CBSA Deleg Amend 2025-07-10",
+        "act_name": "Amendment to the " + _CBSA_DELEG_NAME,
+        "url": _CBSA_DELEG_URL + "irpa-lipr-2025-07-10-eng.html",
+        "effective": "2025-07-10",
+    },
+    # Separate authority: a peace-officer designation under IRPA s. 138(1).
+    # Narrative prose, not a Schedule table -- needs its own parser.
+    "cbsa-peaceofficer": {
+        "code": "cbsa-peaceofficer",
+        "kind": "html-cbsa-narrative",
+        "act_code": "CBSA-IRPA-PEACEOFF-2022-08",
+        "act_short": "CBSA Peace Officer Auth 2022-08-18",
+        "act_name": ("Authorization to have the Authority and Powers of a "
+                     "Peace Officer under the Immigration and Refugee "
+                     "Protection Act (subsection 138(1))"),
+        "url": _CBSA_DELEG_URL + "desig/po-ag_2022-08-eng.html",
+        "effective": "2022-08-18",
+    },
+    "ircc": {
+        "code": "ircc",
+        "kind": "pdf-ircc",
+        "act_code": "IRCC-IL3-DELEG",
+        "act_short": "IRCC IL3",
+        "act_name": ("IL3 — Instrument of Designation and Delegation, "
+                     "Immigration and Refugee Protection Act and Regulations"),
+        "url": ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
+                "resources/manuals/il/il3-eng.pdf"),
+    },
+}
+
+
+def _normalize_refs(text):
+    """Expand the instruments' provision shorthand so the section numbers are
+    searchable as tokens: 'A55(1)' -> 'IRPA 55(1)', 'R39' -> 'IRPR 39'. Both
+    instruments write 'A' for the Act and 'R' for the Regulations."""
+    text = re.sub(r"\bA(?=\d)", "IRPA ", text)
+    text = re.sub(r"\bR(?=\d)", "IRPR ", text)
+    return _norm(text)
+
+
+def _fetch(url, dest, powershell=False):
+    """Fetch a page or file, caching the raw bytes under data/raw/delegation.
+    canada.ca blocks Python's HTTP client, so its PDF is fetched via PowerShell.
+    (refresh.py also calls this for its IL3 version check.)"""
+    return fetch_cached(url, dest, powershell=powershell)
+
+
+# --- CBSA instrument (HTML) ---------------------------------------------------
+
+def _delegates(cell):
+    """Flatten a 'Delegates / Designated officials' table cell into readable
+    lines. The cell groups officer positions under an organisation header
+    (<p class="h4">: CBSA, RCMP, IRCC) and an optional sub-heading
+    (<p class="h5">: a region or a headquarters branch), each followed by a
+    <ul> of position titles."""
+    lines, org, sub, org_used = [], "", "", False
+    for el in cell.find_all(["p", "ul"], recursive=False):
+        if el.name == "p":
+            classes = " ".join(el.get("class") or [])
+            text = _norm(el.get_text())
+            if not text:
+                continue
+            if "h4" in classes:
+                if org and not org_used:   # a previous org with no list of its own
+                    lines.append(org)
+                org, sub, org_used = text.rstrip(": "), "", False
+            elif "h5" in classes:
+                sub = text
+            else:
+                lines.append(text)         # a free-standing note
+        else:                              # <ul> of position titles
+            positions = "; ".join(
+                t for t in (_norm(li.get_text())
+                            for li in el.find_all("li", recursive=False)) if t)
+            if positions:
+                label = f"{org} — {sub}" if sub else org
+                lines.append(f"{label}: {positions}" if label else positions)
+                org_used = True
+    if org and not org_used:               # a trailing org with no list
+        lines.append(org)
+    return "\n".join(lines)
+
+
+def parse_cbsa(html, src):
+    """Parse the CBSA delegation instrument into one chunk per Schedule item,
+    plus one chunk for the preamble."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+    for sup in main.find_all("sup"):       # drop footnote-reference superscripts
+        sup.decompose()
+
+    # The effective date comes from SOURCES, not the first <time> in <main>:
+    # amendment pages quote the base instrument's date ("signed on May 8, 2023")
+    # in their preamble, so the first <time> on an amendment page is the base
+    # instrument's date, not the amendment's own.
+    date = src["effective"]
+
+    chunks = []
+
+    # Preamble: the paragraphs between the title and the Schedule, which set out
+    # the tiers of delegation and designation and how the columns are read.
+    schedule = main.find("h2", id="sch")
+    if schedule:
+        paras = [_norm(p.get_text())
+                 for p in reversed(schedule.find_previous_siblings("p"))]
+        body = "\n".join(p for p in paras if p)
+        if body:
+            chunks.append({
+                "id": f"delegation-{src['code']}-preamble",
+                "doc_type": "delegation",
+                "act_code": src["act_code"],
+                "act_short": src["act_short"],
+                "act_name": src["act_name"],
+                "section": "",
+                "marginal_note": "Tiers of delegation and designation",
+                "part": "",
+                "division": "",
+                "heading": "Instrument of delegation and designation under IRPA",
+                "text": body,
+                "history": "",
+                "last_amended": "",
+                "current_to": date,
+                "citation": f"{src['act_short']} — Preamble",
+                "source_url": src["url"],
+            })
+
+    # One chunk per Schedule item. Two row shapes are accepted:
+    #   (a) four <td> cells: Item | Refs | Power | Delegates (the 2023-05
+    #       restatement and the 2023-09, 2023-11, 2024-03 amendments).
+    #   (b) one <th> + two or three <td> cells: the <th> carries the item
+    #       number and the <td>s carry Refs | Power [| Delegates]. The
+    #       2025-07-10 amendment uses this layout, and may omit the Delegates
+    #       column when an amendment changes only references or descriptions.
+    # Each topical <h3>, if present, names the schedule section the table belongs to.
+    for table in main.find_all("table", class_="table-bordered"):
+        h3 = table.find_previous_sibling("h3")
+        section_name = _norm(h3.get_text()) if h3 else ""
+        for tr in table.find_all("tr"):
+            th_cells = tr.find_all("th", recursive=False)
+            td_cells = tr.find_all("td", recursive=False)
+            if not th_cells and len(td_cells) == 4:
+                item_cell, refs_cell, power_cell, deleg_cell = td_cells
+            elif len(th_cells) == 1 and len(td_cells) in (2, 3):
+                item_cell, refs_cell, power_cell = th_cells[0], td_cells[0], td_cells[1]
+                deleg_cell = td_cells[2] if len(td_cells) == 3 else None
+            else:
+                continue                   # header row or a stray row
+            item_no = _norm(item_cell.get_text()).rstrip(".")
+            refs = _normalize_refs(_norm(refs_cell.get_text()))
+            power = " ".join(_norm(p.get_text())
+                             for p in power_cell.find_all("p")) \
+                or _norm(power_cell.get_text())
+            delegates = _delegates(deleg_cell) if deleg_cell is not None else ""
+            if not item_no or not (power or refs):
+                continue
+            text = power
+            if refs:
+                text += f"\n\nProvisions (IRPA / IRPR): {refs}."
+            if delegates:
+                text += "\n\nDelegated / designated to:\n" + delegates
+            chunks.append({
+                "id": f"delegation-{src['code']}-{item_no}",
+                "doc_type": "delegation",
+                "act_code": src["act_code"],
+                "act_short": src["act_short"],
+                "act_name": src["act_name"],
+                "section": "",
+                "marginal_note": refs or f"Item {item_no}",
+                "part": section_name,
+                "division": "",
+                "heading": section_name,
+                "text": text,
+                "history": "",
+                "last_amended": "",
+                "current_to": date,
+                "citation": f"{src['act_short']}, Item {item_no}",
+                "source_url": src["url"],
+            })
+    return chunks
+
+
+# --- CBSA narrative-prose instrument (e.g. the peace-officer designation) -----
+
+def parse_cbsa_narrative(html, src):
+    """Parse a narrative-prose CBSA designation instrument into a single chunk.
+
+    Used for the peace-officer authorization under IRPA s. 138(1) -- plain prose
+    listing 18 designated officer positions, not a four-column Schedule table,
+    so parse_cbsa's table walker would yield nothing. The whole operative text
+    is a few hundred words, well within a single chunk."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+    for sup in main.find_all("sup"):
+        sup.decompose()
+
+    date = src["effective"]
+
+    # Skip the breadcrumb/title/footer chrome -- only paragraphs and lists in
+    # <main> that carry real content. Lists are rendered as "; "-joined items.
+    parts = []
+    for el in main.find_all(["p", "ul", "ol"]):
+        if el.find_parent(["ul", "ol"]):
+            continue   # nested lists are picked up by their parent
+        if el.name in ("ul", "ol"):
+            items = [_norm(li.get_text()) for li in el.find_all("li")]
+            joined = "; ".join(t for t in items if t)
+            if joined:
+                parts.append(joined)
+        else:
+            text = _norm(el.get_text())
+            if text:
+                parts.append(text)
+    body = "\n".join(parts)
+    if not body:
+        return []
+
+    return [{
+        "id": f"delegation-{src['code']}",
+        "doc_type": "delegation",
+        "act_code": src["act_code"],
+        "act_short": src["act_short"],
+        "act_name": src["act_name"],
+        "section": "",
+        "marginal_note": "Peace-officer authorization — IRPA s. 138(1)",
+        "part": "",
+        "division": "",
+        "heading": src["act_name"],
+        "text": body,
+        "history": "",
+        "last_amended": "",
+        "current_to": date,
+        "citation": src["act_short"],
+        "source_url": src["url"],
+    }]
+
+
+# --- IRCC IL3 instrument (PDF) ------------------------------------------------
+
+# A topical part heading -- a line in full upper case (MINISTERIAL INSTRUCTIONS,
+# MEDICAL, MISCELLANEOUS). Organisation acronyms (CI, CBSA, RCMP) are shorter
+# than the 6-character floor and so are not mistaken for headings.
+_IL3_PART = re.compile(r"[A-Z][A-Z &/,()'.\-]{5,}")
+# An item opens "<n>. <A/R provision>" -- the number must be followed by a
+# provision reference, which rejects ordinary numbered prose.
+_IL3_ITEM = re.compile(r"(\d+)\.\s+(?=[AR]\d)")
+# The word that introduces an item's power description, after the provisions.
+_IL3_POWER = re.compile(r"(?:Delegation|Designation)\s*[-–—]\s*")
+
+
+def _pdf_pages(pdf_bytes):
+    """Extract each page's text, dropping the printed page number that pypdf
+    emits as the page's first line (roman in the front matter, arabic later)."""
+    reader = PdfReader(io.BytesIO(pdf_bytes))
+    pages = []
+    for page in reader.pages:
+        lines = (page.extract_text() or "").split("\n")
+        if lines and re.fullmatch(r"\s*[ivxlcdm\d]+\s*", lines[0] or "", re.I):
+            lines = lines[1:]
+        pages.append("\n".join(lines))
+    return pages
+
+
+def parse_ircc(pdf_bytes, src):
+    """Parse the IRCC IL3 instrument into one chunk per Schedule item, plus a
+    preamble chunk. The PDF flattens the four-column table into a linear text
+    stream, so each item runs from its numbered marker to the next; the power
+    description and the delegated positions are kept together within the item."""
+    pages = _pdf_pages(pdf_bytes)
+    if not pages:
+        return []
+    version_match = re.search(r"(?:Spring|Summer|Fall|Winter)\s+\d{4}", pages[0])
+    version = version_match.group(0) if version_match else ""
+
+    chunks = []
+    preamble = _norm(pages[1]) if len(pages) > 1 else ""
+    if preamble:
+        chunks.append({
+            "id": f"delegation-{src['code']}-preamble",
+            "doc_type": "delegation",
+            "act_code": src["act_code"],
+            "act_short": src["act_short"],
+            "act_name": src["act_name"],
+            "section": "",
+            "marginal_note": "Preamble — designation and delegation under IRPA s. 6",
+            "part": "",
+            "division": "",
+            "heading": "Instrument of designation and delegation under IRPA",
+            "text": preamble,
+            "history": "",
+            "last_amended": "",
+            "current_to": version,
+            "citation": f"{src['act_short']} — Preamble",
+            "source_url": src["url"],
+        })
+
+    # Walk every line: an all-caps line is a topical part heading; a line that
+    # opens "<n>. <A/R provision>" starts a new item. Lines before the first
+    # item (the preamble, definitions and table of contents) are skipped.
+    items, part = [], ""
+    cur_no = cur_part = None
+    cur_lines = []
+    for line in "\n".join(pages).split("\n"):
+        s = _norm(line)
+        if not s:
+            continue
+        if _IL3_PART.fullmatch(s):
+            part = s.title()
+            continue
+        m = _IL3_ITEM.match(s)
+        if m:
+            if cur_no is not None:
+                items.append((cur_part, cur_no, cur_lines))
+            cur_no, cur_part, cur_lines = m.group(1), part, [s]
+        elif cur_no is not None:
+            cur_lines.append(s)
+    if cur_no is not None:
+        items.append((cur_part, cur_no, cur_lines))
+
+    for n, (item_part, item_no, lines) in enumerate(items, start=1):
+        body = re.sub(r"^\d+\.\s*", "", "\n".join(lines)).strip()
+        if not body:
+            continue
+        power = _IL3_POWER.search(body)
+        refs = _normalize_refs(body[:power.start()]) if power else ""
+        chunks.append({
+            "id": f"delegation-{src['code']}-{n}",
+            "doc_type": "delegation",
+            "act_code": src["act_code"],
+            "act_short": src["act_short"],
+            "act_name": src["act_name"],
+            "section": "",
+            "marginal_note": refs or f"Item {item_no}",
+            "part": item_part or "",
+            "division": "",
+            "heading": item_part or "",
+            "text": _normalize_refs(body),
+            "history": "",
+            "last_amended": "",
+            "current_to": version,
+            "citation": (f"{src['act_short']} — {item_part}, Item {item_no}"
+                         if item_part else f"{src['act_short']}, Item {item_no}"),
+            "source_url": src["url"],
+        })
+    return chunks
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of instruments this run could not build.
+
+    Ingestion is a full rebuild, so an instrument that fails its fetch -- or
+    that parses to nothing because CBSA changed the page's shape again -- would
+    silently vanish from the corpus. An instrument withdrawn on purpose is
+    removed from SOURCES instead, and so never lands in `failures`, which is
+    why preservation is restricted to the codes that actually failed here.
+    act_code is the per-instrument identity every parser stamps on its chunks.
+    Returns (chunks, preserved act_codes).
+    """
+    failed = {act_code for act_code, _why in failures}
+    return preserve_dropped(chunks, stored, key=lambda c: c["act_code"],
+                            only=failed)
+
+
+def build(allow_shrink=False):
+    """Fetch, parse and chunk every delegation instrument into delegation.json."""
+    all_chunks, failures = [], []
+    for src in SOURCES.values():
+        print(f"Ingesting {src['act_short']} ...")
+        try:
+            if src["kind"] == "html-cbsa":
+                html = _fetch(src["url"], RAW / f"{src['code']}.html")
+                chunks = parse_cbsa(html, src)
+            elif src["kind"] == "html-cbsa-narrative":
+                html = _fetch(src["url"], RAW / f"{src['code']}.html")
+                chunks = parse_cbsa_narrative(html, src)
+            elif src["kind"] == "pdf-ircc":
+                pdf = _fetch(src["url"], RAW / f"{src['code']}.pdf",
+                             powershell=True)
+                chunks = parse_ircc(pdf, src)
+            else:
+                chunks = []
+        except Exception as exc:
+            print(f"  !! {src['act_short']}: {type(exc).__name__}: {exc}")
+            failures.append((src["act_code"], f"{type(exc).__name__}: {exc}"))
+            continue
+        if not chunks:
+            failures.append((src["act_code"], "no content parsed"))
+        all_chunks.extend(chunks)
+        print(f"  {len(chunks)} chunks")
+    uniquify_ids(all_chunks)
+
+    all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                            stored_chunks(OUT))
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} instrument(s): "
+              f"{', '.join(preserved)}")
+    if not write_corpus(OUT, all_chunks, indent=1, allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} delegation chunks from {len(SOURCES)} "
+          f"instrument(s) -> {OUT}")
+    return True
+
+
+def main():
+    sys.exit(0 if build(allow_shrink="--allow-shrink" in sys.argv) else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/directive.py

@@ -0,0 +1,298 @@
+"""Ingest National Joint Council directives (HTML) into section-level chunks.
+
+NJC directives are negotiated by employer and bargaining-agent representatives;
+their provisions form part of collective agreements (and the rate tables in
+their appendices apply too). Chunks are tagged doc_type="directive".
+"""
+import re
+import sys
+
+from bs4 import BeautifulSoup
+
+from ._common import (MONTHS, fetch_cached, norm_ws as _norm, preserve_dropped,
+                      split_lines, stored_chunks, write_corpus)
+from .config import RAW_DIR, PROCESSED_DIR
+
+INDEX_URL = "https://www.njc-cnm.gc.ca/directive/en"
+BASE = "https://www.njc-cnm.gc.ca"
+DIRECTIVE_DIR = RAW_DIR / "directives"
+OUT_FILE = PROCESSED_DIR / "directives.json"
+_CONTENT_TAGS = {"p", "dl", "table", "h4", "h5", "h6", "blockquote"}
+_NUMBERED = re.compile(r"^(\d+(?:\.\d+)*)\s+(.+)")
+_CODE = re.compile(r"/directive/([^/]+)/")
+
+_PIECE_TARGET = 1800   # keep pieces under the embedder's ~2000-char window
+_EFFECTIVE = re.compile(
+    r"effective[^.<:]{0,20}?([A-Z][a-z]+)\s+(\d{1,2}),?\s+(\d{4})", re.I)
+
+
+def _split(text, target=_PIECE_TARGET):
+    """Split long text into ~target-sized pieces at line boundaries."""
+    return split_lines(text, target)
+
+
+def _page_effective_date(main):
+    """Latest 'effective <Month D, YYYY>' date on a directive page, as ISO.
+    Used as a currency fallback when the index gives no date (e.g. the Foreign
+    Service Directives, whose chunks otherwise render currency as 'n/a')."""
+    best = ""
+    for mon, day, year in _EFFECTIVE.findall(main.get_text(" ", strip=True)):
+        if mon.lower() in MONTHS:
+            iso = f"{int(year):04d}-{MONTHS[mon.lower()]:02d}-{int(day):02d}"
+            best = max(best, iso)
+    return best
+
+
+def _fetch(url, dest, force=False):
+    """Fetch via PowerShell's (.NET) HTTP stack -- the NJC site blocks
+    Python's HTTP client at the TLS layer."""
+    return fetch_cached(url, dest, force=force, powershell=True)
+
+
+def directive_links(force=False):
+    """Return [(url, title, date), ...] for the current NJC directives."""
+    html = _fetch(INDEX_URL, DIRECTIVE_DIR / "_index.html", force=force)
+    soup = BeautifulSoup(html, "html.parser")
+    ul = soup.find("ul", class_="directive-list")
+    if ul is None:
+        return []
+    out = []
+    for li in ul.find_all("li", recursive=False):
+        a = li.find("a", href=True)  # the first <a> is the current directive
+        if not a:
+            continue
+        url = a["href"] if a["href"].startswith("http") else BASE + a["href"]
+        span = li.find("span", class_="date")
+        out.append((url, _norm(a.get_text(" ", strip=True)),
+                    _norm(span.get_text()) if span else ""))
+    return out
+
+
+def _block_text(heading):
+    """Readable text from a heading up to the next h2/h3 (sections unwrapped)."""
+    lines = []
+    for sib in heading.find_next_siblings():
+        if sib.name in ("h2", "h3"):
+            break
+        if sib.name in ("ul", "ol"):
+            for li in sib.find_all("li", recursive=False):
+                item = _norm(li.get_text(" ", strip=True))
+                if item:
+                    lines.append(f"- {item}")
+        elif sib.name in _CONTENT_TAGS:
+            text = _norm(sib.get_text(" ", strip=True))
+            if text:
+                lines.append(text)
+    return "\n".join(lines)
+
+
+def parse_directive(html, url, title, date):
+    """Parse one NJC directive page into one chunk per h2/h3 section."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+    for tag in main.find_all(["section", "div"]):
+        tag.unwrap()  # flatten so each heading and its content become siblings
+
+    code_match = _CODE.search(url)
+    code = code_match.group(1) if code_match else url
+    if not date:                       # recover a currency date from the page
+        date = _page_effective_date(main)
+    chunks = []
+    current_part = ""
+    sec_idx = 0
+    for h in main.find_all(["h2", "h3"]):
+        text = _norm(h.get_text(" ", strip=True))
+        if not text:
+            continue
+        if h.name == "h2":
+            current_part = text
+        body = _block_text(h)
+        if not body:
+            continue
+        numbered = _NUMBERED.match(text)
+        if numbered:
+            number, note = numbered.group(1), numbered.group(2).strip()
+            citation = f"{title}, s. {number}"
+        else:
+            # Un-numbered heading: keep the text in the note only, so the rendered
+            # header (citation + note) does not repeat it.
+            number, note = "", text
+            citation = title
+        sec_idx += 1
+        # Split a long section so each piece fits the embed window; pieces of one
+        # section share the base id 'directive-{code}-{sec_idx}' and take a
+        # '-p<k>' suffix, so the diversity cap treats them as one source.
+        pieces = _split(body)
+        for k, piece in enumerate(pieces, start=1):
+            base = f"directive-{code}-{sec_idx}"
+            chunk_id = base if len(pieces) == 1 else f"{base}-p{k}"
+            pnote = note if len(pieces) == 1 else f"{note} (part {k})"
+            chunks.append({
+                "id": chunk_id,
+                "doc_type": "directive",
+                "act_code": code,
+                "act_short": title,
+                "act_name": f"NJC {title}",
+                "section": number or text,
+                "marginal_note": pnote,
+                "part": current_part if h.name == "h3" else "",
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": "",
+                "current_to": date,
+                "citation": citation,
+                "source_url": url,
+            })
+    return chunks
+
+
+def _print_link(html):
+    """Find a 'Print Full Directive' / 'Print all FSDs' link on a TOC page
+    (or its French label on the /fr editions)."""
+    soup = BeautifulSoup(html, "html.parser")
+    for a in soup.find_all("a", href=True):
+        text = _norm(a.get_text(" ", strip=True)).lower()
+        if ("print full directive" in text or "print all fsd" in text
+                or "imprimer la directive" in text
+                or "imprimer toutes" in text or "imprimez toutes" in text):
+            return a["href"] if a["href"].startswith("http") else BASE + a["href"]
+    return None
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of directives this run could not parse.
+
+    Ingestion is a full rebuild, so a directive whose page 404s or is restyled
+    vanishes from the corpus entirely -- and a directive that is still in force
+    is better served from the copy we already had than not at all. Keyed on
+    act_code (the /directive/<code>/ slug), which survives the title rewordings
+    the NJC index goes through; restricted to the directives that actually
+    failed, so any the NJC has genuinely retired stay gone.
+    Returns (chunks, preserved directive codes).
+    """
+    failed = {code for code, _title, _why in failures}
+    return preserve_dropped(chunks, stored, key=lambda c: c["act_code"],
+                            only=failed)
+
+
+def ingest(force=False, limit=None, allow_shrink=False):
+    directives = directive_links(force=force)
+    if limit:
+        directives = directives[:limit]
+    print(f"Ingesting {len(directives)} NJC directives...")
+    all_chunks, failures = [], []
+    for url, title, date in directives:
+        code_match = _CODE.search(url)
+        code = code_match.group(1) if code_match else "x"
+        try:
+            html = _fetch(url, DIRECTIVE_DIR / f"{code}.html", force=force)
+            chunks = parse_directive(html, url, title, date)
+            if not chunks:
+                # Multi-page directive: the landing page is only a table of
+                # contents -- follow its "Print Full Directive" link.
+                print_url = _print_link(html)
+                if print_url:
+                    full = _fetch(print_url, DIRECTIVE_DIR / f"{code}-full.html",
+                                  force=force)
+                    chunks = parse_directive(full, url, title, date)
+            if chunks:
+                all_chunks.extend(chunks)
+                print(f"  {title}: {len(chunks)} chunks")
+            else:
+                failures.append((code, title, "no content parsed"))
+        except Exception as exc:
+            failures.append((code, title, f"{type(exc).__name__}: {exc}"))
+    if not limit:
+        all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                                stored_chunks(OUT_FILE))
+        if preserved:
+            print(f"  kept the last-good copy of {len(preserved)} unparseable "
+                  f"directive(s): {', '.join(preserved)}")
+    # Before the write, so a refusal does not bury why the run came up short.
+    for _code, title, why in failures:
+        print(f"  FAILED {title}: {why}")
+    # The guard applies to --limit runs too: a smoke test over two directives
+    # must not be able to write a two-directive corpus over the real one.
+    if not write_corpus(OUT_FILE, all_chunks, indent=2,
+                        allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} chunks from {len(directives) - len(failures)} "
+          f"directives -> {OUT_FILE.name}")
+    return True
+
+
+OUT_FILE_FR = OUT_FILE.with_name("directives-fr.json")
+
+
+def ingest_fr(force=False, limit=None, allow_shrink=False):
+    """The French editions of the NJC directives, mirroring ingest().
+
+    The NJC publishes each directive at .../directive/<code>/fr with the
+    same markup as /en, so the parser is reused as-is. The EN index supplies
+    the directive list and identity (act_short stays the English title so
+    act filters behave one way regardless of language); chunks get
+    lang='fr' and a -fr id infix, and language-routed retrieval serves them
+    only to French queries."""
+    directives = directive_links(force=force)
+    if limit:
+        directives = directives[:limit]
+    print(f"Ingesting {len(directives)} directives du CNM (français)...")
+    all_chunks, failures = [], []
+    for url, title, date in directives:
+        code_match = _CODE.search(url)
+        code = code_match.group(1) if code_match else "x"
+        fr_url = re.sub(r"/en$", "/fr", url)
+        try:
+            html = _fetch(fr_url, DIRECTIVE_DIR / f"{code}-fr.html",
+                          force=force)
+            chunks = parse_directive(html, fr_url, title, date)
+            if not chunks:
+                print_url = _print_link(html)
+                if print_url:
+                    full = _fetch(print_url,
+                                  DIRECTIVE_DIR / f"{code}-fr-full.html",
+                                  force=force)
+                    chunks = parse_directive(full, fr_url, title, date)
+            if chunks:
+                all_chunks.extend(chunks)
+                print(f"  {title} (fr): {len(chunks)} chunks")
+            else:
+                failures.append((code, title, "no content parsed"))
+        except Exception as exc:
+            failures.append((code, title, f"{type(exc).__name__}: {exc}"))
+    for c in all_chunks:
+        c["id"] = c["id"].replace("directive-", "directive-fr-", 1)
+        c["lang"] = "fr"
+    if not limit:
+        all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                                stored_chunks(OUT_FILE_FR))
+        if preserved:
+            print(f"  kept the last-good copy of {len(preserved)} "
+                  f"unparseable directive(s)")
+    for _code, title, why in failures:
+        print(f"  FAILED {title} (fr): {why}")
+    if not write_corpus(OUT_FILE_FR, all_chunks, indent=2,
+                        allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} chunks from "
+          f"{len(directives) - len(failures)} directives -> "
+          f"{OUT_FILE_FR.name}")
+    return True
+
+
+def main():
+    force = "--force" in sys.argv
+    limit = next((int(a.split("=", 1)[1]) for a in sys.argv[1:]
+                  if a.startswith("--limit=")), None)
+    runner = ingest_fr if "--fr" in sys.argv else ingest
+    ok = runner(force=force, limit=limit,
+                allow_shrink="--allow-shrink" in sys.argv)
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/dmemo.py

@@ -0,0 +1,368 @@
+"""Ingest CBSA D-Memoranda (HTML) into structured, section-level chunks.
+
+D-Memoranda are CBSA's administrative guidance on how it applies the Customs Act
+and related law. They are persuasive, not binding -- every chunk is tagged
+doc_type="memorandum" so the rest of CanLex can keep them distinct from statute.
+"""
+import io
+import json
+import re
+import sys
+from urllib.parse import urljoin
+
+from bs4 import BeautifulSoup
+from pypdf import PdfReader
+
+from ._common import (POLITE_UA, fetch_cached, norm_ws as _norm,
+                      preserve_dropped, split_lines, stored_chunks,
+                      write_corpus)
+from .config import RAW_DIR, PROCESSED_DIR
+
+INDEX_URL = "https://www.cbsa-asfc.gc.ca/publications/dm-md/d1-d23-eng.html"
+# CBSA rebuilt the index on 2026-07-10: the page now ships an empty WET
+# DataTable and NO memo links at all -- the rows arrive from this JSON
+# datasource. Each row's NUMBER holds the anchor
+# ('<a href="/publications/dm-md/d1/d1-2-1-eng.html">D1-2-1</a>'), and
+# cancelled memos carry STATUS "Cancelled" (usually with the link stripped).
+# Cancelled memos are deliberately not ingested -- same rule the ENF manuals
+# follow: retired guidance must not surface as if it were current.
+DATASOURCE_URL = ("https://www.cbsa-asfc.gc.ca/publications/dm-md/"
+                  "dmemoranda-datasource-eng.json")
+DMEMO_DIR = RAW_DIR / "dmemos"
+OUT_FILE = PROCESSED_DIR / "dmemos.json"
+
+# <h2> headings that are page boilerplate rather than memo content.
+_SKIP_HEADINGS = {"contact us", "related links",
+                  "contactez-nous", "liens connexes"}
+# Bilingual: the French editions live at the same paths with -fra.html, so
+# one pattern serves both ingest passes.
+_MEMO_HREF = re.compile(r"/dm-md/d\d+/d[\d-]+-(?:eng|fra)\.html")
+_URL_NUMBER = re.compile(r"/(d\d+-[\d-]+)-(?:eng|fra)\.html")
+_DS_HREF = re.compile(r'href="([^"]+)"')
+DATASOURCE_URL_FR = DATASOURCE_URL.replace("-eng.json", "-fra.json")
+OUT_FILE_FR = PROCESSED_DIR / "dmemos-fr.json"
+
+
+def _fetch(url, dest, force=False):
+    return fetch_cached(url, dest, ua=POLITE_UA, force=force)
+
+
+def datasource_urls(raw):
+    """Active D-memo URLs from the index's JSON datasource. Pure, for tests."""
+    urls, seen = [], set()
+    for row in json.loads(raw).get("data", []):
+        if _norm(row.get("STATUS", "")).lower() in ("cancelled", "annulé"):
+            continue
+        match = _DS_HREF.search(row.get("NUMBER", ""))
+        if not match:
+            continue
+        full = urljoin(DATASOURCE_URL, match.group(1))
+        if full not in seen:
+            seen.add(full)
+            urls.append(full)
+    return urls
+
+
+def index_urls(html):
+    """D-memo URLs linked from the index page itself. Pure, for tests.
+
+    How the index worked before 2026-07 and how it would work again if CBSA
+    reverts; kept as the fallback for datasource_urls.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+    urls, seen = [], set()
+    for a in soup.find_all("a", href=True):
+        if _MEMO_HREF.search(a["href"]):
+            full = urljoin(INDEX_URL, a["href"])
+            if full not in seen:
+                seen.add(full)
+                urls.append(full)
+    return urls
+
+
+def memo_urls(force=False):
+    """All individual active D-memo URLs, from the CBSA index's datasource,
+    falling back to links on the index page if the datasource is gone."""
+    try:
+        urls = datasource_urls(_fetch(DATASOURCE_URL,
+                                      DMEMO_DIR / "_datasource-eng.json",
+                                      force=force))
+    except Exception as exc:
+        print(f"  datasource unavailable ({type(exc).__name__}: {exc}); "
+              f"falling back to index page links")
+        urls = []
+    if not urls:
+        urls = index_urls(_fetch(INDEX_URL, DMEMO_DIR / "_index.html",
+                                 force=force))
+    return urls
+
+
+def _render_section(h2):
+    """Readable text from an <h2> up to the next <h2> (sections already unwrapped)."""
+    lines = []
+    for sib in h2.find_next_siblings():
+        if sib.name == "h2" or sib.get("id") == "wb-dtmd":
+            break
+        if sib.name in ("ul", "ol"):
+            for li in sib.find_all("li", recursive=False):
+                item = _norm(li.get_text(" ", strip=True))
+                if item:
+                    lines.append(f"- {item}")
+        else:
+            text = _norm(sib.get_text(" ", strip=True))
+            if text:
+                lines.append(text)
+    return "\n".join(lines)
+
+
+def parse_memo(html, url):
+    """Parse one D-memo HTML page into one chunk per <h2> content section."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+    for section in main.find_all("section"):
+        section.unwrap()  # flatten so each <h2> and its content become siblings
+
+    match = _URL_NUMBER.search(url)
+    number = match.group(1).upper() if match else url
+    h1 = main.find("h1")
+    topic = ""
+    if h1:
+        # Pages vary: most carry the memo title in <h1><small>, others as plain
+        # "Memorandum DNN-N-N: Title" h1 text. Use the <small> if present, else
+        # the h1 text, and strip any leading memo-number prefix either way.
+        small = h1.find("small")
+        raw = (small.get_text(" ", strip=True) if small
+               else h1.get_text(" ", strip=True))
+        topic = re.sub(r"^Memorandum\s+D[\w-]+\s*[:–-]\s*", "",
+                       _norm(raw), flags=re.I)
+    dm = main.find("time", attrs={"property": "dateModified"})
+    date = _norm(dm.get("datetime") or dm.get_text()) if dm else ""
+
+    chunks = []
+    for h2 in main.find_all("h2"):
+        heading = _norm(h2.get_text(" ", strip=True))
+        if not heading or heading.lower() in _SKIP_HEADINGS:
+            continue
+        body = _render_section(h2)
+        if not body:
+            continue
+        # One chunk per h2 section. (Splitting long memos to the embed window
+        # was trialled and reverted: it tripled memo chunk-count and gave
+        # D-memoranda enough extra retrieval surface to out-rank the statutes
+        # and sibling memos they relate to -- a measured eval regression. The
+        # memo-tail recall gap is better addressed later by multi-vector
+        # embedding than by more competing chunks.)
+        chunks.append({
+            "id": f"dmemo-{number}-{len(chunks) + 1}",
+            "doc_type": "memorandum",
+            "act_code": "D-Memo",
+            "act_short": "D-Memo",
+            "act_name": "CBSA D-Memoranda",
+            "section": number,
+            "marginal_note": heading,
+            "part": topic,
+            "division": "",
+            "heading": "",
+            "text": body,
+            "history": "",
+            "last_amended": date,
+            "current_to": date,
+            "citation": f"Memorandum {number}",
+            "source_url": url,
+        })
+    return chunks
+
+
+def _pdf_clean(text):
+    text = re.sub(r"[ \t]+", " ", text)
+    text = re.sub(r"\n[ \t]+", "\n", text)
+    return re.sub(r"\n{3,}", "\n\n", text).strip()
+
+
+def _pdf_text(pdf_bytes):
+    try:
+        reader = PdfReader(io.BytesIO(pdf_bytes))
+        return _pdf_clean("\n".join((p.extract_text() or "") for p in reader.pages))
+    except Exception:
+        return ""
+
+
+def _split(text, target=3000):
+    """Split long PDF text into ~target-sized pieces at line boundaries."""
+    return split_lines(text, target)
+
+
+def parse_pdf_memo(html, url):
+    """Fallback for memos whose HTML page is only a stub linking to a PDF."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+    pdf_href = next((a["href"] for a in main.find_all("a", href=True)
+                     if a["href"].lower().endswith(".pdf")), None)
+    if not pdf_href:
+        return []
+    pdf_url = urljoin(url, pdf_href)
+
+    match = _URL_NUMBER.search(url)
+    number = match.group(1).upper() if match else url
+    h1 = main.find("h1")
+    topic = _norm(h1.get_text(" ", strip=True)) if h1 else ""
+    topic = re.sub(r"^Memorandum\s+D[\w-]+\s*[:–-]\s*", "", topic, flags=re.I)
+    dm = main.find("time", attrs={"property": "dateModified"})
+    date = _norm(dm.get("datetime") or dm.get_text()) if dm else ""
+
+    pdf_bytes = _fetch(pdf_url, DMEMO_DIR / "pdf" / pdf_url.rsplit("/", 1)[-1])
+    text = _pdf_text(pdf_bytes)
+    if not text:
+        return []
+
+    parts = _split(text)
+    chunks = []
+    for i, part in enumerate(parts, 1):
+        label = topic or number
+        if len(parts) > 1:
+            label = f"{label} (part {i})"
+        chunks.append({
+            "id": f"dmemo-{number}-pdf{i}",
+            "doc_type": "memorandum",
+            "act_code": "D-Memo",
+            "act_short": "D-Memo",
+            "act_name": "CBSA D-Memoranda",
+            "section": number,
+            "marginal_note": label,
+            "part": topic,
+            "division": "",
+            "heading": "",
+            "text": part,
+            "history": "",
+            "last_amended": date,
+            "current_to": date,
+            "citation": f"Memorandum {number}",
+            "source_url": url,
+        })
+    return chunks
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of memos that could not be fetched.
+
+    Ingestion is a full rebuild, so anything that fails a fetch disappears from
+    the corpus. CBSA's rebuilt index (2026-07) lists several memos as Active
+    whose pages 404, and a transient network error looks the same -- in both
+    cases dropping guidance that is still in force is worse than serving the
+    copy we already had. Restricted to memos that actually failed, so the
+    cancelled ones (filtered out before fetching) are not resurrected.
+    Returns (chunks, preserved memo numbers).
+    """
+    failed = set()
+    for url, _why in failures:
+        match = _URL_NUMBER.search(url)
+        if match:
+            failed.add(match.group(1).upper())
+    return preserve_dropped(chunks, stored,
+                            key=lambda c: c["section"].upper(), only=failed)
+
+
+def ingest(force=False, limit=None, allow_shrink=False):
+    urls = memo_urls(force=force)
+    if limit:
+        urls = urls[:limit]
+    print(f"Ingesting {len(urls)} D-Memoranda...")
+    all_chunks, failures = [], []
+    for i, url in enumerate(urls, 1):
+        try:
+            html = _fetch(url, DMEMO_DIR / url.rsplit("/", 1)[-1], force=force)
+            chunks = parse_memo(html, url) or parse_pdf_memo(html, url)
+            if chunks:
+                all_chunks.extend(chunks)
+            else:
+                failures.append((url, "no content parsed"))
+        except Exception as exc:
+            failures.append((url, f"{type(exc).__name__}: {exc}"))
+        if i % 50 == 0:
+            print(f"  {i}/{len(urls)} ...")
+    if failures:
+        print(f"  {len(failures)} memos with no content / errors:")
+        for url, why in failures[:15]:
+            print(f"    - {url.rsplit('/', 1)[-1]}: {why}")
+
+    if not limit:
+        all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                                stored_chunks(OUT_FILE))
+        if preserved:
+            print(f"  kept the last-good copy of {len(preserved)} unfetchable "
+                  f"memo(s): {', '.join(preserved)}")
+    # The guard applies to --limit runs too: a smoke test over five memos must
+    # not be able to write a five-memo corpus over the real one.
+    if not write_corpus(OUT_FILE, all_chunks, allow_shrink=allow_shrink):
+        return False
+    print(f"  {len(all_chunks)} section-chunks from {len(urls) - len(failures)} memos "
+          f"-> {OUT_FILE.name}")
+    return True
+
+
+def ingest_fr(force=False, limit=None, allow_shrink=False):
+    """The French editions (Mémorandums D), mirroring ingest().
+
+    Same pages at -fra.html, same WET markup, same parser; the chunks are
+    post-processed with lang='fr', a dmemo-fr- id prefix and the French
+    citation so language-routed retrieval (index.query_lang) serves them
+    only to French queries."""
+    try:
+        urls = datasource_urls(_fetch(DATASOURCE_URL_FR,
+                                      DMEMO_DIR / "_datasource-fra.json",
+                                      force=force))
+    except Exception as exc:
+        print(f"  FR datasource unavailable ({type(exc).__name__}: {exc})")
+        urls = []
+    if limit:
+        urls = urls[:limit]
+    print(f"Ingesting {len(urls)} Mémorandums D (français)...")
+    all_chunks, failures = [], []
+    for i, url in enumerate(urls, 1):
+        try:
+            html = _fetch(url, DMEMO_DIR / url.rsplit("/", 1)[-1], force=force)
+            chunks = parse_memo(html, url) or parse_pdf_memo(html, url)
+            if chunks:
+                all_chunks.extend(chunks)
+            else:
+                failures.append((url, "no content parsed"))
+        except Exception as exc:
+            failures.append((url, f"{type(exc).__name__}: {exc}"))
+        if i % 50 == 0:
+            print(f"  {i}/{len(urls)} ...")
+    for c in all_chunks:
+        c["id"] = c["id"].replace("dmemo-", "dmemo-fr-", 1)
+        c["lang"] = "fr"
+        c["citation"] = c["citation"].replace("Memorandum", "Mémorandum", 1)
+    if failures:
+        print(f"  {len(failures)} mémorandums with no content / errors "
+              f"(first: {failures[0][0].rsplit('/', 1)[-1]})")
+    if not limit:
+        all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                                stored_chunks(OUT_FILE_FR))
+        if preserved:
+            print(f"  kept the last-good copy of {len(preserved)} "
+                  f"unfetchable mémorandum(s)")
+    if not write_corpus(OUT_FILE_FR, all_chunks, allow_shrink=allow_shrink):
+        return False
+    print(f"  {len(all_chunks)} section-chunks from "
+          f"{len(urls) - len(failures)} mémorandums -> {OUT_FILE_FR.name}")
+    return True
+
+
+def main():
+    force = "--force" in sys.argv
+    limit = next((int(a.split("=", 1)[1]) for a in sys.argv[1:]
+                  if a.startswith("--limit=")), None)
+    runner = ingest_fr if "--fr" in sys.argv else ingest
+    ok = runner(force=force, limit=limit,
+                allow_shrink="--allow-shrink" in sys.argv)
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/embed.py

@@ -0,0 +1,165 @@
+"""Build semantic embeddings for ingested chunks (local, key-free).
+
+Uses BAAI's bge-small-en-v1.5 sentence-embedding model as ONNX, run on CPU via
+onnxruntime -- no API key. A transformer embedding has far stronger retrieval
+recall than a static one: it can connect a natural-language question to a
+provision even when the two share few exact words.
+"""
+import json
+import os
+
+import numpy as np
+import onnxruntime as ort
+from huggingface_hub import hf_hub_download
+from tokenizers import Tokenizer
+
+from .config import PROCESSED_DIR
+
+# The embedding model is env-configurable so an alternative (e.g. a multilingual
+# model for French support) can be trialled without code edits; the defaults
+# reproduce the production bge-small-en-v1.5 setup exactly. A model swap means
+# matching three things to that model's training: the HF repo, the pooling
+# ('cls' for BGE, 'mean' for E5/MiniLM), and the query/passage prefixes (BGE
+# prefixes only the query; E5 prefixes both with 'query: ' / 'passage: ').
+EMB_REPO = os.environ.get("CANLEX_EMB_REPO", "Xenova/bge-small-en-v1.5")
+_POOLING = os.environ.get("CANLEX_EMB_POOLING", "cls").lower()
+_QUERY_PREFIX = os.environ.get(
+    "CANLEX_EMB_QUERY_PREFIX",
+    "Represent this sentence for searching relevant passages: ")
+_PASSAGE_PREFIX = os.environ.get("CANLEX_EMB_PASSAGE_PREFIX", "")
+EMB_PATH = PROCESSED_DIR / "embeddings.npz"
+_MAX_TOKENS = 512
+_MAX_BODY = 2000   # cap embedded body text so long sections stay topically focused
+
+
+def load_chunks():
+    chunks = []
+    for path in sorted(PROCESSED_DIR.glob("*.json")):
+        chunks.extend(json.loads(path.read_text(encoding="utf-8")))
+    return chunks
+
+
+# A chunk longer than the embed window gets additional vectors for its tail,
+# up to this many per chunk (an 87K-char D-memo section would otherwise need
+# 40+; eight windows cover 16K chars, past which BM25 carries the tail).
+_MAX_WINDOWS = 8
+
+
+def embed_text(chunk):
+    """Compact, retrieval-focused representation of one section (head only)."""
+    # The section title is the strongest topical signal, so it is repeated to
+    # emphasise it. Title selection is doc_type-aware (see index.topical_title):
+    # a D-memo's marginal_note is a generic banner so its actual subject in
+    # 'part' is used; a case-law chunk's marginal_note is just the paragraph
+    # range so the case proposition in 'heading' is used.
+    from .index import topical_title
+    note = topical_title(chunk)
+    body = chunk["text"][:_MAX_BODY]
+    parts = [chunk["act_short"], note, note, chunk["heading"], body]
+    return " . ".join(p for p in parts if p)
+
+
+def embed_texts(chunk):
+    """One representation per embedding window of a chunk.
+
+    The first is the classic head representation; a chunk whose body exceeds
+    the embed window additionally gets tail windows (same topical-title
+    anchor, successive body slices), so a fact buried mid-memo is reachable
+    through semantic recall instead of BM25 alone. All windows share the
+    chunk's id -- the index scores a chunk by its best window."""
+    from .index import topical_title
+    texts = [embed_text(chunk)]
+    body = chunk["text"]
+    if len(body) > _MAX_BODY:
+        note = topical_title(chunk)
+        anchor = " . ".join(p for p in (chunk["act_short"], note) if p)
+        for start in range(_MAX_BODY, len(body), _MAX_BODY):
+            if len(texts) >= _MAX_WINDOWS:
+                break
+            texts.append(f"{anchor} . {body[start:start + _MAX_BODY]}")
+    return texts
+
+
+class Embedder:
+    """Local transformer sentence-embedder: bge-small-en-v1.5 as ONNX on CPU.
+
+    No API key; the model is downloaded once and cached. Produces L2-normalized
+    vectors, so a dot product between them is cosine similarity.
+    """
+
+    def __init__(self):
+        model_path = None
+        for name in ("onnx/model_quantized.onnx", "onnx/model.onnx"):
+            try:
+                model_path = hf_hub_download(EMB_REPO, name)
+                break
+            except Exception:
+                continue
+        if model_path is None:
+            raise RuntimeError(f"Could not download an ONNX model from {EMB_REPO}.")
+        tok_path = hf_hub_download(EMB_REPO, "tokenizer.json")
+        from .rerank import make_ort_session
+        self.session = make_ort_session(model_path)
+        self.input_names = {i.name for i in self.session.get_inputs()}
+        self.tokenizer = Tokenizer.from_file(tok_path)
+        self.tokenizer.enable_truncation(max_length=_MAX_TOKENS)
+
+    def _run(self, texts):
+        """Tokenize, run the encoder, CLS-pool and L2-normalize one batch."""
+        encs = self.tokenizer.encode_batch(list(texts))
+        width = max(len(e.ids) for e in encs)
+        input_ids = np.zeros((len(encs), width), dtype=np.int64)
+        attention = np.zeros((len(encs), width), dtype=np.int64)
+        type_ids = np.zeros((len(encs), width), dtype=np.int64)
+        for row, enc in enumerate(encs):
+            n = len(enc.ids)
+            input_ids[row, :n] = enc.ids
+            attention[row, :n] = enc.attention_mask
+            type_ids[row, :n] = enc.type_ids
+        feed = {"input_ids": input_ids, "attention_mask": attention}
+        if "token_type_ids" in self.input_names:
+            feed["token_type_ids"] = type_ids
+        hidden = np.asarray(self.session.run(None, feed)[0], dtype=np.float32)
+        if hidden.ndim != 3:
+            pooled = hidden                                    # already pooled
+        elif _POOLING == "mean":                               # E5 / MiniLM
+            mask = attention[:, :, None].astype(np.float32)
+            pooled = (hidden * mask).sum(1) / np.maximum(mask.sum(1), 1e-9)
+        else:                                                  # BGE: CLS token
+            pooled = hidden[:, 0, :]
+        norms = np.linalg.norm(pooled, axis=1, keepdims=True)
+        return pooled / np.maximum(norms, 1e-9)
+
+    def encode(self, texts, batch_size=32):
+        """Return L2-normalized embeddings for passages, one row per text."""
+        texts = [_PASSAGE_PREFIX + t for t in texts]
+        if not texts:
+            return np.zeros((0, 384), dtype=np.float32)
+        rows = [self._run(texts[i:i + batch_size])
+                for i in range(0, len(texts), batch_size)]
+        return np.vstack(rows)
+
+    def encode_query(self, text):
+        """Return the L2-normalized embedding for one search query."""
+        return self._run([_QUERY_PREFIX + text])[0]
+
+
+def build():
+    chunks = load_chunks()
+    if not chunks:
+        print(f"No processed data in {PROCESSED_DIR}. Run 'canlex.ingest' first.")
+        return
+    texts, ids = [], []
+    for c in chunks:
+        for t in embed_texts(c):
+            texts.append(t)
+            ids.append(c["id"])
+    print(f"Embedding {len(chunks)} sections as {len(texts)} windows "
+          f"with {EMB_REPO} ...")
+    vectors = Embedder().encode(texts)
+    np.savez(EMB_PATH, ids=np.array(ids), vectors=vectors)
+    print(f"  {vectors.shape[0]} vectors, dim {vectors.shape[1]} -> {EMB_PATH.name}")
+
+
+if __name__ == "__main__":
+    build()

canlex/enf.py

@@ -0,0 +1,295 @@
+"""Ingest the IRCC/CBSA ENF operational-manual chapters.
+
+The ENF chapters are the operational layer between the statute and the
+officer: ENF 4 (port-of-entry examinations), ENF 5 (writing A44(1) reports),
+ENF 6 (Minister's-delegate review), ENF 12 (search and seizure), ENF 20
+(detention) and their siblings are what an immigration-secondary BSO actually
+opens. The legacy PDFs live under the canada.ca manuals directory that
+delegation.py already fetches IL3 from (same PowerShell TLS workaround), and
+are parsed with pypdf, TOC-segmented by their internal numbered headings.
+
+Chapters are probed rather than hardcoded: the numbering has gaps and IRCC
+retires chapters, so anything that 404s is skipped with a note. doc_type is
+'memorandum' -- operational guidance, persuasive not binding, exactly like
+the D-memoranda.
+
+    python -m canlex.enf [--allow-shrink]
+"""
+import re
+import sys
+import urllib.error
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      split_lines, stored_chunks, uniquify_ids, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+BASE = ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
+        "resources/manuals/enf/enf{n:02d}-eng.pdf")
+RAW = RAW_DIR / "enf"
+_CHAPTERS = range(1, 41)      # probe; gaps and retired chapters just skip
+
+# Chapters whose current edition lives under a non-standard filename in the
+# same directory (verified 2026-07-22 against the OBM Active-manuals index).
+# ENF 3's old enf03-eng.pdf still returns 200 but is a STALE 2015-era copy;
+# ENF 20 (Detention) was renamed enf20a-en.pdf (Aug 2024 edition).
+_URL_OVERRIDES = {
+    3: ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
+        "resources/manuals/enf/enf-3-admissibility-(en)-final.pdf"),
+    20: ("https://www.canada.ca/content/dam/ircc/migration/ircc/english/"
+         "resources/manuals/enf/enf20a-en.pdf"),
+}
+
+# Chapters IRCC withdrew from public distribution whose only public text is
+# an ATIP-released copy hosted by a third party (meurrensonimmigration.com,
+# verified live 2026-07-22). Ingested at the maintainer's decision
+# (2026-07-23) with an unmistakable UNOFFICIAL banner on every chunk: the
+# text may lag the internal current version, and nothing here is an official
+# publication. ENF 14/OP 19 is the operational manual behind criminal
+# rehabilitation -- directly relevant to canlex_rehabilitation.
+UNOFFICIAL = {
+    2: {"url": "https://meurrensonimmigration.com/wp-content/uploads/2023/06/"
+               "ENF2-3.pdf",
+        "title": "Evaluating Inadmissibility (unofficial ATIP copy)",
+        "note": "ATIP-released copy dated 2021-08-25; retired from canada.ca "
+                "in 2018 with no public successor"},
+    # ENF 14's ATIP release is a 56-page image scan with no text layer; its
+    # text comes from a Windows-OCR pass cached as enf14-ocr.txt (see the
+    # OCR fallback in build()). Regenerate the cache with pymupdf
+    # rasterization + Windows.Media.Ocr if the source PDF ever changes.
+    14: {"url": "https://meurrensonimmigration.com/wp-content/uploads/2025/03/"
+                "Enforcement-Manual-14-%E2%80%93-Rehabilitation.pdf",
+         "title": "Criminal Rehabilitation (ENF 14/OP 19, unofficial ATIP "
+                  "copy)",
+         "note": "withdrawn from public distribution ~2007; ATIP-released "
+                 "scan, text recovered by OCR (last internal change "
+                 "2008-04-21) -- OCR artifacts possible"},
+}
+_UNOFFICIAL_BANNER = (
+    "[UNOFFICIAL ATIP COPY — this chapter was withdrawn from public "
+    "distribution; this text comes from an access-to-information release "
+    "hosted by a third party and may lag the current internal version. "
+    "Treat as historical guidance; verify against internal sources.] ")
+
+# Curated titles for the core chapters; anything else gets a parsed or
+# generic title. Only used for display -- retrieval works off the text.
+TITLES = {
+    1: "Inadmissibility",
+    2: "Evaluating inadmissibility",
+    3: "Admissibility hearings and detention review proceedings",
+    4: "Port of entry examinations",
+    5: "Writing 44(1) reports",
+    6: "Review of reports under A44(1)",
+    7: "Investigations and arrests",
+    8: "Deposits and guarantees",
+    9: "Judicial review",
+    10: "Removals",
+    11: "Verifying departure",
+    12: "Search, seizure, fingerprinting and photographing",
+    13: "Examining bank drafts and other negotiable instruments",
+    14: "Investigations and arrests (war crimes)",
+    15: "Obligations of transporters",
+    17: "Maintaining confidence in the immigration program",
+    18: "War crimes and crimes against humanity",
+    19: "Appeals before the Immigration Appeal Division",
+    20: "Detention",
+    21: "Recovery of removal costs",
+    23: "Loss of permanent resident status",
+    27: "Permanent resident card",
+    28: "Ministerial opinions on danger to the public",
+    34: "Alternatives to prosecution",
+}
+
+_DATE = re.compile(r"\b(20\d{2}-\d{2}-\d{2})\b")
+# A manual heading line: "5. Departmental policy" / "5.1 Authority to ..."
+_HEADING = re.compile(r"^(\d{1,2}(?:\.\d{1,2})?)\.?\s+([A-Z][^\n]{2,90})$")
+
+
+def _pdf_text(path):
+    from pypdf import PdfReader
+    reader = PdfReader(str(path))
+    pages = []
+    for page in reader.pages:
+        try:
+            pages.append(page.extract_text() or "")
+        except Exception:
+            pages.append("")
+    return pages
+
+
+def _chunks_for(n, pages, url):
+    title = TITLES.get(n, f"ENF {n}")
+    head = "\n".join(pages[:3])
+    dates = _DATE.findall(head)
+    updated = max(dates) if dates else ""
+    text = "\n".join(pages)
+    lines = text.split("\n")
+    # Segment on the manual's own numbered headings when enough are found;
+    # PDF extraction is uneven, so fall back to plain size-splitting.
+    marks = [(i, m.group(1), _norm(m.group(2)))
+             for i, l in enumerate(lines) if (m := _HEADING.match(l.strip()))]
+    segments = []
+    if len(marks) >= 6:
+        for j, (i, num, heading) in enumerate(marks):
+            end = marks[j + 1][0] if j + 1 < len(marks) else len(lines)
+            body = _norm_block(lines[i:end])
+            if len(body) > 120:
+                segments.append((f"{num}", heading, body))
+    if not segments:
+        for k, piece in enumerate(split_lines(text, 1800), start=1):
+            body = _norm_block(piece.split("\n"))
+            if len(body) > 120:
+                segments.append((str(k), f"{title} (part {k})", body))
+    out = []
+    for num, heading, body in segments:
+        pieces = split_lines(body, 1800) if len(body) > 2400 else [body]
+        total = len(pieces)
+        for k, piece in enumerate(pieces, start=1):
+            suffix = "" if total == 1 else f"-p{k}"
+            note = "" if total == 1 else f" (part {k} of {total})"
+            out.append({
+                "id": f"enf-{n}-{num.replace('.', '-')}{suffix}",
+                "doc_type": "memorandum",
+                "act_code": f"ENF-{n}",
+                "act_short": f"ENF {n}",
+                "act_name": f"ENF {n} — {title}",
+                "section": f"ENF {n} s. {num}",
+                "marginal_note": heading,
+                "part": title,
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": updated,
+                "current_to": updated,
+                "citation": f"ENF {n}, s. {num} ({heading}){note}",
+                "source_url": url,
+            })
+    return out
+
+
+def _norm_block(lines):
+    return "\n".join(p for p in (_norm(l) for l in lines) if p)
+
+
+_HTTP_404 = re.compile(r"\b404\b")
+
+
+def is_missing_upstream(exc):
+    """True when a fetch failed because the chapter simply is not there (404).
+
+    _CHAPTERS is a blind range(1, 41) probe, so on a healthy run most numbers
+    404 -- and a chapter IRCC RETIRES starts 404ing too. That is this module's
+    documented signal to drop a chapter, so a 404 must not be preserved
+    through: doing so would pin every withdrawn chapter in the corpus forever.
+    Every other fetch error (TLS, timeout, a dropped connection) is a real
+    failure the stored copy should survive. The two fetch paths report
+    differently -- urllib raises HTTPError with a status, the PowerShell
+    fallback raises CalledProcessError with the status only in its stderr
+    prose ("The remote server returned an error: (404) Not Found."), so that
+    side is a text match.
+    """
+    if isinstance(exc, urllib.error.HTTPError):
+        return exc.code == 404
+    stderr = getattr(exc, "stderr", "") or ""
+    if isinstance(stderr, bytes):
+        stderr = stderr.decode("utf-8", "replace")
+    return bool(_HTTP_404.search(stderr))
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of chapters this run could not produce.
+
+    A chapter that stops fetching for any reason OTHER than a 404 -- a dropped
+    network, a TLS refusal, a PDF that suddenly has no text layer -- has not
+    been retired, it just did not arrive, and losing guidance that is still in
+    force is the worse error. So the stored copy stays and the run says so
+    loudly enough to be investigated. Chapters that 404 (retired, or never
+    existed) never enter `failures` and are not resurrected.
+    Returns (chunks, preserved act_codes).
+    """
+    failed = {f"ENF-{n}" for n, _why in failures}
+    return preserve_dropped(chunks, stored, key=lambda c: c["act_code"],
+                            only=failed)
+
+
+def build(chapters=_CHAPTERS, allow_shrink=False):
+    # Returns the chunks written, or False if the write was refused.
+    all_chunks, skipped, failures = [], [], []
+    for n in chapters:
+        unofficial = UNOFFICIAL.get(n)
+        url = (unofficial["url"] if unofficial
+               else _URL_OVERRIDES.get(n, BASE.format(n=n)))
+        dest = RAW / f"enf{n:02d}.pdf"
+        try:
+            # The ATIP host is an ordinary WordPress site; canada.ca needs
+            # the PowerShell TLS workaround.
+            fetch_cached(url, dest, powershell="canada.ca" in url, pause=1.0)
+        except Exception as exc:
+            skipped.append(n)
+            if not is_missing_upstream(exc):
+                # Not a 404: the chapter may well still be published, so it
+                # counts as a failure and its stored chunks are preserved.
+                failures.append((n, f"fetch failed: {type(exc).__name__}: {exc}"))
+            if dest.exists():
+                dest.unlink()      # don't cache an error page as a PDF
+            continue
+        try:
+            pages = _pdf_text(dest)
+            if sum(len(p) for p in pages) < 500:
+                # Image-scan PDF: fall back to the OCR cache (form-feed
+                # separated pages), produced offline via Windows OCR.
+                ocr = RAW / f"enf{n:02d}-ocr.txt"
+                if ocr.exists():
+                    pages = ocr.read_text(encoding="utf-8").split("\f")
+                    print(f"  ENF {n:2d}: using OCR cache "
+                          f"({sum(len(p) for p in pages)} chars)")
+                else:
+                    print(f"  !! ENF {n}: no text layer and no OCR cache "
+                          f"({ocr.name}); skipping")
+                    failures.append((n, f"no text layer and no {ocr.name}"))
+                    continue
+            chunks = _chunks_for(n, pages, url)
+            if not chunks:
+                # Fetched fine, parsed to nothing: the PDF's heading structure
+                # changed under us. That is the shape the D-memo loss took, so
+                # treat it as a failure and keep the stored chapter.
+                print(f"  !! ENF {n}: no chunks parsed; keeping stored copy")
+                failures.append((n, "no content parsed"))
+                continue
+        except Exception as exc:
+            print(f"  !! ENF {n}: parse failed: {type(exc).__name__}: {exc}")
+            failures.append((n, f"parse failed: {type(exc).__name__}: {exc}"))
+            continue
+        if unofficial:
+            for c in chunks:
+                c["act_name"] = f"ENF {n} — {unofficial['title']}"
+                c["text"] = _UNOFFICIAL_BANNER + c["text"]
+                c["citation"] += " (unofficial ATIP copy)"
+                c["history"] = unofficial["note"]
+        print(f"  ENF {n:2d}: {len(chunks)} chunks"
+              + (" [UNOFFICIAL ATIP]" if unofficial else ""))
+        all_chunks.extend(chunks)
+    uniquify_ids(all_chunks)
+    out = PROCESSED_DIR / "enf.json"
+    # Preserved chunks keep the ids they were stored with, so they are added
+    # after uniquify_ids -- ids embed the chapter number, and a preserved
+    # chapter is by definition absent from this run's chunks.
+    all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                            stored_chunks(out))
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} chapter(s) "
+              f"that failed this run: {', '.join(preserved)}")
+    if not write_corpus(out, all_chunks, indent=2, allow_shrink=allow_shrink):
+        return False
+    print(f"enf: {len(all_chunks)} chunks -> {out.name} "
+          f"(chapters absent upstream: {skipped})")
+    return all_chunks
+
+
+def main():
+    chunks = build(allow_shrink="--allow-shrink" in sys.argv)
+    sys.exit(1 if chunks is False else 0)
+
+
+if __name__ == "__main__":
+    main()

canlex/eval.py

@@ -0,0 +1,171 @@
+"""Measure CanLex retrieval quality against a curated question set.
+
+Each item in data/eval/questions.json pairs a realistic legal question with the
+provision(s) or case(s) that answer it. This runs every question through the
+retrieval index and reports Hit@k and MRR. Re-run it after any retrieval change
+-- a new reranker, different embeddings, a chunking tweak -- to see whether
+quality moved, and read the "Misses" list to see exactly what to fix.
+
+Question schema (per item):
+    query      the question text
+    answers    [[act_short, section], ...]; empty section matches any chunk
+    style      optional tag ('standard' when absent): 'fact-pattern',
+               'keyword', 'typo', 'french', 'no-answer', ... -- metrics are
+               reported per style so a hard new slice cannot silently drag
+               the comparable legacy numbers
+    holdout    optional bool; holdout questions are excluded from parameter
+               sweeps (sweep.py) so knob choices cannot fit their noise --
+               eval.py reports them separately
+    'no-answer' style items carry no answers: they exist to calibrate the
+    confidence signal (a good corpus-coverage score should be LOW on them),
+    reported as a separate table, never as misses.
+
+    py -m canlex.eval
+"""
+import json
+import os
+import sys
+from collections import defaultdict
+
+from .config import ROOT
+from .index import LegislationIndex
+
+QUESTIONS = ROOT / "data" / "eval" / "questions.json"
+EVAL_TOP_K = 20      # search depth, so ranks past the usual 6 are still visible
+
+# The historical baseline slice: styles that existed before the 2026-07
+# expansion. Numbers over this slice stay comparable across commits.
+_LEGACY_STYLES = {"standard"}
+
+
+def load_questions():
+    """The question set with defaults applied; None if the file is absent."""
+    if not QUESTIONS.exists():
+        return None
+    items = json.loads(QUESTIONS.read_text(encoding="utf-8"))
+    for item in items:
+        item.setdefault("style", "standard")
+        item.setdefault("holdout", False)
+    return items
+
+
+def sweep_questions():
+    """The slice parameter sweeps may see (sweep.py calls this): everything
+    except holdout items and the no-answer calibration set."""
+    items = load_questions() or []
+    return [i for i in items
+            if not i["holdout"] and i["style"] != "no-answer"]
+
+
+def _matches(result, answers):
+    """True if a search result is one of the gold answers (act + section).
+
+    A gold answer is [act, section]; an empty section matches any chunk of that
+    act/case (used for case-law answers, whose chunks carry no section number).
+    """
+    r_acts = {result.get("act_short", "").lower(),
+              result.get("act_code", "").lower()}
+    r_sec = result.get("section", "")
+    for act, section in answers:
+        if act.lower() in r_acts and (section == r_sec or section == ""):
+            return True
+    return False
+
+
+def _metrics(ranks):
+    n = len(ranks) or 1
+    hit = lambda k: sum(1 for r in ranks if 0 < r <= k) / n
+    mrr = sum(1.0 / r for r in ranks if r) / n
+    return hit(1), hit(3), hit(5), hit(10), mrr
+
+
+def _report_line(label, ranks):
+    h1, h3, h5, h10, mrr = _metrics(ranks)
+    print(f"  {label:24} n={len(ranks):<4} Hit@1 {h1:.2f}  Hit@3 {h3:.2f}  "
+          f"Hit@5 {h5:.2f}  Hit@10 {h10:.2f}  MRR {mrr:.2f}")
+
+
+def run():
+    items = load_questions()
+    if items is None:
+        print(f"No question set at {QUESTIONS}.", file=sys.stderr)
+        return
+    # Sweep mode (CANLEX_EVAL_SWEEP=1, set by sweep.py): score only the
+    # sweep-visible slice and print the legacy vertical metric block its
+    # parser expects -- holdout questions stay blind to every knob choice.
+    sweep_mode = bool(os.environ.get("CANLEX_EVAL_SWEEP"))
+    if sweep_mode:
+        items = [i for i in items
+                 if not i["holdout"] and i["style"] != "no-answer"]
+    index = LegislationIndex()
+    ranks = []          # (item, rank); rank 0 = miss
+    misses = []
+    no_answer = []      # (query, top confidence) for the calibration table
+    for item in items:
+        if item["style"] == "no-answer":
+            results = index.search(item["query"], top_k=3)
+            conf = results[0].get("confidence") if results else None
+            no_answer.append((item["query"], conf))
+            continue
+        answers = [tuple(a) for a in item["answers"]]
+        results = index.search(item["query"], top_k=EVAL_TOP_K)
+        rank = 0
+        for i, result in enumerate(results, start=1):
+            if _matches(result, answers):
+                rank = i
+                break
+        ranks.append((item, rank))
+        if rank == 0 or rank > 5:
+            top = results[0] if results else None
+            misses.append((item, answers, rank, top))
+
+    all_ranks = [r for _, r in ranks]
+    if sweep_mode:
+        h1, h3, h5, h10, mrr = _metrics(all_ranks)
+        print(f"CanLex retrieval evaluation -- {len(all_ranks)} questions "
+              f"(sweep slice)\n")
+        print(f"  Hit@1:   {h1:.2f}")
+        print(f"  Hit@3:   {h3:.2f}")
+        print(f"  Hit@5:   {h5:.2f}")
+        print(f"  Hit@10:  {h10:.2f}")
+        print(f"  MRR:     {mrr:.2f}")
+        return
+    legacy = [r for it, r in ranks if it["style"] in _LEGACY_STYLES]
+    held = [r for it, r in ranks if it["holdout"]]
+    swept = [r for it, r in ranks if not it["holdout"]]
+    by_style = defaultdict(list)
+    for it, r in ranks:
+        by_style[it["style"]].append(r)
+
+    print(f"CanLex retrieval evaluation -- {len(all_ranks)} scored questions\n")
+    _report_line("ALL", all_ranks)
+    _report_line("legacy (comparable)", legacy)
+    for style in sorted(by_style):
+        if style not in _LEGACY_STYLES:
+            _report_line(f"style: {style}", by_style[style])
+    if held:
+        _report_line("holdout (sweep-blind)", held)
+        _report_line("sweep-visible", swept)
+
+    if no_answer:
+        print("\nNo-answer calibration (confidence should be LOW here; the "
+              "weak-match hedge fires below 0.72):")
+        for query, conf in no_answer:
+            shown = f"{conf:.3f}" if conf is not None else "n/a"
+            flag = "" if (conf is None or conf < 0.72) else "  <-- OVER-CONFIDENT"
+            print(f"  {shown}  {query[:70]}{flag}")
+
+    if misses:
+        print(f"\n{len(misses)} miss(es) -- gold answer ranked >5 or absent:")
+        for item, answers, rank, top in misses:
+            gold = ", ".join(f"{a} s.{s}".rstrip(" s.") for a, s in answers)
+            where = f"ranked #{rank}" if rank else f"absent (searched {EVAL_TOP_K})"
+            got = (f"{top.get('act_short', '')} s.{top.get('section', '')}".rstrip(" s.")
+                   if top else "nothing")
+            print(f"  [{where}] ({item['style']}) {item['query']}")
+            print(f"      gold: {gold}   |   top result: {got}")
+    print()
+
+
+if __name__ == "__main__":
+    run()

canlex/index.py

@@ -0,0 +1,1079 @@
+"""Hybrid retrieval (BM25 + semantic) with cross-encoder reranking."""
+import json
+import math
+import os
+import re
+import sys
+import unicodedata
+from collections import Counter, defaultdict
+
+import snowballstemmer
+
+from .config import PROCESSED_DIR
+from .synonyms import expand_query
+
+K1 = 1.5
+B = 0.75
+RRF_K = 60          # reciprocal-rank-fusion damping constant
+W_SEM = 2.0         # weight on the semantic retriever in the fusion (1.0 = equal; eval-tuned)
+CANDIDATES = 80     # hits each retriever contributes to the fusion
+RERANK_POOL = int(os.environ.get("CANLEX_RERANK_POOL", "16"))
+                    # top fused candidates the cross-encoder rescores. Profiled
+                    # at ~96% of query latency (the pool is cross-encoded on
+                    # CPU), so this is THE latency knob; env-sweepable, and any
+                    # change must be eval-gated (a smaller pool limits which
+                    # low-fusion candidates the promote-only rerank can lift).
+                    # Swept 2026-06: 50 -> 16 (with doc cap 3000 -> 1000) held
+                    # the 159-Q eval exactly (MRR +0.01) at ~5.5x lower latency.
+MN_WEIGHT = float(os.environ.get("CANLEX_MN_WEIGHT", "0.0024"))
+                    # title-match boost per unit of idf-weighted overlap between
+                    # the query and a candidate's marginal note (section title)
+MN_CAP = float(os.environ.get("CANLEX_MN_CAP", "0.012"))
+                    # ceiling on the title-match boost -- it nudges the ranking
+                    # without overriding a strong base score
+REG_PENALTY = float(os.environ.get("CANLEX_REG_PENALTY", "0.004"))
+                    # small fusion penalty on regulation sections, so the Act
+                    # that creates a duty outranks the regulation elaborating it
+                    # (sweep-tuned 2026-05-23 from 0.008 -> 0.004; see sweep.log)
+NIF_PENALTY = float(os.environ.get("CANLEX_NIF_PENALTY", "0.012"))
+                    # fusion penalty on not-in-force amendment chunks, so
+                    # pending law never outranks the live provision it amends
+REPEALED_PENALTY = float(os.environ.get("CANLEX_REPEALED_PENALTY", "0.008"))
+                    # likewise for repealed provisions (kept so lookups still
+                    # inform): a repealed section's on-topic title must not
+                    # beat live law in a close contest -- the CDSA-to-Cannabis
+                    # -Act migration is the live risk zone
+BACKMATTER_PENALTY = float(os.environ.get("CANLEX_BACKMATTER_PENALTY", "0.004"))
+                    # likewise for a collective agreement's back-matter
+                    # (memoranda, letters of understanding) vs its numbered articles
+                    # (sweep-tuned 2026-05-23 from 0.008 -> 0.004)
+SOURCE_CAP = 2      # max chunks one case or memorandum may contribute
+RECALL_CAP = int(os.environ.get("CANLEX_RECALL_CAP", "20"))
+                    # max chunks one capped source (anything _source_key keys:
+                    # a decision, a memorandum) may contribute to EACH
+                    # retriever's CANDIDATES slice at the recall stage. Without
+                    # it, a deep decision whose every paragraph matches a
+                    # specific query (Canfield: 94 paragraphs on device border
+                    # searches) fills the entire fusion pool, so the result-
+                    # stage SOURCE_CAP defers its surplus with nothing left to
+                    # promote and _ensure_primary finds no statute to pull in.
+                    # Generous by design: SOURCE_CAP=2 governs display; this
+                    # only has to leave room for other sources to survive
+                    # recall. Uncapped sources (_source_key None) are unlimited.
+                    # Swept 2026-07-10 over {12,16,20,24} on the 159-Q eval:
+                    # 12 and 20 tie at 0.80/0.94/0.97/0.99/0.87 (4 misses),
+                    # both above the uncapped baseline; 20 chosen as the
+                    # lightest-touch winner (24 regresses Hit@3/@5).
+APPENDIX_CAP = 3    # max referenced appendices co-surfaced into a result set
+
+# Primary instruments -- enacted law, collective agreements, the NJC directives
+# incorporated into them, and the IRPA delegation instruments. Their sections or
+# items are distinct provisions, so (like legislation) they are never collapsed
+# under the diversity cap.
+PRIMARY_DOC_TYPES = frozenset({"legislation", "agreement", "directive", "delegation"})
+# Memorandum families whose act_code is ONE shared constant over many
+# independent items, so the diversity cap keys on `section` instead (see
+# _source_key): a D-memo number, an AMPS contravention code.
+_FLAT_MEMO_CODES = frozenset({"D-Memo", "AMPS"})
+
+_TOKEN = re.compile(r"[a-z0-9]+")
+# A chunk id ending '-p<k>' marks one embeddable piece of a long section that
+# ingest.py split (see _split_text); same-section pieces are capped together.
+_PIECE_ID = re.compile(r"-p\d+$")
+_SECTION_REF = re.compile(r"\bs(?:ec(?:tion)?)?s?\.?\s*(\d+(?:\.\d+)?)")
+# A cross-reference to another provision -- "section 34", "subsection 25(1)",
+# "paragraph 36(1)(a)", "s. 34" -- capturing the top-level section number.
+_XREF = re.compile(
+    r"\b(?:sections?|subsections?|paragraphs?|ss?\.)\s*(\d+(?:\.\d+)?)",
+    re.IGNORECASE)
+
+# A line opening with "(1)", "(a)" or "(b.1)" -- a citable subdivision
+# (subsection, paragraph or subparagraph) of a provision.
+_MARKER = re.compile(r"(?m)^\(([0-9a-zA-Z]+(?:\.\d+)?)\)")
+
+# A D-memorandum's reference to a provision -- "section 32 of the Customs Act",
+# or "section 32 of the Act" (the Act a D-memo administers -- the Customs Act).
+_MEMO_CITE = re.compile(
+    r"\b(?:sub)?sections?\s+(\d+(?:\.\d+)?)(?:\([^)]+\))*\s+of\s+the\s+"
+    r"(Customs Act|Customs Tariff|Act)\b", re.IGNORECASE)
+
+# A directive section's reference to an appendix of the same directive --
+# "as specified in Appendix C". A trailing "of" ("Appendix C of the NJC Travel
+# Directive") marks a cross-directive citation and is deliberately left alone.
+_APPENDIX_REF = re.compile(r"\bAppendi(?:x|ces)\s+([A-Za-z])\b(?!\s+of\b)",
+                           re.IGNORECASE)
+# A directive chunk that *is* an appendix: its title opens "Appendix C ...".
+_APPENDIX_HEAD = re.compile(r"Appendix\s+([A-Za-z])\b", re.IGNORECASE)
+
+
+_STEMMER = snowballstemmer.stemmer("english")
+_STEM_CACHE = {}
+
+
+# Stem pairs Snowball does not merge but that share a legal meaning, so a
+# query naming the verb still matches a provision titled with the noun (or
+# vice versa). Mapped to the verb form on both index and query sides, which
+# is consistent and arbitrary -- the merge is what matters.
+_STEM_NORMALIZE = {
+    "seizur": "seiz",            # seizure -> seize
+    "forfeitur": "forfeit",      # forfeiture -> forfeit
+    "appel": "appeal",           # appellate/appellant -> appeal
+    "detent": "detain",          # detention -> detain
+    "exclus": "exclud",          # exclusion -> exclude
+    "admiss": "admit",           # admission/admissibility -> admit
+    "applic": "appli",           # application -> apply
+    "complianc": "compli",       # compliance -> comply
+    "grievanc": "griev",         # grievance -> grieve
+}
+
+
+def _stem(word):
+    """Snowball-stem a word, memoised -- legal text repeats terms heavily.
+    A small post-stem normalization merges a few verb/noun pairs Snowball
+    leaves apart ('seize'/'seizure', 'forfeit'/'forfeiture')."""
+    stemmed = _STEM_CACHE.get(word)
+    if stemmed is None:
+        stemmed = _STEMMER.stemWord(word)
+        stemmed = _STEM_NORMALIZE.get(stemmed, stemmed)
+        _STEM_CACHE[word] = stemmed
+    return stemmed
+
+
+# Query-language detection for the bilingual corpus: accented characters or
+# two-plus French function words mark a French query. French queries may draw
+# on BOTH languages (guidance and case law exist only in English); English
+# queries never surface French chunks (the English corpus is complete, and a
+# French twin would only duplicate results).
+_FR_HINT = re.compile(r"[àâçéèêëîïôùûüÿœæ]", re.IGNORECASE)
+# ASCII-only entries: tokens are matched pre-folding with [a-zA-Z']+, and any
+# accented query already short-circuits through _FR_HINT above.
+_FR_WORDS = frozenset(
+    "le la les de des une du et est dans pour que qui aux au sur par ne pas "
+    "peut doit selon lors quelle quel cette ces son ses leur d'un d'une "
+    "interdiction fouille saisie renvoi territoire".split())
+
+
+def query_lang(query):
+    """'fr' when the query reads as French, else 'en'."""
+    if _FR_HINT.search(query):
+        return "fr"
+    tokens = re.findall(r"[a-zA-Z']+", query.lower())
+    return "fr" if sum(1 for t in tokens if t in _FR_WORDS) >= 2 else "en"
+
+
+def _fold_accents(text):
+    """ASCII-fold accented characters so a French or mixed query tokenizes
+    usefully instead of shattering: the [a-z0-9] token pattern used to split
+    'détention' into ['d','tention']. NFKD + ASCII-encode drops the combining
+    marks ('détention' -> 'detention'), which both matches the English corpus
+    directly and lets the FR->EN synonym bridges fire."""
+    return unicodedata.normalize("NFKD", text).encode(
+        "ascii", "ignore").decode("ascii")
+
+
+def tokenize(text):
+    """Lower-case, accent-fold, split on word characters, and Snowball-stem
+    each token, so a query matches a provision even when their word forms
+    differ -- 'possession' vs 'possess', 'reporting' vs 'report',
+    'importation' vs 'import', 'détention' vs 'detention'."""
+    return [_stem(w) for w in _TOKEN.findall(_fold_accents(text.lower()))]
+
+
+def _section_refs(query):
+    """Pull explicit section numbers from a query, e.g. 'section 34', 's. 20.1'."""
+    return set(_SECTION_REF.findall(query.lower()))
+
+
+def topical_title(chunk):
+    """Return the chunk's topic-bearing string, used wherever a section's
+    'title' is weighted for retrieval -- BM25 indexing, the title-match boost,
+    and the semantic embedding. Differs by doc_type because the field that
+    carries the topic differs: legislation/agreement/directive/delegation use
+    the marginal_note (section heading); D-memoranda use 'part' because their
+    marginal_note is a generic banner; case-law uses 'heading' because its
+    marginal_note is just the paragraph range ('paras 11-13') and the case
+    proposition lives in heading."""
+    doc_type = chunk.get("doc_type")
+    if doc_type == "memorandum":
+        return chunk.get("part") or chunk["marginal_note"]
+    if doc_type == "caselaw":
+        return chunk.get("heading") or chunk["marginal_note"]
+    return chunk["marginal_note"]
+
+
+def _provision_units(text):
+    """Citable parts of a provision, for pinpoint scoring -- a list of
+    (citation_suffix, scoring_text, snippet). One entry per paragraph, with its
+    subsection chapeau prepended to scoring_text for context, plus one per
+    paragraph-less subsection. Returns [] when the provision is too flat to
+    pinpoint (fewer than two subdivisions)."""
+    marks = list(_MARKER.finditer(text))
+    if len(marks) < 2:
+        return []
+    spans = []
+    for i, m in enumerate(marks):
+        end = marks[i + 1].start() if i + 1 < len(marks) else len(text)
+        spans.append((m.group(1), text[m.start():end].strip()))
+    units, cur_sub, cur_intro = [], "", ""
+    for j, (token, body) in enumerate(spans):
+        if "[Repealed" in body[:40]:
+            if token[0].isdigit():
+                cur_sub, cur_intro = f"({token})", ""
+            continue
+        if token[0].isdigit():
+            cur_sub, cur_intro = f"({token})", body
+            nxt = spans[j + 1][0] if j + 1 < len(spans) else ""
+            if not nxt or nxt[0].isdigit():
+                units.append((cur_sub, body, body))   # subsection has no paragraphs
+            # otherwise the chapeau is emitted via its paragraphs below
+        else:
+            label = f"{cur_sub}({token})" if cur_sub else f"({token})"
+            units.append((label, f"{cur_intro} {body}".strip(), body))
+    return units
+
+
+class LegislationIndex:
+    def __init__(self):
+        self.chunks = []
+        for path in sorted(PROCESSED_DIR.glob("*.json")):
+            self.chunks.extend(json.loads(path.read_text(encoding="utf-8")))
+        if not self.chunks:
+            raise RuntimeError(
+                f"No processed legislation in {PROCESSED_DIR}. Run 'canlex.ingest' first.")
+        self._tri_index = None       # lazy; see _fuzzy_term
+        self._word_vocab = None      # lazy; see _correct_word (typo repair)
+        self._word_tri = None
+        # Cached recall mask for the common case (English query, no
+        # filters): everything but the French twins.
+        self._en_mask = [c.get("lang", "en") != "fr" for c in self.chunks]
+        self._build_bm25()
+        self._build_note_tokens()
+        self._build_xref()
+        self._build_appendix_index()
+        self._load_semantic()
+        self._load_reranker()
+
+    def _build_bm25(self):
+        """BM25 postings plus TWO sets of collection statistics: one over the
+        whole corpus, one over the English scope alone.
+
+        An English query never retrieves the French twins (see _en_mask), yet
+        a single corpus-wide idf/avgdl let those unreachable chunks set the
+        rarity of every shared token (numbers, act codes, proper nouns, folded
+        cognates) and drag the average document length. Ingesting the French
+        D-memos and NJC directives moved legacy Hit@1 0.767 -> 0.750 that way,
+        without a single French chunk ever being retrievable -- the statistics
+        must be scoped to what the query can actually reach.
+        """
+        self.doc_len = []
+        self.postings = defaultdict(list)  # term -> [(doc_idx, term_frequency), ...]
+        df = defaultdict(int)
+        df_en = defaultdict(int)
+        en_docs = en_total_len = 0
+        for idx, c in enumerate(self.chunks):
+            # The topical title is repeated to weight it above body text;
+            # the Act name, code and section are indexed too, so an Act's own
+            # terminology (e.g. "controlled substance") and its codes/numbers
+            # are searchable even when a section's text omits them. The title
+            # is doc_type-aware via topical_title -- for case-law it picks
+            # the case proposition (heading), not the paragraph range
+            # (marginal_note), so a leading case surfaces on a topical query.
+            title = topical_title(c)
+            blob = " ".join((title, title, c["heading"],
+                             c["part"], c["division"], c["act_name"], c["act_code"],
+                             c["section"], c["text"]))
+            counts = Counter(tokenize(blob))
+            self.doc_len.append(sum(counts.values()))
+            is_en = c.get("lang", "en") != "fr"
+            if is_en:
+                en_docs += 1
+                en_total_len += self.doc_len[-1]
+            for term, tf in counts.items():
+                self.postings[term].append((idx, tf))
+                df[term] += 1
+                if is_en:
+                    df_en[term] += 1
+        n = len(self.chunks)
+        self.avgdl = sum(self.doc_len) / n
+        self.idf = {t: math.log(1 + (n - d + 0.5) / (d + 0.5)) for t, d in df.items()}
+        # The English-scoped twins. A term the English corpus never uses is
+        # absent from idf_en rather than zero-weighted: it can only match
+        # chunks the mask already excludes.
+        self.avgdl_en = en_total_len / (en_docs or 1)
+        self.idf_en = {t: math.log(1 + (en_docs - d + 0.5) / (d + 0.5))
+                       for t, d in df_en.items()}
+
+    def _build_note_tokens(self):
+        """Pre-tokenise each chunk's topical title (see topical_title) for the
+        title-match boost in search(). Each chunk is also flagged as a
+        regulation (act codes beginning SOR/C.R.C.) for the Act-over-regulation
+        preference, as collective-agreement back-matter (memoranda and letters
+        with no article number) for the back-matter penalty, and as a
+        not-in-force amendment for the pending-law penalty."""
+        self._note_tokens = []
+        self._is_regulation = []
+        self._is_backmatter = []
+        self._is_nif = []
+        self._is_repealed = []
+        for c in self.chunks:
+            self._note_tokens.append(set(tokenize(topical_title(c))))
+            self._is_regulation.append(
+                c.get("doc_type", "legislation") == "legislation"
+                and c["act_code"].startswith(("SOR", "C.R.C")))
+            self._is_backmatter.append(
+                c.get("doc_type") == "agreement"
+                and not str(c["section"])[:1].isdigit())
+            self._is_nif.append(c.get("status") == "not-in-force")
+            self._is_repealed.append(c.get("status") == "repealed")
+
+    def _build_appendix_index(self):
+        """Index directive appendices by (act_code, letter), so a directive
+        section that cites 'Appendix C' can pull that appendix into the result
+        set -- a directive's rate-table appendices are bare numbers and rank
+        poorly on a natural-language query, yet the section citing them is of
+        little use without them."""
+        self._appendix = defaultdict(list)
+        for idx, c in enumerate(self.chunks):
+            if c.get("doc_type") != "directive":
+                continue
+            m = _APPENDIX_HEAD.match(c["marginal_note"])
+            if m:
+                self._appendix[(c["act_code"], m.group(1).upper())].append(idx)
+
+    def _load_semantic(self):
+        """Load precomputed embeddings and the query embedder.
+
+        Any failure (missing embeddings, or numpy/model unavailable) degrades the
+        index to BM25-only rather than breaking retrieval.
+        """
+        self.semantic = False
+        emb_path = PROCESSED_DIR / "embeddings.npz"
+        if not emb_path.exists():
+            print("CanLex index: no embeddings.npz; using BM25 only "
+                  "(run 'canlex.embed' to enable semantic search).", file=sys.stderr)
+            return
+        try:
+            import numpy as np
+            from .embed import Embedder
+            # Multi-vector layout: rows share a chunk id when the chunk was
+            # embedded as several windows (long D-memo sections etc.); a
+            # chunk's semantic score is the max over its windows. Rows whose
+            # id matches no loaded chunk are dropped; chunks with no rows
+            # count as missing (BM25-only for them).
+            chunk_ordinal = {c["id"]: i for i, c in enumerate(self.chunks)}
+            with np.load(emb_path) as data:
+                all_ids = data["ids"].tolist()
+                all_vecs = data["vectors"]
+            keep = [(chunk_ordinal[i], r) for r, i in enumerate(all_ids)
+                    if i in chunk_ordinal]
+            self._np = np
+            self.vectors = all_vecs[[r for _, r in keep]]
+            self._row_chunk = np.array([o for o, _ in keep], dtype=np.int64)
+            covered = set(self._row_chunk.tolist())
+            missing = len(self.chunks) - len(covered)
+            self.embedder = Embedder()
+            self.semantic = True
+            if missing:
+                print(f"CanLex index: {missing}/{len(self.chunks)} sections lack "
+                      f"embeddings; re-run 'canlex.embed' to refresh.", file=sys.stderr)
+        except Exception as exc:
+            print(f"CanLex index: semantic search disabled ({type(exc).__name__}: "
+                  f"{exc}); using BM25 only.", file=sys.stderr)
+            self.semantic = False
+
+    def _load_reranker(self):
+        """Load the cross-encoder reranker; degrade to the fusion order on failure."""
+        self.reranker = None
+        try:
+            from .rerank import Reranker
+            self.reranker = Reranker()
+        except Exception as exc:
+            print(f"CanLex index: reranker disabled ({type(exc).__name__}: {exc}); "
+                  f"using hybrid fusion order.", file=sys.stderr)
+
+    def _fuzzy_term(self, term):
+        """The closest vocabulary term for a token the corpus has never seen
+        -- a typo bridge ('detension' -> 'detention'), scored by character-
+        trigram Dice similarity with a strict 0.6 floor and a length guard.
+        Only ever consulted for UNKNOWN tokens, so correctly-spelled queries
+        are untouched. The trigram index is built lazily on first miss."""
+        if len(term) < 4:
+            return None
+        if self._tri_index is None:
+            tri = defaultdict(set)
+            for vocab_term in self.idf:
+                for i in range(len(vocab_term) - 2):
+                    tri[vocab_term[i:i + 3]].add(vocab_term)
+            self._tri_index = tri
+        tris = {term[i:i + 3] for i in range(len(term) - 2)}
+        counts = Counter()
+        for t in tris:
+            counts.update(self._tri_index.get(t, ()))
+        best, best_score = None, 0.6
+        for cand, n in counts.most_common(50):
+            # Typos rarely corrupt a word's opening letters; requiring a
+            # shared 2-char prefix kills drift like 'detens' -> 'pretens'
+            # while keeping 'seisur' -> 'seizur'.
+            if abs(len(cand) - len(term)) > 2 or cand[:2] != term[:2]:
+                continue
+            score = 2 * n / (len(tris) + len(cand) - 2)
+            if score > best_score:
+                best, best_score = cand, score
+        return best
+
+    @staticmethod
+    def _edit_distance(a, b, cap=3):
+        """Damerau-Levenshtein distance, early-exited past `cap` -- typos
+        are substitutions, insertions, deletions and transpositions, and a
+        transposition ('undeclraed') must cost 1, not 2."""
+        if abs(len(a) - len(b)) > cap:
+            return cap + 1
+        prev2, prev = None, list(range(len(b) + 1))
+        for i, ca in enumerate(a, start=1):
+            cur = [i]
+            best_row = i
+            for j, cb in enumerate(b, start=1):
+                cost = 0 if ca == cb else 1
+                d = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + cost)
+                if (prev2 is not None and i > 1 and j > 1
+                        and ca == b[j - 2] and a[i - 2] == cb):
+                    d = min(d, prev2[j - 2] + 1)
+                cur.append(d)
+                best_row = min(best_row, d)
+            if best_row > cap:
+                return cap + 1
+            prev2, prev = prev, cur
+        return prev[-1]
+
+    def _correct_word(self, word):
+        """Nearest corpus WORD (not stem) for a misspelling, or None.
+
+        Trigram index generates candidates; Damerau-Levenshtein picks the
+        winner (distance <= 2, shared 2-char prefix, frequency floor of 5
+        so OCR debris in the corpus can never be a 'correction'). The
+        result is real text: it feeds the semantic embedder and the
+        cross-encoder, which score words, not stems."""
+        if self._word_vocab is None:
+            words = Counter()
+            for c in self.chunks:
+                words.update(re.findall(
+                    r"[a-z]{4,}", _fold_accents(c["text"].lower())))
+            tri = defaultdict(set)
+            for w, n in words.items():
+                if n >= 5:                      # junk never enters the index
+                    for i in range(len(w) - 2):
+                        tri[w[i:i + 3]].add(w)
+            self._word_vocab, self._word_tri = words, tri
+        tris = {word[i:i + 3] for i in range(len(word) - 2)}
+        candidates = set()
+        for t in tris:
+            candidates |= self._word_tri.get(t, set())
+        best, best_key = None, (3, 0)           # (distance, -frequency)
+        for cand in candidates:
+            if abs(len(cand) - len(word)) > 2 or cand[:2] != word[:2]:
+                continue
+            dist = self._edit_distance(word, cand, cap=2)
+            if dist > 2:
+                continue
+            key = (dist, -self._word_vocab[cand])
+            if key < best_key:
+                best, best_key = cand, key
+        return best
+
+    def _correct_query(self, query):
+        """The query with misspelled words replaced by their nearest corpus
+        words; the query itself when nothing needed fixing. A word is only
+        'misspelled' when its stem is absent from the whole BM25 vocabulary,
+        so correctly-spelled queries pass through untouched by
+        construction."""
+        out, changed = [], False
+        for tok in query.split():
+            word = _fold_accents(tok.lower()).strip(".,;:!?()'\"")
+            if len(word) >= 4 and word.isalpha():
+                stem = _stem(word)
+                df = len(self.postings.get(stem, ()))
+                if df == 0:
+                    # Unknown everywhere: replace outright.
+                    fix = self._correct_word(word)
+                    if fix and fix != word:
+                        out.append(fix)
+                        changed = True
+                        continue
+                elif df <= 2:
+                    # Nearly unknown -- often a typo that happens to collide
+                    # with OCR debris in the corpus. APPEND a strong
+                    # correction rather than replacing, so a legitimately
+                    # rare term (a chemical in a CDSA schedule, a case name)
+                    # never loses its own recall.
+                    fix = self._correct_word(word)
+                    if (fix and fix != word
+                            and self._word_vocab[fix] >= 100):
+                        out.append(f"{tok} {fix}")
+                        changed = True
+                        continue
+            out.append(tok)
+        return " ".join(out) if changed else query
+
+    def _bm25_scores(self, query, allowed=None, en_only=False):
+        """allowed: optional boolean list by chunk index. Filtered searches
+        pass it so recall competes only within scope -- filtering after
+        recall meant a doc_type='caselaw' search drew from whatever case law
+        happened to survive the open-corpus top-N competition.
+
+        en_only: score with the English-scoped collection statistics (see
+        _build_bm25). Set for every English query, whose mask already bars
+        the French twins from the results they would otherwise be weighting.
+        """
+        idf_map = self.idf_en if en_only else self.idf
+        avgdl = self.avgdl_en if en_only else self.avgdl
+        scores = defaultdict(float)
+        for term in set(tokenize(query)):
+            idf = idf_map.get(term)
+            weight = 1.0
+            if idf is None:
+                # Unknown token: try the typo bridge at a discount, so a
+                # misspelled query still reaches the provision while a real
+                # match always outranks a fuzzy one.
+                alt = self._fuzzy_term(term)
+                if alt is None or alt not in idf_map:
+                    continue
+                term, idf, weight = alt, idf_map[alt], 0.7
+            for idx, tf in self.postings[term]:
+                if allowed is not None and not allowed[idx]:
+                    continue
+                dl = self.doc_len[idx]
+                denom = tf + K1 * (1 - B + B * dl / avgdl)
+                scores[idx] += weight * idf * tf * (K1 + 1) / denom
+        return scores
+
+    def _semantic_ranking(self, query, allowed=None):
+        np = self._np
+        qv = self.embedder.encode_query(query)
+        row_sims = self.vectors @ qv
+        # Aggregate windows to chunks by max: a long section scores as its
+        # best-matching window, so a fact buried mid-memo is reachable.
+        sims = np.full(len(self.chunks), -1.0, dtype=np.float32)
+        np.maximum.at(sims, self._row_chunk, row_sims)
+        if allowed is not None:
+            # Mask out-of-scope chunks so a filtered search ranks (and reports
+            # confidence) within its own scope, not corpus-wide.
+            sims = np.where(np.asarray(allowed, dtype=bool), sims, -1.0)
+        # Over-fetch 4x: the recall-stage source cap (_capped_top) drops the
+        # surplus chunks of any one deep source, and the extra depth is what
+        # lets other sources backfill the freed CANDIDATES slots.
+        order = np.argsort(sims)[::-1][:CANDIDATES * 4]
+        # The top cosine similarity doubles as a corpus-coverage signal: a query
+        # the corpus cannot answer has no passage close to it.
+        return [int(i) for i in order], float(sims.max())
+
+    def _rerank_doc(self, idx):
+        c = self.chunks[idx]
+        return f"{c['citation']} — {c['marginal_note']}\n{c['text']}"
+
+    def _source_key(self, idx):
+        """The parent document a chunk belongs to, for diversity capping. Returns
+        None for primary instruments -- legislation, collective agreements and
+        directives -- whose sections are distinct provisions and are never
+        capped; case law is keyed by citation, memoranda by memo number.
+
+        Benefit-plan booklets also return None: a booklet's sections are
+        distinct topics (eligibility, coverage, claims, appeals...) like
+        statute sections, so capping the whole booklet to SOURCE_CAP chunks
+        would starve a detailed benefits query. They are not in
+        PRIMARY_DOC_TYPES, though -- they are guidance, not governing
+        instruments, so _ensure_primary does not pull them in as if they were
+        enacted law.
+
+        Exception: the pieces of one long section/article that was split for
+        embedding (ids ending '-p<k>') share a key, so the diversity cap stops a
+        single multi-part provision from filling the result set with its own
+        fragments. This applies to the otherwise-uncapped primary types
+        (legislation, agreement, directive, delegation): pieces are keyed by
+        their shared base id, distinct sections keep distinct keys. (Memoranda
+        and case law are already capped at the parent level, so their pieces are
+        bounded without a separate key.)"""
+        c = self.chunks[idx]
+        doc_type = c.get("doc_type", "legislation")
+        m = _PIECE_ID.search(c.get("id", ""))
+        if m and doc_type in PRIMARY_DOC_TYPES:
+            return ("piece", c["id"][:m.start()])  # pieces of one chunk together
+        # Commentary entries are like benefits chunks: each is a distinct
+        # curated topic, so capping the dataset as one source would starve a
+        # multi-entry question; not PRIMARY, so never force-pulled as law.
+        if doc_type in PRIMARY_DOC_TYPES or doc_type in ("benefits", "commentary"):
+            return None
+        if doc_type == "memorandum":
+            # The key must be the parent DOCUMENT, and which field carries that
+            # identity differs across the memorandum families. ENF chapters,
+            # PDI pages and the IRB guidelines each have their own act_code and
+            # use `section` to subdivide it, so act_code is the document.
+            # _FLAT_MEMO_CODES are the two families where one act_code covers
+            # many independent items -- ~360 D-memoranda and ~1,000 AMPS
+            # contraventions -- and `section` (memo number / contravention
+            # code) is the unit a reader would cite.
+            #
+            # Keying every memorandum on `section` (as this did) gave each ENF
+            # section and each PDI page its own source key, so neither cap ever
+            # bound at the document level: the ~150-page PDI refugee tree could
+            # fill the whole candidate pool and push IRPA s. 112 out of a PRRA
+            # query entirely.
+            code = c["act_code"]
+            return ("memorandum",
+                    c["section"] if code in _FLAT_MEMO_CODES else code)
+        return (doc_type, c["act_code"])          # one decision, keyed by citation
+
+    def _capped_top(self, ordered):
+        """The first CANDIDATES indices of `ordered`, keeping at most
+        RECALL_CAP per capped source. The recall-stage counterpart of
+        _diversify: where _diversify reorders the fused candidates so a deep
+        source cannot monopolise the visible results, this stops that source
+        from monopolising the candidate pool itself -- skipped surplus frees
+        slots that later, weaker-ranked chunks of OTHER sources backfill."""
+        out, counts = [], defaultdict(int)
+        for idx in ordered:
+            key = self._source_key(idx)
+            if key is not None:
+                counts[key] += 1
+                if counts[key] > RECALL_CAP:
+                    continue
+            out.append(idx)
+            if len(out) >= CANDIDATES:
+                break
+        return out
+
+    def _diversify(self, ordered):
+        """Reorder so no single case, memorandum, agreement or directive can
+        monopolise the results: once a source has contributed SOURCE_CAP chunks,
+        its remaining chunks are deferred below every other candidate. This stops
+        a heavily paragraph-chunked decision from crowding out the statute it
+        interprets. Legislation is never capped."""
+        kept, deferred, counts = [], [], defaultdict(int)
+        for idx in ordered:
+            key = self._source_key(idx)
+            if key is None:
+                kept.append(idx)
+                continue
+            counts[key] += 1
+            (kept if counts[key] <= SOURCE_CAP else deferred).append(idx)
+        return kept + deferred
+
+    def _ensure_primary(self, ordered, top_k, q_tokens):
+        """Guarantee the governing primary instrument is surfaced: when the
+        natural top_k is monopolised by case law or D-memoranda that interpret
+        a statute, pull the most topically-on-target Act/agreement/directive/
+        delegation section into the top_k, displacing the lowest-ranked
+        secondary sources. The single best match is always kept in place.
+
+        Two changes from the older 'ensure_legislation' guarantee: (i) all
+        primary instruments count, not only legislation -- so an FB-Agreement
+        query that surfaces only FPSLREB case law gets the agreement article
+        pulled in too; (ii) the candidate to promote is chosen by title-match
+        against the query (the section whose marginal_note covers the most of
+        the query's distinctive vocabulary), not by raw fusion rank. The
+        fusion rank surfaces tangentially-on-topic sections that share the
+        Act's general vocabulary; the title-match scorer surfaces the section
+        actually about the topic ('Seizure and forfeiture' over 'Report to
+        President' for a 'seize currency' query)."""
+        if top_k < 3:
+            return ordered
+        def is_primary(i):
+            # A not-in-force amendment is pending law, not a governing
+            # provision -- never promote it as "the statute on point".
+            return (self.chunks[i].get("doc_type", "legislation") in PRIMARY_DOC_TYPES
+                    and self.chunks[i].get("status") != "not-in-force")
+        top, rest = ordered[:top_k], ordered[top_k:]
+        need = 2 - sum(1 for i in top if is_primary(i))
+        if need <= 0:
+            return ordered
+        primary_in_rest = [i for i in rest if is_primary(i)]
+        if not primary_in_rest:
+            return ordered
+        if q_tokens:
+            def title_score(idx):
+                note_tokens = self._note_tokens[idx]
+                if not note_tokens:
+                    return 0.0
+                matched = sum(self.idf.get(t, 0.0)
+                              for t in note_tokens if t in q_tokens)
+                total = sum(self.idf.get(t, 0.0) for t in note_tokens) or 1.0
+                score = matched * matched / total
+                # Mirror the fusion-stage hierarchy preferences for tiebreaks:
+                # the governing Act beats its regulation, and numbered
+                # agreement articles beat their back-matter, when both have
+                # identical titles (e.g. IRPA s. 112 and IRPR s. 160 both
+                # marginal-noted 'Application for protection').
+                if self._is_regulation[idx]:
+                    score -= REG_PENALTY
+                if self._is_backmatter[idx]:
+                    score -= BACKMATTER_PENALTY
+                return score
+            # Sort by title-match descending, then by original fusion order as
+            # a tiebreak (stable sort: keep the original rest order).
+            primary_in_rest.sort(key=title_score, reverse=True)
+        promote = primary_in_rest[:need]
+        drop = [i for i in reversed(top) if not is_primary(i)][:len(promote)]
+        if not drop:
+            return ordered
+        promote = promote[:len(drop)]
+        dropped, promoted = set(drop), set(promote)
+        kept = [i for i in top if i not in dropped]
+        return kept[:1] + promote + kept[1:] + drop + [
+            i for i in rest if i not in promoted]
+
+    def _cosurface_appendices(self, top):
+        """Append the appendices the directive results cite but that retrieval
+        missed. A directive's rate tables ('Appendix C') rank poorly on a
+        natural-language query, yet a section that cites them is of little use
+        without them -- so the appendix travels with it. When more appendices
+        are cited than APPENDIX_CAP allows, the ones cited by the most results
+        win, so a lone off-topic result cannot outvote the relevant ones.
+        Returns `top` extended by up to APPENDIX_CAP appendix chunks."""
+        have = set(top)
+        cited = Counter()
+        for idx in top:
+            c = self.chunks[idx]
+            if c.get("doc_type") != "directive":
+                continue
+            seen = set()        # count an appendix once per citing result
+            for m in _APPENDIX_REF.finditer(c["text"]):
+                key = (c["act_code"], m.group(1).upper())
+                for app in self._appendix.get(key, ()):
+                    if app not in have and app not in seen:
+                        seen.add(app)
+                        cited[app] += 1
+        return top + [app for app, _ in cited.most_common(APPENDIX_CAP)]
+
+    def _highlight(self, query, indices):
+        """For each result chunk, the subsection or paragraph most on point for
+        the query: {result_position: (citation_suffix, snippet)}. Uses the
+        cross-encoder; returns {} if it is unavailable or nothing is structured.
+        Only the first results are scored -- a pinpoint deep in the list is not
+        worth the cross-encoder cost."""
+        if not self.reranker:
+            return {}
+        jobs = []   # (result_position, label, scoring_text, snippet)
+        for pos, idx in enumerate(indices[:8]):
+            c = self.chunks[idx]
+            if c.get("doc_type", "legislation") != "legislation":
+                continue
+            note = c["marginal_note"]
+            for label, scoring, snippet in _provision_units(c["text"]):
+                jobs.append((pos, label, f"{note}. {scoring}", snippet))
+        if not jobs:
+            return {}
+        best = {}   # result_position -> (score, label, snippet)
+        for (pos, label, _, snippet), score in zip(
+                jobs, self.reranker.score(query, [j[2] for j in jobs])):
+            if pos not in best or score > best[pos][0]:
+                best[pos] = (score, label, snippet)
+        return {pos: (label, " ".join(snippet[:240].split()))
+                for pos, (score, label, snippet) in best.items()}
+
+    def _filter_ok(self, c, act, doc_type, qlang="en", court=None,
+                   decided_after=None):
+        """One predicate for both recall masking and the late result filter.
+        The act filter matches short name, code, or full name -- an agent
+        passing 'Immigration and Refugee Protection Act' should not get
+        silence. English queries never see French chunks; French queries see
+        both languages (guidance/case law exist only in English). court /
+        decided_after constrain CASE-LAW chunks only (a case-law chunk's
+        'part' carries the deciding court and current_to the decision date);
+        non-caselaw chunks pass those filters untouched so a court-filtered
+        mixed search still surfaces the governing statute."""
+        if qlang == "en" and c.get("lang", "en") == "fr":
+            return False
+        if act:
+            a = act.lower()
+            if a not in (c["act_short"].lower(), c["act_code"].lower(),
+                         c.get("act_name", "").lower()):
+                return False
+        if doc_type and c.get("doc_type", "legislation") != doc_type:
+            return False
+        if (court or decided_after) and c.get("doc_type") == "caselaw":
+            if court and court.lower() not in (c.get("part") or "").lower():
+                return False
+            if decided_after:
+                year = (c.get("current_to") or "")[:4]
+                if not year.isdigit() or int(year) < decided_after:
+                    return False
+        return True
+
+    def search(self, query, top_k=6, act=None, doc_type=None, court=None,
+               decided_after=None):
+        """Hybrid candidate fusion (BM25 + semantic), then cross-encoder rerank."""
+        # Repair misspellings first (no-op for correctly-spelled queries),
+        # then expand legal abbreviations (PRRA, H&C, ...) into statutory
+        # wording for the recall stages. The corrected text is what feeds
+        # the semantic embedder and the cross-encoder -- the stem-level
+        # fuzzy bridge in _bm25_scores only ever helped BM25, which is why
+        # typo queries used to reach the pool but not the top ranks.
+        corrected = self._correct_query(query)
+        expanded = expand_query(corrected)
+        qlang = query_lang(query)
+        confidence = None
+        fused = defaultdict(float)
+        # A filtered search competes only within its scope: the mask reaches
+        # both recall stages, so results come from the best of the whole
+        # filtered corpus rather than whatever survived open recall. An
+        # unfiltered English query uses the cached English-only mask; a
+        # French query searches both languages unmasked.
+        if act or doc_type or court or decided_after:
+            mask = [self._filter_ok(c, act, doc_type, qlang, court,
+                                    decided_after)
+                    for c in self.chunks]
+        elif qlang == "en":
+            mask = self._en_mask
+        else:
+            mask = None
+        bm25 = self._bm25_scores(expanded, allowed=mask,
+                                 en_only=(qlang == "en"))
+        for rank, idx in enumerate(self._capped_top(
+                sorted(bm25, key=bm25.get, reverse=True))):
+            fused[idx] += 1.0 / (RRF_K + rank)
+        if self.semantic:
+            sem_order, confidence = self._semantic_ranking(expanded, allowed=mask)
+            for rank, idx in enumerate(self._capped_top(sem_order)):
+                fused[idx] += W_SEM / (RRF_K + rank)
+
+        # Ensure explicitly-referenced sections are retrieved even if recall
+        # missed them -- but only for Acts the query actually names. A query
+        # like "IRPA s. 40 misrepresentation defence" uses the section number
+        # topically; pulling every Act's s. 40 into the pool would drown out
+        # the case law that interprets the section the user meant. Substring
+        # check rather than token-overlap because act_codes split into trivial
+        # tokens ("A-8.8" -> {a, 8}) that spuriously match common query words.
+        refs = _section_refs(query)
+        q_lc = query.lower()
+        def _act_in_query(c):
+            short = c["act_short"].lower()
+            code = c["act_code"].lower()
+            return ((short and short in q_lc)
+                    or (code and len(code) >= 3 and code in q_lc))
+        if refs:
+            for idx, c in enumerate(self.chunks):
+                if (c["section"] in refs and idx not in fused
+                        and _act_in_query(c)):
+                    fused[idx] = 0.0
+
+        # Title-match boost: the marginal note is a section's canonical subject.
+        # Reward a candidate by how completely and how specifically the query
+        # matches its marginal note. The overlap is idf-weighted (matching a
+        # distinctive title like "hours of work" counts far more than a generic
+        # one like "Decision"), scaled by coverage, and capped -- so it nudges
+        # ranking toward the provision a question names by topic without
+        # overriding a strong base score.
+        q_tokens = set(tokenize(expanded))
+        for idx in list(fused):
+            note_tokens = self._note_tokens[idx]
+            total = sum(self.idf.get(t, 0.0) for t in note_tokens)
+            if total <= 0:
+                continue
+            matched = sum(self.idf.get(t, 0.0)
+                          for t in note_tokens if t in q_tokens)
+            if matched > 0:
+                fused[idx] += min(MN_WEIGHT * matched * matched / total, MN_CAP)
+
+        # Hierarchy penalties: a topical question should surface the governing
+        # provision, not the supplementary material around it. An Act creates a
+        # duty while a regulation only elaborates procedure; a collective
+        # agreement's numbered articles are its substance while its memoranda
+        # and letters of understanding are back-matter; a not-in-force
+        # amendment is pending law, not the law. Each takes a small fusion
+        # penalty so the governing provision wins a close contest (the NIF
+        # penalty is larger: pending text should surface only when it is the
+        # clearly best match for the subject, never beat the live provision).
+        for idx in list(fused):
+            if self._is_nif[idx]:
+                fused[idx] -= NIF_PENALTY
+            elif self._is_repealed[idx]:
+                fused[idx] -= REPEALED_PENALTY
+            elif self._is_regulation[idx]:
+                fused[idx] -= REG_PENALTY
+            elif self._is_backmatter[idx]:
+                fused[idx] -= BACKMATTER_PENALTY
+
+        candidates = [i for i in sorted(fused, key=fused.get, reverse=True)
+                      if self._filter_ok(self.chunks[i], act, doc_type, qlang,
+                                         court, decided_after)]
+        if not candidates:
+            return []
+        scores = {i: fused[i] for i in candidates}
+
+        # Precision stage: the cross-encoder rescores the top candidate pool, but
+        # may only PROMOTE -- each pooled candidate is placed at the better of its
+        # fusion rank and its rerank rank, never below its fusion rank. The
+        # reranker reliably surfaces a strong answer the fusion ranked low, yet is
+        # unreliable on long statutory text (it can score the right section
+        # negative), so its power to demote a candidate is deliberately removed.
+        if self.reranker:
+            pool = candidates[:RERANK_POOL]
+            docs = [self._rerank_doc(i) for i in pool]
+            ce_orig = self.reranker.score(query, docs)
+            # A repaired query is scored alongside the original and each
+            # candidate keeps its better score -- the cross-encoder cannot
+            # ground 'detension' against text saying 'detention'. Applies
+            # only when a correction fired, so legacy queries never take
+            # the second pass.
+            if corrected != query:
+                ce_fix = self.reranker.score(corrected, docs)
+                ce_orig = [max(a, b) for a, b in zip(ce_orig, ce_fix)]
+            # Optional (CANLEX_RERANK_DUAL=1): likewise for the
+            # synonym-expanded form. Off by default: the 2026-07 eval could
+            # not attribute its effect cleanly on the grown corpus.
+            if expanded != corrected and os.environ.get("CANLEX_RERANK_DUAL"):
+                ce_exp = self.reranker.score(expanded, docs)
+                ce_orig = [max(a, b) for a, b in zip(ce_orig, ce_exp)]
+            ce = dict(zip(pool, ce_orig))
+            fusion_rank = {idx: r for r, idx in enumerate(pool)}
+            rerank_rank = {idx: r for r, idx in enumerate(
+                sorted(pool, key=ce.get, reverse=True))}
+            pool.sort(key=lambda i: (min(fusion_rank[i], rerank_rank[i]),
+                                     fusion_rank[i]))
+            candidates = pool + candidates[RERANK_POOL:]
+
+        # Explicit section references are pinned to the very top -- using the
+        # same Act-mentioned constraint as the recall step above, for the same
+        # reason: a bare "s. 40" without an Act name is usually topical
+        # (e.g. "the IRPA s. 40 misrepresentation defence"), not a lookup.
+        if refs:
+            pinned = [i for i in candidates
+                      if self.chunks[i]["section"] in refs
+                      and _act_in_query(self.chunks[i])]
+            if pinned:
+                pinned_set = set(pinned)
+                candidates = pinned + [i for i in candidates if i not in pinned_set]
+
+        # Cap one-source monopolies, then guarantee a primary instrument on
+        # the topic is represented. The guarantee operates on a fixed visible
+        # window of min(top_k, 5), not the full top_k -- with top_k=20 (the
+        # eval default) the larger window almost always contains incidental
+        # legislation, so the guarantee never fires even when the governing
+        # provision is buried at rank 10+.
+        candidates = self._diversify(candidates)
+        candidates = self._ensure_primary(candidates, min(top_k, 5), q_tokens)
+
+        top = self._cosurface_appendices(candidates[:top_k])
+        highlights = self._highlight(query, top)
+        results = []
+        for pos, i in enumerate(top):
+            result = {**self.chunks[i], "score": round(scores.get(i, 0.0), 4),
+                      "confidence": confidence}
+            if pos in highlights:
+                result["highlight"] = highlights[pos]
+            results.append(result)
+        return results
+
+    def get_section(self, act, section, lang="en"):
+        """The section chunk in the requested language, falling back to the
+        other language rather than returning nothing."""
+        act = act.lower()
+        fallback = None
+        for c in self.chunks:
+            if c["section"] == section and act in (
+                    c["act_short"].lower(), c["act_code"].lower(),
+                    c.get("act_name", "").lower()):
+                if c.get("lang", "en") == lang:
+                    return c
+                fallback = fallback or c
+        return fallback
+
+    def _build_xref(self):
+        """Index legislation by (act, section); find each Act's definitions
+        section; link every regulation to its enabling Act and every
+        D-memorandum to the provisions it cites -- all for cross-referencing."""
+        self._by_section = {}
+        self._defs_section = {}
+        acts, regs = {}, {}        # act_code -> (act_short, act_name)
+        for c in self.chunks:
+            if c.get("doc_type", "legislation") != "legislation":
+                continue
+            if c.get("lang", "en") == "fr":
+                continue    # xref keys are per (act, section); English canonical
+            self._by_section[(c["act_code"], c["section"])] = c
+            if c["act_code"] not in self._defs_section and (
+                    c["marginal_note"].strip().lower() in (
+                        "definitions", "definition", "interpretation")):
+                self._defs_section[c["act_code"]] = c
+            bucket = regs if c["act_code"].startswith(("SOR", "C.R.C")) else acts
+            bucket.setdefault(c["act_code"], (c["act_short"], c["act_name"]))
+
+        # Link a regulation to the Act it is made under by matching their names
+        # ("X Regulations" <-> "X Act").
+        self._enabling_act = {}                 # reg code -> (act_short, act_name)
+        self._regulations = defaultdict(list)   # act code -> [(reg_short, reg_name)]
+        def base(name):
+            return re.sub(r"\b(?:Act|Regulations)\b", "", name).strip().lower()
+        act_by_base = {base(n): (code, s, n) for code, (s, n) in acts.items()}
+        for rcode, (rshort, rname) in regs.items():
+            hit = act_by_base.get(base(rname))
+            if hit:
+                self._enabling_act[rcode] = (hit[1], hit[2])
+                self._regulations[hit[0]].append((rshort, rname))
+
+        # Link D-memoranda to the Customs Act / Customs Tariff provisions they
+        # cite; an unqualified "the Act" in a D-memo means the Customs Act.
+        by_short = {s.lower(): code for code, (s, n) in acts.items()}
+        customs, tariff = by_short.get("customs act"), by_short.get("customs tariff")
+        self._memos_for_section = defaultdict(set)   # (act_code, section) -> memos
+        for c in self.chunks:
+            if c.get("doc_type") != "memorandum":
+                continue
+            for num, which in _MEMO_CITE.findall(c["text"]):
+                code = tariff if which.lower() == "customs tariff" else customs
+                if code:
+                    self._memos_for_section[(code, num)].add(c["section"])
+
+    def related(self, chunk):
+        """Cross-references for a legislation result, as a dict: 'provisions'
+        (intra-Act sections it cites, plus the definitions section),
+        'regulations' (made under this Act), 'enabling_act' (for a regulation,
+        the Act it is made under) and 'memoranda' (D-memo numbers citing this
+        section). Empty dict for case law, memoranda, etc."""
+        if chunk.get("doc_type", "legislation") != "legislation":
+            return {}
+        act = chunk["act_code"]
+        provisions, seen = [], {chunk["section"]}
+        defs = self._defs_section.get(act)
+        if defs and defs["section"] not in seen:
+            provisions.append((defs["section"], defs["marginal_note"]))
+            seen.add(defs["section"])
+        for match in _XREF.finditer(chunk["text"]):
+            sec = match.group(1)
+            if sec in seen:
+                continue
+            target = self._by_section.get((act, sec))
+            if target:
+                provisions.append((sec, target["marginal_note"]))
+                seen.add(sec)
+            if len(provisions) >= 8:
+                break
+        return {
+            "provisions": provisions,
+            "regulations": self._regulations.get(act, []),
+            "enabling_act": self._enabling_act.get(act),
+            "memoranda": sorted(self._memos_for_section.get(
+                (act, chunk["section"]), []))[:6],
+        }
+
+
+def main():
+    if len(sys.argv) < 2:
+        print('usage: python -m canlex.index "your query"')
+        return
+    query = " ".join(sys.argv[1:])
+    index = LegislationIndex()
+    if index.reranker:
+        mode = "hybrid + cross-encoder rerank"
+    elif index.semantic:
+        mode = "hybrid (BM25 + semantic)"
+    else:
+        mode = "BM25 only"
+    print(f"{len(index.chunks)} sections indexed - {mode}. Query: {query!r}\n")
+    for r in index.search(query):
+        print(f"[{r['score']:.3f}] {r['citation']} - {r['marginal_note']}")
+        print(f"          {r['text'].replace(chr(10), ' ')[:160]}")
+    print()
+
+
+if __name__ == "__main__":
+    main()

canlex/ingest.py

@@ -0,0 +1,481 @@
+"""Ingest Justice Laws XML into structured, section-level JSON chunks."""
+import re
+import sys
+import time
+import urllib.request
+import xml.etree.ElementTree as ET
+
+from ._common import norm_ws as _norm, split_lines, write_corpus
+from .config import SOURCES, RAW_DIR, PROCESSED_DIR
+
+LIMS = "{http://justice.gc.ca/lims}"
+BLOCK_TAGS = {"Subsection", "Paragraph", "Subparagraph", "Clause", "Definition"}
+
+# A section whose entire rendered body is a placeholder marker -- "30 [Amendment]",
+# "162 [Amendments]", "5 [Related provision]" -- left where an enacting Act amended
+# another Act. Content-free; dropped at ingest. ([Repealed, ...] is deliberately
+# excluded here so repealed sections are kept and flagged instead.)
+_AMENDMENT_STUB = re.compile(
+    r"^\S+\s*\[(?:Amendments?|Related provisions?)\]\s*$", re.IGNORECASE)
+
+# The semantic embedder (embed.py) only reads the first _MAX_BODY=2000 chars of
+# a chunk, so the tail of a long section is invisible to semantic search. A
+# section whose rendered text exceeds _SPLIT_THRESHOLD is therefore emitted as
+# several piece-chunks, each <= _PIECE_TARGET chars (under the embed window) and
+# split on subsection/line boundaries, so every part is embeddable. Sections at
+# or below the threshold are emitted unchanged -- same id, same embedding --
+# which keeps ~90% of legislation chunks (and their retrieval behaviour) intact.
+_SPLIT_THRESHOLD = 2400
+_PIECE_TARGET = 1800
+
+
+def _split_text(text, target=_PIECE_TARGET):
+    """Split rendered section text into <=target pieces at line boundaries,
+    folding small leading/trailing remnants into a neighbour so no piece is a
+    stub."""
+    return split_lines(text, target, fold_stubs=True)
+
+
+def _itertext(el):
+    return _norm("".join(el.itertext()))
+
+
+def fetch_xml(code, force=False):
+    src = SOURCES[code]
+    RAW_DIR.mkdir(parents=True, exist_ok=True)
+    dest = RAW_DIR / f"{code}.xml"
+    if dest.exists() and not force:
+        return dest
+    print(f"  downloading {src['xml_url']}")
+    req = urllib.request.Request(src["xml_url"], headers={"User-Agent": "CanLex/0.1"})
+    with urllib.request.urlopen(req, timeout=120) as resp:
+        dest.write_bytes(resp.read())
+    time.sleep(1.0)  # be polite to the Justice Laws server between downloads
+    return dest
+
+
+def _heading_text(h):
+    """Render a <Heading> as 'LABEL - TITLE' when both parts are present."""
+    label, title = h.find("Label"), h.find("TitleText")
+    if label is not None or title is not None:
+        parts = [_itertext(e) for e in (label, title) if e is not None]
+        return " - ".join(p for p in parts if p)
+    return _itertext(h)
+
+
+def _render_block(el, is_section=False):
+    """Recursively render a provision element into readable, structured text."""
+    label = note = ""
+    inline, blocks = [], []
+    for child in el:
+        tag = child.tag
+        if tag == "Label":
+            label = _itertext(child)
+        elif tag == "MarginalNote":
+            if not is_section:  # a section's own marginal note is its title
+                note = _itertext(child)
+        elif tag == "Text":
+            inline.append(_itertext(child))
+        elif tag == "HistoricalNote":
+            continue
+        elif tag in BLOCK_TAGS:
+            blocks.append(_render_block(child))
+        else:
+            nested = _render_block(child)
+            if nested:
+                blocks.append(nested)
+    head = " ".join(p for p in (label, f"[{note}]" if note else "", " ".join(inline)) if p)
+    if blocks:
+        return (head + "\n" if head else "") + "\n".join(b for b in blocks if b)
+    return head
+
+
+def _table_text(tablegroup):
+    """Render a CALS <TableGroup> as caption/title lines plus one
+    pipe-separated line per row. Column structure survives as ' | '
+    separators, which is enough for BM25/semantic matching and readable in a
+    result pane; no attempt is made to preserve spanned cells."""
+    out = []
+    for t in (tablegroup.find("Caption"), tablegroup.find("table/title")):
+        if t is not None:
+            txt = _itertext(t)
+            if txt:
+                out.append(txt)
+    for row in tablegroup.iter("row"):
+        cells = [_itertext(e) for e in row.findall("entry")]
+        line = " | ".join(cells).strip(" |")
+        if line:
+            out.append(line)
+    return "\n".join(out)
+
+
+def _sched_body(el):
+    """Render a <Schedule>'s content elements into readable text.
+
+    Observed shapes across the corpus: List/Item trees (CDSA I-IV, IRPR
+    1.1), DocumentInternal/Group/Provision prose (CDSA V-VI, IRPA's Refugee
+    Convention), and TableGroup grids (IRPR Schedule 2 violations, the AMPS
+    designated-provisions tables). Lists, groups and provisions reuse
+    _render_block, whose Label/Text recursion matches their structure;
+    tables get the dedicated renderer above so rows keep their shape."""
+    parts = []
+    for child in el:
+        tag = child.tag
+        if tag in ("ScheduleFormHeading", "HistoricalNote"):
+            continue
+        if tag == "TableGroup":
+            parts.append(_table_text(child))
+        elif tag == "GroupHeading":
+            parts.append(_heading_text(child))
+        elif tag in ("DocumentInternal", "Group", "RegulationPiece", "BillPiece"):
+            parts.append(_sched_body(child))
+        else:
+            rendered = _render_block(child)
+            if rendered:
+                parts.append(rendered)
+    return "\n".join(p for p in parts if p)
+
+
+# Schedule ids with dedicated handling: NifProvs is ingested separately as
+# not-in-force chunks; RelatedProvs (transitional provisions of past
+# amendments) is deliberately excluded -- see _nif_chunks.
+_SCHED_SKIP_IDS = {"NifProvs", "RelatedProvs"}
+
+
+def _schedule_chunks(root, code, src, current_to):
+    """Chunks for an Act's substantive Schedules.
+
+    Previously every Schedule except NifProvs was silently dropped -- which
+    meant the CDSA's controlled-substance lists, the AMPS designated-
+    provisions tables and IRPA's Refugee Convention articles were absent
+    from the corpus even though the sections that depend on them were
+    ingested. A source can opt out with 'skip_schedules': True (the Customs
+    Tariff does: its schedule is the full tariff nomenclature, served
+    separately by tariff_schedule.py from the CBSA edition)."""
+    if src.get("skip_schedules"):
+        return []
+    chunks = []
+    for n, sched in enumerate(root.findall("Schedule"), start=1):
+        if sched.get("id") in _SCHED_SKIP_IDS:
+            continue
+        h = sched.find("ScheduleFormHeading")
+        label = _norm(h.findtext("Label") or "") if h is not None else ""
+        title = _norm(h.findtext("TitleText") or "") if h is not None else ""
+        oref = _norm(h.findtext("OriginatingRef") or "") if h is not None else ""
+        label = label or (f"SCHEDULE ({n})" if not title else title)
+        repealed = sched.find("Repealed") is not None
+        body = "[Repealed]" if repealed else _sched_body(sched)
+        if not body:
+            continue
+        header = " — ".join(p for p in (title, oref) if p)
+        text = (header + "\n" if header else "") + body
+        base = {
+            "act_code": code,
+            "act_short": src["short"],
+            "act_name": src["name"],
+            "section": label.title(),
+            "marginal_note": title or label.title(),
+            "part": "SCHEDULES",
+            "division": "",
+            "heading": label.title(),
+            "history": "",
+            "last_amended": sched.get(f"{LIMS}lastAmendedDate", ""),
+            "in_force": sched.get(f"{LIMS}inforce-start-date", ""),
+            "status": "repealed" if repealed else "in force",
+            "current_to": current_to,
+            "citation": f"{src['short']}, {label.title()}",
+            "source_url": src["web_base"],
+        }
+        pieces = (_split_text(text)
+                  if len(text) > _SPLIT_THRESHOLD else [text])
+        total = len(pieces)
+        for k, piece in enumerate(pieces, start=1):
+            if total == 1:
+                chunks.append({**base, "id": f"{code}-sched{n}", "text": piece})
+            else:
+                chunks.append({
+                    **base, "id": f"{code}-sched{n}-p{k}", "text": piece,
+                    "citation": f"{base['citation']} (part {k} of {total})"})
+    return chunks
+
+
+def _history(section):
+    note = section.find("HistoricalNote")
+    if note is None:
+        return ""
+    return "; ".join(_itertext(i) for i in note.iter("HistoricalNoteSubItem"))
+
+
+def parse_legislation(xml_path, code):
+    src = SOURCES[code]
+    data = xml_path.read_bytes()
+    if data[:3] == b"\xef\xbb\xbf":  # some Justice Laws XML files carry a UTF-8 BOM
+        data = data[3:]
+    root = ET.fromstring(data)
+    current_to = root.get(f"{LIMS}current-date", "")
+    body = root.find("Body")
+    if body is None:
+        # A schedule-only instrument (e.g. the Import Control List) has no
+        # <Body> at all -- its entire substance is in Schedules.
+        if not root.findall("Schedule"):
+            raise ValueError(f"{code}: no <Body> element (root <{root.tag}>)")
+        body = ()
+
+    headings, chunks = {}, []
+    for el in body:
+        if el.tag == "Heading":
+            try:
+                level = int(el.get("level", "0"))
+            except ValueError:
+                level = 0
+            headings = {k: v for k, v in headings.items() if k < level}
+            headings[level] = _heading_text(el)
+            continue
+        if el.tag != "Section":
+            continue
+
+        label_el = el.find("Label")
+        number = _itertext(label_el) if label_el is not None else ""
+        note_el = el.find("MarginalNote")
+        marginal = _itertext(note_el) if note_el is not None else ""
+        body_text = _render_block(el, is_section=True)
+
+        # Drop content-free amendment/related-provision stubs. When an Act
+        # amends OTHER Acts, the consolidation keeps a placeholder section whose
+        # whole body is just a marker -- e.g. "30 [Amendment]" -- the actual
+        # text living in the Act being amended. These carry no substance, yet
+        # were being indexed and rendered with an 'in force' currency line as if
+        # they were live provisions (117 in the CBSA Act alone). Skip them.
+        # (Repealed sections, "<n> [Repealed, ...]", are NOT dropped -- they are
+        # flagged status='repealed' and kept so a section lookup still informs.)
+        if _AMENDMENT_STUB.match(body_text):
+            continue
+
+        if number:
+            chunk_id = f"{code}-s{number}"
+            citation = f"{src['short']}, s. {number}"
+            source_url = f"{src['web_base']}section-{number}.html"
+        else:
+            # An unnumbered <Section> is a preamble or enacting recital. Keep it
+            # unless it has no text; the LIMS id keeps the chunk id unique when
+            # an Act contains more than one such block.
+            if not body_text:
+                continue
+            number = marginal or "Preamble"
+            marginal = marginal or "Preamble"
+            lims_id = el.get(f"{LIMS}id") or str(len(chunks))
+            chunk_id = f"{code}-pre-{lims_id}"
+            citation = f"{src['short']}, {number}"
+            source_url = src["web_base"]
+
+        part = next((v for _, v in sorted(headings.items())
+                     if v.upper().startswith("PART")), "")
+        division = next((v for _, v in sorted(headings.items())
+                         if v.upper().startswith("DIVISION")), "")
+        nearest = headings[max(headings)] if headings else ""
+
+        # Per-section coming-into-force date (lims:inforce-start-date is present
+        # on every section) and repeal status. A wholly-repealed section is a
+        # <Section> whose own <Text> is a <Repealed> marker (e.g. "[Repealed,
+        # 2018, c. 27, s. 174]"); a live section with a repealed subsection has
+        # the <Repealed> nested deeper, so we key on the direct Text/Repealed to
+        # flag only sections that are themselves no longer law.
+        in_force = el.get(f"{LIMS}inforce-start-date", "")
+        status = "repealed" if el.find("Text/Repealed") is not None else "in force"
+
+        base = {
+            "act_code": code,
+            "act_short": src["short"],
+            "act_name": src["name"],
+            "section": number,
+            "marginal_note": marginal,
+            "part": part,
+            "division": division,
+            "heading": nearest,
+            "history": _history(el),
+            "last_amended": el.get(f"{LIMS}lastAmendedDate", ""),
+            "in_force": in_force,
+            "status": status,
+            "current_to": current_to,
+            "source_url": source_url,
+        }
+
+        # Short sections (the large majority) stay one chunk with their
+        # original id and citation -- unchanged embedding and retrieval. Long
+        # ones are split into embeddable pieces; each piece keeps the section
+        # number (so citation lookups, section-ref pinning and the eval still
+        # match) and notes its part in the citation.
+        pieces = (_split_text(body_text)
+                  if len(body_text) > _SPLIT_THRESHOLD else [body_text])
+        n = len(pieces)
+        for k, piece in enumerate(pieces, start=1):
+            if n == 1:
+                chunks.append({**base, "id": chunk_id, "text": piece,
+                               "citation": citation})
+            else:
+                chunks.append({**base, "id": f"{chunk_id}-p{k}", "text": piece,
+                               "citation": f"{citation} (part {k} of {n})"})
+
+    chunks.extend(_schedule_chunks(root, code, src, current_to))
+    chunks.extend(_nif_chunks(root, code, src, current_to))
+    return chunks
+
+
+def _nif_chunks(root, code, src, current_to):
+    """Chunks for the Act's AMENDMENTS NOT IN FORCE schedule.
+
+    Justice Laws consolidations carry enacted-but-not-yet-in-force amendments
+    in a Schedule with id='NifProvs' (each entry a RelatedOrNotInForce node:
+    a citation heading for the amending Act plus the amendment text as regular
+    Section elements). Previously the parser dropped these silently -- the
+    corpus's biggest currency blind spot, since a Royal-Assented amendment
+    days from changing the law was invisible until consolidation. They are
+    ingested with status='not-in-force': retrieval down-weights them below
+    live law and the server banners them, but a search for their subject
+    matter now surfaces the pending law instead of nothing.
+
+    The sibling Schedule id='RelatedProvs' (transitional related provisions)
+    is deliberately NOT ingested here: those are in-force companion rules of
+    past amendments, not pending law, and would need their own treatment.
+
+    Note the Label inside an entry is the AMENDING Act's section number, so
+    the chunk's 'section' is left empty -- these must never satisfy a lookup
+    or pin for the target Act's own section numbers.
+    """
+    nif = next((s for s in root.findall("Schedule")
+                if s.get("id") == "NifProvs"), None)
+    if nif is None:
+        return []
+    chunks = []
+    for n, entry in enumerate(nif.iter("RelatedOrNotInForce"), start=1):
+        title = entry.find("Heading/TitleText")
+        cite = _norm("".join(title.itertext())) if title is not None else ""
+        cite = cite.lstrip("—- ").strip()
+        sections = entry.findall("Section")
+        body = "\n".join(p for p in (_render_block(s, is_section=True)
+                                     for s in sections) if p)
+        if not body:
+            continue
+        note_el = sections[0].find("MarginalNote") if sections else None
+        marginal = _norm("".join(note_el.itertext())) if note_el is not None else ""
+        pieces = (_split_text(body)
+                  if len(body) > _SPLIT_THRESHOLD else [body])
+        total = len(pieces)
+        for k, piece in enumerate(pieces, start=1):
+            suffix = "" if total == 1 else f"-p{k}"
+            part_note = "" if total == 1 else f" (part {k} of {total})"
+            chunks.append({
+                "id": f"{code}-nif-{n}{suffix}",
+                "act_code": code,
+                "act_short": src["short"],
+                "act_name": src["name"],
+                "section": "",
+                "marginal_note": marginal or f"Amendment not in force — {cite}",
+                "part": "AMENDMENTS NOT IN FORCE",
+                "division": "",
+                "heading": "Amendments not in force",
+                "text": piece,
+                "history": cite,
+                "last_amended": "",
+                "in_force": "",
+                "status": "not-in-force",
+                "current_to": current_to,
+                "citation": (f"{src['short']} — amendment not in force "
+                             f"({cite}){part_note}"),
+                "source_url": src["web_base"],
+            })
+    return chunks
+
+
+def ingest(code, force=False, allow_shrink=False):
+    print(f"Ingesting {code} ({SOURCES[code]['short']})...")
+    xml_path = fetch_xml(code, force=force)
+    chunks = parse_legislation(xml_path, code)
+    out = PROCESSED_DIR / f"{code}.json"
+    # An Act whose XML schema shifts parses to a fraction of its sections; the
+    # guard turns that into a refusal instead of a quietly gutted statute.
+    if not write_corpus(out, chunks, indent=2, allow_shrink=allow_shrink):
+        raise RuntimeError(f"{code}: refused to write a collapsed corpus")
+    print(f"  {len(chunks)} sections -> {out.name}")
+    return chunks
+
+
+# --- French corpus (phase 3 of bilingual support) ----------------------------
+# Justice Laws publishes every consolidation as parallel French XML in the
+# SAME schema (English element names, French text, identical section
+# numbering), so the whole parser -- schedules included -- works unchanged.
+# French chunks carry lang='fr', a '-fr-' id infix, French citation style
+# ('art.' / 'annexe') and the French web URL; retrieval filters by query
+# language (index.py), and get_section/xref/list_acts stay English-first.
+
+def _fr_xml_url(src):
+    return (src["xml_url"]
+            .replace("/eng/XML/", "/fra/XML/")
+            .replace("/XML/SOR-", "/XML/DORS-")  # règlements: SOR -> DORS
+            .replace("_c._", "_ch._"))        # C.R.C. chapters: c. -> ch.
+
+
+def _frenchify(chunks, code):
+    out = []
+    for c in chunks:
+        c = dict(c)
+        c["id"] = c["id"].replace(f"{code}-", f"{code}-fr-", 1)
+        c["lang"] = "fr"
+        c["citation"] = (c["citation"]
+                         .replace(", s. ", ", art. ")
+                         .replace(", Schedule", ", annexe"))
+        c["source_url"] = (c["source_url"]
+                           .replace("/eng/acts/", "/fra/lois/")
+                           .replace("/eng/regulations/", "/fra/reglements/"))
+        out.append(c)
+    return out
+
+
+def ingest_fr(code, force=False, allow_shrink=False):
+    src = SOURCES[code]
+    dest = RAW_DIR / f"{code}-fr.xml"
+    if not dest.exists() or force:
+        url = _fr_xml_url(src)
+        print(f"  downloading {url}")
+        req = urllib.request.Request(url, headers={"User-Agent": "CanLex/0.1"})
+        with urllib.request.urlopen(req, timeout=120) as resp:
+            dest.write_bytes(resp.read())
+        time.sleep(1.0)
+    chunks = _frenchify(parse_legislation(dest, code), code)
+    out = PROCESSED_DIR / f"{code}-fr.json"
+    if not write_corpus(out, chunks, indent=2, allow_shrink=allow_shrink):
+        raise RuntimeError(f"{code}: refused to write a collapsed corpus")
+    print(f"  {len(chunks)} articles -> {out.name}")
+    return chunks
+
+
+def main():
+    force = "--force" in sys.argv
+    french = "--fr" in sys.argv
+    allow_shrink = "--allow-shrink" in sys.argv
+    only = [a for a in sys.argv[1:] if not a.startswith("-")]
+    codes = only or list(SOURCES)
+    failures = []
+    for code in codes:
+        if code not in SOURCES:
+            print(f"  SKIP {code}: not a known source")
+            continue
+        try:
+            if french:
+                print(f"Ingesting {code} (français)...")
+                ingest_fr(code, force=force, allow_shrink=allow_shrink)
+            else:
+                ingest(code, force=force, allow_shrink=allow_shrink)
+        except Exception as exc:
+            failures.append((code, exc))
+            print(f"  FAILED {code}: {type(exc).__name__}: {exc}")
+    print(f"\nDone: {len(codes) - len(failures)}/{len(codes)} ingested.")
+    for code, exc in failures:
+        print(f"  FAILED {code}: {exc}")
+    # One collapsed Act must not pass as a clean run: refresh.py and the
+    # RUNBOOK chain this into `&& py -m canlex.embed`.
+    sys.exit(1 if failures else 0)
+
+
+if __name__ == "__main__":
+    main()

canlex/irb_guidelines.py

@@ -0,0 +1,149 @@
+"""Ingest the IRB Chairperson's Guidelines the border workflow actually meets.
+
+Guideline 2 (Detention) is what Immigration Division members apply at every
+48-hour/7-day/30-day review an officer attends; Guidelines 3 (child
+claimants) and 8 (vulnerable persons) govern how detained or referred minors
+and vulnerable people are handled. Soft law: issued under IRPA
+s. 159(1)(h), members are expected to apply them or explain why not --
+ingested as doc_type='memorandum' (guidance), like the D-memoranda.
+
+Pages are public HTML on irb-cisr.gc.ca (same host caselaw.py already
+parses for jurisprudential guides).
+
+    python -m canlex.irb_guidelines
+"""
+import re
+import sys
+
+from bs4 import BeautifulSoup
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      split_lines, stored_chunks, uniquify_ids, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "irb_guidelines"
+
+GUIDELINES = [
+    {"num": "2", "file": "guideline2",
+     "url": "https://irb.gc.ca/en/legal-policy/policies/Pages/GuideDir02.aspx",
+     "title": "Detention"},
+    {"num": "3", "file": "guideline3",
+     "url": "https://irb.gc.ca/en/legal-policy/policies/Pages/GuideDir03.aspx",
+     "title": "Child Refugee Claimants: Procedural and Evidentiary Issues"},
+    {"num": "8", "file": "guideline8",
+     "url": "https://irb.gc.ca/en/legal-policy/policies/Pages/GuideDir08.aspx",
+     "title": "Accessibility to IRB Proceedings — Procedural Accommodations "
+              "and Substantive Considerations"},
+]
+
+# "Effective date: May 31, 2021" (or amended/revised) in the page prose.
+_EFFECTIVE = re.compile(
+    r"(?:effective|amended|revised)[^.:\n]*:?\s*((?:January|February|March|"
+    r"April|May|June|July|August|September|October|November|December)\s+"
+    r"\d{1,2},\s+\d{4})", re.I)
+_MONTHS = {m: i for i, m in enumerate(
+    ("January", "February", "March", "April", "May", "June", "July",
+     "August", "September", "October", "November", "December"), start=1)}
+
+
+def _iso(prose):
+    m = re.match(r"(\w+)\s+(\d{1,2}),\s+(\d{4})", prose or "")
+    if not m or m.group(1) not in _MONTHS:
+        return ""
+    return f"{m.group(3)}-{_MONTHS[m.group(1)]:02d}-{int(m.group(2)):02d}"
+
+
+def _sections(main):
+    """Yield (label, heading, text) per numbered heading in the page body."""
+    heads = main.find_all(["h2", "h3"])
+    for i, h in enumerate(heads):
+        heading = _norm(h.get_text())
+        m = re.match(r"(\d+(?:\.\d+)*)\.?\s+(.*)", heading)
+        label = m.group(1) if m else str(i + 1)
+        title = m.group(2) if m else heading
+        parts = []
+        for sib in h.next_siblings:
+            if getattr(sib, "name", None) in ("h2", "h3"):
+                break
+            if getattr(sib, "get_text", None):
+                parts.append(sib.get_text("\n"))
+        body = "\n".join(p for p in (_norm(x) for x in
+                                     "\n".join(parts).split("\n")) if p)
+        if len(body) > 80:
+            yield label, title, body
+
+
+def build(allow_shrink=False):
+    chunks, failures = [], []
+    for g in GUIDELINES:
+        try:
+            html = fetch_cached(g["url"], RAW / f"{g['file']}.html",
+                                pause=1.0).decode("utf-8", "replace")
+        except Exception as exc:
+            print(f"  !! Guideline {g['num']}: fetch failed "
+                  f"({type(exc).__name__}: {exc})")
+            failures.append(f"IRB-G{g['num']}")
+            continue
+        soup = BeautifulSoup(html, "html.parser")
+        main = soup.find("main") or soup
+        eff = _EFFECTIVE.search(main.get_text())
+        effective = _iso(eff.group(1)) if eff else ""
+        n = 0
+        for label, title, body in _sections(main):
+            pieces = split_lines(body, 1800) if len(body) > 2400 else [body]
+            total = len(pieces)
+            for k, piece in enumerate(pieces, start=1):
+                suffix = "" if total == 1 else f"-p{k}"
+                note = "" if total == 1 else f" (part {k} of {total})"
+                chunks.append({
+                    "id": f"irbg-{g['num']}-{label.replace('.', '-')}{suffix}",
+                    "doc_type": "memorandum",
+                    "act_code": f"IRB-G{g['num']}",
+                    "act_short": f"IRB Guideline {g['num']}",
+                    "act_name": f"IRB Chairperson's Guideline {g['num']} — "
+                                f"{g['title']}",
+                    "section": f"G{g['num']} s. {label}",
+                    "marginal_note": title,
+                    "part": g["title"],
+                    "division": "",
+                    "heading": "",
+                    "text": piece,
+                    "history": "",
+                    "last_amended": effective,
+                    "current_to": effective,
+                    "citation": f"Chairperson's Guideline {g['num']}, "
+                                f"s. {label} ({title}){note}",
+                    "source_url": g["url"],
+                })
+                n += 1
+        if not n:
+            # Fetched, parsed to nothing: the IRB page template moved.
+            print(f"  !! Guideline {g['num']}: no content parsed; "
+                  f"keeping stored copy")
+            failures.append(f"IRB-G{g['num']}")
+            continue
+        print(f"  Guideline {g['num']} ({g['title'][:40]}): {n} chunks, "
+              f"effective {effective or '?'}")
+    out = PROCESSED_DIR / "irb_guidelines.json"
+    chunks, preserved = preserve_dropped(chunks, stored_chunks(out),
+                                         key=lambda c: c["act_code"],
+                                         only=set(failures))
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} guideline(s) "
+              f"that failed this run: {', '.join(preserved)}")
+    uniquify_ids(chunks)
+    if not write_corpus(out, chunks, indent=2, allow_shrink=allow_shrink):
+        return False
+    print(f"irb_guidelines: {len(chunks)} chunks -> {out.name}")
+    return chunks
+
+
+def main():
+    # `is not False` rather than truthiness: a first-ever run that writes an
+    # empty corpus succeeded, a refused write did not.
+    ok = build(allow_shrink="--allow-shrink" in sys.argv) is not False
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/loganalyze.py

@@ -0,0 +1,81 @@
+"""Mine the query log for real-world retrieval gaps.
+
+The MCP server appends one JSON line per search to CANLEX_LOG_DIR/queries.jsonl
+when CANLEX_LOG_DIR is set (see server._log_query). Real user queries -- the
+ones that returned nothing or scored a weak match -- are the only unbiased
+source of corpus and synonym gaps, since the 155-question eval set is authored
+by the maintainer. This tool surfaces the weakest and most frequent of them as
+candidates for new eval questions, synonyms, or corpus additions.
+
+    py -m canlex.loganalyze [path/to/queries.jsonl]   # defaults to CANLEX_LOG_DIR
+
+By default it lists no-result queries, then the lowest-confidence queries, then
+the most-repeated ones. Privacy: query text may contain personal facts -- treat
+the log as sensitive and do not commit it.
+"""
+import json
+import os
+import sys
+from collections import Counter
+from pathlib import Path
+
+# Mirror server.HEDGE_THRESHOLD's intent without importing the server: a query
+# whose best result scored below this is a weak match worth reviewing.
+_WEAK = 0.72
+
+
+def _load(path):
+    rows = []
+    for line in Path(path).read_text(encoding="utf-8").splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            rows.append(json.loads(line))
+        except ValueError:
+            continue
+    return rows
+
+
+def report(path):
+    rows = _load(path)
+    if not rows:
+        print(f"No logged queries in {path}.")
+        return
+    print(f"{len(rows)} logged queries in {path}\n")
+
+    no_result = [r for r in rows if r.get("n_results", 0) == 0]
+    weak = sorted((r for r in rows
+                   if r.get("confidence") is not None
+                   and r["confidence"] < _WEAK and r.get("n_results")),
+                  key=lambda r: r["confidence"])
+    freq = Counter(r.get("query", "").strip().lower() for r in rows)
+
+    print(f"== No results ({len(no_result)}) -- likely corpus or vocabulary gaps ==")
+    for r in no_result[:25]:
+        print(f"  {r.get('query','')}")
+
+    print(f"\n== Weak matches (confidence < {_WEAK}; {len(weak)}) -- candidate "
+          f"eval questions / synonyms ==")
+    for r in weak[:25]:
+        top = (r.get("top_citations") or ["?"])[0]
+        print(f"  {r['confidence']:.3f}  {r.get('query','')[:70]:70}  -> {top}")
+
+    print("\n== Most repeated queries (demand signal) ==")
+    for q, n in freq.most_common(15):
+        if n > 1:
+            print(f"  {n:>3}x  {q}")
+
+
+def main():
+    path = (sys.argv[1] if len(sys.argv) > 1
+            else os.path.join(os.environ.get("CANLEX_LOG_DIR", "."), "queries.jsonl"))
+    if not Path(path).exists():
+        print(f"No query log at {path}. Set CANLEX_LOG_DIR on the server (and "
+              f"pass the path here) to collect one.", file=sys.stderr)
+        sys.exit(1)
+    report(path)
+
+
+if __name__ == "__main__":
+    main()

canlex/pdi.py

@@ -0,0 +1,250 @@
+"""Ingest scoped IRCC Program Delivery Instructions (PDI) pages.
+
+The PDIs are the current-policy successor to the legacy chapter manuals.
+This module ingests the specific PDI sets that replace retired ENF chapters
+the corpus needs: the PR-card/status section (successor to ENF 27) and the
+Tran serious-criminality page (partial successor to ENF 2). Each named
+index page is fetched, its same-section sub-pages discovered, and every
+page chunked by heading -- doc_type='memorandum', like all guidance.
+
+    python -m canlex.pdi [--allow-shrink]
+"""
+import re
+import sys
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      split_lines, stored_chunks, uniquify_ids, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "pdi"
+OUT_FILE = PROCESSED_DIR / "pdi.json"
+_OBM = ("https://www.canada.ca/en/immigration-refugees-citizenship/corporate/"
+        "publications-manuals/operational-bulletins-manuals/")
+
+# Each set is one PDI category tree, crawled breadth-first from its index to
+# _MAX_DEPTH, capped at _MAX_PAGES pages (dropped pages are LOGGED, never
+# silently skipped). Scope fragments keep the crawl inside the category. The
+# selection is the border-relevant slice of the OBM tree: temporary
+# residents (visitors/eTA/TRV/TRPs), refugee protection (port-of-entry
+# claims), the cross-cutting standard requirements (includes the Tran
+# serious-criminality page), officer designation/delegation, biometrics and
+# identity, and the PR-card section (ENF 27's successor). Deliberately out:
+# citizenship, fees, office service-delivery, and the year-by-year
+# operational-bulletin archives (mostly expired instructions).
+_MAX_DEPTH = 2
+_MAX_PAGES = 160    # 60 capped out: the Temporary Residents tree alone has
+                    # ~150 in-scope pages (2026-07 crawl)
+
+SETS = [
+    {"code": "pdi-prcard",
+     "short": "PDI PR Card",
+     "name": "PDI — Permanent resident card and PR status (successor to ENF 27)",
+     "index": _OBM + "permanent-residence/card.html",
+     "scope": "/permanent-residence/card"},
+    {"code": "pdi-tr",
+     "short": "PDI Temporary Residents",
+     "name": "PDI — Temporary Resident Program (visitors, eTA, TRV, permits)",
+     "index": _OBM + "temporary-residents.html",
+     "scope": "/temporary-residents"},
+    {"code": "pdi-refugee",
+     "short": "PDI Refugee Protection",
+     "name": "PDI — Refugee Protection (claims, eligibility, port of entry)",
+     "index": _OBM + "refugee-protection.html",
+     "scope": "/refugee-protection"},
+    {"code": "pdi-standard",
+     "short": "PDI Standard Requirements",
+     "name": "PDI — Standard requirements (incl. Tran serious-criminality "
+             "assessment)",
+     "index": _OBM + "standard-requirements.html",
+     "scope": "/standard-requirements"},
+    {"code": "pdi-delegation",
+     "short": "PDI Designation-Delegation",
+     "name": "PDI — Designation of officers and delegation of authority",
+     "index": _OBM + "legislation.html",
+     "scope": "operational-bulletins-manuals/legislation"},
+    {"code": "pdi-identity",
+     "short": "PDI Biometrics-Identity",
+     "name": "PDI — Biometrics and identity management",
+     "index": _OBM + "identity-management.html",
+     "scope": "/identity-management"},
+]
+
+_MODIFIED = re.compile(r'<time[^>]*>(\d{4}-\d{2}-\d{2})</time>')
+_TITLE = re.compile(r"<h1[^>]*>(.*?)</h1>", re.S)
+_LINK = re.compile(r'href="([^"#?]*?\.html)"')
+
+
+def _strip_tags(html):
+    html = re.sub(r"</t[dh]>", " | ", html)
+    html = re.sub(r"</(?:p|li|tr|h\d|caption)>", "\n", html)
+    html = re.sub(r"<[^>]+>", " ", html)
+    return [l for l in (_norm(x).strip("| ").strip()
+                        for x in html.split("\n")) if l]
+
+
+def _main(html):
+    m = re.search(r"<main[^>]*>(.*?)</main>", html, re.S)
+    return m.group(1) if m else html
+
+
+def _page_chunks(src, page_url, html, page_no):
+    content = _main(html)
+    title_m = _TITLE.search(content)
+    title = (_norm(re.sub(r"<[^>]+>", " ", title_m.group(1)))
+             if title_m else src["short"])
+    mod = _MODIFIED.search(html)
+    modified = mod.group(1) if mod else ""
+    # Split on h2 headings; each becomes a section-like unit.
+    parts = re.split(r"<h2[^>]*>", content[title_m.end():] if title_m else content)
+    chunks = []
+    for i, part in enumerate(parts):
+        if i == 0:
+            heading, body_html = title, part
+        else:
+            head_end = part.find("</h2>")
+            heading = _norm(re.sub(r"<[^>]+>", " ", part[:head_end]))
+            body_html = part[head_end:]
+        body = "\n".join(_strip_tags(body_html))
+        if len(body) < 120:
+            continue
+        pieces = split_lines(body, 1800) if len(body) > 2400 else [body]
+        total = len(pieces)
+        for k, piece in enumerate(pieces, start=1):
+            suffix = "" if total == 1 else f"-p{k}"
+            note = "" if total == 1 else f" (part {k} of {total})"
+            chunks.append({
+                "id": f"{src['code']}-{page_no}-{i}{suffix}",
+                "doc_type": "memorandum",
+                "act_code": src["code"].upper(),
+                "act_short": src["short"],
+                "act_name": src["name"],
+                "section": f"{title[:60]} — {heading[:60]}" if i else title[:80],
+                "marginal_note": heading,
+                "part": src["name"],
+                "division": "",
+                "heading": "",
+                "text": piece,
+                "history": "",
+                "last_amended": modified,
+                "current_to": modified,
+                "citation": f"{src['short']}: {heading}{note}",
+                "source_url": page_url,
+            })
+    return chunks
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of sub-pages this run could not fetch.
+
+    Ingestion is a full rebuild, so a sub-page that 404s or drops its
+    connection silently vanishes from the corpus; serving the copy we already
+    had beats losing instructions that are still in force. Keyed on source_url,
+    the only per-page identity a PDI chunk carries that survives pages coming
+    and going -- the chunk id embeds the page's position in the index listing,
+    which shifts whenever one page drops out. Restricted to pages that actually
+    errored, so a sub-page IRCC retired on purpose is not resurrected.
+    Returns (chunks, preserved page URLs).
+    """
+    return preserve_dropped(chunks, stored, key=lambda c: c["source_url"],
+                            only={url for url, _why in failures})
+
+
+def _crawl(src, failures):
+    """Breadth-first crawl of one PDI category: index -> in-scope links ->
+    their in-scope links, to _MAX_DEPTH, capped at _MAX_PAGES pages.
+    Returns [(url, html)]; pages dropped by the cap are LOGGED, and fetch
+    failures land in `failures` so preserve_failed keeps their stored
+    copies."""
+    index_html = fetch_cached(
+        src["index"], RAW / f"{src['code']}-index.html",
+        powershell=True, pause=0.8).decode("utf-8", "replace")
+    pages = [(src["index"], index_html)]
+    if not src["scope"]:
+        return pages
+    seen = {src["index"]}
+    frontier = [(src["index"], index_html)]
+    dropped = 0
+    for _depth in range(_MAX_DEPTH):
+        next_frontier = []
+        for _url, html in frontier:
+            for href in _LINK.findall(_main(html)):
+                if src["scope"] not in href:
+                    continue
+                url = href if href.startswith("http") else \
+                    "https://www.canada.ca" + href
+                if url in seen:
+                    continue
+                seen.add(url)
+                if len(pages) >= _MAX_PAGES:
+                    dropped += 1
+                    continue
+                name = url.rsplit("/", 1)[1].replace(".html", "")
+                try:
+                    page = fetch_cached(
+                        url, RAW / f"{src['code']}-{name}.html",
+                        powershell=True, pause=0.8).decode("utf-8", "replace")
+                except Exception as exc:
+                    failures.append((url, f"{type(exc).__name__}: {exc}"))
+                    print(f"  !! {name}: {type(exc).__name__}: {exc}")
+                    continue
+                pages.append((url, page))
+                next_frontier.append((url, page))
+        frontier = next_frontier
+    if dropped:
+        print(f"  !! {src['short']}: page cap {_MAX_PAGES} hit -- {dropped} "
+              f"in-scope pages NOT ingested (raise _MAX_PAGES to cover them)")
+    return pages
+
+
+def build(allow_shrink=False):
+    """Rebuild pdi.json. Returns the chunks, or False if the write was refused."""
+    all_chunks, failures = [], []
+    for src in SETS:
+        try:
+            pages = _crawl(src, failures)
+        except Exception as exc:
+            failures.append((src["index"], f"{type(exc).__name__}: {exc}"))
+            print(f"  !! {src['short']}: index fetch failed "
+                  f"({type(exc).__name__}: {exc})")
+            continue
+        n = 0
+        for page_no, (url, html) in enumerate(pages):
+            chunks = _page_chunks(src, url, html, page_no)
+            if not chunks:
+                # Downloaded fine, parsed to nothing: the page template moved
+                # under us. That is exactly how the D-memo corpus was lost, so
+                # it counts as a failure and the stored page is kept.
+                print(f"  !! {url.rsplit('/', 1)[-1]}: no content parsed; "
+                      f"keeping stored copy")
+                failures.append((url, "no content parsed"))
+                continue
+            all_chunks.extend(chunks)
+            n += len(chunks)
+        print(f"  {src['short']}: {len(pages)} pages, {n} chunks")
+
+    all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                            stored_chunks(OUT_FILE))
+    if preserved:
+        names = ", ".join(u.rsplit("/", 1)[-1] for u in preserved)
+        print(f"  kept the last-good copy of {len(preserved)} unfetchable "
+              f"page(s): {names}")
+    # After preservation, not before: a preserved page brings back the ids it
+    # was written with, and this run may have handed those ids to other pages
+    # (an id embeds the page's position in the index listing).
+    uniquify_ids(all_chunks)
+    if not write_corpus(OUT_FILE, all_chunks, indent=2,
+                        allow_shrink=allow_shrink):
+        return False
+    print(f"pdi: {len(all_chunks)} chunks -> {OUT_FILE.name}")
+    return all_chunks
+
+
+def main():
+    # `is not False` rather than truthiness: a first-ever run that writes an
+    # empty corpus succeeded, a refused write did not.
+    ok = build(allow_shrink="--allow-shrink" in sys.argv) is not False
+    sys.exit(0 if ok else 1)
+
+
+if __name__ == "__main__":
+    main()

canlex/refresh.py

@@ -0,0 +1,753 @@
+"""Detect when CanLex's ingested sources have gone stale upstream.
+
+    py -m canlex.refresh --check
+
+For every Justice Laws instrument in config.SOURCES, this fetches just the
+opening of the XML (the root <Statute>/<Regulation> element carries the
+consolidation date in its lims:current-date attribute, so a few KB is enough)
+and compares that upstream date against the current_to stored in the matching
+data/processed/<code>.json. Anything where upstream is newer is reported as
+drift, and the process exits non-zero -- so this can gate a deploy or run on a
+schedule.
+
+It also checks the two delegation instruments against upstream (check_delegation):
+the CBSA IRPA delegation -- by diffing the dated 'irpa-lipr-YYYY-MM-DD' instruments
+the CBSA legislation index publishes against those delegation.SOURCES ingests, so
+a newly-published amendment is flagged -- and the IRCC IL3 instrument, by comparing
+the version stamp in the live PDF against the stored one. The NJC directives get
+the same treatment (check_directives): the effective date each chunk stores in
+current_to comes from the NJC directive index (the span.date beside each entry),
+so one fresh index fetch re-reads it for nearly every directive, with the
+'effective <Month D, YYYY>' prose on the directive's own page as fallback for
+entries the index leaves undated (the Foreign Service Directives) -- exactly the
+two signals ingestion uses. Finally it prints the stored currency of the
+remaining non-XML sources (D-memos, case law, agreements, benefits, tariff
+schedule), which expose no comparable upstream signal, with the command to
+refresh each; and flags any tracked pending bill whose coming-into-force date
+has passed.
+
+This is the staleness detection the corpus previously lacked: legislation was
+refreshed by hand, so e.g. Bill C-12 only landed when someone thought to look.
+
+    py -m canlex.refresh --check          # human-readable; exit 1 on drift
+    py -m canlex.refresh --check --json    # machine-readable for CI
+"""
+import datetime
+import json
+import re
+import sys
+import tempfile
+import time
+import urllib.request
+from pathlib import Path
+
+from bs4 import BeautifulSoup
+
+from . import delegation, directive, dmemo
+from ._common import BROWSER_UA, MONTHS
+from .config import SOURCES, PROCESSED_DIR
+
+_UA = "CanLex-refresh/0.1"
+_HEAD_BYTES = 16384      # the root element's attributes sit at the very top
+_CURRENT_DATE = re.compile(rb'current-date="(\d{4}-\d{2}-\d{2})"')
+
+# Aggregate (non-Justice-Laws) sources: (label, processed filename, refresh
+# command, description). These carry no single upstream consolidation date, so
+# the check reports their STORED currency (newest item) and how to refresh them
+# rather than diffing against upstream. (Delegation and the NJC directives are
+# checked against upstream separately, by check_delegation and check_directives,
+# because their sources DO expose a comparable signal -- the CBSA instrument
+# index and IL3 version string, and the NJC directive index dates.)
+_NON_XML = [
+    ("memorandum", "dmemos.json", "py -m canlex.dmemo", "CBSA D-Memoranda"),
+    ("caselaw", "caselaw.json", "py -m canlex.caselaw", "Court & tribunal decisions"),
+    ("agreement", "agreements.json", "py -m canlex.agreement", "FB collective agreement"),
+    ("benefits", "benefits.json", "py -m canlex.benefits", "Benefit-plan booklets"),
+    ("tariff", "tariff_schedule.json", "py -m canlex.tariff_schedule",
+     "Customs Tariff Schedule ch. 98-99"),
+    ("enf", "enf.json", "py -m canlex.enf", "ENF operational manuals"),
+    ("amps", "amps.json", "py -m canlex.amps", "AMPS Master Penalty Document"),
+    ("charter", "charter.json", "py -m canlex.charter", "Charter / Constitution"),
+    ("commentary", "commentary.json", "py -m canlex.commentary",
+     "Curated commentary chunks"),
+    ("union", "ciu.json", "py -m canlex.ciu", "CIU/PSAC union documents"),
+    ("pdi", "pdi.json", "py -m canlex.pdi",
+     "IRCC PDI trees (TR/refugee/standard/identity/PR-card)"),
+    ("memorandum-fr", "dmemos-fr.json", "py -m canlex.dmemo --fr",
+     "Mémorandums D (français)"),
+    ("directive-fr", "directives-fr.json", "py -m canlex.directive --fr",
+     "Directives du CNM (français)"),
+    ("irb-guidelines", "irb_guidelines.json", "py -m canlex.irb_guidelines",
+     "IRB Chairperson's Guidelines"),
+]
+
+
+_CIU_DECISIONS = "https://ciu-sdi.ca/en/important-decisions/"
+_CIU_DEC_MARKER = PROCESSED_DIR.parent / "raw" / "ciu" / "decisions-page-date.txt"
+
+
+def check_ciu_decisions():
+    """New-case-law tripwire: the CIU curates its Important Decisions page,
+    so its dateModified moving is a cheap signal that a new decision worth
+    ingesting has issued. This is the only automated freshness signal the
+    case-law corpus has -- courts publish no 'new leading case' feed -- and
+    it complements the quarterly gap-audit ritual rather than replacing it."""
+    try:
+        req = urllib.request.Request(
+            _CIU_DECISIONS, headers={"User-Agent": BROWSER_UA})
+        with urllib.request.urlopen(req, timeout=60) as resp:
+            html = resp.read().decode("utf-8", "replace")
+        m = re.search(r'"dateModified":"(\d{4}-\d{2}-\d{2})', html)
+        remote = m.group(1) if m else ""
+    except Exception as exc:
+        return {"stored": "", "remote": "", "status": "error",
+                "error": f"{type(exc).__name__}: {exc}"}
+    if not remote:
+        return {"stored": "", "remote": "", "status": "no-date", "error": ""}
+    stored = (_CIU_DEC_MARKER.read_text(encoding="utf-8").strip()
+              if _CIU_DEC_MARKER.exists() else "")
+    if not stored:
+        # First run: record the baseline rather than alarming.
+        _CIU_DEC_MARKER.parent.mkdir(parents=True, exist_ok=True)
+        _CIU_DEC_MARKER.write_text(remote, encoding="utf-8")
+        stored = remote
+    status = "stale" if remote > stored else "ok"
+    return {"stored": stored, "remote": remote, "status": status, "error": ""}
+
+
+def check_union():
+    """Upstream drift for the CIU/PSAC documents (incl. every site VSSA).
+
+    The union reposts amended documents at the same URLs, so the drift
+    signal is each PDF's Last-Modified header against our cached copy's
+    fetch time. New/removed VSSAs surface as fetch errors or new links on
+    the next canlex.ciu run; this check covers content changes to what we
+    already hold."""
+    import email.utils
+    from . import ciu
+    rows = []
+    targets = [(short, lang, ciu.RAW / f"{code}{'-fr' if lang == 'fr' else ''}.pdf", url)
+               for code, short, name, url, lang in ciu.DOCS]
+    try:
+        targets += [(f"VSSA {stem}", "en",
+                     ciu.RAW / f"vssa-{stem.lower()}.pdf", url)
+                    for url, stem in ciu._vssa_links()]
+    except Exception as exc:
+        rows.append({"short": "VSSA index", "lang": "en", "status": "error",
+                     "remote": "", "error": f"{type(exc).__name__}: {exc}"})
+    for short, lang, cache, url in targets:
+        try:
+            req = urllib.request.Request(url, method="HEAD",
+                                         headers={"User-Agent": BROWSER_UA})
+            with urllib.request.urlopen(req, timeout=30) as resp:
+                lm = resp.headers.get("Last-Modified", "")
+        except Exception as exc:
+            rows.append({"short": short, "lang": lang, "status": "error",
+                         "remote": "", "error": f"{type(exc).__name__}: {exc}"})
+            continue
+        if not lm:
+            rows.append({"short": short, "lang": lang, "status": "no-date",
+                         "remote": "", "error": ""})
+            continue
+        remote_ts = email.utils.parsedate_to_datetime(lm).timestamp()
+        local_ts = cache.stat().st_mtime if cache.exists() else 0
+        rows.append({"short": short, "lang": lang,
+                     "status": "stale" if remote_ts > local_ts else "ok",
+                     "remote": lm[:25], "error": ""})
+        time.sleep(0.3)
+    return rows
+
+# Curated datasets age: state legislatures amend deferred-adjudication
+# statutes every session, so a reviewed date older than this many days earns
+# a warning even though nothing upstream can be diffed automatically.
+_CURATED_MAX_AGE_DAYS = 180
+
+
+def check_curated(today):
+    """Age-check the curated datasets and detect edit-without-regen drift.
+
+    Two signals: (1) each data/curated/*.json 'reviewed' date older than
+    _CURATED_MAX_AGE_DAYS -> aging (re-verify against the state sources);
+    (2) the newest commentary.json current_to differing from the newest
+    curated 'reviewed' -> the curated file was edited without rerunning
+    canlex.commentary + canlex.embed, so production serves stale chunks."""
+    from .config import DATA_DIR
+    rows = []
+    newest_reviewed = ""
+    for path in sorted((DATA_DIR / "curated").glob("*.json")):
+        try:
+            reviewed = json.loads(
+                path.read_text(encoding="utf-8")).get("reviewed", "")
+        except Exception as exc:
+            rows.append({"file": path.name, "reviewed": "",
+                         "status": "error", "error": f"{type(exc).__name__}: {exc}"})
+            continue
+        newest_reviewed = max(newest_reviewed, reviewed or "")
+        status = "ok"
+        if reviewed:
+            age = (datetime.date.fromisoformat(today)
+                   - datetime.date.fromisoformat(reviewed)).days
+            if age > _CURATED_MAX_AGE_DAYS:
+                status = "aging"
+        rows.append({"file": path.name, "reviewed": reviewed,
+                     "status": status, "error": ""})
+    commentary = PROCESSED_DIR / "commentary.json"
+    if newest_reviewed and commentary.exists():
+        chunks = json.loads(commentary.read_text(encoding="utf-8"))
+        newest_chunk = max((c.get("current_to", "") for c in chunks), default="")
+        if newest_chunk != newest_reviewed:
+            rows.append({"file": "commentary.json",
+                         "reviewed": newest_chunk, "status": "drift",
+                         "error": f"curated reviewed {newest_reviewed} but "
+                                  f"chunks say {newest_chunk or '?'} -- rerun "
+                                  f"canlex.commentary && canlex.embed"})
+    return rows
+
+
+def check_tariff_edition(today):
+    """Probe whether CBSA has published a Customs Tariff edition newer than
+    the ingested one. The tariff pages live under a year-stamped path
+    (.../tariff-tarif/<year>/html/...), so probing the ingested-edition-plus-
+    one URL (and the current calendar year's) is a reliable new-edition
+    signal without parsing anything."""
+    from . import tariff_schedule
+    src = next(iter(tariff_schedule.SOURCES.values()))
+    have = int(src["edition"])
+    candidates = sorted({have + 1, int(today[:4])} - {have})
+    for year in candidates:
+        if year <= have:
+            continue
+        probe = src["url"].replace(f"/{src['edition']}/", f"/{year}/")
+        try:
+            req = urllib.request.Request(
+                probe, method="HEAD", headers={"User-Agent": BROWSER_UA})
+            with urllib.request.urlopen(req, timeout=30) as resp:
+                if resp.status == 200:
+                    return {"stored": str(have), "remote": str(year),
+                            "status": "stale", "error": ""}
+        except Exception:
+            continue                       # 404 = that edition doesn't exist yet
+    return {"stored": str(have), "remote": "", "status": "ok", "error": ""}
+
+
+def dmemo_drift(raw, stored):
+    """(newest active LAST_UPDATED, memos newer than `stored`) from the CBSA
+    D-memo datasource JSON. Pure, for testability."""
+    rows = [r for r in json.loads(raw).get("data", [])
+            if str(r.get("STATUS", "")).strip().lower() != "cancelled"]
+    remote = max((r.get("LAST_UPDATED", "") for r in rows), default="")
+    drifted = sorted(_dmemo_number(r) for r in rows
+                     if stored and r.get("LAST_UPDATED", "") > stored)
+    return remote, drifted
+
+
+def _dmemo_number(row):
+    """'D1-2-1' out of a datasource NUMBER cell (an anchor, or bare text)."""
+    cell = str(row.get("NUMBER", ""))
+    inner = re.search(r">([^<]+)<", cell)
+    return (inner.group(1) if inner else re.sub(r"<[^>]+>", "", cell)).strip()
+
+
+def check_dmemo_index():
+    """Newest per-memo 'Last updated' upstream vs stored currency.
+
+    This reads the JSON datasource behind the index table, NOT dates in the
+    index HTML: since CBSA rebuilt the page (2026-07-10) the newest date in the
+    markup is the page's own 'Date modified', which moves when the page is
+    rebuilt and no memo has changed -- that would report stale forever. The
+    datasource carries a LAST_UPDATED per memo, so drift is both true and named.
+    """
+    stored = ""
+    path = PROCESSED_DIR / "dmemos.json"
+    if path.exists():
+        chunks = json.loads(path.read_text(encoding="utf-8"))
+        stored = max((c.get("current_to", "") for c in chunks), default="")
+    try:
+        req = urllib.request.Request(
+            dmemo.DATASOURCE_URL, headers={"User-Agent": BROWSER_UA})
+        with urllib.request.urlopen(req, timeout=60) as resp:
+            remote, drifted = dmemo_drift(resp.read(), stored)
+    except Exception as exc:
+        return {"stored": stored, "remote": "", "drifted": [], "status": "error",
+                "error": f"{type(exc).__name__}: {exc}"}
+    status = "stale" if (remote and stored and remote > stored) else \
+        ("ok" if remote else "no-date")
+    return {"stored": stored, "remote": remote, "drifted": drifted,
+            "status": status, "error": ""}
+
+# NJC directive dates as published: the index prints prose ('June  1, 1993' --
+# double spaces occur in the raw markup) while chunks may store either that
+# prose or the ISO fallback directive.py derives from page text, so both sides
+# are normalized to ISO before comparing.
+_NJC_PROSE_DATE = re.compile(r"([A-Za-z]+)\s+(\d{1,2}),?\s+(\d{4})")
+_ISO_DATE = re.compile(r"\d{4}-\d{2}-\d{2}")
+
+# CBSA publishes its IRPA delegation as a base instrument plus incremental
+# amendments (no consolidation), each a dated 'irpa-lipr-YYYY-MM-DD-eng.html'
+# linked from the CBSA legislation index. A date on the index that we have not
+# ingested means a new amendment to add to delegation.SOURCES.
+_CBSA_INDEX_URL = ("https://www.cbsa-asfc.gc.ca/agency-agence/actreg-loireg/"
+                   "legislation-eng.html")
+_CBSA_INSTRUMENT_RE = re.compile(r"irpa-lipr-(\d{4}-\d{2}-\d{2})-eng\.html")
+# The IRCC IL3 instrument is reissued with a seasonal version stamp on page 1.
+_IL3_VERSION_RE = re.compile(r"(?:Spring|Summer|Fall|Winter)\s+\d{4}")
+
+# Bills known to be heading into force that affect the corpus. Surfaced once
+# their in-force date has passed, with the codes to re-ingest. (The legislation
+# drift check above will independently flag the affected Act once Justice Laws
+# consolidates the amendment; this list just names the cause.)
+_PENDING_BILLS = [
+    {"bill": "C-16", "name": "Protecting Victims Act",
+     "in_force": "2026-07-18", "affects": ["C-46"],
+     "note": "Criminal Code: coercive control, child-protection reporting "
+             "(coercive-control provisions come into force later)."},
+]
+
+
+def _extract_current_date(raw):
+    """Pull the lims:current-date (YYYY-MM-DD) from the head of a Justice Laws
+    XML document. Pure and offline, for testability. Returns '' if absent."""
+    if raw[:3] == b"\xef\xbb\xbf":
+        raw = raw[3:]
+    m = _CURRENT_DATE.search(raw)
+    return m.group(1).decode("ascii") if m else ""
+
+
+def _local_current_to(code):
+    """The current_to stored in data/processed/<code>.json (first chunk), or ''."""
+    path = PROCESSED_DIR / f"{code}.json"
+    if not path.exists():
+        return None
+    chunks = json.loads(path.read_text(encoding="utf-8"))
+    return chunks[0].get("current_to", "") if chunks else ""
+
+
+def _remote_current_date(url):
+    """Fetch only the head of the XML and extract its consolidation date.
+    Returns (date, error): date is '' on a clean fetch that lacked the
+    attribute, error is a short string when the fetch itself failed."""
+    req = urllib.request.Request(url, headers={"User-Agent": _UA})
+    try:
+        with urllib.request.urlopen(req, timeout=60) as resp:
+            head = resp.read(_HEAD_BYTES)
+        return _extract_current_date(head), ""
+    except Exception as exc:                         # network / HTTP / timeout
+        return "", f"{type(exc).__name__}: {exc}"
+
+
+def check_legislation():
+    """Compare each Justice Laws source's upstream date to the stored corpus.
+    Returns a list of row dicts."""
+    rows = []
+    for code, src in SOURCES.items():
+        local = _local_current_to(code)
+        remote, error = _remote_current_date(src["xml_url"])
+        if error:
+            status = "error"
+        elif local is None:
+            status = "missing-local"      # configured but never ingested
+        elif remote and remote > local:
+            status = "stale"              # upstream newer than our corpus
+        elif not remote:
+            status = "no-date"            # fetched, but no date found (unusual)
+        else:
+            status = "ok"
+        rows.append({"code": code, "short": src["short"], "local": local or "",
+                     "remote": remote, "status": status, "error": error})
+        time.sleep(0.3)                   # be polite to Justice Laws
+    return rows
+
+
+def non_xml_currency():
+    """Stored currency (newest item) of the aggregate sources."""
+    rows = []
+    for label, fname, cmd, desc in _NON_XML:
+        path = PROCESSED_DIR / fname
+        if not path.exists():
+            rows.append({"label": label, "desc": desc, "cmd": cmd,
+                         "newest": "", "chunks": 0, "present": False})
+            continue
+        chunks = json.loads(path.read_text(encoding="utf-8"))
+        newest = max((c.get("current_to", "") for c in chunks), default="")
+        rows.append({"label": label, "desc": desc, "cmd": cmd,
+                     "newest": newest, "chunks": len(chunks), "present": True})
+    return rows
+
+
+def _cbsa_ingested_dates():
+    """The effective dates of the CBSA dated delegation instruments we ingest."""
+    return sorted({s["effective"] for s in delegation.SOURCES.values()
+                   if s.get("kind") == "html-cbsa" and s.get("effective")})
+
+
+def _ircc_stored_version():
+    """The IL3 version string recorded in the ingested corpus, or ''."""
+    path = PROCESSED_DIR / "delegation.json"
+    if not path.exists():
+        return ""
+    chunks = json.loads(path.read_text(encoding="utf-8"))
+    versions = [c.get("current_to", "") for c in chunks
+                if c.get("act_short") == "IRCC IL3" and c.get("current_to")]
+    return max(versions) if versions else ""
+
+
+def check_delegation():
+    """Compare the delegation instruments against upstream. Returns row dicts.
+
+    CBSA: diff the dated instruments published on the legislation index against
+    those we ingest -- a new date means an un-ingested amendment. IRCC: compare
+    the IL3 version stamp published in the live PDF against the stored one."""
+    rows = []
+
+    # --- CBSA: index diff (cheap; one HTML fetch) ---
+    have = _cbsa_ingested_dates()
+    try:
+        req = urllib.request.Request(_CBSA_INDEX_URL,
+                                     headers={"User-Agent": BROWSER_UA})
+        with urllib.request.urlopen(req, timeout=60) as resp:
+            html = resp.read().decode("utf-8", "replace")
+        published = sorted(set(_CBSA_INSTRUMENT_RE.findall(html)))
+        new = sorted(set(published) - set(have))
+        rows.append({"name": "CBSA IRPA delegation (+ amendments)",
+                     "status": "stale" if new else "ok",
+                     "have": have, "published": published, "new": new,
+                     "error": ""})
+    except Exception as exc:
+        rows.append({"name": "CBSA IRPA delegation (+ amendments)",
+                     "status": "error", "have": have, "published": [],
+                     "new": [], "error": f"{type(exc).__name__}: {exc}"})
+
+    # --- IRCC IL3: version-stamp compare (re-fetches the PDF via PowerShell) ---
+    stored = _ircc_stored_version()
+    try:
+        with tempfile.TemporaryDirectory() as td:
+            data = delegation._fetch(delegation.SOURCES["ircc"]["url"],
+                                     Path(td) / "il3.pdf", powershell=True)
+        pages = delegation._pdf_pages(data)
+        m = _IL3_VERSION_RE.search(pages[0]) if pages else None
+        published = m.group(0) if m else ""
+        if not published:
+            status = "no-version"
+        elif published != stored:
+            status = "stale"
+        else:
+            status = "ok"
+        rows.append({"name": "IRCC IL3 instrument", "status": status,
+                     "have": [stored], "published": [published], "new": [],
+                     "error": ""})
+    except Exception as exc:
+        rows.append({"name": "IRCC IL3 instrument", "status": "error",
+                     "have": [stored], "published": [], "new": [],
+                     "error": f"{type(exc).__name__}: {exc}"})
+    return rows
+
+
+def _stored_directives():
+    """Per-directive stored currency from data/processed/directives.json:
+    {code: (title, current_to, source_url)}, one entry per act_code."""
+    path = PROCESSED_DIR / "directives.json"
+    if not path.exists():
+        return {}
+    chunks = json.loads(path.read_text(encoding="utf-8"))
+    out = {}
+    for c in chunks:
+        code = c.get("act_code")
+        if code and code not in out:
+            out[code] = (c.get("act_short", ""), c.get("current_to", ""),
+                         c.get("source_url", ""))
+    return out
+
+
+def _directive_iso(text):
+    """Normalize an NJC directive date ('June  1, 1993' or already ISO) to
+    YYYY-MM-DD. Pure, for testability. Returns '' when not a recognizable date."""
+    text = (text or "").strip()
+    if _ISO_DATE.fullmatch(text):
+        return text
+    m = _NJC_PROSE_DATE.fullmatch(text)
+    if m and m.group(1).lower() in MONTHS:
+        return (f"{int(m.group(3)):04d}-{MONTHS[m.group(1).lower()]:02d}-"
+                f"{int(m.group(2)):02d}")
+    return ""
+
+
+def _njc_index_dates(html):
+    """{code: date text} parsed from the NJC directive index, mirroring
+    directive.directive_links: the first <a> in each top-level li of
+    ul.directive-list is the current directive (nested lis are archived
+    versions), and the sibling span.date its effective date ('' if absent)."""
+    soup = BeautifulSoup(html, "html.parser")
+    ul = soup.find("ul", class_="directive-list")
+    out = {}
+    if ul is None:
+        return out
+    for li in ul.find_all("li", recursive=False):
+        a = li.find("a", href=True)
+        if not a:
+            continue
+        m = directive._CODE.search(a["href"])
+        if not m:
+            continue
+        span = li.find("span", class_="date")
+        out[m.group(1)] = span.get_text(strip=True) if span else ""
+    return out
+
+
+def _njc_page_effective(url, td):
+    """Fresh-fetch a directive's own page and pull its 'effective <Month D,
+    YYYY>' prose (directive._page_effective_date), following the 'Print Full
+    Directive' link when the landing page is only a table of contents (the
+    FSDs). Returns (iso_date, error)."""
+    try:
+        html = directive._fetch(url, td / "probe.html", force=True)
+        soup = BeautifulSoup(html, "html.parser")
+        main = soup.find("main")
+        date = directive._page_effective_date(main) if main else ""
+        if not date:
+            print_url = directive._print_link(html)
+            if print_url:
+                full = directive._fetch(print_url, td / "probe-full.html",
+                                        force=True)
+                fmain = BeautifulSoup(full, "html.parser").find("main")
+                date = directive._page_effective_date(fmain) if fmain else ""
+        return date, ""
+    except Exception as exc:
+        return "", f"{type(exc).__name__}: {exc}"
+
+
+def check_directives():
+    """Compare each ingested NJC directive's effective date against upstream.
+    Returns row dicts.
+
+    The date directive.py stores in current_to comes from the NJC directive
+    index (the span.date beside each entry), so one fresh index fetch covers
+    nearly every directive; entries the index leaves undated (the Foreign
+    Service Directives) fall back to the 'effective <Month D, YYYY>' prose on
+    the directive's own page -- the same two signals ingestion uses. Probe
+    fetches go to a temp dir, never data/raw, so a stale probe can't mask a
+    later real refresh; directive._fetch already sleeps 0.5s after each
+    download, so the probes stay polite without extra pauses here."""
+    stored = _stored_directives()
+    rows = []
+    if not stored:
+        return rows
+    with tempfile.TemporaryDirectory() as td:
+        td = Path(td)
+        try:
+            html = directive._fetch(directive.INDEX_URL, td / "_index.html",
+                                    force=True)
+            index_dates, index_error = _njc_index_dates(html), ""
+        except Exception as exc:
+            index_dates, index_error = {}, f"{type(exc).__name__}: {exc}"
+        for code, (title, stored_date, url) in sorted(stored.items()):
+            upstream, error = index_dates.get(code, ""), index_error
+            if not upstream and not error:
+                # Undated on the index: read the date off the page itself.
+                upstream, error = _njc_page_effective(url, td)
+            local, remote = _directive_iso(stored_date), _directive_iso(upstream)
+            if error:
+                status = "error"
+            elif not remote:
+                status = "no-date"
+            elif remote != local:
+                status = "stale"      # NJC reissues in place: any drift counts
+            else:
+                status = "ok"
+            rows.append({"code": code, "title": title, "stored": stored_date,
+                         "local": local, "remote": remote, "status": status,
+                         "error": error})
+    return rows
+
+
+def pending_in_force(today):
+    """Pending bills whose in-force date has passed as of `today` (ISO str)."""
+    return [b for b in _PENDING_BILLS if b["in_force"] <= today]
+
+
+def run(as_json=False):
+    today = datetime.date.today().isoformat()
+    leg = check_legislation()
+    deleg = check_delegation()
+    dirs = check_directives()
+    nonxml = non_xml_currency()
+    curated = check_curated(today)
+    dmemo_idx = check_dmemo_index()
+    tariff = check_tariff_edition(today)
+    union = check_union()
+    ciu_dec = check_ciu_decisions()
+    pending = pending_in_force(today)
+    stale = [r for r in leg if r["status"] in ("stale", "missing-local")]
+    errors = [r for r in leg if r["status"] == "error"]
+    deleg_stale = [r for r in deleg if r["status"] == "stale"]
+    dirs_stale = [r for r in dirs if r["status"] == "stale"]
+    curated_flag = [r for r in curated if r["status"] in ("aging", "drift")]
+    dmemo_stale = dmemo_idx["status"] == "stale"
+    tariff_stale = tariff["status"] == "stale"
+    union_stale = [r for r in union if r["status"] == "stale"]
+    ciu_dec_stale = ciu_dec["status"] == "stale"
+
+    if as_json:
+        print(json.dumps({"checked": today, "legislation": leg,
+                          "delegation": deleg, "directives": dirs,
+                          "non_xml": nonxml, "curated": curated,
+                          "dmemo_index": dmemo_idx, "tariff": tariff,
+                          "union": union, "ciu_decisions": ciu_dec,
+                          "pending_in_force": pending,
+                          "stale_count": (len(stale) + len(deleg_stale)
+                                          + len(dirs_stale)
+                                          + len(curated_flag)
+                                          + len(union_stale)
+                                          + (1 if dmemo_stale else 0)
+                                          + (1 if tariff_stale else 0)),
+                          "error_count": len(errors)}, indent=2))
+    else:
+        print(f"CanLex corpus staleness check — {today}\n")
+        print(f"Justice Laws legislation ({len(leg)} instruments):")
+        for r in sorted(leg, key=lambda x: (x["status"] != "stale", x["short"])):
+            if r["status"] == "stale":
+                print(f"  STALE  {r['short']:34} local {r['local']}  "
+                      f"upstream {r['remote']}")
+            elif r["status"] == "missing-local":
+                print(f"  ABSENT {r['short']:34} configured but not ingested "
+                      f"(upstream {r['remote']})")
+            elif r["status"] == "error":
+                print(f"  ERROR  {r['short']:34} {r['error'][:50]}")
+            elif r["status"] == "no-date":
+                print(f"  ?      {r['short']:34} no current-date found upstream")
+        if not stale and not errors:
+            print(f"  all {len(leg)} current (local matches or exceeds upstream)")
+        if stale:
+            codes = " ".join(r["code"] for r in stale)
+            print(f"\n  -> {len(stale)} drifted. Refresh: "
+                  f"py -m canlex.ingest --force {codes}\n"
+                  f"     and the French twins: py -m canlex.ingest --fr "
+                  f"--force {codes}\n"
+                  f"     then: py -m canlex.embed && py -m canlex.eval")
+
+        print("\nDelegation instruments (upstream check):")
+        for r in deleg:
+            if r["status"] == "stale" and r["new"]:
+                print(f"  STALE  {r['name']}: new upstream {', '.join(r['new'])} "
+                      f"not ingested")
+            elif r["status"] == "stale":
+                print(f"  STALE  {r['name']}: upstream {r['published'][0] or '?'} "
+                      f"!= stored {r['have'][0] or '?'}")
+            elif r["status"] == "error":
+                print(f"  ERROR  {r['name']}: {r['error'][:50]}")
+            elif r["status"] == "no-version":
+                print(f"  ?      {r['name']}: no version stamp found upstream")
+            elif len(r["published"]) > 1:        # CBSA: a set of dated instruments
+                print(f"  ok     {r['name']}: current "
+                      f"({len(r['published'])} instruments, latest "
+                      f"{max(r['published'])})")
+            else:                                # IRCC: a single version stamp
+                shown = r["published"][0] if r["published"] else "current"
+                print(f"  ok     {r['name']}: current ({shown})")
+        if deleg_stale:
+            print("     -> re-ingest: add any new CBSA dates to delegation.SOURCES, "
+                  "then py -m canlex.delegation && py -m canlex.embed")
+
+        print(f"\nNJC directives (upstream check, {len(dirs)} ingested):")
+        for r in dirs:
+            if r["status"] == "stale":
+                print(f"  STALE  {r['title']}: upstream {r['remote']} "
+                      f"!= stored {r['local'] or r['stored'] or '?'}")
+            elif r["status"] == "error":
+                print(f"  ERROR  {r['title']}: {r['error'][:50]}")
+            elif r["status"] == "no-date":
+                print(f"  ?      {r['title']}: no effective date found upstream")
+            else:
+                print(f"  ok     {r['title']}: current ({r['remote']})")
+        if dirs_stale:
+            print("     -> re-ingest: py -m canlex.directive --force && "
+                  "py -m canlex.embed")
+
+        print("\nOther sources (stored currency; re-run the ingester to refresh):")
+        for r in nonxml:
+            mark = " " if r["present"] else "!"
+            print(f" {mark}{r['label']:14} {str(r['chunks']):>5} chunks  "
+                  f"newest {r['newest'] or 'n/a':12}  {r['cmd']}")
+
+        print("\nCurated datasets (age + regen-drift check):")
+        for r in curated:
+            if r["status"] == "aging":
+                print(f"  AGING  {r['file']}: reviewed {r['reviewed']} — older "
+                      f"than {_CURATED_MAX_AGE_DAYS} days; re-verify the state "
+                      f"rows against their statutes and bump 'reviewed'")
+            elif r["status"] == "drift":
+                print(f"  DRIFT  {r['file']}: {r['error']}")
+            elif r["status"] == "error":
+                print(f"  ERROR  {r['file']}: {r['error'][:60]}")
+            else:
+                print(f"  ok     {r['file']}: reviewed {r['reviewed'] or '?'}")
+
+        if dmemo_idx["status"] == "stale":
+            drifted = dmemo_idx.get("drifted") or []
+            named = (f" ({', '.join(drifted[:6])}"
+                     f"{', …' if len(drifted) > 6 else ''})" if drifted else "")
+            print(f"\nD-memo index: STALE — newest upstream 'Last updated' "
+                  f"{dmemo_idx['remote']} > stored {dmemo_idx['stored']}"
+                  f"{named}; "
+                  f"run py -m canlex.dmemo --force && py -m canlex.embed")
+        elif dmemo_idx["status"] == "error":
+            print(f"\nD-memo index: ERROR {dmemo_idx['error'][:60]}")
+        else:
+            print(f"\nD-memo index: ok (upstream {dmemo_idx['remote'] or '?'} "
+                  f"<= stored {dmemo_idx['stored'] or '?'})")
+
+        if tariff["status"] == "stale":
+            print(f"\nCustoms Tariff edition: STALE — CBSA has published the "
+                  f"{tariff['remote']} edition (ingested: {tariff['stored']}); "
+                  f"bump the year in tariff_schedule.SOURCES and re-ingest")
+        else:
+            print(f"\nCustoms Tariff edition: ok ({tariff['stored']} is "
+                  f"current)")
+
+        n_union_ok = sum(1 for r in union if r["status"] == "ok")
+        print(f"\nCIU/PSAC union documents (upstream Last-Modified check, "
+              f"{len(union)} documents):")
+        for r in union:
+            if r["status"] == "stale":
+                print(f"  STALE  {r['short']} ({r['lang']}): upstream "
+                      f"modified {r['remote']} after our fetch — run "
+                      f"py -m canlex.ciu && py -m canlex.embed")
+            elif r["status"] == "error":
+                print(f"  ERROR  {r['short']}: {r['error'][:55]}")
+        if not union_stale and not any(r["status"] == "error" for r in union):
+            print(f"  all {n_union_ok} current (upstream unchanged since fetch)")
+
+        if ciu_dec["status"] == "stale":
+            print(f"\nCIU Important-Decisions page: UPDATED "
+                  f"({ciu_dec['remote']} > {ciu_dec['stored']}) — the union "
+                  f"may have flagged a new decision; review "
+                  f"{_CIU_DECISIONS} and, after ingesting anything new, "
+                  f"update {_CIU_DEC_MARKER.name} to {ciu_dec['remote']}")
+        elif ciu_dec["status"] == "error":
+            print(f"\nCIU Important-Decisions page: ERROR "
+                  f"{ciu_dec['error'][:55]}")
+        else:
+            print(f"\nCIU Important-Decisions page: ok (last modified "
+                  f"{ciu_dec['remote'] or '?'})")
+
+        if pending:
+            print("\nPending bills now in force (re-ingest the affected Acts):")
+            for b in pending:
+                print(f"  Bill {b['bill']} ({b['name']}) in force {b['in_force']} "
+                      f"-> {', '.join(b['affects'])}")
+                print(f"     {b['note']}")
+                print(f"     run: py -m canlex.ingest --force {' '.join(b['affects'])}")
+
+    return 1 if (stale or errors or deleg_stale or dirs_stale
+                 or curated_flag or dmemo_stale or tariff_stale
+                 or union_stale or ciu_dec_stale) else 0
+
+
+def main():
+    args = sys.argv[1:]
+    as_json = "--json" in args
+    # --check is the only mode; accepted (and default) for forward-compatibility.
+    sys.exit(run(as_json=as_json))
+
+
+if __name__ == "__main__":
+    main()

canlex/rehab.py

@@ -0,0 +1,143 @@
+"""Criminal-rehabilitation assessment under IRPA s. 36(3)(c) and IRPR ss. 17-18.1.
+
+Pure rule logic, verified against the corpus text of IRPR ss. 17, 18 and 18.1
+(SOR-2002-227, current consolidation) and IRPA s. 36(3). The rules:
+
+  Deemed rehabilitation (IRPR s. 18(2)) -- automatic, no application:
+    (a) ONE foreign conviction, indictable-equivalent, punishable in Canada by
+        a maximum of LESS THAN 10 years: 10 years after sentence completion,
+        with the cleanliness conditions (i)-(vii).
+    (b) TWO OR MORE foreign convictions that would all be summary offences:
+        5 years after the sentences were completed, with conditions.
+    (c) ONE committed act (s. 36(1)(c)/(2)(c)), <10-year maximum: 10 years
+        after the commission, with conditions.
+  Serious criminality (>=10-year maximum equivalent) can NEVER be deemed
+  rehabilitated -- only an individual application.
+
+  Individual rehabilitation (IRPA s. 36(3)(c), period per IRPR s. 17):
+  eligible to APPLY 5 years after sentence completion (conviction branches)
+  or 5 years after commission (act branches). Available for serious
+  criminality too.
+
+  In-Canada convictions are outside the rehabilitation scheme entirely --
+  the cure is a record suspension (IRPA s. 36(3)(b)) -- EXCEPT the IRPR
+  s. 18.1 prescribed class: inadmissible SOLELY for two or more in-Canada
+  summary-only convictions, exempt from s. 36(2)(a) five years after the
+  sentences were completed.
+"""
+
+BASES = ("foreign-conviction", "foreign-act", "canada-conviction")
+COUNTS = ("one", "two-plus-summary-only", "multiple")
+
+
+def assess(basis, max_penalty_lt_10, count, years_elapsed, clean_record):
+    """Assess rehabilitation status. Returns a dict:
+    {status, provision, explanation, caveats}.
+
+    basis: how the inadmissibility arises (BASES).
+    max_penalty_lt_10: True if the equivalent Canadian offence's maximum term
+        is under 10 years (False = serious criminality territory).
+    count: 'one' | 'two-plus-summary-only' | 'multiple' (COUNTS).
+    years_elapsed: full years since sentence completion (conviction bases)
+        or since commission (act basis).
+    clean_record: True if there are no other convictions or committed acts
+        (the s. 18(2) (i)-(vii) conditions, simplified -- see caveats).
+    """
+    caveats = [
+        "The 'clean record' answer stands in for the detailed conditions in "
+        "IRPR s. 18(2)(i)-(vii) (no Canadian indictable conviction ever; no "
+        "recent summary convictions; at most one older summary conviction; no "
+        "other committed acts) -- confirm each condition against the full "
+        "record before relying on deemed rehabilitation.",
+        "A temporary resident permit (IRPA s. 24) remains available at any "
+        "time regardless of rehabilitation status.",
+        "Time runs from the day AFTER completion of the ENTIRE sentence -- "
+        "including probation, fines and driving prohibitions -- not from the "
+        "conviction date.",
+    ]
+
+    if basis == "canada-conviction":
+        if count == "two-plus-summary-only":
+            if years_elapsed >= 5:
+                return _r("exempt", "IRPR s. 18.1",
+                          "Inadmissible solely for two or more in-Canada "
+                          "summary-only convictions, and at least five years "
+                          "have passed since the sentences were completed: "
+                          "the person is in the prescribed class exempt from "
+                          "IRPA s. 36(2)(a).", caveats)
+            return _r("not-yet", "IRPR s. 18.1",
+                      "The in-Canada two-summary-only exemption requires five "
+                      "years since the sentences were completed "
+                      f"({years_elapsed} elapsed).", caveats)
+        return _r("record-suspension-route", "IRPA s. 36(3)(b)",
+                  "Rehabilitation does not apply to in-Canada convictions. "
+                  "The cure is a record suspension under the Criminal Records "
+                  "Act (IRPA s. 36(3)(b)); once ordered and not revoked, the "
+                  "conviction cannot ground inadmissibility.", caveats)
+
+    since = ("sentence completion" if basis == "foreign-conviction"
+             else "the commission of the act")
+
+    if not max_penalty_lt_10:
+        # Serious criminality: deemed rehabilitation is never available.
+        if years_elapsed >= 5:
+            return _r("eligible-to-apply", "IRPA s. 36(3)(c); IRPR s. 17",
+                      "The equivalent Canadian offence carries a maximum of "
+                      "10 years or more (serious criminality), so deemed "
+                      "rehabilitation is NOT available -- but at least five "
+                      f"years have passed since {since}, so the person is "
+                      "eligible to APPLY for individual rehabilitation.",
+                      caveats)
+        return _r("not-yet", "IRPR s. 17",
+                  "Serious criminality: deemed rehabilitation is not "
+                  "available, and the five-year eligibility period to apply "
+                  f"for individual rehabilitation has not yet run "
+                  f"({years_elapsed} of 5 years since {since}).", caveats)
+
+    if count == "multiple":
+        if years_elapsed >= 5:
+            return _r("eligible-to-apply", "IRPA s. 36(3)(c); IRPR s. 17",
+                      "More than one indictable-equivalent conviction/act: "
+                      "deemed rehabilitation is not available (IRPR s. 18(2) "
+                      "covers only a single such conviction or act), but the "
+                      "five-year period to apply for individual "
+                      "rehabilitation has run.", caveats)
+        return _r("not-yet", "IRPR s. 17",
+                  "Multiple indictable-equivalent matters: only individual "
+                  "rehabilitation is possible, and its five-year eligibility "
+                  f"period has not run ({years_elapsed} of 5 years).", caveats)
+
+    if count == "two-plus-summary-only":
+        para = "IRPR s. 18(2)(b)"
+        need = 5
+    elif basis == "foreign-act":
+        para = "IRPR s. 18(2)(c)"
+        need = 10
+    else:
+        para = "IRPR s. 18(2)(a)"
+        need = 10
+
+    if years_elapsed >= need and clean_record:
+        return _r("deemed-rehabilitated", para,
+                  f"All {para} requirements appear met: {years_elapsed} years "
+                  f"since {since} (needs {need}), a qualifying single/summary "
+                  "profile, and a clean record. The person is deemed "
+                  "rehabilitated -- the matter cannot ground inadmissibility "
+                  "under IRPA s. 36(3)(c).", caveats)
+    if years_elapsed >= 5:
+        why = (f"only {years_elapsed} of {need} years have run"
+               if years_elapsed < need else
+               "the record is not clean, which defeats deemed rehabilitation")
+        return _r("eligible-to-apply", f"{para}; IRPR s. 17",
+                  f"Deemed rehabilitation is not (yet) made out ({why}), but "
+                  "five years have passed, so the person is eligible to APPLY "
+                  "for individual rehabilitation.", caveats)
+    return _r("not-yet", f"{para}; IRPR s. 17",
+              f"Neither deemed rehabilitation ({years_elapsed} of {need} "
+              "years) nor application eligibility (five years) has been "
+              "reached.", caveats)
+
+
+def _r(status, provision, explanation, caveats):
+    return {"status": status, "provision": provision,
+            "explanation": explanation, "caveats": caveats}

canlex/rerank.py

@@ -0,0 +1,119 @@
+"""Cross-encoder reranker over ONNX Runtime (local, key-free)."""
+import json
+import os
+
+import numpy as np
+import onnxruntime as ort
+from huggingface_hub import hf_hub_download
+from tokenizers import Tokenizer
+
+RERANK_REPO = "Xenova/bge-reranker-base"
+RERANK_ONNX = "onnx/model_quantized.onnx"  # int8: ~3x faster on CPU, negligible quality loss
+MAX_TOKENS = 512
+_MAX_DOC_CHARS = int(os.environ.get("CANLEX_RERANK_DOC_CHARS", "2000"))
+                    # 1000 -> 2000 adopted 2026-07-23: the doubled reading
+                    # window held every 203-Q eval metric and lifted Hit@5
+                    # +0.01; the cross-encoder now judges a full-size piece
+                    # rather than 60% of one.
+                       # cap doc text before tokenizing. Cross-encoder cost is
+                       # ~linear in tokens per doc, so halving this ~halves the
+                       # dominant query cost; env-sweepable, eval-gated (a
+                       # tighter cap scores long sections on less of their text
+                       # -- fine in practice: the median section is ~520 chars,
+                       # and sub-chunked pieces cap at 1800). Swept 2026-06:
+                       # 3000 -> 1000 (with pool 50 -> 16) held the eval exactly.
+
+
+def make_ort_session(model_path):
+    """A CPU InferenceSession tuned for shared, small-vCPU hosts.
+
+    Two deviations from onnxruntime defaults: (1) spin-waiting between requests
+    is disabled -- by default ORT worker threads busy-spin after an inference
+    for a microsecond-scale latency win, which on the Space's shared 2 vCPUs
+    burns quota between queries; (2) the intra-op thread count is
+    env-configurable (CANLEX_ORT_THREADS) so concurrent sessions can be capped
+    below core count if contention is ever observed. Unset, ORT's default
+    (all cores) is kept, so single-query numerics and speed are unchanged."""
+    opts = ort.SessionOptions()
+    threads = int(os.environ.get("CANLEX_ORT_THREADS", "0"))
+    if threads:
+        opts.intra_op_num_threads = threads
+    opts.add_session_config_entry("session.intra_op.allow_spinning", "0")
+    opts.add_session_config_entry("session.inter_op.allow_spinning", "0")
+    return ort.InferenceSession(model_path, sess_options=opts,
+                                providers=["CPUExecutionProvider"])
+
+
+class Reranker:
+    """Cross-encoder that scores (query, section) pairs for relevance.
+
+    Loads a BGE reranker as ONNX and runs it on CPU -- no API key; the model is
+    downloaded once and cached. A cross-encoder reads the query and section
+    jointly, so it judges true relevance far better than the BM25/embedding
+    similarities used to build the candidate pool.
+    """
+
+    def __init__(self):
+        model_path = hf_hub_download(RERANK_REPO, RERANK_ONNX)
+        tok_path = hf_hub_download(RERANK_REPO, "tokenizer.json")
+        cfg_path = hf_hub_download(RERANK_REPO, "config.json")
+        with open(cfg_path, encoding="utf-8") as fh:
+            self.pad_id = json.load(fh).get("pad_token_id", 0)
+
+        self.session = make_ort_session(model_path)
+        self.input_names = {i.name for i in self.session.get_inputs()}
+
+        self.tokenizer = Tokenizer.from_file(tok_path)
+        # only_second keeps the query intact and truncates the (longer) section.
+        self.tokenizer.enable_truncation(max_length=MAX_TOKENS, strategy="only_second")
+
+    def score(self, query, documents):
+        """Return a relevance logit for each document paired with the query.
+
+        Higher means more relevant. The returned list is aligned with `documents`.
+        """
+        if not documents:
+            return []
+        encs = self.tokenizer.encode_batch(
+            [(query, doc[:_MAX_DOC_CHARS]) for doc in documents])
+        width = max(len(e.ids) for e in encs)
+        input_ids = np.full((len(encs), width), self.pad_id, dtype=np.int64)
+        attention = np.zeros((len(encs), width), dtype=np.int64)
+        type_ids = np.zeros((len(encs), width), dtype=np.int64)
+        for row, enc in enumerate(encs):
+            n = len(enc.ids)
+            input_ids[row, :n] = enc.ids
+            attention[row, :n] = enc.attention_mask
+            type_ids[row, :n] = enc.type_ids
+
+        feed = {"input_ids": input_ids, "attention_mask": attention}
+        if "token_type_ids" in self.input_names:
+            feed["token_type_ids"] = type_ids
+        logits = self.session.run(None, feed)[0]
+        return np.asarray(logits, dtype=np.float32).reshape(-1).tolist()
+
+
+def main():
+    import time
+    print(f"Loading {RERANK_REPO} ({RERANK_ONNX}) ...")
+    reranker = Reranker()
+    query = "powers of arrest without warrant"
+    docs = [
+        "Arrest without warrant. A peace officer may arrest without warrant a "
+        "person who has committed a criminal offence.",
+        "Definitions. In this Act, fish means any fish and includes shellfish, "
+        "crustaceans and marine animals.",
+        "Importation. It is prohibited to import cannabis except as authorized "
+        "under this Act.",
+    ]
+    start = time.perf_counter()
+    scores = reranker.score(query, docs)
+    elapsed = (time.perf_counter() - start) * 1000
+    print(f"\nQuery: {query!r}")
+    for doc, score in sorted(zip(docs, scores), key=lambda x: x[1], reverse=True):
+        print(f"  {score:8.3f}  {doc[:62]}")
+    print(f"\n{elapsed:.0f} ms for {len(docs)} documents")
+
+
+if __name__ == "__main__":
+    main()

canlex/server.py

@@ -0,0 +1,1237 @@
+#!/usr/bin/env python3
+"""MCP server exposing CanLex Canadian legislation retrieval as tools.
+
+Runs over stdio. Launched by an MCP client (Claude Code / Claude Desktop);
+it requires no API keys -- retrieval is fully local.
+"""
+import datetime
+import json
+import os
+import re
+import sys
+import time
+from pathlib import Path
+from typing import Optional
+
+# Allow this file to be launched directly by an MCP client, cwd-independent.
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from typing import Annotated, Literal
+
+from pydantic import Field
+from mcp.server.fastmcp import FastMCP
+
+from canlex.index import LegislationIndex
+
+mcp = FastMCP("canlex_mcp", host="0.0.0.0",
+              port=int(os.environ.get("PORT", "8000")))
+
+_READONLY = {
+    "readOnlyHint": True,
+    "destructiveHint": False,
+    "idempotentHint": True,
+    "openWorldHint": False,
+}
+
+# Prepended to every result so the model grounds, cites, and hedges correctly.
+GROUNDING_NOTE = (
+    "ANSWERING INSTRUCTIONS: Base the answer only on the material below. Cite "
+    "specific provisions and quote key operative words (e.g. 'IRPA s. 34(1)(c)'). "
+    "When a result lists related provisions, regulations or D-memoranda, fetch "
+    "any that bear on the question -- the definitions section, an exception, a "
+    "cross-referenced rule, the regulation that adds detail -- with "
+    "canlex_get_section or canlex_search_legislation before answering. "
+    "Distinguish the kinds of source: enacted law (Acts and regulations) is binding; "
+    "CBSA D-Memoranda are administrative guidance -- persuasive only, not binding, "
+    "and a court may disagree with them; collective agreements and the National "
+    "Joint Council directives they incorporate are binding employment-terms "
+    "instruments for a bargaining unit; court decisions interpret and apply the "
+    "law and are binding precedent depending on the court and jurisdiction -- "
+    "name the deciding court and the date, and do not assume a decision is still "
+    "good law if it may have been overtaken (the canlex_case tool checks a "
+    "decision's later treatment on CanLII -- give it the neutral citation). "
+    "Always state the date the source is current to, and that the answer "
+    "reflects the law only as of that date -- for a time-sensitive matter, tell "
+    "the reader to verify no amendment has come into force since. If the material "
+    "below does not fully resolve the question -- including where it turns on case "
+    "law or facts not present here -- say so explicitly. This is legal information, "
+    "not legal advice."
+)
+
+HEDGE_THRESHOLD = 0.72   # max semantic similarity below which results are weak
+
+WEAK_MATCH_NOTE = (
+    "RETRIEVAL CAUTION: the material below is only a weak match for this query "
+    "— CanLex may not contain a provision or decision that directly answers it. "
+    "Read it critically; if it does not actually address the question, say so "
+    "plainly rather than stretching it to fit, and consider canlex_list_acts to "
+    "check what the corpus covers."
+)
+
+_INDEX: Optional[LegislationIndex] = None
+
+
+def _index() -> LegislationIndex:
+    """Load and cache the legislation index on first use."""
+    global _INDEX
+    if _INDEX is None:
+        _INDEX = LegislationIndex()
+    return _INDEX
+
+
+_GUIDANCE_AGE_YEARS = 5          # warn when guidance is older than this
+_YEAR = re.compile(r"\b(?:19|20)\d{2}\b")
+
+
+def _log_query(query, top_k, act, doc_type, results):
+    """Append one JSON line per search to CANLEX_LOG_DIR/queries.jsonl, if that
+    env var is set. Real user queries -- especially weak-match or no-result ones
+    -- are the only unbiased source of corpus gaps (the eval set is
+    maintainer-authored); scripts/loganalyze.py mines them. Logs the query text
+    and outcome only, never anything else; opt-in via the env var, off by
+    default. Best-effort: a logging failure never affects the response."""
+    log_dir = os.environ.get("CANLEX_LOG_DIR")
+    if not log_dir:
+        return
+    try:
+        top = results[0] if results else None
+        record = {
+            "ts": int(time.time()),
+            "query": query,
+            "top_k": top_k,
+            "act": act,
+            "doc_type": doc_type,
+            "n_results": len(results),
+            "confidence": (round(top["confidence"], 4)
+                           if top and top.get("confidence") is not None else None),
+            "top_citations": [r.get("citation", "") for r in results[:3]],
+        }
+        path = Path(log_dir)
+        path.mkdir(parents=True, exist_ok=True)
+        with (path / "queries.jsonl").open("a", encoding="utf-8") as fh:
+            fh.write(json.dumps(record, ensure_ascii=False) + "\n")
+    except Exception:
+        pass
+
+
+def _stated(value: str) -> str:
+    """A currency value for display, or an honest placeholder when the source
+    states none -- so a reader sees 'not stated by the source' rather than a
+    bare 'n/a' that reads as an oversight."""
+    return value or "not stated by the source"
+
+
+def _guidance_age_note(current_to: str) -> str:
+    """A warning when administrative guidance is several years old, since its
+    figures and procedures may have changed (e.g. duty-free limits in a 2013
+    D-memo). Only fires when a year can be read from the currency value."""
+    years = _YEAR.findall(current_to or "")
+    if not years:
+        return ""
+    newest = max(int(y) for y in years)
+    age = datetime.date.today().year - newest
+    if age >= _GUIDANCE_AGE_YEARS:
+        return (f"_⚠ This guidance is about {age} years old (dated {newest}); "
+                "its figures or procedures may have changed -- verify against "
+                "the current published version._")
+    return ""
+
+
+def _format_section(c: dict, related=None) -> str:
+    """Render one chunk (legislation, D-Memo, or agreement) as cited Markdown."""
+    doc_type = c.get("doc_type", "legislation")
+    header = f"### {c['citation']} — {c['marginal_note']}".rstrip(" —")
+    location = " > ".join(p for p in (c["part"], c["division"]) if p)
+    lines = [header]
+    if location:
+        lines.append(f"*{location}*")
+    if doc_type == "memorandum":
+        lines.append("_CBSA administrative guidance — persuasive, not binding law._")
+        lines.append(f"(modified {_stated(c['current_to'])})")
+        age = _guidance_age_note(c["current_to"])
+        if age:
+            lines.append(age)
+    elif doc_type == "agreement":
+        lines.append("_Treasury Board collective agreement — a binding contract for "
+                     "this bargaining unit._")
+        lines.append(f"(in force to {_stated(c['current_to'])})")
+    elif doc_type == "directive":
+        lines.append("_National Joint Council directive — forms part of collective "
+                     "agreements; binding for the matters it covers._")
+        lines.append(f"(effective {_stated(c['current_to'])})")
+        age = _guidance_age_note(c["current_to"])
+        if age:
+            lines.append(age)
+    elif doc_type == "caselaw":
+        if "Immigration and Refugee Board" in c["part"]:
+            lines.append("_Immigration and Refugee Board jurisprudential guide "
+                         "— IRB members apply its reasoning to similar cases or "
+                         "explain why not; persuasive, and subject to revocation "
+                         "or to review by the Federal Court._")
+        elif "Board" in c["part"]:
+            lines.append("_Labour-board decision — a federal administrative "
+                         "tribunal's ruling; persuasive within the board's own "
+                         "jurisprudence, and subject to judicial review by the "
+                         "Federal Court of Appeal._")
+        else:
+            lines.append("_Court decision — binding precedent depending on the "
+                         "court and jurisdiction; confirm it has not been "
+                         "overturned on appeal or overtaken by later authority._")
+        lines.append(f"(decided {c['current_to'] or 'n/a'})")
+        if c["heading"]:
+            lines.append(f"Subject: {c['heading']}")
+    elif doc_type == "delegation":
+        lines.append("_Instrument of delegation and designation — it records "
+                     "which officials the Minister has delegated powers to, or "
+                     "designated for functions, under IRPA and the IRPR. "
+                     "Administrative; confirm it is still the current version._")
+        lines.append(f"(dated {c['current_to'] or 'n/a'})")
+    elif doc_type == "benefits":
+        lines.append("_Benefit-plan member booklet — a plain-language summary "
+                     "published by the plan administrator (Canada Life). The "
+                     "governing terms are in the plan's directive or rules, "
+                     "which prevail in any discrepancy; quote the booklet for "
+                     "what it says but flag that the plan rules control._")
+        lines.append(f"(booklet edition: {_stated(c['current_to'])})")
+        age = _guidance_age_note(c["current_to"])
+        if age:
+            lines.append(age)
+    elif doc_type == "union":
+        lines.append("_Union document (CIU/PSAC) — internal governance "
+                     "rules, member guides or locally negotiated scheduling "
+                     "instruments published by the bargaining agent. They "
+                     "bind the union and its members under the union's own "
+                     "rules; they are NOT employer policy and NOT part of "
+                     "the collective agreement itself (a VSSA operates "
+                     "under the FB agreement's variable-shift provisions). "
+                     "The union site may lag renegotiations — verify "
+                     "currency with the Branch._")
+        lines.append(f"(document currency: {_stated(c['current_to'])})")
+        age = _guidance_age_note(c["current_to"])
+        if age:
+            lines.append(age)
+    elif doc_type == "commentary":
+        lines.append("_**CURATED ANALYSIS — CanLex commentary, not a source "
+                     "of law.** A curated synthesis of the cited authorities; "
+                     "where it says no authority exists, the analysis is "
+                     "reasoned interpretation only. Always verify against the "
+                     "cited cases and guidance before relying on it._")
+        lines.append(f"(entry reviewed {_stated(c['current_to'])})")
+    elif c.get("status") == "not-in-force":
+        # Enacted but not yet in force: pending law from the Act's AMENDMENTS
+        # NOT IN FORCE schedule. Make it impossible to mistake for current law.
+        lines.append("_**⚠ NOT YET IN FORCE — this is an enacted amendment "
+                     "that has not come into force.** It is NOT current law; "
+                     "the provision it amends (if any) continues to read as "
+                     "consolidated. Cite it only as pending law, naming the "
+                     f"amending Act: {c['history'] or 'see text'}._ "
+                     f"(consolidation current to {c['current_to'] or 'n/a'})")
+    elif c.get("status") == "repealed":
+        # A repealed provision is no longer law; say so unmistakably so the
+        # model never presents the (near-empty) stub as if it were in force.
+        lines.append("_**⚠ REPEALED — this provision is no longer in force.** "
+                     "The repeal citation is in the text below; do not rely on "
+                     "it as current law, and tell the reader it has been "
+                     f"repealed._ (text current to {c['current_to'] or 'n/a'})")
+    else:
+        meta = [f"in force; text current to {c['current_to'] or 'n/a'}"]
+        if c.get("in_force"):
+            meta.append(f"this provision in force since {c['in_force']}")
+        if c["last_amended"]:
+            meta.append(f"last amended {c['last_amended']}")
+        lines.append(f"**Currency:** {'; '.join(meta)}. Does not reflect any "
+                     f"amendment that came into force after the 'current to' date.")
+    hl = c.get("highlight")
+    if hl:
+        label, snippet = hl
+        lines.append(f"**Most on point for this query:** "
+                     f"{c['citation']}{label} — {snippet}")
+    lines.append("")
+    lines.append(c["text"])
+    lines.append("")
+    if related:
+        provisions = related.get("provisions")
+        if provisions:
+            refs = "; ".join(f"s. {s} ({n})" if n else f"s. {s}"
+                             for s, n in provisions)
+            lines.append(f"Related provisions in this Act: {refs}")
+        regs = related.get("regulations")
+        if regs:
+            lines.append("Regulations made under this Act: "
+                         + "; ".join(f"{n} ({s})" for s, n in regs))
+        enabling = related.get("enabling_act")
+        if enabling:
+            lines.append(f"Made under: {enabling[1]} ({enabling[0]})")
+        memos = related.get("memoranda")
+        if memos:
+            lines.append("CBSA D-memoranda citing this section (guidance, not "
+                         "binding): " + ", ".join(memos))
+    if c["history"]:
+        if doc_type == "caselaw":
+            lines.append(f"Also reported: {c['history']}")
+        elif doc_type == "legislation":
+            lines.append(f"Amendment history: {c['history']}")
+        else:
+            lines.append(f"History: {c['history']}")
+    lines.append(f"Source: {c['source_url']}")
+    return "\n".join(lines)
+
+
+# The eight doc_type values, as a Literal so every client schema carries them
+# as an enum rather than free text (regex 'pattern' constraints are dropped by
+# several function-calling translators, notably Gemini's).
+DocType = Literal["legislation", "memorandum", "agreement", "directive",
+                  "caselaw", "delegation", "benefits", "commentary", "union"]
+
+# Longest alternative first, and 's'/'ss' only at a word boundary or before
+# a dotted number -- so 'section 20.1' loses the whole word (not its first
+# letter) and a real section value like 'Schedule I' is left alone.
+_SECTION_PREFIX = re.compile(
+    r"^(?:sections?\b|ss?\.(?=\s*\d)|ss?\b|§+)\s*", re.IGNORECASE)
+
+
+def _normalize_section(section: str) -> str:
+    """Accept the forms other tool outputs teach a client to write -- 's. 34',
+    'section 34', '34(1)(c)' -- and reduce them to the bare section number the
+    index keys on. Subsection parentheticals are truncated (the full section
+    text contains its subsections anyway)."""
+    s = _SECTION_PREFIX.sub("", section.strip())
+    s = s.split("(", 1)[0].strip().rstrip(".")
+    return s or section.strip()
+
+
+@mcp.tool(name="canlex_search_legislation",
+          annotations={"title": "Search Canadian Legislation", **_READONLY})
+def canlex_search_legislation(
+    query: Annotated[str, Field(
+        description="Natural-language legal question or keywords, e.g. "
+        "'detention review timelines' or 'inadmissibility for serious "
+        "criminality'.", min_length=2, max_length=500)],
+    top_k: Annotated[int, Field(
+        description="Number of sections to return (1-20). Use more for broad "
+        "questions.", ge=1, le=20)] = 6,
+    act: Annotated[Optional[str], Field(
+        description="Optional filter to a single Act, by short name, code or "
+        "full name (e.g. 'IRPA', 'I-2.5', 'Customs Act'). Omit to search "
+        "every loaded instrument.")] = None,
+    doc_type: Annotated[Optional[DocType], Field(
+        description="Optional filter by source type: 'legislation' (Acts and "
+        "regulations, incl. their Schedules), 'memorandum' (CBSA D-Memoranda "
+        "and operational guidance), 'agreement' (collective agreements), "
+        "'directive' (NJC directives), 'caselaw' (court and tribunal "
+        "decisions), 'delegation' (IRPA/IRPR delegation instruments), "
+        "'benefits' (benefit-plan member booklets), or 'commentary' "
+        "(CanLex's curated analysis datasets, e.g. US dispositions vs the "
+        "IRPA 'conviction' concept). Omit to search all.")] = None,
+    court: Annotated[Optional[str], Field(
+        description="Optional: restrict CASE-LAW results to a deciding "
+        "court/tribunal by name fragment, e.g. 'Supreme', 'Federal Court "
+        "of Appeal', 'Immigration and Refugee Board', 'FPSLREB'. Other "
+        "source types are unaffected.", max_length=60)] = None,
+    decided_after: Annotated[Optional[int], Field(
+        description="Optional: restrict CASE-LAW results to decisions from "
+        "this year onward, e.g. 2015. Other source types are unaffected.",
+        ge=1950, le=2100)] = None,
+) -> str:
+    """Search Canadian federal law, CBSA D-Memoranda, agreements, NJC directives,
+    and leading court decisions.
+
+    The CanLex corpus has nine kinds of source: federal Acts and regulations
+    with their Schedules (immigration, customs, criminal, firearms, drugs,
+    food/health, labour, privacy and more); CBSA D-Memoranda (administrative
+    guidance on how the CBSA applies customs and border law); Treasury Board
+    collective agreements (currently the FB / Border Services group); National
+    Joint Council directives (travel, relocation, isolated posts and more);
+    leading decisions of the courts and federal tribunals (Supreme Court,
+    Federal Court of Appeal and Federal Court, the IRB, and the FPSLREB and
+    CIRB labour boards); instruments of delegation and designation under IRPA
+    and the IRPR; member booklets of the public-service benefit plans;
+    CanLex's curated commentary (US-disposition and equivalency analysis);
+    and union documents (the CIU by-laws/regulations/branch rules, the PSAC
+    Constitution, member guides, and every published site VSSA).
+    Use this for ANY question about that material. It ranks results by
+    relevance and returns their full text so the answer can cite the actual
+    wording; an explicit section reference (e.g. "section 34") is always
+    surfaced. Each result is marked with its source type.
+
+    Returns:
+        str: Markdown with answering instructions followed by the matching
+        sections -- citation, Part/Division, currency dates, full text,
+        amendment history and source URL per result. Returns a "No ...
+        matched" message if nothing is found.
+
+    Examples:
+        - "What are the security grounds for inadmissibility?" -> query about s. 34.
+        - "detention review 48 hours" -> finds the detention-review provisions.
+        - Don't use to look up a known section number verbatim -- use canlex_get_section.
+    """
+    index = _index()
+    results = index.search(query, top_k=top_k, act=act, doc_type=doc_type,
+                           court=court, decided_after=decided_after)
+    _log_query(query, top_k, act, doc_type, results)
+    if not results:
+        if act:
+            known = sorted({c["act_short"] for c in index.chunks})
+            act_lc = act.strip().lower()
+            if not any(act_lc in (c["act_short"].lower(), c["act_code"].lower(),
+                                  c.get("act_name", "").lower())
+                       for c in index.chunks):
+                return (f"No results: the act filter '{act}' matches no loaded "
+                        f"instrument (short names accepted: "
+                        f"{', '.join(known)}). Retry without the filter, or "
+                        f"with one of those names.")
+        scope = f" in '{act}'" if act else ""
+        return (f"No results matched '{query}'{scope}. "
+                f"Try broader or different keywords, or call canlex_list_acts to see "
+                f"what is currently loaded.")
+    blocks = []
+    weak = results[0].get("confidence")
+    if weak is not None and weak < HEDGE_THRESHOLD:
+        blocks += [WEAK_MATCH_NOTE, ""]
+    blocks += [GROUNDING_NOTE, "",
+               f'{len(results)} relevant section(s) for: "{query}"']
+    for c in results:
+        blocks.append("")
+        blocks.append("---")
+        blocks.append("")
+        blocks.append(_format_section(c, index.related(c)))
+    return "\n".join(blocks)
+
+
+@mcp.tool(name="canlex_get_section",
+          annotations={"title": "Get a Legislation Section", **_READONLY})
+def canlex_get_section(
+    act: Annotated[str, Field(
+        description="Act short name or code, e.g. 'IRPA' or 'I-2.5'.",
+        min_length=1, max_length=60)],
+    section: Annotated[str, Field(
+        description="Section number, e.g. '34', '20.1', 's. 34' or "
+        "'34(1)(c)' -- prefixes and subsection parentheticals are accepted "
+        "and normalized.", min_length=1, max_length=40)],
+) -> str:
+    """Retrieve one specific section of legislation verbatim, by Act and section number.
+
+    Use this when the exact citation is already known (e.g. the user asks about
+    "IRPA section 34" or a provision cross-references "subsection 25.1(1)").
+
+    Returns:
+        str: Markdown with answering instructions followed by the section's citation,
+        Part/Division, 'current to' and 'last amended' dates, full provision text,
+        amendment history, and source URL. Returns an actionable error message (listing
+        the loaded Acts) if the Act or section is not found.
+
+    Examples:
+        - "Show me IRPA s. 34" -> act='IRPA', section='34'.
+        - "What does subsection 20.1 say" -> act='IRPA', section='20.1'.
+    """
+    index = _index()
+    result = index.get_section(act, section)
+    if result is None and section != _normalize_section(section):
+        result = index.get_section(act, _normalize_section(section))
+    if result is None:
+        acts = sorted({c["act_short"] for c in index.chunks})
+        return (f"Error: no section '{section}' found in '{act}'. "
+                f"Loaded Acts: {', '.join(acts) or 'none'}. Check the section number, "
+                f"or use canlex_search_legislation to locate the provision by topic.")
+    return GROUNDING_NOTE + "\n\n" + _format_section(result, index.related(result))
+
+
+def _load_dispositions():
+    """The curated US-dispositions dataset (data/curated/us_dispositions.json)."""
+    from canlex.config import DATA_DIR
+    path = DATA_DIR / "curated" / "us_dispositions.json"
+    if not path.exists():
+        return None
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+_STATE_NAMES = {
+    "al": "alabama", "ak": "alaska", "az": "arizona", "ar": "arkansas",
+    "ca": "california", "co": "colorado", "ct": "connecticut", "de": "delaware",
+    "fl": "florida", "ga": "georgia", "hi": "hawaii", "id": "idaho",
+    "il": "illinois", "in": "indiana", "ia": "iowa", "ks": "kansas",
+    "ky": "kentucky", "la": "louisiana", "me": "maine", "md": "maryland",
+    "ma": "massachusetts", "mi": "michigan", "mn": "minnesota",
+    "ms": "mississippi", "mo": "missouri", "mt": "montana", "ne": "nebraska",
+    "nv": "nevada", "nh": "new hampshire", "nj": "new jersey",
+    "nm": "new mexico", "ny": "new york", "nc": "north carolina",
+    "nd": "north dakota", "oh": "ohio", "ok": "oklahoma", "or": "oregon",
+    "pa": "pennsylvania", "ri": "rhode island", "sc": "south carolina",
+    "sd": "south dakota", "tn": "tennessee", "tx": "texas", "ut": "utah",
+    "vt": "vermont", "va": "virginia", "wa": "washington",
+    "wv": "west virginia", "wi": "wisconsin", "wy": "wyoming",
+}
+
+
+def _match_dispositions(data, disposition, state):
+    """Score dataset entries against the query; return the best matches.
+
+    Token-overlap over the entry's names/aliases, with a bonus when the
+    entry's state_variations cover the requested state. Deliberately simple:
+    the vocabulary is small and controlled."""
+    q_tokens = set(re.findall(r"[a-z0-9]+", disposition.lower()))
+    state_lc = (state or "").strip().lower()
+    state_name = _STATE_NAMES.get(state_lc, state_lc)
+    scored = []
+    for e in data.get("dispositions", []):
+        best = 0.0
+        for name in e["names"]:
+            n_tokens = set(re.findall(r"[a-z0-9]+", name.lower()))
+            if not n_tokens:
+                continue
+            overlap = len(q_tokens & n_tokens) / len(n_tokens)
+            best = max(best, overlap)
+        if state_name and any(
+                state_name in (v.get("state", "").lower())
+                or v.get("state", "").lower() in (state_lc, state_name)
+                for v in e.get("state_variations", [])):
+            best += 0.15
+        if best > 0.3:
+            scored.append((best, e))
+    scored.sort(key=lambda t: -t[0])
+    return [e for _, e in scored[:2]]
+
+
+def _load_equivalency():
+    from canlex.config import DATA_DIR
+    path = DATA_DIR / "curated" / "us_equivalency.json"
+    if not path.exists():
+        return None
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+# Tool registration order mirrors the s. 36 screening chain (step 1
+# disposition, step 2 equivalency, step 3 rehabilitation) so clients that
+# list tools in registration order present the workflow forwards.
+@mcp.tool(name="canlex_us_disposition",
+          annotations={"title": "US Disposition vs IRPA 'Conviction'", **_READONLY})
+def canlex_us_disposition(
+    disposition: Annotated[str, Field(
+        description="The US disposition, in the traveller's or record's own "
+        "words, e.g. 'deferred adjudication', 'nolo contendere plea', "
+        "'withheld adjudication', 'expunged conviction', 'ACD' -- or 'all' "
+        "for the complete breakdown of every covered disposition type and "
+        "state variation.", min_length=1, max_length=200)],
+    state: Annotated[Optional[str], Field(
+        description="Optional US state (name or 2-letter code), e.g. 'TX' or "
+        "'Florida' -- the analysis can vary by state.",
+        max_length=30)] = None,
+) -> str:
+    """Assess whether a US criminal disposition is a "conviction" for IRPA
+    s. 36 inadmissibility purposes, from CanLex's curated, authority-cited
+    dataset of US disposition types (deferred adjudication, withheld
+    adjudication, nolo pleas, expungements, state pardons, diversion, and
+    more), with state-by-state variations.
+
+    This is STEP 1 of a s. 36(1)(b)/(2)(b) screening (is there a conviction
+    at all?); step 2 is offence equivalency (canlex_us_equivalency) and step
+    3 is rehabilitation (canlex_rehabilitation). Every answer is CURATED
+    COMMENTARY, not law: entries cite their authorities, and entries with no
+    authority say so and label their reasoning as interpretation.
+
+    Returns:
+        str: Markdown -- the matching entry/entries with authorities and
+        status flags, or the governing principles plus an honest no-entry
+        statement when nothing matches.
+    """
+    from canlex.commentary import BANNER, _entry_text
+    data = _load_dispositions()
+    if data is None:
+        return ("The curated US-dispositions dataset is not present in this "
+                "deployment. Use canlex_search_legislation (doc_type "
+                "'caselaw') for the governing principles: Canada (MCI) v "
+                "Saini, 2001 FCA 311, and IRPA s. 36(3).")
+    if disposition.strip().lower() in ("all", "list", "list all", "*"):
+        lines = [BANNER, "",
+                 "Every covered disposition type (and state variation), with "
+                 "its conviction verdict for IRPA s. 36. Query a specific "
+                 "disposition for the full analysis and authorities.", ""]
+        for e in data.get("dispositions", []):
+            lines.append(f"- {e['names'][0]} -- conviction: "
+                         f"{e['is_conviction'].upper()} "
+                         f"[{e['status']}]")
+            for v in e.get("state_variations", []):
+                if v.get("state", "").lower() == "general":
+                    continue
+                flag = (v.get("is_conviction") or e["is_conviction"]).upper()
+                lines.append(f"    - {v['state']}: {flag}")
+        lines.append("")
+        lines.append("Reminder: even a non-conviction disposition can ground "
+                     "inadmissibility under the act branch, IRPA "
+                     "s. 36(1)(c)/(2)(c).")
+        return "\n".join(lines)
+    matches = _match_dispositions(data, disposition, state)
+    parts = []
+    if matches:
+        for e in matches:
+            parts.append(_entry_text(e))
+    else:
+        parts.append(BANNER)
+        parts.append(
+            f"No curated entry matches '{disposition}'"
+            + (f" ({state})" if state else "") + ". "
+            "That does not mean the disposition is or is not a conviction -- "
+            "it means CanLex has not analyzed this disposition type. Apply "
+            "the governing principles (below) to the disposition's actual "
+            "features: was guilt admitted or found? is the disposition final? "
+            "does an analogous Canadian non-conviction regime exist?")
+    for m in data.get("methodology", []):
+        if m.get("id") == "conviction-framework":
+            parts.append("---\n\nGoverning framework:\n\n" + m["text"])
+            break
+    parts.append("Confirm any cited decision's current status with "
+                 "canlex_case, and retrieve its full text with "
+                 "canlex_search_legislation (doc_type 'caselaw').")
+    return "\n\n".join(parts)
+
+
+@mcp.tool(name="canlex_us_equivalency",
+          annotations={"title": "US Offence Equivalency", **_READONLY})
+def canlex_us_equivalency(
+    offense: Annotated[str, Field(
+        description="The US offence as charged or described, e.g. 'DUI', "
+        "'petty theft', 'aggravated assault', 'possession of a controlled "
+        "substance', 'disorderly conduct'.",
+        min_length=2, max_length=200)],
+) -> str:
+    """Map a common US offence to its Canadian equivalent and the resulting
+    IRPA s. 36 inadmissibility branch (serious criminality vs criminality vs
+    none), from CanLex's curated pairing table with corpus-verified maximum
+    penalties.
+
+    This is step 2 of a s. 36 analysis (step 1: is the disposition a
+    conviction -- canlex_us_disposition; step 3: rehabilitation --
+    canlex_rehabilitation, which takes this tool's max-penalty answer as
+    input). Covers the COMMON pairings (DUI, theft, assault, fraud, drugs,
+    cannabis, disorderly conduct...); anything unusual gets the Hill
+    methodology and an honest no-entry statement.
+
+    Returns:
+        str: Markdown -- the pairing with penalty, branch, caveats and
+        authorities, or the Hill framework when nothing matches.
+    """
+    from canlex.commentary import BANNER
+    data = _load_equivalency()
+    if data is None:
+        return ("The equivalency dataset is not present in this deployment. "
+                "Apply Hill v Canada (MEI) (1987), 73 NR 315 (FCA): compare "
+                "essential elements, or the evidence of what was done, or "
+                "both; remember IRPA s. 36(3)(a) hybrid deeming.")
+    q = set(re.findall(r"[a-z0-9]+", offense.lower()))
+    scored = []
+    for p in data.get("pairings", []):
+        best = 0.0
+        for t in p["us_terms"]:
+            toks = set(re.findall(r"[a-z0-9]+", t.lower()))
+            if toks:
+                best = max(best, len(q & toks) / len(toks))
+        if best > 0.4:
+            scored.append((best, p))
+    scored.sort(key=lambda x: -x[0])
+    parts = [BANNER]
+    if scored:
+        for _, p in scored[:2]:
+            lines = [f"US offence: {'; '.join(p['us_terms'][:5])}",
+                     f"Canadian equivalent: {p['canadian_offence']}",
+                     f"Maximum penalty (verified): {p['penalty']}",
+                     f"Inadmissibility branch: {p['branch']}", "",
+                     p["analysis"]]
+            if p.get("caveats"):
+                lines.append("")
+                lines += [f"- CAVEAT: {c}" for c in p["caveats"]]
+            if p.get("authorities"):
+                lines += ["", "Authorities:"]
+                lines += [f"- {a['cite']} ({a['court']}): {a['holding']}"
+                          for a in p["authorities"]]
+            parts.append("\n".join(lines))
+    else:
+        parts.append(
+            f"No curated pairing matches '{offense}'. Apply the Hill "
+            "framework (below) to the actual foreign statute -- compare "
+            "essential elements, or the evidence of what was done, or both.")
+    for m in data.get("methodology", []):
+        if m["id"] == "equivalency-framework":
+            parts.append("---\n\nGoverning framework:\n\n" + m["text"])
+            break
+    parts.append("Retrieve the Canadian provisions with canlex_get_section "
+                 "to verify elements and penalties; feed the max-penalty "
+                 "answer into canlex_rehabilitation for step 3.")
+    return "\n\n".join(parts)
+
+
+@mcp.tool(name="canlex_rehabilitation",
+          annotations={"title": "Criminal Rehabilitation Assessment", **_READONLY})
+def canlex_rehabilitation(
+    basis: Annotated[Literal["foreign-conviction", "foreign-act",
+                             "canada-conviction"], Field(
+        description="How the inadmissibility arises: 'foreign-conviction' "
+        "(IRPA s. 36(1)(b)/(2)(b)), 'foreign-act' (s. 36(1)(c)/(2)(c)), or "
+        "'canada-conviction' (s. 36(1)(a)/(2)(a)).")],
+    max_penalty_lt_10: Annotated[bool, Field(
+        description="True if the equivalent Canadian offence's maximum term "
+        "of imprisonment is LESS THAN 10 years (remember IRPA s. 36(3)(a): "
+        "hybrid offences count as indictable at their indictable maximum).")],
+    count: Annotated[Literal["one", "two-plus-summary-only", "multiple"], Field(
+        description="'one' (a single conviction or act), "
+        "'two-plus-summary-only' (two or more, all summary-equivalent), or "
+        "'multiple' (more than one indictable-equivalent matter).")],
+    years_elapsed: Annotated[int, Field(
+        ge=0, le=100,
+        description="Full years since the day after completion of the ENTIRE "
+        "imposed sentence (conviction bases) or since the act was committed "
+        "(act basis).")],
+    clean_record: Annotated[bool, Field(
+        description="True if the person has no other convictions or "
+        "committed acts anywhere (the IRPR s. 18(2)(i)-(vii) conditions, "
+        "simplified -- the answer carries a caveat to confirm in detail).")],
+) -> str:
+    """Assess whether a person is deemed rehabilitated, eligible to apply for
+    individual rehabilitation, or neither, under IRPA s. 36(3)(c) and IRPR
+    ss. 17-18.1 -- the time-based cures for criminal inadmissibility.
+
+    Rules verified against the current IRPR consolidation: deemed
+    rehabilitation (automatic) for a single <10-year-maximum foreign
+    conviction after 10 clean years, two-plus summary-only convictions after
+    5 years, or a single committed act after 10 years; individual
+    rehabilitation applications after 5 years (available even for serious
+    criminality, which can never be deemed); record suspensions for in-Canada
+    convictions. This is step 3 of a s. 36 screening: step 1 is whether the
+    disposition is a conviction (canlex_us_disposition), step 2 is offence
+    equivalency (which also supplies the max-penalty input here).
+
+    Returns:
+        str: The assessment with the governing provision, explanation and
+        mandatory caveats. Curated logic -- verify against the cited
+        provisions via canlex_get_section.
+    """
+    from canlex.rehab import assess
+    from canlex.commentary import BANNER
+    r = assess(basis, max_penalty_lt_10, count, years_elapsed, clean_record)
+    lines = [BANNER, "",
+             f"REHABILITATION STATUS: {r['status'].upper().replace('-', ' ')}",
+             f"Governing provision(s): {r['provision']}", "",
+             r["explanation"], "", "Caveats:"]
+    lines += [f"- {c}" for c in r["caveats"]]
+    lines.append("")
+    lines.append("Retrieve the provisions with canlex_get_section (act "
+                 "'IRPR', sections '17'/'18'/'18.1'; act 'IRPA', section "
+                 "'36').")
+    return "\n".join(lines)
+
+
+@mcp.tool(name="canlex_list_acts",
+          annotations={"title": "List Loaded Legislation", **_READONLY})
+def canlex_list_acts() -> str:
+    """List what the CanLex corpus contains -- Acts and regulations, CBSA
+    D-Memoranda, collective agreements, NJC directives, leading cases,
+    delegation instruments, benefit-plan booklets, and CanLex's curated
+    commentary datasets.
+
+    Use this to learn the scope and currency of the corpus before searching, or to
+    report it to the user.
+
+    Returns:
+        str: Markdown grouped by source type.
+    """
+    index = _index()
+    acts: dict[str, dict] = {}
+    agreements: dict[str, dict] = {}
+    directives: dict[str, dict] = {}
+    cases: dict[str, dict] = {}
+    delegations: dict[str, dict] = {}
+    benefits: dict[str, dict] = {}
+    commentary: dict[str, dict] = {}
+    union_docs: dict[str, dict] = {}
+    memo_numbers: set[str] = set()
+    memo_chunks = 0
+    memo_date = ""
+    fr_chunks = 0
+    fr_acts: set[str] = set()
+    for c in index.chunks:
+        if c.get("lang", "en") == "fr":
+            # French twins share their English instrument's identity; count
+            # them once as coverage rather than doubling every act's rows.
+            fr_chunks += 1
+            fr_acts.add(c["act_code"])
+            continue
+        doc_type = c.get("doc_type", "legislation")
+        if doc_type == "memorandum":
+            memo_numbers.add(c["section"])
+            memo_chunks += 1
+            memo_date = max(memo_date, c["current_to"] or "")
+        elif doc_type == "agreement":
+            entry = agreements.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        elif doc_type == "directive":
+            entry = directives.setdefault(c["act_code"], {
+                "short": c["act_short"], "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        elif doc_type == "caselaw":
+            entry = cases.setdefault(c["act_code"], {
+                "name": c["act_name"], "decided": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        elif doc_type == "delegation":
+            entry = delegations.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        elif doc_type == "benefits":
+            entry = benefits.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        elif doc_type == "commentary":
+            entry = commentary.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        elif doc_type == "union":
+            entry = union_docs.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+        else:
+            entry = acts.setdefault(c["act_code"], {
+                "short": c["act_short"], "name": c["act_name"],
+                "code": c["act_code"], "current_to": c["current_to"], "count": 0,
+            })
+            entry["count"] += 1
+    lines = ["# CanLex corpus", "", "## Enacted law"]
+    if fr_chunks:
+        lines.insert(1, f"_Bilingual: {len(fr_acts)} instruments also carry "
+                        f"their parallel French text ({fr_chunks} articles); "
+                        f"a French-language query searches both._")
+    for a in sorted(acts.values(), key=lambda x: x["short"]):
+        lines.append(f"- **{a['short']}** — {a['name']} ({a['code']}): "
+                     f"{a['count']} sections, current to {a['current_to'] or 'n/a'}")
+    if memo_numbers:
+        lines += ["", "## CBSA guidance",
+                  f"- **D-Memoranda** — {len(memo_numbers)} memoranda "
+                  f"({memo_chunks} sections), modified up to {memo_date or 'n/a'}. "
+                  f"Administrative guidance on customs and border law; "
+                  f"persuasive, not binding."]
+    if agreements:
+        lines += ["", "## Collective agreements"]
+        for a in sorted(agreements.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} articles, "
+                         f"in force to {a['current_to'] or 'n/a'}")
+    if directives:
+        lines += ["", "## NJC directives"]
+        for a in sorted(directives.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}**: {a['count']} sections, "
+                         f"effective {a['current_to'] or 'n/a'}")
+    if cases:
+        lines += ["", "## Case law"]
+        for cite, a in sorted(cases.items(), key=lambda kv: kv[1]["decided"]):
+            lines.append(f"- **{a['name']}**, {cite}: {a['count']} excerpts, "
+                         f"decided {a['decided'] or 'n/a'}")
+    if delegations:
+        lines += ["", "## Delegation instruments"]
+        for a in sorted(delegations.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} items, "
+                         f"dated {a['current_to'] or 'n/a'}")
+    if benefits:
+        lines += ["", "## Benefit plans"]
+        for a in sorted(benefits.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} "
+                         f"sections, booklet edition {a['current_to'] or 'n/a'}. "
+                         f"Plain-language member guide; the plan rules govern.")
+    if union_docs:
+        lines += ["", "## Union documents (CIU/PSAC)"]
+        for a in sorted(union_docs.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} "
+                         f"sections, currency {a['current_to'] or 'n/a'}. "
+                         f"Bargaining-agent document; not employer policy.")
+    if commentary:
+        lines += ["", "## Curated commentary (not law)"]
+        for a in sorted(commentary.values(), key=lambda x: x["short"]):
+            lines.append(f"- **{a['short']}** — {a['name']}: {a['count']} "
+                         f"entries, reviewed {a['current_to'] or 'n/a'}. "
+                         f"CanLex-authored analysis of the cited authorities; "
+                         f"clearly bannered, never a source of law.")
+    lines += ["", "Search with canlex_search_legislation; filter by doc_type "
+              "(legislation / memorandum / agreement / directive / caselaw / "
+              "delegation / benefits / commentary). Fetch a known provision "
+              "with canlex_get_section, or a case's citations with canlex_case."]
+    return "\n".join(lines)
+
+
+@mcp.tool(name="canlex_whats_new",
+          annotations={"title": "What Changed Recently", **_READONLY})
+def canlex_whats_new(
+    since: Annotated[str, Field(
+        description="ISO date (YYYY-MM-DD): report amendments and additions "
+        "on or after this date.", pattern=r"^\d{4}-\d{2}-\d{2}$")],
+) -> str:
+    """List what has changed in the corpus's law since a date: provisions
+    amended or newly in force on or after it, plus each guidance source's
+    newest modification date.
+
+    Use for questions like "has IRPA changed since March?" or "what did the
+    last refresh pick up?". Changes reflect the ingested consolidations --
+    an amendment newer than the corpus's 'current to' dates is invisible
+    here; say so when the date asked about is very recent.
+
+    Returns:
+        str: Markdown -- amended/new provisions grouped by Act, then
+        guidance sources' newest dates.
+    """
+    index = _index()
+    amended: dict[str, list] = {}
+    guidance_newest: dict[str, str] = {}
+    for c in index.chunks:
+        if c.get("lang", "en") == "fr":
+            continue
+        if c.get("doc_type", "legislation") == "legislation":
+            stamp = max(c.get("last_amended") or "", c.get("in_force") or "")
+            if stamp >= since and c.get("section"):
+                amended.setdefault(c["act_short"], []).append(
+                    (c["section"], c.get("marginal_note", ""), stamp))
+        else:
+            key = c.get("act_short") or c.get("doc_type")
+            cur = c.get("current_to") or ""
+            if cur > guidance_newest.get(key, ""):
+                guidance_newest[key] = cur
+    lines = [f"# Changes since {since}", ""]
+    if amended:
+        lines.append("## Provisions amended or newly in force")
+        for act in sorted(amended):
+            secs = sorted({(s, n, d) for s, n, d in amended[act]},
+                          key=lambda t: t[2], reverse=True)
+            lines.append(f"**{act}** ({len(secs)} provisions):")
+            for s, n, d in secs[:25]:
+                lines.append(f"- s. {s} ({n}) — {d}")
+            if len(secs) > 25:
+                lines.append(f"- ...and {len(secs) - 25} more.")
+    else:
+        lines.append("No ingested provision shows an amendment or "
+                     "coming-into-force date on or after that date.")
+    recent_guidance = {k: v for k, v in guidance_newest.items() if v >= since}
+    if recent_guidance:
+        lines += ["", "## Guidance sources modified since then"]
+        for k, v in sorted(recent_guidance.items(), key=lambda kv: -ord(kv[1][0])
+                           if kv[1] else 0):
+            lines.append(f"- {k}: {v}")
+    lines += ["", "_Reflects the ingested consolidations only; verify "
+              "anything newer than the corpus's 'current to' dates upstream._"]
+    return "\n".join(lines)
+
+
+@mcp.tool(name="canlex_citing",
+          annotations={"title": "What Cites This Provision", **_READONLY})
+def canlex_citing(
+    act: Annotated[str, Field(
+        description="Act short name or code, e.g. 'IRPA'.",
+        min_length=1, max_length=60)],
+    section: Annotated[str, Field(
+        description="Section number, e.g. '34' (prefixes accepted).",
+        min_length=1, max_length=40)],
+) -> str:
+    """Reverse cross-reference lookup: everything in the corpus that cites a
+    given provision -- related provisions in the same Act, regulations made
+    under it, and the CBSA D-memoranda/guidance citing the section --
+    without fetching the section's own text first.
+
+    Returns:
+        str: Markdown lists of citing/related material, or a not-found
+        message.
+    """
+    index = _index()
+    result = index.get_section(act, section)
+    if result is None and section != _normalize_section(section):
+        result = index.get_section(act, _normalize_section(section))
+    if result is None:
+        return (f"No section '{section}' in '{act}'. Use canlex_list_acts "
+                f"for the loaded instruments.")
+    related = index.related(result) or {}
+    lines = [f"# Material citing {result['citation']}", ""]
+    memos = related.get("memoranda")
+    if memos:
+        lines.append("**Guidance citing this section** (persuasive, not "
+                     "binding): " + ", ".join(memos))
+    provisions = related.get("provisions")
+    if provisions:
+        lines.append("**Related provisions in this Act:** "
+                     + "; ".join(f"s. {s} ({n})" if n else f"s. {s}"
+                                 for s, n in provisions))
+    regs = related.get("regulations")
+    if regs:
+        lines.append("**Regulations made under this Act:** "
+                     + "; ".join(f"{n} ({s})" for s, n in regs))
+    enabling = related.get("enabling_act")
+    if enabling:
+        lines.append(f"**Made under:** {enabling[1]} ({enabling[0]})")
+    if len(lines) == 2:
+        lines.append("Nothing in the corpus cross-references this section.")
+    lines += ["", "For case law interpreting it, search with "
+              "canlex_search_legislation (doc_type 'caselaw')."]
+    return "\n".join(lines)
+
+
+@mcp.tool(name="canlex_toc",
+          annotations={"title": "Act Table of Contents", **_READONLY})
+def canlex_toc(
+    act: Annotated[str, Field(
+        description="Act short name or code, e.g. 'IRPA' or 'Customs Act'.",
+        min_length=1, max_length=60)],
+) -> str:
+    """Browse an Act's structure: its Parts/Divisions and the sections under
+    each, with their marginal notes. Use when you know the Act but not the
+    section number, or to survey what an Act covers.
+
+    Returns:
+        str: Markdown outline. Very large Acts (e.g. the Criminal Code)
+        return Parts with section ranges rather than every section.
+    """
+    index = _index()
+    a = act.strip().lower()
+    secs = [c for c in index.chunks
+            if c.get("doc_type", "legislation") == "legislation"
+            and c.get("lang", "en") == "en"
+            and a in (c["act_short"].lower(), c["act_code"].lower(),
+                      c.get("act_name", "").lower())
+            and c.get("section") and not c["id"].endswith(tuple(
+                f"-p{k}" for k in range(2, 40)))]
+    if not secs:
+        known = sorted({c["act_short"] for c in index.chunks
+                        if c.get("doc_type", "legislation") == "legislation"})
+        return (f"No legislation matches '{act}'. Loaded: "
+                + ", ".join(known))
+    by_part: dict[str, list] = {}
+    seen = set()
+    for c in secs:
+        if c["section"] in seen:
+            continue
+        seen.add(c["section"])
+        by_part.setdefault(c.get("part") or "(no Part)", []).append(
+            (c["section"], c.get("marginal_note", "")))
+    name = secs[0]["act_name"]
+    total = sum(len(v) for v in by_part.values())
+    lines = [f"# {name} — contents ({total} sections)", ""]
+    detailed = total <= 150
+    for part, entries in by_part.items():
+        lines.append(f"## {part}" if part != "(no Part)" else "## —")
+        if detailed:
+            for s, n in entries:
+                lines.append(f"- s. {s}" + (f" — {n}" if n else ""))
+        else:
+            first, last = entries[0][0], entries[-1][0]
+            lines.append(f"- ss. {first}–{last} ({len(entries)} sections)")
+    lines += ["", "Fetch any section with canlex_get_section."]
+    return "\n".join(lines)
+
+
+_CITATOR = None
+
+
+def _citator():
+    """Load and cache the CanLII citator on first use."""
+    global _CITATOR
+    if _CITATOR is None:
+        from canlex.citator import Citator
+        _CITATOR = Citator()
+    return _CITATOR
+
+
+def _format_case(report: dict) -> str:
+    """Render a CanLII citator report as Markdown."""
+    meta = report["meta"]
+    lines = [f"## {meta.get('title', '(untitled case)')}",
+             meta.get("citation", "").strip()]
+    facts = []
+    if meta.get("decisionDate"):
+        facts.append(f"decided {meta['decisionDate']}")
+    if meta.get("docketNumber"):
+        facts.append(f"docket {meta['docketNumber']}")
+    if facts:
+        lines.append("(" + "; ".join(facts) + ")")
+    if meta.get("keywords"):
+        lines.append(f"**Keywords:** {meta['keywords']}")
+    if meta.get("topics"):
+        lines.append(f"**Topics:** {meta['topics']}")
+    if meta.get("url"):
+        lines.append(f"CanLII: {meta['url']}")
+    lines.append("")
+    lines.append("_CanLII citator data (live). This is case metadata and the "
+                 "citation graph only -- not the judgment text; follow the CanLII "
+                 "link to read the decision. A case is binding precedent depending "
+                 "on the court and jurisdiction._")
+
+    def case_list(label, block):
+        rows = [f"\n### {label}: {block['total']}"]
+        for item in block["items"]:
+            rows.append(f"- {item.get('title', '')} — {item.get('citation', '')}")
+        extra = block["total"] - len(block["items"])
+        if extra > 0:
+            rows.append(f"  ...and {extra} more.")
+        return rows
+
+    lines += case_list("Cited by (later cases citing this one)", report["citingCases"])
+    lines += case_list("Cites (authorities this case relies on)", report["citedCases"])
+
+    legis = report["citedLegislations"]
+    lines.append(f"\n### Legislation cited: {legis['total']}")
+    for item in legis["items"]:
+        kind = item.get("type", "")
+        lines.append(f"- {item.get('title', '')} — {item.get('citation', '')}"
+                     + (f" [{kind}]" if kind else ""))
+    extra = legis["total"] - len(legis["items"])
+    if extra > 0:
+        lines.append(f"  ...and {extra} more.")
+    return "\n".join(lines)
+
+
+@mcp.tool(name="canlex_case",
+          annotations={"title": "CanLII Case Citator", "readOnlyHint": True,
+                       "destructiveHint": False, "idempotentHint": True,
+                       "openWorldHint": True})
+def canlex_case(
+    case_url: Annotated[str, Field(
+        description="A Canadian case, given either as a full canlii.org URL "
+        "or -- for a Supreme Court, Federal Court of Appeal, Federal Court, "
+        "FPSLREB or CIRB decision -- its neutral citation (e.g. '2019 SCC "
+        "65', '2016 FCA 93', '2023 FPSLREB 12'). For other courts, supply "
+        "the canlii.org URL; find it by web search if you only have the "
+        "case name.", min_length=8, max_length=400)],
+) -> str:
+    """Look up a Canadian case on CanLII and return its citation graph.
+
+    Returns the case's metadata plus its citator: the cases it cites, the cases
+    that cite it (its treatment and how leading it is), and the legislation it
+    cites -- live from the CanLII API. Use it to gauge whether a decision is
+    still good law -- how heavily and how recently it has been cited.
+
+    Supply either a canlii.org URL or, for a Supreme Court / Federal Court of
+    Appeal / Federal Court decision, its neutral citation (e.g. '2019 SCC 65') --
+    the citation a canlex_search_legislation result already shows. This returns
+    metadata and the citation graph only, NOT the judgment text -- follow the
+    CanLII link for that. A call takes ~15-20 seconds (the API is rate-limited).
+
+    Args:
+        params (CaseInput): contains case_url -- a canlii.org URL or a neutral
+            citation.
+
+    Returns:
+        str: Markdown -- the case's title, neutral citation, date, docket and
+        topics; how many cases cite it (with examples); how many it cites; and the
+        legislation it cites. Returns an error message if the URL is not a
+        recognized canlii.org case URL, or if the CanLII API is unavailable.
+    """
+    try:
+        citator = _citator()
+    except Exception:
+        return ("This deployment has no citator configured (the live CanLII "
+                "lookup is unavailable here). Fall back to a web search for "
+                "the case's CanLII page and note that its later treatment "
+                "was not verified.")
+    try:
+        report = citator.case_report(case_url)
+    except Exception as exc:
+        return f"CanLII lookup failed: {type(exc).__name__}: {exc}"
+    if "error" in report:
+        return f"Error: {report['error']}"
+    return _format_case(report)
+
+
+@mcp.prompt(name="us_record_screening",
+            description="Guided IRPA s. 36 screening of a US criminal record: "
+            "disposition -> equivalency -> rehabilitation, using the three "
+            "CanLex screening tools in order with the right hand-offs.")
+def us_record_screening(disposition: str, state: str = "",
+                        offense: str = "", sentence_completed_year: str = "",
+                        ) -> str:
+    """The three-step s. 36 screening chain as a reusable client prompt."""
+    facts = [f"Disposition: {disposition}"]
+    if state:
+        facts.append(f"State: {state}")
+    if offense:
+        facts.append(f"Offence: {offense}")
+    if sentence_completed_year:
+        facts.append(f"Sentence completed: {sentence_completed_year}")
+    return (
+        "Screen this US criminal record for inadmissibility under IRPA "
+        "s. 36, using CanLex's three screening tools IN ORDER and carrying "
+        "each step's output into the next:\n\n" + "\n".join(facts) + "\n\n"
+        "STEP 1 — canlex_us_disposition (pass the disposition and state): is "
+        "this a 'conviction' at all? If the verdict is NO or LIKELY NO, note "
+        "that the act branch (s. 36(1)(c)/(2)(c)) may still apply and say "
+        "so; a NO verdict usually ends the conviction-branch analysis.\n"
+        "STEP 2 — canlex_us_equivalency (pass the offence): what Canadian "
+        "offence does it equate to, and is the branch serious criminality "
+        "(s. 36(1)(b)) or criminality (s. 36(2)(b))? Note the maximum "
+        "penalty — whether it is under 10 years feeds step 3.\n"
+        "STEP 3 — canlex_rehabilitation: compute full years since sentence "
+        "completion from today's date, pass basis='foreign-conviction', the "
+        "max-penalty answer from step 2, the count, the elapsed years, and "
+        "whether the record is otherwise clean.\n\n"
+        "Then give a single bottom-line assessment with every authority "
+        "cited, every uncertainty flagged, and the reminder that this is "
+        "legal information, not legal advice, and that an officer's "
+        "determination governs at the port of entry.")
+
+
+@mcp.resource("canlex://grounding-note",
+              name="CanLex answering instructions",
+              description="The grounding/citation/hedging instructions "
+              "prepended to every search result.")
+def _res_grounding() -> str:
+    return GROUNDING_NOTE
+
+
+@mcp.resource("canlex://corpus",
+              name="CanLex corpus inventory",
+              description="Everything the corpus contains, grouped by source "
+              "type, with currency dates.")
+def _res_corpus() -> str:
+    return canlex_list_acts()
+
+
+@mcp.resource("canlex://methodology/conviction",
+              name="US disposition conviction framework",
+              description="The governing framework for whether a US "
+              "disposition is an IRPA s. 36 'conviction'.")
+def _res_conviction_framework() -> str:
+    data = _load_dispositions()
+    if data:
+        for m in data.get("methodology", []):
+            if m.get("id") == "conviction-framework":
+                return m["text"]
+    return "The dispositions dataset is not present in this deployment."
+
+
+@mcp.resource("canlex://methodology/equivalency",
+              name="Offence equivalency framework (Hill)",
+              description="The Hill v MEI methodology for equating a foreign "
+              "offence to a Canadian one, with the s. 36(3)(a) and Tran rules.")
+def _res_equivalency_framework() -> str:
+    data = _load_equivalency()
+    if data:
+        for m in data.get("methodology", []):
+            if m.get("id") == "equivalency-framework":
+                return m["text"]
+    return "The equivalency dataset is not present in this deployment."
+
+
+if __name__ == "__main__":
+    try:
+        idx = _index()
+        print(f"CanLex MCP: {len(idx.chunks)} sections loaded.", file=sys.stderr)
+    except Exception as exc:  # surfaced in the MCP client's server logs
+        print(f"CanLex MCP: failed to load legislation index: {exc}\n"
+              f"Run 'py -m canlex.ingest' first.", file=sys.stderr)
+        sys.exit(1)
+    # CANLEX_HTTP switches on the remote (web) transport; default is local stdio.
+    if os.environ.get("CANLEX_HTTP"):
+        print("CanLex MCP: serving over streamable-HTTP.", file=sys.stderr)
+        mcp.run(transport="streamable-http")
+    else:
+        mcp.run()

canlex/sweep.py

@@ -0,0 +1,132 @@
+"""Coordinate-descent tuning sweep for the four retrieval knobs.
+
+Runs the 141-question eval repeatedly, varying one knob at a time while holding
+the others at their current best. Picks the value that maximises Hit@5 (with
+MRR as the tiebreak) and continues to the next knob. One pass through all four
+knobs is usually enough to settle.
+
+The knobs are read from env vars at index-load time (see canlex/index.py), so
+each combination is exercised in a fresh subprocess of canlex.eval. Outputs go
+to data/eval/sweep.log and a compact JSON summary to data/eval/sweep.json.
+
+    py -m canlex.sweep
+"""
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+LOG = ROOT / "data" / "eval" / "sweep.log"
+SUMMARY = ROOT / "data" / "eval" / "sweep.json"
+
+# (env var, list of candidate values, current default). The defaults match the
+# literals in canlex/index.py; values bracket each on a roughly geometric grid.
+KNOBS = [
+    ("CANLEX_MN_WEIGHT",         [0.0012, 0.0024, 0.005, 0.01],   0.0024),
+    ("CANLEX_MN_CAP",            [0.006, 0.012, 0.024, 0.05],     0.012),
+    ("CANLEX_REG_PENALTY",       [0.004, 0.008, 0.016, 0.032],    0.008),
+    ("CANLEX_BACKMATTER_PENALTY",[0.004, 0.008, 0.016, 0.032],    0.008),
+]
+
+
+_METRIC_RE = re.compile(
+    r"Hit@1:\s*([\d.]+).*?Hit@3:\s*([\d.]+).*?Hit@5:\s*([\d.]+).*?"
+    r"Hit@10:\s*([\d.]+).*?MRR:\s*([\d.]+)", re.S)
+
+
+def _run_eval(env_overrides: dict[str, float]) -> dict[str, float]:
+    """Run canlex.eval once with the given env overrides; return metrics dict."""
+    env = dict(os.environ)
+    env["CANLEX_EVAL_SWEEP"] = "1"   # holdout questions stay sweep-blind
+    for k, v in env_overrides.items():
+        env[k] = f"{v}"
+    proc = subprocess.run(
+        [sys.executable, "-u", "-m", "canlex.eval"],
+        capture_output=True, text=True, env=env, cwd=ROOT,
+    )
+    if proc.returncode != 0:
+        raise RuntimeError(f"eval failed (exit {proc.returncode}):\n"
+                           f"{proc.stderr[-800:]}")
+    m = _METRIC_RE.search(proc.stdout)
+    if not m:
+        raise RuntimeError(f"could not parse eval output:\n{proc.stdout[-800:]}")
+    h1, h3, h5, h10, mrr = (float(x) for x in m.groups())
+    n_misses = proc.stdout.count("miss(es)")          # 0 if we end up at 100
+    return {"hit1": h1, "hit3": h3, "hit5": h5, "hit10": h10, "mrr": mrr,
+            "stdout": proc.stdout}
+
+
+def _score(metrics: dict[str, float]) -> tuple[float, float]:
+    """Order by Hit@5 then MRR (both higher is better)."""
+    return (metrics["hit5"], metrics["mrr"])
+
+
+def main():
+    LOG.parent.mkdir(parents=True, exist_ok=True)
+    log = LOG.open("w", encoding="utf-8")
+    log.write(f"# CanLex tuning sweep -- {time.strftime('%Y-%m-%d %H:%M:%S')}\n")
+    print(f"# CanLex tuning sweep -- writing log to {LOG}\n")
+
+    current = {name: default for name, _values, default in KNOBS}
+    all_runs: list[dict] = []
+
+    # Baseline at current defaults.
+    print("Baseline:", current)
+    log.write(f"\nBaseline: {current}\n")
+    baseline = _run_eval(current)
+    print(f"  Hit@5={baseline['hit5']:.3f}  MRR={baseline['mrr']:.3f}")
+    log.write(f"  Hit@5={baseline['hit5']:.3f}  MRR={baseline['mrr']:.3f}\n")
+    all_runs.append({"values": dict(current), "metrics":
+                     {k: baseline[k] for k in ("hit1", "hit3", "hit5", "hit10", "mrr")}})
+    best_metrics = baseline
+
+    for name, values, _default in KNOBS:
+        print(f"\nSweeping {name} in {values} (others held at {current})...")
+        log.write(f"\nSweeping {name} in {values} (others held at {current})\n")
+        local_best = (current[name], _score(best_metrics), best_metrics)
+        for v in values:
+            if v == current[name]:
+                # Re-use the already-measured baseline at this knob value.
+                metrics = best_metrics
+            else:
+                run_values = dict(current, **{name: v})
+                metrics = _run_eval(run_values)
+            row = (f"  {name}={v!r:<8s} -> Hit@1={metrics['hit1']:.3f} "
+                   f"Hit@3={metrics['hit3']:.3f} Hit@5={metrics['hit5']:.3f} "
+                   f"Hit@10={metrics['hit10']:.3f} MRR={metrics['mrr']:.3f}")
+            print(row); log.write(row + "\n"); log.flush()
+            all_runs.append({"values": dict(current, **{name: v}),
+                             "metrics": {k: metrics[k] for k in
+                                         ("hit1", "hit3", "hit5", "hit10", "mrr")}})
+            score = _score(metrics)
+            if score > local_best[1]:
+                local_best = (v, score, metrics)
+        if local_best[0] != current[name]:
+            print(f"  ! {name}: {current[name]} -> {local_best[0]}  "
+                  f"(Hit@5 {best_metrics['hit5']:.3f} -> {local_best[2]['hit5']:.3f})")
+            log.write(f"  ! {name}: {current[name]} -> {local_best[0]}\n")
+        current[name] = local_best[0]
+        best_metrics = local_best[2]
+
+    print(f"\nBest: {current}")
+    print(f"  Hit@1={best_metrics['hit1']:.3f}  Hit@3={best_metrics['hit3']:.3f}  "
+          f"Hit@5={best_metrics['hit5']:.3f}  Hit@10={best_metrics['hit10']:.3f}  "
+          f"MRR={best_metrics['mrr']:.3f}")
+    log.write(f"\nBest: {current}\n  Hit@1={best_metrics['hit1']:.3f} "
+              f"Hit@5={best_metrics['hit5']:.3f}  MRR={best_metrics['mrr']:.3f}\n")
+    log.close()
+    SUMMARY.write_text(json.dumps({
+        "best": current,
+        "best_metrics": {k: best_metrics[k] for k in
+                         ("hit1", "hit3", "hit5", "hit10", "mrr")},
+        "runs": all_runs,
+    }, indent=2), encoding="utf-8")
+    print(f"\nLog: {LOG}\nSummary: {SUMMARY}")
+
+
+if __name__ == "__main__":
+    main()

canlex/synonyms.py

@@ -0,0 +1,125 @@
+"""Query-side expansion of legal abbreviations and informal terms.
+
+Statutes use formal wording -- "application for protection", "removal order" --
+but users (and a model drafting a search) reach for everyday shorthand: "PRRA",
+"H&C", "deportation". Before retrieval, expand_query() appends the canonical
+statutory terms for any abbreviation or nickname it recognises, so the BM25 and
+semantic stages can match the provision's actual language. It only ever ADDS
+words -- the user's own phrasing is left untouched -- and the cross-encoder
+reranker still sees the original query, so precision is unaffected.
+
+    python -m canlex.synonyms "PRRA eligibility and an H&C application"
+"""
+import re
+import sys
+
+# (trigger, canonical terms to append). The trigger is a regex fragment matched
+# case-insensitively as a whole word. Keep this list high-precision: an entry
+# earns its place only when the shorthand is unambiguous in Canadian border,
+# immigration, customs, financial-crime or labour law.
+_SYNONYMS = [
+    # Immigration and refugee law
+    (r"prra", "pre-removal risk assessment application for protection"),
+    (r"pre[- ]removal risk assessment", "application for protection"),
+    (r"h\s*&\s*c", "humanitarian and compassionate"),
+    (r"rad", "refugee appeal division"),
+    (r"rpd", "refugee protection division"),
+    (r"iad", "immigration appeal division"),
+    (r"irb", "immigration and refugee board"),
+    (r"trp", "temporary resident permit"),
+    (r"deportation", "removal order"),
+    # "No right to appeal (a removal order)" is how users ask about IRPA
+    # s. 64, but the section itself never says "removal order" or "right" --
+    # its vocabulary is "no appeal may be made ... inadmissible on grounds
+    # of ... serious criminality". Bridge the gap.
+    (r"no (?:right (?:to|of) )?appeal",
+     "no appeal may be made inadmissibility serious criminality"),
+    # "Standard of proof" is how users ask about IRPA s. 33, whose own words
+    # are "reasonable grounds to believe" that the facts "have occurred".
+    (r"standard of proof", "reasonable grounds to believe facts constitute"),
+    # "Apply for (criminal) rehabilitation" points at IRPR s. 17, a tiny
+    # section titled only "Prescribed period" whose vocabulary is the
+    # five-year period after "completion of an imposed sentence".
+    (r"appl(?:y|ication) for (?:criminal )?rehabilitation",
+     "prescribed period five years completion of imposed sentence"),
+    (r"misrep", "misrepresentation"),
+    (r"ircc", "immigration refugees and citizenship canada"),
+    (r"eta", "electronic travel authorization"),
+    (r"trv", "temporary resident visa"),
+    (r"arc", "authorization to return to canada removal order"),
+    (r"danger opinion", "danger to the public refoulement protection"),
+    (r"ministerial relief", "national interest exception inadmissibility"),
+    (r"poe", "port of entry"),
+    (r"flagpol(?:e|ing)", "leaving canada and reentering port of entry"),
+    # The single most common US-screening term: bridge DUI shorthand to the
+    # Criminal Code s. 320.14 vocabulary the equivalency analysis turns on.
+    (r"(?:dwi|dui|drunk driving)",
+     "operation while impaired blood alcohol concentration conveyance"),
+    (r"personal exemption",
+     "returning persons exemption duty free allowance"),
+    (r"(?:nexus|trusted traveller)",
+     "presentation of persons alternative manner authorization"),
+    # Border and customs
+    (r"cbsa", "canada border services agency"),
+    (r"bsos?", "border services officer"),
+    (r"amps", "administrative monetary penalty system"),
+    # Financial-crime and labour
+    (r"fintrac", "financial transactions and reports analysis centre"),
+    (r"njc", "national joint council"),
+    # French -> English bridges. The corpus is English-only, but officers work
+    # bilingually and drafting agents emit French terms; with the tokenizer's
+    # accent folding these triggers match either accented or folded forms.
+    # High-precision statutory anchors only.
+    (r"erar", "pre-removal risk assessment application for protection"),
+    (r"asfc", "canada border services agency"),
+    (r"interdiction de territoire", "inadmissibility inadmissible"),
+    (r"grande criminalit[ée]", "serious criminality"),
+    (r"criminalit[ée]", "criminality convicted offence"),
+    (r"renvoi", "removal order"),
+    (r"saisie?s?", "seizure seized goods"),
+    # 'devises' is the everyday word for currency; the PCMLTFA's French text
+    # says 'espèces et effets' -- bridge to both statutory vocabularies.
+    (r"devises", "currency monetary instruments espèces effets"),
+    (r"r[ée]sidents? permanents?", "permanent resident"),
+    (r"[ée]trangers?", "foreign national"),
+    (r"d[ée]tention", "detention detained review"),
+    (r"r[ée]fugi[ée]s?", "refugee protection convention"),
+    (r"douanes?", "customs duties"),
+    (r"marchandises?", "goods imported"),
+    (r"d[ée]claration", "report declaration"),
+    (r"armes? [àa] feu", "firearm weapon"),
+    (r"contr[ôo]le", "examination search"),
+    (r"agents? des services frontaliers", "border services officer"),
+    (r"permis de travail", "work permit"),
+    (r"permis d'[ée]tudes", "study permit"),
+    (r"visa de r[ée]sident temporaire", "temporary resident visa"),
+    (r"r[ée]adaptation", "rehabilitation prescribed period"),
+    (r"expulsion", "deportation removal order"),
+    (r"passeurs?", "human smuggling organized entry"),
+    (r"fouille", "search of the person strip search"),
+    (r"griefs?", "grievance adjudication"),
+    (r"convention collective", "collective agreement"),
+    (r"heures suppl[ée]mentaires", "overtime hours of work"),
+]
+
+_COMPILED = [(re.compile(rf"\b{trigger}\b", re.IGNORECASE), expansion)
+             for trigger, expansion in _SYNONYMS]
+
+
+def expand_query(query):
+    """Return `query` with canonical statutory terms appended for every legal
+    abbreviation it contains; return it unchanged if it contains none."""
+    additions = [exp for pattern, exp in _COMPILED if pattern.search(query)]
+    if not additions:
+        return query
+    return f"{query} {' '.join(additions)}"
+
+
+def main():
+    query = " ".join(sys.argv[1:]) or "PRRA eligibility and an H&C application"
+    print(f"query:    {query}")
+    print(f"expanded: {expand_query(query)}")
+
+
+if __name__ == "__main__":
+    main()

canlex/tariff_schedule.py

@@ -0,0 +1,285 @@
+"""Ingest Chapters 98 and 99 of the Customs Tariff Schedule.
+
+The Customs Tariff's Schedule is the Harmonized System classification of goods
+-- chapters 1-97 categorise every imported good for duty purposes. Those 97
+chapters are huge and outside CanLex's scope, but **chapters 98 and 99 are
+different**: they carry Canada's "special classification" provisions, which
+matter for almost every CBSA border interaction:
+
+  - Chapter 98 (non-commercial): traveller exemptions, settler's effects,
+    Canadian goods returned, ancestral household effects, conveyances
+    temporarily imported by a resident, etc.
+  - Chapter 99 (commercial): temporary importations, end-use programs,
+    government imports, reduced-rate goods for specific industries.
+
+Source: the CBSA's HTML edition of the current Customs Tariff. The Justice
+Laws XML for the Act (C-54.011) does NOT include the Schedule.
+
+Chunking is one chunk per 4-digit HEADING (98.01, 98.02, ...) plus one chunk
+per chapter for its Notes and Subheading Notes -- a heading is the natural
+unit of legal classification (the eight- and ten-digit items below it are the
+same rule with finer rate granularity).
+
+    py -m canlex.tariff_schedule [--allow-shrink]
+"""
+import re
+import sys
+from collections import defaultdict
+
+from bs4 import BeautifulSoup
+
+from ._common import (fetch_cached, norm_ws as _norm, preserve_dropped,
+                      stored_chunks, write_corpus)
+from .config import PROCESSED_DIR, RAW_DIR
+
+RAW = RAW_DIR / "tariff_schedule"
+OUT = PROCESSED_DIR / "tariff_schedule.json"
+
+# The 2026 edition is the current Customs Tariff at the time of writing. To
+# refresh: bump the year in the URL and `edition` once CBSA publishes the next.
+SOURCES = {
+    "ch98": {
+        "code": "ch98",
+        "chapter": "98",
+        "title": "Special classification provisions — non-commercial",
+        "url": ("https://www.cbsa-asfc.gc.ca/trade-commerce/tariff-tarif/"
+                "2026/html/00/ch98-eng.html"),
+        "edition": "2026",
+    },
+    "ch99": {
+        "code": "ch99",
+        "chapter": "99",
+        "title": "Special classification provisions — commercial",
+        "url": ("https://www.cbsa-asfc.gc.ca/trade-commerce/tariff-tarif/"
+                "2026/html/00/ch99-eng.html"),
+        "edition": "2026",
+    },
+}
+
+
+def _fetch(url, dest):
+    return fetch_cached(url, dest)
+
+
+def _heading_of(item):
+    """Map a tariff-item code to its 4-digit heading. The schedule uses three
+    levels:
+      - '98.01'       (4 digits)           -- heading
+      - '9801.10'     (6 digits)           -- subheading
+      - '9801.10.10'  (8 digits)           -- tariff item
+    All three roll up to heading '98.01'."""
+    digits = re.sub(r"\D", "", item)
+    if len(digits) < 4:
+        return None
+    return f"{digits[:2]}.{digits[2:4]}"
+
+
+def _chapter_part(chapter):
+    """The `part` every chunk of a chapter carries -- and, because nothing else
+    on a chunk names its chapter, the per-chapter identity preserve_failed
+    matches on. Derived in one place so the two stay in step."""
+    return f"Schedule, Chapter {chapter}"
+
+
+def _collect_notes(main):
+    """Return the chapter's Notes + Subheading Notes as a single text block."""
+    out = []
+    for label in ("Notes", "Subheading Notes"):
+        h2 = main.find(
+            "h2",
+            string=lambda s, lbl=label: s and lbl in s and (
+                lbl != "Notes" or "Subheading" not in s))
+        if not h2:
+            continue
+        parts = []
+        sib = h2.find_next_sibling()
+        while sib and sib.name != "h2":
+            t = _norm(sib.get_text(" ", strip=True))
+            if t:
+                parts.append(t)
+            sib = sib.find_next_sibling()
+        if parts:
+            out.append(f"{label}:\n" + "\n".join(parts))
+    return "\n\n".join(out)
+
+
+def parse_chapter(html, src):
+    """Parse one Customs Tariff Schedule chapter into chunks."""
+    soup = BeautifulSoup(html, "html.parser")
+    main = soup.find("main")
+    if main is None:
+        return []
+    for sup in main.find_all("sup"):
+        sup.decompose()
+
+    chunks = []
+    chapter = src["chapter"]
+    citation_root = f"Customs Tariff, Sched., Ch. {chapter}"
+
+    # Chapter Notes + Subheading Notes -- one chunk per chapter.
+    notes_body = _collect_notes(main)
+    if notes_body:
+        chunks.append({
+            "id": f"tariff-sched-ch{chapter}-notes",
+            "doc_type": "legislation",
+            "act_code": "C-54.011",
+            "act_short": "Customs Tariff",
+            "act_name": "Customs Tariff",
+            "section": f"Sch-Ch{chapter}-Notes",
+            "marginal_note": (f"Chapter {chapter} Notes — "
+                              f"{src['title']}"),
+            "part": _chapter_part(chapter),
+            "division": "",
+            "heading": src["title"],
+            "text": notes_body,
+            "history": "",
+            "last_amended": "",
+            "current_to": src["edition"],
+            "citation": f"{citation_root}, Notes",
+            "source_url": src["url"],
+        })
+
+    # Walk every row in the schedule table, grouping by 4-digit heading.
+    table = main.find("table")
+    if table is None:
+        return chunks
+
+    rows_by_heading = defaultdict(list)   # heading -> list of (item, ss, desc, unit, mfn, pref)
+    heading_desc = {}                     # heading -> the 4-digit row's description
+
+    for tr in table.find_all("tr"):
+        cells = tr.find_all(["td", "th"], recursive=False)
+        if not cells:
+            continue
+        first = _norm(cells[0].get_text(" ", strip=True))
+        if not first or first == "Tariff Item":
+            continue                       # header row or blank
+        heading = _heading_of(first)
+        if heading is None:
+            continue
+
+        def col(i):
+            return _norm(cells[i].get_text(" ", strip=True)) if i < len(cells) else ""
+
+        ss, desc, unit, mfn, pref = col(1), col(2), col(3), col(4), col(5)
+
+        # A 4-digit row carries only the heading number and description -- store
+        # it once. Otherwise it's a subheading/item row; remember its rate cells.
+        digits = re.sub(r"\D", "", first)
+        if len(digits) == 4 and desc and heading not in heading_desc:
+            heading_desc[heading] = desc
+        rows_by_heading[heading].append((first, ss, desc, unit, mfn, pref))
+
+    for heading in sorted(rows_by_heading):
+        desc = heading_desc.get(heading, "")
+        rows = rows_by_heading[heading]
+        if not desc:
+            # No bare 4-digit row -- fall back to the first row's description.
+            desc = next((r[2] for r in rows if r[2]), "")
+
+        # Format the chunk: heading description first, then the items as a
+        # readable list. Suppress description on item/subheading rows that just
+        # repeat the heading desc verbatim (very common in this Schedule).
+        lines = []
+        for item, ss, item_desc, unit, mfn, pref in rows:
+            d = re.sub(r"\D", "", item)
+            if len(d) == 4:
+                continue                   # the heading row, already in desc
+            label = f"  {item}"
+            if ss:
+                label += f" (SS {ss})"
+            extras = []
+            if item_desc and item_desc != desc:
+                extras.append(item_desc)
+            rate = []
+            if mfn:
+                rate.append(f"MFN {mfn}")
+            if pref:
+                rate.append(f"Pref: {pref}")
+            if unit and unit != "-":
+                rate.append(f"Unit {unit}")
+            if rate:
+                extras.append(" | ".join(rate))
+            tail = " — " + "; ".join(extras) if extras else ""
+            lines.append(label + tail)
+
+        body = f"Heading {heading} — {desc}"
+        if lines:
+            body += "\n\nTariff items:\n" + "\n".join(lines)
+
+        chunks.append({
+            "id": f"tariff-sched-{heading.replace('.', '-')}",
+            "doc_type": "legislation",
+            "act_code": "C-54.011",
+            "act_short": "Customs Tariff",
+            "act_name": "Customs Tariff",
+            "section": f"Sch-{heading}",
+            "marginal_note": desc[:200],
+            "part": _chapter_part(chapter),
+            "division": "",
+            "heading": src["title"],
+            "text": body,
+            "history": "",
+            "last_amended": "",
+            "current_to": src["edition"],
+            "citation": f"Customs Tariff, Sched., heading {heading}",
+            "source_url": src["url"],
+        })
+
+    return chunks
+
+
+def preserve_failed(chunks, failures, stored):
+    """Re-attach the last-good chunks of chapters this run could not rebuild.
+
+    Only two chapters make up this corpus, so losing one is losing a third to
+    two thirds of it -- and a CBSA page that has been reshaped (or a fetch that
+    died mid-run) is indistinguishable from a chapter that legitimately
+    vanished until a human looks. Serving last week's copy of ch. 98 beats
+    serving no traveller exemptions at all. Restricted to the chapters that
+    actually failed. Returns (chunks, preserved chapter identities).
+    """
+    failed = {_chapter_part(chapter) for chapter, _why in failures}
+    return preserve_dropped(chunks, stored, key=lambda c: c["part"],
+                            only=failed)
+
+
+def build(allow_shrink=False):
+    all_chunks, failures = [], []
+    for src in SOURCES.values():
+        print(f"Ingesting Customs Tariff Schedule {src['code']} ...")
+        try:
+            html = _fetch(src["url"], RAW / f"{src['code']}.html")
+            chunks = parse_chapter(html, src)
+        except Exception as exc:
+            print(f"  !! {src['code']}: {type(exc).__name__}: {exc}")
+            failures.append((src["chapter"], f"{type(exc).__name__}: {exc}"))
+            continue
+        if not chunks:
+            # A chapter never parses to nothing legitimately -- an empty result
+            # means the page moved or changed shape, so treat it as a failure
+            # rather than letting it silently shrink the corpus.
+            print(f"  !! {src['code']}: no chunks parsed")
+            failures.append((src["chapter"], "no chunks parsed"))
+            continue
+        all_chunks.extend(chunks)
+        print(f"  {len(chunks)} chunks")
+
+    all_chunks, preserved = preserve_failed(all_chunks, failures,
+                                            stored_chunks(OUT))
+    if preserved:
+        print(f"  kept the last-good copy of {len(preserved)} chapter(s): "
+              f"{', '.join(preserved)}")
+    if not write_corpus(OUT, all_chunks, indent=1, allow_shrink=allow_shrink):
+        return False
+    print(f"\n{len(all_chunks)} tariff-schedule chunks from "
+          f"{len(SOURCES) - len(failures)} chapter(s) -> {OUT}")
+    return True
+
+
+def main():
+    sys.exit(0 if build(allow_shrink="--allow-shrink" in sys.argv) else 1)
+
+
+if __name__ == "__main__":
+    main()

data/curated/drafts/equivalency-pairings-draft.json

@@ -0,0 +1,52 @@
+{
+  "_draft": "DRAFT FOR MAINTAINER REVIEW 2026-07-23 — three new pairings for data/curated/us_equivalency.json. Every Canadian maximum verified against the corpus text of the current C-46/IRPA consolidations. NOT deployed: append to the 'pairings' array, bump 'reviewed', then py -m canlex.commentary && py -m canlex.embed && eval && push.",
+  "pairings": [
+    {
+      "id": "weapons-carry",
+      "us_terms": ["unlawful carrying of a weapon", "CCW violation", "carrying a concealed weapon", "unlawful possession of a firearm", "felon in possession", "UCW"],
+      "canadian_offence": "Criminal Code s. 90 (carrying a concealed weapon); s. 91 (unauthorized possession of a firearm); s. 92 (possession knowing possession is unauthorized); s. 88 (possession of a weapon for a purpose dangerous to the public peace)",
+      "penalty": "Hybrid throughout; indictable maximums verified: s. 90 five years; s. 91 five years; s. 92 ten years; s. 88 ten years",
+      "branch": "SPLITS ON THE EQUIVALENT: ss. 90/91 (5 years) -> CRIMINALITY, IRPA s. 36(2)(b); ss. 92/88 (10 years) -> SERIOUS CRIMINALITY, s. 36(1)(b)",
+      "status": "guidance-only",
+      "analysis": "The most common US weapons conviction — carrying without a permit — usually equates to s. 90 or s. 91 (five-year maximums): criminality, deemed rehabilitation available after 10 years. The branch escalates to serious criminality where the record shows KNOWLEDGE the possession was unauthorized in a s. 92 sense, or a dangerous purpose (s. 88). 'Felon in possession' convictions typically map to s. 91/92 — the s. 92 knowledge element is often satisfied on those facts. The Hill evidence branch matters more here than for most pairings: US statutes bundle conduct Canada splits across four provisions.",
+      "caveats": [
+        "Constitutional-carry states: lawful-there conduct can still equate to a Canadian offence, but with no US conviction only the act branch (s. 36(2)(c)) is in play.",
+        "A weapon lawful in the US but prohibited in Canada (e.g. brass knuckles) engages the classification regulations (SOR/98-462) — check the device, not just the conduct.",
+        "Using a firearm in the commission of an offence (s. 85) carries consecutive mandatory minimums — an aggravated US charge deserves its own equivalency analysis."
+      ],
+      "authorities": []
+    },
+    {
+      "id": "sexual-assault",
+      "us_terms": ["sexual assault", "criminal sexual conduct", "sexual battery", "rape", "statutory rape", "sexual abuse"],
+      "canadian_offence": "Criminal Code s. 271 (sexual assault); s. 151 (sexual interference, complainant under 16); ss. 272/273 (weapon/bodily harm/aggravated)",
+      "penalty": "s. 271 hybrid, indictable maximum 10 years (14 where the complainant is under 16); s. 151 hybrid, 14 years with a one-year mandatory minimum; verified from the corpus text",
+      "branch": "SERIOUS CRIMINALITY — IRPA s. 36(1)(b) in effectively every configuration (every plausible equivalent carries a 10-year-or-greater maximum)",
+      "status": "guidance-only",
+      "analysis": "Any US sexual-offence conviction equates to a 10-years-or-more Canadian offence: serious criminality, no deemed rehabilitation ever, individual rehabilitation application (5 years post-sentence) or TRP the only routes. 'Statutory rape' variants map to s. 151 (14-year maximum) — age thresholds differ by state, so verify the victim's age against Canada's under-16 line (and the close-in-age exceptions in s. 150.1) via the Hill evidence branch.",
+      "caveats": [
+        "US registration-only dispositions (e.g. a deferred adjudication that still requires registration) remain step-1 questions — registration does not itself make a disposition a conviction.",
+        "Canada's s. 150.1 close-in-age defences can defeat equivalency for peer-age statutory-rape convictions — this is the classic Li point that available Canadian defences form part of the comparison.",
+        "Misdemeanor 'sexual battery' in some states covers non-consensual touching that still equates to s. 271 — the 10-year branch does not depend on the US felony label."
+      ],
+      "authorities": [
+        {"cite": "Li v Canada (MCI), [1997] 1 FC 235 (CA)", "court": "Federal Court of Appeal", "holding": "Equivalency compares essential elements including available defences; no retrial of the foreign case."}
+      ]
+    },
+    {
+      "id": "immigration-fraud",
+      "us_terms": ["visa fraud", "immigration fraud", "false statement on immigration application", "alien smuggling", "harboring an alien", "marriage fraud"],
+      "canadian_offence": "IRPA s. 127 (misrepresentation offence; penalty s. 128); IRPA ss. 122-123 (document offences); IRPA s. 117 (organizing entry — smuggling); Criminal Code s. 368 (use of forged document)",
+      "penalty": "s. 128: hybrid, indictable maximum 5 years; s. 123: up to 14 years for document use/trafficking; s. 117: up to life (with mandatory minimums where aggravated); verified from the corpus text",
+      "branch": "SPLITS: false statements/marriage fraud -> s. 127/128 (5 years) = CRIMINALITY, s. 36(2)(b); document fraud with use/trafficking -> s. 123 (14 years) = SERIOUS CRIMINALITY; alien smuggling -> s. 117 = SERIOUS CRIMINALITY",
+      "status": "guidance-only",
+      "analysis": "A US conviction for lying on an immigration form (18 USC 1546 false-statement branch) generally equates to IRPA s. 127: five-year maximum, criminality only — deemed rehabilitation available. The same US statute's document-use branch reaches s. 123's 14-year maximum: serious criminality. Alien smuggling (8 USC 1324) equates to IRPA s. 117 — serious criminality, and s. 37 organized-criminality may also be in play. NOTE: a misrepresentation to CANADIAN authorities separately grounds s. 40 inadmissibility without any conviction; this pairing addresses foreign convictions only.",
+      "caveats": [
+        "18 USC 1546 is a classic Hill problem: one US provision straddles two Canadian branches — the record of what was actually done decides between s. 127 (5 years) and s. 123 (14 years).",
+        "'Harboring' convictions vary widely; minor-assistance facts may equate to s. 117's lower tiers or fail equivalency entirely — evidence branch required.",
+        "Marriage fraud: the US conviction equates to s. 127 (criminality), but the underlying facts may independently engage s. 40 misrepresentation if made to IRCC/CBSA."
+      ],
+      "authorities": []
+    }
+  ]
+}

data/curated/us_equivalency.json

@@ -0,0 +1,285 @@
+{
+ "reviewed": "2026-07-15",
+ "_provenance": "Authored 2026-07-12. Every Canadian maximum penalty verified against the corpus text of the Criminal Code (C-46) and CDSA (C-38.8), current consolidations; branch classifications apply IRPA s. 36(3)(a) (hybrid offences deemed indictable). Tran v Canada, 2017 SCC 50 (in-corpus) supplies the commission-date rule for assessing maximum penalties. Equivalency methodology per Hill v Canada (MEI) (1987), 73 NR 315 (FCA), as set out in IRB Legal Services, Sponsorship Appeals Ch. 2 (Criminal Refusals). These are the COMMON pairings only -- an unusual offence needs the full Hill analysis against the actual foreign statute.",
+ "methodology": [
+  {
+   "id": "equivalency-framework",
+   "title": "How offence equivalency works (step 2 of the s. 36 analysis)",
+   "text": "Equivalency asks whether the foreign offence, if committed in Canada, would constitute a federal offence of the required seriousness. The governing test is Hill v Canada (MEI) (1987), 73 NR 315 (FCA): equivalency can be established (1) by comparing the precise wording and essential elements of the two provisions; (2) by examining the evidence of what the person actually did, to see whether it would satisfy the essential elements of the Canadian offence; or (3) by a combination of both. Key rules: (a) HYBRID DEEMING -- IRPA s. 36(3)(a): an offence prosecutable either way in Canada is deemed indictable, at its indictable maximum, even if it would likely proceed summarily; (b) TIMING -- Tran v Canada, 2017 SCC 50: the maximum penalty is assessed as of the DATE OF COMMISSION, so a later increase in the Canadian maximum does not retroactively convert criminality into serious criminality; (c) THRESHOLDS -- Canadian property offences split at $5,000 (theft, fraud, mischief), so the value in the foreign record often decides the branch; (d) the comparison is with the CANADIAN offence's elements -- a foreign offence broader than its Canadian counterpart requires the evidence branch of Hill to see what was actually done; (e) purely SUMMARY Canadian equivalents (e.g. causing a disturbance, s. 175) cannot ground s. 36(2)(b) inadmissibility on a single conviction -- only the two-or-more-convictions branch reaches them."
+  }
+ ],
+ "pairings": [
+  {
+   "id": "dui",
+   "us_terms": [
+    "DUI",
+    "DWI",
+    "OWI",
+    "OUI",
+    "driving under the influence",
+    "driving while intoxicated",
+    "operating under the influence",
+    "drunk driving",
+    "impaired driving"
+   ],
+   "canadian_offence": "Criminal Code s. 320.14 (operation while impaired / over the legal limit); punishment s. 320.19",
+   "penalty": "Hybrid; indictable maximum 10 years (s. 320.19(1)(a): 'liable to imprisonment for a term of not more than 10 years')",
+   "branch": "SERIOUS CRIMINALITY -- IRPA s. 36(1)(b): via s. 36(3)(a) hybrid deeming, a single ordinary US DUI conviction equates to an offence punishable by 10 years",
+   "status": "judicially-considered",
+   "analysis": "The single most consequential everyday pairing: one US misdemeanor DUI grounds serious criminality, because Canada's impaired-driving offence is hybrid with a 10-year indictable maximum since December 18, 2018. Deemed rehabilitation is therefore unavailable (10-year-or-more maximum); only individual rehabilitation or a TRP cures it.",
+   "caveats": [
+    "TIMING (Tran, 2017 SCC 50): for offences COMMITTED before December 18, 2018, the equivalent was the former s. 253 (maximum 5 years) -- criminality under s. 36(2)(b) only, not serious criminality, and deemed rehabilitation remains available.",
+    "Zero-tolerance administrative per-se findings are not convictions (see the DUI entry in the dispositions helper); this pairing addresses criminal DUI convictions.",
+    "A US 'wet reckless' plea-down maps to careless/dangerous driving analysis, not s. 320.14 -- check what was actually convicted."
+   ],
+   "authorities": [
+    {
+     "cite": "Tran v Canada (Public Safety), 2017 SCC 50",
+     "court": "Supreme Court of Canada",
+     "holding": "The maximum term for equivalency/seriousness purposes is assessed as of the date of commission; subsequent increases do not apply retrospectively."
+    }
+   ]
+  },
+  {
+   "id": "theft-under",
+   "us_terms": [
+    "petty theft",
+    "petit larceny",
+    "shoplifting",
+    "misdemeanor theft",
+    "theft under"
+   ],
+   "canadian_offence": "Criminal Code s. 334(b) (theft not exceeding $5,000)",
+   "penalty": "Hybrid; indictable maximum 2 years (s. 334(b)(i))",
+   "branch": "CRIMINALITY -- IRPA s. 36(2)(b) (foreign national); a permanent resident is NOT inadmissible on this alone",
+   "status": "guidance-only",
+   "analysis": "Ordinary shoplifting equates to theft under $5,000: hybrid, deemed indictable, 2-year maximum. One conviction makes a foreign national inadmissible for criminality; two convictions not arising from a single occurrence also engage s. 36(2)(b). Deemed rehabilitation after 10 years (single) per IRPR s. 18(2)(a).",
+   "caveats": [
+    "Convert the stolen value at the offence date: a US 'felony grand theft' at a low state threshold (some states: $500+) may still be theft UNDER $5,000 in Canadian terms -- the CANADIAN threshold controls the branch.",
+    "Value over $5,000 -> s. 334(a), 10-year maximum -> serious criminality (see theft-over)."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "theft-over",
+   "us_terms": [
+    "grand theft",
+    "grand larceny",
+    "felony theft",
+    "auto theft",
+    "grand theft auto"
+   ],
+   "canadian_offence": "Criminal Code s. 334(a) (theft exceeding $5,000); motor vehicle theft s. 333.1",
+   "penalty": "Indictable maximum 10 years (s. 334(a): 'not more than 10 years'); motor vehicle theft (s. 333.1) hybrid, 10 years",
+   "branch": "SERIOUS CRIMINALITY -- IRPA s. 36(1)(b), where the value genuinely exceeds $5,000 (or the subject is a motor vehicle under s. 333.1)",
+   "status": "guidance-only",
+   "analysis": "Theft of property over $5,000 (Canadian dollars, at the offence date) is a 10-year offence: serious criminality; no deemed rehabilitation.",
+   "caveats": [
+    "The US felony label does not control -- verify the actual value against the Canadian $5,000 line; borderline values with exchange-rate movement deserve the evidence branch of Hill."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "assault-simple",
+   "us_terms": [
+    "simple assault",
+    "misdemeanor assault",
+    "battery",
+    "simple battery",
+    "assault in the fourth degree"
+   ],
+   "canadian_offence": "Criminal Code s. 266 (assault)",
+   "penalty": "Hybrid; indictable maximum 5 years (s. 266(a): 'not exceeding five years')",
+   "branch": "CRIMINALITY -- IRPA s. 36(2)(b)",
+   "status": "guidance-only",
+   "analysis": "US simple assault/battery generally equates to s. 266: hybrid, 5-year maximum -- criminality, not serious criminality. Deemed rehabilitation available after 10 years (single conviction).",
+   "caveats": [
+    "Bodily injury, a weapon, or choking moves the equivalent to s. 267 (10 years) -> serious criminality (see assault-aggravated).",
+    "Domestic-violence variants equate to the same assault provisions; the DV character matters to sentencing and to some US dispositions, not to the Canadian equivalency.",
+    "Some US assault statutes cover mere threats -- threat-only conduct maps to s. 264.1 (uttering threats, 5 years), same branch."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "assault-aggravated",
+   "us_terms": [
+    "aggravated assault",
+    "assault with a deadly weapon",
+    "ADW",
+    "felony assault",
+    "assault causing injury",
+    "domestic battery with injury"
+   ],
+   "canadian_offence": "Criminal Code s. 267 (assault with a weapon / causing bodily harm / choking); s. 268 (aggravated assault)",
+   "penalty": "s. 267: hybrid, indictable maximum 10 years; s. 268: indictable, maximum 14 years",
+   "branch": "SERIOUS CRIMINALITY -- IRPA s. 36(1)(b)",
+   "status": "guidance-only",
+   "analysis": "Weapon involvement or bodily harm lifts the Canadian equivalent to s. 267 (10 years) or s. 268 (wounding/maiming, 14 years): serious criminality either way; no deemed rehabilitation.",
+   "caveats": [
+    "US 'aggravated' labels sometimes rest only on the victim's status (e.g. assault on an officer) -- if no weapon/injury, the closer equivalent may be s. 270 (assaulting an officer, hybrid 5 years) -> criminality only. Use the evidence branch."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "fraud",
+   "us_terms": [
+    "fraud",
+    "bad checks",
+    "check fraud",
+    "insufficient funds",
+    "credit card fraud",
+    "identity theft",
+    "wire fraud"
+   ],
+   "canadian_offence": "Criminal Code s. 380 (fraud); s. 342 (credit card offences); s. 402.2 (identity theft)",
+   "penalty": "s. 380(1)(a): indictable maximum 14 years where value exceeds $5,000; s. 380(1)(b): hybrid, 2 years where value $5,000 or less",
+   "branch": "SERIOUS CRIMINALITY over $5,000 (36(1)(b)); CRIMINALITY at or under $5,000 (36(2)(b))",
+   "status": "guidance-only",
+   "analysis": "Fraud splits on the $5,000 line exactly like theft. Bad-check convictions are usually small-value: hybrid 2-year equivalents -> criminality. Larger frauds (14 years) are serious criminality.",
+   "caveats": [
+    "Many US bad-check statutes criminalize mere non-payment without dishonesty elements -- where intent to defraud is not an element, the Canadian equivalent may fail on essential elements; use the Hill evidence branch."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "drug-possession",
+   "us_terms": [
+    "drug possession",
+    "possession of a controlled substance",
+    "simple possession",
+    "cocaine possession",
+    "meth possession",
+    "narcotics possession"
+   ],
+   "canadian_offence": "CDSA s. 4(1) (possession); punishment s. 4(3)-(6) by schedule",
+   "penalty": "Schedule I (cocaine, heroin, meth, fentanyl): hybrid, indictable maximum 7 years (s. 4(3)(a): 'not exceeding seven years'); Schedule III: hybrid, 3 years",
+   "branch": "CRIMINALITY -- IRPA s. 36(2)(b) (7-year maximum is below the 10-year serious-criminality line)",
+   "status": "guidance-only",
+   "analysis": "Simple possession of hard drugs equates to CDSA s. 4(1): hybrid, 7-year maximum for Schedule I -- criminality, NOT serious criminality. Deemed rehabilitation available after 10 clean years for a single conviction.",
+   "caveats": [
+    "CANNABIS IS DIFFERENT: see the cannabis entry -- simple possession within legal limits is no longer an offence in Canada at all.",
+    "Possession WITH INTENT to distribute equates to trafficking (see drug-trafficking): a completely different branch."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "cannabis-possession",
+   "us_terms": [
+    "marijuana possession",
+    "cannabis possession",
+    "possession of marijuana",
+    "weed possession"
+   ],
+   "canadian_offence": "None for simple possession within legal limits -- the Cannabis Act legalized adult possession of up to 30 g of dried cannabis (public)",
+   "penalty": "Not an offence in Canada (within limits); over-limit possession: Cannabis Act s. 8, hybrid, maximum 5 years less a day",
+   "branch": "NO INADMISSIBILITY for simple possession equivalents: with no equivalent Canadian offence, neither s. 36(1)(b) nor s. 36(2)(b) can be made out",
+   "status": "guidance-only",
+   "analysis": "A US conviction for simple marijuana possession (an amount lawful in Canada) has no Canadian equivalent offence since October 17, 2018 -- there is nothing to equate, so the conviction cannot ground criminal inadmissibility. This is the clearest 'NO' in the pairing table.",
+   "caveats": [
+    "Quantities beyond Canadian legal limits, distribution, sale, cultivation beyond limits, and import/export DO have Cannabis Act equivalents (ss. 8-14; import/export max 14 years -> serious criminality).",
+    "The equivalency is assessed against Canadian law at the ADMISSIBILITY DECISION date for the would-it-be-an-offence question -- post-legalization assessments of old possession convictions find no equivalent offence.",
+    "US felony marijuana TRAFFICKING remains serious criminality (Cannabis Act s. 9/10 or CDSA analysis)."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "drug-trafficking",
+   "us_terms": [
+    "drug trafficking",
+    "possession with intent to distribute",
+    "PWID",
+    "drug distribution",
+    "sale of a controlled substance",
+    "drug dealing"
+   ],
+   "canadian_offence": "CDSA s. 5 (trafficking / possession for the purpose)",
+   "penalty": "Schedule I or II: indictable, maximum LIFE (s. 5(3)(a): 'liable to imprisonment for life'); Schedule III/V: hybrid, 10 years",
+   "branch": "SERIOUS CRIMINALITY -- IRPA s. 36(1)(b) for any schedule (all branches reach at least 10 years)",
+   "status": "guidance-only",
+   "analysis": "Any trafficking or possession-with-intent conviction equates to CDSA s. 5: serious criminality at every schedule level. No deemed rehabilitation; individual rehabilitation after 5 years or TRP only.",
+   "caveats": [
+    "Also engages potential organized-criminality analysis (s. 37) where a criminal organization is involved."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "disorderly-conduct",
+   "us_terms": [
+    "disorderly conduct",
+    "disturbing the peace",
+    "public disturbance",
+    "breach of the peace",
+    "public intoxication"
+   ],
+   "canadian_offence": "Criminal Code s. 175 (causing a disturbance)",
+   "penalty": "SUMMARY ONLY -- s. 175 is a pure summary conviction offence",
+   "branch": "A SINGLE conviction grounds NO inadmissibility: s. 36(2)(b) requires an indictable-equivalent, and s. 175 is not one. TWO OR MORE convictions (not from a single occurrence) engage s. 36(2)(b)'s second branch",
+   "status": "guidance-only",
+   "analysis": "The most common harmless record: one US disorderly conduct/disturbing-the-peace conviction, equated to s. 175 (summary only), makes NOBODY inadmissible -- not even a foreign national. The two-or-more branch changes that, and IRPR s. 18(2)(b) then offers deemed rehabilitation five years after the sentences.",
+   "caveats": [
+    "Check what the conviction actually covers: some US disorderly statutes bundle assaultive or threatening conduct that would equate to hybrid offences -- the evidence branch of Hill controls where the statute is broader than s. 175.",
+    "Public intoxication alone typically has no federal Canadian equivalent at all (provincial regulatory matter) -- even cleaner than s. 175."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "resisting-arrest",
+   "us_terms": [
+    "resisting arrest",
+    "obstruction",
+    "obstructing an officer",
+    "evading arrest",
+    "failure to comply with police"
+   ],
+   "canadian_offence": "Criminal Code s. 129 (resisting or obstructing a peace officer)",
+   "penalty": "Hybrid; indictable maximum 2 years (s. 129(d): 'not exceeding two years')",
+   "branch": "CRIMINALITY -- IRPA s. 36(2)(b)",
+   "status": "guidance-only",
+   "analysis": "Non-violent resisting/obstruction equates to s. 129: hybrid, 2-year maximum -- criminality only.",
+   "caveats": [
+    "Vehicle flight equates to s. 320.17 (flight from police: hybrid, 10 years) -> SERIOUS criminality -- distinguish on-foot obstruction from vehicular flight.",
+    "Assaulting the officer moves it to s. 270 or s. 267 territory."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "threats",
+   "us_terms": [
+    "criminal threatening",
+    "terroristic threats",
+    "menacing",
+    "harassment threats",
+    "intimidation"
+   ],
+   "canadian_offence": "Criminal Code s. 264.1 (uttering threats)",
+   "penalty": "Hybrid; indictable maximum 5 years for threats to cause death or bodily harm (s. 264.1(2)(a))",
+   "branch": "CRIMINALITY -- IRPA s. 36(2)(b)",
+   "status": "guidance-only",
+   "analysis": "Threat offences equate to s. 264.1: hybrid, 5-year maximum -- criminality. Criminal harassment courses of conduct equate to s. 264 (10 years) -> serious criminality instead.",
+   "caveats": [
+    "'Terroristic threats' is a state-law label with no terrorism content -- do not confuse with IRPA s. 34 security grounds absent actual terrorism facts.",
+    "Repeated/stalking conduct: s. 264 criminal harassment (10 years) -> serious criminality."
+   ],
+   "authorities": []
+  },
+  {
+   "id": "mischief-vandalism",
+   "us_terms": [
+    "vandalism",
+    "criminal mischief",
+    "malicious mischief",
+    "property damage",
+    "graffiti"
+   ],
+   "canadian_offence": "Criminal Code s. 430 (mischief)",
+   "penalty": "Value over $5,000: hybrid, indictable maximum 10 years (s. 430(3)); $5,000 or under: hybrid, 2 years (s. 430(4))",
+   "branch": "CRIMINALITY at or under $5,000 (36(2)(b)); SERIOUS CRIMINALITY over $5,000 (36(1)(b))",
+   "status": "guidance-only",
+   "analysis": "Vandalism splits on the $5,000 damage value like the other property offences. Ordinary graffiti/vandalism records are 2-year equivalents -> criminality.",
+   "caveats": [
+    "The damage figure in the US record (restitution ordered is a good proxy) decides the branch -- convert to CAD at the offence date."
+   ],
+   "authorities": []
+  }
+ ]
+}

data/eval/questions.json

@@ -0,0 +1,2042 @@
+[
+ {
+  "query": "How soon must the Immigration Division review the detention of a foreign national?",
+  "answers": [
+   [
+    "IRPA",
+    "57"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "On what security grounds is a foreign national inadmissible to Canada?",
+  "answers": [
+   [
+    "IRPA",
+    "34"
+   ]
+  ]
+ },
+ {
+  "query": "When is a permanent resident inadmissible for serious criminality?",
+  "answers": [
+   [
+    "IRPA",
+    "36"
+   ]
+  ]
+ },
+ {
+  "query": "What makes a person inadmissible for organized criminality?",
+  "answers": [
+   [
+    "IRPA",
+    "37"
+   ]
+  ]
+ },
+ {
+  "query": "Can someone be found inadmissible to Canada for misrepresentation?",
+  "answers": [
+   [
+    "IRPA",
+    "40"
+   ]
+  ]
+ },
+ {
+  "query": "Is a foreign national inadmissible on health grounds?",
+  "answers": [
+   [
+    "IRPA",
+    "38"
+   ]
+  ]
+ },
+ {
+  "query": "Inadmissibility for violating human or international rights",
+  "answers": [
+   [
+    "IRPA",
+    "35"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Can a person be inadmissible to Canada for financial reasons?",
+  "answers": [
+   [
+    "IRPA",
+    "39"
+   ]
+  ]
+ },
+ {
+  "query": "Is someone inadmissible because an accompanying family member is inadmissible?",
+  "answers": [
+   [
+    "IRPA",
+    "42"
+   ]
+  ]
+ },
+ {
+  "query": "What humanitarian and compassionate relief can the Minister grant a foreign national?",
+  "answers": [
+   [
+    "IRPA",
+    "25"
+   ],
+   [
+    "IRPA",
+    "25.1"
+   ]
+  ]
+ },
+ {
+  "query": "When can an officer arrest and detain a foreign national without a warrant?",
+  "answers": [
+   [
+    "IRPA",
+    "55"
+   ]
+  ]
+ },
+ {
+  "query": "Who prepares a report that a permanent resident is inadmissible?",
+  "answers": [
+   [
+    "IRPA",
+    "44"
+   ]
+  ]
+ },
+ {
+  "query": "Must a person appear for an examination when seeking to enter Canada?",
+  "answers": [
+   [
+    "IRPA",
+    "18"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What must a person establish to be allowed to enter Canada?",
+  "answers": [
+   [
+    "IRPA",
+    "20"
+   ]
+  ]
+ },
+ {
+  "query": "What is the definition of a Convention refugee?",
+  "answers": [
+   [
+    "IRPA",
+    "96"
+   ]
+  ]
+ },
+ {
+  "query": "Who qualifies as a person in need of protection?",
+  "answers": [
+   [
+    "IRPA",
+    "97"
+   ]
+  ]
+ },
+ {
+  "query": "When is a refugee claim ineligible to be referred to the Refugee Protection Division?",
+  "answers": [
+   [
+    "IRPA",
+    "101"
+   ]
+  ]
+ },
+ {
+  "query": "What is a pre-removal risk assessment and who can apply for one?",
+  "answers": [
+   [
+    "IRPA",
+    "112"
+   ]
+  ]
+ },
+ {
+  "query": "When does a removal order become enforceable?",
+  "answers": [
+   [
+    "IRPA",
+    "48"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Is it an offence to organize the illegal entry of people into Canada?",
+  "answers": [
+   [
+    "IRPA",
+    "117"
+   ]
+  ]
+ },
+ {
+  "query": "Must a person report to a customs officer when arriving in Canada?",
+  "answers": [
+   [
+    "Customs Act",
+    "11"
+   ]
+  ]
+ },
+ {
+  "query": "What is the duty to report goods imported into Canada?",
+  "answers": [
+   [
+    "Customs Act",
+    "12"
+   ]
+  ]
+ },
+ {
+  "query": "Can a customs officer examine imported goods?",
+  "answers": [
+   [
+    "Customs Act",
+    "99"
+   ]
+  ]
+ },
+ {
+  "query": "When can a customs officer search a person at the border?",
+  "answers": [
+   [
+    "Customs Act",
+    "98"
+   ]
+  ]
+ },
+ {
+  "query": "What happens when goods are seized for a customs contravention?",
+  "answers": [
+   [
+    "Customs Act",
+    "110"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is ascertained forfeiture under the Customs Act?",
+  "answers": [
+   [
+    "Customs Act",
+    "124"
+   ]
+  ]
+ },
+ {
+  "query": "When do imported goods become forfeit after a customs contravention?",
+  "answers": [
+   [
+    "Customs Act",
+    "122"
+   ]
+  ]
+ },
+ {
+  "query": "How can a person appeal a customs seizure or penalty decision to the Federal Court?",
+  "answers": [
+   [
+    "Customs Act",
+    "135"
+   ]
+  ]
+ },
+ {
+  "query": "Advance information about commercial goods before they arrive in Canada",
+  "answers": [
+   [
+    "Customs Act",
+    "12.1"
+   ]
+  ]
+ },
+ {
+  "query": "How is the value for duty of imported goods determined?",
+  "answers": [
+   [
+    "Customs Act",
+    "46"
+   ],
+   [
+    "Customs Act",
+    "47"
+   ],
+   [
+    "Customs Act",
+    "48"
+   ]
+  ]
+ },
+ {
+  "query": "How are imported goods classified under the Customs Tariff?",
+  "answers": [
+   [
+    "Customs Tariff",
+    "10"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Which tariff heading in the Customs Tariff Schedule covers a settler's effects on first arrival?",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.07"
+   ]
+  ]
+ },
+ {
+  "query": "What allowance of alcohol and tobacco may a traveller bring back to Canada duty free?",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.27"
+   ]
+  ]
+ },
+ {
+  "query": "Conveyance temporarily imported by a non-resident visiting Canada -- customs tariff treatment",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.03"
+   ]
+  ]
+ },
+ {
+  "query": "Tariff classification of prohibited firearms and weapons under the Customs Tariff",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.98"
+   ]
+  ]
+ },
+ {
+  "query": "Tariff treatment of articles imported for personal use of representatives of foreign countries",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.08"
+   ]
+  ]
+ },
+ {
+  "query": "Must travellers report large amounts of currency when crossing the border?",
+  "answers": [
+   [
+    "PCMLTFA",
+    "12"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Can an officer seize currency that was not reported at the border?",
+  "answers": [
+   [
+    "PCMLTFA",
+    "18"
+   ]
+  ]
+ },
+ {
+  "query": "How does someone appeal the forfeiture of seized currency to the Federal Court?",
+  "answers": [
+   [
+    "PCMLTFA",
+    "30"
+   ]
+  ]
+ },
+ {
+  "query": "Is simple possession of a controlled substance an offence?",
+  "answers": [
+   [
+    "CDSA",
+    "4"
+   ]
+  ]
+ },
+ {
+  "query": "What is the offence of trafficking in a controlled substance?",
+  "answers": [
+   [
+    "CDSA",
+    "5"
+   ]
+  ]
+ },
+ {
+  "query": "Is it an offence to import or export a controlled substance?",
+  "answers": [
+   [
+    "CDSA",
+    "6"
+   ]
+  ]
+ },
+ {
+  "query": "Is possession of cannabis an offence?",
+  "answers": [
+   [
+    "Cannabis Act",
+    "8"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Can cannabis be imported into or exported from Canada?",
+  "answers": [
+   [
+    "Cannabis Act",
+    "11"
+   ]
+  ]
+ },
+ {
+  "query": "When can a peace officer arrest a person without a warrant?",
+  "answers": [
+   [
+    "Criminal Code",
+    "495"
+   ]
+  ]
+ },
+ {
+  "query": "What right does a person have to access their own personal information held by a government institution?",
+  "answers": [
+   [
+    "Privacy Act",
+    "12"
+   ]
+  ]
+ },
+ {
+  "query": "When may a government institution disclose someone's personal information?",
+  "answers": [
+   [
+    "Privacy Act",
+    "8"
+   ]
+  ]
+ },
+ {
+  "query": "Can an employee refuse to do work that presents a danger?",
+  "answers": [
+   [
+    "Canada Labour Code",
+    "128"
+   ]
+  ]
+ },
+ {
+  "query": "What are the standard hours of work for an employee?",
+  "answers": [
+   [
+    "Canada Labour Code",
+    "169"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is the standard of review of an administrative decision on judicial review?",
+  "answers": [
+   [
+    "Vavilov",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How does the Refugee Appeal Division review a decision of the Refugee Protection Division?",
+  "answers": [
+   [
+    "Huruglica",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "To get back currency seized at the border, what must the claimant show about the money?",
+  "answers": [
+   [
+    "Sellathurai",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What factors shape the content of the duty of procedural fairness in an administrative decision?",
+  "answers": [
+   [
+    "Baker",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can Canada remove a person to a country where they face a substantial risk of torture?",
+  "answers": [
+   [
+    "Suresh",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is the security certificate regime for detaining and removing non-citizens consistent with the Charter?",
+  "answers": [
+   [
+    "Charkaoui",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Do refugee claimants have a right to an oral hearing under the Charter?",
+  "answers": [
+   [
+    "Singh",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What is a particular social group in the definition of a Convention refugee?",
+  "answers": [
+   [
+    "Ward",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "When is a person excluded from refugee protection for acts contrary to the purposes of the United Nations?",
+  "answers": [
+   [
+    "Pushpanathan",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What degree of involvement makes a person complicit in international crimes and excluded from refugee protection?",
+  "answers": [
+   [
+    "Ezokola",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can a person be denied refugee protection for a serious crime committed abroad before claiming asylum?",
+  "answers": [
+   [
+    "Febles",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How must a decision-maker weigh the best interests of a child in a humanitarian and compassionate application?",
+  "answers": [
+   [
+    "Kanthasamy",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What does the national interest mean when the Minister grants relief from security inadmissibility?",
+  "answers": [
+   [
+    "Agraira",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Does helping fellow asylum seekers enter a country illegally amount to people smuggling for inadmissibility?",
+  "answers": [
+   [
+    "B010",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is the offence of human smuggling unconstitutionally overbroad if it captures humanitarian aid?",
+  "answers": [
+   [
+    "Appulonappa",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Does a conditional sentence count as a term of imprisonment for serious criminality?",
+  "answers": [
+   [
+    "Tran",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can an immigration detainee challenge their detention through habeas corpus?",
+  "answers": [
+   [
+    "Chhina",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What kinds of border search engage the Charter protection against unreasonable search and seizure?",
+  "answers": [
+   [
+    "Simmons",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Does inadmissibility for membership in a terrorist organization require a complicity analysis?",
+  "answers": [
+   [
+    "Kanagendren",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How broadly is a criminal organization interpreted for organized criminality inadmissibility?",
+  "answers": [
+   [
+    "Sittampalam",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What principles govern a finding of inadmissibility for misrepresentation?",
+  "answers": [
+   [
+    "Goburdhun",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What is the innocent-misrepresentation defence to inadmissibility under IRPA s. 40?",
+  "answers": [
+   [
+    "Wang",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Must a misrepresentation under IRPA s. 40 be intentional, deliberate or negligent to be material?",
+  "answers": [
+   [
+    "Bellido",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "At an immigration detention review, who bears the onus and how are earlier detention rulings treated?",
+  "answers": [
+   [
+    "Thanabalasingham",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is the test for admitting new evidence in a pre-removal risk assessment?",
+  "answers": [
+   [
+    "Raza",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Are gold coins currency or monetary instruments that must be reported when imported?",
+  "answers": [
+   [
+    "Hociung",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How do the courts review a customs tariff classification decision?",
+  "answers": [
+   [
+    "Best Buy",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What does CBSA policy say about how the value for duty of imported goods is established?",
+  "answers": [
+   [
+    "D-Memo",
+    "D13-1-1"
+   ]
+  ]
+ },
+ {
+  "query": "What are CBSA's requirements for marking imported goods with their country of origin?",
+  "answers": [
+   [
+    "D-Memo",
+    "D11-3-1"
+   ]
+  ]
+ },
+ {
+  "query": "What is CBSA's guidance on importing or exporting cannabis and controlled substances?",
+  "answers": [
+   [
+    "D-Memo",
+    "D19-9-2"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What personal exemptions can a resident claim when returning to Canada, per CBSA guidance?",
+  "answers": [
+   [
+    "D-Memo",
+    "D2-3-1"
+   ]
+  ]
+ },
+ {
+  "query": "What is CBSA's guidance on cross-border currency and monetary instruments reporting?",
+  "answers": [
+   [
+    "D-Memo",
+    "D19-14-1"
+   ]
+  ]
+ },
+ {
+  "query": "How does CBSA decide whether imported material is obscene?",
+  "answers": [
+   [
+    "D-Memo",
+    "D9-1-1"
+   ]
+  ]
+ },
+ {
+  "query": "What proof of origin does CBSA require for imported goods?",
+  "answers": [
+   [
+    "D-Memo",
+    "D11-4-2"
+   ]
+  ]
+ },
+ {
+  "query": "What is the Canadian Goods Abroad Program for goods sent outside Canada for repair?",
+  "answers": [
+   [
+    "D-Memo",
+    "D8-2-1"
+   ]
+  ]
+ },
+ {
+  "query": "How does the FB Border Services collective agreement deal with discipline of an employee?",
+  "answers": [
+   [
+    "FB Agreement",
+    "17"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is the grievance procedure under the FB collective agreement?",
+  "answers": [
+   [
+    "FB Agreement",
+    "18"
+   ]
+  ]
+ },
+ {
+  "query": "What are the hours of work under the FB Border Services collective agreement?",
+  "answers": [
+   [
+    "FB Agreement",
+    "25"
+   ]
+  ]
+ },
+ {
+  "query": "How is overtime compensated under the FB collective agreement?",
+  "answers": [
+   [
+    "FB Agreement",
+    "28"
+   ]
+  ]
+ },
+ {
+  "query": "How much vacation leave with pay do FB-group employees earn?",
+  "answers": [
+   [
+    "FB Agreement",
+    "34"
+   ]
+  ]
+ },
+ {
+  "query": "What is the bilingualism bonus and who is eligible to receive it?",
+  "answers": [
+   [
+    "Bilingualism Bonus Directive",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What assistance is available to a federal employee who faces unusual daily commuting costs?",
+  "answers": [
+   [
+    "Commuting Assistance Directive",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "When may the Immigration Division order the release of a detained person?",
+  "answers": [
+   [
+    "IRPA",
+    "58"
+   ]
+  ]
+ },
+ {
+  "query": "When is there no right to appeal a removal order to the Immigration Appeal Division?",
+  "answers": [
+   [
+    "IRPA",
+    "64"
+   ]
+  ]
+ },
+ {
+  "query": "Is distributing cannabis an offence?",
+  "answers": [
+   [
+    "Cannabis Act",
+    "9"
+   ]
+  ]
+ },
+ {
+  "query": "What is the offence of smuggling goods into Canada under the Customs Act?",
+  "answers": [
+   [
+    "Customs Act",
+    "159"
+   ]
+  ]
+ },
+ {
+  "query": "Is an employee entitled to medical leave under the Canada Labour Code?",
+  "answers": [
+   [
+    "Canada Labour Code",
+    "239"
+   ]
+  ]
+ },
+ {
+  "query": "How should an employer apply progressive discipline to a border services officer?",
+  "answers": [
+   [
+    "Menzies",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is the burden and standard of proof when an employer defends a disciplinary grievance?",
+  "answers": [
+   [
+    "Basra (2007)",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What factors determine whether an employer had just cause to discipline or terminate an employee?",
+  "answers": [
+   [
+    "Pepper",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What is the test for whether a union breached its duty of fair representation?",
+  "answers": [
+   [
+    "McRaeJackson",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can a union breach the duty of fair representation in how it handles a vaccination-policy grievance?",
+  "answers": [
+   [
+    "Watson",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is it consistent with the Charter of Rights to deport a permanent resident convicted of serious crimes?",
+  "answers": [
+   [
+    "Chiarelli",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "When the Immigration Appeal Division hears an appeal from a removal order, may it weigh the hardship the person would face in the country of removal?",
+  "answers": [
+   [
+    "Chieu",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Does a non-citizen have a Charter right to remain in Canada?",
+  "answers": [
+   [
+    "Medovarski",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What must be established for inadmissibility for incitement to genocide or a crime against humanity?",
+  "answers": [
+   [
+    "Mugesera",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is the IRPA security-certificate regime, with its special advocate scheme, consistent with the Charter?",
+  "answers": [
+   [
+    "Harkat",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How does the principle of non-refoulement constrain Canada when it extradites a person who holds refugee protection?",
+  "answers": [
+   [
+    "Németh",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What procedural fairness is owed to a sponsor before the government enforces a sponsorship-undertaking debt?",
+  "answers": [
+   [
+    "Mavi",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can a sentencing court take the immigration consequences of a sentence into account?",
+  "answers": [
+   [
+    "Pham",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Can a fear of coerced sterilization under a one-child population-control policy support a refugee claim?",
+  "answers": [
+   [
+    "Chan",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can a customs officer stop and search a vehicle near the border on reasonable suspicion of a customs offence?",
+  "answers": [
+   [
+    "Jacques",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is an ascertained-forfeiture proceeding under the Customs Act penal enough to engage the right against self-incrimination?",
+  "answers": [
+   [
+    "Martineau",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Does a person's age at the time of involvement matter to inadmissibility for membership in a terrorist organization?",
+  "answers": [
+   [
+    "Poshteh",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can a deserter from the United States military succeed in a refugee or humanitarian and compassionate claim in Canada?",
+  "answers": [
+   [
+    "Hinzman",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Does a Refugee Protection Division guideline that sets the order of questioning improperly fetter a member's discretion?",
+  "answers": [
+   [
+    "Thamotharem",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Can a person with no immigration status obtain coverage under the Interim Federal Health Program?",
+  "answers": [
+   [
+    "Toussaint",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What does a finding of no credible basis mean for a refugee claim?",
+  "answers": [
+   [
+    "Rahaman",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How is the reasonableness of a security certificate assessed on a re-determination after Charkaoui?",
+  "answers": [
+   [
+    "Almrei",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can a permanent resident be found inadmissible for acts of violence that endanger safety in Canada even without a conviction?",
+  "answers": [
+   [
+    "Mason",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How much deference does the Immigration Appeal Division receive when it declines to grant discretionary relief from removal?",
+  "answers": [
+   [
+    "Khosa",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is lengthy immigration detention consistent with the Charter protection against arbitrary detention?",
+  "answers": [
+   [
+    "Brown",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Is the designation of the United States as a safe third country for refugee claimants constitutional?",
+  "answers": [
+   [
+    "CCRf",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Can an employee with an addiction be dismissed for breach of a drug and alcohol policy in a safety-sensitive workplace?",
+  "answers": [
+   [
+    "Elk Valley",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "When is random alcohol testing permitted in a unionized safety-sensitive workplace?",
+  "answers": [
+   [
+    "Irving",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Do RCMP members have a Charter right to collective bargaining?",
+  "answers": [
+   [
+    "MPAO",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is the right to strike protected by the Charter?",
+  "answers": [
+   [
+    "SFL",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What meal and incidental allowances can a federal employee claim while travelling on government business?",
+  "answers": [
+   [
+    "Travel Directive",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is a federal employee entitled to when the employer requires them to wear a uniform?",
+  "answers": [
+   [
+    "Uniforms Directive",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What protections does an indeterminate employee have when their position is declared surplus?",
+  "answers": [
+   [
+    "Work Force Adjustment Directive",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is there an allowance for an employee required to administer first aid to the general public?",
+  "answers": [
+   [
+    "First Aid to the General Public - Allowance for Employees",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What allowances are available to a federal employee posted at an isolated post?",
+  "answers": [
+   [
+    "Isolated Posts and Government Housing Directive",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What relocation expenses are reimbursed when a federal employee must move for work?",
+  "answers": [
+   [
+    "NJC Relocation Directive",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What occupational health and safety obligations does the NJC directive place on the employer?",
+  "answers": [
+   [
+    "Occupational Health and Safety Directive",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What coverage does the Public Service Health Care Plan provide?",
+  "answers": [
+   [
+    "Public Service Health Care Plan Directive",
+    ""
+   ],
+   [
+    "PSHCP",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What does the PSHCP member booklet say about prescription drug coverage?",
+  "answers": [
+   [
+    "PSHCP",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What dental services are covered under the Public Service Dental Care Plan for employees?",
+  "answers": [
+   [
+    "PSDCP",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What dental coverage do federal pensioners get under the Pensioners' Dental Services Plan?",
+  "answers": [
+   [
+    "PDSP",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What life and disability insurance does the Public Service Management Insurance Plan provide for excluded employees?",
+  "answers": [
+   [
+    "PSMIP",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What support do the Foreign Service Directives give an employee posted outside Canada?",
+  "answers": [
+   [
+    "Foreign Service Directives",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Who has a right to appeal a decision to the Immigration Appeal Division?",
+  "answers": [
+   [
+    "IRPA",
+    "63"
+   ]
+  ]
+ },
+ {
+  "query": "What are the objectives of the Immigration and Refugee Protection Act?",
+  "answers": [
+   [
+    "IRPA",
+    "3"
+   ]
+  ]
+ },
+ {
+  "query": "Can a non-unionized federally regulated employee complain that they were unjustly dismissed?",
+  "answers": [
+   [
+    "Canada Labour Code",
+    "240"
+   ]
+  ]
+ },
+ {
+  "query": "What does CBSA disclose about normal values and export prices under the Special Import Measures Act?",
+  "answers": [
+   [
+    "D-Memo",
+    "D14-1-2"
+   ]
+  ]
+ },
+ {
+  "query": "What are CBSA's requirements for importing or exporting cultural property?",
+  "answers": [
+   [
+    "D-Memo",
+    "D19-4-1"
+   ]
+  ]
+ },
+ {
+  "query": "When assessing criminal inadmissibility for a conviction abroad, must a decision-maker consider whether the defence of duress was practically available in that country?",
+  "answers": [
+   [
+    "Rodriguez Anzola",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What is the mandate of the Canada Border Services Agency?",
+  "answers": [
+   [
+    "CBSA Act",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How is a grievance referred to adjudication in the federal public service?",
+  "answers": [
+   [
+    "FPSLRA",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What is the Federal Public Sector Labour Relations and Employment Board?",
+  "answers": [
+   [
+    "FPSLREB Act",
+    ""
+   ],
+   [
+    "FPSLRA",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What are the requirements to become a Canadian citizen by grant?",
+  "answers": [
+   [
+    "Citizenship Act",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How are dumping and subsidizing of imported goods investigated and what duties apply?",
+  "answers": [
+   [
+    "SIMA",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "When is a permit required to export goods on the Export Control List?",
+  "answers": [
+   [
+    "EIPA",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What authorization is required to import a firearm into Canada?",
+  "answers": [
+   [
+    "Firearms Act",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What are the procedures for an admissibility hearing before the Immigration Division?",
+  "answers": [
+   [
+    "Immigration Division Rules",
+    ""
+   ],
+   [
+    "ENF 3",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What rules govern the hearing of a refugee protection claim before the RPD?",
+  "answers": [
+   [
+    "RPD Rules",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What are the Refugee Appeal Division Rules for filing and perfecting an appeal?",
+  "answers": [
+   [
+    "RAD Rules",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What is the right to withdraw from a preclearance area, and what powers does a US preclearance officer have in Canada?",
+  "answers": [
+   [
+    "Preclearance Act",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What does WAPPRIITA prohibit for international trade in protected wild animals and plants?",
+  "answers": [
+   [
+    "WAPPRIITA",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "How must imported goods be reported to a customs officer?",
+  "answers": [
+   [
+    "Reporting of Imported Goods Regs",
+    ""
+   ],
+   [
+    "Customs Act",
+    "12"
+   ]
+  ]
+ },
+ {
+  "query": "Returning Persons Exemption Regulations prescribed value of goods for a returning resident",
+  "answers": [
+   [
+    "Returning Persons Exemption Regs",
+    ""
+   ],
+   [
+    "D-Memo",
+    "D2-3-1"
+   ]
+  ]
+ },
+ {
+  "query": "What tariff item covers goods acquired abroad by a returning resident of Canada for personal use?",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.04"
+   ]
+  ]
+ },
+ {
+  "query": "Under what tariff heading are a settler's household effects imported free of duty?",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-98.07"
+   ]
+  ]
+ },
+ {
+  "query": "Can goods classified in Chapter 99 also qualify under Chapter 98 of the tariff schedule?",
+  "answers": [
+   [
+    "Customs Tariff",
+    "Sch-Ch98-Notes"
+   ],
+   [
+    "Customs Tariff",
+    "Sch-Ch99-Notes"
+   ]
+  ]
+ },
+ {
+  "query": "What does the public service health care plan cover for out-of-province emergency travel?",
+  "answers": [
+   [
+    "PSHCP",
+    ""
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Can border officers search a traveller's cell phone without any grounds under the Customs Act?",
+  "answers": [
+   [
+    "Canfield",
+    ""
+   ],
+   [
+    "Pike",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What limits did the Supreme Court place on searching a cell phone incident to arrest?",
+  "answers": [
+   [
+    "Fearon",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "What did the Ontario Court of Appeal decide about suspicionless searches of personal electronic devices at the border?",
+  "answers": [
+   [
+    "Pike",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "How does the RAD jurisprudential guide interpret the generalized risk exception for gang extortion claims?",
+  "answers": [
+   [
+    "TC1-05038",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Which CBSA officers are designated to form an inadmissibility report under IRPA section 44?",
+  "answers": [
+   [
+    "CBSA Deleg 2023-05-08",
+    ""
+   ],
+   [
+    "CBSA Deleg Amend 2025-07-10",
+    ""
+   ],
+   [
+    "CBSA Deleg Amend 2023-09-08",
+    ""
+   ]
+  ]
+ },
+ {
+  "query": "Is a Texas deferred adjudication a conviction for Canadian immigration purposes?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "deferred-adjudication"
+   ],
+   [
+    "US Dispositions Helper",
+    "state-texas"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "Does a Florida withheld adjudication make someone inadmissible to Canada?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "withheld-adjudication"
+   ],
+   [
+    "US Dispositions Helper",
+    "state-florida"
+   ]
+  ]
+ },
+ {
+  "query": "How does Canada treat a US state pardon for immigration inadmissibility?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "state-pardon"
+   ]
+  ]
+ },
+ {
+  "query": "What US criminal dispositions does Georgia law treat as non-convictions?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "state-georgia"
+   ]
+  ]
+ },
+ {
+  "query": "Is a juvenile delinquency adjudication in the United States a conviction under IRPA?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "juvenile-adjudication"
+   ]
+  ]
+ },
+ {
+  "query": "What is the standard of proof for criminal inadmissibility findings under IRPA?",
+  "answers": [
+   [
+    "IRPA",
+    "33"
+   ],
+   [
+    "US Dispositions Helper",
+    "conviction-framework"
+   ]
+  ]
+ },
+ {
+  "query": "Does a single US DUI conviction make a person inadmissible to Canada for serious criminality?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "equiv-dui"
+   ]
+  ],
+  "holdout": true
+ },
+ {
+  "query": "What does a US shoplifting conviction equate to under Canadian law for inadmissibility?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "equiv-theft-under"
+   ]
+  ]
+ },
+ {
+  "query": "Is an old US marijuana possession conviction still a ground of inadmissibility to Canada?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "equiv-cannabis-possession"
+   ]
+  ]
+ },
+ {
+  "query": "Does one disorderly conduct conviction make a foreign national criminally inadmissible?",
+  "answers": [
+   [
+    "US Dispositions Helper",
+    "equiv-disorderly-conduct"
+   ]
+  ]
+ },
+ {
+  "query": "When is a person deemed rehabilitated after a single foreign conviction?",
+  "answers": [
+   [
+    "IRPR",
+    "18"
+   ]
+  ]
+ },
+ {
+  "query": "How long after completing a sentence can someone apply for criminal rehabilitation?",
+  "answers": [
+   [
+    "IRPR",
+    "17"
+   ]
+  ]
+ },
+ {
+  "query": "Traveller referred to secondary arriving from Amsterdam; a white powder in their bag field-tests as ketamine. Which CDSA schedule lists ketamine and what does the possession offence say?",
+  "answers": [
+   [
+    "CDSA",
+    "Schedule I"
+   ],
+   [
+    "CDSA",
+    "4"
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "Secondary examination turns up a 2019 Texas deferred adjudication for misdemeanor theft, successfully completed in 2021. Is the traveller criminally inadmissible on the conviction branch?",
+  "answers": [
+   [
+    "IRPA",
+    "36"
+   ],
+   [
+    "US Dispositions Helper",
+    ""
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "A permanent resident was detained at the port of entry on Tuesday morning. When must the Immigration Division review the detention, and when are the later reviews?",
+  "answers": [
+   [
+    "IRPA",
+    "57"
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "Traveller declares nothing but a currency detector dog indicates; officers find US$12,000 in an envelope. What reporting duty was breached and what can be seized?",
+  "answers": [
+   [
+    "PCMLTFA",
+    "12"
+   ],
+   [
+    "PCMLTFA",
+    "18"
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "A NEXUS member used the members-only lane while transporting a firearm part they failed to declare. What regulations govern presentation in an alternative manner and cancellation of the authorization?",
+  "answers": [
+   [
+    "Presentation of Persons Regs",
+    ""
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "Baggage exam finds brass knuckles bought as a souvenir. Where are they classified as a prohibited weapon?",
+  "answers": [
+   [
+    "Firearms Classification Regs",
+    ""
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "A traveller detained for a strip search at secondary asks to call a lawyer before it happens. What Charter right is engaged?",
+  "answers": [
+   [
+    "Charter",
+    "10"
+   ],
+   [
+    "Dehghani",
+    ""
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "An officer believes a foreign national in secondary is inadmissible for serious criminality and wants to write the report. Which operational manual covers writing A44(1) reports?",
+  "answers": [
+   [
+    "ENF 5",
+    ""
+   ]
+  ],
+  "style": "fact-pattern"
+ },
+ {
+  "query": "currency seizure threshold",
+  "answers": [
+   [
+    "PCMLTFA",
+    "12"
+   ],
+   [
+    "PCMLTFA",
+    "18"
+   ]
+  ],
+  "style": "keyword"
+ },
+ {
+  "query": "detention review timelines",
+  "answers": [
+   [
+    "IRPA",
+    "57"
+   ]
+  ],
+  "style": "keyword"
+ },
+ {
+  "query": "ketamine schedule",
+  "answers": [
+   [
+    "CDSA",
+    "Schedule I"
+   ]
+  ],
+  "style": "keyword"
+ },
+ {
+  "query": "personal exemption returning resident",
+  "answers": [
+   [
+    "Customs Tariff",
+    ""
+   ],
+   [
+    "D-Memo",
+    "D2-3-1"
+   ]
+  ],
+  "style": "keyword"
+ },
+ {
+  "query": "AMPS contravention penalty amounts",
+  "answers": [
+   [
+    "AMPS",
+    ""
+   ]
+  ],
+  "style": "keyword"
+ },
+ {
+  "query": "IAD appeal rules",
+  "answers": [
+   [
+    "IAD Rules",
+    ""
+   ],
+   [
+    "ENF 19",
+    ""
+   ]
+  ],
+  "style": "keyword"
+ },
+ {
+  "query": "detension review timlines permanent resident",
+  "answers": [
+   [
+    "IRPA",
+    "57"
+   ]
+  ],
+  "style": "typo"
+ },
+ {
+  "query": "innadmissibility for serius criminality",
+  "answers": [
+   [
+    "IRPA",
+    "36"
+   ]
+  ],
+  "style": "typo"
+ },
+ {
+  "query": "seisure of undeclraed goods",
+  "answers": [
+   [
+    "Customs Act",
+    "110"
+   ]
+  ],
+  "style": "typo"
+ },
+ {
+  "query": "criminal rehabilitaton aplication period",
+  "answers": [
+   [
+    "IRPR",
+    "17"
+   ]
+  ],
+  "style": "typo"
+ },
+ {
+  "query": "délai de contrôle de la détention d'un résident permanent",
+  "answers": [
+   [
+    "IRPA",
+    "57"
+   ]
+  ],
+  "style": "french"
+ },
+ {
+  "query": "interdiction de territoire pour grande criminalité",
+  "answers": [
+   [
+    "IRPA",
+    "36"
+   ]
+  ],
+  "style": "french"
+ },
+ {
+  "query": "saisie de devises non déclarées à la frontière",
+  "answers": [
+   [
+    "PCMLTFA",
+    "18"
+   ]
+  ],
+  "style": "french"
+ },
+ {
+  "query": "renvoi d'un étranger interdit de territoire",
+  "answers": [
+   [
+    "IRPA",
+    "48"
+   ],
+   [
+    "ENF 10",
+    ""
+   ]
+  ],
+  "style": "french"
+ },
+ {
+  "query": "fouille d'une personne à la frontière par l'agent des douanes",
+  "answers": [
+   [
+    "Customs Act",
+    "98"
+   ]
+  ],
+  "style": "french"
+ },
+ {
+  "query": "How many demerit points for speeding in Ontario?",
+  "answers": [],
+  "style": "no-answer"
+ },
+ {
+  "query": "What is the process to renew a US H-1B visa?",
+  "answers": [],
+  "style": "no-answer"
+ },
+ {
+  "query": "How is child custody decided after divorce in Quebec?",
+  "answers": [],
+  "style": "no-answer"
+ },
+ {
+  "query": "What notice must an Ontario landlord give to evict a tenant?",
+  "answers": [],
+  "style": "no-answer"
+ },
+ {
+  "query": "What do the CIU by-laws say about how branch elections are conducted?",
+  "answers": [
+   [
+    "CIU By-Laws",
+    ""
+   ],
+   [
+    "CIU Elections Kit",
+    ""
+   ],
+   [
+    "CIU Branch Rules",
+    ""
+   ]
+  ],
+  "style": "union"
+ },
+ {
+  "query": "What is the variable shift schedule agreement for Windsor Bridge?",
+  "answers": [
+   [
+    "VSSA",
+    ""
+   ]
+  ],
+  "style": "union"
+ },
+ {
+  "query": "How does the PSAC constitution deal with membership discipline and expulsion?",
+  "answers": [
+   [
+    "PSAC Constitution",
+    ""
+   ]
+  ],
+  "style": "union"
+ },
+ {
+  "query": "What are the timelines for presenting a grievance according to the union's grievance guide?",
+  "answers": [
+   [
+    "CIU Grievance Guide",
+    ""
+   ]
+  ],
+  "style": "union"
+ }
+]

data/eval/sweep.json

@@ -0,0 +1,272 @@
+{
+  "best": {
+    "CANLEX_MN_WEIGHT": 0.0024,
+    "CANLEX_MN_CAP": 0.012,
+    "CANLEX_REG_PENALTY": 0.004,
+    "CANLEX_BACKMATTER_PENALTY": 0.004
+  },
+  "best_metrics": {
+    "hit1": 0.79,
+    "hit3": 0.96,
+    "hit5": 0.99,
+    "hit10": 0.99,
+    "mrr": 0.88
+  },
+  "runs": [
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.87
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0012,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.78,
+        "hit3": 0.94,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.86
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.87
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.005,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.78,
+        "hit3": 0.94,
+        "hit5": 0.97,
+        "hit10": 0.99,
+        "mrr": 0.86
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.01,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.75,
+        "hit3": 0.94,
+        "hit5": 0.96,
+        "hit10": 0.99,
+        "mrr": 0.84
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.006,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.77,
+        "hit3": 0.93,
+        "hit5": 0.96,
+        "hit10": 0.99,
+        "mrr": 0.85
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.87
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.024,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.77,
+        "hit3": 0.94,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.86
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.05,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.77,
+        "hit3": 0.94,
+        "hit5": 0.97,
+        "hit10": 0.99,
+        "mrr": 0.86
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.004,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.88
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.008,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.87
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.016,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.88
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.032,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.88
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.004,
+        "CANLEX_BACKMATTER_PENALTY": 0.004
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.99,
+        "hit10": 0.99,
+        "mrr": 0.88
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.004,
+        "CANLEX_BACKMATTER_PENALTY": 0.008
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.88
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.004,
+        "CANLEX_BACKMATTER_PENALTY": 0.016
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.98,
+        "hit10": 0.99,
+        "mrr": 0.87
+      }
+    },
+    {
+      "values": {
+        "CANLEX_MN_WEIGHT": 0.0024,
+        "CANLEX_MN_CAP": 0.012,
+        "CANLEX_REG_PENALTY": 0.004,
+        "CANLEX_BACKMATTER_PENALTY": 0.032
+      },
+      "metrics": {
+        "hit1": 0.79,
+        "hit3": 0.96,
+        "hit5": 0.99,
+        "hit10": 0.99,
+        "mrr": 0.87
+      }
+    }
+  ]
+}

data/eval/sweep.log

@@ -0,0 +1,33 @@
+# CanLex tuning sweep -- 2026-05-23 12:59:35
+
+Baseline: {'CANLEX_MN_WEIGHT': 0.0024, 'CANLEX_MN_CAP': 0.012, 'CANLEX_REG_PENALTY': 0.008, 'CANLEX_BACKMATTER_PENALTY': 0.008}
+  Hit@5=0.980  MRR=0.870
+
+Sweeping CANLEX_MN_WEIGHT in [0.0012, 0.0024, 0.005, 0.01] (others held at {'CANLEX_MN_WEIGHT': 0.0024, 'CANLEX_MN_CAP': 0.012, 'CANLEX_REG_PENALTY': 0.008, 'CANLEX_BACKMATTER_PENALTY': 0.008})
+  CANLEX_MN_WEIGHT=0.0012   -> Hit@1=0.780 Hit@3=0.940 Hit@5=0.980 Hit@10=0.990 MRR=0.860
+  CANLEX_MN_WEIGHT=0.0024   -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.870
+  CANLEX_MN_WEIGHT=0.005    -> Hit@1=0.780 Hit@3=0.940 Hit@5=0.970 Hit@10=0.990 MRR=0.860
+  CANLEX_MN_WEIGHT=0.01     -> Hit@1=0.750 Hit@3=0.940 Hit@5=0.960 Hit@10=0.990 MRR=0.840
+
+Sweeping CANLEX_MN_CAP in [0.006, 0.012, 0.024, 0.05] (others held at {'CANLEX_MN_WEIGHT': 0.0024, 'CANLEX_MN_CAP': 0.012, 'CANLEX_REG_PENALTY': 0.008, 'CANLEX_BACKMATTER_PENALTY': 0.008})
+  CANLEX_MN_CAP=0.006    -> Hit@1=0.770 Hit@3=0.930 Hit@5=0.960 Hit@10=0.990 MRR=0.850
+  CANLEX_MN_CAP=0.012    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.870
+  CANLEX_MN_CAP=0.024    -> Hit@1=0.770 Hit@3=0.940 Hit@5=0.980 Hit@10=0.990 MRR=0.860
+  CANLEX_MN_CAP=0.05     -> Hit@1=0.770 Hit@3=0.940 Hit@5=0.970 Hit@10=0.990 MRR=0.860
+
+Sweeping CANLEX_REG_PENALTY in [0.004, 0.008, 0.016, 0.032] (others held at {'CANLEX_MN_WEIGHT': 0.0024, 'CANLEX_MN_CAP': 0.012, 'CANLEX_REG_PENALTY': 0.008, 'CANLEX_BACKMATTER_PENALTY': 0.008})
+  CANLEX_REG_PENALTY=0.004    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.880
+  CANLEX_REG_PENALTY=0.008    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.870
+  CANLEX_REG_PENALTY=0.016    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.880
+  CANLEX_REG_PENALTY=0.032    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.880
+  ! CANLEX_REG_PENALTY: 0.008 -> 0.004
+
+Sweeping CANLEX_BACKMATTER_PENALTY in [0.004, 0.008, 0.016, 0.032] (others held at {'CANLEX_MN_WEIGHT': 0.0024, 'CANLEX_MN_CAP': 0.012, 'CANLEX_REG_PENALTY': 0.004, 'CANLEX_BACKMATTER_PENALTY': 0.008})
+  CANLEX_BACKMATTER_PENALTY=0.004    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.990 Hit@10=0.990 MRR=0.880
+  CANLEX_BACKMATTER_PENALTY=0.008    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.880
+  CANLEX_BACKMATTER_PENALTY=0.016    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.980 Hit@10=0.990 MRR=0.870
+  CANLEX_BACKMATTER_PENALTY=0.032    -> Hit@1=0.790 Hit@3=0.960 Hit@5=0.990 Hit@10=0.990 MRR=0.870
+  ! CANLEX_BACKMATTER_PENALTY: 0.008 -> 0.004
+
+Best: {'CANLEX_MN_WEIGHT': 0.0024, 'CANLEX_MN_CAP': 0.012, 'CANLEX_REG_PENALTY': 0.004, 'CANLEX_BACKMATTER_PENALTY': 0.004}
+  Hit@1=0.790 Hit@5=0.990  MRR=0.880

data/processed/A-8.8-fr.json

@@ -0,0 +1,942 @@
+[
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "1",
+    "marginal_note": "Titre abrégé",
+    "part": "",
+    "division": "",
+    "heading": "Titre abrégé",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-1.html",
+    "id": "A-8.8-fr-s1",
+    "text": "1 Loi sur les sanctions administratives pécuniaires en matière d’agriculture et d’agroalimentaire.",
+    "citation": "AAAMPA, art. 1",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "2",
+    "marginal_note": "Définitions",
+    "part": "",
+    "division": "",
+    "heading": "Définitions",
+    "history": "1995, ch. 40, art. 2; 1997, ch. 21, art. 30; 2002, ch. 28, art. 82; 2005, ch. 38, art. 30 et 145; 2012, ch. 24, art. 98; 2015, ch. 2, art. 113",
+    "last_amended": "2019-01-15",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-2.html",
+    "id": "A-8.8-fr-s2",
+    "text": "2 Les définitions qui suivent s’appliquent à la présente loi.\nCommission La Commission de révision prorogée par le paragraphe 27(1). (Tribunal)\nloi agroalimentaire La Loi sur la médiation en matière d’endettement agricole, la Loi relative aux aliments du bétail, la Loi sur les engrais, la Loi sur la santé des animaux, la Loi sur les produits antiparasitaires, la Loi sur la protection des végétaux, la Loi sur la salubrité des aliments au Canada ou la Loi sur les semences. (agri-food Act)\nministre Le ministre de l’Agriculture et de l’Agroalimentaire ou :\na) le ministre de la Santé, pour toute violation relative à une contravention :\n(i) soit à la Loi sur les produits antiparasitaires,\n(ii) soit à une disposition portant sur la salubrité alimentaire dans une loi agroalimentaire ou dans un règlement pris en vertu d’une telle loi;\nb) le ministre de la Sécurité publique et de la Protection civile, pour toute question relative aux procès-verbaux relatifs aux contraventions à la législation frontalière visée au paragraphe 11(5) de la Loi sur l’Agence canadienne d’inspection des aliments. (Minister)\nsanction Sanction administrative pécuniaire infligée pour une violation. (penalty)",
+    "citation": "AAAMPA, art. 2",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "3",
+    "marginal_note": "Principe",
+    "part": "",
+    "division": "",
+    "heading": "Objet",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-3.html",
+    "id": "A-8.8-fr-s3",
+    "text": "3 La présente loi a pour objet d’établir, comme solution de rechange au régime pénal et complément aux autres mesures d’application des lois agroalimentaires déjà en vigueur, un régime juste et efficace de sanctions administratives pécuniaires.",
+    "citation": "AAAMPA, art. 3",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "4",
+    "marginal_note": "Pouvoir réglementaire",
+    "part": "",
+    "division": "",
+    "heading": "Attributions du ministre",
+    "history": "1995, ch. 40, art. 4; 2012, ch. 24, art. 99; 2015, ch. 2, art. 114; 2016, ch. 9, art. 70 et 72",
+    "last_amended": "2019-01-15",
+    "in_force": "2016-12-12",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-4.html",
+    "id": "A-8.8-fr-s4",
+    "text": "4\n(1) Le ministre peut, par règlement :\na) désigner comme violation punissable au titre de la présente loi la contravention — si elle constitue une infraction à une loi agroalimentaire :\n(i) aux dispositions spécifiées d’une loi agroalimentaire ou de ses règlements,\n(ii) à tout arrêté spécifié pris par le ministre au titre de la Loi sur la protection des végétaux, ou à toute catégorie de ces arrêtés,\n(iii) à toute obligation ou catégorie d’obligations spécifiée — par refus ou omission de l’accomplir — découlant de la Loi sur la santé des animaux, de la Loi sur la protection des végétaux, de la Loi sur les produits antiparasitaires ou de la Loi sur la salubrité des aliments au Canada;\nb) qualifier les violations, selon le cas, de mineures, de graves ou de très graves;\nb.1) établir le sommaire caractérisant la violation dans les procès-verbaux;\nc) fixer le montant — notamment par barème — de la sanction applicable à chaque violation;\nd) prévoir les critères de majoration ou de minoration — notamment pour les transactions — de ce montant, ainsi que les modalités de cette opération;\ne) régir la détermination d’un montant inférieur à la sanction infligée dont le paiement, dans le délai et selon les modalités réglementaires, vaut règlement;\nf) prévoir les cas dans lesquels la Commission peut procéder, dans le cadre du paragraphe 14(1), par écrit ou par la tenue d’une audience;\ng) régir, notamment par l’établissement de présomptions et de règles de preuve, la notification des documents autorisés ou exigés par la présente loi;\nh) prendre toute mesure d’ordre réglementaire prévue par la présente loi;\ni) prendre toute autre mesure d’application de la présente loi.\n(2) [Plafond de la sanction] Le plafond de la sanction est de 2 000 $ pour toute violation commise par une personne physique, sauf dans le cadre d’une entreprise ou à des fins lucratives, et, dans les autres cas, de 5 000 $, 15 000 $ ou 25 000 $, selon que la violation est mineure, grave ou très grave.\n(3) [Critères] Figurent parmi les critères prévus par le ministre au titre de l’alinéa (1)d) notamment :\na) la nature de l’intention ou de la négligence du contrevenant;\nb) la gravité du tort causé par la violation;\nc) les antécédents du contrevenant relatifs aux violations d’une loi agroalimentaire ou aux condamnations pour infraction à une telle loi dans les cinq ans précédant la violation.",
+    "citation": "AAAMPA, art. 4",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "5",
+    "marginal_note": "Précision",
+    "part": "",
+    "division": "",
+    "heading": "Attributions du ministre",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-5.html",
+    "id": "A-8.8-fr-s5",
+    "text": "5 Tout acte ou omission qualifiable à la fois de violation et d’infraction peut être réprimé soit comme violation par le ministre, soit, sur sa recommandation, comme infraction, les poursuites pour violation et celles pour infraction s’excluant toutefois mutuellement.",
+    "citation": "AAAMPA, art. 5",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "6",
+    "marginal_note": "Pouvoir du ministre : agents verbalisateurs",
+    "part": "",
+    "division": "",
+    "heading": "Attributions du ministre",
+    "history": "1995, ch. 40, art. 6; 2015, ch. 2, art. 115",
+    "last_amended": "2015-02-27",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-6.html",
+    "id": "A-8.8-fr-s6",
+    "text": "6 Le ministre peut désigner, individuellement ou par catégorie, les agents verbalisateurs.",
+    "citation": "AAAMPA, art. 6",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "7",
+    "marginal_note": "Violation",
+    "part": "",
+    "division": "",
+    "heading": "Ouverture de la poursuite",
+    "history": "1995, ch. 40, art. 7; 2012, ch. 24, art. 100(A); 2015, ch. 2, art. 116(A); 2016, ch. 9, art. 71(A) et 72(A)",
+    "last_amended": "2019-01-15",
+    "in_force": "2016-12-12",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-7.html",
+    "id": "A-8.8-fr-s7",
+    "text": "7\n(1) Toute contravention désignée au titre de l’alinéa 4(1)a) constitue une violation pour laquelle le contrevenant s’expose à l’avertissement ou à la sanction prévus par la présente loi.\n(2) [Verbalisation] L’agent verbalisateur qui a des motifs raisonnables de croire qu’une violation a été commise peut dresser un procès-verbal qu’il fait notifier au contrevenant. Le procès-verbal comporte, outre le nom du contrevenant et les faits reprochés, soit un avertissement, soit le montant, établi en application du règlement, de la sanction à payer — auquel cas il précise le délai et les modalités de paiement — et, sous réserve des règlements, le montant inférieur de la sanction infligée prévu au procès-verbal dont le paiement, dans le délai et selon les modalités, vaut règlement.\n(3) [Sommaire des droits] Figurent aussi au procès-verbal en langage clair un sommaire des droits et obligations du contrevenant prévus par la présente loi, notamment le droit de contester les faits reprochés auprès du ministre ou de la Commission et la procédure pour le faire.",
+    "citation": "AAAMPA, art. 7",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "8",
+    "marginal_note": "Option",
+    "part": "",
+    "division": "",
+    "heading": "Avertissements",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-8.html",
+    "id": "A-8.8-fr-s8",
+    "text": "8\n(1) Si le procès-verbal comporte un avertissement, le contrevenant peut, dans le délai et selon les modalités réglementaires, contester les faits reprochés auprès du ministre ou de la Commission.\n(2) [Présomption] Le défaut du contrevenant d’exercer l’option dans le délai et selon les modalités prévus vaut déclaration de responsabilité à l’égard de la violation.",
+    "citation": "AAAMPA, art. 8",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "9",
+    "marginal_note": "Paiement",
+    "part": "",
+    "division": "",
+    "heading": "Sanctions",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-9.html",
+    "id": "A-8.8-fr-s9",
+    "text": "9\n(1) Si le procès-verbal inflige une sanction et que le contrevenant paie, dans le délai et selon les modalités réglementaires, le montant de celle-ci — ou, sous réserve des règlements, le montant inférieur prévu au procès-verbal — , le paiement, que le ministre accepte en règlement, vaut déclaration de responsabilité à l’égard de la violation et met fin à la poursuite.\n(2) [Option] À défaut d’effectuer le paiement, le contrevenant peut, dans le délai et selon les modalités réglementaires :\na) si la sanction est de 2 000 $ ou plus, demander au ministre de conclure une transaction en vue de la bonne application de la loi agroalimentaire ou du règlement en cause;\nb) contester auprès du ministre les faits reprochés;\nc) demander à la Commission de l’entendre sur les faits reprochés.\n(3) [Présomption] Le défaut du contrevenant d’exercer l’option visée au paragraphe (2) dans le délai et selon les modalités prévus vaut déclaration de responsabilité à l’égard de la violation.",
+    "citation": "AAAMPA, art. 9",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "10",
+    "marginal_note": "Conclusion d’une transaction",
+    "part": "",
+    "division": "",
+    "heading": "Transactions",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-10.html",
+    "id": "A-8.8-fr-s10",
+    "text": "10\n(1) Sur demande du contrevenant, le ministre peut conclure une transaction qui, d’une part, est subordonnée aux conditions qu’il estime indiquées, notamment au dépôt d’une caution raisonnable — dont le montant et la nature doivent lui agréer — , en garantie de l’exécution de la transaction et, d’autre part, peut prévoir la réduction partielle ou complète du montant de la sanction.\n(2) [Présomption] La conclusion de la transaction par le ministre vaut déclaration de responsabilité à l’égard de la violation.\n(3) [Avis d’exécution] La notification à l’intéressé d’un avis du ministre déclarant que celui-ci estime la transaction exécutée met fin à la poursuite; dès lors la caution est remise à l’intéressé.\n(4) [Avis de défaut d’exécution] S’il estime la transaction inexécutée, le ministre fait notifier à l’intéressé un avis de défaut qui l’informe soit qu’il doit payer, au lieu du montant de la sanction initiale et sans qu’il soit tenu compte du plafond fixé au paragraphe 4(2), le double de ce montant, soit qu’il y aura confiscation de la caution au profit de Sa Majesté du chef du Canada.\n(5) [Effet de l’inexécution] Sur notification de l’avis, l’intéressé perd tout droit à la compensation pour les sommes exposées dans le cadre de la transaction. Aux termes de l’avis, il est tenu de payer le montant mentionné, ou la confiscation de la caution s’opère au profit de Sa Majesté du chef du Canada, ce qui met fin à la poursuite.\n(6) [Paiement] Le paiement, que le ministre accepte en règlement, met fin à la poursuite.",
+    "citation": "AAAMPA, art. 10",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "11",
+    "marginal_note": "Option en cas de refus de transiger",
+    "part": "",
+    "division": "",
+    "heading": "Transactions",
+    "history": "1995, ch. 40, art. 11; 2015, ch. 2, art. 117(A)",
+    "last_amended": "2015-02-27",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-11.html",
+    "id": "A-8.8-fr-s11",
+    "text": "11\n(1) Si le ministre refuse de transiger, le contrevenant peut, dans le délai et selon les modalités réglementaires, soit payer le montant de la sanction infligée initialement, soit demander à la Commission de l’entendre sur les faits reprochés.\n(2) [Paiement] Le paiement, que le ministre accepte en règlement, vaut déclaration de responsabilité à l’égard de la violation et met fin à la poursuite.\n(3) [Présomption] Le défaut du contrevenant d’exercer l’option dans le délai et selon les modalités prévus vaut déclaration de responsabilité à l’égard de la violation.",
+    "citation": "AAAMPA, art. 11",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "12",
+    "marginal_note": "Décision du ministre : avertissement",
+    "part": "",
+    "division": "",
+    "heading": "Contestation devant le ministre",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-12.html",
+    "id": "A-8.8-fr-s12",
+    "text": "12\n(1) Saisi d’une contestation au titre de l’article 8, le ministre détermine la responsabilité du contrevenant et lui fait notifier sa décision.\n(2) [Demande de révision] Le contrevenant peut alors, dans le délai et selon les modalités réglementaires, demander à la Commission de l’entendre sur la décision du ministre.",
+    "citation": "AAAMPA, art. 12",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "13",
+    "marginal_note": "Décision du ministre : sanction",
+    "part": "",
+    "division": "",
+    "heading": "Contestation devant le ministre",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-13.html",
+    "id": "A-8.8-fr-s13",
+    "text": "13\n(1) Saisi d’une contestation au titre de l’alinéa 9(2)b), le ministre détermine la responsabilité du contrevenant et lui fait notifier sa décision. S’il juge que le montant de la sanction n’a pas été établi en application des règlements, il y substitue le montant qu’il estime conforme.\n(2) [Option] Le contrevenant peut, dans le délai et selon les modalités réglementaires, soit payer le montant mentionné — paiement que le ministre accepte en règlement et qui met fin à la poursuite — , soit demander à la Commission de l’entendre sur la décision du ministre.",
+    "citation": "AAAMPA, art. 13",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "14",
+    "marginal_note": "Pouvoir de la Commission",
+    "part": "",
+    "division": "",
+    "heading": "Révision par la Commission",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-14.html",
+    "id": "A-8.8-fr-s14",
+    "text": "14\n(1) Saisie d’une affaire au titre de la présente loi, la Commission, par ordonnance et selon le cas, soit confirme, modifie ou annule la décision du ministre, soit détermine la responsabilité du contrevenant; en outre, si elle estime que le montant de la sanction n’a pas été établi en application des règlements, elle y substitue le montant qu’elle juge conforme. Elle fait notifier l’ordonnance à l’intéressé et au ministre.\n(2) [Paiement] Le paiement du montant conformément à l’ordonnance, que le ministre accepte en règlement, met fin à la poursuite.",
+    "citation": "AAAMPA, art. 14",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "15",
+    "marginal_note": "Créance de Sa Majesté",
+    "part": "",
+    "division": "",
+    "heading": "Exécution des sanctions",
+    "history": "1995, ch. 40, art. 15; 2012, ch. 24, art. 101; 2015, ch. 2, art. 118(F)",
+    "last_amended": "2019-01-15",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-15.html",
+    "id": "A-8.8-fr-s15",
+    "text": "15\n(1) Constitue une créance de Sa Majesté du chef du Canada, dont le recouvrement peut être poursuivi à ce titre devant la Cour fédérale :\na) le montant de la sanction, à compter de la date de notification du procès-verbal;\nb) tout montant prévu dans une transaction conclue au titre du paragraphe 10(1), à compter de la date de sa conclusion;\nc) le montant mentionné dans l’avis de défaut notifié au titre du paragraphe 10(4), à compter de la date de sa notification;\nd) le montant mentionné dans la décision notifiée au titre du paragraphe 13(1), à compter de la date de sa notification;\ne) le montant mentionné dans l’ordonnance visée au paragraphe 14(1), à compter de l’expiration du délai fixé par la Commission pour le payer;\nf) le montant des frais raisonnables visés à l’article 22, à compter de la date où ils ont été faits.\n(2) [Prescription] Le recouvrement de la créance se prescrit par cinq ans à compter de la date à laquelle elle est devenue exigible.\n(3) [Conditions de révision] La créance est définitive et n’est susceptible de contestation ou de révision que dans la mesure et selon les modalités prévues aux articles 9 à 14.",
+    "citation": "AAAMPA, art. 15",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "16",
+    "marginal_note": "Certificat de non-paiement",
+    "part": "",
+    "division": "",
+    "heading": "Exécution des sanctions",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-16.html",
+    "id": "A-8.8-fr-s16",
+    "text": "16\n(1) Le ministre peut établir un certificat de non-paiement pour la partie impayée des créances visées au paragraphe 15(1).\n(2) [Enregistrement en Cour fédérale] L’enregistrement à la Cour fédérale confère au certificat la valeur d’un jugement de cette juridiction pour la somme visée et les frais afférents.",
+    "citation": "AAAMPA, art. 16",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "17",
+    "marginal_note": "Précision",
+    "part": "",
+    "division": "",
+    "heading": "Règles spécifiques aux violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-17.html",
+    "id": "A-8.8-fr-s17",
+    "text": "17 Les violations n’ont pas valeur d’infractions; en conséquence nul ne peut être poursuivi à ce titre sur le fondement de l’article 126 du Code criminel.",
+    "citation": "AAAMPA, art. 17",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "18",
+    "marginal_note": "Exclusion de certains moyens de défense",
+    "part": "",
+    "division": "",
+    "heading": "Règles spécifiques aux violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-18.html",
+    "id": "A-8.8-fr-s18",
+    "text": "18\n(1) Le contrevenant ne peut invoquer en défense le fait qu’il a pris les mesures nécessaires pour empêcher la violation ou qu’il croyait raisonnablement et en toute honnêteté à l’existence de faits qui, avérés, l’exonéreraient.\n(2) [Principes de la common law] Les règles et principes de la common law qui font d’une circonstance une justification ou une excuse dans le cadre d’une poursuite pour infraction à une loi agroalimentaire s’appliquent à l’égard d’une violation sauf dans la mesure où ils sont incompatibles avec la présente loi.",
+    "citation": "AAAMPA, art. 18",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "19",
+    "marginal_note": "Charge de la preuve",
+    "part": "",
+    "division": "",
+    "heading": "Règles spécifiques aux violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-19.html",
+    "id": "A-8.8-fr-s19",
+    "text": "19 En cas de contestation devant le ministre ou de révision par la Commission, portant sur les faits, il appartient au ministre d’établir, selon la prépondérance des probabilités, la responsabilité du contrevenant.",
+    "citation": "AAAMPA, art. 19",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "20",
+    "marginal_note": "Responsabilité indirecte : titulaires",
+    "part": "",
+    "division": "",
+    "heading": "Règles spécifiques aux violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-20.html",
+    "id": "A-8.8-fr-s20",
+    "text": "20\n(1) Le titulaire d’un agrément — licence, permis ou autre type d’autorisation — délivré en vertu d’une loi agroalimentaire est responsable de la violation commise dans le cadre des activités ou des obligations visées par l’agrément, que l’auteur de la contravention soit ou non connu ou poursuivi aux termes de la présente loi.\n(2) [Responsabilité indirecte : employeurs et mandants] L’employeur ou le mandant est responsable de la violation commise, dans le cadre de son emploi ou du mandat, par un employé ou un mandataire, que l’auteur de la contravention soit ou non connu ou poursuivi aux termes de la présente loi.",
+    "citation": "AAAMPA, art. 20",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "21",
+    "marginal_note": "Violation continue",
+    "part": "",
+    "division": "",
+    "heading": "Règles spécifiques aux violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-21.html",
+    "id": "A-8.8-fr-s21",
+    "text": "21 Il est compté une violation distincte pour chacun des jours au cours desquels se continue une violation.",
+    "citation": "AAAMPA, art. 21",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "22",
+    "marginal_note": "Confiscation",
+    "part": "",
+    "division": "",
+    "heading": "Règles spécifiques aux violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-22.html",
+    "id": "A-8.8-fr-s22",
+    "text": "22 Il y a confiscation au profit de Sa Majesté du chef du Canada — si elle en décide ainsi — de tout objet détenu ou saisi, relativement à une violation, au titre d’une loi agroalimentaire dès lors que le contrevenant est déclaré ou réputé être responsable de la violation et que, dans ce dernier cas, il n’a pas, dans le délai et selon les modalités réglementaires, saisi la Commission d’une demande de révision; il en est alors disposé, aux frais du saisi, conformément, sous réserve des instructions du ministre, au règlement pris au titre de la loi agroalimentaire en cause.",
+    "citation": "AAAMPA, art. 22",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "23",
+    "marginal_note": "Dossiers",
+    "part": "",
+    "division": "",
+    "heading": "Dispositions générales",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-23.html",
+    "id": "A-8.8-fr-s23",
+    "text": "23\n(1) Sur demande du contrevenant, toute mention relative à une violation est rayée du dossier que le ministre tient à son égard cinq ans après la date soit du paiement de toute créance visée au paragraphe 15(1), soit de la notification d’un procès-verbal comportant un avertissement, à moins que celui-ci estime que ce serait contraire à l’intérêt public ou qu’une autre mention ait été portée au dossier au sujet de l’intéressé par la suite, mais n’ait pas été rayée.\n(2) [Notification] Le ministre fait notifier un avis de radiation à l’intéressé.",
+    "citation": "AAAMPA, art. 23",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "24",
+    "marginal_note": "Notification",
+    "part": "",
+    "division": "",
+    "heading": "Dispositions générales",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-24.html",
+    "id": "A-8.8-fr-s24",
+    "text": "24 Toute notification autorisée ou exigée par la présente loi s’effectue conformément au règlement, par remise à personne ou de toute autre manière qui y est autorisée.",
+    "citation": "AAAMPA, art. 24",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "25",
+    "marginal_note": "Admissibilité du procès-verbal de violation",
+    "part": "",
+    "division": "",
+    "heading": "Dispositions générales",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-25.html",
+    "id": "A-8.8-fr-s25",
+    "text": "25 Dans les poursuites pour violation ou pour infraction, le procès-verbal censé délivré en application de la présente loi est admissible en preuve sans qu’il soit nécessaire de prouver l’authenticité de la signature ni la qualité officielle du signataire.",
+    "citation": "AAAMPA, art. 25",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "26",
+    "marginal_note": "Prescription",
+    "part": "",
+    "division": "",
+    "heading": "Dispositions générales",
+    "history": "1995, ch. 40, art. 26; 2015, ch. 2, art. 119",
+    "last_amended": "2015-02-27",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-26.html",
+    "id": "A-8.8-fr-s26",
+    "text": "26 Les poursuites pour violation se prescrivent par six mois à compter de la date à laquelle elle a été commise, lorsque celle-ci est mineure, et par deux ans, lorsqu’elle est grave ou très grave.",
+    "citation": "AAAMPA, art. 26",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "27",
+    "marginal_note": "Prorogation de la Commission",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, ch. 40, art. 27; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-27.html",
+    "id": "A-8.8-fr-s27",
+    "text": "27\n(1) Est prorogée la Commission de révision prorogée par le paragraphe 4.1(1) la Loi sur les produits agricoles au Canada, chapitre 20 du 4e supplément des Lois révisées du Canada (1985).\n(2) [Composition] La Commission est composée des membres, dont le président, nommés par le gouverneur en conseil.",
+    "citation": "AAAMPA, art. 27",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "28",
+    "marginal_note": "Compétences",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, ch. 40, art. 28; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-28.html",
+    "id": "A-8.8-fr-s28",
+    "text": "28 Les membres sont nommés en raison de leurs connaissances ou de leur expérience dans le domaine de l’agriculture ou de l’agroalimentaire et le président et au moins un autre membre sont obligatoirement choisis parmi les avocats ou notaires inscrits respectivement, depuis au moins dix ans, au barreau d’une province ou à la Chambre des notaires du Québec.",
+    "citation": "AAAMPA, art. 28",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "29",
+    "marginal_note": "Exercice des fonctions",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, ch. 40, art. 29; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-29.html",
+    "id": "A-8.8-fr-s29",
+    "text": "29 Le président exerce ses fonctions à temps plein; les autres membres, à temps plein ou à temps partiel.",
+    "citation": "AAAMPA, art. 29",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "30",
+    "marginal_note": "Mandat",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, ch. 40, art. 30; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-30.html",
+    "id": "A-8.8-fr-s30",
+    "text": "30\n(1) Les membres occupent leur poste à titre inamovible pour un mandat maximal de cinq ans, sous réserve de révocation motivée prononcée par le gouverneur en conseil.\n(2) [Nouveau mandat] Les membres peuvent recevoir un nouveau mandat, aux fonctions identiques ou non.",
+    "citation": "AAAMPA, art. 30",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "31",
+    "marginal_note": "Incompatibilité de fonctions",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, ch. 40, art. 31; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-31.html",
+    "id": "A-8.8-fr-s31",
+    "text": "31 La charge de membre est incompatible avec d’autres fonctions dans l’administration publique fédérale.",
+    "citation": "AAAMPA, art. 31",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "32",
+    "marginal_note": "Conflits d’intérêts",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, ch. 40, art. 32; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-32.html",
+    "id": "A-8.8-fr-s32",
+    "text": "32 Les membres ne peuvent accepter ni occuper de charge ou d’emploi incompatibles avec leurs fonctions, ni se saisir d’une affaire dans laquelle ils ont un intérêt.",
+    "citation": "AAAMPA, art. 32",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "33",
+    "marginal_note": "Fonctions du président",
+    "part": "",
+    "division": "",
+    "heading": "Président",
+    "history": "1995, ch. 40, art. 33; 2012, ch. 24, art. 102; 2014, ch. 20, art. 480",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-33.html",
+    "id": "A-8.8-fr-s33",
+    "text": "33\n(1) Le président de la Commission répartit les tâches entre les membres.\n(2) [Absence ou empêchement] En cas d’absence ou d’empêchement du président, ou de vacance de son poste, les autres membres confient l’intérim à l’un des membres dotés de la compétence juridique prévue à l’article 28. Cependant, l’intérim ne peut dépasser soixante jours sans l’approbation du gouverneur en conseil.",
+    "citation": "AAAMPA, art. 33",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "34",
+    "marginal_note": "Indemnités",
+    "part": "",
+    "division": "",
+    "heading": "Indemnités et frais",
+    "history": "1995, ch. 40, art. 34; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-34.html",
+    "id": "A-8.8-fr-s34",
+    "text": "34\n(1) Les membres à temps plein reçoivent le traitement, et les autres membres reçoivent les honoraires ou toute autre rémunération, que fixe le gouverneur en conseil.\n(2) [Frais] Les membres sont indemnisés des frais de déplacement et de séjour entraînés par l’exercice de leurs attributions hors de leur lieu habituel soit de travail, s’ils sont à temps plein, soit de résidence, s’ils sont à temps partiel, conformément aux directives du Conseil du Trésor.",
+    "citation": "AAAMPA, art. 34",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "35",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "Indemnités et frais",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-35.html",
+    "id": "A-8.8-fr-s35",
+    "text": "35 [Abrogé, 2014, ch. 20, art. 480]",
+    "citation": "AAAMPA, art. 35",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "36",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "Indemnités et frais",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-36.html",
+    "id": "A-8.8-fr-s36",
+    "text": "36 [Abrogé, 2014, ch. 20, art. 480]",
+    "citation": "AAAMPA, art. 36",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "37",
+    "marginal_note": "Siège",
+    "part": "",
+    "division": "",
+    "heading": "Siège",
+    "history": "1995, ch. 40, art. 37; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-37.html",
+    "id": "A-8.8-fr-s37",
+    "text": "37\n(1) Le siège de la Commission est fixé dans la région de la capitale nationale au sens de l’article 2 de la Loi sur la capitale nationale.\n(2) [Réunions] La Commission siège en tout lieu du Canada fixé par le gouverneur en conseil.",
+    "citation": "AAAMPA, art. 37",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "38",
+    "marginal_note": "Commission",
+    "part": "",
+    "division": "",
+    "heading": "Compétence",
+    "history": "1995, ch. 40, art. 38; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-38.html",
+    "id": "A-8.8-fr-s38",
+    "text": "38\n(1) La Commission a compétence exclusive pour les affaires relevant des domaines qui lui sont attribués sous le régime de la présente loi ou de toute autre loi fédérale.\n(2) [Révision en Cour fédérale] Les ordonnances de la Commission ne sont susceptibles de révision qu’au titre de la Loi sur les Cours fédérales.",
+    "citation": "AAAMPA, art. 38",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "39",
+    "marginal_note": "Président",
+    "part": "",
+    "division": "",
+    "heading": "Compétence",
+    "history": "1995, ch. 40, art. 39; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-39.html",
+    "id": "A-8.8-fr-s39",
+    "text": "39\n(1) Le président de la Commission a seul compétence pour entendre :\na) les demandes touchant les avertissements formées au titre des paragraphes 8(1) ou 12(2);\nb) les demandes touchant les sanctions de moins de 2 000 $ formées au titre de l’alinéa 9(2)c) ou du paragraphe 13(2).\n(2) [Délégation] Le président peut déléguer aux autres membres ses attributions relativement aux demandes mentionnées au paragraphe (1) s’ils sont dotés de la compétence juridique prévue à l’article 28.",
+    "citation": "AAAMPA, art. 39",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "40",
+    "marginal_note": "Révision",
+    "part": "",
+    "division": "",
+    "heading": "Compétence",
+    "history": "1995, ch. 40, art. 40; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-40.html",
+    "id": "A-8.8-fr-s40",
+    "text": "40 Les demandes de révision sont entendues par un membre seul.",
+    "citation": "AAAMPA, art. 40",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "41",
+    "marginal_note": "Cour d’archives",
+    "part": "",
+    "division": "",
+    "heading": "Pouvoirs",
+    "history": "1995, ch. 40, art. 41; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-41.html",
+    "id": "A-8.8-fr-s41",
+    "text": "41\n(1) La Commission est une cour d’archives; elle a un sceau officiel, dont l’authenticité est admise d’office.\n(2) [Interrogatoire des témoins] En outre, la Commission a, pour la comparution, la prestation des serments et l’interrogatoire des témoins, ainsi que pour la production et l’examen des documents et autres pièces, l’exécution de ses ordonnances et toutes autres questions relevant de sa compétence, les attributions d’une juridiction supérieure d’archives. Elle peut notamment :\na) par citation adressée aux personnes ayant connaissance de faits se rapportant à l’affaire dont elle est saisie, leur enjoindre de comparaître comme témoins aux date, heure et lieu indiqués et d’apporter et de produire tous documents, livres ou pièces utiles à l’affaire, dont elles ont la possession ou la responsabilité;\nb) faire prêter serment et interroger sous serment;\nc) recevoir en cours d’audition les éléments de preuve supplémentaires qu’elle estime utiles et dignes de foi.",
+    "citation": "AAAMPA, art. 41",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "42",
+    "marginal_note": "Règles",
+    "part": "",
+    "division": "",
+    "heading": "Règles",
+    "history": "1995, ch. 40, art. 42; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-42.html",
+    "id": "A-8.8-fr-s42",
+    "text": "42 La Commission peut, avec l’agrément du gouverneur en conseil, établir des règles régissant :\na) la pratique et la procédure des audiences;\nb) les modalités, y compris les délais, d’établissement des demandes et des avis à donner;\nc) de façon générale, l’exercice de ses activités sous le régime de la présente loi ou de toute autre loi fédérale.",
+    "citation": "AAAMPA, art. 42",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "43",
+    "marginal_note": "Consultations",
+    "part": "",
+    "division": "",
+    "heading": "Généralités",
+    "history": "1995, ch. 40, art. 43; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-43.html",
+    "id": "A-8.8-fr-s43",
+    "text": "43 Les membres peuvent, dans le cadre des affaires dont la Commission est saisie, consulter d’autres membres.",
+    "citation": "AAAMPA, art. 43",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "44",
+    "marginal_note": "Règles en matière de preuve",
+    "part": "",
+    "division": "",
+    "heading": "Généralités",
+    "history": "1995, ch. 40, art. 44; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-44.html",
+    "id": "A-8.8-fr-s44",
+    "text": "44 La Commission n’est pas liée par les règles juridiques ou techniques applicables en matière de preuve lors des audiences. Dans la mesure où les circonstances, l’équité et la justice naturelle le permettent, il lui appartient d’agir rapidement et sans formalités.",
+    "citation": "AAAMPA, art. 44",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "45",
+    "marginal_note": "Exception en matière de preuve",
+    "part": "",
+    "division": "",
+    "heading": "Généralités",
+    "history": "1995, ch. 40, art. 45; 2012, ch. 24, art. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-45.html",
+    "id": "A-8.8-fr-s45",
+    "text": "45 La Commission ne peut recevoir ni admettre en preuve les éléments protégés par le droit de la preuve et rendus, de ce fait, inadmissibles en justice devant un tribunal judiciaire.",
+    "citation": "AAAMPA, art. 45",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "46 à 89",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "Généralités",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-46 à 89.html",
+    "id": "A-8.8-fr-s46 à 89",
+    "text": "46 à 89 [Abrogés, 2012, ch. 24, art. 102]",
+    "citation": "AAAMPA, art. 46 à 89",
+    "lang": "fr"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "90",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "Généralités",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/fra/lois/a-8.8/section-90.html",
+    "id": "A-8.8-fr-s90",
+    "text": "90 [Abrogé, 2012, ch. 24, art. 102]",
+    "citation": "AAAMPA, art. 90",
+    "lang": "fr"
+  }
+]

data/processed/A-8.8.json

@@ -0,0 +1,933 @@
+[
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "1",
+    "marginal_note": "Short title",
+    "part": "",
+    "division": "",
+    "heading": "Short Title",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-1.html",
+    "id": "A-8.8-s1",
+    "text": "1 This Act may be cited as the Agriculture and Agri-Food Administrative Monetary Penalties Act.",
+    "citation": "AAAMPA, s. 1"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "2",
+    "marginal_note": "Definitions",
+    "part": "",
+    "division": "",
+    "heading": "Interpretation",
+    "history": "1995, c. 40, s. 2; 1997, c. 21, s. 30; 2002, c. 28, s. 82; 2005, c. 38, ss. 30, 145; 2012, c. 24, s. 98; 2015, c. 2, s. 113",
+    "last_amended": "2019-01-15",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-2.html",
+    "id": "A-8.8-s2",
+    "text": "2 In this Act,\nagri-food Act means the Farm Debt Mediation Act, the Feeds Act, the Fertilizers Act, the Health of Animals Act, the Pest Control Products Act, the Plant Protection Act, the Safe Food for Canadians Act or the Seeds Act; (loi agroalimentaire)\nMinister means the Minister of Agriculture and Agri-Food, except that\n(a) it means the Minister of Health in relation to a violation involving a contravention of\n(i) the Pest Control Products Act, or\n(ii) a provision relating to food safety of an agri-food Act or of a regulation made under such an Act, and\n(b) it means the Minister of Public Safety and Emergency Preparedness in relation to a notice of violation issued in respect of the contravention of program legislation referred to in subsection 11(5) of the Canadian Food Inspection Agency Act; (ministre)\npenalty means an administrative monetary penalty imposed under this Act for a violation; (sanction)\nprescribed means prescribed by regulation; (Version anglaise seulement)\nTribunal means the Review Tribunal continued by subsection 27(1); (Commission)",
+    "citation": "AAAMPA, s. 2"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "3",
+    "marginal_note": "Purpose of Act",
+    "part": "",
+    "division": "",
+    "heading": "Purpose of Act",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-3.html",
+    "id": "A-8.8-s3",
+    "text": "3 The purpose of this Act is to establish, as an alternative to the existing penal system and as a supplement to existing enforcement measures, a fair and efficient administrative monetary penalty system for the enforcement of the agri-food Acts.",
+    "citation": "AAAMPA, s. 3"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "4",
+    "marginal_note": "Regulations",
+    "part": "",
+    "division": "",
+    "heading": "Powers of Minister",
+    "history": "1995, c. 40, s. 4; 2012, c. 24, s. 99; 2015, c. 2, s. 114; 2016, c. 9, ss. 70, 72",
+    "last_amended": "2019-01-15",
+    "in_force": "2016-12-12",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-4.html",
+    "id": "A-8.8-s4-p1",
+    "text": "4\n(1) The Minister may make regulations\n(a) designating as a violation that may be proceeded with in accordance with this Act\n(i) the contravention of any specified provision of an agri-food Act or of a regulation made under an agri-food Act,\n(ii) the contravention of any specified order, or class of orders, made by the Minister under the Plant Protection Act, or\n(iii) the refusal or neglect to perform any specified duty, or class of duties, imposed by or under the Plant Protection Act, the Health of Animals Act, the Pest Control Products Act or the Safe Food for Canadians Act;\nif the contravention, or the failure or neglect to perform the duty, as the case may be, is an offence under an agri-food Act;\n(b) classifying each violation as a minor violation, a serious violation or a very serious violation;\n(b.1) establishing, in respect of each violation, a short-form description to be used in notices of violation;\n(c) fixing a penalty, or a range of penalties, in respect of each violation;\n(d) respecting the circumstances under which, the criteria by which and the manner in which a penalty may be increased or reduced, including the reduction of a penalty pursuant to a compliance agreement under subsection 10(1);\n(e) respecting the determination of a lesser amount that may be paid in complete satisfaction of a penalty if paid within the prescribed time and manner;\n(f) respecting the circumstances under which reviews under this Act by the Tribunal shall be oral or in writing;",
+    "citation": "AAAMPA, s. 4 (part 1 of 2)"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "4",
+    "marginal_note": "Regulations",
+    "part": "",
+    "division": "",
+    "heading": "Powers of Minister",
+    "history": "1995, c. 40, s. 4; 2012, c. 24, s. 99; 2015, c. 2, s. 114; 2016, c. 9, ss. 70, 72",
+    "last_amended": "2019-01-15",
+    "in_force": "2016-12-12",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-4.html",
+    "id": "A-8.8-s4-p2",
+    "text": "(g) respecting the service of documents required or authorized to be served under this Act including, without restricting the generality of the foregoing, the manner of serving such documents, the proof of their service and the circumstances under which such documents shall be deemed to have been served;\n(h) prescribing anything that by this Act is to be prescribed; and\n(i) generally, for carrying out the purposes and provisions of this Act.\n(2) [Maximum penalties] The maximum penalty for a violation is\n(a) in the case of a violation that is committed by an individual otherwise than in the course of a business and that is not committed to obtain a financial benefit, $2,000; and\n(b) in any other case, $5,000 for a minor violation, $15,000 for a serious violation and $25,000 for a very serious violation.\n(3) [Criteria] Without restricting the generality of paragraph (1)(d), in making regulations respecting the criteria for increasing or reducing the amount of the penalty for a violation, the Minister shall include the following in any such criteria:\n(a) the degree of intention or negligence on the part of the person who committed the violation;\n(b) the harm done by the violation; and\n(c) the history of the person who committed the violation of prior violations or convictions under agri-food Acts within the five year period immediately before the violation.",
+    "citation": "AAAMPA, s. 4 (part 2 of 2)"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "5",
+    "marginal_note": "How act or omission may be proceeded with",
+    "part": "",
+    "division": "",
+    "heading": "Powers of Minister",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-5.html",
+    "id": "A-8.8-s5",
+    "text": "5 Where any act or omission can be proceeded with as a violation or as an offence, the Minister may commence proceedings in respect of that act or omission as a violation or recommend that it be proceeded with as an offence, but proceeding with it as a violation precludes proceeding with it as an offence, and proceeding with it as an offence precludes proceeding with it as a violation.",
+    "citation": "AAAMPA, s. 5"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "6",
+    "marginal_note": "Power of Minister — notices of violation",
+    "part": "",
+    "division": "",
+    "heading": "Powers of Minister",
+    "history": "1995, c. 40, s. 6; 2015, c. 2, s. 115",
+    "last_amended": "2015-02-27",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-6.html",
+    "id": "A-8.8-s6",
+    "text": "6 The Minister may designate persons, or classes of persons, who are authorized to issue notices of violation.",
+    "citation": "AAAMPA, s. 6"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "7",
+    "marginal_note": "Commission of violation",
+    "part": "",
+    "division": "",
+    "heading": "Proceedings",
+    "history": "1995, c. 40, s. 7; 2012, c. 24, s. 100(E); 2015, c. 2, s. 116(E); 2016, c. 9, ss. 71(E), 72(E)",
+    "last_amended": "2019-01-15",
+    "in_force": "2016-12-12",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-7.html",
+    "id": "A-8.8-s7",
+    "text": "7\n(1) Every person who\n(a) contravenes any provision of an agri-food Act or of a regulation made under an agri-food Act,\n(b) contravenes any order made by the Minister under the Plant Protection Act, or\n(c) refuses or neglects to perform any duty imposed by or under the Plant Protection Act, the Health of Animals Act, the Pest Control Products Act or the Safe Food for Canadians Act\nthe contravention of which, or the refusal or neglect of which, is designated to be a violation by a regulation made under paragraph 4(1)(a) commits a violation and is liable to a warning or to a penalty in accordance with this Act.\n(2) [Issuance of notice of violation] If a person designated under section 6 has reasonable grounds to believe that a person has committed a violation, the designated person may issue, and shall cause to be served on the person, a notice of violation that names the person, identifies the violation and\n(a) contains a warning that the person has committed a violation; or\n(b) sets out\n(i) the penalty, established in accordance with the regulations, for the violation that the person is liable to pay,\n(ii) particulars concerning the time for paying and the manner of paying the penalty, and\n(iii) subject to the regulations, a lesser amount that may be paid in complete satisfaction of the penalty if paid within the time and manner specified in the notice.\n(3) [Summary of rights] A notice of violation must clearly summarize, in plain language, the rights and obligations under this Act of the person on whom it is served, including the right to have the facts of the violation reviewed by the Minister or the Tribunal, and the procedure for requesting such a review.",
+    "citation": "AAAMPA, s. 7"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "8",
+    "marginal_note": "Notices with warning — request for review",
+    "part": "",
+    "division": "",
+    "heading": "Notices of Violation with Warning",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-8.html",
+    "id": "A-8.8-s8",
+    "text": "8\n(1) Where a notice of violation contains a warning, the person named in the notice may, in the prescribed time and manner, request a review of the facts of the violation by the Minister or the Tribunal.\n(2) [Deeming] Where a person who is served with a notice of violation that contains a warning does not request a review under subsection (1) in the prescribed time and manner, the person is deemed to have committed the violation identified in the notice of violation.",
+    "citation": "AAAMPA, s. 8"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "9",
+    "marginal_note": "Notices with penalty — payment",
+    "part": "",
+    "division": "",
+    "heading": "Notices of Violation with Penalty",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-9.html",
+    "id": "A-8.8-s9",
+    "text": "9\n(1) Where a notice of violation sets out a penalty and the person named in the notice pays, in the prescribed time and manner, the amount of the penalty or, subject to the regulations, the lesser amount set out in the notice that may be paid in lieu of the penalty,\n(a) the person is deemed to have committed the violation in respect of which the amount is paid;\n(b) the Minister shall accept that amount as and in complete satisfaction of the penalty; and\n(c) the proceedings commenced in respect of the violation under section 7 are ended.\n(2) [Alternatives to payment] Instead of paying the penalty set out in a notice of violation or, where applicable, the lesser amount that may be paid in lieu of the penalty, the person named in the notice may, in the prescribed time and manner,\n(a) if the penalty is $2,000 or more, request to enter into a compliance agreement with the Minister that ensures the person’s compliance with the agri-food Act or regulation to which the violation relates;\n(b) request a review by the Minister of the facts of the violation; or\n(c) request a review by the Tribunal of the facts of the violation.\n(3) [Deeming] Where a person who is served with a notice of violation that sets out a penalty does not pay the penalty in the prescribed time and manner or, where applicable, the lesser amount that may be paid in lieu of the penalty, and does not exercise any right referred to in subsection (2) in the prescribed time and manner, the person is deemed to have committed the violation identified in the notice.",
+    "citation": "AAAMPA, s. 9"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "10",
+    "marginal_note": "Compliance agreements",
+    "part": "",
+    "division": "",
+    "heading": "Compliance Agreements",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-10.html",
+    "id": "A-8.8-s10-p1",
+    "text": "10\n(1) After considering a request under paragraph 9(2)(a), the Minister may enter into a compliance agreement, as described in that paragraph, with the person making the request on such terms and conditions as are satisfactory to the Minister, which terms may\n(a) include a provision for the giving of reasonable security, in a form and in an amount satisfactory to the Minister, as a guarantee that the person will comply with the compliance agreement; and\n(b) provide for the reduction, in whole or in part, of the penalty for the violation.\n(2) [Deeming] A person who makes a request under paragraph 9(2)(a) and who enters into a compliance agreement with the Minister shall, on entering into the compliance agreement, be deemed to have committed the violation in respect of which the compliance agreement was entered into.\n(3) [Where compliance agreement complied with] Where the Minister is satisfied that a person who has entered into a compliance agreement has complied with the agreement, the Minister shall cause a notice to that effect to be served on the person and, on the service of that notice,\n(a) the proceedings commenced in respect of the violation under section 7 are ended; and\n(b) any security given under the compliance agreement by the person shall be returned to the person.\n(4) [Where compliance agreement not complied with] Where the Minister is of the opinion that a person who has entered into a compliance agreement has not complied with the agreement, the Minister shall cause a notice of default to be served on the person to the effect that",
+    "citation": "AAAMPA, s. 10 (part 1 of 2)"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "10",
+    "marginal_note": "Compliance agreements",
+    "part": "",
+    "division": "",
+    "heading": "Compliance Agreements",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-10.html",
+    "id": "A-8.8-s10-p2",
+    "text": "(a) instead of the penalty set out in the notice of violation in respect of which the compliance agreement was entered into, the person is liable to pay twice the amount of that penalty and, for greater certainty, subsection 4(2) does not apply in respect of that amount; or\n(b) the security, if any, given under the compliance agreement by the person shall be forfeited to Her Majesty in right of Canada.\n(5) [Effect of notice of default] On the service of a notice under subsection (4), the person served has no right of set-off against any amount spent by the person under the compliance agreement and\n(a) the person served is liable to pay the amount set out in the notice; or\n(b) where the notice of default provides for the forfeiture of the security given under the compliance agreement, that security is forfeited to Her Majesty in right of Canada and the proceedings commenced in respect of the violation under section 7 are ended.\n(6) [Effect of payment] Where a person pays the amount set out in a notice of default under subsection (4) in the prescribed time and manner,\n(a) the Minister shall accept the amount as and in complete satisfaction of the amount owing; and\n(b) the proceedings commenced in respect of the violation under section 7 are ended.",
+    "citation": "AAAMPA, s. 10 (part 2 of 2)"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "11",
+    "marginal_note": "Refusal to enter into compliance agreement",
+    "part": "",
+    "division": "",
+    "heading": "Compliance Agreements",
+    "history": "1995, c. 40, s. 11; 2015, c. 2, s. 117(E)",
+    "last_amended": "2015-02-27",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-11.html",
+    "id": "A-8.8-s11",
+    "text": "11\n(1) Where the Minister refuses to enter into a compliance agreement pursuant to a request under paragraph 9(2)(a), the person who made the request may, in the prescribed time and manner,\n(a) pay the amount of the penalty; or\n(b) request a review by the Tribunal of the facts of the violation.\n(2) [Effect of payment] Where a person pays the amount referred to in paragraph (1)(a),\n(a) the person is deemed to have committed the violation in respect of which the payment is made;\n(b) the Minister shall accept the amount as and in complete satisfaction of the penalty; and\n(c) the proceedings commenced in respect of the violation under section 7 are ended.\n(3) [Deeming] If a person does not, in the prescribed time and manner, either pay the amount referred to in paragraph (1)(a) or request a review under paragraph (1)(b), the person is deemed to have committed the violation identified in the notice of violation.",
+    "citation": "AAAMPA, s. 11"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "12",
+    "marginal_note": "Review under section 8",
+    "part": "",
+    "division": "",
+    "heading": "Review by Minister",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-12.html",
+    "id": "A-8.8-s12",
+    "text": "12\n(1) After concluding a review requested under section 8, the Minister shall determine whether or not the person committed the violation, and the Minister shall cause a notice of any decision under this subsection to be served on the person who requested the review.\n(2) [Right to review] Where the Minister decides under subsection (1) that a person has committed a violation, the person may, in the prescribed time and manner, request a review of the Minister’s decision by the Tribunal.",
+    "citation": "AAAMPA, s. 12"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "13",
+    "marginal_note": "Review under paragraph 9(2)(b)",
+    "part": "",
+    "division": "",
+    "heading": "Review by Minister",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-13.html",
+    "id": "A-8.8-s13",
+    "text": "13\n(1) After concluding a review requested under paragraph 9(2)(b), the Minister shall determine whether or not the person requesting the review committed a violation and, where the Minister decides that the person committed a violation but considers that the amount of the penalty for the violation was not established in accordance with the regulations, the Minister shall correct the amount of the penalty for the violation, and the Minister shall cause a notice of any decision under this subsection to be served on the person who requested the review.\n(2) [Payment or right to review] Where the Minister decides under subsection (1) that a person has committed a violation, the person may, in the prescribed time and manner,\n(a) pay the amount of the penalty set out in the notice referred to in subsection (1), in which case\n(i) the Minister shall accept the amount as and in complete satisfaction of the penalty, and\n(ii) the proceedings commenced in respect of the violation under section 7 are ended; or\n(b) request a review of the Minister’s decision by the Tribunal.",
+    "citation": "AAAMPA, s. 13"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "14",
+    "marginal_note": "Review by Tribunal",
+    "part": "",
+    "division": "",
+    "heading": "Review by Tribunal",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-14.html",
+    "id": "A-8.8-s14",
+    "text": "14\n(1) After concluding a review requested under this Act, the Tribunal shall, by order, as the case may be,\n(a) confirm, vary or set aside any decision of the Minister under section 12 or 13, or\n(b) determine whether or not the person requesting the review committed a violation and, where the Tribunal decides that the person committed a violation but considers that the amount of the penalty for the violation, if any, was not established in accordance with the regulations, the Tribunal shall correct the amount of the penalty,\nand the Tribunal shall cause a notice of any order made under this subsection to be served on the person who requested the review, and on the Minister.\n(2) [Payment] Where the Tribunal decides under subsection (1) that a person has committed a violation, the person is liable for the amount of the penalty as set out in the order of the Tribunal and, on the payment of that amount in the time and manner specified in the order,\n(a) the Minister shall accept the amount as and in complete satisfaction of the penalty; and\n(b) the proceedings commenced in respect of the violation under section 7 are ended.",
+    "citation": "AAAMPA, s. 14"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "15",
+    "marginal_note": "Debts to Her Majesty",
+    "part": "",
+    "division": "",
+    "heading": "Enforcement",
+    "history": "1995, c. 40, s. 15; 2012, c. 24, s. 101; 2015, c. 2, s. 118(F)",
+    "last_amended": "2019-01-15",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-15.html",
+    "id": "A-8.8-s15",
+    "text": "15\n(1) The following amounts constitute debts due to Her Majesty in right of Canada that may be recovered as such in the Federal Court:\n(a) the amount of a penalty, from the time the notice of violation setting out the penalty is served;\n(b) every amount undertaken to be paid pursuant to a compliance agreement entered into with the Minister under subsection 10(1), from the time the compliance agreement is entered into;\n(c) the amount set out in a notice of default referred to in subsection 10(4), from the time the notice is served;\n(d) the amount of a penalty as set out in a decision of the Minister under subsection 13(1), from the time the notice under that subsection is served;\n(e) the amount of a penalty as set out in an order of the Tribunal under subsection 14(1), from the expiration of the time specified in the order for the payment of that amount; and\n(f) the amount of any reasonable expenses incurred pursuant to section 22, from the date they are incurred.\n(2) [Time limit] No proceedings to recover a debt referred to in subsection (1) may be commenced later than five years after the debt became payable.\n(3) [Debt final] A debt referred to in subsection (1) is final and not subject to review or to be restrained, prohibited, removed, set aside or otherwise dealt with except to the extent and in the manner provided by sections 9 to 14.",
+    "citation": "AAAMPA, s. 15"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "16",
+    "marginal_note": "Certificate of default",
+    "part": "",
+    "division": "",
+    "heading": "Enforcement",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-16.html",
+    "id": "A-8.8-s16",
+    "text": "16\n(1) Any debt referred to in subsection 15(1) in respect of which there is a default of payment, or the part of any such debt that has not been paid, may be certified by the Minister.\n(2) [Judgments] On production to the Federal Court, a certificate made under subsection (1) shall be registered in that Court and, when registered, has the same force and effect, and all proceedings may be taken on the certificate, as if it were a judgment obtained in that Court for a debt of the amount specified in the certificate and all reasonable costs and charges attendant in the registration of the certificate.",
+    "citation": "AAAMPA, s. 16"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "17",
+    "marginal_note": "Violations not offences",
+    "part": "",
+    "division": "",
+    "heading": "Rules of Law About Violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-17.html",
+    "id": "A-8.8-s17",
+    "text": "17 For greater certainty, a violation is not an offence and, accordingly, section 126 of the Criminal Code does not apply.",
+    "citation": "AAAMPA, s. 17"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "18",
+    "marginal_note": "Certain defences not available",
+    "part": "",
+    "division": "",
+    "heading": "Rules of Law About Violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-18.html",
+    "id": "A-8.8-s18",
+    "text": "18\n(1) A person named in a notice of violation does not have a defence by reason that the person\n(a) exercised due diligence to prevent the violation; or\n(b) reasonably and honestly believed in the existence of facts that, if true, would exonerate the person.\n(2) [Common law principles] Every rule and principle of the common law that renders any circumstance a justification or excuse in relation to a charge for an offence under an agri-food Act applies in respect of a violation to the extent that it is not inconsistent with this Act.",
+    "citation": "AAAMPA, s. 18"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "19",
+    "marginal_note": "Burden of proof",
+    "part": "",
+    "division": "",
+    "heading": "Rules of Law About Violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-19.html",
+    "id": "A-8.8-s19",
+    "text": "19 In every case where the facts of a violation are reviewed by the Minister or by the Tribunal, the Minister must establish, on a balance of probabilities, that the person named in the notice of violation committed the violation identified in the notice.",
+    "citation": "AAAMPA, s. 19"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "20",
+    "marginal_note": "Vicarious liability — licence holders, etc.",
+    "part": "",
+    "division": "",
+    "heading": "Rules of Law About Violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-20.html",
+    "id": "A-8.8-s20",
+    "text": "20\n(1) The holder of a licence, certificate, letter of accreditation, permit, notice or other document issued under an agri-food Act is liable for a violation that is committed in respect of any matter relating to any activity or requirement under that document, whether or not the person who actually committed the violation is identified or proceeded against in accordance with this Act.\n(2) [Vicarious liability — acts of employees and agents] A person is liable for a violation that is committed by any employee or agent of the person acting in the course of the employee’s employment or the scope of the agent’s authority, whether or not the employee or agent who actually committed the violation is identified or proceeded against in accordance with this Act.",
+    "citation": "AAAMPA, s. 20"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "21",
+    "marginal_note": "Continuing violation",
+    "part": "",
+    "division": "",
+    "heading": "Rules of Law About Violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-21.html",
+    "id": "A-8.8-s21",
+    "text": "21 A violation that is continued on more than one day constitutes a separate violation in respect of each day during which it is continued.",
+    "citation": "AAAMPA, s. 21"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "22",
+    "marginal_note": "Forfeiture",
+    "part": "",
+    "division": "",
+    "heading": "Rules of Law About Violations",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-22.html",
+    "id": "A-8.8-s22",
+    "text": "22 Where\n(a) a person is deemed by this Act to have committed a violation, or\n(b) the Minister, pursuant to a review under this Act, has decided that a person has committed a violation and no request to review the Minister’s decision has been made to the Tribunal in the prescribed time and manner,\nanything seized and detained under an agri-food Act in relation to the violation is, at the election of Her Majesty in right of Canada, immediately forfeited to Her Majesty in right of Canada and may be disposed of, at the expense of the person from whom it was seized, in accordance with the regulations made under the applicable agri-food Act unless the Minister directs otherwise.",
+    "citation": "AAAMPA, s. 22"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "23",
+    "marginal_note": "Notations removed",
+    "part": "",
+    "division": "",
+    "heading": "General Provisions",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-23.html",
+    "id": "A-8.8-s23",
+    "text": "23\n(1) Any notation of a violation shall, on application by the person who committed the violation, be removed from any records that may be kept by the Minister respecting that person after the expiration of five years from\n(a) where the notice of violation contained a warning, the date the notice was served, or\n(b) in any other case, the payment of any debt referred to in subsection 15(1),\nunless the removal from the record would not in the opinion of the Minister be in the public interest or another notation of a violation has been recorded by the Minister in respect of that person after that date and has not been removed in accordance with this subsection.\n(2) [Duty to notify] The Minister shall cause a notice of removal to be served on the person in respect of whom a notation is removed pursuant to subsection (1).",
+    "citation": "AAAMPA, s. 23"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "24",
+    "marginal_note": "Service of documents",
+    "part": "",
+    "division": "",
+    "heading": "General Provisions",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-24.html",
+    "id": "A-8.8-s24",
+    "text": "24 Every document required or authorized to be served under this Act shall be served in accordance with the regulations, either personally or in such other manner as may be authorized in the regulations.",
+    "citation": "AAAMPA, s. 24"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "25",
+    "marginal_note": "Evidence",
+    "part": "",
+    "division": "",
+    "heading": "General Provisions",
+    "history": "",
+    "last_amended": "2002-12-31",
+    "in_force": "2002-12-31",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-25.html",
+    "id": "A-8.8-s25",
+    "text": "25 In any proceeding for a violation or for a prosecution for an offence, a notice of violation purporting to be issued pursuant to this Act is admissible in evidence without proof of the signature or official character of the person appearing to have signed the notice of violation.",
+    "citation": "AAAMPA, s. 25"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "26",
+    "marginal_note": "Limitation period",
+    "part": "",
+    "division": "",
+    "heading": "General Provisions",
+    "history": "1995, c. 40, s. 26; 2015, c. 2, s. 119",
+    "last_amended": "2015-02-27",
+    "in_force": "2015-02-27",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-26.html",
+    "id": "A-8.8-s26",
+    "text": "26 No proceedings in respect of a violation may be commenced later than\n(a) six months after the day on which the subject matter of the proceedings arises, in the case of a minor violation; or\n(b) two years after the day on which the subject matter of the proceedings arises, in the case of a serious violation or a very serious violation.",
+    "citation": "AAAMPA, s. 26"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "27",
+    "marginal_note": "Review Tribunal continued",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, c. 40, s. 27; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-27.html",
+    "id": "A-8.8-s27",
+    "text": "27\n(1) The Review Tribunal, continued by subsection 4.1(1) of the Canada Agricultural Products Act, chapter 20 of the 4th Supplement to the Revised Statutes of Canada, 1985, is continued.\n(2) [Composition] The Tribunal consists of members to be appointed by the Governor in Council, one of whom is to be appointed as Chairperson.",
+    "citation": "AAAMPA, s. 27"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "28",
+    "marginal_note": "Qualifications",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, c. 40, s. 28; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-28.html",
+    "id": "A-8.8-s28",
+    "text": "28 A person is not eligible to be appointed as a member unless the person is knowledgeable about or has experience related to agriculture or agri-food and the Chairperson and at least one other member must, in addition, be a lawyer of at least ten years’ standing at the bar of any province or a notary of at least ten years’ standing at the Chambre des notaires du Québec.",
+    "citation": "AAAMPA, s. 28"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "29",
+    "marginal_note": "Membership",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, c. 40, s. 29; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-29.html",
+    "id": "A-8.8-s29",
+    "text": "29 The Chairperson is to be appointed as a full-time member and the other members are to be appointed as either full-time members or part-time members.",
+    "citation": "AAAMPA, s. 29"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "30",
+    "marginal_note": "Tenure",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, c. 40, s. 30; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-30.html",
+    "id": "A-8.8-s30",
+    "text": "30\n(1) Each member is to be appointed for a term of not more than five years and holds office during good behaviour, but may be removed by the Governor in Council for cause.\n(2) [Re-appointment] Each member may be re-appointed as a member in the same or another capacity.",
+    "citation": "AAAMPA, s. 30"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "31",
+    "marginal_note": "No other federal public administration",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, c. 40, s. 31; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-31.html",
+    "id": "A-8.8-s31",
+    "text": "31 A member must not hold any other office in the federal public administration.",
+    "citation": "AAAMPA, s. 31"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "32",
+    "marginal_note": "Conflict of interest",
+    "part": "",
+    "division": "",
+    "heading": "Composition",
+    "history": "1995, c. 40, s. 32; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-32.html",
+    "id": "A-8.8-s32",
+    "text": "32 A member must not accept or hold any office or employment that is inconsistent with the member’s duties or take part in any matter before the Tribunal in which the member has an interest.",
+    "citation": "AAAMPA, s. 32"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "33",
+    "marginal_note": "Duties of Chairperson",
+    "part": "",
+    "division": "",
+    "heading": "Chairperson",
+    "history": "1995, c. 40, s. 33; 2012, c. 24, s. 102; 2014, c. 20, s. 480",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-33.html",
+    "id": "A-8.8-s33",
+    "text": "33\n(1) The Chairperson apportions work among the Tribunal’s members.\n(2) [Absence or incapacity of Chairperson] If the Chairperson is absent or unable to act or the Chairperson’s position becomes vacant, the members must designate a member with the legal qualifications described in section 28 to act as Chairperson pending the appointment of a replacement, but no person may so act for a period exceeding 60 days without the approval of the Governor in Council.",
+    "citation": "AAAMPA, s. 33"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "34",
+    "marginal_note": "Remuneration",
+    "part": "",
+    "division": "",
+    "heading": "Remuneration and Expenses",
+    "history": "1995, c. 40, s. 34; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-34.html",
+    "id": "A-8.8-s34",
+    "text": "34\n(1) Each full-time member is to be paid the salary that is fixed by the Governor in Council and each part-time member is entitled to be paid the fees or other remuneration that is fixed by the Governor in Council.\n(2) [Travel and living expenses] Members are entitled to be paid, in accordance with Treasury Board directives, reasonable travel and living expenses incurred in the performance of their duties and functions while absent from their ordinary place of work, in the case of full-time members, or from their ordinary place of residence, in the case of part-time members.",
+    "citation": "AAAMPA, s. 34"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "35",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "Remuneration and Expenses",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-35.html",
+    "id": "A-8.8-s35",
+    "text": "35 [Repealed, 2014, c. 20, s. 480]",
+    "citation": "AAAMPA, s. 35"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "36",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "Remuneration and Expenses",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-36.html",
+    "id": "A-8.8-s36",
+    "text": "36 [Repealed, 2014, c. 20, s. 480]",
+    "citation": "AAAMPA, s. 36"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "37",
+    "marginal_note": "Head office",
+    "part": "",
+    "division": "",
+    "heading": "Head Office",
+    "history": "1995, c. 40, s. 37; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-37.html",
+    "id": "A-8.8-s37",
+    "text": "37\n(1) The head office of the Tribunal is to be in the National Capital Region as defined in section 2 of the National Capital Act.\n(2) [Sittings] The Tribunal is to sit at the places in Canada that may be specified by the Governor in Council.",
+    "citation": "AAAMPA, s. 37"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "38",
+    "marginal_note": "Tribunal",
+    "part": "",
+    "division": "",
+    "heading": "Jurisdiction",
+    "history": "1995, c. 40, s. 38; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-38.html",
+    "id": "A-8.8-s38",
+    "text": "38\n(1) The Tribunal has sole and exclusive jurisdiction to hear and determine all questions of fact or law in relation to any matter over which it is given jurisdiction under this Act or any other Act of Parliament.\n(2) [Review by Federal Court] An order of the Tribunal may only be reviewed under the Federal Courts Act.",
+    "citation": "AAAMPA, s. 38"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "39",
+    "marginal_note": "Chairperson",
+    "part": "",
+    "division": "",
+    "heading": "Jurisdiction",
+    "history": "1995, c. 40, s. 39; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-39.html",
+    "id": "A-8.8-s39",
+    "text": "39\n(1) The jurisdiction of the Tribunal in relation to the following matters is to be exercised by the Chairperson:\n(a) requests under subsection 8(1) or 12(2) for a review in respect of a notice of violation that contains a warning; and\n(b) requests under paragraph 9(2)(c) or 13(2)(b) for a review in respect of a notice of violation that sets out a penalty of less than $2,000.\n(2) [Other legally qualified members] The jurisdiction of the Tribunal in relation to a matter referred to in subsection (1) may be exercised, if the Chairperson so directs, by any member of the Tribunal with the legal qualifications described in section 28.",
+    "citation": "AAAMPA, s. 39"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "40",
+    "marginal_note": "Reviews",
+    "part": "",
+    "division": "",
+    "heading": "Jurisdiction",
+    "history": "1995, c. 40, s. 40; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-40.html",
+    "id": "A-8.8-s40",
+    "text": "40 Reviews by the Tribunal are to be heard by a single member.",
+    "citation": "AAAMPA, s. 40"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "41",
+    "marginal_note": "Court of record",
+    "part": "",
+    "division": "",
+    "heading": "Powers",
+    "history": "1995, c. 40, s. 41; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-41.html",
+    "id": "A-8.8-s41",
+    "text": "41\n(1) The Tribunal is a court of record with an official seal that must be judicially noticed.\n(2) [Examination of witnesses, etc.] In addition to the powers conferred by subsection (1), the Tribunal has, with respect to the appearance, swearing and examination of witnesses, the production and inspection of documents and other things, the enforcement of its orders and other matters necessary or proper for the due exercise of its jurisdiction, all the powers, rights and privileges that are vested in a superior court of record and, without limiting the generality of the foregoing, it may\n(a) issue a summons requiring a person\n(i) to appear at the time and place stated in the summons to testify to all matters within the person’s knowledge relative to any subject matter before the Tribunal, and\n(ii) to bring and produce any document, book or paper in the person’s possession or under the person’s control relative to that subject matter;\n(b) administer oaths and examine any person on oath; and\n(c) during a hearing, receive any evidence that it considers relevant and trustworthy.",
+    "citation": "AAAMPA, s. 41"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "42",
+    "marginal_note": "Rules",
+    "part": "",
+    "division": "",
+    "heading": "Rules",
+    "history": "1995, c. 40, s. 42; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-42.html",
+    "id": "A-8.8-s42",
+    "text": "42 The Tribunal may, with the approval of the Governor in Council, make rules governing\n(a) the practice and procedure in respect of hearings;\n(b) the time and manner in which applications and notices must be made or given; and\n(c) the work of the Tribunal under this or any other Act of Parliament.",
+    "citation": "AAAMPA, s. 42"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "43",
+    "marginal_note": "Consultations",
+    "part": "",
+    "division": "",
+    "heading": "General",
+    "history": "1995, c. 40, s. 43; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-43.html",
+    "id": "A-8.8-s43",
+    "text": "43 The members of the Tribunal may consult with other members of the Tribunal in respect of any matter before it.",
+    "citation": "AAAMPA, s. 43"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "44",
+    "marginal_note": "Rules of evidence",
+    "part": "",
+    "division": "",
+    "heading": "General",
+    "history": "1995, c. 40, s. 44; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-44.html",
+    "id": "A-8.8-s44",
+    "text": "44 The Tribunal is not bound by any legal or technical rules of evidence in conducting any matter that comes before it. It must deal with matters that come before it as informally and expeditiously as the circumstances and considerations of fairness and natural justice permit.",
+    "citation": "AAAMPA, s. 44"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "45",
+    "marginal_note": "Privileged evidence not receivable",
+    "part": "",
+    "division": "",
+    "heading": "General",
+    "history": "1995, c. 40, s. 45; 2012, c. 24, s. 102",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-45.html",
+    "id": "A-8.8-s45",
+    "text": "45 The Tribunal is not entitled to receive or accept as evidence anything that would be inadmissible in a court by reason of any privilege under the law of evidence.",
+    "citation": "AAAMPA, s. 45"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "46 to 89",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "General",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "in force",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-46 to 89.html",
+    "id": "A-8.8-s46 to 89",
+    "text": "46 to 89 [Repealed, 2012, c. 24, s. 102]",
+    "citation": "AAAMPA, s. 46 to 89"
+  },
+  {
+    "act_code": "A-8.8",
+    "act_short": "AAAMPA",
+    "act_name": "Agriculture and Agri-Food Administrative Monetary Penalties Act",
+    "section": "90",
+    "marginal_note": "",
+    "part": "",
+    "division": "",
+    "heading": "General",
+    "history": "",
+    "last_amended": "2019-01-15",
+    "in_force": "2019-01-15",
+    "status": "repealed",
+    "current_to": "2022-03-22",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/a-8.8/section-90.html",
+    "id": "A-8.8-s90",
+    "text": "90 [Repealed, 2012, c. 24, s. 102]",
+    "citation": "AAAMPA, s. 90"
+  }
+]

data/processed/C-1.4.json

@@ -0,0 +1,743 @@
+[
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "1",
+    "marginal_note": "Short title",
+    "part": "",
+    "division": "",
+    "heading": "Short Title",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-1.html",
+    "id": "C-1.4-s1",
+    "text": "1 This Act may be cited as the Canada Border Services Agency Act.",
+    "citation": "CBSA Act, s. 1"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "2",
+    "marginal_note": "Definitions",
+    "part": "",
+    "division": "",
+    "heading": "Interpretation",
+    "history": "2005, c. 38, ss. 2, 145; 2012, c. 24, s. 107; 2022, c. 10, s. 167",
+    "last_amended": "2022-09-01",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-2.html",
+    "id": "C-1.4-s2",
+    "text": "2 The following definitions apply in this Act.\nAgency means the Canada Border Services Agency established under subsection 3(1). (Agence)\nMinister means the Minister of Public Safety and Emergency Preparedness. (ministre)\nPresident means the President of the Agency appointed under subsection 7(1). (président)\nprogram legislation means any other Act of Parliament or any instrument made under it, or any part of such an Act or instrument,\n(a) that the Governor in Council or Parliament authorizes the Minister, the Agency, the President or an employee of the Agency to administer and enforce, including the Excise Act, the Special Import Measures Act, the Customs Act, the Customs Tariff, the Immigration and Refugee Protection Act, the Excise Act, 2001 and the Select Luxury Items Tax Act;\n(b) that the Governor in Council or Parliament authorizes the Minister, the Agency, the President or an employee of the Agency to enforce, including the Agriculture and Agri-Food Administrative Monetary Penalties Act, the Feeds Act, the Fertilizers Act, the Health of Animals Act, the Plant Protection Act, the Safe Food for Canadians Act and the Seeds Act;\n(c) under which the Minister or another minister authorizes the Agency, the President or an employee of the Agency to administer a program or carry out an activity; or\n(d) under which duties or taxes collected and paid pursuant to the Customs Act are imposed. (législation frontalière)",
+    "citation": "CBSA Act, s. 2"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "3",
+    "marginal_note": "Agency established",
+    "part": "",
+    "division": "",
+    "heading": "Establishment and Mandate of the Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-3.html",
+    "id": "C-1.4-s3",
+    "text": "3\n(1) The Canada Border Services Agency is established as a body corporate.\n(2) [Agent of Her Majesty] The Agency is for all purposes an agent of Her Majesty in right of Canada.",
+    "citation": "CBSA Act, s. 3"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "4",
+    "marginal_note": "Head office",
+    "part": "",
+    "division": "",
+    "heading": "Establishment and Mandate of the Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-4.html",
+    "id": "C-1.4-s4",
+    "text": "4 The head office of the Agency is to be in the National Capital Region described in the schedule to the National Capital Act.",
+    "citation": "CBSA Act, s. 4"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "5",
+    "marginal_note": "Mandate of Agency",
+    "part": "",
+    "division": "",
+    "heading": "Establishment and Mandate of the Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-5.html",
+    "id": "C-1.4-s5",
+    "text": "5\n(1) The Agency is responsible for providing integrated border services that support national security and public safety priorities and facilitate the free flow of persons and goods, including animals and plants, that meet all requirements under the program legislation, by\n(a) supporting the administration or enforcement, or both, as the case may be, of the program legislation;\n(b) implementing agreements between the Government of Canada or the Agency and a foreign state or a public body performing a function of government in a foreign state to carry out an activity, provide a service or administer a tax or program;\n(c) implementing agreements between the Government of Canada or the Agency and the government of a province or other public body performing a function of the Government in Canada to carry out an activity, provide a service or administer a tax or program;\n(d) implementing agreements or arrangements between the Agency and departments or agencies of the Government of Canada to carry out an activity, provide a service or administer a program; and\n(e) providing cooperation and support, including advice and information, to other departments and agencies of the Government of Canada to assist them in developing, evaluating and implementing policies and decisions in relation to program legislation for which they have responsibility.\n(2) [Support] The Agency may provide support, through the provision of services, to departments and agencies for which the Minister is responsible, in accordance with agreements or arrangements entered into with those departments and agencies.",
+    "citation": "CBSA Act, s. 5"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "6",
+    "marginal_note": "Minister responsible",
+    "part": "",
+    "division": "",
+    "heading": "Minister",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-6.html",
+    "id": "C-1.4-s6",
+    "text": "6\n(1) The Minister is responsible for the Agency.\n(2) [Delegation by Minister] The Minister may delegate to any person any power, duty or function conferred on the Minister under this Act or under the program legislation.\n(3) [Exception] Subsection (2) does not apply if an Act of Parliament other than this Act authorizes the Minister to delegate the power, duty or function to any person or authorizes any person to exercise or perform it.\n(4) [Limitation] Subsection (2) does not apply in respect of a power to make regulations.",
+    "citation": "CBSA Act, s. 6"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "7",
+    "marginal_note": "Appointment",
+    "part": "",
+    "division": "",
+    "heading": "President and Executive Vice-president",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-7.html",
+    "id": "C-1.4-s7",
+    "text": "7\n(1) The Governor in Council shall appoint a President of the Agency to hold office during pleasure for a term of not more than five years, which term may be renewed for one or more further terms.\n(2) [Executive Vice-president] The Governor in Council may appoint an Executive Vice-president of the Agency to hold office during pleasure for a term of not more than five years, which term may be renewed for one or more further terms.",
+    "citation": "CBSA Act, s. 7"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "8",
+    "marginal_note": "Role of President",
+    "part": "",
+    "division": "",
+    "heading": "President and Executive Vice-president",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-8.html",
+    "id": "C-1.4-s8",
+    "text": "8\n(1) The President, under the direction of the Minister, has the control and management of the Agency and all matters connected with it.\n(2) [Rank of deputy head] The President has the rank and all the powers of a deputy head of a department.\n(3) [Executive Vice-president’s powers] The Executive Vice-president shall exercise the powers and perform the duties and functions that the President may assign and shall act as President if that office is vacant or if the President is absent or incapacitated.",
+    "citation": "CBSA Act, s. 8"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "9",
+    "marginal_note": "Delegation by President",
+    "part": "",
+    "division": "",
+    "heading": "President and Executive Vice-president",
+    "history": "2005, c. 38, s. 9; 2012, c. 24, s. 108",
+    "last_amended": "2019-01-15",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-9.html",
+    "id": "C-1.4-s9",
+    "text": "9\n(1) The President may delegate to any person any power, duty or function that the President is authorized to exercise or perform under this Act or any other enactment.\n(2) [Designation of officers] The President may designate any person, or person within a class of persons,\n(a) as an officer as defined in subsection 2(1) of the Customs Act to exercise any powers or perform any duties and functions of an officer under that Act that the President may specify; or\n(b) as an inspector or a veterinary inspector or other officer for the enforcement of any Act or instrument made under it, or any part of an Act or instrument, that the Governor in Council or Parliament authorizes the Minister, the Agency, the President or an employee of the Agency to enforce, including the Agriculture and Agri-Food Administrative Monetary Penalties Act, the Feeds Act, the Fertilizers Act, the Health of Animals Act, the Plant Protection Act, the Safe Food for Canadians Act and the Seeds Act.\n(3) [Designation power] The President may exercise any power that the Minister has to designate officers under subsection 6(1) of the Immigration and Refugee Protection Act.",
+    "citation": "CBSA Act, s. 9"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "10",
+    "marginal_note": "Remuneration",
+    "part": "",
+    "division": "",
+    "heading": "President and Executive Vice-president",
+    "history": "2005, c. 38, ss. 10, 144(E)",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-10.html",
+    "id": "C-1.4-s10",
+    "text": "10\n(1) The President and the Executive Vice-president shall be paid the remuneration that is fixed by the Governor in Council.\n(2) [Expenses] The President and the Executive Vice-president are entitled to be paid reasonable travel and living expenses incurred by them in the course of performing their duties while absent from their ordinary place of work.\n(3) [Deemed employment] The President and the Executive Vice-president are deemed to be employed in the public service for the purposes of the Public Service Superannuation Act and to be employed in the federal public administration for the purposes of the Government Employees Compensation Act and any regulations made under section 9 of the Aeronautics Act.",
+    "citation": "CBSA Act, s. 10"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "11",
+    "marginal_note": "Officers and employees",
+    "part": "",
+    "division": "",
+    "heading": "Human Resources",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-11.html",
+    "id": "C-1.4-s11",
+    "text": "11 Officers and employees necessary for the proper conduct of the work of the Agency shall be appointed in accordance with the Public Service Employment Act.",
+    "citation": "CBSA Act, s. 11"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "12",
+    "marginal_note": "Exercise of powers conferred on Minister",
+    "part": "",
+    "division": "",
+    "heading": "Powers of the Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-12.html",
+    "id": "C-1.4-s12",
+    "text": "12\n(1) Subject to any direction given by the Minister, the Agency may exercise the powers, and shall perform the duties and functions, that relate to the program legislation and that are conferred on, or delegated, assigned or transferred to, the Minister under any Act or regulation.\n(2) [Officers and employees] An officer or employee of the Agency may exercise any power or perform any duty or function referred to in subsection (1) if the officer or employee is appointed to serve in the Agency in a capacity appropriate to the exercise of the power or the performance of the duty or function, and, in so doing, shall comply with any general or special direction given by the Minister.\n(3) [Exception] Subsection (1) does not include\n(a) any power, duty or function of the Minister under this Act; or\n(b) a power to make regulations.\n(4) [Non-application of Statutory Instruments Act] A direction given by the Minister under subsection (1) or (2) is not a statutory instrument for the purposes of the Statutory Instruments Act.",
+    "citation": "CBSA Act, s. 12"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "13",
+    "marginal_note": "Agreements",
+    "part": "",
+    "division": "",
+    "heading": "Powers of the Agency",
+    "history": "2005, c. 38, s. 13; 2006, c. 12, s. 46",
+    "last_amended": "2007-02-10",
+    "in_force": "2007-02-10",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-13.html",
+    "id": "C-1.4-s13",
+    "text": "13\n(1) Subject to sections 38 and 38.1 of the Proceeds of Crime (Money Laundering) and Terrorist Financing Act, the Agency may, with the approval of the Governor in Council, on the recommendation of the Minister and the Minister of Foreign Affairs, enter into an agreement with a foreign state or an international organization, for the purposes of carrying out the mandate of the Agency.\n(2) [Arrangements and agreements] The Agency may, for the purposes of carrying out its mandate,\n(a) enter into an arrangement with a foreign state or an international organization; or\n(b) enter into an agreement or arrangement with the government of a province, a department or agency of the Government of Canada or any person or organization.",
+    "citation": "CBSA Act, s. 13"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "14",
+    "marginal_note": "Agreements to administer a tax",
+    "part": "",
+    "division": "",
+    "heading": "Powers of the Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-14.html",
+    "id": "C-1.4-s14",
+    "text": "14\n(1) The Agency may enter into or amend an agreement with a provincial or territorial government to administer a tax or other fiscal measure if the agreement is in accordance with guidelines relating to agreements of that kind established jointly by the Minister and the Minister of Finance.\n(2) [Application of the Federal-Provincial Fiscal Arrangements Act] Parts III and III.1 of the Federal-Provincial Fiscal Arrangements Act do not apply to an agreement entered into or amended under subsection (1).",
+    "citation": "CBSA Act, s. 14"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "15",
+    "marginal_note": "Appropriation Acts",
+    "part": "",
+    "division": "",
+    "heading": "Expenditures",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-15.html",
+    "id": "C-1.4-s15",
+    "text": "15 An appropriation Act may provide that the balance of money appropriated by Parliament for the use of the Agency that remains unexpended at the end of the fiscal year, after the adjustments referred to in section 37 of the Financial Administration Act are made, lapses at the end of the following fiscal year.",
+    "citation": "CBSA Act, s. 15"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "15.1",
+    "marginal_note": "Annual report",
+    "part": "",
+    "division": "",
+    "heading": "Annual Report",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-15.1.html",
+    "id": "C-1.4-s15.1",
+    "text": "15.1\n(1) The Minister shall, as soon as possible after the end of each fiscal year but no later than the end of the calendar year in which that fiscal year ends, cause to be laid before each House of Parliament a report of the operations and performance of the Agency for that fiscal year.\n(2) [Reports required by Treasury Board] The obligation imposed by subsection (1) may be satisfied by the tabling of any reports of the operations and performance of the Agency required by the Treasury Board that contain the information required by that subsection.",
+    "citation": "CBSA Act, s. 15.1"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "16",
+    "marginal_note": "Definitions",
+    "part": "",
+    "division": "",
+    "heading": "Definitions",
+    "history": "2005, c. 38, ss. 16, 144(E)",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-16.html",
+    "id": "C-1.4-s16",
+    "text": "16 The following definitions apply in sections 17 to 19 and 21 to 28.\nformer agency means the portion of the federal public administration known as the Canada Border Services Agency. (ancienne agence)\nnew agency means the Canada Border Services Agency established under subsection 3(1). (nouvelle agence)\norder P.C. 2003-2064 means Order in Council P.C. 2003-2064 of December 12, 2003, registered as SI/2003-216. (décret C.P. 2003-2064)",
+    "citation": "CBSA Act, s. 16"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "17",
+    "marginal_note": "President and Executive Vice-president",
+    "part": "",
+    "division": "",
+    "heading": "Former Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-17.html",
+    "id": "C-1.4-s17",
+    "text": "17\n(1) The persons occupying the positions of President and Executive Vice-president of the former agency on the day on which this section comes into force become the President and Executive Vice-president of the new agency on that day and are deemed to have been appointed under section 7.\n(2) [Positions] Nothing in this Act is to be construed as affecting the status of an employee who, immediately before the coming into force of this section, occupied a position in the former agency, except that the employee shall, on the coming into force of this section, occupy his or her position in the new agency under the direction of the President.\n(3) [Definition of employee] In subsection (2), employee has the same meaning as in subsection 2(1) of the Public Service Employment Act.",
+    "citation": "CBSA Act, s. 17"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "18",
+    "marginal_note": "Transfer of appropriations",
+    "part": "",
+    "division": "",
+    "heading": "Former Agency",
+    "history": "2005, c. 38, ss. 18, 144(E)",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-18.html",
+    "id": "C-1.4-s18-p1",
+    "text": "18\n(1) Any amount appropriated, for the fiscal year in which this section comes into force, by an appropriation Act based on the Estimates for that year for defraying the charges and expenses of the federal public administration for the former agency that, on the day on which this section comes into force, is unexpended is deemed, on that day, to be an amount appropriated for defraying the charges and expenses of the federal public administration for the new agency.\n(2) [Transfer of powers, duties and functions] Wherever under any Act, order, rule or regulation, or any contract, lease, licence or other document, any power, duty or function is vested in or exercisable by the President of the former agency or an employee of the former agency, the power, duty or function is vested in and shall be exercised by the President of the new agency or an employee of the new agency unless the Governor in Council by order designates a deputy minister or an officer of the federal public administration to exercise that power or perform that duty or function.\n(3) [Continuation of proceedings] Any action, suit or other legal or administrative proceeding to which the former agency or its President is a party that is pending on the coming into force of this section may be continued by or against the new agency or its President in a similar manner and to the same extent as it would have been continued by or against the former agency or its President.\n(4) [Deeming] Decisions made by the President of the former agency are deemed to be decisions made by the President of the new agency.",
+    "citation": "CBSA Act, s. 18 (part 1 of 2)"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "18",
+    "marginal_note": "Transfer of appropriations",
+    "part": "",
+    "division": "",
+    "heading": "Former Agency",
+    "history": "2005, c. 38, ss. 18, 144(E)",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-18.html",
+    "id": "C-1.4-s18-p2",
+    "text": "(5) [Validity of documents] All orders, rules, regulations, decisions, determinations and re-determinations, directions, licences, authorizations, certificates, consents, approvals, declarations, designations, permits, registrations, rates or other documents that are in force on the coming into force of this section and that are made or issued by the President of the former agency or any person under his or her authority continue in force as if they were made or issued by the President of the new agency or a person under his or her authority, as the case may be, until they expire or are repealed, replaced, rescinded or altered.\n(6) [Continuation of evidentiary presumption] Every affidavit sworn, or document purporting to be certified, by an employee of the former agency before the day on which this section comes into force has the same probative value as if it were sworn or certified by an employee of the new agency after that day.",
+    "citation": "CBSA Act, s. 18 (part 2 of 2)"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "19",
+    "marginal_note": "References",
+    "part": "",
+    "division": "",
+    "heading": "Former Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-19.html",
+    "id": "C-1.4-s19",
+    "text": "19\n(1) A reference to the former agency in any of the following is deemed to be a reference to the new agency:\n(a) Schedule I to the Access to Information Act under the heading “Other Government Institutions”;\n(b) the schedule to the Privacy Act;\n(c) Part I of Schedule I to the Public Service Staff Relations Act;\n(d) any order of the Governor in Council made under paragraph (b) of the definition head in section 3 of the Access to Information Act;\n(e) any direction of the Governor in Council made under subsection 24(3) of the Auditor General Act;\n(f) any order of the Governor in Council made under paragraph 29(e) of the Canadian Security Intelligence Service Act;\n(g) any order of the Governor in Council made under paragraph (b) of the definition head in section 3 of the Privacy Act; and\n(h) any order of the Governor in Council made under the definition department in subsection 2(1) of the Public Service Employment Act.\n(2) [Deputy head] The designation of a person as deputy head of the former agency in any of the following is deemed to be a designation of the President of the new agency as deputy head of that agency:\n(a) any order of the Governor in Council made under paragraph 29(e) of the Canadian Security Intelligence Service Act; and\n(b) any order of the Governor in Council made under the definition deputy head in subsection 2(1) of the Public Service Employment Act.",
+    "citation": "CBSA Act, s. 19"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "20",
+    "marginal_note": "Positions",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-20.html",
+    "id": "C-1.4-s20",
+    "text": "20 Nothing in this Act shall be construed as affecting the status of an employee who, immediately before the coming into force of this section, occupied a position in the Canada Customs and Revenue Agency, except that the employee shall occupy that position in the Canada Revenue Agency.",
+    "citation": "CBSA Act, s. 20"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "21",
+    "marginal_note": "Continuation of rights and property",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-21.html",
+    "id": "C-1.4-s21",
+    "text": "21\n(1) Subject to subsection (2), all rights and property of the Canada Customs and Revenue Agency continue as the rights and property of the Canada Revenue Agency.\n(2) [Transfer to new agency] All rights and property of the Canada Customs and Revenue Agency that are in respect of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064 are transferred to the new agency.",
+    "citation": "CBSA Act, s. 21"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "22",
+    "marginal_note": "Continuation of obligations and liabilities",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-22.html",
+    "id": "C-1.4-s22",
+    "text": "22\n(1) Subject to subsection (2), all obligations and liabilities of the Canada Customs and Revenue Agency continue as obligations and liabilities of the Canada Revenue Agency.\n(2) [Transfer to new agency] All obligations and liabilities of the Canada Customs and Revenue Agency that were incurred in respect of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064 are transferred to the new agency.",
+    "citation": "CBSA Act, s. 22"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "23",
+    "marginal_note": "Real property, immovables and licences",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-23.html",
+    "id": "C-1.4-s23",
+    "text": "23\n(1) The administration of any real property or immovable, and the administrative responsibility for any licence in respect of any real property or immovable, that was under the administration or administrative responsibility of the Canada Customs and Revenue Agency immediately before the coming into force of this section and that was used for or in support of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064 are transferred to the Minister.\n(2) [List] As soon as practicable after the coming into force of this section, the Minister of National Revenue shall publish in the Canada Gazette a list of the real property and immovables the administration of which was transferred under subsection (1) in such a way that each is sufficiently identified.\n(3) [Title] Where the title of any real property or immovable was held in the name of the Canada Customs and Revenue Agency immediately before the coming into force of this section and that real property or immovable was used for or in support of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064, the title to that real property or immovable is deemed to be held in the name of Her Majesty in right of Canada.\n(4) [Other real property, immovables and licences — Canada Revenue Agency] The administration of any real property or immovable, and the administrative responsibility for any licence in respect of any real property or immovable, that is not referred to in subsection (1) and that was under the administration or administrative responsibility of the Canada Customs and Revenue Agency immediately before the coming into force of this section continues under the administration or administrative responsibility, as the case may be, of the Canada Revenue Agency.",
+    "citation": "CBSA Act, s. 23"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "24",
+    "marginal_note": "Continuation of legal proceedings: Canada Revenue Agency",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-24.html",
+    "id": "C-1.4-s24",
+    "text": "24\n(1) Subject to subsection (2), any action, suit or other legal or administrative proceeding to which the Canada Customs and Revenue Agency is a party that is pending on the coming into force of this section may be continued by or against the Canada Revenue Agency in the same manner and to the same extent as it could have been continued by or against the Canada Customs and Revenue Agency.\n(2) [Continuation of legal proceedings: new agency] Any action, suit or other legal or administrative proceeding to which the Canada Customs and Revenue Agency is a party that is pending on the coming into force of this section may be continued by or against the new agency in the same manner and to the same extent as it could have been continued by or against the Canada Customs and Revenue Agency in respect of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064.",
+    "citation": "CBSA Act, s. 24"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "25",
+    "marginal_note": "Validity of documents: Canada Revenue Agency",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-25.html",
+    "id": "C-1.4-s25",
+    "text": "25\n(1) Subject to subsection (2), all orders, rules, regulations, decisions, determinations and re-determinations, directions, licences, authorizations, certificates, consents, approvals, declarations, designations, permits, registrations, rates or other documents that are in force on the coming into force of this section and that were made or issued by the Minister of National Revenue or by the Commissioner of Customs and Revenue or any person under their authority continue in force as if they were made or issued by the Minister of National Revenue or the Commissioner of Revenue or any person under their authority, as the case may be, until they expire or are repealed, replaced, rescinded or altered.\n(2) [Validity of documents] All orders, rules, regulations, decisions, determinations and re-determinations, directions, licences, authorizations, certificates, consents, approvals, declarations, designations, permits, registrations, rates or other documents that are in force on the coming into force of this section and that were made or issued by the Minister of National Revenue or by the Commissioner of Customs and Revenue or any person under their authority that are in respect of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064 continue in force as if they were made or issued by the Minister, the President of the new agency or a person under their authority, as the case may be, until they expire or are repealed, replaced, rescinded or altered.",
+    "citation": "CBSA Act, s. 25"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "26",
+    "marginal_note": "Continuation of evidentiary presumption: Canada Revenue Agency",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-26.html",
+    "id": "C-1.4-s26",
+    "text": "26\n(1) Subject to subsection (2), every affidavit sworn, or document purporting to be certified, by an employee of the Canada Customs and Revenue Agency before the day on which this section comes into force has the same probative value as if it were sworn or certified by an employee of the Canada Revenue Agency after that day.\n(2) [Continuation of evidentiary presumption: Canada Border Services Agency] Every affidavit sworn, or document purporting to be certified, by an employee of the Canada Customs and Revenue Agency before the day on which this section comes into force that was sworn or was purported to be certified in respect of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064 has the same probative value as if it were sworn or certified by an employee of the new agency after that day.",
+    "citation": "CBSA Act, s. 26"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "27",
+    "marginal_note": "References",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-27.html",
+    "id": "C-1.4-s27",
+    "text": "27\n(1) Subject to subsection (2), every reference to the Canada Customs and Revenue Agency, the Commissioner of Customs and Revenue, the Deputy Commissioner of Customs and Revenue or any person under their authority in a document issued in the name of the Canada Customs and Revenue Agency, the Commissioner of Customs and Revenue or the Deputy Commissioner of Customs and Revenue is to be read, unless the context otherwise requires, as a reference to the Canada Revenue Agency, the Commissioner of Revenue, the Deputy Commissioner of Revenue or a person under their authority, as the case may be.\n(2) [References] Every reference to the Canada Customs and Revenue Agency, the Commissioner of Customs and Revenue, the Deputy Commissioner of Customs and Revenue or any person under their authority in a document issued in the name of the Canada Customs and Revenue Agency, the Commissioner of Customs and Revenue or the Deputy Commissioner of Customs and Revenue is to be read in respect of those documents that relate to those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064, unless the context otherwise requires, as a reference to the new agency, the President of the new agency, the Executive Vice-president of the new agency or a person under their authority, as the case may be.",
+    "citation": "CBSA Act, s. 27"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "28",
+    "marginal_note": "References in documents and other provisions: Canada Revenue Agency",
+    "part": "",
+    "division": "",
+    "heading": "Canada Customs and Revenue Agency",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-28.html",
+    "id": "C-1.4-s28",
+    "text": "28\n(1) Subject to subsection (2), any expression referring to the Deputy Minister of National Revenue, the Department of National Revenue, the Commissioner of Customs and Revenue or the Canada Customs and Revenue Agency in any document, any instrument made under an Act of Parliament or any provision of an Act of Parliament not amended by this Act is to be read, unless the context otherwise requires, as a reference to the Commissioner of Revenue or the Canada Revenue Agency, as the case may be.\n(2) [References in documents and other provisions: Canada Border Services Agency] Any expression referring to the Deputy Minister of National Revenue, the Department of National Revenue, the Commissioner of Customs and Revenue or the Canada Customs and Revenue Agency in any document, any instrument made under an Act of Parliament or any provision of an Act of Parliament not amended by this Act and the document, instrument or provision is in respect of those portions of the Canada Customs and Revenue Agency the control and supervision of which were transferred to the former agency by order P.C. 2003-2064 is to be read, unless the context otherwise requires, as a reference to the President of the new agency or the new agency, as the case may be.",
+    "citation": "CBSA Act, s. 28"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "29",
+    "marginal_note": "Royal Canadian Mint Act",
+    "part": "",
+    "division": "",
+    "heading": "Royal Canadian Mint Act",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-29.html",
+    "id": "C-1.4-s29",
+    "text": "29 The Minister of National Revenue is the Minister for the purposes of the Royal Canadian Mint Act until another member of the Queen’s Privy Council for Canada is designated under section 2.1 of that Act, as enacted by section 130 of this Act.",
+    "citation": "CBSA Act, s. 29"
+  },
+  {
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "*147",
+    "marginal_note": "Order in council",
+    "part": "",
+    "division": "",
+    "heading": "Coming into Force",
+    "history": "",
+    "last_amended": "2005-12-12",
+    "in_force": "2005-12-12",
+    "status": "in force",
+    "current_to": "2024-11-11",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/section-*147.html",
+    "id": "C-1.4-s*147",
+    "text": "*147 This Act, except for sections 144 to 146, comes into force on a day to be fixed by order of the Governor in Council.\n* [Note: Sections 144 to 146 in force on assent November 3, 2005; Act, except sections 144 to 146, in force December 12, 2005, see SI/2005-119.]",
+    "citation": "CBSA Act, s. *147"
+  },
+  {
+    "id": "C-1.4-nif-1",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 108",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "108 Section 2 of the Canada Border Services Agency Act is amended by adding the following in alphabetical order:\nCommission means the Public Complaints and Review Commission established by subsection 3(1) of the Public Complaints and Review Commission Act. (Commission)",
+    "history": "2024, c. 25, s. 108",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 108)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  },
+  {
+    "id": "C-1.4-nif-2",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 109",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "109 Subsection 9(1) of the Act is replaced by the following:\n9 [Delegation by President]\n(1) Subject to subsection (1.1), the President may delegate to any person any power, duty or function that the President is authorized to exercise or perform under this Act or any other enactment.\n(1.1) [Exception] The President shall not delegate the power to make rules under section 39 of the Public Complaints and Review Commission Act and the power to enter into a memorandum of understanding under subsection 17(7) of that Act.",
+    "history": "2024, c. 25, s. 109",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 109)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  },
+  {
+    "id": "C-1.4-nif-3",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 110",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "110 Section 13 of the Act is amended by adding the following after subsection (2):\n(3) [Restriction] The Agency may enter into an agreement or arrangement with the government of a province respecting the detention of persons on behalf of the Agency only if the Minister is of the opinion that there is in the province an independent individual or body that is empowered to receive and deal with complaints about the treatment and conditions of detention of detained persons.\n(4) [Exception] If the Minister is of the opinion that there is an urgent need to provide for the detention of persons on a temporary basis, the Agency may, with the Minister’s approval, enter into an agreement or arrangement referred to in subsection (3) with the government of a province even if the Minister is of the opinion that there is no independent individual or body referred to in that subsection in the province.\n(5) [Duty to provide information] If the Agency is notified that a person who is or has been detained on behalf of the Agency under an agreement or arrangement referred to in subsection (3) has made a complaint respecting their treatment during detention — or their conditions of detention — to any competent authority in the province where they are or were detained, the Agency shall, as soon as feasible, provide the Commission with all of the information that the Agency has received or receives in relation to the complaint.\n(6) [Duty to provide copy] If the Agency enters into an agreement or arrangement respecting the monitoring of the treatment and of the conditions of detention of persons detained by or on behalf of the Agency or the inspection of any facility where persons are detained by or on behalf of the Agency, the Agency shall, as soon as feasible, provide the Commission with a copy of every document containing findings or recommendations that it receives under that agreement or arrangement.",
+    "history": "2024, c. 25, s. 110",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 110)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  },
+  {
+    "id": "C-1.4-nif-4-p1",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 111",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "111 The Act is amended by adding the following after section 14:\n14.1 [Definitions]\n(1) The following definitions apply in this section and sections 14.2 to 14.9.\nemployee, in relation to the Agency, includes any person who assists the Agency in the exercise of any of its powers or the performance of any of its duties and functions under this Act. (employé)\nserious incident means an incident in which the actions of an officer or employee of the Agency in the exercise of its powers or the performance of its duties and functions under this Act\n(a) may have resulted in serious injury to or the death of any person; or\n(b) may have constituted an offence under federal or provincial law that the Minister or the President decides would be in the public interest to be investigated by the Agency. (incident grave)\nserious injury means a prescribed physical or psychological injury. (blessure grave)\n(2) [Regulations] The Governor in Council may make regulations prescribing physical or psychological injuries for the purposes of the definition serious injury in subsection (1).\n14.2 [Notification] As soon as feasible after becoming aware that a serious incident is alleged to have occurred, the President shall notify the police force that has jurisdiction at the location where it is alleged to have occurred and the Commission of the alleged serious incident.\n14.3 [Investigation]\n(1) Subject to subsection (2), as soon as feasible after the President becomes aware that a serious incident is alleged to have occurred, the Agency shall investigate the alleged serious incident.",
+    "history": "2024, c. 25, s. 111",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 111) (part 1 of 4)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  },
+  {
+    "id": "C-1.4-nif-4-p2",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 111",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "(2) [Limit on investigation] If the alleged serious incident involves the actions of one or more persons, the Agency shall not, in the investigation, investigate the actions of any person who assists the Agency in the exercise of any of its powers or the performance of any of its duties and functions under this Act by reason only of an agreement or arrangement referred to in subsection 13(3).\n14.4 [Copy of records] As soon as feasible after completing the investigation, the President shall provide the Commission with a copy of any records in its possession relating to the investigation. The President shall also provide the Commission with a copy of any record in respect of every other lawfully authorized investigation conducted into the alleged serious incident that the Agency receives after the completion of its own investigation.\n14.5 [Observer]\n(1) The Agency shall permit an observer appointed by the Commission under subsection (2) to assess the impartiality of the investigation.\n(2) [Appointment of observer] The Commission may appoint an observer to assess the impartiality of the investigation.\n(3) [Observers are compellable] Despite subsection 11(3) of the Public Complaints and Review Commission Act, but subject to section 14.8(1), every observer is a compellable witness in every criminal, civil or administrative action or proceeding, or inquiry, in respect of any matter coming to the knowledge of the observer as a result of exercising a power or performing a duty or function under this Part.",
+    "history": "2024, c. 25, s. 111",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 111) (part 2 of 4)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  },
+  {
+    "id": "C-1.4-nif-4-p3",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 111",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "(4) [No observer appointed] If no observer is appointed to an investigation of a serious incident under subsection (2), the President shall provide the Chairperson of the Commission with a report that sets out all measures that have been or will be taken by the Agency to ensure the impartiality of the investigation.\n14.6 [Recommendations] If the observer has concerns with the impartiality of the investigation, the observer may inform the Agency of their concerns and may make any recommendations to the Agency that they consider appropriate to address those concerns.\n14.7 [Report]\n(1) The observer shall, in accordance with the regulations, provide a report respecting the impartiality of the investigation of a serious incident to the Chairperson of the Commission and the President.\n(2) [Response] If the observer’s report identifies concerns with respect to the impartiality of an investigation, the President shall provide to the Chairperson of the Commission a written response to the observer’s report that includes a description of what actions have or will be taken by the Agency to address those concerns.\n(3) [Report on response] If the Chairperson of the Commission is not satisfied with the response of the President, the Chairperson shall issue a report to that effect to the Minister.\n14.8 [Information subject to privilege]\n(1) Nothing in this Act authorizes a person to disclose to an observer privileged information, as defined in subsection 17(1) of the Public Complaints and Review Commission Act, and an observer shall not use or disclose that information if it is disclosed.\n(2) [Offence to disclose certain information] Every person who contravenes subsection (1) is\n(a) guilty of an indictable offence and liable to imprisonment for a term of not more than five years; or",
+    "history": "2024, c. 25, s. 111",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 111) (part 3 of 4)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  },
+  {
+    "id": "C-1.4-nif-4-p4",
+    "act_code": "C-1.4",
+    "act_short": "CBSA Act",
+    "act_name": "Canada Border Services Agency Act",
+    "section": "",
+    "marginal_note": "Amendment not in force — 2024, c. 25, s. 111",
+    "part": "AMENDMENTS NOT IN FORCE",
+    "division": "",
+    "heading": "Amendments not in force",
+    "text": "(b) guilty of an offence punishable on summary conviction and liable to a fine of not more than $5,000 or to imprisonment for a term of not more than six months, or to both.\n(3) [Limitation period] Summary conviction proceedings in respect of an offence under subsection (2) may be instituted at any time within but not later than two years after the time when the subject matter of the proceedings arose.\n14.9 [Regulations] The Governor in Council may make regulations\n(a) respecting the criteria and procedures for the appointment of an observer;\n(b) respecting the scope of an observer’s role;\n(c) respecting an observer’s reporting obligations;\n(d) respecting the access to, and use of, the notes, reports or other material prepared by an observer in relation to an investigation;\n(e) prescribing the period within which the President is to provide a response under subsection 14.7(2); and\n(f) generally for carrying out the purposes of sections 14.5 to 14.8.",
+    "history": "2024, c. 25, s. 111",
+    "last_amended": "",
+    "in_force": "",
+    "status": "not-in-force",
+    "current_to": "2024-11-11",
+    "citation": "CBSA Act — amendment not in force (2024, c. 25, s. 111) (part 4 of 4)",
+    "source_url": "https://laws-lois.justice.gc.ca/eng/acts/c-1.4/"
+  }
+]