diff --git "a/RUNBOOK.md" "b/RUNBOOK.md" new file mode 100644--- /dev/null +++ "b/RUNBOOK.md" @@ -0,0 +1,3862 @@ +# CanLex maintainer runbook + +Everything a future maintainer needs that the code alone does not say. + +## The refresh choreography + +Order matters — chunks feed embeddings feed the eval: + +``` +py -m canlex.ingest [--force] [CODES] # Justice Laws XML -> per-Act JSON (incl. Schedules) +py -m canlex.ingest --fr # parallel French corpus (same schema; SOR->DORS URLs) +py -m canlex.pdi [--force] # IRCC PDI trees (TR / refugee / standard / + # identity / PR-card; capped BFS, drops logged) +py -m canlex.ciu [--force] # CIU/PSAC union documents + site VSSAs +py -m canlex.dmemo [--fr] [--force] # CBSA D-Memoranda (EN; --fr = Mémorandums D) +py -m canlex.enf [--fr] [--force] # ENF operational manuals (canada.ca PDFs; --fr = guides ENF) +py -m canlex.amps [--force] # AMPS Master Penalty Document +py -m canlex.charter [--force] # Charter / Constitution HTML +py -m canlex.caselaw # Lexum/Decisia courts + cached CanLII texts +py -m canlex.delegation [--force] # CBSA/IRCC delegation instruments +py -m canlex.directive [--fr] [--force] # NJC directives (EN; --fr = éditions françaises) +py -m canlex.agreement [--fr] [--force] # FB collective agreement (--fr = convention collective) +py -m canlex.benefits [--force] # benefit-plan booklets +py -m canlex.tariff_schedule [--force] # Customs Tariff ch. 98/99 (CBSA edition) +py -m canlex.irb_papers [--force] # IRB Legal Services papers (equivalency, + # criminal refusals) -- PDF via PyMuPDF +py -m canlex.irb_guidelines [--force] # IRB Chairperson's Guidelines (irb_guidelines.json) +py -m canlex.conduct [--force] # policy pages: employer instruments EN/FR, CBSA + # detention policy, STCA, IRCC rehabilitation + # pages (policy.json) +py -m canlex.commentary # curated datasets -> commentary chunks +py -m canlex.embed # re-embed what changed (run after ANY of the above; + # incremental by content hash -- seconds for a refresh; + # --full after a model/pooling/prefix/embed_texts change) +py -m unittest discover -s tests # all offline +py -m canlex.eval # gate: legacy and holdout (undated) no lower than the + # last committed run (dated slices such as criminality + # join ALL and holdout (sweep-blind), so those lines move) +``` + +Every ingester above (and `canlex.commentary`) also takes `--allow-shrink`: +the corpus-write guard refuses a write below 80% of the stored chunk count +("The corpus-write guard", below), and `--allow-shrink` lets an understood +drop through. `tests/test_docs_drift.py` holds this list to the ingesters +in canlex/. + +`canlex.embed` is mandatory after any chunk change: a chunk missing from +embeddings.npz scores -1.0 on the semantic side (index.py fills -1.0, never +zero, so it can never rank semantically) and quietly degrades to BM25-only +(a stderr warning is the only signal). + +## Deploying to the Hugging Face Space + +The Space (`space` remote, `Beemer0/CanLex`) has a **1 GB LFS storage cap**, and +*every version ever pushed counts against it* -- `embeddings.npz` alone is ~40 MB +per push. On 2026-08-31 that cap was reached and the push was rejected outright. + +Two things were done, and they change how you deploy: + +1. The Space's history was **super-squashed** (74 commits -> 1) to free the + accumulated LFS versions. This is non-revertible, and it permanently diverged + the Space's history from local `main`. +2. Deploys now go out as a **single squashed commit on the `deploy` branch**, + never as a push of `main`. + +`main` keeps the full commit-by-commit history and is the source of truth. It is +the only copy -- there is no other remote. **Never force-push `main` to the +Space**: its 82-commit history references every historical `embeddings.npz`, and +pushing it would re-upload all of them and blow the cap again. + +**Deploy the composed release, never `main` as it stands** (2026-09-12). +`main` carries curated legal content that is HELD FOR REVIEW; the approved +baseline is recorded in `data/curated/review_state.json` (c32bd27, approved +2026-09-16; before it f5cf54d, 2026-09-15). The composed release is main's code with the approved curated +JSON: + +``` +py tools/compose_release.py # detached worktree; tests run on the composed tree +git fetch space main +TREE=$(git rev-parse release-composed^{tree}) +PARENT=$(git rev-parse space/main) # write it down: this is the rollback point +NEW=$(git commit-tree $TREE -p $PARENT -m "Deploy: ") +git branch -f deploy $NEW +git lfs ls-files -s deploy # see below +git push space deploy:main # the pre-push hook runs tools/predeploy.py on $NEW +py tools/smoke_space.py # once the Space has rebuilt +``` + +What the composed tree ships: + +- `canlex/`, `tests/`, `tools/` and everything outside `data/curated`: + main's. +- `data/processed/*`: main's official documents -- legislation, ENF / PDI / + IRB guidance, Guide 5312, case law, the Income Tax Act -- EXCEPT + `commentary.json`, which is re-rendered from the approved curated files, and + the commentary rows of `embeddings.npz`, re-embedded to match + (`canlex.embed` is content-hash keyed, so only changed commentary chunks + are encoded). The memo's pointers to the IRB Legal Services paper and Guide + 5312 assume these local-only documents ship; they do. +- `data/curated/*.json`: the approved commit's. A curated JSON file the + approved commit lacks is held and left out; `review_state.json` ships as + metadata. +- `data/curated/drafts/` (since the code audit of 2026-09-21): the approved + commit's subtree, swapped into the worktree's index after the composed + tree's tests ran against main's files. Review memos and the held batches' + source drafts are held with their batch: the review check + (`predeploy.check_review`) holds any difference from the approved + commit's subtree, so a memo cannot reach the public Space before its batch + is approved. Tests that read a draft (`tests/test_step1_rows.py`, + `tests/test_tran_limbs.py`) skip where the file is absent. +- `data/curated/held_manifest.json` (2026-09-15, the wave-2 verifier's held + manifest): written by the composer, never on main. It lists the ids and + MATCHING WORDS of what main holds that the approved files lack -- disposition + entries and pairings (held `entry` / `pairing`), words added to approved ones + (`names` / `terms`), statute-code readings -- and nothing of their analysis. + When a record's words reach held content ahead of everything loaded, the + tools print **HELD FOR REVIEW -- not in this deployment** (naming it and + what the answer used instead), suppress any 'closest by meaning' guess, and + CANLEX_RESULT carries `held` and `assumed: held-content`. predeploy does not + count the file as curated content (`predeploy.HELD_MANIFEST`). + +`compose_release.py` never touches the main working tree: it builds in a +detached `git worktree` (default `%TEMP%\canlex-compose`, removed afterwards +unless `--keep`), runs the offline suite there, commits on the worktree's +HEAD, runs predeploy's tree checks on that commit and only then moves the +`release-composed` branch. A failing composed-tree test blocks it: tests that +assert held content skip on the composed tree, so a failure there is a real +difference between main's code and the approved data (the bare-'pending' +disposition was the first). `--no-embed` builds and tests without +re-embedding, and then does not commit; with `--no-commit` it is a dry run +read from its test summary -- the stale-vector count it prints is a warning +there, and it exits 0 when the composed tree's tests pass (the review of batch +1, round 5). Record the embed's runtime here the first time it runs. `--review-state PATH` composes against another review +state -- a trial baseline, for instance to see what an older approval would +ship and announce -- and ships that file in the tree; it requires +`--no-commit`. Build one with `py tools/predeploy.py --print-review-state +` merged into a copy of review_state.json. + +After composing the curated files it prints **STALE APPROVED CONTENT** (wave 2 +verifier, 2026-09-13): each approved pairing whose verify section now states +a maximum the corpus did not state when the table was approved, and that the +pairing does not quote (2026, c. 19: s. 173(2)(a) 2 -> 10 years, s. 286.1(2) +10 -> 14). `tests/test_refresh_drift.py` fails on the same list, so such a +tree is not committed: approve the held fix first. It also prints +**RE-READ**: every verify section whose text changed since approval (s. 271's +summary maximum, for one) -- not a failure, but read them before approving. + +`py tools/predeploy.py --tree release-composed` runs the same gate by hand; +without `--tree` it checks HEAD and, on `main`, fails the review check by +design. To approve new curated content: `py tools/predeploy.py +--print-review-state ` prints the `approved` and `curated_files` +blocks for `review_state.json`; commit that, and the next compose ships it. + +**Legal text in code has its own check, `review-code`** (2026-09-12). The +composed release ships main's CODE, so officer-facing legal wording and +routing that live in code -- the allegation, removal-path, s. 44, family, IAD +and fee lines, the tie map, the publisher labels, the rehabilitation rules, +caveats and methodology; the list is `canlex/reviewed_code.py` -- would +otherwise reach the Space unreviewed. predeploy fingerprints each registered +definition (its AST: formatting and comments do not count, words and logic +do) in the deploy tree and compares it with the `approved_code` commit in +`review_state.json` (read it from the file: it moves at every approval -- +c32bd27 as of 2026-09-16; before that d784d67, 2026-09-09 to 2026-09-16). +Anything new or +changed is HELD, and the check FAILS -- a composed release included -- until +the maintainer reviews it (the check prints the `git diff` to read) and +records the reviewed commit as `approved_code` (`--print-review-state` prints +that block too). `py tools/compose_release.py --allow-held-code` (or +predeploy's `--allow-held`, or its `--allow-held-code` for this check alone) +downgrades it to a warning, only on the +maintainer's explicit say-so. When new legal wording goes into code, add its +definition to the registry; `tests/test_reviewed_code.py` fails on an entry +that names nothing. Since the code audit of 2026-09-21: the docstring and +decorator keywords of every `mcp.tool` / `mcp.prompt` / `mcp.resource` +function count in its fingerprint (FastMCP publishes the docstring verbatim +as the tool's description, the text the client model reads, and +canlex_screen_record's states law), every such function is registered +(`tests/test_reviewed_code.py` fails otherwise), the registry is read at +BOTH commits and the union checked -- an entry may be added, never dropped, +without a HELD line naming the drop, and a tree with no +`canlex/reviewed_code.py` while the approved code has one FAILS (a rollback +to a pre-registry release needs `CANLEX_PREDEPLOY_ALLOW_HELD=1`) -- and a +tuple-unpacked constant (`STRUCTURED_LIMB, STATED_LIMB = ...`) is a +definition the gate sees. + +Since 2026-09-13 (wave 2 verifier) each registered definition is fingerprinted +TOGETHER WITH every canlex definition it reaches -- the names its body uses, +in its own module or imported from another canlex module, transitively -- so a +change to a helper (a unit table, a negation window, a history reader) holds +the registered name that relies on it; the HELD line names the reached +definitions that changed ("via ..."). Every reached definition must be +classified: registered in `REVIEWED` if it carries legal wording or routing, +else listed in `PLUMBING` (search, loading, parsing, formatting). +`tests/test_reviewed_code.py` fails on a reached definition that is neither. + +**The authority registry prints only when it says so** (2026-09-15). +`data/curated/authorities.json` links every decision the curated datasets cite +to the corpus. The tools print its pointer under each cited authority ('full +text in CanLex: canlex_search_legislation doc_type caselaw, act=...' or 'not in +the CanLex corpus; quoted in ...'; never `canlex_case`), the statute sentence +behind each verified maximum (`VERIFIED TEXT`), and its Saini +subsequent-history note -- but only when the file carries `"render": true`. +The f5cf54d registry (approved 2026-09-15) recorded corrections to the curated +holdings that had never been applied, so a release composed on that baseline +printed no pointers and kept `server.SUPERSEDED_CITES`; the corrections and +the render flag were approved with baseline c32bd27 (2026-09-16). A row's `corrected` lists each +correction applied to a curated file (the old and new strings); +`tests/test_authorities.py` checks them against the files, every pin and +decision key against `data/processed`, and that every curated cite resolves. +Reviewer text goes in a pin's `review_note`, never in the printed `pin`. + +That recipe ships exactly the composed tree while uploading only the LFS +objects the Space does not already hold. Check `git lfs ls-files -s deploy` +before pushing: if it lists more than the files you actually changed, you are +about to spend storage on blobs nobody needs. (A composed release rewrites +`embeddings.npz` whenever the approved commentary differs from main's -- one +~55 MB LFS version per deploy against the 1 GB cap.) + +### Rollback + +Re-push the previous deploy (the `PARENT` you wrote down) as a fast-forward, +so nothing is force-pushed and no LFS object is uploaded -- the Space already +holds them: + +``` +git fetch space main +PREV= +NEW=$(git commit-tree $(git rev-parse $PREV^{tree}) -p $(git rev-parse space/main) -m "Rollback to $PREV") +git push space $NEW:main +py tools/smoke_space.py +``` + +If the approved baseline has moved since `PREV` was deployed, the hook's +review check refuses the older curated content; `CANLEX_PREDEPLOY_ALLOW_HELD=1` +downgrades that one check to a warning for the rollback push. + +When the cap is hit again, re-run the squash (`HfApi().super_squash_history`) +and rebuild `deploy` on the new base. + +**Hugging Face also rejects any NON-LFS file over 10 MiB** (10,485,760 +bytes) on push. `.gitattributes` tracks exactly the processed files above +that line -- `caselaw.json`, `embeddings.npz`, and since 2026-09-10 the +Income Tax Act's `I-3.3.json` (12.2 MB) and `I-3.3-fr.json` (14.0 MB). Track +files by exact path, never with a glob over `data/processed/`: converting +every JSON to LFS would re-upload all of them and spend the 1 GB cap at +once. Watch `dmemos-fr.json` -- 10.27 MB, just under the limit; a D-memo +refresh that grows it past 10 MiB will make the next deploy bounce, and the +fix is to add its path here before committing the refresh. Before any +deploy, this lists every oversized plain blob: + +``` +for f in data/processed/*.json; do s=$(stat -c %s "$f"); [ "$s" -gt 10485760 ] && echo "$s $f"; done +``` + +and each path it prints must appear in `git lfs ls-files`. + +### `--force` and the raw cache + +Every ingester caches its downloads under `data/raw/` and **reuses them on the +next run**. That is deliberate politeness, but it means a plain re-run cannot +notice a document upstream revised *in place at the same URL* — it rebuilds +identical chunks from the stale file. `--force` re-downloads instead. + +Use it whenever `refresh.py --check` reports a source as STALE against an +upstream `Last-Modified` or version stamp. The CIU/PSAC and IRCC IL3 documents +are the ones that actually get reposted at fixed URLs; most other sources move +their URL when the content changes, so the cache turns over on its own. + +This bit us on 2026-08-31: CIU Regulations was flagged STALE, `py -m canlex.ciu` +produced a byte-identical `ciu.json`, and the row could not clear because the +cache mtime never moved. `CacheInvalidationContractTests` in `tests/test_common.py` +now fails the suite if any ingester loses its `--force` wiring (since +2026-09-21 it also sees the one-level wrappers, `dmemo._fetch` and friends: +the D-memo PDF fallback had no `force` of its own). + +Two cache rules from the code audit of 2026-09-21. (1) A download lands in +`.part` and is renamed into place only once it succeeded, on the urllib +and the PowerShell paths alike, and `canlex.ingest` caches a Justice Laws +body only when it starts with `` / `` and evicts a +cached XML it cannot parse ("cache removed; re-run"): a stalled canada.ca +response used to leave a 0-byte page in `data/raw` that every later plain run +reused, and a maintenance page cached as an Act failed identically on every +run. (2) `canlex.pdi` names each cached page by its path below the OBM root +(`pdi-pr-permanent-residence-non-economic-classes-construction-admissibility.html`), +not by its file name: the five `...//admissibility.html` pages of +the PR tree shared one cache file, and a plain run served one pathway's text +under all five URLs (7 colliding names, 12 pages, 87 chunks). The index page +keeps `{code}-index.html` (the refresh watch reads it). The first `py -m +canlex.pdi` after that change re-downloads every page (nothing is cached +under the new names); the stored `pdi.json` is unaffected until then. + +### The corpus-write guard + +Every ingester above rebuilds its whole processed file from a fresh scrape, so +an upstream page that changes shape turns into data loss at the write. All of +them go through `_common.write_corpus` (`canlex.ciu` only since the code +audit of 2026-09-21: until then it wrote `ciu.json` directly -- no guard, no +atomic write, no preserve-on-failure -- and a run with the union site down +wrote an empty list over the 635 stored chunks and exited 0; +`tests/test_ciu.py` now holds every ingester to the guard), which **refuses +to write a corpus below 80% of the stored one** and makes the process exit +non-zero: + +``` + REFUSING to write: 0 chunks would replace 1870 already in dmemos.json. ... +``` + +That is not a transient error — it means the scrape broke. Diagnose it, don't +retry. Once the drop is understood and intended (a source genuinely shrank), +re-run the same module with `--allow-shrink`. Because the runbook chains these +as `py -m canlex.X && py -m canlex.embed`, a refusal now correctly stops the +embed instead of propagating a gutted corpus. + +Most ingesters also **preserve on failure**: an item that errors keeps its +last-good chunks (`_common.preserve_dropped`), restricted to items that +actually failed, so anything removed on purpose stays gone. Deliberate +exception: `enf.py` treats an HTTP 404 as IRCC retiring a chapter — its +documented policy — so a 404 drops the chapter while any other error preserves +it. Writes are atomic (tmp + `os.replace`): a killed run cannot leave truncated +JSON that would read as an empty corpus and disable the guard next time. + +## Deploys + +CanLex is an MCP server only, and there is one Space to deploy: the MCP Space +(`Beemer0/CanLex`). **2026-09-16:** the Gradio chat page (the separate +`Canlex-web` Space) and its Google Gemini API integration were retired: the +maintainer pauses or deletes that Space and revokes its API key; `webapp/`, +its tests, `webapp/decls.py`'s parity check and its deploy step were removed +from the repo (the git history keeps them). + +1. **Compose** the release: `py tools/compose_release.py` (the recipe in + "Deploying to the Hugging Face Space" above). +2. **Commit-tree** the composed tree onto `space/main` and point `deploy` at + it -- never `git push space main`: the Space's history diverged at the + 2026-08-31 squash, so that push is rejected, and forcing it would + re-upload every historical `embeddings.npz`. +3. **Push** `deploy` to the `space` remote (`git push space deploy:main`; the + pre-push hook runs predeploy on it). The Space rebuilds automatically + after the push. `embeddings.npz` ships in-repo via git-LFS — a fresh clone + needs `git lfs install`. +4. **Smoke test** once the Space has rebuilt: `py tools/smoke_space.py` + (key-free; `--url` or `CANLEX_MCP_URL` overrides the default + `https://beemer0-canlex.hf.space/mcp`) -- tools/list on the live endpoint, + every tool `canlex.server` registers (14) checked present on the LIVE + server, and five golden read-only calls. A non-zero exit means: fix + forward or roll the Space back ("Rollback" above). + +**The pre-push hook.** `tools/hooks/pre-push` is committed; install it once +per clone (it replaces the stock LFS-only hook and keeps its +`git lfs pre-push` call): + +``` +cp tools/hooks/pre-push .git/hooks/pre-push +``` + +On a push to the `space` remote (or its URL) it runs +`tools/predeploy.py --tree ` for every commit pushed and refuses the +push on any failure: the offline suite, no non-LFS blob over 10 MiB, +`commentary.json` exactly as rendered from the tree's own curated JSON with a +current vector for every commentary chunk, the tree's curated JSON equal to +the approved baseline and its officer-facing legal text in code equal to the +approved code, so held content cannot ship by accident. Pushes to any other +remote only run `git lfs pre-push`. + +### Limits and guards (the code audit of 2026-09-21) + +What the server refuses or bounds, and why each number is where it is. Name +the guard in a commit before raising any of them. + +- **Request body.** `_Utf8Body` (canlex/server.py, `MAX_BODY_BYTES`) refuses + any POST over 1 MiB with 413 before buffering or decoding it. The largest + legitimate body is about 40 KB (20 entries at the 24,000-character + `entries_json` cap). Before the cap a 64 MB body cost +390..450 MB of + working set per request. +- **Sessions.** In HTTP mode (`CANLEX_HTTP=1`, the Space) an MCP session the + client never closes expires after `SESSION_IDLE_SECONDS` (1800 s) idle; + before, each lived until restart with its initialize params. There is no + global cap on open sessions (`SESSION_MAX_OPEN = None`); each client may + open `CANLEX_SESSION_OPEN_LIMIT` (60) new sessions per 10 minutes instead + (see "Verification fixes (2026-09-22): F3" at the end). +- **One tool at a time.** `offload_sync_tools` runs the 14 tools in a worker + thread under `anyio.CapacityLimiter(1)`, so a 2-7 s search no longer stalls + pings, the notice page or another session's cheap call -- but execution + stays serialized: the alias table, the `_analyse` cache and the lazy index + tables are module-level state that has not been reviewed for concurrency. + Do not raise the limit before `load_aliases` is idempotent and those caches + are locked. +- **Per-field caps.** `query` is 500 characters (the reranker's ~509-token + bound; the tool cannot reach the tokenizer's truncation error); + offence/disposition fields are 200 characters (`screening.record_order` is + quadratic in segments: ~1 ms at 200 characters, 3-12 s at 5,000). Every + module-level regex was timed on 100 KB adversarial input under 0.25 s. +- **Call log.** `calllog.ARGS_BUDGET` keeps at most 64,000 characters of + argument text per record (a valid call carrying a 4 MB extra key once wrote + a 4.01 MB line); `RESULT_KEYS` lists the headline fields retained; client + tags are bounded to 256 names of 80 characters with control characters + dropped. +- **Host/Origin.** `FastMCP(..., host="0.0.0.0")` switches off the DNS-rebinding + (Host/Origin) check, which the Space needs behind Hugging Face's ingress. A + local HTTP run (`CANLEX_HTTP=1` on the laptop) must be reached only from the + same machine -- `canlii_key.txt` is the one local asset a rebinding page + could reach (the Space runs key-free). +- **`CANLEX_SEE_ALSO_DRAFT`** (canlex/index.py) co-surfaces an UNREVIEWED + see-also draft after search results: local diagnostics only, never set it + on the Space. +- **Review-gate scope.** Since 2026-09-21 the code fingerprint counts a + published tool's docstring and `description=` (the MCP description the + client model reads), tuple-unpacked constants and module-level mutations; + the registry is read at BOTH the approved and the deploy commit (a dropped + entry is a HELD line, a missing registry FAILS); the composed release ships + `data/curated/drafts/` from the APPROVED commit, not from main, so held + review memos do not reach the public Space. The approved-code baseline is + `c32bd27` (2026-09-16). + +## Secrets and variables (set in the Space's settings) + +| Where | Name | Purpose | +|---|---|---| +| MCP Space | `CANLII_API_KEY` | optional, enables canlex_case | +| MCP Space | `CANLEX_LOG_DATASET` | optional, the private dataset repo the call log pushes to (e.g. `Beemer0/canlex-call-log`); needs `CANLEX_LOG_TOKEN` too | +| MCP Space | `CANLEX_LOG_TOKEN` | optional, a fine-grained token with read and write access to that one dataset repo | +| MCP Space (variable) | `CANLEX_LOG_EVERY_MIN` | optional, minutes between call-log pushes (default 10) | + +Locally, the CanLII key lives in `canlii_key.txt` (gitignored — never +commit it). + +### Environment variables + +Every environment variable the code reads (`grep -rn "environ" canlex tools`), +plus the two the Dockerfile sets for the Hugging Face library. +`tests/test_docs_drift.py` fails when the code reads one this table does not +list, or the table lists a `CANLEX_*` name nothing reads or sets. Unset means +the default; nothing needs to be set for the local stdio server. + +| Name | Read by | Default | What it does | +|---|---|---|---| +| `PORT` | canlex/server.py | `8000` | HTTP port under `CANLEX_HTTP`; the Dockerfile sets `7860` (the Space's port) | +| `CANLEX_HTTP` | canlex/server.py | unset (stdio) | any value serves streamable HTTP on `0.0.0.0:PORT` (the Dockerfile sets `1`). **Local use only on this machine**: the host binding switches off the Host/Origin check ("Limits and guards") | +| `CANLII_API_KEY` | canlex/citator.py | unset | enables canlex_case (a Space secret); locally `canlii_key.txt` is read instead | +| `CANLEX_LOG_DIR` | canlex/calllog.py, canlex/loganalyze.py | unset (no local log); `.` for loganalyze | the local call log `calls.jsonl` ("Call log"); the Dockerfile sets `/tmp/canlex-logs`, `.mcp.json` sets `data/logs` | +| `CANLEX_LOG_DATASET` | canlex/calllog.py | unset | the private dataset the durable call log pushes to (needs `CANLEX_LOG_TOKEN`) | +| `CANLEX_LOG_TOKEN` | canlex/calllog.py, canlex/loganalyze.py | unset | the dataset token (a Space secret); loganalyze also reads it as a Windows user variable | +| `CANLEX_LOG_EVERY_MIN` | canlex/calllog.py | `10` | minutes between the durable log's pushes | +| `HF_HUB_OFFLINE` | huggingface_hub (the Dockerfile sets it) | `1` on the Space | model files come only from the baked cache; `calllog.allow_dataset_requests` lets the dataset sink through | +| `HF_HOME` | huggingface_hub (the Dockerfile sets it) | `/app/.hf_cache` on the Space | where the baked model cache lives | +| `CANLEX_EMB_REPO` | canlex/embed.py | `Xenova/bge-small-en-v1.5` | embedding model; any of the four `CANLEX_EMB_*` changed means `py -m canlex.embed --full` and an eval | +| `CANLEX_EMB_POOLING` | canlex/embed.py | `cls` | `cls` (BGE) or `mean` (E5/MiniLM) | +| `CANLEX_EMB_QUERY_PREFIX` | canlex/embed.py | `Represent this sentence for searching relevant passages: ` | the query instruction prefix | +| `CANLEX_EMB_PASSAGE_PREFIX` | canlex/embed.py | empty | the passage prefix | +| `CANLEX_ORT_THREADS` | canlex/rerank.py | `0` (all cores) | ONNX Runtime intra-op threads for the reranker and embedder sessions | +| `CANLEX_RERANK_DOC_CHARS` | canlex/rerank.py | `2000` | characters of each candidate the cross-encoder reads | +| `CANLEX_RERANK_POOL` | canlex/index.py | `16` | fused candidates the cross-encoder rescores (the latency knob) | +| `CANLEX_RERANK_DUAL` | canlex/index.py | unset (off) | also rerank against the synonym-expanded query | +| `CANLEX_POOL_CAP` | canlex/index.py | `8` | chunks one decision or memorandum may hold in the rerank pool | +| `CANLEX_HIGHLIGHT_DEPTH` | canlex/index.py | `3` | results that get a 'Most on point' pinpoint (the audit follow-up of 2026-09-21 lowered it from 8; eval-gated) | +| `CANLEX_CACHE_DIR` | canlex/bm25.py | `data/cache` | where the BM25 postings cache (about 47 MB, keyed by the corpus files and the tokenizer code) is written; a read-only disk falls back to an in-memory build | +| `CANLEX_RECALL_CAP` | canlex/index.py | `20` | chunks one capped source may give each retriever at recall | +| `CANLEX_MN_WEIGHT` | canlex/index.py | `0.0024` | title-match boost per unit of idf-weighted overlap | +| `CANLEX_MN_CAP` | canlex/index.py | `0.024` | ceiling on the title-match boost | +| `CANLEX_REG_PENALTY` | canlex/index.py | `0.004` | fusion penalty on regulation sections | +| `CANLEX_NIF_PENALTY` | canlex/index.py | `0.012` | fusion penalty on not-in-force amendments | +| `CANLEX_MEMO_PENALTY` | canlex/index.py | `0.008` | fusion penalty on guidance | +| `CANLEX_REPEALED_PENALTY` | canlex/index.py | `0.008` | fusion penalty on repealed provisions | +| `CANLEX_INAPPLICABLE_PENALTY` | canlex/index.py | `0.004` | fusion penalty on provisions the audience is excluded from | +| `CANLEX_BACKMATTER_PENALTY` | canlex/index.py | `0.004` | fusion penalty on a collective agreement's back-matter | +| `CANLEX_COMMENTARY_PENALTY` | canlex/index.py | `0` | fusion penalty on CanLex's curated commentary | +| `CANLEX_ATIP_PENALTY` | canlex/index.py | `0.004` | extra penalty on unofficial ATIP copies (ENF 2, ENF 14) | +| `CANLEX_APPLICANT_PENALTY` | canlex/index.py | `0` | extra penalty on IRCC's applicant-facing pages | +| `CANLEX_COURT_HINT` | canlex/index.py | `0.004` | fusion bonus for the court a query names | +| `CANLEX_ACT_HINT` | canlex/index.py | `0.012` | fusion bonus for the instrument a query names | +| `CANLEX_CRIMINALITY_ROUTE` | canlex/index.py | `1` | `0` turns off the s. 36 primary-provision route (roadmap T5.2) | +| `CANLEX_STATE_ONE_SOURCE` | canlex/index.py | `1` | `0` counts the 51 US-state pages as separate sources (roadmap T5.3) | +| `CANLEX_SEE_ALSO_DRAFT` | canlex/index.py | unset | `1` (or a path) co-surfaces the UNREVIEWED see-also draft when no curated see-also exists: **local diagnostics only, never set it on the Space** | +| `CANLEX_EVAL_SWEEP` | canlex/eval.py (set by canlex/sweep.py) | unset | sweep mode: holdout questions stay blind to the knob being swept | +| `CANLEX_PREDEPLOY_ALLOW_HELD` | tools/predeploy.py | unset | `1` is `--allow-held`: ship with HELD content (never by default) | +| `CANLEX_MCP_URL` | tools/smoke_space.py | `https://beemer0-canlex.hf.space/mcp` | the endpoint the smoke test calls | +| `CANLEX_RATE_LIMIT_PER_MIN` | canlex/ratelimit.py | unset (off) | requests a minute per client on the HTTP transport; unset or invalid = no limit | +| `CANLEX_RATE_LIMIT_BURST` | canlex/ratelimit.py | one minute's allowance | the burst a client may spend at once | +| `CANLEX_SESSION_OPEN_LIMIT` | canlex/ratelimit.py | `60` (on over HTTP) | new MCP sessions one client may open per 10 minutes (then one each 10 s); `0` = no per-client cap; not a whole number = the default | + +The retrieval knobs (`CANLEX_RERANK_DOC_CHARS` through +`CANLEX_STATE_ONE_SOURCE`) are read once at import (`CANLEX_RERANK_DUAL` at +each search) and are eval-gated: `canlex.sweep` sets them for each eval run +it launches, and a change to a default ships only with a before/after eval. + +## Call log (2026-09-16) + +The maintainer wants to see what people ask through the MCP tools. +`canlex/calllog.py` wraps FastMCP's tool manager once at startup (the +`__main__` block of `canlex/server.py`), so every tool call -- all 14 tools, +whether it returns or raises -- becomes one JSON record. No tool function was +edited, so no REVIEWED fingerprint moved; the new definitions are PLUMBING. + +**What a record holds:** `ts` (UTC ISO 8601); `tool`; `arguments` exactly as +the client sent them (a string over 4,000 characters is cut, marked +`[truncated: N chars]` and its path listed in `truncated`); `duration_ms`; +`ok`; `error` (the exception's type name only, e.g. `ValidationError` -- +never its message or traceback); `output_chars`; for +`canlex_search_legislation` `n_results`, `confidence` and `top_citations` +(the fields the old `queries.jsonl` carried; `server._log_query` now hands +them to the record); for a tool whose output ends in a CANLEX_RESULT line +(the screening tools) `result` with the CANLEX_RESULT fields named in +`calllog.RESULT_KEYS`, each only when the line carries it: `branch`, +`rehab_status`, `status`, `verdict`, `needs` (the questions, at most 10 of +500 characters each), and since the step-1 review of batch 1 which entry +answered and how -- `disposition_entry`, `entry`, `footing`, `act_branch`, +`question_flag`, `disposition_flags` and `allegation_paragraphs`; and +`dropped_before` when calls went unrecorded before it. +`tests/test_docs_drift.py` fails when `RESULT_KEYS` gains a field this +paragraph does not name. + +**What it never holds:** IP addresses, request headers, user agents, MCP +session ids, cookies, tokens or any other auth material -- the wrapper sees +only the tool name and its arguments. The tool's output text is not stored +(only its length and the CANLEX_RESULT fields above). But the arguments are +what the user typed: offences, dates, places, and sometimes a person's name +or date of birth (`canlex_screen`, `canlex_screen_record` and +`canlex_screen_history` accept `date_of_birth`, and the record keeps it as +sent). Open maintainer decision: whether the record should keep only the birth +year (or the age) instead of the full date -- a change to the record only, +never to what the tool receives. + +**Sinks** (each opt-in; with none set nothing is wrapped, no thread starts, +nothing is logged -- the local stdio server with no env vars is unchanged): + +- `CANLEX_LOG_DIR` -- appends to `CANLEX_LOG_DIR/calls.jsonl`. The Dockerfile + sets `/tmp/canlex-logs` on the Space, where it is lost on every restart; + `.mcp.json` sets `data/logs` (gitignored) for the maintainer's own stdio + server, so that server now records every call there, not only searches. + `queries.jsonl` is no longer written; `loganalyze` still reads old copies. +- `CANLEX_LOG_DATASET` + `CANLEX_LOG_TOKEN` (both) -- the durable log: + `huggingface_hub.CommitScheduler` pushes `logs/YYYY-MM-DD/.jsonl` + (one file per server process per UTC day) to that dataset every + `CANLEX_LOG_EVERY_MIN` minutes (default 10), and once more at a graceful + shutdown. It is opened on a background thread, so startup never waits on + the Hub. The dataset must be **private**: `CommitScheduler(private=True)` + only applies when it creates the repo, so once the scheduler is up the sink + reads the dataset's info and, if it is not private, stops the scheduler + before any record reaches it ("the durable call log is off: the dataset ... + is not private"). The check runs at startup only: never make the dataset + public while the Space runs. When the dataset is open and private the log + says `CanLex call log: the dataset call log is open: '' (private)`. + One variable without the other, a token the Hub refuses, or a public + dataset: one stderr line ("the durable call log is off ...") and the server + keeps serving. A network or 5xx error is retried after 30 s, 2 min and + 10 min, so that "off" line can come about 12.5 minutes after startup. The + token is never printed or logged. +- Best-effort throughout: a tool call only puts a small tuple on a bounded + queue (1,000); a background thread builds and writes the records. A full + queue drops the call and counts it (`dropped_before` on the next record). + A sink that raises or hangs cannot slow, change or break a tool response. + A batch a sink refuses is retried one record at a time, so one bad record + costs only itself; a sink's failures go to stderr by exception type, the + first of each type at once and repeats at most once an hour (with a count). + A lone UTF-16 surrogate in an argument (valid JSON a client can send, not + valid UTF-8) is recorded as its backslash escape, e.g. `\ud800`. +- `HF_HUB_OFFLINE=1` (the Dockerfile sets it so model files come only from + the baked cache) blocks every huggingface_hub request. When the dataset sink + starts in offline mode, `calllog.allow_dataset_requests` installs a Hub + client whose request hook still refuses every Hub request except repo + creation and requests for that one dataset repo -- a model download's first + request (`//resolve/...`) is still refused. + +**Maintainer setup** (only the maintainer can do these; done 2026-09-16 -- +the live Space carries the call log and serves the notice): + +1. On huggingface.co: **+ New -> Dataset**, owner `Beemer0`, name e.g. + `canlex-call-log`, visibility **Private**, Create. +2. **Settings -> Access Tokens -> Create new token -> Fine-grained**. Name it + (e.g. `canlex-call-log-writer`); under *Repositories permissions* pick the + dataset `Beemer0/canlex-call-log` and tick both **Read access to contents + of selected repos** and **Write access to contents/settings of selected + repos** (the scheduler reads the repo's info as well as committing to it); + tick nothing else (no user, org, inference or other-repo permissions). + Copy the token once, straight into step 3. +3. On the `Beemer0/CanLex` Space: **Settings -> Variables and secrets -> New + secret** `CANLEX_LOG_DATASET` = `Beemer0/canlex-call-log`, and **New + secret** `CANLEX_LOG_TOKEN` = the token. Optionally a variable + `CANLEX_LOG_EVERY_MIN`. +4. The Space restarts. Its container log first shows `CanLex call log: + recording every tool call to: local call log (...), dataset call log + (Beemer0/canlex-call-log, every 10 min; opening in the background)` -- that line only + says the sink is starting. The durable log is working once a later line + says `CanLex call log: the dataset call log is open: + 'Beemer0/canlex-call-log' (private)`. A "durable call log is off" line + means it is not (a refused token or a public dataset says so within + seconds; a network error only after the retries, about 12.5 minutes). + After a tool call and up to 10 minutes, the dataset's Files tab shows + `logs//.jsonl` from a "Scheduled Commit". + +**Reading it:** + +``` +py -m canlex.loganalyze --dataset Beemer0/canlex-call-log # asks for the token; --out DIR keeps the download +py -m canlex.loganalyze --days 7 --dataset Beemer0/canlex-call-log data/logs + # the last 7 days, dataset + local log +py -m canlex.loganalyze data/logs # the local stdio server's log +``` + +With `--dataset` it asks for a token that can read the dataset, with the +input hidden. Never type or paste the token itself into a command line (for +example `$env:CANLEX_LOG_TOKEN = "hf_..."`): Windows PowerShell 5.1 with +PSReadLine 2.0 saves every line typed, in plain text, to +`%APPDATA%\Microsoft\Windows\PowerShell\PSReadLine\ConsoleHost_history.txt` +(PSReadLine only started leaving out lines that look secret in 2.2). If +`CANLEX_LOG_TOKEN` is already set in the environment it is used without +asking; to set it for one session without typing it, copy the token and run +`$env:CANLEX_LOG_TOKEN = Get-Clipboard`. + +**Unattended reading (the weekly check, 2026-09-16).** A scheduled run has no +terminal to ask in, so `loganalyze` also reads `CANLEX_LOG_TOKEN` as a +**Windows user environment variable**, straight from the registry (so no app +restart is needed after setting it). Set it once in Windows' own dialog, never +a shell: Windows search -> "Edit environment variables for your account" -> +New -> name `CANLEX_LOG_TOKEN`, value = a fine-grained token with **read** +access to `Beemer0/canlex-call-log` only (a separate token from the Space's +read/write one, so the copy on this computer cannot change the log). It is +never read from huggingface_hub's general saved login (`hf auth login`, +`HF_TOKEN`): that one may be broad. Without the variable the weekly check +reports the local log only and says the dataset was not read. + +It reports calls per tool and per day, the most common question texts and +offence strings, no-result and weak-match searches, NOT SETTLED screening +results (an open branch -- `depends`, `tie`, `tran-open`, `limb-open`, or +`unknown` when no pairing matched -- or a `not-settled` / `undetermined` +rehabilitation status; `undetermined` and `unknown` were counted as settled +until 2026-09-21), NOT ASSESSED transborder results (branch `table-held`: +the s. 19(1)(c)-(g) table is held on the deployment; its own line) and errors. +The `--dataset` download goes to a temporary folder deleted afterwards. Or +browse the private dataset's Files tab on the website. + +**Privacy and retention.** The log may hold personal facts about travellers. +Keep the dataset private and add no collaborators; keep the token only in the +Space secret and hidden prompts (never a command line or a file); never +commit a log or paste a report anywhere public. Delete old days periodically +(e.g. keep 90 days): delete the `logs//` folders on the website, or in +`py`: `HfApi(token=getpass.getpass()).delete_folder("logs/", +repo_id="Beemer0/canlex-call-log", repo_type="dataset")` (after `import +getpass` and `from huggingface_hub import HfApi`; the token needs write +access). A deleted file stays in the dataset's git history until you run +`HfApi(token=getpass.getpass()).super_squash_history("Beemer0/canlex-call-log", +repo_type="dataset")`. Delete only past days: the running Space re-uploads +today's file whenever it grows. To stop the durable log, delete the two +secrets (the Space restarts) or revoke the token (the push fails with an +error line in the Space log; after the next restart, one warning). + +**The notice.** While logging is on (on the Space always, since the +Dockerfile sets `CANLEX_LOG_DIR`), `calllog.announce` tells the server's users +so in two places MCP users can actually see: the MCP `instructions` every +client receives when it connects, and a plain-text page at the root URL +(`https://beemer0-canlex.hf.space/`, which is what the Space page's frame +shows -- a Docker Space's page shows the running app, not README.md). Both +carry `calllog.NOTICE`: "Tool calls to this server, including their +arguments, are logged by the operator." With no logging variable set there is +no notice, no page and no instructions. README.md ("Call log (2026-09-16)") +says the same, but on the Space it is only readable under Files -> +README.md. The notice is a plain statement of fact and gives no advice about +which arguments to send: advice such as "do not pass dates of birth" would +steer the screening tools' `date_of_birth` input, which decides the youth +rules, so any such wording belongs in the review gate (REVIEWED +officer-facing text, like `GROUNDING_NOTE`), not in plumbing. + +## The staleness watch + +`py -m canlex.refresh` (alias `--check`) diffs upstream: Justice Laws +consolidation dates, CBSA/IRCC delegation instruments, NJC directive dates, +the newest per-memo `LAST_UPDATED` in the D-memo datasource, curated-dataset +age (>180 days → AGING) and curated-vs-chunk regen drift. Non-zero exit = +something drifted. Two coverage guards (2026-09-08) keep the watch honest +about the corpus itself: every `config.SOURCES` instrument must have its +`-fr.json` twin at the same consolidation date (ABSENT/DRIFT rows), and any +`data/processed/*.json` that no watch claims — SOURCES, `_NON_XML`, or a +dedicated check — prints as UNWATCHED and fails; `tests/test_refresh.py` +asserts the same against the real corpus, so a new family cannot land +without its watch entry. +The weekly watch is a **Claude Code scheduled task**, not a Windows task: +`canlex-weekly-staleness-check` (Mondays 03:07 local, cron `0 3 * * 1`; its +prompt is `C:\Users\Luca_\.claude\scheduled-tasks\canlex-weekly-staleness-check\SKILL.md`; +last run 2026-09-21 07:01 UTC). The run executes `refresh --check`, refreshes +drifted sources autonomously (the 2026-08-24 standing instruction), re-embeds, +evals against the frozen slices, re-checks, then does the call-log review +(`loganalyze --days 7 --dataset Beemer0/canlex-call-log data/logs`, report +only). + +**Both routines are hands-free since 2026-09-23** ("I want the Canlex routine +to automatically implement the fixes and updates. I want it hands-free with a +summary of what changed this week"). The weekly watch now COMMITS what it +refreshed -- only the `data/processed` paths it regenerated, named one by one, +never `-A`, never `data/curated`, `canlex/`, the tests or the docs -- and then +DEPLOYS the composed release by the recipe above, ending in +`tools/smoke_space.py`; a failed smoke test rolls the Space back to the +recorded PARENT before anything is reported. The quarterly case-law audit +(`canlex-quarterly-caselaw-audit`) likewise ingests the essential and strong +decisions its reviewers verified at the issuing court and whose text Lexum or +Decisia serves, then commits and deploys them; a CanLII-only decision still +needs the maintainer's browser capture, and a decision's ANALYSIS -- what it +holds, which pairing cites it -- stays curated content nobody automates. +Both stop rather than ship when a gate refuses: a frozen eval slice that +drops, a failing suite, `embed --check` not 0/0, an UNWATCHED row, a rejected +push, the LFS cap in sight, or curated and officer-facing code the maintainer +has not approved (`compose_release` prints "no release produced"; predeploy +FAILs on review or review-code). Neither may pass `--allow-held`, +`--allow-test-failures`, `CANLEX_PREDEPLOY_ALLOW_HELD`, `git push space main` +or a force-push. Each run ends in a summary -- what changed, the counts, the +eval line, the commit and deploy shas with the rollback point, what waits on +the maintainer -- sent as a push notification as well, and the tasks notify on +completion. `Get-ScheduledTask` finds nothing named +canlex because nothing needs to be there -- the 2026-09-21 code audit's +"no task is registered" looked in the wrong scheduler. The run needs the +Claude desktop app on this machine; if it did not fire, the task's run list +(the scheduled-tasks panel) shows the gap. The guidance watch (ENF/PDI/IRB +pages) takes its last-seen time from the cached PDF's mtime (`data/raw/enf`), +and only a chapter with no cache is recorded in +`data/raw/guidance/last-seen.json` -- so that file is normally absent and its +absence means nothing (the 2026-09-21 audit read it as "never seeded"; the +check of 2026-09-21 21:00 reported 41 of 41 chapters and 9 of 9 PDI pages +unchanged). Exit status: since +2026-09-21 a probe ERROR on ANY watch fails the check (exit 1) -- before, +only the legislation rows' errors did, a broken PowerShell probe printed 52 +ERROR rows in the guidance watch and the check passed, and +`check_tariff_edition` read a network failure as "ok" -- so a watch that +cannot reach its source can't pass silently. + +`_PENDING_BILLS` in canlex/refresh.py names bills heading into force that +affect corpus Acts — edit that list when a new one appears (source-code edit, +deliberate). Once its in-force date has passed, the check prints the bill +with, for each affected Act, the consolidation date stored for it +(`refresh.pending_consolidation`, since 2026-09-22): **on or after** the +in-force date means what came into force that day is already in the corpus +and no re-ingest is needed for it; **before** it prints the re-ingest +commands (`py -m canlex.ingest --force ` and its `--fr` twin), which +only help once Justice Laws has consolidated the change (the legislation +rows say STALE when it has). Until 2026-09-22 it printed "run: py -m +canlex.ingest --force C-46" either way. C-16 came into force 2026-07-18 and +the stored Criminal Code is current to 2026-07-21, so the check now says no +re-ingest is needed; the bill stays listed because its coercive-control +provisions come into force later (its `note`) -- delete the entry once they +have and the Criminal Code consolidation covers them. A pending bill never +changes the exit status. + +## The CBSA D-memo index (rebuilt 2026-07-10) + +CBSA replaced the static index with a WET DataTable: the page ships **no memo +links at all**, the rows come from `dmemoranda-datasource-eng.json` beside it. +`dmemo.memo_urls` reads that datasource and falls back to scraping the page +only if it disappears. Two consequences worth knowing: + +- Cancelled memos (`STATUS: "Cancelled"`, 34 of 398) are skipped — retired + guidance must not retrieve as current. D10-14-48 left the corpus this way. +- Eight memos are listed Active but their pages 404 (D8-2-2, D8-9-3, D10-2-3, + D10-14-3, D10-14-18, D10-14-65, D10-15-16, D10-15-25) — CBSA's own broken + links. `dmemo.preserve_failed` re-attaches the last-good chunks of any memo + that fails to fetch, so a 404 (or a dropped connection) can no longer delete + still-in-force guidance. The two D8s were never ingestable and are absent. + +This is the incident the write guard above exists for: the first run after the +rebuild scraped zero memos and wrote an empty dmemos.json straight over 1,870 +good chunks, recovered from git. Every ingester now carries the guard +(`canlex.ciu` since 2026-09-21; it was the one that did not). + +## Curated-dataset review protocol + +The four files `review_state.json` names under `curated_files` -- +`data/curated/us_dispositions.json`, `us_equivalency.json`, +`transborder.json` and `authorities.json` -- are the hand-maintained legal +content (the first two are the ones `canlex.commentary` renders; the +IRPR s. 19(1)(c)-(g) table and the authority registry are read at run time). +To change any of them: edit → bump `"reviewed"` → +`py -m canlex.commentary` → `py -m canlex.embed` → eval → commit with HELD FOR +REVIEW in the message. The change does not deploy until the maintainer +approves it and `data/curated/review_state.json` names the approving commit +(`py tools/predeploy.py --print-review-state `); until then +`tools/compose_release.py` ships the previously approved files. The refresh +check flags any drift between the curated `reviewed` date and the rendered +chunks. Provenance and the verbatim-verification log live inside the JSON +files themselves; keep them current. + +`tests/test_curated_schema.py` validates both the working files and the +approved ones: closed key sets, types, enums, ISO dates, sorted Tran +histories, tie groups of two or more, alias targets that reach a pairing, +and a `verify` row behind every structured maximum. A new field must be added +to its key set there and must stay optional (the approved files lack it). +Its allowlists (dead aliases, 'depends' row verdicts) may only shrink. + +Provenance (roadmap T3.5, since 2026-09-13): a state row may carry `source` +(one object or a list: cite, URL, a quote of at most 300 characters, retrieval +date, and a clearance row's evidence `grade` or a pardon row's quote `check`), +`verified_by`, `unverifiable` (why a row the verification pass could not reach +keeps no verified date), `characterisation` (CanLex's own reading of the +state's law, held for the maintainer) and `open_question` (what decides a +split row); every row prints its provenance tier. A curated alias may be an +object -- `plain`, `jurisdiction`, `cite`, `catchline`, `cite_url`, +`verified`, `note`, and dated readings through `from`/`until`/`dated` +(canlex/screening.py documents them); a plain string stays valid. Every fired +alias prints which statute the code was read as ("STATUTE CODE: ...") and is +carried in CANLEX_RESULT `aliases_read`; a bare section number is not applied +when the record names another state (with no state it applies, labelled +"jurisdiction assumed"); a date-dependent code (the UCMJ articles renumbered on +2019-01-01) is applied only when the offence date chooses its reading, and +otherwise prints every reading and asks for `offence_date`. Ratchets: unverified +clearance rows, unverified state-pardon rows and unverified curated aliases may +only fall. +`tests/data/screening_cases.json` holds whole-record screening fact patterns +with a cited source per expectation; its expected values also await the +maintainer's sign-off, and a case marked `known_defect` flips when the +roadmap item it names is fixed. `tests/gold/disposition_gold.json` rows run +through `_jurisdiction` and `_step1` (disposition roadmap T1.1, since +2026-09-16): a row may carry `country` and expect the step-1 verdict +(`expected_verdict`), the `footing` (conviction / act / both), the governing +state row (`expected_row`) or a two-readings `caveat`; a row marked +`known_defect` runs as its own expected failure, outside the ratchet, until +its fix removes the marker (a fix held in curated JSON adds the held content +to the row's `requires`). `tests/test_step1_footing.py` sweeps every state row +through `canlex_screen_record` for the verdict-to-footing rule. +Record wording is read in record order before any entry sees it +(`screening.DISPOSITION_CUE_TABLE` and `record_order`, disposition roadmap +T1.6, since 2026-09-16): routing rules rewrite the words (a not-guilty plea, a +trial set or a new trial ordered with no outcome after it, a refused relief or +offer, a transfer to adult court, a stayed term, a programme revoked then guilt +adjudicated, a later judgment, no disposition reported), and the question flags +(`screening.QUESTION_FLAGS`) leave step 1 FACT-SPECIFIC with their deciding +question (split counts, sentencing pending, conviction words then a dismissal, +a set-aside with no ground, a penal order not shown final, a dropped matter +brought back, and -- raised by the server from the curated place gates -- a +place-gated name with no place named). CANLEX_RESULT carries +`disposition_flags` (and `question_flag`, `gate_places`). Every rule needs a positive and a +negative fixture in `tests/data/disposition_cues.json` (one table, reused by +T6.2 -- never a second one); `tests/test_disposition_fixtures.py` also checks +that every curated name reaches its own entry, that no folded name sits in two +footing classes, and that accented names match without their accents, with +shrink-only allowlists per baseline. + +## Backups + +The HF Space remote is also production. A full-history bundle lives at +`C:\projects\CanLex-backup.bundle` (recreate: `git bundle create +../CanLex-backup.bundle --all`). For a real offsite mirror, create a private +GitHub repo and: + +``` +git remote add mirror https://github.com//CanLex.git +git push mirror --all +``` + +(GitHub needs git-lfs enabled for embeddings.npz, or add it to a +`.lfsconfig`-excluded push.) + +## Known gaps / parked + +- ENF coverage (researched and URL-verified 2026-07-22): ENF 20 Detention + and the current ENF 3 edition live under non-standard filenames — handled + by `_URL_OVERRIDES` in canlex/enf.py (the old enf03-eng.pdf still serves a + STALE 2015-era copy; never remove the override). ENF 7 was never publicly + released; ENF 13/16/25 never existed; ENF 9 (Judicial Reviews) and ENF 11 + (Verifying Departure) are retired with no successor (Wayback copies exist + but retired guidance is deliberately not ingested). ENF 27 (PR Card) was + replaced by the PDI section at + operational-bulletins-manuals/permanent-residence/card.html — ingested by + `canlex.pdi` (the `pdi-prcard` set). ENF 2 (Evaluating Inadmissibility) + and ENF 14/OP 19 (Criminal Rehabilitation) exist publicly only as + ATIP-released copies hosted by a private law firm — NOT ingested pending + the maintainer's call on unofficial sources; the Tran PDI page now comes + in with the whole `standard-requirements` tree and partially covers + ENF 2's ground. +- PDI scope: `canlex.pdi` crawls six category trees breadth-first + (`_MAX_DEPTH` 2, `_MAX_PAGES` 160 per tree). Deliberately out of scope: + citizenship, fees, office service delivery, and the year-by-year + operational-bulletin archives (mostly expired instructions). If a tree + ever hits the page cap the run PRINTS how many in-scope pages it dropped + — raise `_MAX_PAGES` rather than letting the silence read as coverage. +- CanLII capture workflow (used for the Hill/Brannson/Li trilogy, + 2026-07-22): open the URL a missing-file stub prints in a real browser, + copy the page text to the named `data/raw/caselaw/*.txt`, re-run + `py -m canlex.caselaw && py -m canlex.embed`. +- Dag (FCA) parses to 2 chunks because it genuinely is a 9-paragraph bench + judgment — correct, not a bug. +- A weekly citator sweep over the ~245 corpus decisions (negative-treatment + detection) was considered and deliberately skipped: ~13 minutes of + rate-limited CanLII API per run (an estimate, extrapolated from the ~6 + minutes measured at the 2026-07 count of ~110 decisions) for marginal + signal. Run `canlex_case` on a decision before relying on it instead. +- French support is now phase 3: the full parallel French legislation corpus + is ingested (`--fr`; lang='fr' chunks, '-fr-' id infix, 'art.'/'annexe' + citations). Retrieval is language-routed: English queries never see French + chunks; French queries (detected by accents or French function words) + search both languages. get_section/xref/list_acts stay English-canonical. + Phase 2 (a multilingual embedder) was tried and REJECTED 2026-07-22 — + regressed every slice and broke confidence calibration; don't retry + without re-calibrating HEDGE_THRESHOLD. French guidance (D-memos are + published bilingually) is the remaining bilingual frontier. +- ENF 2 and ENF 14/OP 19 are both ingested from their ATIP releases + (authorized 2026-07-23, bannered UNOFFICIAL on every chunk). ENF 14's + image-scan PDF was recovered by OCR (pymupdf rasterization + Windows OCR; + cached at data/raw/enf/enf14-ocr.txt — enf.py auto-falls-back to + `*-ocr.txt` when a PDF extracts under 500 chars). +- Typo robustness, two layers: unknown STEMS get the trigram bridge in + `index._fuzzy_term` (BM25 recall only), and misspelled WORDS are repaired + by `index._correct_query` (trigram candidates, Damerau-Levenshtein <= 2, + shared 2-char prefix, frequency floor 5 so OCR debris can't win) -- the + repaired text feeds the semantic embedder and the reranker. Ultra-rare + stems (df <= 2) get the correction APPENDED, never substituted, so rare + legitimate terms keep their recall. The `typo` eval slice tracks it. +- Employer policy (doc_type 'policy', `canlex.conduct`, 2026-09-08): the CBSA + Code of Conduct (cbsa-asfc.gc.ca, 86 sections incl. the 16 expected + standards and the conflict-of-interest chapter) and the Values and Ethics + Code for the Public Sector (tbs-sct.canada.ca policy suite, 21 sections). + The TBS host rejects some clients; `fetch_cached`'s browser UA over urllib + is accepted (verified 2026-09-08), and the page wraps each chapter in +

, which the parser flattens. Both pages carry a + dateModified stamp that `refresh.check_policy` watches. The TBS Directive + on Terms and Conditions of Employment (id=15772; appendix headings sit in +

, also flattened) joined them the same day, so the FB + agreement's pay-administration references resolve. Still absent: the CBSA + discipline/attestation directives, which are intranet-only. +- 2026-09-09 expansion (all watched by the weekly check through SOURCES, + `_NON_XML` or `check_policy`): 15 Justice Laws instruments (FPSLR + Regulations, PSER, FAA, PSDPA, OLA, Interpretation Act, Criminal Records + Act, YCJA, SEMA, UN Act, JVCFOA, Preclearance in Canada Regs, Passenger + Information Regs, Reporting of Exported Goods Regs, SOR/98-209), EN + FR; + IRB Chairperson's Guidelines 4, 7 and 9; the IRCC PDI permanent-residence + tree (`pdi-pr`, excluding the PR-card subtree and the economic classes); + seven Treasury Board people-management instruments EN + FR (discipline, + duty to accommodate, performance management, harassment and violence, + leave and special working arrangements, government security, security + screening); CBSA detention policy as doc_type 'memorandum' (National + Immigration Detention Framework, the minors directive, the six-chapter + Detention Standards, the 2017 Ministerial Direction on minors); the Safe + Third Country Agreement and its 2023 Additional Protocol as doc_type + 'legislation' (a treaty, named as such; in-force dates pinned in DOCS + because the canada.ca pages carry no dateModified); and French editions + of the CBSA Code, the FB collective agreement (`canlex.agreement --fr`, + agreements-fr.json) and the ENF chapters (`canlex.enf --fr`, enf-fr.json; + ENF 2/3/14/20 have no verified French URL and skip). The TBS host accepts + the ingesters' browser UA; the CBSA detention pages and publicsafety.gc.ca + answer plain urllib; the STCA pages need the PowerShell fetch. +- Bilingual guidance: the D-memos and NJC directives now carry their French + editions (`--fr` modes; dmemos-fr.json / directives-fr.json) alongside + the French legislation, all lang-routed. Remaining English-only: case + law, ENF/AMPS/PDI, benefits booklets, and the curated commentary. + +## Two retrieval invariants worth not re-breaking (2026-07-29) + +Ingesting the French guidance and the wider PDI trees exposed both of these. +Each was measured question-by-question against the previous corpus, not +guessed at — the harness is a per-question rank dump over two corpus +variants, which is how a 5-question shuffle gets attributed instead of +argued about. It lives in the repo as `tools/rankdump.py`: run it from the +repo root for the current tree, and from a `git worktree add ../CanLex-prev +` checkout for the baseline (`py tools/rankdump.py +data/eval/questions.json ` in each; the script imports the +`canlex` of its working directory), then diff the two JSON files. Use it +for any corpus change above ~5% or any ranking-knob change: the summary +metrics round away single-question moves. + +- **BM25 collection statistics are language-scoped.** `_build_bm25` keeps + `idf`/`avgdl` (whole corpus) AND `idf_en`/`avgdl_en` (English scope); + every English query scores with the English pair. An English query can + never retrieve the French twins, but a single corpus-wide idf let those + unreachable chunks set the rarity of every shared token (numbers, act + codes, folded cognates) and drag the average length: adding the French + D-memos and directives alone moved legacy Hit@1 0.767 → 0.750 without a + single French chunk being retrievable. French queries still use the + whole-corpus pair — they search both languages. +- **The diversity cap keys memoranda on the parent DOCUMENT.** `_source_key` + used `section` for every memorandum, which is the document identity only + for the flat families (`_FLAT_MEMO_CODES`: one act_code across ~360 + D-memos / ~1,000 AMPS contraventions). ENF chapters, PDI pages and IRB + guidelines carry a per-document act_code and use `section` to subdivide + it, so each ENF section and each PDI page was its own "source" and + neither RECALL_CAP nor SOURCE_CAP ever bound: the ~150-page PDI refugee + tree filled the candidate pool and pushed IRPA s. 112 out of the top 20 + of "what is a pre-removal risk assessment" entirely. If a new corpus + family lands, check which field carries its document identity before + trusting the caps. +- **The rerank pool is source-capped (`POOL_CAP`, 2026-09-06).** The + cross-encoder only ever sees `RERANK_POOL` (16) fused candidates, and + `RECALL_CAP` (20) let one deep decision take 14 of those slots: on "can an + officer search a traveller's cell phone at the port of entry without a + warrant" the pool was 14 Fearon paragraphs plus two unrelated "without + warrant" marginal notes, while Canfield, Pike and ENF 12 s. 6.15 sat at + fused ranks 19-32 and were never scored. `_fill_pool` now admits at most + `POOL_CAP` (8; env `CANLEX_POOL_CAP` -- the value was never 5, the commit + that introduced it set 8) chunks per capped source while filling the pool; `_diversify` + still governs display afterwards. Eval-gated: no slice moved. The + device-vocabulary bridge in synonyms.py is the other half of that fix. +- **Some law in the corpus does not bind its audience (`_INAPPLICABLE`).** + Canada Labour Code Part III excludes federal departments (s. 167(1)(d)), + and FPSLRA Part 2.1 is RCMP-only, yet both out-BM25'd the FB agreement on + steward questions (CLC s. 173.1 over FB art. 25.21 for shift-change + notice). Those chunks take `INAPPLICABLE_PENALTY` in the fusion and carry + an APPLICABILITY caveat whenever they are rendered. Extend the table when a + new instrument with a public-service carve-out lands. +- **Guidance that PARAPHRASES a provision needs a bigger hierarchy penalty + than guidance that quotes it (`MEMO_PENALTY` refit 0.004 -> 0.008, + 2026-09-10).** IRCC's rehabilitation guide (Guide 5312) and the IRB Legal + Services papers joined the corpus and immediately took rank 1 on four + rehabilitation questions whose gold is IRPR s. 17 or s. 18 -- the guide + answers the question in the reader's own words, so it out-matches the + provision's statutory phrasing by more than the old penalty could bridge + (0.0115 versus 0.004). Swept 0.006 / 0.008 / 0.010 on the full set: 0.008 + restores ALL Hit@1 and ADDS four to Hit@3 (legacy Hit@3 0.95 -> 0.96, + holdout 0.93 -> 0.95); 0.010 costs the French slice (0.58 -> 0.42) and + Hit@5. Expect to re-sweep whenever a new plain-language guidance family + lands -- the knob is fitted to the guidance mix, not to a constant. + +- **The Income Tax Act (2026-09-10) and what a huge, off-domain statute does + to retrieval.** Added so the tax-evasion pairing verifies against text + (s. 239(2)(b): 5 years on indictment; s. 238: summary only, 12 months). + It is the largest instrument in the corpus -- 4,691 English sections and + 5,006 French, +17% -- and it moved the eval in three measurable ways: + (1) its vocabulary ('allowance', 'employee', 'benefit', 'officer', + 'report') lowered the rarity of words border and labour questions depend + on, shuffling four near-ties with no ITA section anywhere in their top 5; + (2) `_ensure_primary` promoted ITA s. 231.3 ('Search warrant' -- a CRA + audit power) to rank 2 of the device-search question on its title alone, + now prevented by `_PROMOTION_GATES` (the Act is promoted only for tax + queries); (3) 'evasion' and the Act's 'evaded' stem apart and s. 239 is + titled 'Other offences and punishment', so a synonym bridge supplies its + wording. Even then the Luxury Tax Act s. 122 and the Excise Act -- the same + evasion boilerplate -- outranked s. 239 on "penalty for tax evasion under + the Income Tax Act". **`ACT_HINT`** (the court hint's twin: a query that + names an instrument boosts it) fixed that. Swept 0.004 to 0.016: every + weight beat no hint on every slice; 0.012 set, lifting ALL Hit@1 from 212 + to 215 over the pre-ITA baseline. 0.016 was ~3 questions better still, so + the ceiling is unmeasured -- sweep 0.016 / 0.020 / 0.024 next time. + Re-embedding the Act takes ~20 minutes, and almost every federal budget + amends it: expect it to be the most frequent drift in the weekly watch. + +- **A query that names the deciding court gets that court's decisions + (`COURT_HINT`, 2026-09-09).** "What did the Ontario Court of Appeal decide + about suspicionless device searches" surfaced Canfield (ABCA) above Pike + (ONCA) on a fusion near-tie after the corpus grew. `court_hints()` in + index.py maps the ways a court is written (full name, "ONCA", "FCA", + "Federal Court" meaning the trial court, the Board names) to a predicate + over the case-law chunk's `part`, and matching chunks take `COURT_HINT` + (0.004, the same scale as the hierarchy penalties) in the fusion. The hint + fires only beside a decision word (decided, held, ruled, case, ...): "what + is the FPSLREB" names the Board as an institution and wants the Act that + creates it, and the first cut pushed that gold from 2 to 4. Add a + row to `_COURT_HINTS` when a new court joins the corpus; the `part` values + in use are listed by grepping the processed case-law files. The `court=` + filter now also accepts the neutral-citation abbreviation ("ONCA") as well + as a name fragment. When the hint fires, `_ensure_primary` is skipped: the + decision is the governing source the user asked for, and the guarantee + was pulling the CBSA Code of Conduct's "Use of Personal Electronic Devices + at Work" into rank 2 on the Pike question (policy joined + PRIMARY_DOC_TYPES on 2026-09-08 and its section titles match traveller + questions by accident). Eval-gated: no slice moved. +- **Policy instruments are promoted by `_ensure_primary` only on + employment queries (2026-09-09).** A diagnostic over the eval set + (`py tools/diag_policy_promote.py`: any policy chunk in the top + 5, with its fused score as a ratio of the #1 hit) found the CBSA Code of + Conduct in the top 5 of 67 of 297 questions, most at 15-30% of the top + score -- cultural property, customs valuation, Fearon, Chan, Elk Valley. + Excluding policy outright cut that to 38 (all naturally ranked) but + dropped the Security Screening Directive from 2 to 8 on the + reliability-status question, where it IS the governing instrument. So + `policy` stays in PRIMARY_DOC_TYPES (uncapped as a source, primary + banner) and the promotion admits it only when `employment_query()` sees + employment vocabulary in the query (employee, discipline, grievance, + reliability status, social media, code de conduite...). Extend the word + list when a new employer instrument lands; re-run the diagnostic when a + new primary family lands -- a governing-instrument guarantee only helps + if the candidate pool is actually governing law. + +## The s. 36 screening chain (2026-09-09, rebuilt after the 2026-09-10 audit) + +- **The record's own words settle step 1.** `_match_dispositions` matches a + curated entry's names as PHRASES first (longest wins, so 'not guilty' beats + 'guilty' and 'adjudication withheld' beats 'probation'), then by content-token + overlap; `screening.disposition_cues` reads the string for conviction words + ('convicted', 'pled guilty', 'sentenced') and for non-conviction phrases, and + a curated `conviction` entry carries the plain case. Before this, 'convicted' + matched nothing and 'guilty' matched the dismissed entry through its 'not + guilty' name -- and `canlex_screen_record` then ran the whole memo on the ACT + branch, clocking rehabilitation from the offence date. When the verdict is + still unknown or fact-specific the memo computes BOTH footings rather than + choosing one. +- **A pairing whose branch depends on a fact never asserts a band.** + `_resolve_pairing` emits `max_penalty_lt_10: null` when the maximum is + unknown, and resolves it when the record supplies the fact: a named substance + against the pairing's `schedule_max`, `value_over_5000` against `value_max`. + Schedule IV possession resolves to 'no offence' (CDSA s. 4(1) reaches + Schedules I-III only). +- **Ties are reported, never resolved.** Two pairings can match one label + ('larceny' until the value is known; 'evading arrest' until it is clear + whether the flight was on foot or in a vehicle). Those declare a shared + `tie_group`; `screening.is_tie` detects the tie and the memo prints both + pairings with a notice. Phrase matches outrank token-coverage matches + (`TOKEN_CAP`), so 'sexual battery' beats 'battery' and a state-qualified + degree label ('assault in the third degree' + NY) beats the generic token + match. `tests/test_curated_penalties.py` fails any term shared WITHOUT a + tie_group. +- **s. 463 attempts change the number, not just the prose.** `attempt_max` + implements s. 463: life -> 14, 14-or-less -> half, summary stays summary. The + structured result and the branch carry the adjusted maximum. +- **s. 36(2) reaches foreign nationals only, including its two-offence limb.** + `branch_for` collapses every sub-10-year outcome to `not-inadmissible-pr` for + a permanent resident, and `rehab.assess` does the same regardless of count. +- **The two clocks differ by a day.** IRPR s. 18(2) says 'the day after the + completion'; s. 17 says 'after the completion'. `rehab.assess` runs the + deeming test from the day after and the application test from completion + itself -- IRCC's own Guide 5312 (in the corpus) works its examples that way. +- **The memo ends with what the officer does next**: the allegation in ENF 5 + wording for the paragraph the branch names, the Hill comparison instruction, + the s. 44(1) / ENF 6 route, the TRP and rehabilitation fees READ FROM the + IRPR text (`_fee`, so a fee change in the corpus flows through), s. 52, s. 42, + s. 36(3)(e) and the foreign-pardon check. +- **A multi-entry record goes to `canlex_screen_history`**, which derives the + s. 18(2) count from the entries that actually engage a branch and runs the + s. 17 clock from the most recent sentence completion. `canlex_screen_record` + refuses a string that names several offences and says so. + +## Transborder criminality and s. 37 (2026-09-12, roadmap T2.1 / T2.5; HELD FOR REVIEW) + +- **`canlex_transborder`** screens an offence committed ON ENTERING Canada + under IRPA s. 36(2.1) (s. 36(2)(d) is repealed, 2023, c. 32, s. 53). IRPR + s. 19(1)(a)/(b) -- any indictable Criminal Code / IRPA offence, hybrids + deemed indictable by s. 19(2) -- are open classes, so `canlex/transborder.py` + applies them as a rule on the offence's mode (given by the officer, or read + from the curated pairing's `hybrid`/`summary_only` flags for an offence + named in words). Only s. 19(1)(c)-(g) are tabulated, in + `data/curated/transborder.json` (HELD: keyed by Act, the provision + contravened and, for the Customs Act, the goods class; `prescribed: false` + rows record the traps -- s. 12 cannabis, Cannabis Act s. 12). A composed + release ships without the file: the (a)/(b) rules still run and the tool + says the (c)-(g) table awaits review (branch `table-held`). +- The removal path is a code rule pinned to the Regulations: the delegate's + deportation order for the IRPR s. 228(1)(a.01)(i)-(v) offences + (`MD_ORDERABLE`; tests/test_transborder.py re-reads the list from the R228 + text), otherwise the Immigration Division under s. 229(1)(d.1). No + rehabilitation (s. 36(3)(c) lists only (1)(b),(c) and (2)(b),(c)); a + permanent resident is not reached; the s. 36(3)(e)(i) contraventions + carve-out is checked against the SOR-96-313 schedules in the corpus (one + Act per schedule, read from the schedule's first chunk: 25 live schedules at + the 2026-02-04 consolidation, none for any of the six Acts -- the test fails + first if a refresh adds one). s. 36(3)(b)'s acquittal words attach to 'a + conviction ... in respect of which' and s. 36(2.1) needs no conviction, so + the memo prints both readings of a final acquittal (`ACQUITTAL_NOTE`, NOT + SETTLED). tests/test_curated_penalties.py checks every table phrase and that + the table's delegate flags follow `MD_ORDERABLE`. +- Guidance that still says 'A36(2)(d)' (ENF 1 s. 3, ENF 2 ATIP, ENF 3 + s. 10.8) gets an AMENDED LAW note where a search result shows it + (`AMENDED_REFERENCES`); the ENF 2/3 'present tense' limit is printed as + historical and NOT SETTLED under s. 36(2.1). +- **s. 37 also to consider.** An optional curated `also_engages` on a pairing + (HELD) and the optional `organization` fact on `canlex_screen_record` print + a block whatever the s. 36 branch, permanent residents included: the + s. 37(1)(a)/(b) elements to establish, no conviction needed (s. 33), no + rehabilitation, s. 42.1 relief for foreign nationals only, no IAD appeal + (s. 64(1)), the s. 42(2) family reach and the ID referral (R229(1)(e)), + citing Sittampalam (caselaw fca-35313) and B010 (scc-15647). It always reads + 'consider -- not engaged'. `canlex_us_equivalency` prints a one-line + pointer; `canlex_screen_history` one block for the record. + +## Canadian convictions, youth and finality (2026-09-12, roadmap T2.2 / T2.6 / T2.7; HELD FOR REVIEW) + +- **A Canadian entry is screened, not refused.** `canlex_screen_record` (country + 'Canada' or a province as the state) and `canlex_screen_history` (an entry + with jurisdiction 'canada', country 'Canada' or a province) run + `rehab.assess_canadian`: s. 36(1)(a) (the 10-year and the six-month limbs, for + anyone), s. 36(2)(a) (foreign nationals), IRPR s. 18.1, the record suspension + as the only cure, the removal path (IRPR s. 228(1)(a) delegate for a foreign + national, s. 229(1)(c) Immigration Division for a permanent resident) and the + s. 64(2) appeal line ('at least six months'). It reuses the T1.8 rules in + `_assess_canada` (now tagged with the limb applied), never restates them. + Before them: a discharge is not a conviction (C-46 s. 730(3)); a + contravention and a youth sentence are outside s. 36 (s. 36(3)(e)); a record + suspension that stands bars it (s. 36(3)(b)); a conditional sentence is no + custody (Tran); an intermittent sentence is NOT settled (both readings). The + maximum and mode come from the officer (`canada_max_years`, + `canada_offence_mode`) or the curated pairing's maximum at the offence date + and flags; where the maximum, mode or custody is missing and decides the + answer, every reading is printed (branch 'depends', canada_branch + 'not-settled') and `needs` names the input. In screen_history the + two-offence limb and s. 18.1 count Canadian convictions only; a Canadian + conviction is a 'subsequent offence' for the s. 17 clock and defeats the + foreign deeming answer (limb-by-limb is T2.3). +- **Youth (T2.6).** `date_of_birth` / `age_at_offence` (screen_record; screen_history + top level and per entry): 12-17 at the offence prints a YOUTH block for any + disposition, STATUTE (s. 36(3)(e): Canadian YOA/YCJA dispositions only) and + POLICY (ENF 14 ss. 12.1/24, withdrawn 2008 ATIP copy citing the repealed YOA; + no verdict) kept apart; under 12 prints Criminal Code s. 13. + CANLEX_RESULT carries `youth_screen` (screen_history: `youth_entries`). +- **Finality (T2.7).** A disposition entry may carry `rehab_clock: + 'not-started-unless-served'` (the held `conviction-in-absentia` entry): + `_clocks` and screen_history then start no conviction clock from the + completion date given and say why (ENF 14 s. 16), printing the 'if served' + reading only as a condition. The held batch also moves conviction-under-appeal + to likely-yes and adds `conviction-reversed-on-appeal` (NO). On the approved + baseline none of this data exists and the tools behave as before. +- **Audit of the step (re-run).** (1) An answer every reading settles on the + same ground is not 'not-settled': more than six months imposed with no + maximum given is s. 36(1)(a) on the six-month limb whatever the maximum, and + an intermittent term beside a 10-year maximum is s. 36(1)(a) on the 10-year + limb. `canada_limbs` holds the limbs common to every reading, rehab's + `either_way` lists the readings, the memo prints SETTLED EITHER WAY, and a + permanent resident's IAD appeal is 'depends' where the readings differ on it. + An intermittent term over six months has no `custody_counted` figure. + (2) Canadian disposition words: 'appeal dismissed' or 'conviction affirmed on + appeal' is a conviction and not a pending appeal; acquittal, withdrawal, stay + or dismissal words beside conviction words are a conviction when they name + other counts ('remaining charges', 'count 2') and unsettled otherwise; 'appeal + allowed' is unsettled (it was read as no conviction, or as a pending appeal). + (3) The foreign chain reads the same decided-appeal phrases the same way + (`screening.strip_appeal_upheld`, in `_cue_text`): 'convicted, appeal + dismissed' no longer ranks the dismissed entry first, on either baseline. + (4) The held in-absentia entry also cites IRB ROA ch. 8 s. 10 (Timis: the + Minister proceeded on s. 36(1)(c)) and s. 15 (Drake: the conviction was later + vacated). +- **Re-render**: the two new disposition entries render into commentary.json: + re-render and re-embed before a release that approves them. + +## The whole record, IRPR s. 18(2) limb by limb (2026-09-12, roadmap T2.3; HELD FOR REVIEW) + +- **`rehab.assess_record(record)` and `rehab.check_limbs(paragraph, record)`** + test every limb of IRPR s. 18(2)(a)-(c) and s. 18.1 against the record's + entries: pass / fail / unknown, the entries that decide each limb, the date it + starts or stops holding, and its IRPR chunk id (SOR-2002-227-s18-p1, -p2, -p3, + -s18.1). deemed_on is the first day every limb holds at once -- the latest + start, unless a limb stops first ((vi) with two old summary convictions). + `canlex_screen_history` builds the record from its rows (`_record_entries`: + conviction / act / unsettled verdict, the classes each equivalent can be, the + dates) and prints one limb table per paragraph whose chapeau can hold. The old + `clean = count != one or len(engaged) == 1` collapse and the interim 'a + Canadian conviction defeats deeming' note are gone. `assess()` and + canlex_rehabilitation are unchanged (a new `solely` kwarg defaults to None). +- **Two readings are not settled** (roadmap DECISIONS (a), (b)) and each is run: + a hybrid equivalent as also a 'summary conviction offence' in (iv)/(vi), or + not; 'within / before the last N years' on the conviction date or the offence + date. A conviction leaves the window on the day N years have elapsed since the + day after it. When the readings give different answers the status is + `not-settled` and every reading is printed with its own table (CANLEX_RESULT + `s18_readings`; `s18` is null); when only the date differs, + `s18_dates_differ`. A missing offence date leaves the offence-date reading + open wherever a conviction is still inside a window. +- **Open facts are run every way they can go**: a missing conviction date + (between the offence and the completion -- the new optional entry key + `conviction_date`; the dates shown are the latest they can be, with a + warning), an unresolved equivalent, an unsettled verdict, a Canadian offence's + mode or conviction and -- not settled in CanLex -- a record-suspended Canadian + conviction or a youth disposition in a limb with no YCJA exception. A fact that + decides a limb makes it UNKNOWN and is named in `needs`; the status is + `undetermined`. A Canadian contravention is outside every limb. +- **s. 18.1 'solely'** is read from the other entries, and the Canadian answers + follow it: two summary-only Canadian convictions beside a foreign ground are + s. 36(2)(a) two offences, not EXEMPT. +- **Fixtures**: tests/data/record_cases.json (hand-worked records, pending the + maintainer's sign-off, the arithmetic in each case, no in-test oracle); + tests/test_record_limbs.py pins every quoted phrase; the engine sweep runs a + whole-record sweep on both curated baselines. +- **Audit of the step (re-run, 2026-09-12; HELD FOR REVIEW)**: + - *A Canadian hybrid conviction in (iii) / (b)(ii)* was always read as 'an + indictable offence'. On the 'summary' reading of roadmap DECISION (a) + (IRPA s. 36(3)(a) governs only s. 36), the Interpretation Act applies to the + Regulations 'unless a contrary intention appears' (I-21 s. 3(1)), and + s. 34(1)(c) says a person convicted of a hybrid on summary conviction is not + 'convicted of an indictable offence'. So on that reading the Crown's election + decides: new optional entry key `canada_prosecuted` ('summarily' | + 'indictment'; engine key `prosecuted`), asked in `needs` where it decides. + s. 18.1 'solely' and the Canadian entry's own ground still read s. 36(3)(a) + (IRPA s. 36). + - *Permanent residents*: assess_record no longer clears a PR + ('not-inadmissible-pr') when a foreign entry's maximum, equivalent or youth + status is open (now 'undetermined' with needs), nor when an entry is serious + whether a conviction or an act. canlex_screen_history was not affected (it + reaches the engine for a PR only with a serious entry). + - *One occurrence*: assess_record reads IRPA s. 36(2)(b)'s 'not arising out of + a single occurrence' for two summary-equivalent convictions (nothing to + rehabilitate on one occurrence; an open occurrence is warned and asked). + - *Wording*: the youth / record-suspension questions read as questions; the + youth note names every limb without the YCJA exception (checked against the + corpus: `YCJA_EXCEPTION_LIMBS`); a foreign youth disposition excepted by + analogy in (a)(v), (b)(iv), (c)(v) is now flagged; a record with too many + open facts prints no empty table (and no s. 18.1 table without a Canadian + entry); with no deeming and no s. 17 date the status is 'undetermined', not + 'not-yet'. + +## Foreign pardons, element resolvers and the gold misses (2026-09-12, roadmap T2.8 / T2.10; HELD FOR REVIEW) + +- **Relief by effect (T2.8)**: five new disposition entries in + us_dispositions.json -- `foreign-rehabilitation-law` (spent convictions: the + UK Act per Barnett, the Hong Kong ordinance per Lui; fact-specific), + `us-presidential-pardon` and `commutation` (likely-yes, CanLex's own reading), + `foreign-pardon-other` (indulto, amnesty, royal pardon; fact-specific; a + remission-only pardon stays a conviction -- inferred) and + `discharge-no-conviction-recorded` (Burgon; fact-specific). New aliases: 'def + pros' (pretrial-diversion), 'charge(s) refused', 'prosecution declined', 'not + prosecuted' (dismissed-nolle). The matcher reads a name that is itself a + cancelled-conviction phrase ('no conviction recorded', 'spent conviction') as + a phrase on the record's words; a name whose only word is 'refused' matches + as a phrase only ('pardon refused' is never a dismissal); and for a NON-US + record the US-framed relief entries yield to their counterparts + (`server._FOREIGN_RELIEF`: state / presidential pardon -> foreign-pardon-other, + conditional-discharge -> discharge-no-conviction-recorded). +- **The Saini checklist**: `screening.saini_checklist(entry)` prints Saini + para 24's three elements, motivating considerations and seriousness, each + quoted from its chunk (`SAINI_QUOTES`: fca-31531-*, enf-3-10-9; sources also + irb-ls-roa8-16..18, enf-14-27), for an entry that carries the optional + curated `pardon_test` (its note under each item). state-pardon and the five + new entries carry it, so the approved baseline prints no checklist. + canlex_us_disposition, screen_record step 1 and the commentary renderer show + it; CANLEX_RESULT carries `saini_checklist`. +- **Element resolvers (T2.10)**: optional curated `resolvers` on + domestic-violence (weapon / bodily harm / choking -> s. 267, none -> s. 266, + not an assault -> nothing), identity-theft (used -> s. 403, obtained or + possessed -> s. 402.2) and weapons-possession (s. 95 / s. 92 / ss. 90-91; + ss. 117.01 and 88 listed but not resolved -- no point-in-time-checked + maximum yet). Each resolving outcome is one of the pairing's own verify rows; + each Canadian element is a verbatim fragment with its chunk id + (tests/test_pardons_resolvers.py finds it). `screening.resolve_elements` reads + an explicit answer in the officer's facts ('harm-weapon-choking: none'), else + the element's own words in the offence text or the facts ('weapon', 'bodily + harm', 'choked'), never when negated or doubted ('no weapon', 'did not + choke', 'alleged'). An element in force only from a later date (s. 267(c), + 2019-09-19, pinned by the point-in-time fixtures; s. 402.2, 2010-01-08) is + applied only when the offence date is given and on or after it. Otherwise the + branch stays 'depends' and CANLEX_RESULT `needs` names the fact (parameter + 'facts'; canlex_us_equivalency says tool 'canlex_screen_record'), the question + and every option with its maximum and branch. A resolver never yields 'no + equivalent'; sexual-offences-child may carry none until the maintainer signs + off (`RESOLVER_SIGN_OFF` in tests/test_curated_schema.py). canlex_screen_history + reads each entry's `facts` too (no new parameter anywhere). +- **Gold misses and dead aliases**: code -- singular/plural matching + (`_singular`), 'possesion' / 'posession' (never Spanish 'posesion'), 'sm amt', + Texas 'PG N' dropped from a drug charge with a note, an amount dropped from + any drug charge, 'PC 664/NNN' read as an attempt of section NNN (is_attempt + reads the statute code too). Curated -- terms for OWI, WA theft 3, trafficking + in a controlled substance (with a Florida weight-trafficking caveat), a small + amount of marijuana, resisting without violence (and 'with violence' + negatives), criminal threats, dealing in stolen property (with an s. 355.2 / + 355.5 caveat), taking a motor vehicle without permission, the Texas DPS + 'assault causes bodily injury' (the same element split, term note alias); + 'ipc 498a' -> domestic violence, 'pc 459.5' -> shoplifting, 'pc 664' removed, + 'para venta' negatives on drug-possession. Dead aliases left: 18 USC 1708 + (needs an s. 356 pairing) and 31 USC 5324 (no equivalent named). The gold + baseline was ratcheted (working 155/155 offence, 94/94 disposition; approved + 112/121, 64/66). + +## Inchoate and party liability, and curated text hygiene (2026-09-13, roadmap T2.4; HELD FOR REVIEW) + +- **Inchoate and party liability (T2.4)**: `screening.inchoate(offense)` reads + the kind (attempt, accessory, counselling, conspiracy, party), the cue words + and the object offence from the record's words. `screening.inchoate_match` + matches the OBJECT on its own, and step 2 (`server._step2`, so all three + tools) resolves that pairing with the Criminal Code rule applied to its + maximum (`screening.inchoate_max`): s. 463(a)-(d) for an attempt and an + accessory after the fact; s. 464(a)/(b) for counselling an offence that is + not committed (the attempt punishment); s. 465(1)(c)/(d) for a conspiracy + (the object's punishment); ss. 21(1)/22(1) for a party. `EXPRESS_EXCEPTIONS` + gives life where the object is murder -- an attempt (s. 239), an accessory + after the fact (s. 240), a conspiracy (s. 465(1)(a)), counselling (s. 464(a) + through s. 239) -- keyed on a curated limb's `provision` ('C-46 s. 235'), + never on the foreign word. Every phrase is pinned (`INCHOATE_PINS`, + tests/test_inchoate.py). A pairing term that names a complete offence with + the cue in it ('solicitation of prostitution', 'solicitation of a minor', + 'inciting a riot', 'attempt to evade or defeat tax') is not read as inchoate. +- **Two readings**: counselling (the offence committed or not: s. 22(1) or + s. 464) and a bare 'accessory' (before or after the fact) are resolved both + ways; readings with different branches give 'limb-open' and an + `inchoate-role` need, answered in the facts (`inchoate-role: + not-committed`). CANLEX_RESULT carries `inchoate` {kind, cue, object, + readings (rule, section, maximum, branch, exception), answered, open}; + `attempt` stays (kind attempt). A direct `_resolve_pairing` call (no step-2 + reading) reads the record against that pairing alone. +- **Held curated**: homicide gains limbs `murder` (provision C-46 s. 235) and + `other-homicide`, verify rows for ss. 235, 239, 240, 464 and 465(1)(a), penalty + prose and a caveat; prostitution gains murder / kill / homicide negatives; + party-accessory gains caveats (the exceptions; US solicitation statutes that + are complete offences; a bare accessory) and verify rows (ss. 464, 240); + conspiracy a verify row (s. 465(1)(a)). The limb key `provision` is optional + (schema). ss. 235, 240 and 464 are unamended since before the 2003 + point-in-time floor (corpus history); s. 239 was amended in 2008 and 2009 and + its point-in-time walk is pending. On the approved baseline the object is + still matched (code) but no exception applies: attempted murder is 14 years + (s. 463(a)), still serious criminality, and the memo's EXPRESS EXCEPTIONS + note says so. +- **Matcher**: a curated term also matches with its abbreviations expanded + (`_best_match`): the record's 'attempt' is read as 'attempted', so the term + 'attempt to evade or defeat tax' was unreachable. +- **Curated text hygiene (held)**: the six '[Corrected ...]' compiler markers on + deferred-adjudication rows (MI, MS, NV, SD, TX, WY; all verified 2026-07-15) + are plain statements now, so commentary prints no OPEN QUESTION line for + them; the conviction methodology quotes ENF 2 s. 15.2 (enf-2-15-2-p1/-p2, a + withdrawn ATIP copy) and ENF 3 s. 10.9 instead of 'cited here by name only'; + `_state_survey` states the five verification rounds instead of 'PENDING' and + documents the seven legacy 'depends' rows (not converted; allowlist in + tests/test_curated_schema.py). +- **Re-render**: us_equivalency.json and us_dispositions.json changed -- re-render + commentary.json and re-embed (main loop). + +## Read next and the ENF 2 s. 3 policy caveat (2026-09-15, roadmap T3.6; HELD FOR REVIEW) + +- **see_also**: every disposition entry, both methodology items, every + pairing and every transborder row carries an optional `see_also` -- the + official guidance and statute to read next, as STABLE keys + (`|s.
` for a statute, `|`, + `|` for IRB papers, PDI pages and Guide 5312, + `|ENF 1 App. A|`), the chunk each one quotes and an + optional verbatim quote (150 characters at most). Seeded from + `data/curated/drafts/see-also-draft-2026-09-12.json`, re-run on the current + tree (all 145 keys and 294 items still resolve), plus the seven entries the + draft predates (from their own guidance references), YCJA s. 119 on the + juvenile entry (s. 119(9)(a) limits the s. 82 deeming it already names), + IRPA s. 37 on the pairings with `also_engages`, and each transborder row's + own sections. The draft's prose 'why' lines are not carried: the printed + 'why' is the source's own words. Official sources come before the unofficial + ATIP copies (ENF 2, ENF 14), which are flagged. +- **Rendering** (`canlex/readnext.py`): canlex_us_disposition and + canlex_us_equivalency print a READ NEXT block (corpus label, chunk id, + status, quote; a statute-only pairing adds the framework's items); + canlex_screen_record prints a line per step; canlex_transborder the matched + row's sections. CANLEX_RESULT carries `read_next`. canlex/index.py + co-surfaces the same items after search results (at most 3, official first). + A key the corpus stops resolving prints "not in this deployment's corpus"; + after any re-ingest (T3.8 IRB headings, an ENF re-parse) run + `tests/test_read_next.py`, which resolves every key and pins every quote. +- **Policy caveat**: the six entries whose bottom lines keep the act branch + open after a non-conviction (withheld and deferred adjudication, pretrial + diversion, ACD/CWOF, youthful offender, expungement) carry `policy_caveats`: + ENF 2 s. 3 (enf-2-3-p7/-p8, an ATIP copy) says the committing-an-act + provisions "would generally not be applied" after, for example, a discharge, + deferral or expungement, and that unusual situations may make their use + "entirely appropriate". Both readings print (statute first / policy first); + the maintainer decides. Rendered in the tools and in commentary (re-render + in the main loop). +- **Caveats by footing** (disposition roadmap T1.5, T1.10 and the critic's + items, 2026-09-16, HELD): `commentary.CAVEAT_CLASSES` says, by curated caveat + id, what a caveat bears on. `act` (ENF 2 s. 3, now also on uk-caution, DPA, + discharge-no-conviction-recorded, state-pardon and foreign-rehabilitation-law): + the act-branch allegation is drafted, the caveat prints only when the act + footing is in play, and the card's Action ends "POLICY CAVEAT (ENF 2 s. 3): + NOT SETTLED"; while the record says a programme still runs + (`screening.programme_running`) the concluded-trial condition prints instead + of the completed-programme readings. `act-not-settled` (enf-6-s23-1-acquittal: + the new acquittal entry, ngri, conviction-reversed-on-appeal, + mexico-dismissal): no act-branch allegation, s. 44(1) skeleton, removal path + or card report is drafted -- ENF 6 s. 23.1 against the statute and Magtibay. + `act-if-acquitted` (pending-charge): the readings apply if the charge ends in + an acquittal. `conviction` (the conviction entry's Rodriguez Anzola 2026 FCA + 90 caveat): printed on the conviction footing, its bottom-line and card + clauses only when the record's words say the plea was coerced + (`screening.duress_words`). A new caveat id needs a class + (`tests/test_step1_caveats.py` fails otherwise). CANLEX_RESULT carries + `footing` and `act_branch`; canlex_screen_history's carries `step1` per entry. +- **State rows and non-US routing** (disposition roadmap T1.8, T1.9, + 2026-09-16, HELD): place gates in `us_dispositions.json` -- an entry's + `countries` and its `gated_names` ([{names, places}]; places are US state + names or countries and regions) meet a record only from a listed place + (`server._place_gate`); with no place named, a gated name meets it only as a + full phrase and yields a tie to an ungated one; a US record's state, or a + non-US record's country and state field (`server._record_places`, passed to + `_step1` / `_match_dispositions`), decide. 'deferred sentence' is state-aware + this way: the new deferred-sentence entry (Washington, Iowa, New Mexico + post-judgment rows, both readings, "was judgment entered before the + deferral?") answers unless the state's deferred-adjudication row describes its + deferred sentence (Vermont, Rhode Island, Colorado, D.C., Louisiana, Montana, + North Dakota). Alaska SEJ / SIS rows are filed by mechanism, WA RCW 10.05 is a + pretrial-diversion row, 'withheld judgment' is an Idaho name. Non-US: penal + orders (Germany's Strafbefehl LIKELY YES on StPO § 410(3), labelled + INTERPRETATION; others FACT-SPECIFIC), three Korean entries, Scottish fiscal + disposals on uk-caution, 'conviction not recorded' wording and country-gated + NSW s. 10 / NZ s. 106 / Probation of Offenders Act names on + discharge-no-conviction-recorded, 'procedimiento abreviado' on conviction + (Mexico); 'penal' carries no name alone; a non-US 'expunged' reaches + foreign-rehabilitation-law. Every foreign quote comes from + `data/curated/drafts/batch1-{us,nonus}-sources-2026-09-16.json` and is pinned + there by `tests/test_step1_rows.py` (blocked hosts are marked NOT READ, never + quoted); `tests/test_disposition_fixtures.py` checks each gated name from its + places. After a curated edit re-run `python tests/test_gold.py --write-baseline`. +- **Row selection by mechanism and the step-1 headline** (disposition roadmap + T1.11, T6.1, 2026-09-16, HELD): a state row is found by its state, and + optional curated fields narrow that choice (`server._select_row`, shared by + every tool through `_resolve_step1`): on a row, `covers` (read only when two + rows of one entry share a state; never defaulted from the label), + `other_programs` ([{label, cites, words?, entry | verdict, note?}] -- the NY + CPL 160.50 / 160.55 / 440.10 and NM § 31-20-3 / § 31-20-13 seeds), `not_for` + and `mechanism`; on an entry, `mechanism_cues` and `ground_not_stated` + (expungement and set-aside-1203-4: a legal-error vacatur, and 'vacated' with + no ground). A set-aside row prints one ROW NOT APPLIED line, floors the + verdict at FACT-SPECIFIC unless the readings share a footing, and + CANLEX_RESULT carries `row_mismatch`; a cite in a refused segment ('440.10 + MOTION DENIED') counts for nothing. canlex_us_disposition opens with STEP 1 + FOR THIS RECORD / CONSEQUENCE / WHAT DECIDES IT (the governing verdict, as + CANLEX_RESULT), demotes the entry verdict, counts the other states when a + state is given and prints the full conviction framework only with no match + or 'all'. `tests/test_step1_row_selection.py` sweeps every state row for + text-and-result agreement and holds the SHRINK-ONLY `ROW_REACH_MISSES` counts + (curated names that do not reach one of their entry's rows); lower a count + when it falls. + +## The pure chain, decisive facts, confidence and the result card (2026-09-15, roadmap T4.1 / T4.2 / T4.5 / T4.7; HELD FOR REVIEW) + +- **The chain** (`_chain_entry` for one foreign entry; `_history_rows` and + `_history_answer` for a record): steps 1-3 as fields, on the shared + `_step1` / `_step2` and `rehab.assess` / `assess_record`. canlex_screen_record + and canlex_screen_history render them. A refactor with no verdict change: + before/after every screening case (123) and a 60-record set gave the same + CANLEX_RESULT fields and the same memo text once the new blocks are removed + (the check: every screening case and each record run through the tools at + the parent commit and at this one, the CANLEX_RESULT lines compared field + by field and the memo text with the new blocks cut out -- a one-off + comparison, not kept in the repo; the two records that differ carry a + `conviction_date`, the one new input read). +- **THE ANSWER CHANGES WITH THESE FACTS** (top of both memos; CANLEX_RESULT + `decisive_missing`, `decisive_checked`): the open result's own questions (a + tie fact, the offence date of a straddled maximum, the limb, quantity or + inchoate role, an element or specification answer) and each assumed fact + (status, relief -- the disposition's words plus 'later pardoned' / 'later + expunged' --, another record, the age) are re-run with every plausible value; + only facts whose values give a different branch, allegation paragraph or + rehabilitation status are listed, each value with its answer. The ASSUMED + line a decisive fact replaces says "DECISIVE HERE"; `assumed` is unchanged. + One fact is varied at a time. screen_history re-runs each entry's questions, + each conviction's relief, the status and the age over the whole record (not + the limb or quantity readings, and not 'record-complete'). +- **conviction_date** (screen_record parameter; screen_history's entry key now + read here too): with no offence date it bounds the Tran maximum (offence <= + conviction <= completion, the earlier bound used), and the memo says which + bound set it. Out-of-order dates print a WARNING. +- **IF SERVED EXACTLY AS IMPOSED** (`sentence_imposed` + `sentence_date`, no + `sentence_completed`): `screening.parse_sentence` reads '3 YRS PROB', '30 DAYS + JAIL SUSP', '$500 FINE', '12 MOS DL SUSP', '180 DAYS JAIL, 170 SUSP'; + `projected_completion` applies Guide 5312's rules (policy-imm-5312-12, + verbatim, pinned by tests/test_decisive_card.py) to the part that ends last. + A fine or an unreadable part blocks it; a driving suspension prints both + readings (court-imposed or administrative). The footing gives the dates that + completion would give -- never a status -- and a MARGIN line within 12 + months of the assessment date. CANLEX_RESULT `completion_basis` + ('sentence_completed' / 'if-served-as-imposed' / 'not-projected') and + `if_served_as_imposed`. +- **confidence** (every screening tool): `step1` [{entry, entry_status, + row_verified, row_tier}], `step2` [{pairing, authority_status, + authorities_in_corpus 'n/m', maximum_source current / history / timeline / + limb / resolver}], `step3` {limbs_tested, limbs_unknown}. +- **detail 'card'** (screen_record, screen_history): about ten lines -- the + result or NOT SETTLED with the deciding fact, step 1 and its tier, the dates, + the action (a s. 44(1) report, a TRP, allowing the person to leave -- or no + ground, or deemed), the facts that change the answer -- then the same + CANLEX_RESULT line plus `detail: "card"`. 'full' stays the default. + +## canlex_screen and the s. 44(1) narrative skeleton (2026-09-15, roadmap T4.4 / T4.6; HELD FOR REVIEW) + +- **canlex_screen** (the 14th tool): the record as typed entries -- `entries` + is a list of objects (`ScreenEntry` in canlex/server.py: the keys + canlex_screen_history's entries_json takes, plus `relief`), with `status`, + `application`, `country`, `date_of_birth`, `relief`, `today`, `detail` + (default **'card'**) and `language`. FastMCP (mcp 1.27.1) renders it as an + array of objects with `$defs`/`$ref`; a JSON string is still accepted. An + unknown key is refused (pydantic `extra="forbid"`). No free-text rap-sheet + segmentation on the Space: the identifiers in a pasted rap sheet would + already have reached the client's model before any server-side stripping + could run. canlex_screen and canlex_screen_history share one pipeline, + `_screen_entries` (moved out of canlex_screen_history unchanged: the same + one-off before/after comparison as for the pure chain above -- every + screening case and each record at the parent commit and at this one -- + gives the same CANLEX_RESULT fields and the same memo text once the + skeleton is removed; the only other difference is the corrected cannabis + penalty text below). +- **relief**: an entry's `relief` (or canlex_screen's record-level one) is + appended to the disposition's words, which step 1 matches on (a pardon, + an expungement...); 'none' confirms there is none. `no-foreign-relief` is + assumed only while some entry has no relief given, and the decisive-facts + relief re-run skips an entry whose relief was given. +- **The DRAFT s. 44(1) narrative skeleton** (`canlex/narrative.py`; the end + of the full memo of canlex_screen_record, canlex_screen_history and + canlex_screen; `language` 'en' / 'fr' / 'both'; CANLEX_RESULT `narrative`): + ENF 5 s. 13.1's frames verbatim in both languages (tests/test_narrative.py + pins them and shows the rendered sentence IS the ENF 5 sample with its + brackets filled; the French sample's missing bracket after 'É.-U.' is + closed); the IRPA paragraph `_allegation_paras` proposes, with its words + (French: `FR_ALLEGATION`, verbatim from I-2.5-fr-s36-p1); the Canadian + provision from the pairing's new optional **`report_provision`** (curated, + HELD: 130 provisions on 66 pairings, seeded from + data/curated/drafts/narrative-draft-2026-09-12.json, every EN and FR line + verbatim in its chunk); the maximum in the statute's own words in each + language (ONE rule: the words the provision's line uses for the term -- + 'not more than 10 years' / 'de dix ans'; ENF 5's French sample recasts it + as 'ne dépassant pas', CanLex does not). `narrative.choose` picks the + provision by the matched limb and the maximum at commission: a maximum at + commission that differs from today's provision (a Tran history row; a + provision not in force at commission) prints [CONFIRM] with today's + provision for reference, because the former text is not in the corpus; + candidate limbs with different provisions print 'CONFIRM which'; an + inchoate reading, a derived maximum (ss. 463, 465) and a maximum no curated + provision carries stay [CONFIRM]. A tie, a straddled maximum, an open limb + or an unsettled conviction question gets one allegation per reading, each + with its condition (ENF 5 s. 13.4: a separate report per allegation) -- + never one settled allegation. Name, aliases, place and date of issue, + addressee, city, status evidence and signature stay [CONFIRM]; the date is + prefilled with the conviction and offence dates given, and ENF 5's silence + on which date it means is said. The title, the allegation sentence, the + act-branch and two-offence narratives are CanLex's wording, marked + '(drafted)'. A conviction in Canada gets no skeleton (ENF 5's sample is for + a conviction outside Canada). On the approved baseline (no + report_provision) the provision and maximum stay [CONFIRM] and the block + says the provisions are held. +- **Curated fixes (HELD)**, from the prep3 critic: failure-to-appear's primary + report provision is s. 145(2) (failure to attend court or surrender), with a + new verify row for its 'not exceeding two years' and the penalty text naming + it; s. 145(3)(a) is an alternative. Burglary's provisions follow the matched + limb (348(1)(d) dwelling-house, 348(1)(e) otherwise; no place, both). The + cannabis-distribution penalty text cites s. 12(9)(a), not 12(7)(a). + The curated JSON was edited by a one-off patch script, not kept in the + repo, that wrote nothing unless every line was verbatim, every provision + was a verify section and both languages stated the same maximum. + Commentary renders the penalty texts: + re-render commentary.json in the main loop. +- **Held manifest**: report_provision is a new FIELD on approved pairings; + the manifest names new entries, pairings and words, not changed fields + (the A13 decision), so a composed release simply lacks it and the skeleton + says so itself. + +## The s. 36 screening chain (2026-09-09) + +`canlex/screening.py` holds the shared, unit-tested logic behind +canlex_us_equivalency, canlex_rehabilitation and the one-call +canlex_screen_record: the offence matcher, Tran timing, branch codes, date +maths and the CANLEX_RESULT hand-off line each tool appends. + +- **The matcher refuses to guess.** A pairing is returned only when a term + matches as a phrase, or the query's content words cover >= 60% of the term + AND the term covers >= 50% of the query, with an overall floor of 0.8 + (`MIN_SCORE`); `negative_terms` veto. Statute codes (`ALIASES`: VC 23152, + PC 647(f), VTL 1192 ...) resolve to plain charges first, and drug names + (`SUBSTANCES`) generalise to 'controlled substance' with the CDSA schedule + carried alongside. Below the floor the embedder may offer a 'closest by + meaning' suggestion (labelled) at cosine >= 0.83; otherwise the Hill + framework alone. The first matcher scored partial word overlap and put + 'reckless driving' on DUI and 'sexual assault' on simple assault. +- **Structured fields drive the branch.** Each pairing carries `max_years`, + `hybrid`, `summary_only`, `equivalent`, `depends_on` and an optional + `penalty_history` (the Tran rule: DUI before 2018-12-18 was 5 years). + Since 2026-09-12 (roadmap T1.13 / T3.1, all optional, HELD until reviewed): + `limbs` names the Canadian provisions of one pairing whose maxima have + different histories (child pornography possession vs making; cannabis + trafficking by quantity; s. 173(1) vs 173(2)); a history row with `limb` + applies to that limb only, and a record whose words select no limb gets + every reading ('limb-open') wherever they differ in branch. History rows + from the point-in-time research carry `pit_url` + a verbatim `contains`, + pinned against the trimmed page in tests/fixtures/pit (tests/test_tran_limbs.py). + `in_force` dates a provision's first point-in-time version (an earlier + offence gets the NOT IN FORCE AT COMMISSION note, both readings); + `pit_checked` records the walk (the count without it may only fall); + `pit_floor` moves the "maximum before 2003-01-01 not verified" line. + Regenerate the research with `python tools/pit_history.py --offline`. + tests/test_curated_penalties.py checks every `verify` phrase against the + corpus's own statute text, checks the fields agree with the prose branch, + and that no two pairings share a term. Add a `verify` entry with any new + maximum. +- **Rehabilitation takes dates.** `sentence_completed` (or the act date) + plus an optional `today`; the tool computes full years from the day after + and reports the earliest deemed-rehabilitation and application dates. + `status='permanent-resident'` collapses a sub-10-year matter to + 'not-inadmissible-pr' (s. 36(2) reaches foreign nationals only). +- **Dispositions dataset**: 19 entries incl. youthful-offender (adjudication + vs sentencing states), ny-violation, conditional-discharge (the NY/IL + sentence vs Connecticut AR), court-martial tiers, dismissed/nolle/stet, + conviction-under-appeal, deferred-prosecution-agreement. + +## Audit follow-up implementation (2026-09-21): group A -- retrieval performance and retrieval residue + +Every number below was measured on the maintainer's laptop with other jobs +running (timings are noisy; memory and ranks are not). Rank identity was +checked with `PYTHONHASHSEED=0` on both sides (the BM25 term loop iterates a +Python set, so its tie order and float sums follow the hash seed -- as they +always have) and `tools/rankdump.py` over the 325 scored eval questions. + +**ONNX arena shrinkage** (canlex/rerank.py `make_run_options`, used by +`Reranker.score` and `Embedder._run`). Every `session.run` passes RunOptions +with `memory.enable_memory_arena_shrinkage=cpu:0`, so ORT returns what a run +allocated instead of keeping its high-water mark. First search after load: ++472 MB RSS / +1,237 MB private before, +12 MB / +40 MB after. Logits (320 +sampled pairs) and embeddings (64 passages, 20 queries) bit-identical. + +**Model files from the local cache first** (`rerank.hub_file`, used by both +models). `hf_hub_download(..., local_files_only=True)` first; the network +only on `LocalEntryNotFoundError`. A cold `Embedder()` + `Reranker()` sent 7 +HEAD requests to huggingface.co before and none after; with the network +refused it took 17.5 s before and 7.0 s after. `.mcp.json` now also sets +`HF_HUB_OFFLINE=1` for the stdio server. **A fresh clone must fetch the two +models once** before that server can start (it refuses every Hub request): + + .venv\Scripts\python.exe -c "from canlex.embed import Embedder; Embedder(); from canlex.rerank import Reranker; Reranker()" + +(run it without `HF_HUB_OFFLINE`; the files land in the Hugging Face cache). + +**BM25 postings as arrays, cached** (canlex/bm25.py; `LegislationIndex. +_build_bm25`, `_bm25_accumulate`, `_bm25_ranking`). The postings are CSR +arrays (per-term offsets into one chunk-index array and one tf array) with +the corpus statistics and the typo corrector's word counts, cached in +`data/cache/bm25-.npz` (gitignored; about 47 MB). The key is the +processed files' (name, size, mtime_ns) AND a sha256 of the source of +`tokenize`, `_stem`, `_fold_accents`, `topical_title`, `_bm25_blob`, the +corrector word functions, `bm25.build`, `_STEM_NORMALIZE`, the token +patterns, the snowballstemmer version and the stemmer's class (module and +qualname; verification 2026-09-22 -- a stemmer swap kept the key before) and +`bm25.CACHE_VERSION` -- a corpus change or a tokenizer change rebuilds it, +never serves stale postings. Bump `CACHE_VERSION` for a change the +fingerprint cannot see (a dtype). A cache that cannot be read or written +(read-only disk, corrupt file, a zero-byte file an unclean shutdown left -- +which raised EOFError and failed every start before 2026-09-22 -- or arrays +that do not fit the corpus) is rebuilt in process with a stderr line and +rewritten; `CANLEX_CACHE_DIR` moves it (e.g. to /tmp on the +Space, whose image does not ship it: the first start there builds it). The +scorer adds each term's postings with numpy in the old term order and +arithmetic, so scores are bit-identical and ties keep their old order. The +typo bridge's trigram index and the corrector vocabularies are built at load. + +| | before | after | +|---|---|---| +| `LegislationIndex()` (cache hit) | 12.5 s | 3.8 s | +| `LegislationIndex()` (cache miss: build + write) | 12.5 s | 14.4 s, once | +| RSS / private after load | 1,986 / 2,465 MB | 1,454 / 1,932 MB | +| RSS / private after the 325 eval searches | 3,402 / 4,822 MB | 1,489 / 1,992 MB (with the arena item) | +| BM25 scoring, English eval question, p50 / max | 56 / 126 ms | 8 / 16 ms | +| search(top_k=6) p50 over 40 questions | 1,548 ms | 1,488 ms (reranker-bound) | +| rankdump, 325 questions | -- | identical (every result id, score and pinpoint too) | + +**Retrieval residue** (HELD; `_SECTION_REF`, `_french_short_title`, +`LegislationIndex._french_acts_named`, `_correct_query`). The section pin +now reads `sect. 36`, `art. 36`, `article 36`, `l'article 36`, `articles 17 +et 18` (as canlex_get_section does; `Article 1E` of the Refugee Convention is +no section) and an Oxford comma (`ss. 17, and 18` read `{17}` before, `{17, +18}` now). Since 2026-09-22 (`_article_cites_section`) an article form in +English cites a section only in a query that names no treaty (Convention, +Covenant, Protocol, Treaty, ICCPR, CAT, the Rome Statute ... -- not +'Convention refugee', not 'Agreement', whose FB articles are sections), and +never as a dotless `art`/`arts` (`martial arts 5 years`); in French syntax +it does unless a treaty follows (`l'article 33 de la Convention`). A French +Act's short title, read from its French twin's `Titre abrégé` section +(accents and apostrophes folded), names that Act for the pin only -- never +for the fusion's act hint: `Loi sur l'immigration et la +protection des réfugiés article 36` answered IRPR s. 338 (fr) before and IRPA +s. 36 (fr, fr, en) now. Regulations have no French short-title section, and +the French acronyms (LIPR, RIPR) are not read: `article 36 de la LIPR` still +pins nothing (it is in the lookup slice, as a known miss). A French query's +ACCENTED word is corrected only into a word of the French chunks, accents +kept: `la détension du résident permanent` became `la detention du ...` +before and `la détention du ...` now. The accent trade-off stands: a lone +accented word in an English query is English and corrected as before. + +**The eval 'lookup' slice** (data/eval/questions.json, style `lookup`, added +2026-09-21, 14 questions): section lookups in the officer's phrasing -- +`deemed rehabilitation section 18`, `section 36 of the Immigration and +Refugee Protection Act`, `ss. 17 and 18 of IRPR`, `IRPA 36(1)(a)`, `article +36 de la LIPR`, `IRPR ss. 17, and 18`, `sect. 44 IRPA report` ... -- with the +pinned provision as gold (every gold id checked against the corpus by +tests/test_eval_lookup.py). `py -m canlex.eval` reports it as `style: +lookup`; dated, so outside legacy. + +**Embedding currency** (`LegislationIndex.embedding_missing`, +`canlex.embed.currency`, `py -m canlex.embed --check`). The index counts the +chunks with no vector (None on a synthetic index, and None when the store or +the query embedder fails to load -- `LegislationIndex.semantic` is then False, +keyword search only; it said 0 before 2026-09-22); `currency()` returns +(missing, stale) over the corpus, stale meaning a vector embedded from other +text. tools/predeploy.py `check_embeddings` now FAILS a deploy tree when +either is above 0 anywhere in its corpus (it read the commentary chunks +only); a corpus it cannot read (an LFS object not in the local store) WARNs, +and a corpus file that does not parse FAILs, naming it (verification +2026-09-22, A-EMB-3: until then both PASSed). The tree at the start of +this work: 68,560 chunks, 0 missing, 0 stale. + +**Eval-gated retrieval changes (Optimization 7, Missing 11).** Each change +was measured on its own against the same baseline (the tree after the items +above, 339 scored questions), with `py -m canlex.eval` and +`tools/rankdump.py`'s rank rule over the same searches (scratch harness; the +eval report and rankdump output were produced from one pass). The rule: ADOPT +only when legacy (n=192) and holdout (undated) (n=40) Hit@1, Hit@3 and MRR +do not drop at all and no other slice's Hit@1 drops by more than 0.03; +otherwise the code stays as it was. Runs used `CANLEX_ORT_THREADS=4` (logits +verified bit-identical to the default thread count) and `PYTHONHASHSEED=0`. + +| change | legacy H@1/H@3/MRR | holdout (undated) H@1/H@3/MRR | ALL MRR | worst other-slice H@1 | ranks better/worse | top-6 moved | verdict | +|---|---|---|---|---|---|---|---| +| baseline | .7760/.9583/.8651 | .7500/.9250/.8383 | .8043 | -- | -- | -- | -- | +| (a) one reranker run per document | .7760/.9688/.8668 | .7500/.9500/.8404 | .8063 | 0 | 8/3 | 209 | ADOPTED | +| (b) pinpoints on results 1-3 only | unchanged | unchanged | .8043 | 0 | 0/0 | 0 | ADOPTED | +| (c) RERANK_POOL 8 | .7760/.9427/.8597 | .7500/.9000/.8363 | .7991 | 0 | 7/14 | 305 | rejected | +| (c) RERANK_POOL 12 | .7760/.9531/.8631 | .7500/.9250/.8383 | .8021 | 0 | 6/7 | 249 | rejected | +| (c) RERANK_POOL 20 | .7760/.9583/.8644 | .7500/.9000/.8300 | .8046 | 0 | 8/8 | 249 | rejected | +| (d) idf_en in the MN boost | .7760/.9635/.8678 | .7500/.9500/.8446 | .8015 | -.143 keyword | 8/8 | 106 | rejected | +| (d) idf_en in the MN boost and _title_score | .7760/.9583/.8673 | .7500/.9500/.8488 | .8012 | -.143 keyword | 9/9 | 115 | rejected | +| (e) topical_title in the rerank header | .7760/.9688/.8671 | .7500/.9500/.8404 | .8047 | 0 | 5/3 | 233 | ADOPTED | +| (f) MN_WEIGHT .0012 | .7969/.9635/.8810 | .7500/.9250/.8467 | .8074 | -.250 typo (also french, guidance) | 18/17 | 220 | rejected | +| (f) MN_WEIGHT .0036 | .7708/.9635/.8602 | .7750/.9250/.8494 | .7953 | -.250 union | 9/17 | 201 | rejected | +| (f) MN_CAP .012 | .7917/.9688/.8764 | .7750/.9500/.8613 | .8171 | -.059 fact-pattern | 16/7 | 111 | rejected | +| (f) MN_CAP .036 | .7708/.9531/.8625 | .7500/.9000/.8350 | .7967 | -.083 french | 1/6 | 17 | rejected | +| (g) idf >= 1.0 floor on the route guard | unchanged | unchanged | .8043 | 0 | 0/0 | 0 | ADOPTED | +| **adopted together (a+b+e+g)** | **.7760/.9688/.8656** | **.7500/.9500/.8404** | **.8050** | 0 | 7/5 | 271 | gate PASS | + +- (a) `Reranker.score` runs every (query, document) pair alone. The int8 + reranker quantizes activations per run over the whole input tensor + (`DynamicQuantizeLinear`, 50 nodes), so in a padded batch a pair's logit + moved with its batch companions -- all 192 logits of a 12 x 16 sample, + by up to 8.8. Now a pair's score is the pair's alone, and there is no + padding to compute. +- (b) `HIGHLIGHT_DEPTH` (env `CANLEX_HIGHLIGHT_DEPTH`, default 3, was 8): + results 4-8 no longer carry a 'Most on point' pinpoint (159 of 339 eval + answers lose one or more); no rank can move. +- (e) `_rerank_doc`: the cross-encoder reads a D-memo's subject ('part') and + a decision's proposition ('heading') instead of the generic banner or the + paragraph range. +- (g) `_ROUTE_IDF_FLOOR`: 'of' or 'the' in a title no longer lets a + pairing-allowlisted section through the criminality route's overlap guard. +- Rejected, code unchanged: every RERANK_POOL value (16 stays); idf_en for the + title-match boost (one keyword question loses rank 1, both scopes); + every MN_WEIGHT/MN_CAP step. MN_CAP .012 is the strongest candidate + measured (legacy MRR .8651 -> .8764, holdout .8383 -> .8613, ALL +.0128) + and fails the rule on one fact-pattern question ('traveller has USD 9,500 + in cash ...', 1 -> 4) -- a maintainer call, not adopted here. +- Speed, `search(top_k=6)` p50 over 40 eval questions, one process, default + threads: 1,425 ms before, 951 ms with (a)+(e)+(g), 877 ms with all four. + +**The adopted baseline** (the tree after these changes; `py -m canlex.eval` +prints the same numbers rounded to two decimals): + +| slice | n | Hit@1 | Hit@3 | Hit@5 | MRR | +|---|---|---|---|---|---| +| ALL | 339 | .7021 | .9086 | .9558 | .8050 | +| legacy | 192 | .7760 | .9688 | .9948 | .8656 | +| holdout (undated) | 40 | .7500 | .9500 | 1.0000 | .8404 | +| holdout (all) | 50 | .6600 | .8400 | .9000 | .7479 | +| caselaw-found | 28 | .8214 | .9286 | 1.0000 | .8851 | +| criminality | 30 | .3000 | .6000 | .7000 | .4512 | +| fact-pattern | 17 | .5294 | .8235 | .9412 | .6986 | +| french | 12 | .5833 | .9167 | 1.0000 | .7528 | +| guidance | 19 | .6316 | .8947 | .9474 | .7654 | +| keyword | 7 | .5714 | 1.0000 | 1.0000 | .7619 | +| lookup | 14 | .8571 | .8571 | .9286 | .8750 | +| typo | 4 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | +| union | 4 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | + +**Token-aware embedding windows -- measured and REVERTED (2026-09-22).** +Group A implemented Optimization 11 (each tail window starts where the +embedder's 512-token truncation stopped reading the window before it) in its +own commit 1ec9d65, after every eval above, and did not measure it. The +integration measured it on the regenerated corpus (486cfc9), with the same +code and data both ways and only the window policy changed (group A's +evalrank harness, PYTHONHASHSEED=0, 339 questions): + +| slice (H@1 / H@3 / H@5, MRR) | old windows (74,743 vectors) | token-aware (85,679 vectors) | +|---|---|---| +| legacy n=192 | .78 / .97 / .99, .87 | .78 / .97 / .99, .87 | +| holdout (undated) n=40 | .75 / .93 / 1.00, .84 | .75 / .93 / .95, .83 | +| criminality n=30 | .30 / .60 / .70, .45 | .27 / .60 / .67, .44 | +| french n=12 | .58 / .83 / 1.00, .75 | .58 / .83 / .92, .74 | +| ALL n=339 | .71 / .91 / .96, .81 | .70 / .91 / .95, .80 | + +It fails the adoption rule used for every other ranking change (legacy and +holdout (undated) must not drop; no slice's Hit@1 may drop more than 0.03): +holdout (undated) MRR and criminality Hit@1 drop. Six questions moved down +(for example 'Quels facteurs justifient le maintien en détention selon les +directives de la CISR?' 3 -> 7, 'What removal order does a foreign national +get for A36(1)(b) serious criminality' 5 -> 7) and two moved up. More text +embedded per long chunk dilutes the vectors of the long guidance chunks the +questions target. The commit is reverted (the embedder is back to fixed +2,000-character tail steps) and the store was rebuilt with the old windows +(74,743 vectors, `canlex.embed --check` 0 missing, 0 stale). The unread-tail +measurement stands (12,147 chunks, 3.85 M characters no window reads); a +future attempt should try a smaller change (a single extra tail window, or +the tail of the section only) and must be eval-gated before adoption. + +## Audit follow-up implementation (2026-09-21): group B -- engine and server performance, robustness and wording + +What changed in the engine and the server after the code audit of 2026-09-21 +(data/curated/drafts/audit-2026-09-21-code.md, sections 4-6). Officer-facing +answers are byte-identical except where an item says otherwise; everything +that a REVIEWED name reaches is HELD by the review gate until approved. + +**Screening caches (optimization 1).** `screening.load_aliases` returns early +when the merged, lower-cased table equals (a deep copy of) the installed one: +every screening call still calls `_load_equivalency()`, which now costs a dict +comparison instead of a 17 ms rebuild and a cleared expansion cache. The +comparison is of state, so a different table is still installed and the next +curated load undoes it. A test that installs a table itself (the approved +baseline) should `self.addCleanup(sc.load_aliases, None)`. `_section_text` +parses a processed file once per mtime (the IRPR fee lines), `_disp_fold` is +memoised and each curated pairing term is read once (`_term_reading`). +Measured warm on this laptop, before and after interleaved under the same load: +canlex_screen_record 133-140 -> 16-18 ms (DUI, TX) and 111-115 -> 14-16 ms +(theft, deferred adjudication), canlex_us_equivalency 34-40 -> 5.7 ms, a +one-entry canlex_screen card 111-114 -> 23-26 ms. + +**The record step inside one call (optimization 4).** `_record_assessment` +runs `rehab.assess_record` once per distinct input inside the tool call's step +memo (every caller gets a deep copy), and the base record step runs in the +memo its decisive re-runs share. The four-entry 'possession of a controlled +substance' / 'deferred sentence' record: 11-12 assess_record runs -> 1-2, and +7.96-8.00 s -> 0.65-0.79 s through canlex_screen (interleaved, same load). +Outside a step memo nothing is cached. + +**Concurrency (optimization 10).** Over HTTP, `offload_sync_tools` runs the +curated screening tools (`SCREENING_TOOLS`: us_disposition, us_equivalency, +rehabilitation, screen_record, screen_history, screen, transborder) up to +`SCREENING_LIMIT` = 3 at a time; every other tool -- search, sections, toc, +list_acts, citing, case, whats_new -- still runs one at a time. `warm_tables()` +builds the lazy tables before the first call (the startup log says how many): +the curated datasets, the alias table, the alias keys, the held manifest, the +disposition-name folds and, when the index offers them, its typo and trigram +tables. The small module caches publish whole values, so a second thread never +reads one half-filled. tests/test_concurrency.py runs 8 threads x 20 mixed +screening calls against their serial answers. Assumptions: the curated files +do not change under a running HTTP server (a change is picked up, but a call +running at that moment may see the old table and the new one; what it caches +from the old table is keyed by the old alias generation, so no later call +reads it; restart after editing curated data on a served process all the +same); the index tools keep one token +(their lazy tables are group A's). To go back to one-at-a-time for every tool, +set `SCREENING_LIMIT = 1` in canlex/server.py. + +**Suggest-by-meaning needs a loaded index (optimization 12).** When no curated +pairing matches, `_step2_match` asks the embedder for a 'closest by meaning' +suggestion only if `server._INDEX` is already loaded; otherwise one stderr line +(`closest-by-meaning suggestion unavailable: index not loaded`) and no +suggestion. The served processes load the index in `__main__`, so nothing +changes there. In library or test use, call `server._index()` first if the +suggestion is wanted (it costs the 30-60 s index load, which is what a no-match +offence used to pay inside the call). + +**Tool error guard (missing 8).** Where the call log is installed +(CANLEX_LOG_DIR, or CANLEX_LOG_DATASET with CANLEX_LOG_TOKEN), a tool's own +exception no longer reaches the client as `str(exc)`: the client reads `Error: +CanLex could not complete this call (); reference `, the traceback +is printed to stderr (the Space's container log) under the same uuid, and the +call-log record carries `error_id`. Search the log for the uuid an officer +quotes. An unknown tool, arguments that failed validation and a cancellation +pass through unchanged. + +**Optional rate limit (missing 14).** canlex/ratelimit.py, wrapped around +`_Utf8Body` on the HTTP transport; OFF unless set: + +| variable | meaning | +|---|---| +| `CANLEX_RATE_LIMIT_PER_MIN` | requests a minute per client (a positive integer; unset or invalid = off) | +| `CANLEX_RATE_LIMIT_BURST` | the burst a client may spend at once (default: one minute's allowance) | + +A suggested lenient start for the Space: `CANLEX_RATE_LIMIT_PER_MIN=120` (a +tool call is one to three HTTP requests). Over the limit: 429, a plain-text +body and `Retry-After`. `GET /` (the notice page, health probes) is never +counted. The client is the rightmost `X-Forwarded-For` entry (the address +Hugging Face's ingress saw; repeated header lines are read as one list), else +the socket peer, cut to 64 characters; at most 10,000 clients are tracked, +idle ones dropped first. The startup log prints the limit when on. The key +assumes an appending proxy in front, as Hugging Face's ingress is: served +without one, the header is the client's to write, so it can dodge its own +limit or spend another address's. + +**Response-size options (missing 13; the defaults are the answers as they +stood).** `canlex_list_acts(doc_type=..., compact=True)` and the +`canlex://corpus/{view}` resource (`compact`, or a doc_type) -- the full list +is about 44,000 characters, compact about 4,000; `canlex_us_disposition('all', +detail='entries')` -- the disposition types without the 50-state rows (with +the given state's rows when `state` is given) and the framework in brief, +about 4,300 characters instead of 23,900; `canlex_get_section(..., +grounding='brief')` -- a three-line reminder (GROUNDING_BRIEF, REVIEWED) in +place of the 2,146-character GROUNDING_NOTE, which stays the default and stays +unchanged (decision taken as the audit recommended: offer `brief`, keep `full` +the default). + +**canlex_toc (missing 7).** An Act of up to `TOC_FULL_LISTING` = 400 sections +is listed in full (IRPA's 285 are; they were collapsed at 150). A larger Act +-- the IRPR's 569, the Criminal Code -- prints each Part's range and, under +it, each Division's range. +`part=` expands one Part or Division in full: 'Part 1', 'Part VIII', +'Division 6', a bare number (a Part's), or words of a heading ('detention'). +Not gated. + +**Embedding currency (missing 15, this group's half).** canlex_list_acts prints +one `_Embeddings: ..._` line under the title from the index's +`embedding_missing` (a count or the missing ids) when the index offers that +property (group A adds it); without it, the listing is unchanged. + +**s. 17 second reading (engine-s17; HELD wording).** When the most recent +conviction is a FOREIGN entry with no curated pairing, the memo now reads the +clock both ways, as it already did for a pairing whose branch is 'none': +"entry #N's Canadian equivalent is not settled (no curated pairing). If it has +no Canadian federal equivalent and such a conviction is not a subsequent +offence for s. 17 (CanLex does not settle this), the clock runs from and +the earliest application is ." (the sentence approved in the +audit memo's Decisions, taken as recommended). A conviction in Canada gets +this reading too when the entry does not show it to be under an Act of +Parliament (no maximum, no hybrid or indictable mode, no curated federal +pairing): a provincial or municipal offence is not 'an offence under an Act of +Parliament', the same question (corrected 2026-09-22, LEGAL-8 -- this line had +said a conviction in Canada is its own equivalent). Both readings go into +CANLEX_RESULT `s17_readings`: {"subsequent-offence": {entry, from, apply_on}, +"not-a-subsequent-offence": {from, apply_on}}, present only when there are two +readings. Fixture: burglary 2010 (completed 2011-03-01) and jaywalking 2020 +(completed 2020-04-01): 2020-04-01 / 2025-04-01, or 2011-03-01 / 2016-03-01. +Since the G3 fixes of 2026-09-22 (below): "subsequent-offence" is always the +headline (the strictest date), a clock entry whose verdict is open may add +"a-committed-act": {from, apply_on}, the card prints each other reading as a +NOTE, and `s17_clock_kind: "offence-date"` marks a clock that runs from a +committed act's offence date (`s17_clock_entry` / `s17_clock_from` name it). + +**Internal-API guards (engine-info-trio).** `rehab.assess` raises ValueError +for `max_penalty_lt_10=None` (it silently meant serious criminality); +`screening.analyse` and `match_pairings` read a missing offence as ''; +`authorities_in_corpus` catches only OSError / ValueError / KeyError / +TypeError, answers '?/n' and says so on stderr (anything else now raises); +`maxima_stated` reads 'up to N years' as narrative.term_words does, and the two +parsers share one number-word table (a bounded agreement test covers one to +fourteen years in every form). + +**HELD by the review gate** (tools/predeploy.py check_review_code against the +approved-code commit): 339 registered names held at the group's last commit +against 321 on main at cd5109b. Newly held by this group (18): +narrative choose and term_years (via _EN_NUM); readnext compact_line, lines, +resolved, with_fallback and server _pairing_read_next_items (via _pool); +rehab assess, assess_canadian and server _chain_step3 (the None guard); +screening _STATED_MAX, maxima_stated, stated_limbs ('up to N years') and +authorities_in_corpus; server _fee (via _section_text), _held_alias (via +_held_manifest / _alias_keys_read); and the new GROUNDING_BRIEF and +_res_corpus_view. 59 names already held on main now also reach this group's +changes (among them load_aliases' callers _best_match, match_pairings, +_step2_match, _history_answer, the screening tools, canlex_toc, +canlex_list_acts, canlex_get_section, canlex_us_disposition and +canlex_search_legislation). Wording changes among them: the s. 17 sentence +and s17_readings, GROUNDING_BRIEF, the search docstring, the new tool +parameters' descriptions; everything else is behaviour-preserving. + +## Audit follow-up implementation (2026-09-21): group C -- tests, deploy gates, dependencies + +The code audit's memo (`data/curated/drafts/audit-2026-09-21-code.md`, sections +4-6) listed the dependency residue, the suite never running on the shipped +interpreter, and four areas the suite never executed. What changed, and the +routine it adds: + +**Before each deploy, run the suite on Python 3.12** (the image's and CI's +interpreter; the venv is 3.14): + +``` +git lfs checkout # LFS files must be content, not pointers +sh tools/test_in_docker.sh # Git Bash, in the MAIN checkout; Docker Desktop running +sh tools/test_in_docker.sh tests.test_server tests.test_calllog # a subset +``` + +It runs `python:3.12-slim` with the checkout mounted read-only and copied +inside the container (without `.venv`, `.claude`, the git-ignored +`data/raw` -- its tests skip, as in CI -- and the 2.5 GB LFS store, which is +linked back read-only; nothing is written to the host checkout), the Hugging +Face cache mounted read-only with `HF_HUB_OFFLINE=1` (nothing is downloaded; +model-backed tests use the models on this machine), `pip install +--require-hashes -r requirements.lock`, then the suite. Run it from the main +checkout: a `git worktree`'s `.git` file points at a Windows path the +container cannot follow, and the tests that read the approved baseline with +`git show` would skip. **Not yet run**: Docker Desktop was not running when +this was written (2026-09-22); record the first run's outcome here. + +**The locked dependency set.** `requirements.txt` still names the direct +dependencies (pinned); the image installs `requirements.lock`: all 52 +packages, transitives included, each with its sha256 hashes, resolved for +Linux / Python 3.12. The Dockerfile's `pip install --require-hashes` refuses +any distribution but the locked one, so every build installs the same set +(until the audit, starlette, anyio and the other transitives were re-resolved +at each build). Regenerate it after any change to `requirements.txt`, with uv +in a throwaway venv OUTSIDE the repo (never in `.venv`): + +``` +py -3.14 -m venv %TEMP%\uvenv +%TEMP%\uvenv\Scripts\python.exe -m pip install uv +%TEMP%\uvenv\Scripts\uv.exe pip compile requirements.txt --python-platform linux --python-version 3.12 --generate-hashes -o requirements.lock +``` + +(the command is the lock's second line). Add `--upgrade-package NAME` to move +one transitive (an advisory fix) without touching the rest, `--upgrade` to +move them all. The 2026-09-22 lock (uv 0.12.17) recompiles byte-identical, +and every pin has a manylinux cp312 wheel (`--only-binary :all: +--python-platform x86_64-manylinux_2_36` resolves). The lock carries every +platform's hashes, so the Windows venv can take the same set: +`.venv\Scripts\python.exe -m pip install --no-deps --require-hashes -r +requirements.lock` (`--no-deps`: mcp's Windows-only pywin32 is not in a Linux +lock and is already installed), then `pip check` and the suite. + +**mcp 1.30.0.** `requirements.txt` moves mcp 1.27.2 -> 1.30.0 (clears +GHSA-vj7q-gjh5-988w, fixed in 1.28.1; the WebSocket transport is not served +here). The 1.30.0 wheel was downloaded to a temporary directory and read, not +installed: `ToolManager.call_tool(name, arguments, context=, convert_result=)` +and `Tool` (`fn`, `is_async`, `run`) are unchanged from 1.27.2, as are +`ToolManager._tools`, `FastMCP._session_manager` / `_mcp_server` / +`_event_store` / `_retry_interval`, `StreamableHTTPSessionManager(..., +session_idle_timeout=)` and the client's `streamablehttp_client`. New +defaults in the session manager: a 30-minute idle timeout (CanLex passes its +own 1800 s), a 4 MiB body cap (`_Utf8Body` refuses 1 MB first) and at most +10,000 sessions (CanLex passes `max_sessions=None`: one client could fill the +10,000 and every other client's initialize got 503 for 30 minutes; the +per-client cap on new sessions bounds it instead -- verification fixes +2026-09-22, F3, SEC-1). **The shared `.venv` was upgraded on 2026-09-22** to +`requirements.lock` (`.venv\Scripts\python.exe -m pip install --no-deps +--require-hashes --only-binary :all: -r requirements.lock`): mcp 1.30.0 +(installed 08:51), `pip check` clean; the OSV check then found 10 advisories, +all on pip and setuptools, which were upgraded (pip 26.2.1, setuptools +84.0.0, 08:55); `tests/test_calllog.py` and `tests/test_server.py` passed on +it, then the full suite. + +**Advisory scan (OSV, key-free).** `tools/osv_check.py` POSTs a pinned set to +`https://api.osv.dev/v1/querybatch` (only names and versions leave the +machine), fetches each advisory, merges a GHSA with its PYSEC twin, and prints +the version that fixes it; exit 1 on any advisory, 2 when the API or the set +cannot be read. + +``` +py tools/osv_check.py --lock requirements.lock # what the image installs +py tools/osv_check.py # the checkout's .venv set (pip freeze, read-only) +py tools/osv_check.py --python PATH # another interpreter's set +``` + +`--python` defaults to the checkout's own venv interpreter +(`.venv\Scripts\python.exe`, or `.venv/bin/python`) when it exists, else the +interpreter running the tool (verification 2026-09-22, C-DOC-5: it was always +the latter, so `py tools/osv_check.py` without the venv activated scanned the +py launcher's Python 3.14 and its packages). The first output line names the +interpreter; a `git worktree` has no `.venv`, so pass `--python +C:\projects\CanLex\.venv\Scripts\python.exe` there. + +First run, 2026-09-22: the lock PASS (0 advisories on 52 packages); the venv +(before its upgrade to the lock, above) FAIL, 31 distinct advisories on 10 +of 65 packages -- anyio 4.13.0 (2, +one CRITICAL: TLS host-name encoding), cryptography 48.0.0 (4), mcp 1.27.2 +(1), pip 26.0.1 (4), pydantic-settings 2.14.1 (1), pyjwt 2.12.1 (5), +python-multipart 0.0.29 (4), setuptools 82.0.1 (1), soupsieve 2.8.3 (4), +starlette 1.0.0 (5, two HIGH). Every one except pip's and setuptools' is +fixed at the lock's versions; pip and setuptools are the venv's own tools +(the image's pip comes from the base image and is not scanned). **The weekly +routine** should run the `--lock` check; its prompt lives outside the repo +(`canlex-weekly-staleness-check`, rewritten hands-free on 2026-09-23) and does +not yet carry that line -- add it there. An advisory on the lock: regenerate with `--upgrade-package`, +re-run the check and the 3.12 suite, deploy. + +**smoke_space tells a failed rebuild from a live one.** After the golden +calls (which wake a sleeping Space), `tools/smoke_space.py` GETs +`https://huggingface.co/api/spaces/Beemer0/CanLex` (key-free) and FAILS +unless `runtime.stage` is `RUNNING` and both the repo head (`sha`) and the +commit the running container was built from (`runtime.sha`) equal `git +rev-parse space/main` (the push updates that ref); it prints the stage, the +hardware and all three commits. When a build fails, Hugging Face keeps the +previous container answering, so the tool checks alone printed PASS for a +deploy that never went live. Right after a push the check FAILS with the +stage `BUILDING` / `RUNNING_BUILDING` and the old running sha: re-run it once +the build has finished; a `BUILD_ERROR` stage means read the Space's build +log and roll back or fix forward. `--no-sha-check` (a clone without the +`space` remote) keeps the stage check only; `--no-space-check` skips it (a +local server given with `--url`). On 2026-09-22 it read RUNNING, cpu-basic, +901bf8c for all three. + +**Gold-set and fixture hygiene.** The gold ratchet +(`tests/test_gold.py`) now FAILS when the number of counted rows falls, not +only when accuracy does (two rows vanished unnoticed before); a deliberate +drop is recorded with `python tests/test_gold.py --write-baseline`. Each row's +`approved` in `tests/gold/*.json` and `tests/data/disposition_cues.json` is a +bool, and the file's header must say PENDING THE MAINTAINER'S SIGN-OFF while +any row is still `false` (and no longer has to once none is) -- sign rows off +one at a time by setting `"approved": true`, and drop PENDING from the header +once all are. (The working-tree check that no gold row's requirement is unmet +was already in place since 7499848.) + +**Tests the suite never had.** `tests/test_tools_offline.py` (canlex_toc, +canlex_whats_new, canlex_citing, canlex_case and `_format_case` over a stub +index and a stub citator -- every statement of the five (and `_citator`) runs; +before, 26/27, 32/33, 24/25, 11/12 and 36/37 were missed), +`tests/test_eval.py` (the release gate's scorer and the sweep: eval.py +160/161 and sweep.py 69/70 statements, from 0), `tests/test_predeploy_gates.py` +(check_review, check_blobs, check_commentary, check_embeddings, the +review-code registry refusals, tree_checks and main over a fake repository +with no tree check mocked; main's two working-tree checks, the suite +(`run_unittest`) and the curated drift (`check_curated_working`), were mocks +in every test until the 2026-09-22 verification (C-GATE-1), and now also run +for real with only the subprocess answer or `refresh.check_curated`'s rows +faked), `tests/test_compose_release.py` (compose_curated and +main's refusals: failing composed tests, stale vectors, a release failing its +checks), `tests/test_osv_check.py` and `tests/test_deploy_image.py` (the +Dockerfile's layer order and install line, the lock against +requirements.txt). None of them touches the network, a model or the real +git history. + +**Dockerfile layer order.** The model prefetch (`Embedder()`, `Reranker()`, +331 MB of model files) now runs right after `COPY canlex/`, with `USER app` +and the ENV block before it, and before the corpus, embeddings and curated +COPYs; `ENV HF_HUB_OFFLINE=1` stays after them. Docker rebuilds every layer +after the first changed input, so every compose (which rewrites +embeddings.npz) and every corpus or curated change used to re-download the +models; now only a change under `canlex/` or to the lock does. The final +image's instructions are the same set as before, reordered, with the lock in +place of requirements.txt. Whether Hugging Face's builder keeps layers +between builds is not visible from here. + +**Agent worktrees and LFS.** A `git worktree` made for an agent can hold LFS +pointers instead of content (`data/processed/caselaw.json`, +`embeddings.npz`, the Income Tax Act): run `git lfs checkout` in it (local +store only, no download) before the suite -- with pointers the commentary +render differs from the stored `commentary.json` on 31 chunks. +## Audit follow-up implementation (2026-09-21): Group D -- ingesters, corpus hygiene, documentation + +Implemented 2026-09-22 from the code audit memo +(data/curated/drafts/audit-2026-09-21-code.md, sections 4.6, 6.6 and 6.12). +Everything here is ingest-side plumbing except the `canlex_list_acts` line +(HELD). The corpus changes land only when the main loop regenerates +data/processed; the counts below were measured on the 2026-09-21 corpus +without writing anything. + +- **Invisible format characters.** `_common.norm_ws` drops the soft hyphen + U+00AD, the zero-width space, non-joiner and joiner U+200B-U+200D, the word + joiner U+2060 and a stray U+FEFF (`_common.FORMAT_CHARS`, + `strip_format_chars`) before collapsing whitespace, so 'assist\xadance' + indexes as 'assistance'. 237 chunks carry one in their text (238 in any + field): caselaw 98, irb_guidelines 49, I-3.3 27 (+6 French), C-46 11 (+3), + SOR-96-263 7 (+7), enf 6, pdi 3, directives 1 (+2 French) and a dozen more + Acts. They change at the next `ingest`, `ingest --fr`, `caselaw`, `pdi`, + `enf`, `directive [--fr]` and `irb_guidelines`; `canlex.embed` picks them + up by content hash. +- **Line-break hyphens in PDF text.** `_common.dehyphenate(text, words)` + closes up a word hyphenated at a line end by moving the continuation onto + the hyphenated line (line structure is otherwise kept; a line left empty + goes). The hyphen stays ('cross-examine') unless the document's own + vocabulary (`document_words`) has the joined word and the continuation is + lower-case ('immigration'); inside a URL it always stays. A line-final + soft hyphen is a PDF's printed hyphen, so `mark_soft_breaks` turns it into + '-' at extraction, before `norm_ws` would drop it. Wired into ENF (the + chunk text only, after the titles, ids and pins are fixed), the CIU + documents and VSSAs, the benefit booklets' PDF path and the D-memo PDF + fallback. Approximated on the stored text with each document's vocabulary: + 497 chunks change (enf 141, enf-fr 124, dmemos 75, dmemos-fr 60, ciu 49, + benefits 48); 576 breaks keep the hyphen, 65 drop it. Not touched: the + Federal Courts Reports keep `caselaw._join_lines` (its ratio rule), and + `irb_papers._join` still drops a line-end hyphen whenever the next line + opens lower-case (see "Deferred" below). +- **Corpus hygiene test** (`tests/test_corpus_hygiene.py`): no C0/C1 control + character except newline and tab in any string field of any chunk + (passes today), and no FORMAT_CHARS character in any chunk's text -- an + EXPECTED FAILURE until data/processed is regenerated. After the main loop + regenerates the corpus, delete the `@unittest.expectedFailure` line (an + unexpected success fails the suite). LFS pointer files are skipped. +- **Transient fetch errors are retried once.** `_common.with_retry` / + `is_transient`, in `fetch_cached` (urllib and PowerShell paths) and + `ingest.fetch_xml`: two attempts, `FETCH_RETRY_DELAY` (5 s) between them, + for an HTTP 5xx, a URLError without a status, a timeout (urllib or the + PowerShell subprocess), a reset or dropped connection, a truncated body, or + a PowerShell failure whose status line `CANLEX_HTTP_STATUS ` (written by + `powershell_download`'s script from the exception's response) is a 5xx, or + that got no HTTP answer and reports a connection-level error. The status is + never read from the page: Windows PowerShell 5.1's stderr is CLIXML carrying + the error page's text, not '(503)' (verification 2026-09-22, D-2: a 503 + page with a body was not retried, a 404 page quoting '(514) 555-0100' was). + Never a 4xx: `enf.is_missing_upstream` still reads the first 404 as + a retired chapter, and a non-XML body from Justice Laws is refused at once. + The retry prints `retrying in 5s after `. Not changed: + `caselaw._get` keeps its own 403/429 retry; the staleness watch's probes + are not retried (a probe ERROR still fails the check). +- **Atomic marker and cache writes.** `_common.atomic_write_text` / + `atomic_write_bytes` (a temporary file renamed into place) now write the + guidance last-seen marker (`refresh._write_guidance_marker`: a kill + mid-write used to leave truncated JSON that read as {}, re-seeding the ENF + baseline silently), the CIU Important-Decisions marker, the raw case-law + page cache (`caselaw._get`) and the Justice Laws XML cache + (`ingest._download_xml`, already atomic, now through the shared helper). + Other write sites outside this group's files were left alone (see + "Deferred"). +- **French structural labels** (`ingest._FR_LABELS`, `_frenchify`): the + French twins now say ANNEXES, MODIFICATIONS NON EN VIGUEUR / Modifications + non en vigueur / 'Modification non en vigueur — ', '(partie k de n)', + Préambule and 'Annexe (n)' where the parser wrote its own English labels. + At the next `ingest --fr`, 9,371 French legislation chunks change a label + (citation 9,204, part 1,297, heading 452, marginal_note 416, section 1); + about 452 not-in-force chunks re-embed (their heading and marginal note + feed the embedding). Ids do not change; `server._PART_NOTE` already strips + '(partie k de n)'. Every French legislation chunk also gains `section_en`, + its English twin's section label ('55 et 56' -> '55 and 56', 'Annexe 1' -> + 'Schedule 1', 'Formule 6' -> 'Form 6'); three French labels stay unpaired + (C-29 'Annexe'; CRC-604 'Annexes I, Ii Et Iii' and its list's title). + Nothing reads `section_en` yet. Run `py -m canlex.eval` after the French + re-ingest (a French query's ranking of pending law can move). +- **The pending-bill line** (`refresh.pending_consolidation`, + `pending_lines`): see "The staleness watch" above. For C-16 the check now + says the stored Criminal Code (current to 2026-07-21) is on or after the + 2026-07-18 in-force date: no re-ingest needed. +- **`canlex_list_acts`** (canlex/server.py, HELD: `canlex_list_acts` is + REVIEWED): the 'Bilingual' line counts French legislation only. On the + stored corpus: '124 instruments ... (26829 articles)' before, '80 + instruments ... (20680 articles)' after; the rest of the listing is + byte-identical (a before/after run over all 68,560 chunks with a stub + index). +- **Documentation**: the "Environment variables" table (38 names, grep-backed + by `tests/test_docs_drift.py`, CANLEX_HTTP and CANLEX_SEE_ALSO_DRAFT + marked local-only); the choreography's `canlex.irb_guidelines` and + `canlex.conduct` rows, the `--fr` flags of enf and agreement and the + `--allow-shrink` note; the full `RESULT_KEYS` list under "Call log"; the + three lines that cited the a14/a15 before/after harness now describe the + comparison instead of naming files that are not in the repo; the pending- + bill paragraph. The sentence the audit cited as saying the live Space + lacked the call log is already gone ("Maintainer setup" says it carries + it; the test holds that too). +- **Deferred.** `irb_papers._join` drops a line-end hyphen blindly (moving + it to `dehyphenate` changes the IRB papers' text; do it with an eval); + `tools/pit_history.py`'s 404 marker and point-in-time body cache, + `citator.py`'s dbmap and cache and `curated_checks`' draft file are still + plain writes (other owners' files); the `server.py` module docstring still + says "Runs over stdio"; `cp tools/hooks/pre-push .git/hooks/pre-push` is a + maintainer action (the installed hook differs from the committed one). + +### Verification fixes (2026-09-22): F2 -- retrieval, BM25 cache and embedding currency + +- **A treaty's article is no section** (A-PIN-ARTICLE; `_article_cites_section`, + `_TREATY`; see "Retrieval residue" above for the rule). Before, `IRPA 98 + exclusion under Article 1 of the Refugee Convention` pinned IRPA s. 1 'Short + title' to rank 1, `Vienna Convention on Consular Relations article 36 IRPA + detention` IRPA s. 36, `ICCPR article 14 fair trial IRPA` IRPA s. 14 + 'Regulations', `Article 33 of the Refugee Convention ... IRPA s. 115` IRPA + s. 33 at rank 2 and `martial arts 5 years Criminal Code` the Code's s. 5; now + none does (on an IRPA + IRPR index s. 98, s. 57 and s. 115 lead, as before + group A). Every question + of the eval set yields the same section references as before, and `py -m + canlex.eval` (PYTHONHASHSEED=0) printed byte-identical output before and + after: ALL n=339 Hit@1 0.71 MRR 0.81, legacy 0.78/0.87, lookup n=14 + 0.86/0.88, holdout (undated) 0.75/0.84. One English form is given up: a + dotless `art 37` (`IRPA art 37`) no longer pins; `art. 37`, `article 37` and + every French form still do. +- **The BM25 cache** (canlex/bm25.py). A killed writer's `bm25-..tmp` + older than `_TMP_STALE_SECONDS` (600 s) is removed by the next save; a + loaded cache whose arrays do not fit the corpus (a chunk index outside it, + decreasing offsets, a df per term missing, a non-integer array) is rebuilt + (SEC-3). The stemmer class in the code key changes the key once: the first + start after this commit rebuilds the cache (about 14 s), locally and on the + Space. +- **Semantic search off** (A-EMB-2, the index half). `LegislationIndex.semantic` + is a class attribute now (None on a synthetic index), and + `embedding_missing` is None when the query embedder fails to load, so the + listing no longer calls a BM25-only index current. Saying 'semantic search + is off' in canlex_list_acts is the server's half. +### Verification fixes (2026-09-22): F3 -- served-path infrastructure + +The HTTP plumbing in canlex/server.py (`session_manager_with_timeout`, the new +`http_app` that `__main__` serves), canlex/ratelimit.py, the alias-table and +`_analyse` cache in canlex/screening.py, and two served texts. Before/after: +the verification critic's probes, run against this tree. + +- **SEC-1, one client locked everyone out (major).** mcp 1.30.0's session + manager refuses new sessions with 503 once 10,000 are open, whoever opened + them; nothing limited one client. `session_manager_with_timeout` now passes + `max_sessions=SESSION_MAX_OPEN` (None: the 1.27.2 behaviour, bounded by the + 1800 s idle timeout), and `http_app` wraps `_Utf8Body` in + `ratelimit.SessionOpenLimit`, ON by default over HTTP: a POST with no + `mcp-session-id` header (the only request that opens a session) counts + against the client's bucket -- 60 per 10 minutes, bursts of 60, then one + each 10 s; over it, 429 with a plain body and `Retry-After`, and the request + never reaches the manager. Requests inside an open session, `GET /` and + lifespan are never counted. `CANLEX_SESSION_OPEN_LIMIT` sets the number + (`0` = off). Live probe (uvicorn, 127.0.0.1), before, the cap scaled from + 10,000 to 5: one client's six initializes `[200, 200, 200, 200, 200, 503]`, + another client's `503 Too many open sessions`. After, as served with the + per-client cap scaled to 5: one client's twelve `[200 x 5, 429 x 7]`, + another client's 200, six sessions open; and with the critic's global cap of + 5 kept and the per-client cap at 3: `[200 x 3, 429 x 3]`, the other client + 200. The trade-off: with no global cap, memory is bounded per client (at + most 60 + 6 a minute sessions opened, each held until 1800 s idle -- about + 240 in the idle window, 41 KB each for a plain initialize, up to the 1 MiB + body cap for a padded one), not globally; many addresses can still grow it. + **Shared egress:** every client behind one address (an office NAT, or a + hosted MCP client connecting from its provider's servers) shares one + bucket; if officers see 'Too many new sessions', raise + `CANLEX_SESSION_OPEN_LIMIT` in the Space's variables (the startup log prints + the cap). tests/test_http_sessions.py drives the real stack in-process. +- **SEC-2, the rate-limit key.** `client_key` now joins repeated + `X-Forwarded-For` lines before taking the rightmost entry (before: the + client's own first line won -- `['6.6.6.6', '1.2.3.4']` -> `6.6.6.6`; after: + `1.2.3.4`) and cuts the key to `MAX_KEY_CHARS` = 64 (before: a 16,000- + character header was the key; 10,000 such buckets held 162 MB). The module + and `client_key` docstrings, and the rate-limit paragraph above, now state + the appending-proxy assumption. Not changed: the header is still trusted + from any peer (a forged header from any peer is still the key; rotating it + still dodges the limit off Hugging Face) -- no trusted-proxy list was added. +- **B-CACHE-1, a stale statute-code reading outlived an alias edit.** + `load_aliases` bumps `_ALIAS_GENERATION` under `_ALIAS_LOCK` when it + installs a table, and `analyse()` passes it to `_analyse` as part of the + cache key, so an expansion a concurrent call computed from the old table + lands under a key no later call reads. Probe (thread A paused inside + `_alias_reading` while `{'zzq 123': 'criminal trespass'}` was replaced by + `{'zzq 123': 'burglary'}`), before: every later call `criminal trespass + docket 925 lorem`; after: `burglary docket 925 lorem` (the straddling call + itself still reads the old table). The stale comment above + `_INCHOATE_STEM` ('cleared, per tool call') and the concurrency + assumption in group B's section are corrected. +- **BI-3, canlex_toc's description.** It said the IRPR is listed in full; at + 569 sections it is over `TOC_FULL_LISTING` = 400 and comes back as 24 range + lines with the `part=` hint. The description now names only IRPA (285 + sections) as listed in full and the IRPR among the larger Acts; the + threshold and every outline are unchanged. `canlex_toc` is REVIEWED, so its + new description is HELD until approved. +- **A-EMB-2 (server half), the list_acts embeddings line.** An index whose + `semantic` is False now reads `_Embeddings: semantic search is off (keyword + search only)._` (before, with the embedder failing offline: `every chunk has + its stored vector (semantic search is current)`); with every vector present + it reads `_Embeddings: every chunk has a stored vector._` -- currency by + content is `canlex.embed.currency`'s to count, not this line's. The index + half (`embedding_missing` left at 0 when the embedder fails) is + canlex/index.py's, outside this group; the line no longer depends on it. +- **Answers unchanged.** 3,600 screening calls (the byte-identity lens's + workers: 300 corpus records through screen_record, screen, + screen_history, us_equivalency and us_disposition, and 200 generated 2-4 + entry records through canlex_screen card/full and canlex_screen_history) + before and after these changes: identical. +### Verification fixes (2026-09-22): F4 -- deploy tools, ingest helpers, documentation + +The implementation verification of the audit follow-up (critic verdicts +A-EMB-3, C-GATE-1, C-GATE-2, D-2, D-3, D-5, C-DOC-4, C-DOC-5). Tooling and +ingest plumbing only: no answer, curated file, corpus file or embedding +changes. The corrected RUNBOOK lines are edited in place above (group A's +"Embedding currency", group C's mcp, OSV and test-list paragraphs, group D's +retry bullet); what is new: + +- **predeploy's embeddings check (A-EMB-3, C-GATE-3).** A processed corpus + file that does not parse (truncated JSON, a byte-order mark, not UTF-8, + not a JSON list) now FAILS the check naming the file -- the Space's index + reads every `data/processed` file at start. A corpus that cannot be read + here (an LFS object not in the local store) WARNs instead of PASSing, the + same as an npz absent from the store. On the tree at 8c22978, unpatched: + PASS, 'whole corpus: 0 missing, 0 stale vectors'. +- **The deploy gate's own tests (C-GATE-1).** `tests/test_predeploy_gates.py` + now runs the real `run_unittest` (subprocess answer faked) and the real + `check_curated_working` (`refresh.check_curated` rows faked: drift and + error FAIL, aging WARNs), alone and through `main`. Mutating either gate + to always PASS now fails 3-4 tests; before, the whole suite stayed green. +- **osv_check refuses a malformed answer (C-GATE-2).** A querybatch answer + that is not a list of exactly one result object per query (a missing or + null `results`, a short or long list, a paged follow-up with none) exits 2, + 'the OSV API could not be queried: ValueError: malformed OSV querybatch + answer ...'. Before, it was zipped against the queries and printed + 'PASS -- 0 advisories on 0 of 6 packages', exit 0, for six vulnerable pins. +- **Atomic writes wait out a brief hold (D-3).** On Windows a rename onto a + file another process holds open (Python's `open()`, a scanner, an indexer, + an editor) fails with [WinError 5] Access is denied, where the plain + `write_text` the atomic helpers replaced succeeded. `atomic_write_text` / + `atomic_write_bytes` now retry the rename `REPLACE_ATTEMPTS` (5) times, + `REPLACE_RETRY_DELAY` (0.2 s) apart; a hold that outlasts that (about + 0.8 s) still raises PermissionError, the old file whole and no temporary + file left. If `py -m canlex.refresh` or `canlex.caselaw` stops on it, close + whatever holds the marker or cache file and re-run. +- **pypdf 6.16.2 splits some words (D-5).** The upgrade (7499848, pypdf + 6.11.0 -> 6.16.2) was first realised by 486cfc9's corpus regeneration: + `enf-29-app-c` now reads 'Recommend ex amination for' where it read + 'Recommend examination for'. Raw pypdf 6.16.2 on `data/raw/enf/enf29.pdf` + gives 'Recommend ex amination' once and 'Recommend examination' zero times, + so no ingest code caused it; tokenized, 'ex amination' is 'ex' + 'amin', so + BM25 loses 'examination' at that spot. The verification also found + `enf-fr-17-8-1-p1` 'requises' -> 'requi ses' and `ciu-regs-12-p1` + 'one-page' -> 'one- page', and some breaks the new version closes ('1 st' + -> '1st'). Accepted and recorded here; the next pypdf upgrade follows the + checklist below. +- **pypdf-upgrade checklist** (add to any upgrade of `pypdf` in + `requirements.txt` / `requirements.lock`): + - [ ] Before committing a corpus regenerated under the new pypdf, + re-extract every PDF-derived source (ENF and ENF-fr, the CIU documents + and VSSAs, delegation, the benefit booklets' PDF path, the D-memo PDF + fallback) with the old and the new version, and diff them word by word + (tokens split or joined, not only changed lines). + - [ ] Record the splits and joins here, with the chunk ids, and decide: + accept, pin the old version, or repair at extraction. + - [ ] Run `py -m canlex.eval` on the regenerated corpus. +### Verification fixes (2026-09-22): F1 -- the officer-facing legal readings + +HELD for the maintainer's review: these answers change on purpose. The rule +for all three readers is the same -- they are conservative, and do not try to +recognise every phrasing: what they cannot place goes the strict way. + +- **Drug amounts (LEGAL-1/2/3, then verification round G1: Q1-Q11, Q13; + `screening.settled_quantity`, `select_limbs`, `possession_limit_open`; + tests/test_verification_g1.py).** F1 opened the quantity when the text + carried a word from a list (`_AMOUNT_QUALIFIER`: over, at least, approx., + about, each, per, gross, '+', '<' ...). A list of doubtful words is never + complete: the adversarial pass on F1 ran 678 realistic phrasings and 189 of + them still settled the 3-kg-or-less limb ('est. 2.9 kg', 'GT 2 KG', 'five + packs of 1 kg', '2 kg sample of larger load', 'four kilograms 200 grams', + 'THC concentrate 500 g', '4.000 g'). The reader is now an ALLOWLIST. An amount + settles a quantity limb only when (1) the offence text holds exactly ONE + amount that `quantity_kg` reads -- its word list stays, as a pre-check that + only opens, and a dot followed by exactly three digits ('4.000 g', '2.500 kg', + '3.000,5 g') names no amount, being a decimal or a continental thousands + separator (Q9); (2) outside that amount the text has no digit and no symbol + (letters, spaces and ( ) . , ; : ' " / only); (3) its words, read as the + matcher reads them, are nothing but the pairing's own offence words + (`quantity_terms`: its terms and its limbs' terms, never a term that itself + names a number, a unit or a qualifier), the two substances former Schedules + VII and VIII name (cannabis resin, cannabis (marihuana)) and 'a', 'an', + 'the', 'of', 'in' -- a function word the terms use ('to', 'with', 'for') + counts only inside a run of words the terms write in that order ('with intent + to distribute'), and the one inchoate cue ('attempted') and the military + jurisdiction ('UCMJ') the matcher reads are set aside; and (4) the text does + not end on a function word after the amount ('2 kg to', '2 kg of': cut off). + Everything else is 'NOT SETTLED -- how much was involved?', both readings + shown. The derived words and the shorthand that reaches them are pinned by + tests/test_verification_g1.py. The instruction now asks the officer to + 'write the EXACT amount the record states ... a bound ('over 5 lbs'), an + estimate ('approx. 2 kg') or a part of the total cannot be written as an + amount' (Q10/Q13; the facts are still not read for the quantity). + Former CDSA Schedule VII listed only 'Cannabis resin 3 kg' and 'Cannabis + (marihuana) 3 kg', and s. 5(4) (later s. 5(3)(a.1)) reached a Schedule II + substance 'in an amount that does not exceed the amount set out for that + substance in Schedule VII': THC and the other preparations + (`SCHEDULE_VII_FORMS`: oil, concentrate, extract, wax, shatter, dab, vape, + cartridge, edible, tincture, rosin, budder, distillate) never settle the + 3-kg-or-less limb -- the card asks for the substance's form and quotes + Schedule VII -- while an exact amount over 3 kg still settles the over-3-kg + limb, s. 5(3)(a) either way ('edibles 5 kg', Q5). canlex_us_equivalency + settles a split part's amount only when every other part is a plain offence + of its own (`plain_offence`): 'DUI; sale of marijuana 2 kg' settles, 'sale of + marijuana 2 kg and over' does not, and the multi-offence refusals of + canlex_screen_record and canlex_screen_history now say not to enter the + amount alone (Q4). Cannabis possession keeps NO CANADIAN FEDERAL EQUIVALENT + only for no amount in plain offence words, or ONE exact amount within former + Schedule VIII (30 g; resin 1 g); otherwise DEPENDS -- 'was the cannabis within + the Canadian limits?', quoting former CDSA s. 4(4)/(5) and Cannabis Act + s. 8(1)(a)/(b), 8(2)(a) (Q6). A permanent resident gets that NOT SETTLED too, + as for every depends-on-elements pairing, although neither reading reaches + s. 36(1). The audit fixtures read as before ('5,000 grams', '5 000 g', '2,5 + kg', '1/2 oz', '1 1/2 lbs', '2 kg', '4 kg'; tests/test_tran_limbs.py + unchanged). Measured on the quantity lens's 678 phrasings (worker.py, three + calls each -- a permanent resident and a foreign national in 2015, a foreign + national in 2019 -- 41eaf96 against this branch): lenient 3-kg-or-less + settles 189 -> 0 (approximation 69, repetition or count 51, lower bound 36, + Schedule VII form 12, partial weight 9, dot notation 7, written number 2, + other 3); 504 calls moved from settled to NOT SETTLED, the strict direction -- + 48 of the lens's 75 'exact' phrasings now open (statute citations, dates, + counts, case numbers, 'net weight', 'in total', '5 lb's'; several were not + exact after all: 'appx 2.9 kg', '2 kg/wk', '(wet weight)', 'criminal sale of + marihuana 2nd degree 16 oz'), and 7 of 22 over-3-kg settles open ('LT 50 LBS', + Q11; '3.500 kg'; 'GT 25 LBS'); 246 calls of 2019 keep the same branch with the + limb no longer named; no call moved from open to settled. Side probes: 23 + split phrasings in canlex_us_equivalency settled -> 0; 11 of 26 shorthand + phrasings -> 0; 12 of 12 over-limit possession runs NO -> DEPENDS. Q12 (read a + statute citation, a date or '1-1/2' beside an exact amount) was closed in round + J2 below; Q14 (Texas 'DEL MARIJ' aliases) is still open, and those records stay + NOT SETTLED. +- **Drug amounts, the holes in that allowlist (verification round H1: + GQ-1..GQ-6; `screening.quantity_limb_terms`, `one_offence_clause`, + `fold_erases_word`, `quantity_context_open`, `plain_offence`, + `possession_limit_open`; tests/test_verification_h1.py).** The adversarial + pass on G1 confirmed six lenient classes the allowlist of WORDS could not + see, each settling the 3-kg-or-less limb (a permanent resident 'NOT REACHED', + a foreign national 'CRIMINALITY ... DEEMED REHABILITATED'). Nothing new is + recognised: each fix narrows the allowlist, requires structure, or opens the + reading. + (GQ-1) A part of a split record whose content words are ONLY substance words + is no offence of its own -- `expand()` rewrites a bare substance to + 'possession of ' (CANNABIS_BARE_NOTE), so 'hashish', 'THC' and + 'hash' passed `plain_offence` and 'sale of marijuana 2 kg and hashish' + settled the first part while reading the second as simple possession with no + equivalent; following the refusal gave the PR 'nothing on this record engages + s. 36' and the FN DEEMED REHABILITATED. The amount is now held + (`quantity_held`) and the refusal carries SPLIT_AMOUNT_CAUTION. + (GQ-2) The allowlist is derived from the QUANTITY limbs' terms only + (`quantity_limb_terms`), not from `quantity_terms`' us_terms and every limb: + 'manufacture', 'cultivation' and 'growing' belong to the production limb + (former CDSA s. 7(2)(b): 7 years to 2012-11-05, 14 after), so + 'manufacture delivery of marijuana 2 kg' and 'growing sale of marijuana 2 kg' + leave the quantity open instead of settling trafficking and dropping the + production reading. A record naming production alone still reads as + production. + (GQ-3) The amount must sit in ONE offence clause (`one_offence_clause`): + only one clause -- a newline, '.' or ':' ends one -- may name offence words, + no term's head word may occur twice, and the record may name at most one + substance group. Before, 'sale of marijuana 2 kg\nsale of marijuana' (two + counts), 'sale of 2 kg of marijuana with hashish' (a second substance with no + amount) and 'sale of marijuana 1 lb a sale' / '1 lb/sale' (an amount per + transaction) all settled. + (GQ-4) A character `_fold` would erase opens the quantity + (`fold_erases_word`): `_plain_chars` accepts every Unicode letter while + `_fold` keeps only a-z, digits and ( ) . / & ', so '2 kg 以上' ('or more'), + '約 2 kg' ('approx.') and '(приблизительно)' vanished before the allowlist + saw them. Accents still fold, so an accented Latin record is read, not opened. + (GQ-5) The facts and the disposition are read for the quantity + (`quantity_context_open`, `server._quantity_context`): an amount there that + the offence text does not state, or a qualifier beside an amount or a weight + word, opens it -- the officer's documented answer 'quantity: over 3 kg' in + the facts, 'the traveller admits selling more than 5 kg in total' and + 'convicted -- lab weight 10 kg' each left a 3-kg-or-less settle standing with + 'The answer changes with: none found'. They never settle a quantity, only + open one; CanLex still reads the amount from the OFFENCE TEXT alone + (QUANTITY_CONTEXT_NOTE says so). Known strict cost: a money amount in the + disposition written in pounds ('FINE 200 POUNDS') is read as a weight and + opens the quantity. + (GQ-6) Possession with NO amount is DEPENDS for a resin or concentrate word + ('hashish', 'possession of cannabis resin', 'possession of THC'): the former + Schedule VIII line for resin is 1 g, and the Cannabis Act counts a + concentrate against the 30 g limit as its dried equivalent (Schedule 3), so + the words do not show the possession was within either. Plain marijuana + possession with no amount keeps the curated NO (Q6). The question is now + printed on the canlex_screen and canlex_screen_history cards too + (`server._possession_limit_notes`, CARD_POSSESSION_LIMIT_BRANCH), which had + shown only 'NO CANADIAN FEDERAL EQUIVALENT' or 'NOT SETTLED (value, schedule + or elements)'. + Measured. The quantity lens's 678 phrasings x 3 calls (f6a0f83 against this + branch): 2,034 of 2,034 identical -- the H1 fixes cost nothing on that + corpus, lenient 0, and the 42 remaining settles are the exact amounts G1 + accepted ('2 KILOGRAMS', '2,5 kg', '1 1/2 lbs', '5,000 grams', '105 oz', + 'cannabis resin 2 kg', 'edibles 5 kg'). Identity harness (worker.py over the + 3,601-record corpus, worker2 and the three worker3 variants, 42,610 calls): + 42,328 identical, 282 differing, LENIENT 0. Of those, 14 are answer moves on + 5 records, all GQ-5 reading the disposition -- '<2OZ', '<1G' and '<28G' in a + Texas charge line, and 'FINE 200 POUNDS' -- which turn a settled 'no + equivalent' into the limits question, and a settled serious-criminality or + criminality quantity into NOT SETTLED; 268 are wording (266 the GQ-6 card + note and entry label on entries whose limit was already open, 2 the + confidence 'maximum_source' where the limb is no longer named). + Not closed: two substance words juxtaposed with no separator ('sale of + marijuana hashish 2 kg') still read as one substance group, because + 'cannabis resin' has the same shape and a name list would be recognition; + every separator the splitter reads (',', ';', '/', '&', 'and') already holds + the amount. +- **Canadian record suspension or pardon -- an ALLOWLIST (G2, 2026-09-22; + replaces F1's LEGAL-4/5 word lists; `rehab.relief_reading`, + `record_suspension_in_force`, `relief_dates_stand`, `canada_disposition`).** + F1 read relief with lists of losses, negations and neutral tokens, and the + adversarial pass on it still found 151 phrasings (of 291 that do not say a + record suspension stands) answered NOT A GROUND or 'nothing on this record + engages s. 36' in some tool: a later 'not revoked' revived a ceased relief + ('pardon ceased to have effect 2010, not revoked', REL-1), an unplaceable + later clause left it standing ('pardon granted 1998 - rev. 2004', 'pardon + granted 1998; convicted of robbery 2003', 'Pardon granted - N', REL-4 group), + and neutral tokens stood ('Pardon:', 'Record suspension: --', 'pardon may have + been granted', 'pardon 1999-2004', REL-7 group). The reader now settles only + on a positive grammar. A record suspension stands (s. 36(3)(b), NOT A GROUND) + only when the relief words, TAKEN AS A WHOLE -- the disposition's relief + clauses (from the first clause naming a record suspension or pardon to the + end) and the relief key/field read together, never one after the other + (REL-3) -- are ONE naming clause (`_CANADA_RS_NAMING`): + `record suspension | pardon | pardoned` `[granted | ordered | issued | in force]` + `[[on | in] YYYY-MM-DD | YYYY]`, with a grant word, a date or 'pardoned' (a + bare 'Pardon' or 'Record suspension' is an empty form field); nothing before + it in the disposition but the conviction words (`_CANADA_RS_CONVICTION`: + convicted, found guilty, pleaded/pled guilty, guilty plea, guilty, + conviction, with one date or year); nothing after it but a keep clause + (`_CANADA_RS_KEEP`: 'not revoked', 'has not been revoked or ceased to have + effect', 'which has not been revoked' -- it keeps a relief that stands and + never makes one; 'not revoked and ceased' is no keep clause); the same + naming clause in both the disposition and the key counts once, two + different ones never stand. Then it is placed against the entry: dated + before the offence, the conviction or the sentence's completion, or after + the assessment date, it is not this conviction's (REL-11; CRA s. 4(1) bars + an application until after the sentence expires); a year in the + disposition's own words that is not before the relief's ('convicted 2010; + pardon granted 2009') takes it away; with the relief in the key alone, a + disposition clause carrying another relief word, a loss word or another + conviction's words ('RS revoked 2019', 'subsequent conviction') takes it + away. Everything else is NOT in force: the conviction is screened as it + stands and `RECORD_SUSPENSION_NOT_IN_FORCE_TEXT` states the grammar and asks + for 'record suspension ordered ' as the entry's relief + (canlex_screen, canlex_screen_history) or as the disposition 'convicted; + record suspension ordered ' (canlex_screen_record); it now also + prints for shorthand the grammar does not read -- 'RS', 'R/S', 'R.S.', + 'record susp.' (REL-13). NOT SETTLED instead, the conviction staying a + conviction and only the relief open (`relief_open`): a bare 'pardon' or + 'pardoned' whose date may fall after 2012-03-12 (AI-4: + `PARDON_AFTER_2012_TEXT`, quoting the Criminal Records Act's definition of + pardon '[Repealed, 2012, c. 1, s. 109]', 'record suspension means a measure + ordered by the Board under section 4.1' and s. 9; the corpus has s. 4.1 in + force from 2012-03-13; card question `CANADA_PARDON_AFTER_2012_QUESTION`), + and a record suspension that another conviction on the record may post-date + (REL-5: `RELIEF_LATER_CONVICTION_TEXT`, quoting CRA s. 7.2(a) and s. 7(a); + `server._canada_later_convictions` -- another Canadian entry that is or may + be a conviction counts unless its conviction date, or failing one its + sentence completion, falls before the record suspension's date; an undated + record suspension beside any other Canadian conviction is open; in + canlex_screen_record `other_record` or a count other than 'one' opens it, + and the decisive-facts block names other_record). A relief key naming a + record suspension or pardon, the Board, the Act or 'RS' presupposes the + conviction whether or not it stands (as the key did when it was appended and + read), so an entry never leaves the record's counts or s. 17 clock because + its relief no longer stands. The key is no longer wrapped as 'record + suspension (...)': Parole Board or CRA words alone print + CANADA_RELIEF_NOT_APPLIED, 'which names no record suspension or pardon' + (REL-12). The F1 word-list definitions are gone (`_CANADA_RS_NOT_IN_FORCE`, + `_CANADA_RS_GRANT`, `_CANADA_RS_NEGATION`, `_CANADA_RS_NEUTRAL`, + `_CANADA_RS_NEGATED_LOSS`, `_CANADA_RS_LATER_GRANT`, `_rs_clause_stands`); + `_CANADA_RS_LOSS` survives only as a strict-only check beside a key-only + relief. Strict costs, accepted: 'RS: Y', 'Pardon: Yes', 'record suspension + refused 2018; granted 2021', 'record suspension revoked; reinstated', + 'pardoned by the National Parole Board in 2009', 'record suspension ordered + 2015-03-02 (PBC file 12345)', 'Record suspension granted Sept. 3, 2015' and + 'pardon granted 1998; no subsequent convictions' are now screened as they + stand (the note says how to re-run); 'convicted; conditional pardon 1990; + record suspension ordered 2019' is the prerogative NOT SETTLED reading, no + longer NOT A GROUND. +- **Prerogative pardon (LEGAL-6; `rehab.prerogative_pardon`, `PREROGATIVE_PARDON_TEXT`).** + 'conditional pardon', 'free pardon', 'royal prerogative', 'prerogative of + mercy', 'Governor General', 'Governor in Council' or 's. 748' beside pardon, + mercy or clemency is not a Criminal Records Act record suspension: the + Canadian entry is NOT SETTLED, quoting IRPA s. 36(3)(b) and Criminal Code + s. 748(2)/(3) from the corpus (tests/test_verification_f1.py checks the + quotes). The record card asks `CANADA_PREROGATIVE_QUESTION`; the screen and + history cards print `CANADA_PREROGATIVE_NOTE`. G2: the words anywhere in the + relief -- in two clauses too ('conditional pardon, granted 2010', 'Pardon + granted 2010; Royal Prerogative of Mercy', PREROG-1: NOT A GROUND before) -- + give this reading, and a record suspension beside them no longer bars the + ground (not ONE naming clause). G2 also keeps the entry a conviction + (`relief_open` 'prerogative'; F1 made it a conviction whose status was not + settled, and the record's s. 17 clock dropped it -- PREROG-2): the record's + counts and s. 17 clock read it, and the card note says so. + G3 (PREROG-4): + `PREROGATIVE_PARDON_TEXT` had said 'each reading is shown (the conviction + screened as it stands, and no ground on it)', but only canlex_screen_record + drafts the no-ground reading; it now says what every tool prints -- 'the entry + is screened as the conviction it names' -- and the notes and the record card's + question say 'the royal prerogative of mercy or Criminal Code s. 748' (Criminal + Code s. 749 keeps the prerogative apart from the s. 748 power; they had said + 'the royal prerogative of mercy (Criminal Code s. 748)'). The history card + whose only possible ground is such an entry (or any conviction in Canada + whose branch is open) now answers NOT SETTLED with the entry's question; it + had printed 'Action: no s. 36 ground on this matter'. +- **The relief key (BI-5, G2; `server._canada_relief`).** A Canadian entry's own + relief (canlex_screen's key, canlex_screen_history's field) is applied only + when its words alone are ONE naming clause of the grammar dated on or after + the entry's offence, conviction and sentence completion (a prerogative + pardon goes to the NOT SETTLED reading); anything else prints + CANADA_RELIEF_NOT_APPLIED ('', on or after the offence date). + Whatever is applied, the disposition reads the key TOGETHER with its own + words (`_canada_relief_ctx` on the entry; `server._canada_entry_disposition`), + so a revocation in the disposition still ends a grant in the key, and the + card prints the flag when the two together do not stand + (`_canada_card_relief_notes`). Before (F1): the key was applied whenever the + word-list reader said True -- 98 lenient key phrasings (REL-2) -- and was + appended after the disposition, so the last clause decided (REL-3). +- **Measured (G2).** The canadian-relief lens's full phrase lists + (tests/data/relief_phrases_g2.json: 291 NOT, 30 PREROG, 36 OTHER, 100 + STAND; offence 2005-01-01) through probe.py, on 41eaf96 and on G2: lenient + (a NOT phrasing answering as a record suspension in force) 151 -> 0, in each + path (canlex_screen_record 145 -> 0, canlex_screen relief key 151 -> 0, its + disposition 145 -> 0, canlex_screen_history field 151 -> 0, its disposition + 145 -> 0); PREROG in force 1 -> 0; strict (a STAND phrasing not in force) + 38 -> 88 of 100 -- 3 correct reads (a pardon dated before the offence) and + 85 over-reads of wording outside the grammar, all with the re-run note. + probe3.py, probe4.py and the critic's r1.py/r2.py: every REL-1, REL-3, + REL-4 group, REL-5, REL-7 group, REL-11, REL-12, PREROG-1 and AI-4 repro now + answers a ground or NOT SETTLED; tests/test_verification_g2.py pins them + (each test fails on 41eaf96). Every phrasing the grammar can build (10,800 + from its parts, generated) was read on both trees: 292 stand now that did + not on 41eaf96, all one family, ' which has not been + revoked' (REL-14's strict reading, undone -- it only adds 'not revoked' to a + clause that stands alone, so it cannot over-read); 280 stood on 41eaf96 and + do not now (a bare 'pardon' or 'record suspension' after the conviction + words). The answer-identity harness (41eaf96 exported with git archive + against G2, stub index): the corpus run (3,601 records x 10 calls) is + identical -- its 11 Canadian records carry no relief wording; worker2's 1,800 generated + calls differ on 246 (144 a Canadian entry moves from record-suspension to a + ground or NOT SETTLED -- the grammar 117, REL-11 21, REL-5 9 entry-level; + 102 wording only, the new not-in-force note); a worker3 run that adds + Canadian relief keys and the wordings above (AI-7) differs on 595 of 1,800 + (507 record-suspension -> ground or NOT SETTLED -- grammar 444, REL-11 54, + PREROG-1 45, AI-4 18, REL-5 18, REL-12 6; 82 wording only; 3 NO CONVICTION + -> s. 36(1)(a), a quashed conviction whose relief key names a record + suspension; 3 ELIGIBLE TO APPLY -> UNDETERMINED, a prerogative-pardoned + conviction with no completion date now on the s. 17 clock). No card's + Action moved to a less adverse one (17 'no s. 36 ground' -> a s. 44(1) + report, 2 -> NOT SETTLED, 1 NOT SETTLED -> report) and no earliest + application moved earlier. An earlier cut of G2 that left an open relief's + conviction None moved four records' earliest application earlier (the + entry left the s. 17 clock); the conviction now stays a conviction. +- **The relief grammar's holes (H2, 2026-09-22: G2-REL-A..H; `rehab._rs_naming`, + `relief_reading`, `relief_dates_stand`, `pardon_after_2012`, + `relief_before_cra`, `relief_unplaced`, `canada_disposition`; + `server._canada_later_convictions`, `_jurisdiction`, `_canada_place`; + tests/test_verification_h2.py).** The adversarial pass on G2 found the + allowlist still settling eight ways, each LENIENT (NOT A GROUND, or 'nothing + on this record engages s. 36'). Each is closed by narrowing the grammar or + opening the reading, never by recognising more wording. + - **G2-REL-A, an UNDATED 'pardon' / 'pardoned'.** `pardon_after_2012` now + reads a MISSING date as 'may fall after 2012-03-12': 'convicted; pardoned' + answered NOT A GROUND in all four paths while 'convicted; pardoned 2016' + answered NOT SETTLED, so strictly less information gave the more + favourable answer. NOT SETTLED now, `relief_open` 'pardon-undated' + (`PARDON_UNDATED_TEXT`, `CANADA_PARDON_UNDATED_QUESTION`). + - **G2-REL-B, a relief dated before the Criminal Records Act.** 'pardon + granted 1965' (and, with no entry dates for REL-11 to place them against, + 'pardon granted 0001' and 'record suspension ordered 1000') stood as + s. 36(3)(b) record suspensions. The naming clause's date is now floored at + `_CRA_FIRST_DAY` = 1985-01-01, and the reading is opened, not settled + (`relief_open` 'relief-before-cra', `RELIEF_BEFORE_CRA_TEXT`). The corpus + carries NO enactment date for the Act -- the history line of its s. 1 is + 'R.S., c. 12 (1st Supp.), s. 1', a Revised Statutes supplement with no + date, and the earliest date its own history lines carry is 'R.S., 1985, + c. C-47, s. 2' -- so the note says exactly that and asks the officer to + confirm the instrument. The floor is deliberately LATER than the 1970 the + critic's fix implied: 1970 is nowhere in the corpus, and the strict side of + an uncertain floor is the later one. Nothing in the repo or the gold rows + carries a relief dated before 1985. + - **G2-REL-C, the relief key's disposition.** With the relief in the entry's + relief key the disposition was screened by four word lists, not by the + conviction allowlist: 13 of the lens's 17 clauses saying the relief had + ended or that a later conviction followed ('has been convicted since', + 'suspension terminated', 'set aside by the Board', 'suspension pending') + left the key standing in both key paths, while the same words written + inline in the disposition were refused 17 of 17. `relief_reading` now + applies `_CANADA_RS_CONVICTION` to the disposition beside a key-only relief + too, the word lists staying as a refusal on top. Accepted strict cost: a + disposition carrying a sentence or an appeal clause beside a key ('pled + guilty, 12 months probation') stops the key standing; + `CANADA_RELIEF_NOT_APPLIED` now also says to give the disposition as the + conviction words alone. + - **G2-REL-D, a bare date and no grant word.** 'record suspension 2019', + 'pardon 1998', 'record suspension on 2019-05-01' settled the relief as + ordered and in force, one punctuation mark from the refused control + 'record suspension: 2019'. `_rs_naming` now needs a grant word or the + participle 'pardoned' -- IRPA s. 36(3)(b) reaches a conviction only where a + record suspension 'has been ordered', the same reason a bare 'Record + suspension' never stood. tests/test_verification_g2.py, test_verification_f1.py + and test_canada_youth_finality.py pinned the old reading; each now pins the + new one and says so. + - **G2-REL-E, an undated record suspension.** `relief_dates_stand(None)` is + True, so REL-11 never ran on it and one ordered for an earlier conviction + stood on this one. `canada_disposition` now opens it ('relief-undated', + `RELIEF_UNDATED_TEXT`, quoting CRA s. 4(1)) whenever the entry carries an + offence, conviction or completion date to place it against; with no entry + date at all it stands, as before. + - **G2-REL-F, a year-only relief in the current year.** All three REL-11 + comparisons used the EARLIEST day the date can mean -- the strict side + against the entry's own dates, the lenient side against today, so 'record + suspension ordered 2026' stood on 2026-09-22. `relief_dates_stand` now + compares the LATEST day with today and keeps the earliest for the rest. + - **G2-REL-G, a later Canadian conviction the reader could not see.** + `_jurisdiction` reads 'CAN', 'CA', 'Can.', 'Canada (Ontario)' and a state + wrapped in the country ('Ontario, Canada', 'ON, Canada') as Canada + (`_canada_place`; a wrapper counts only around a province the tool reads, + so 'Santa Ana, CA' and 'Canada Bay' are untouched and a US state value 'CA' + is still California). `_canada_later_convictions` also counts another + entry whose OFFENCE date falls on or after the relief's earliest day: a + conviction cannot precede its own offence, so an offence of 2012 typed + beside a completion of 2009 is a possible later conviction. The offence + date only ever ADDS an entry -- it bounds the other conviction from below. + Bounding the entry by `max(conviction, completion, offence)` instead, as + the first cut did, turns an UNKNOWN upper bound into a known one and is + lenient: the identity harness caught 23 calls moving INTO + 'record-suspension' (an entry with an offence date and no conviction date + stopped counting), and tests/test_verification_h2.py pins the case. + - **G2-REL-H, a second undated conviction clause.** 'convicted 2005; + convicted; record suspension ordered 2010' stood -- the one wording the + conviction allowlist cannot place in time. Every conviction clause after + the first must now carry a year `_CANADA_RS_YEAR` can compare. + - Each open reading leaves the entry a CONVICTION and only the relief open + (G2's PREROG-2 rule), so the record's counts and s. 17 clock still read it; + the card asks that reading's own question and prints its note + (`_CANADA_RELIEF_OPEN`, `CANADA_RELIEF_OPEN_NOTE`, `_RELIEF_OPEN_LINES` for + the readings that are not only about a pardon). + `RECORD_SUSPENSION_NOT_IN_FORCE_TEXT` adds the two new ways a relief fails + to stand ('a year that may not have run out by the assessment date', a + disposition clause beside a key) to the list it already prints. +- **Measured (H2).** The relief-grammar lens's own generator (every phrasing the + grammar can build from its parts x six conviction prefixes, 214,200, read with + entry dates that expose each family): settled as a record suspension in force + 125,970 -> 48,450; 77,520 phrasings moved OUT and 0 moved in; undated pardon + 18,360 -> 0, pre-Act 22,950 -> 0, bare date 21,420 -> 0, undated record + suspension 8,160 -> 0, current year 7,650 -> 0. Its tool probe (each accepted + phrasing through canlex_screen_record with and without dates, and + canlex_screen / canlex_screen_history in the disposition and in the relief + key): in force per path 2,310 -> 60, the 60 being an undated record suspension + in canlex_screen_record with NO dates given (REL-E's documented scope); the + 'clean' phrasings hold at 1,710 per path and none moved into force. Its + key-only probe (17 clause groups x 2 keys x 2 tools): key-path in force + 118 -> 0, the four controls and the disposition path already 0. The + canadian-relief lens's full lists (291 NOT, 30 PREROG, 36 OTHER, 100 STAND, + and the 94-row second list) in all five paths: lenient 0 -> 0 (G2 had cleared + it) and none appears; strict on STAND 88 -> 91, the three being undated record + suspensions now placed against the entry's 2005 offence. The answer-identity + harness (f6a0f83 exported with git archive against this branch, stub index, + 42,610 calls a tree: the 3,601-record corpus x 10 calls, worker2's 1,800 + generated records and the G1, G2 and G3 worker3 variants): the corpus run is + byte-identical (its 11 Canadian records carry no relief wording); worker2 + differs on 192 of 1,800 and the G1 variant on 200 of 1,800, wording only (the + not-in-force note's new tail); the G2 relief variant differs on 558 of 1,800 + (9 answer moves, all a Canadian entry leaving 'record-suspension' for NOT + SETTLED -- G2-REL-C, a relief key beside 'appeal allowed; new trial ordered'; + 549 wording only) and the G3 variant on 24 of 1,200 (4 answer moves: 2 + G2-REL-C, 2 G2-REL-F; 20 wording only). 13 answer moves in all, every one in + the strict direction, 0 lenient, and no record's earliest application or + earliest deemed date moved at all. + +### Verification fixes (2026-09-22): J2 -- the quantity allowlist's last hole, and the relief residue + +The three items section 6 of the implementation memo left open in the quantity +and Canadian-relief readers. All three stay inside the ALLOWLIST rule the whole +codebase follows: a reader settles an officer-facing answer only on a small +positive grammar it is sure of, and everything else answers NOT SETTLED or is +screened as it stands. Everything here is HELD by the review gate. + +- **Q12 -- a record-keeping token beside an exact amount (`screening.quantity_strip`, + `_citation_designators`, `_citation_spans`, `_Q_DATE`, `_Q_DOCKET`, `_Q_COUNT_ID`, + `_Q_BARE_CITATION`, `_MIXED_HYPHEN`, `_Q_CODE_DESIGNATORS`; + tests/test_verification_j2.py).** G1's + allowlist asks that outside the one amount the raw text hold no digit + (`_plain_chars`), so an EXACT amount beside a statute citation, an ISO or slash + date, a docket or case number or a count identifier stayed NOT SETTLED although + the offence's own words settle it -- 'SALE OF CANNABIS 893.13(1)(a)(2) 2 KG', + 'sale of marijuana 2 kg 2015-06-01', 'sale of marijuana 2 kg on 06/01/2015', + 'sale of marijuana 2 kg Case 2015CF001234', 'sale of marijuana 2 kg Count 1'. + Nothing new is recognised as an amount or as an offence word: the tokens are + RECOGNISED and REMOVED before the allowlist reads the words, so what it reads is + the offence's own words plus one amount, and a token the grammar cannot classify + still opens the quantity. The citation designators are an EXPLICIT ALLOWLIST of + US statute-code designators (`_Q_CODE_DESIGNATORS`, intersected with the alias + table so a designator the table does not carry is still no citation): 'A MRSA', + 'HS', 'HSC', 'MCL', 'PC', 'TX PC', 'PL', 'RCW', 'UCMJ article', 'USC', 'VC', + 'VSA', 'VTL' -- dotted initials included ('21 U.S.C. 841(a)(1)') -- and NOT + 'ORS', 'KRS', 'ARS', 'G.S.' or 'R.C.', which keep opening the quantity. (K2, + 2026-09-23, finding J2-L4: J2 derived the vocabulary from `_code_prefix` over + the whole alias table, which put OFFENCE NAMES and other legal systems' statutes + in it -- 'estafa', 'IPC', 'RA', 'NDPS Act', 'Motor Vehicle Act' and the bare word + 'article'. A citation-shaped fragment of one of those names a SECOND offence, + which CanLex has not screened, so removing it was wrong: 'sale of marijuana 2 kg + estafa 315' and '... IPC 302' answered NOT REACHED for a permanent resident where + the record's own digits had kept them NOT SETTLED. The strip's premise -- each + removable token carries no information about HOW MUCH -- holds for a statute-code + citation and not for the name of another offence.) A bare cited + section needs a dotted number with subsections ('893.13(1)(a)(2)') or two or more + bare subsections ('(b)(1)(D)'), so one group alone is never a citation and + '(THC)', '(F3)' and '(10 packs)' still open. A token that overlaps an `_AMOUNT` + match is left alone, so 'sale of marijuana 1 G.S. 90.95' -- where `_AMOUNT` reads + '1 G' as one gram -- keeps opening; and (K2, finding J2-I6) so is a token that + would CUT an `_AMOUNT_QUALIFIER` match of the words as written in half, because a + citation's optional title number could otherwise carry a multiplier's operand + away with it ('sale of marijuana 2 kg x 2 HS 11360' stripped to '... 2 kg x', + which no longer carries the qualifier that must open the quantity). A qualifier + match lying wholly INSIDE a removable token is part of that token, not a + qualifier of the amount ('Docket 15-1234', '2015-06-01'), and does not stop its + removal. Two deliberate limits: the CARDINAL count + ('2 kg, 3 counts') is NOT removed, because it says the offence happened more than + once, exactly the reading `_AMOUNT_QUALIFIER` opens for 'on three occasions', and + neither is a count followed by 'of' ('Count 2 of 3'), which states the same + cardinal fact -- the same reading `_COUNT_SUFFIX` already takes, where 'theft, 2 + counts of burglary' names a second offence. Only the bare identifier ('Count 1', + 'Count #2', 'Count 2:') says which count this entry is and is removed. (K2, + finding J2-L3: J2 removed the whole 'Count 2 of 3' span, so that phrasing + answered NOT REACHED for a permanent resident while '2 kg, 3 counts' stayed NOT + SETTLED.) A bare year ('during 2014', 'Jan-Jun 2014') is no date here. + '1-1/2 lbs' and '2-1/2 kilos' are read as mixed numbers (`_AMOUNT`, + `_amount_number`, `_MIXED_HYPHEN`), which is the LARGEST reading the characters + allow -- a range would have to run from the larger number down to the smaller, + and a per-unit count is smaller still -- and only with a PROPER fraction, so + '3-3/2 lbs' names no amount and opens. + **Q12 moves TWO officer-facing readers, not one** (K2, finding J2-L5). The J2 + entry described the quantity LIMBS only, but `settled_quantity` is also what + cannabis-possession's former-Schedule VIII limit answer runs on + (`possession_limit_open` passes the Schedule VII forms to it), so a 20 g + possession beside a citation, a slash date, a case number or a count identifier + settles at 0.02 kg, is under the 30 g marihuana limit, and the curated 'no + equivalent within the legal limits' answer stands -- NO CANADIAN FEDERAL + EQUIVALENT, one of the named lenient answers. That reader is now in the battery + and in tests/test_verification_j2.py, with an over-limit control ('possession of + marijuana 50 g Count 1' stays NOT SETTLED) and the J2-L3 twin ('possession of + marijuana 20 g Count 2 of 3' is back to NOT SETTLED). Two phrasings this reader + never settled and still does not: a possession amount beside an ISO date or a + hyphenated docket number, because the raw digit run reads as a qualifier before + the strip runs at all. +- **REL-15 -- a standing record suspension beside a prerogative pardon + (`rehab._rs_relief_part`, `prerogative_clause`, `_CANADA_RS_PREROGATIVE_CLAUSE`, + `_CANADA_RS_PREROGATIVE`, `_CANADA_RS_MERCY`, `server._canada_relief`).** IRPA + s. 36(3)(b) excepts a conviction 'in respect of which a record suspension has + been ordered and has not been revoked or ceased to have effect under the + Criminal Records Act': what decides is the Board's order, and a pardon under the + royal prerogative of mercy or Criminal Code s. 748 is a separate instrument that + can only add relief (CRA s. 9: 'Nothing in this Act in any manner limits or + affects Her Majesty's royal prerogative of mercy'). A record suspension naming + clause is now read wherever it sits among the relief clauses, and a clause that + is NOTHING BUT a prerogative pardon -- the name, an optional grant word, an + optional date, and no loss word and no later conviction's words -- does not take + it away: 'record suspension ordered 2015; conditional pardon 2010' and 'free + pardon granted; record suspension ordered 2015' both stand, where before the + first read NOT SETTLED and the second turned on clause order. A naming clause + that names only a PARDON keeps the old rule (it must be the first clause, and + everything after it a keep clause), because that pardon may itself be the + prerogative one; and a prerogative clause sitting BEFORE the relief block, among + the conviction clauses ('royal prerogative of mercy exercised 2010; record + suspension ordered 2015'), still opens -- the accepted strict cost. 'royal + pardon', 'RPM' and 'R.P.M.' are read as prerogative words, and 'grace royale' + with them; before, 'royal pardon 2010' was read as neither a record suspension + nor a prerogative pardon and the conviction was screened as it stands. 'RPM' is + never read after a digit, so an engine speed is not a pardon. +- **REL-16 -- the Criminal Records Act's French relief wording (`rehab._RS_FR_NAME`, + `_RS_FR_GRANT`, `_CANADA_RS_NAME_READ`, `rs_name_read`, `_CANADA_RS_NAMING`, + `_CANADA_RECORD_SUSPENSION`, `_CANADA_RS_LOSS`).** The French wording was unread, + so a French record's relief was screened as it stands -- strict, and useless for + a bilingual corpus. The SAME grammar now reads it, on the Act's own French words: + its French text defines 'suspension du casier' as 'Mesure ordonnee par la + Commission en vertu de l'article 4.1' and its 'rehabilitation' -- the name the + relief carried before 2012 -- as '[Abrogee, 2012, ch. 1, art. 109]', exactly as + the English 'pardon'; s. 2.1(1) gives the Board the power 'pour ordonner, refuser + ou revoquer la suspension du casier'; and IRPA s. 36(3)(b) in French excepts 'en + cas de suspension du casier -- sauf cas de revocation ou de nullite -- au titre + de la Loi sur le casier judiciaire'. So 'suspension du casier' (and 'suspension + du casier judiciaire') is the record suspension, 'rehabilitation' is the pardon + and takes the pardon's post-2012-03-12 reading (AI-4), and 'revoquee', 'annulee' + and 'nullite' are its losses. A French name still needs a French GRANT WORD + ('accordee', 'octroyee', 'ordonnee', 'en vigueur'), so a bare 'suspension du + casier' or 'suspension du casier : oui' stands no more than a bare 'Record + suspension' does; the French conviction words ('declare coupable', 'condamne') + join `_CANADA_RS_CONVICTION`; and the UNACCENTED 'rehabilitation' is an ordinary + English word, read as the Act's name only when a French grant word follows it + (the Act writes rehabilitation-as-reintegration 'readaptation', s. 4.1(1)). + **That last rule was only half applied until K2** (2026-09-23, finding J2-L1). + It lived in `_CANADA_RECORD_SUSPENSION`, which is consulted only to find where + the DISPOSITION's relief clauses start; the relief KEY is read by a separate path + (`_rs_relief_part` -> `_CANADA_RS_NAMING`), whose French name left the accent + optional while its grant group accepts the ENGLISH words. So a Canadian entry + whose relief field read 'rehabilitation granted 2005' -- the ordinary English + word, most naturally a court-ordered treatment condition -- was read as the + Criminal Records Act's pre-2012 pardon, and the conviction answered 'no s. 36 + ground on this matter'. IRPA s. 36(3)(b) excepts only a conviction 'in respect of + which a record suspension has been ordered ... under the Criminal Records Act'; + rehabilitation is s. 36(3)(c)'s separate instrument, and does not reach a + conviction IN Canada at all. `_RS_FR_NAME` now requires the accent, or a French + grant word after the unaccented spelling, so 'réhabilitation accordée 2016' and + 'rehabilitation ordonnee 2005' still read as the pardon and 'rehabilitation + granted / ordered / issued / in force' never names the relief. + **The French prerogative wording is a recorded gap, accepted and strict** (K2, + finding J2-I7). REL-16 gave the French twin of the relief NAMING clause but not + of the prerogative one: `_CANADA_RS_PREROGATIVE_CLAUSE` carries 'grâce royale' + and 'clémence' and not 'pardon conditionnel', 'pardon absolu', 'prérogative + royale de clémence' or 'clémence royale'. So 'suspension du casier ordonnée 2015; + pardon conditionnel 2010' does NOT stand where its English twin 'record + suspension ordered 2015; conditional pardon 2010' does -- the clause is neither a + keep clause nor a prerogative one, and the reading opens. That is the strict + direction (the conviction is screened as it stands), so it is left as it is and + recorded here rather than closed: adding the French prerogative forms is a + wording change to a HELD grammar and belongs with the maintainer's next relief + review. +- **The pre-1985 pardon floor: unchanged, and the maintainer's call.** A relief + dated before the Criminal Records Act as the corpus carries it (`_CRA_FIRST_DAY` + = 1985-01-01, G2-REL-B) still opens the reading, and nothing in this round + touches it. Reading a 1970-1984 Criminal Records Act pardon as a record + suspension is a legal call with a source to cite; the corpus carries no + enactment date for the Act. +- **Measured (J2).** The quantity lens's 678 realistic phrasings (verify-legal/ + quantity, permanent resident, offence 2015-06-01): LENIENT 3-kg-or-less settles + 0 -> 0, strict (an exact amount whose words are the offence's own, no limb named) + 254 -> 236; 18 phrasings moved open -> settled, every one an exact amount beside + a citation, a date, a docket or case number, a count identifier or a hyphenated + mixed number, none moved settled -> open, and no limb changed. The G1/H1 + allowlist repro (verify-allow/quantity/repro.py, GQ-1..GQ-6) is byte-identical + base to head, so every GQ finding stays closed. The canadian-relief lens's full + lists (457 rows, five paths): lenient 0 -> 0 and strict 182 -> 188, the rise + being the four French phrasings moving out of OTHER, where the reader could not + read them at all, into the STAND class their English twins are in; three rows + moved into force and nothing moved out, and every French phrasing reads exactly + as its English twin (20 twin pairs, disposition and relief key, including the + loss, negation, pre-1985, current-year, undated and post-2012 readings). The + answer-identity harness (d05a345 exported with git archive, stub index, 44,410 + calls a tree over six drivers: the 600-record corpus x 10 calls, the generated + multi-entry records, and the G1, G2, G3 and a new J2 worker3 variant carrying + this round's wordings): 42,941 calls byte-identical, 1,469 differ. The corpus + run, the generated-record run and the G2 and G3 variants are byte-identical -- + every difference is in the G1 quantity variant (144 calls) and the J2 variant + (1,325). 25 card Actions moved to a less adverse category, every one accounted + for by the entry that moved: 16 Q12 alone (an exact amount beside a citation, a + date, a docket number or a count identifier now settles the 3-kg-or-less limb -- + 4 in the G1 variant, 12 in the J2 one), 2 REL-15 alone (a standing record + suspension beside a prerogative pardon), 1 REL-16 alone (French relief read), + and 6 where two entries of one record moved (4 Q12 with REL-15, one of them + 'RPM' read as a prerogative word; 2 Q12 with REL-16). No date and no rehabilitation status moved to + a more favourable value anywhere: 0 lenient date or status moves across all + 1,469 differing calls, and no record's earliest application or earliest deemed + date got earlier. +- **Corrected and re-measured (K2, 2026-09-23).** Of the 18 phrasings J2 moved + open -> settled, ONE is reverted: `sale of marijuana 2 kg Count 2 of 3` (and + its possession twin) is open again, so the count is 17. The designator + vocabulary, the qualifier guard and the second reader are as described in the + Q12 bullet above. What K2 measured, base = bf4b0b6 exported with `git + archive`, head = this tree: + - **The quantity and possession battery, 4,835 phrasings** (the J2 lens's own + generator plus the K2 families: a count identifier with an 'of N' tail, a + citation naming another offence, a multiplier beside a citation, and + cannabis possession within and over the 30 g limit): 3,961 identical, 874 + differ. 173 card Actions moved INTO NOT SETTLED -- the reader refusing to + settle, which is the direction this round is for. By the K2 rule's own list + of lenient answers, 25 phrasings moved, every one of a single shape: a + withdrawn cannabis wording beside a statute-code alias, where the alias now + decides the record. All 25 answer exactly as d05a345 does, so nothing is + newly lenient -- they are the pre-J3 answers restored (see the J3 + subsection). + - **The relief keys, end to end:** 6 of the 8 unaccented-'rehabilitation' + keys move from 'no s. 36 ground on this matter' to a s. 44(1) report on + s. 36(1)(a), and 2 more through `canlex_screen_history`; nothing moves the + other way, and the French phrasings and the REL-15 prerogative edges are + byte-identical. + - **The split-record guard:** the three parts J3's terms had made plain + offences are plain offences no longer, all five split records are back to + `limb-open` with `quantity_kg` null, and the Q4 split-amount caution prints + again. + - **The strip's invariants, 32,130 generated phrasings:** `quantity_strip` + still never creates an amount (0) and never destroys one (0); the + phrasings whose written qualifier was carried away with a record-keeping + token fall from 420 to 0 (the 510 that remain are a date's or a docket's + own digit run, which is part of the token, not a qualifier of the amount), + and no phrasing in either tree settles an amount after a qualifier was + swallowed. + - **The answer-identity harness** (bf4b0b6 exported with `git archive`, stub + index, 46,210 calls a tree over seven drivers -- the 3,601-record + disposition corpus x 10 calls, the generated multi-entry records, the G1, + G2, G3 and J1 variants, and a new K2 variant carrying this round's + wordings in the offence slot and in the Canadian relief slot): 44,755 + calls byte-identical, 1,455 differ, 0 timeouts, 0 exceptions. Every + differing call is in the K2 variant, and **0 are lenient**: no call gained + a favourable Action ('no s. 36 ground' or 'deemed'), none gained DEEMED + REHABILITATED, none moved a date earlier, none dropped to a lower branch + while still settling an answer, none newly settled the lower quantity limb + and no record's conviction count fell. The 1,455 are 36 Actions moving + into NOT SETTLED, 177 losing a fixed rehabilitation status into NOT + SETTLED, 254 whose conviction count moved between fixed and open, 159 + whose unresolved list changed and 829 wording. +- **IRPR s. 17 -- the clock (LEGAL-7, then G3 on 2026-09-22: S17-1, S17-2/AI-8, + S17-3/AI-2, PREROG-2/AI-1, PREROG-3/AI-3, AI-6, INFO-1; `rehab._s17`, + `server._history_answer`).** One rule, not a list of cues: the clock runs from + the LATEST date any reading of the record allows, and that strictest date is + the headline; every other reading is a NOT SETTLED sentence (memo and card) + and an `s17_readings` entry, never the headline. On the clock: the completion + of every conviction of any kind (s. 17 excepts only contraventions and Young + Offenders Act offences); the offence date of every committed act s. 17(b) runs + from ('after committing an offence', the matters in s. 36(1)(c) and (2)(c)), + on the conviction basis too (S17-1: TX burglary completed 2012-01-01 and a + simple assault pending from 2023-05-01 answered ELIGIBLE TO APPLY 2017-01-01; + now NOT YET 2028-05-01, 's. 17 clock from entry #2's offence date'); and + every entry whose conviction status is open, read as a conviction (PREROG-3: + a later 'DISPOSITION UNKNOWN' entry completed 2024-01-01 had been left off -- + ELIGIBLE TO APPLY 2015-01-15, now NOT YET 2029-01-01 with the act reading + 2025-02-01; PREROG-2: a later conviction in Canada under a prerogative pardon + had been left off by LEGAL-6 -- ELIGIBLE TO APPLY 2016-03-01, now NOT YET + 2028-01-01). A later conviction still moves an act (LEGAL-7). The one phrase + the reader recognises: 'conditional pardon' with no 'free' counts the pardoned + conviction plainly (Criminal Code s. 748(2): 'to any person who has been + convicted of an offence'; s. 748(3) speaks of a free pardon only); any other + prerogative wording keeps the strict date and gives the date without it. + Recognising it can only remove a favourable second reading -- the headline is + the same either way -- so a misreading cannot move a date earlier. A date the + clock needs that is not given leaves it open ('application not before + '): a conviction's completion (as before), an open entry's completion, + and a committed act's offence date (S17-2/AI-8: the undated act had been + dropped, leaving a settled ELIGIBLE TO APPLY; now NOT SETTLED -- the offence + date of entry #N). The floor and the second reading run from rehab's clock, + not the deeming clock (S17-3/AI-2: a permanent resident's DWI completed + 2011-01-01, no ground, had set the second reading 2016-01-01 on the act + basis; now the robbery act, 2020-01-01; the AI-2 record's floor + 2020-12-19 is now 2029-01-06); a conviction date bounds the floor too + (INFO-1: 2027-01-01 -> 2029-06-01). When the floor is after today every + reading is NOT YET, so the answer is NOT YET with 'application not before + ' and the report Action (AI-6), not NOT SETTLED. On the conviction + basis a permanent resident's act whose maximum is known to be under 10 years + is not on the clock (s. 36(1)(c) is the only act ground that reaches a + permanent resident, and an act restarts nothing); on the act basis every + indictable-equivalent act counts, as before. Outside this round, and left as + they were: a YCJA youth sentence and a foreign juvenile disposition stay off + the clock with the 'confirm that reading' note (now printed on the act basis + too), and a deferred adjudication, probation before judgment or 'adjudication + withheld' is placed as step 1 places it (a committed act, on the clock at its + offence date since S17-1). +- **The second s. 17 reading (BI-4, LEGAL-8, LEGAL-9; narrowed by G3, AI-5; + `server._s17_open_question`, `_s17_moved_clause`, `_s17_reading_text`).** A + reading leaves out only the clock entry -- one reading per open entry. BI-4 + had left out every later conviction that raised the question and called the + others 'the same open question', but an unpaired entry's question is its own + (whether THAT offence has a federal equivalent): TX burglary 2011, an unpaired + 'felony injury to a child' completed 2019-06-01 and CA jaywalking 2023 gave + 2011-03-01 / 2016-03-01; now 2019-06-01 / 2024-06-01. Only convictions whose + pairing SAYS they have no federal equivalent share one legal question and + are left out together, and named ('-- on that reading entry #2 (no Canadian + federal equivalent) is not a subsequent offence either, the same open + question'). A conviction in Canada the entry does not show to be under an + Act of Parliament (no maximum, no hybrid or indictable mode, no federal + pairing) is read both ways like a foreign one (LEGAL-8). 'even though it + grounds no inadmissibility itself' prints only for a conviction whose pairing + says it has no federal equivalent; an unpaired one reads 'whatever it grounds + (its equivalent is not settled)', a conviction in Canada or an engaged one + ends at 'restarts the five years.' (LEGAL-9). +- **Measured.** The answer-identity harness (3,601 records x 10 calls, main + 8c22978 against this branch, stub index) differs on 322 of 36,010 calls, all + canlex_screen_history / canlex_screen two-entry records: 203 are the LEGAL-9 + wording only, 63 LEGAL-7 only, 56 both (attributed against a tree carrying + LEGAL-7 alone). Of the LEGAL-7 calls, the earliest application moves later on + 115, two turn ELIGIBLE TO APPLY into UNDETERMINED (a later conviction in + absentia), and none moves earlier ON THE HARNESS -- which is not the same as + none moving earlier (AI-7, corrected by G3): LEGAL-6 took a later conviction in + Canada under a prerogative pardon off the s. 17 clock (PREROG-2/AI-1: NOT YET + 2028-01-01 / 2029-01-01 became ELIGIBLE TO APPLY 2016-03-01 / 2016-01-15) and + LEGAL-7's floor ran from the deeming clock (S17-3/AI-2: 'application not + before 2020-12-19' where the act allows 2029-01-06), both found by hand and + undone by G3. 'No quantity or relief phrasing of the harness changed' measured + nothing: the harness cannot see LEGAL-1 to LEGAL-6 or BI-5 -- its generator + writes no Canadian relief key (`if rel and kind != "canada"`), its Canadian + dispositions read the same under both readers, and its corpus has no qualified + amount ('approx', 'over ', 'at least') and no prerogative wording (AI-7). The relief changes are measured under 'Measured (G2)' above. Verification round G1 (the quantity allowlist, 41eaf96 against the G1 + branch) re-ran it: 2 of 36,010 corpus calls differ, both Q6 in + canlex_us_equivalency, NO EQUIVALENT -> DEPENDS ('CHARGE: POSS MARIJ <2OZ; + ...', where under 2 oz may exceed 30 g; and 'COURT: CHARGE-POSSESSION OF + MARIJUANA; ...', a strict over-read: no amount, but 'court', 'charge' and '-' + are not offence words); 0 of 3,000 generated multi-entry calls differ; and the + generator extended with 20 quantity phrasings (600 records x 3 calls) differs + on 1,201 of 1,800: 972 have more entries unresolved (every card among them + answers 'Result: NOT SETTLED'; on 2 records the history tool, which leaves an + unresolved entry off the s. 17 clock, now prints an earliest application + computed without it), 229 keep the record-level answer (the new instruction + wording, the confidence block's maximum_source, re-run readings that move to + limb-open), and none moves to a more favourable settled answer. A + before/after table of 79 quantity and relief phrasings (both + statuses for the quantities; canlex_screen_record and canlex_screen's relief + key for the relief): 52 lenient -> strict (20 quantities now open, 28 relief + words now screened as they stand, 4 prerogative pardons now NOT SETTLED), 6 + settled over-3-kg -> open (LEGAL-3), 1 flagged -> NOT SETTLED (a prerogative + pardon in the relief key), 19 controls unchanged, and 1 strict -> lenient: + 'pardon granted 2009' in the relief key now reads as the record suspension the + disposition already read it as (BI-5; ENF 6 s. 23.2 in the corpus: 'record + suspensions (formerly known as "pardons")'). +- **Measured, G3 (the s. 17 clock, 2026-09-22).** The lens's s. 17 battery (123 + hand-worked records x canlex_screen_history and canlex_screen, each with the + most favourable reasonable date and, where two readings stand, the strict + one): on 41eaf96, 20 records answered a date before the most favourable + reading allows (ELIGIBLE TO APPLY where every reading is later), and 35 gave + a headline before the strict reading; now 0 and 8. The 8 are outside this + round and named above (a YCJA youth sentence or foreign juvenile disposition + off the clock; a deferred adjudication, probation before judgment or + 'adjudication withheld' placed as step 1 places it); the comparer's two other + flags are the lens's own false positives (a floor below the hand date on an + open answer; a hedged second reading beside a strict headline). The critic's + probes (S17-1a/b, S17-2 x3, S17-3, AI-2, AI-6, PREROG-2 x3, AI-1, PREROG-3 + P01/P05, AI-3 x2, AI-5) all answer the strict date or NOT SETTLED. The + harness -- the corpus run above, 4,000 generated multi-entry records x 3 calls, + and 3,000 new s. 17 records x 2 calls (`worker3.py`: committed acts dated and + undated, later convictions paired / unpaired / no-equivalent, open verdicts + with and without completion dates, convictions in Canada under conditional, + free and unstated prerogative pardons, record suspensions, appeals allowed, + youth sentences, contraventions, provincial offences, juvenile adjudications, + convictions in absentia), 54,010 calls, 41eaf96 against this branch -- differs + on 14,269: 6,664 strict (the earliest application later, ELIGIBLE TO APPLY -> + NOT YET or NOT SETTLED, a fixed date -> an open one whose floor is no earlier, + a second reading later, a card 'no s. 36 ground' -> NOT SETTLED), 7,605 wording + only (PREROG-4, the act named on the Dates line, the other readings as card + NOTEs, a relief no longer listed as changing the answer because the entry now + stays on the clock either way), 0 lenient. 22 calls (10 records) offer a + second reading earlier than any date offered before, each beside an unchanged + headline: the reading LEGAL-8 gives, now reaching records where no engaged + foreign conviction has a completion date (F1's gate had hidden it). Every + result keeps its s17_readings at or before its headline (53,958 checked). +- **The Canadian outcome words -- an ALLOWLIST (H3, 2026-09-22; S17-INT-1, NEW-1; + `rehab.canada_outcome_stands`, `canada_clauses`, `canada_disposition`; + tests/test_verification_h3.py).** Two of `_CANADA_SENTENCE_WORDS` decide the + ANSWER rather than describe the sentence: a discharge is not a conviction + (Criminal Code s. 730(3)) and a YCJA youth sentence is no ground at all (IRPA + s. 36(3)(e)(iii)). F1 read them ANYWHERE in the disposition, so the word beat + the record's explicit conviction words. 'convicted; request for a conditional + discharge refused; 8 months imprisonment' answered 'NOT A CONVICTION -- a + Canadian discharge (Criminal Code s. 730(3))' and 'no s. 36 ground on this + matter', and 'convicted as an adult; application for a youth sentence + refused; 3 years imprisonment' 'NOT A GROUND -- a youth sentence (IRPA + s. 36(3)(e))'; beside a TX burglary each record answered ELIGIBLE TO APPLY + 2016-03-01 where the control (the same record without the word) answers NOT + YET 2028-01-01 and 2029-01-01 -- the entry left the IRPR s. 17 clock as well. + Six discharge wordings and two youth wordings did it (S17-INT-1: 'conditional + discharge; discharge revoked and conviction entered'; 'Crown appeal allowed: + conditional discharge set aside, conviction entered'; 'found guilty; absolute + discharge denied; convicted and sentenced to 8 months jail'; 'convicted on + count 1, 8 months; absolute discharge on count 2'; 'convicted, 8 months + imprisonment (no conditional discharge)'. NEW-1: the refused application, and + 'adult sentence imposed, not a youth sentence'). Each word now settles only + when it is the disposition's OWN outcome clause: exactly one clause names it + (`canada_clauses` splits on `; : . , ( ) [ ] /` and dashes), that clause is + nothing but the order itself (`_CANADA_DISCHARGE_CLAUSE`, + `_CANADA_YOUTH_CLAUSE`: the accused / offender, a grant word, the phrase, and + at most a s. 730 citation, a probation term, the YCJA named or a length), and + no other clause convicts, revokes, refuses, denies, appeals, sets aside, + names another count or a term of imprisonment (`_CANADA_DISCHARGE_CONTRA`), or + refuses, sets aside or contradicts the youth sentence, including 'as an adult' + and 'adult sentence' (`_CANADA_YOUTH_CONTRA`). Criminal Code s. 730(1) lets + the court, 'instead of convicting the accused', direct that an accused who + 'pleads guilty to or is found guilty of an offence' be discharged, so the + guilt words the subsection itself requires are NO contradiction while + 'convicted' is; s. 730(3)(b) lets the Attorney General appeal 'the decision + of the court not to convict the offender'; s. 730(4) lets the court 'revoke + the discharge, convict the offender ... and impose any sentence'. IRPA + s. 36(3)(e)(iii) excepts an offence for which the person 'received a youth + sentence under the Youth Criminal Justice Act'. Where the word does not + settle it is neither the answer nor the sentence (`sentence` is None): the + conviction words decide, and where they do not the entry is NOT SETTLED -- + either way it stays on the record's counts and its s. 17 clock. The officer's + own `canada_sentence` parameter is a datum, not a reading, and still wins + ('convicted' + `canada_sentence` 'absolute-discharge' is a discharge, as + before). A conditional, suspended or intermittent sentence describes a + sentence and decides no branch, so it has no allowlist and reads as it did + ('the co-accused received a conditional sentence' was already refuted). + Strict costs, accepted: a disposition that both convicts and discharges in + its own words -- 'convicted; absolute discharge', 'convicted; conditional + discharge' -- is now the conviction it names, not a discharge (s. 730(1) + discharges *instead of* convicting; 'pled guilty; absolute discharge' and + 'found guilty, conditional discharge' still read as the discharge), and any + discharge clause carrying an appeal, a negation or another count is read as + it stands. +- **IRPR s. 17 -- the entries step 1 places but does not settle, and the youth + exception as a READING (H3, 2026-09-22; S17-INT-5, S17-INT-6, S17-INT-7; + `rehab._s17_is_open`, `_s17_entries`, `_s17_other_readings`, + `server._record_entries` 's17_open', `_s17_open_lead`).** s. 17 prescribes the + five years only 'if the person has not been convicted of a subsequent offence + other than an offence designated as a contravention under the Contraventions + Act or an offence under the Young Offenders Act' -- the only two exceptions it + states. Three kinds of entry were off the clock or on it at the wrong date. + (a) An entry whose conviction status step 1 places without settling -- a + verdict the curated disposition itself calls likely ('likely-no': a deferred + or withheld adjudication, probation before judgment, an ACD/CWOF) -- left a + permanent resident's clock entirely and silently (S17-INT-5: TX burglary + completed 2011-03-01 plus a TX 'deferred adjudication' completed 2023-01-01 + answered ELIGIBLE TO APPLY 2016-03-01, eleven years earlier than the same + record for a foreign national), and for a foreign national ran from its + OFFENCE date, a year before its completion, with no second reading + (S17-INT-6: NOT YET 2027-01-01 where the conviction reading gives + 2028-01-01). (b) A juvenile or youthful-offender disposition did the same. + (c) A YCJA youth sentence was off the clock altogether, so the favourable + reading was the headline (S17-INT-7: ELIGIBLE TO APPLY 2016-03-01, 's. 17 + clock from entry #1'). Such an entry is now an OPEN entry read as a conviction + ('s17_open', set by `server._record_entries` and read by `rehab._s17` alone -- + no limb reads it): the headline is its sentence completion (2028-01-01 for + every case above, on either status), and the favourable placement is an + `s17_readings` entry with its own NOT SETTLED sentence -- 'a-committed-act' + where s. 17(b) would run from it (2027-01-01), 'not-a-subsequent-offence' + where it would not (2016-03-01). The youth exception is read as a READING, + not a rule: IRPR s. 17 excepts 'an offence under the Young Offenders Act' + while IRPA s. 36(3)(e)(ii) and (iii) except a Young Offenders Act finding of + guilt and a YCJA youth sentence from being a GROUND -- two different tests -- + so the entry stays on the clock for the headline and the excepted date comes + with the existing 'confirm that reading' sentence (G3 had printed that note + while the entry was OFF the clock, which made the favourable reading the + headline; tests/test_verification_g3.py's youth test now pins the reading). + A contravention is still out, the one exception s. 17 states in its own words, + and a YCJA youth sentence is still no s. 36 ground of its own + ('excluded-youth'). An open entry carries BOTH its dated terms -- its + completion and, where s. 17(b) would run from it, its offence date -- because + the headline is the latest date ANY reading allows: an entry whose completion + preceded its offence date had otherwise moved a headline three months earlier + once it became open. The s. 17 lines name a youth entry by what is open about + it ('entry #2 is a conviction in Canada whose sentence is a youth sentence + under the Youth Criminal Justice Act, which IRPR s. 17's exception does not + name'), not as an unsettled conviction (`_s17_open_lead`). +- **The s. 17 clock on the act basis with no dated act (H3, REG-3; + `rehab._s17`).** The clock returned apply_on=None and forgot its undated + entries, so ADDING a matter to a record made the answer more favourable: two + entries (an Idaho withheld judgment whose probation was revoked and a Canadian + conviction completed 2019-06-15) answered 'NOT SETTLED -- a sentence-completion + date', 's. 17 clock not fixed'; the same two plus one older undated-completion + act answered ELIGIBLE TO APPLY 2024-06-15. The completions are now ranked on + either basis, and only the entries that genuinely lack a date (`undated`, + `undated_open`, `undated_acts`) leave it open: the two-entry record answers + 2024-06-15 and the three-entry one is open with 'application not before + 2024-06-15'. 'moved' (a later conviction moved the committed act) is False + when the record holds no dated act. +- **The card's Action is computed from the grounds that hold on EVERY reading + (H3, S17-INT-4, merging REG-1; `server._card_report_holds`, + `_card_deemed_open`, `_CARD_GROUND_OPEN_FACTS`, `CARD_OPEN_FACT_NOTE`).** The + card answered 'Action: NOT SETTLED -- settle the deciding fact first; a + s. 44(1) report, a TRP and allowing the person to leave all turn on it' on + records where a report stands on every reading: an open quantity (G1), an + unresolved entry, an undated act or an open s. 17 date was treated as if it + could remove a settled ground, while the Entries line still read '#1 burglary + ... SERIOUS CRIMINALITY (s. 36(1))'. IRPA s. 36(3)(c) removes the s. 36(1)(b) + and (c) and (2)(b) and (c) grounds only for a person who 'satisfies the + Minister that they have been rehabilitated' or who 'is a member of a + prescribed class that is deemed to have been rehabilitated'; being eligible to + APPLY, or not knowing when the application date falls, cures nothing. The + Action now keeps the report when a settled entry grounds a paragraph the card + reports, its act branch is not itself open, no reading of IRPR s. 18(2) cures + it today (`_card_deemed_open`: a paragraph whose outcome is 'deemed' or + 'open'), and no decisive fact can take that ground away -- relief on any entry + (s. 36(3)(b); Saini), the age at the offence (s. 36(3)(e)), a fact of the + ground entry itself, or the person's status, which while open leaves the + report naming the s. 36(1) paragraphs alone on a serious-criminality or + Canadian s. 36(1)(a) ground (s. 36(2) reaches a foreign national only). A + 'criminality-two-plus-only' entry is never such a ground: it needs a second + offence, which the single-occurrence question can take away. The open fact + then prints as a NOTE on the Dates line ('NOTE: entry #2 unresolved -- not + settled, and no reading of it removes the s. 44(1) report, which stands on + every reading of this record'), and the Result line still says what is not + settled. NOT SETTLED is kept as the Action for every record some reading + clears. +- **REG-7: the Dates line's deemed date names the entries it reaches + (`server.CARD_DEEMED_FOREIGN_ONLY`).** IRPR s. 18(2) reaches only the IRPA + s. 36(1)(b) and (c) and (2)(b) and (c) matters -- 'convicted outside Canada', + 'committed ... an act outside Canada'. A conviction IN Canada is cured by a + record suspension (s. 36(3)(b)), not by time, as `_canada_history_notes` + already says in the memo. The one-line summary had read 'Dates: s. 17 clock + not fixed; earliest deemed 2029-12-31; ...' beside a Canadian conviction; it + now reads 'earliest deemed 2029-12-31 (the foreign entries only -- a + conviction in Canada is cured by a record suspension, IRPA s. 36(3)(b), not by + IRPR s. 18(2))' whenever a Canadian entry is reportable on some reading. +- **Measured, H3 (2026-09-22).** The s17-interaction lens's battery (560 + hand-worked records -- 194 plain, 174 quantity, 131 relief, 27 youth, 27 + discharge, 16 deferred, 14 pardon-era -- through canlex_screen_history as JSON + and as a card and canlex_screen, f6a0f83 against this branch): records whose + HEADLINE is more favourable than the hand-worked strict date 45 -> 0 (youth + 18, discharge 15, deferred 12, relief 2, quantity 1), 0 new; records whose + card Action is NOT SETTLED where a report is supported on every reading + 147 -> 48, 0 new; no `s17_readings` entry is later than its headline. Card + Action base -> head: 173 NOT SETTLED -> report, 264 report -> report, 116 NOT + SETTLED -> NOT SETTLED, 2 deemed -> report (the two Canadian discharge + wordings S17-INT-1 named: the entry is a conviction, so IRPR s. 18(2)(a)(iii) + fails and the clock runs from it), 3 deemed -> deemed, 2 no-ground -> + no-ground; none moved to a less adverse Action. The s17-prerogative lens's + cases (123 records x canlex_screen_history and canlex_screen): headline + lenient 0 -> 0, 22 headlines later, 0 earlier, 0 fixed -> open; the + prerogative pardon's second reading that f6a0f83 restored is still given + (tests/test_verification_h3.py pins it). The critic's own probes (cs.py + S17-INT-1/4/5/6/7 and REG-3, cy.py Y1, cz.py Z2) all answer the strict date, + the report Action or the qualified deemed date. +- **Measured, H3 -- the answer-identity harness.** f6a0f83 exported with + `git archive` against this branch, stub index, five drivers, 42,610 calls per + tree, no timeouts: the corpus run (3,601 records x 10 calls), worker2's 600 + generated multi-entry records x 3 calls, and the three worker3 variants (G1's + quantity phrasings, G2's Canadian relief keys, G3's s. 17 shapes -- 600 + records x 3, 3 and 2 calls). 35,151 calls identical, 7,459 differ. By + direction: the earliest application moves LATER on 1,419 and to an open date + with a floor on 394 (S17-INT-1/5/6/7 -- the entry is on the clock, or its + completion is now needed); it moves from NO date at all to a fixed one on + 1,306 (REG-3: base printed neither a date nor a floor on any of them, checked + call by call, so nothing was told earlier than before), never earlier than a + date base told; the floor moves later on 147. Status: 944 undetermined -> + eligible-to-apply and 122 undetermined -> not-yet (REG-3's fixed date), 300 + eligible-to-apply -> undetermined and 53 -> not-yet, 20 deemed-rehabilitated + -> a weaker answer (eligible-to-apply 9, not-settled 6, not-yet 3, + undetermined 2 -- the Canadian discharge wordings are convictions now, so + IRPR s. 18(2)(a)(iii)/(iv) fails). Card Action: 1,661 NOT SETTLED -> report + (S17-INT-4), 11 no-ground -> report and 4 deemed -> report (S17-INT-1/NEW-1), + 2 no-ground -> NOT SETTLED and 1 deemed -> NOT SETTLED, and 2 report -> NOT + SETTLED. Those 2 are the only calls a mechanical 'less adverse Action' + ranking flags, and both are strict in substance: each is a record whose + settled ELIGIBLE TO APPLY became NOT SETTLED with a floor no earlier (one + 2021-08-15 -> not before 2026-09-14, one 2019-05-17 -> not before 2022-10-08) + because a deferred adjudication or a Canadian 'convicted; absolute discharge' + entry is now open, and the card withholds the report only because a decisive + fact that can take the ground away is open (relief on an entry; the person's + status with no s. 36(1) paragraph). No call clears a person the branch did + not clear before: 0 earliest applications earlier, 0 floors earlier, 0 moves + to 'no s. 36 ground', and every `s17_readings` entry is at or before its + headline. 43 calls differ only in the readings and 2,672 only in wording (the + new NOTE on the Dates line, the s. 17 lines naming a youth entry by what is + open about it, the qualified deemed date). + +### Verification fixes (2026-09-23): J1 -- the Canadian outcome readings + +HELD for the maintainer's review: these answers change on purpose, and they +close the last item section 6 of the implementation memo listed for the +Canadian side -- "the other Canadian outcome readings that still key on a word +anywhere (NCR, appeal, stay)". The rule is the one the quantity reader and the +relief reader already follow: a reader may SETTLE an officer-facing answer only +on an ALLOWLIST, and everything else answers NOT SETTLED or is screened as it +stands. + +- **The five Canadian outcome READINGS -- an ALLOWLIST (J1, 2026-09-23; + `rehab.canada_reading_stands`, `_canada_cue_settles`, `_set_aside`, + `_no_conviction_words`, `canada_disposition`; tests/test_verification_j1.py).** + H3 gave the discharge and the youth sentence an allowlist. Five readings in + `canada_disposition` were left keying on a WORD ANYWHERE in the disposition: + the NCR verdict (`_CANADA_NCR`), a set-aside (`_CANADA_SET_ASIDE`), the + no-conviction words (`_CANADA_NO_CONVICTION`: acquittal, withdrawal, + dismissal, stay of proceedings), an appeal allowed (`_CANADA_APPEAL_ALLOWED`) + and a new trial (`_CANADA_NEW_TRIAL`). Each now settles only as the + disposition's OWN outcome clause: exactly one clause names its words + (`canada_clauses`, as in H3), that clause is nothing but the outcome itself, + and no other clause contradicts it -- refuses it, reverses it, negates it, + appeals it, scopes it to another count, another file, another charge or + another person, or names a conviction or a term of imprisonment. (That + second half was a hand-written denylist, and K1 -- the section below -- + replaced it with an allowlist as well: every other clause must ALSO be one + the grammar can place beside the outcome.) A clause + that is nothing but an empty or negative form value (`_CANADA_EMPTY_VALUE`: + 'NCR: no', 'NCR - not applicable', 'Record: --') contradicts every reading; + it is anchored, so it never reads a value inside a sentence ('found not + criminally responsible; no criminal record entered' still settles). Where a + reading does not stand it settles NOTHING: the conviction words decide, and + where they do not the entry is NOT SETTLED (conviction None), which keeps it + on the record's counts and its IRPR s. 17 clock and prints the deciding + question. + The one clause a contradiction test skips (the keep, in the shape of G2's + `_CANADA_RS_KEEP`) is another of these readings' own clause where the + Criminal Code writes the two together. s. 686(2): 'Where a court of appeal + allows an appeal under paragraph (1)(a), it shall quash the conviction and + (a) direct a judgment or verdict of acquittal to be entered; or (b) order a + new trial' -- so 'appeal allowed; conviction quashed' and 'conviction + quashed; acquittal entered' are ONE outcome in two clauses. A keep is the + other reading's own fullmatch grammar, so it cannot read a refusal ('new + trial refused') or a negation. (K1 narrowed the keep further: s. 686(2) + quashes the CONVICTION, so the keep beside a no-conviction word or a new + trial is `_CANADA_CONVICTION_QUASHED_CLAUSE`, whose noun is mandatory, and + the set-aside reading honours its keeps only where the clause says it was + the conviction that was set aside.) The keep is NOT symmetric: a quashed + conviction keeps a new trial ordered (the new-trial reading then answers -- + the charge is live again, NOT SETTLED), but a new trial anywhere beside a + set-aside takes the settled 'no conviction' away. + What the corpus says the words mean: **NCR** -- Criminal Code s. 672.35, + 'Where a verdict of not criminally responsible on account of mental disorder + is rendered, the accused shall not be found guilty or convicted of the + offence'; an application for that verdict dismissed, withdrawn or set aside + is the opposite, and an assessment ordered under Part XX.1 is no verdict. + **Set aside** -- s. 686(1)(a)/(2) above, with s. 686(1)(b) ('may dismiss the + appeal') and s. 686(3) (a verdict substituted), so a refused or pending + motion to quash, a sentence or an ancillary order quashed, another count's or + another person's conviction, and one restored on further appeal, leave THIS + conviction. **No conviction** -- s. 579(1) stays proceedings on the Attorney + General's direction and s. 579(2) lets them 'be recommenced', so a stay on + one count, a refused stay application or a withdrawn plea is no acquittal. + **Appeal allowed / new trial** -- s. 686(2)(b) and s. 686(4)(b)(i). + The shared non-conviction cue table is held to the same rule on a Canadian + entry (`_canada_cue_settles`): `screening._NON_CONVICTION_CUES` was compiled + for US records and reads 'appeal', 'dismissed', 'pending', 'stayed', + 'quashed' or 'reversed' ANYWHERE, so it settled 'new trial refused by the + Court of Appeal; 2 years' and 'the request for a re-trial was dismissed; + 5 months' as NO CONVICTION. It now settles a Canadian entry only where the + routed words are ONE clause the Canadian grammar can place -- the record's + own no-conviction outcome clause, or the routing's own 'not prosecuted' + (`_CANADA_ROUTED_NO_CONVICTION`, which `record_order` writes for 'convicted; + new trial ordered; Crown declined to retry'). (K1 added the other half of + that test: 'not prosecuted' replaces the WHOLE record, so it settles only + where the record's own words carried no term of imprisonment and no count + scope for the rewrite to drop.) + `_appeal_pending` is deliberately NOT narrowed. It settles no branch, no date + and no Action: it only adds the note 'An appeal is noted: a conviction stands + until it is set aside ...' to a card that already says the entry is a + conviction. For a caution the strict direction is to print it, so the + allowlist rule -- which governs what SETTLES an answer -- does not apply, and + narrowing it would only take a caution away. `strip_appeal_upheld` is + unchanged for the same reason and because it is shared with every US record. +- **Measured, J1 -- the lens battery.** 392 hand-written Canadian dispositions + (tests/test_verification_j1.py carries the repros): 300 in which one of the + five words appears but the outcome it names is refused, reversed, pending, + another count's, another case's, another person's or contradicted by + conviction words (60 for each reading), and 92 in which the outcome IS the + record's own (52 in one clause, 40 in more than one). Each through + `rehab.canada_disposition` and canlex_screen_record (country Canada, hybrid, + canada_max_years 10, custody_imposed_months 3) as a card for a foreign + national and for a permanent resident and as the full memo -- 1,176 calls a + tree, d05a345 against this branch. LENIENT answers (conviction False, card + verdict 'no', or an Action of 'no s. 36 ground' / 'deemed') on the 300 + refuted phrasings: **66 -> 0** (set-aside 35, NCR 17, no-conviction 11, new + trial 2, appeal allowed 1). Of the 92 that state the outcome, 90 settled + before and 88 settle now: the strict price is two multi-clause phrasings, + 'stay of proceedings; charges not recommenced' and 'not guilty; the Crown did + not appeal', which are now NOT SETTLED because the second clause is a keep + the grammar does not carry. (Two more are counted 'opened' only because the + battery expects an appeal allowed to leave the conviction open: 'appeal + allowed and the conviction quashed' and 'appeal succeeded; conviction + quashed' answer NO CONVICTION on both trees, through the set-aside reading.) +- **Measured, J1 -- the answer-identity harness.** d05a345 exported with + `git archive` against this branch, stub index, six drivers, 44,410 calls per + tree, no timeouts: the corpus run (3,601 records x 10 calls), worker2's 600 + generated multi-entry records x 3 calls, the three worker3 variants (G1's + quantity phrasings, G2's Canadian relief keys, G3's s. 17 shapes) and a new + J1 variant whose Canadian entries carry the battery's 392 dispositions + (600 records x 3 calls). **43,774 calls identical, 636 differ, LENIENT 0** -- + no card Action moved to a less adverse one, no earliest application or floor + moved earlier, and no call moved to 'no s. 36 ground'. worker2 and the G1, + G2 and G3 variants are byte-identical (6,600 calls): the J1 readings need + Canadian outcome wording, which those corpora do not carry. + The corpus run differs on 27 calls over 6 records, all Canadian entries whose + outcome word is contradicted later in the record: 'DISMISSED; RE-ARRESTED AND + RE-CHARGED', 'CHARGE WITHDRAWN; CHARGE RELAID', 'APPEAL ALLOWED; CONVICTION + SET ASIDE; RETRIAL TO BE HELD', 'EXPUNGED' and 'CONVICTION QUASHED; ORDER FOR + A NEW TRIAL'. Five calls move 'no s. 36 ground' -> NOT SETTLED, two 'no + s. 36 ground' -> a s. 44(1) report, four move the earliest application later + (2015-01-15 -> 2025-04-01, 2016-03-01 -> 2025-04-01: the entry is on the + s. 17 clock now, so a later entry sets it), two add the record-suspension + route to a status that had none, and 15 differ in wording only. + The J1 variant differs on 609 of 1,800: 505 wording or readings only, and by + direction -- Action 38 NOT SETTLED -> report, 5 no-ground -> report, 4 + no-ground -> NOT SETTLED, 1 deemed -> NOT SETTLED; the earliest application + 15 later and 21 from a fixed date to an open one with a floor, the floor 18 + later; status 9 eligible-to-apply -> undetermined, 12 undetermined -> not-yet, + 6 deemed-rehabilitated -> undetermined, 3 undetermined -> NOT SETTLED, and 6 + undetermined -> eligible-to-apply, each of which keeps the same date or a + later one (2019-05-15 -> 2020-05-15) and a MORE adverse Action, because the + entry that was open is now settled as a conviction. +- **Strict costs, accepted.** A record that both names an outcome and + contradicts it is screened as the conviction its words name, or NOT SETTLED: + 'APPEAL ALLOWED; CONVICTION SET ASIDE; RETRIAL TO BE HELD' is read as a + conviction, because 'retrial to be held' takes the set-aside away and the + phrase 'conviction set aside' carries the conviction cue the shared reader + sees. A retrial the record does not say was ORDERED is not a pending charge, + so that wording is the hardest strict case in the corpus; the next round can + take it (the fix is to read 'retrial to be held' as a new trial, which moves + the answer from a conviction to NOT SETTLED, never the other way). + 'convicted, charges withdrawn' still answers NOT SETTLED, as the batch before + this one pinned: the no-conviction words beside conviction words read as they + did (other counts: a conviction; otherwise NOT SETTLED), and only what they + SETTLE on their own changed. + +### Verification fixes (2026-09-23): K1 -- the other clauses beside a Canadian outcome + +HELD for the maintainer's review: these answers change on purpose. J1 (above) +read the NAMING clause of the five Canadian outcome readings on an allowlist +but cleared every OTHER clause on a hand-written DENYLIST of 40-60 +alternatives -- the shape this RUNBOOK records six earlier rounds failing on, +each closing the phrasings it was shown and leaving new ones. The outcomes lens +measured it on J1's own tree: of 265 distinct Canadian dispositions, **157 still +answered NO CONVICTION**, every one of them because a second clause used +vocabulary no contra table happened to carry. + +- **The OTHER clauses -- an ALLOWLIST too (K1, 2026-09-23; + `rehab.canada_reading_stands`, `_CANADA_COMPATIBLE_CLAUSE`, + `_CANADA_SHARED_CONTRA`, `_CANADA_CONVICTION_QUASHED_CLAUSE`, + `_CANADA_REVIEW_BOARD_CLAUSE`, `_canada_cue_settles`, `canada_disposition`; + `screening.record_order`; tests/test_verification_k1.py).** An outcome now + settles only where EVERY other clause is one the grammar can PLACE beside it: + a bare date; a court; a statute, section or citation reference (the clause + split breaks 's. 672.11' into 's', '672' and '11', so a bare section number + is one of these); a style of cause; a bare party the routing left where it + cut what was said about them ('acquitted; Crown appeal dismissed' keeps the + acquittal and the word 'crown'); a bare offence name; or a keep. Anything the + grammar cannot place leaves the reading NOT SETTLED. An offence name or a + court the grammar does not carry is not a licence to settle -- it is simply + NOT SETTLED, which is the strict side. + The five denylists stay as a second, independent guard and now share one + floor (`_CANADA_SHARED_CONTRA`), so they cannot drift apart again. Its + custodial and sentencing vocabulary is DERIVED from the tables screening and + rehab already read a written sentence with -- `screening._SENT_UNITS` and + `_CANADA_SENTENCE_WORDS` -- so a term in weeks cannot be read by the sentence + reader and missed by the outcome reader. The floor also carries the orders a + Canadian court may make only on a finding of guilt (a conditional sentence + order is available only 'if a person is convicted of an offence' -- Criminal + Code s. 742.1; the victim surcharge is paid by 'an offender who is convicted, + or discharged under section 730' -- s. 737(1); community service, restitution, + a DNA order, a s. 109 weapons prohibition, a SOIRA order, forfeiture); the + words that say the charge has not ended (s. 579(2) lets stayed proceedings + 'be recommenced'); the words that say the outcome did not survive (s. 686(3) + lets a court of appeal 'substitute the verdict that in its opinion should + have been found'); the scoping a record writes (s. 686(1)(b)(i) contemplates + an appellant 'not properly convicted on a count or part of the indictment + [but] properly convicted on another count'); and the relief vocabulary, which + presupposes the conviction it would suspend (Criminal Records Act s. 2.1(1): + the Board's 'exclusive jurisdiction ... to order, refuse to order or revoke a + record suspension'; IRPA s. 36(3)(b) excepts a conviction only where the + record suspension 'has not been revoked or ceased to have effect'). + The keep is narrower. s. 686(2) quashes the CONVICTION, so the keep beside a + no-conviction word or a new trial is `_CANADA_CONVICTION_QUASHED_CLAUSE`, + whose noun group is 'convictions?' and is mandatory, and the set-aside + reading honours its own keeps only where the clause says it was the + conviction that was set aside. An acquittal or an NCR verdict set aside is + that verdict DESTROYED, not s. 686(2): s. 676(1)(a) lets the Attorney General + appeal 'against a judgment or verdict of acquittal or a verdict of not + criminally responsible on account of mental disorder', and s. 686(4)(b) lets + the court 'allow the appeal, set aside the verdict and (i) order a new trial, + or (ii) ... enter a verdict of guilty'. The NCR reading gains one keep in the + other direction: the Part XX.1 disposition s. 672.54 makes the court or + Review Board give INSTEAD of a sentence -- 'discharged absolutely', + 'discharged subject to such conditions', 'detained in custody in a hospital' + -- is that verdict's own consequence, not a contradiction of it. + Three routing seams close with it. (1) `_CUE['appeal-against-conviction-allowed']` + required no object, so 'summary conviction appeal allowed' was rewritten to + 'conviction quashed on appeal' -- the allowlist's own grammar, written by the + ROUTING -- with no flag and no question. The object must be stated now; + s. 813(a) lets the defendant appeal '(i) from a conviction or order made + against him, (ii) against a sentence passed on him, or (iii) against a verdict + of ... not criminally responsible', so the proceeding's name does not say + which was attacked, and s. 822(1) applies s. 686, whose (2)(b) may order a new + trial. The bare forms fall to the appeal-allowed cue, which still raises the + flag and asks the question. (2) The retry-declined branch looked for a count + scope only inside the declination's own segment, so 'convicted on both counts; + new trial ordered; Crown declined to retry; 2 years on count 2' was replaced + by ' ; not prosecuted ; ' -- the stated conviction and the term on the count + the new trial never touched erased. The whole remainder is searched now, and a + count scope raises split-counts; `_canada_cue_settles` also refuses 'not + prosecuted' where the record's own words carried a term or a count scope the + rewrite dropped. (3) A victim surcharge was read as another person's outcome + and cut ('acquitted; victim fine surcharge $200' became a bare acquittal); + s. 737(1) makes it the OFFENDER's own sentence, so it is no longer cut. + The relief presupposition is displaced only by a no-conviction reading that + actually STANDS (it was displaced by the mere presence of an acquittal word), + and, where the routing cut the relief clause out of the words the reading + reads, it is not displaced at all: 'acquitted; record suspension revoked 2019' + answered 'no s. 36 ground' while 'convicted; record suspension revoked 2019' + answered a s. 44(1) report on the same relief. +- **Measured, K1 -- the outcomes lens.** 265 distinct Canadian dispositions + (245 battery rows and 20 seam probes), each through + `rehab.canada_disposition`, canlex_screen_record as a card for a foreign + national and for a permanent resident (country Canada, hybrid, + canada_max_years 10, custody_imposed_months 3) and canlex_screen_history -- + 1,060 calls a tree. LENIENT answers (conviction False, card verdict 'no', or + an Action of 'no s. 36 ground' / 'deemed') on the 240 that should not settle: + **157 -> 0**, with all 25 that should settle still settling. +- **Measured, K1 -- the J1 battery.** The same 392 hand-written dispositions J1 + was measured on, through the same four calls: LENIENT on the 300 refuted + phrasings **0 -> 0**, and of the 92 that state their outcome **88 still + settle, the same 88 as before K1** -- not one of the outcome-stating + phrasings was lost, and no call in that battery moved at all. +- **Measured, K1 -- the answer-identity harness.** bf4b0b6 exported with + `git archive` against this branch, stub index, six drivers, 44,410 calls per + tree, no timeouts: the corpus run (3,601 records x 10 calls), worker2's 600 + generated multi-entry records x 3 calls and the four worker3 variants (G1's + quantity phrasings, G2's Canadian relief keys, G3's s. 17 shapes, J1's + Canadian outcome wordings). **44,401 calls identical, 9 differ, LENIENT 0** -- + no card Action moved to a less adverse one, no earliest application or floor + moved earlier, and no call moved to 'no s. 36 ground'. worker2 and all four + worker3 variants are byte-identical (8,400 calls), the J1 variant included: + every disposition in that battery answers exactly as it did. + The 9 differing calls are all on two corpus records carrying one Canadian + disposition, 'CHARGE WITHDRAWN; RE-LAID; ACQUITTED'. Two card Actions move + 'no s. 36 ground' -> NOT SETTLED (one of them also gains the + record-suspension route, which a status with no conviction had none of), and + seven differ in wording only, where the entry line moves 'no-conviction' -> + 'not-settled'. The cause is the class K1 closes: the contra table carried + 'relaid' and the record writes 're-laid', so nothing refused the reading; + the grammar cannot place 're-laid' beside an acquittal, and the charge laid + again is exactly the live matter s. 579(2) contemplates. +- **Strict costs, accepted.** A clause the grammar cannot place is NOT SETTLED + even where a reader would call it harmless, and the lens's own numbers put + the price at zero on the 92 outcome-stating phrasings and at nine calls in + 44,410 on the corpus. Where a wording turns out to be common and harmless, + the fix is to add it to `_CANADA_COMPATIBLE_CLAUSE` -- a positive grammar + that is reviewed and pinned -- never to take a word out of the denylist. + +### Curated matching gaps (2026-09-23): J3 -- WITHDRAWN by K2; the Q14 item stays OPEN + +**The six terms J3 added were withdrawn on 2026-09-23 (round K2, finding +J3-1).** `data/curated/us_equivalency.json` is byte-identical to d05a345 again. +The audit memo's section 6 item "Texas `DEL MARIJ` aliases (they need curated +terms)" is OPEN, and no term may be added to cannabis-distribution until the +matcher can report a tie between two pairings that phrase-match on DISJOINT +spans of the query instead of resolving it by term length. The eval questions +J3 added STAY (see the end of this subsection). + +**Why (J3-1, a blocker).** A cannabis wording that sits inside a HARDER +offence's one-line charge title displaces that offence. `_best_match` scores +every term the query contains as a phrase at 1.0 and `match_pairings` breaks +the tie by the LONGEST matching term, while `is_tie` reports a tie only when +the two lengths are EQUAL -- so the longer term wins silently. Five of the six +added terms are among the longest phrases cannabis-distribution can match on a +single charge line: `distribution of marijuana` (25 characters), `trafficking +in marijuana` (24), `sell/deliver marijuana` (22), `dealing marijuana` (17), +`DELIVERY MARIJ` (14). They beat `murder` (6), `robbery` (7), `rape` (4), +`sex trafficking` (15), `human trafficking` (17), `drug trafficking` (16), +`money laundering` (16) and `armed criminal action` (21). A hyphen is not a +clause boundary and not a separator `split_offences` reads -- `_fold` maps it +to a space -- so `MURDER 2ND DEGREE - DISTRIBUTION OF MARIJUANA` is screened as +ONE offence. With the terms in, that record answered a Cannabis Act quantity +question: branch `limb-open`, allegation `['not-settled']`, no maximum and no +rehabilitation date, and the card body never named homicide (`homicide` +survived only inside `CANLEX_RESULT`'s `also_matched`). Without them it is +SERIOUS CRIMINALITY on s. 36(1)(b) with a 99-year maximum and an earliest +application date, as it always was. Measured over every curated pairing term +paired with each new wording in one clause: 3,806 strings in which a NAMED +pairing was displaced by cannabis-distribution with the terms in, 0 without +them, and 0 displacements introduced in the other direction. This is exactly +the test J3 itself applied to REFUSE `MAN/DEL` and `PWID MARIJ`. + +**A term may widen which pairing matches; it must never widen the allowlist +that settles an amount.** That part of J3's safety argument is true and still +holds: it rests on `screening.quantity_limb_terms`, and for a pairing that HAS +quantity limbs the allowlist `settled_quantity` reads is built from the LIMBS' +terms alone, never from `us_terms`. cannabis-distribution has three limbs, so +its allowlist was untouched by the terms and is untouched by their withdrawal. +The G1 pinned-allowlist tripwire is back to its pre-J3 word set, and +`test_the_limb_allowlist_excludes_the_new_terms` is kept as the standing guard +that the settling set stays a strict subset of the pairing-wide one. + +**But the safety argument did not cover the whole surface, and two of J3's +published claims were false.** + +- **J3-2: "0 amounts newly settled -- every differing call ends with + `quantity_kg = null`" was not true.** Settling an amount needs BOTH a matched + pairing and `quantity_strip`: J3 supplied a pairing where d05a345 had none + (an ISO date or a citation dilutes the token score below the 0.5 threshold, + so only the new PHRASE match reached the pairing at all) and J2's + `quantity_strip` then removed the date or citation so the amount read. Twelve + offence strings, 81 calls, moved `quantity_kg` from null to a settled amount + -- `trafficking in marijuana 2 kg 2015-06-01`, `DELIVERY MARIJ 893.13(1)(a)(2) + 2 KG` and their neighbours -- four of them on strings that had NO pairing at + all on d05a345, where the card also dropped the over-3-kg (LIFE) reading. The + withdrawal closes the seam: those strings reach no pairing and settle nothing, + and `tests/test_curated_matching_j3.py` pins them. +- **J3-3: the identity harness's own classifier could not see the J3-1 move.** + It scored the Action PREFIX and four date/status fields, and both Actions + begin 'NOT SETTLED', so a record dropping from branch `serious-criminality` + to `limb-open`, allegation `['36(1)(b)','36(1)(c)']` to `['not-settled']`, + `max_years` 99 to null and `rehab_status` `eligible-to-apply` to null was + filed under 'wording'. The comparator now carries a BRANCH ladder (favourable + < unresolved < criminality < serious-criminality), `LENIENT-allegation-lost`, + `LENIENT-max` and a lost status or fixed date counted as a loss rather than + neutral, and it reports separately how many LENIENT calls END in NOT SETTLED + (the reader refusing to settle, which is the safe answer) and how many end in + a settled favourable answer. + +**One further consequence of the withdrawal, recorded because it is the +answer an officer sees.** With the cannabis wording reaching no curated term, +a statute-code alias written in the same line decides the record: `trafficking +in marijuana 2 kg VC 23152` reads as the California DUI (NOT REACHED for a +permanent resident), and `trafficking in marijuana 2 kg motor vehicle act 185` +as India's drink-driving offence. That is the d05a345 answer, restored +unchanged -- 25 such phrasings in the K2 battery, every one matching d05a345 +exactly -- and it is one more reason the Q14 item needs the matcher fixed +rather than more terms. + +The G1 round left a tripwire for exactly this change -- +`test_verification_g1.test_the_derived_allowlist_is_pinned` pins the words the +curated terms lend -- and it fired when the terms went in, as intended: the +PAIRING-WIDE set (`quantity_terms`) gained `dealing`, `del` and `deliver`, all +offence words, and `in`, already a `QUANTITY_NEUTRAL_WORD`. K2 withdrew the +terms, so the pin is the pre-J3 one again. + +The pairing-wide set has a SECOND consumer J3's safety sentence did not cover +(J3-5): `plain_offence`, which the Q4 split-amount guard runs on. Because the +terms put `dealing`, `del` and `deliver` into that set, `dealing marijuana`, +`DEL MARIJ` and `sell/deliver marijuana` became plain offences of their own, +the guard stopped holding the amount beside them, and `sale of marijuana 2 kg +and dealing marijuana` settled the 3-kg-or-less limb with the +`SPLIT_AMOUNT_CAUTION` gone (finding J2-L2: a permanent resident moved from NOT +SETTLED to NOT REACHED). Withdrawing the terms closes it -- measured: all three +parts are back to `plain_offence` False, all five split records are back to +`limb-open` with `quantity_kg` null, and the caution prints again. + +Not a J3 move, and pinned by a test so it is not read as one: `DELIVERY MARIJ +2 kg` settles at 2.0 kg, on d05a345, with the terms in, and here. +`delivery of marijuana` is +a term of the 3-kg-or-less limb, so the string is one exact amount inside a +plain offence clause -- the allowlist working as designed. The same string with +a bound (`DELIVERY MARIJ OVER 5 LBS`) stays open. + +**Two of the audit's strings J3 itself refused**, each on the ground that now +refuses all six -- adding it moves an officer-facing answer: + +- `MAN/DEL` is Texas shorthand for manufacture/delivery of ANY controlled + substance (Penalty Group 1 is cocaine, heroin, methamphetamine). Adding any + cannabis-qualified form (`MAN/DEL MARIJ`) makes the BARE shorthand + token-match cannabis-distribution at 0.833 -- two of its three content words + -- so a record that may name no cannabis at all would be answered under the + Cannabis Act. Its pairing is not unambiguous, so it stays unmatched. +- `PWID MARIJ` already reads as drug-trafficking (1.0). A cannabis PWID term + gives cannabis-distribution the same 1.0, and the longest matching term then + decides; routing it to cannabis-distribution would take a pre-2018 record off + SERIOUS CRIMINALITY and onto the 3-kg-or-less limb (5 years less a day, + CRIMINALITY) -- a LENIENT move, and a legal call that is the maintainer's. + +**The call-log sweep found nothing to add.** `data/logs/` holds only +`queries.jsonl` (two retrieval queries, no offence string); there is no +`calls.jsonl` in this checkout, so no unmatched offence string was available to +sweep. Nothing was read from it beyond those two query lines, and no personal +identifier appears anywhere in this work. + +**What J3's proof measured, and why it missed J3-1.** The identity harness +against d05a345: 36,010 calls over the 3,601-record disposition corpus, all +identical, because that corpus names none of the added wordings except +`POSS MARIJ`, which did not change. A second pass over a 172-record corpus +built from the Q14 strings ran each string as a SEPARATE record, so no split +string exercised `plain_offence` (J2-L2) and no charge-title string put a +cannabis wording beside a harder offence (J3-1); and its 'unknown -> limb-open, +0 lenient' reading came from a classifier that could not see a branch move +(J3-3). Three probes, none of which covered the shape the finding used. The +record-level battery that does cover it is in +`tests/test_curated_matching_j3.py`: the displacement strings, the split +records and the quantity seam. + +**The eval questions STAY.** Five questions dated 2026-09-23, one per wording +family, with gold checked against the corpus (`US Dispositions Helper +s. equiv-cannabis-distribution`, `Cannabis Act s. 10`; ranks 1 to 4 of 20). +They ask what the CORPUS says about distributing cannabis, which is a retrieval +question and does not depend on the matcher, so withdrawing the terms does not +make them wrong -- they read as honest misses on the screening side until the +Q14 item is solved properly. Both frozen slices keep their size: legacy is +"style `standard` and no `added` date", holdout (undated) is "holdout and no +`added` date", so a dated question is structurally outside both -- legacy +n=192, holdout (undated) n=40, unchanged. A gap worth recording: the bare Texas +shorthand (`a Texas rap sheet says DEL MARIJ ...`) retrieves nothing in the top +20, because no corpus document explains US state abbreviations. That is a +corpus gap as well as a matcher gap, and the matcher side is now back open. + +Tests: tests/test_curated_matching_j3.py (21 tests).