ibteda-org
/

pageimprover

@@ -47,122 +47,141 @@ audit regenerates this card and fails on any difference. -->
 # Model card — pageimprover geometry model
-This card describes the deployed page-geometry model: what it predicts, how it was measured, and the places it
-is known to fail. Every performance number here is a row in [`docs/claims.md`](https://github.com/ibteda-org/pageimprover/blob/main/docs/claims.md), which names the
-cohort, the calibration state and the evidence behind it; the figures are rendered from that ledger and a build
-gate refuses any that drift from it. Read that file alongside this one.
-The short version: **the model locates a text block well inside its own archive and does not, on its own, produce
-a finished crop.** A per-book calibration of ten operator crops and a review queue are the other two thirds of
-the system, and the numbers below only mean anything as a trio.
-## What it does
-Given one photographed spread, the model predicts two rectangles — the left and right page crops — as a single
-fixed-length vector. It is geometric regression, not detection: no anchors, no proposals, no class labels
-(ADR 0003). The trunk is a ResNet-34 and the input is a 704<!--claim:model_input--> px tile. That input size was
-chosen to fit a T4's memory, **not** for accuracy, and has never been tuned against the acceptance bar
-(ADR 0007) — a larger input is an untested lever, not a rejected one.
-A prediction is scored by its **worst corner**. A page side passes when the worst of its four corners lands
-within 80<!--claim:pass_bar--> px of the operator's crop, measured in the fixed 2048×2897 finishing frame. This
-is `metrics.PASS_PX`, a geometry rule and nothing more: not an OCR score, not an aesthetic judgement, and not
-"accuracy" in any sense a reader should generalise (ADRs 0005-0010).
-## Evaluation
-Two held-out cohorts, both inside the source archive. "Confirmation" is a 192-book unexposed cohort; "sealed" is
-a 54-book sensitivity test held back from every design decision. All figures are `pass@80`.
-### Raw geometry, before anything
-On the sealed cohort the deployed model reaches a raw pass@80 of 0.6770<!--claim:geo_raw_sealed--> uncalibrated
-and unrouted. That is the honest floor of the model alone. It is not a corpus-wide figure and it is not what the
-system delivers; it is what the network predicts before the two human-in-the-loop stages that follow.
-### Per-book calibration
-The dominant residual is a per-book **size** constant — the crop a given book was finished at leaves no trace in
-a single image (ADR 0001), so no model recovers it at any capacity (ADR 0007). Ten operator crops do. Five
-spreads, both sides, 10<!--claim:cal_crops--> crops per book, pooled into one offset (`calibrate/pooled_median.py`).
-On the confirmation cohort, held-out pass@80 rises from 0.7107<!--claim:cal_baseline--> uncalibrated to
-0.8271<!--claim:cal_pooled--> with that single pooled offset — a gain bought with five spreads of operator work
-per book. The offset is applied only when its magnitude reaches a gate of 40<!--claim:cal_gate--> canonical px
-(`refusals.CALIBRATION_GATE_PX`); below that, an offset is indistinguishable from an operator's own
-self-consistency and can only add noise (ADR 0005).
-Two design choices are recorded as costs, not victories:
-- **Pooled, not per-side.** Splitting the same ten crops by page side harmed 8<!--claim:cal_harm_perside-->
-  books on held-out data; pooling them harmed only 2<!--claim:cal_harm_pooled--> — same crops, estimator the only
-  difference — which is why the pooled estimator is the default (ADR 0010).
-- **k=5, not k=7.** Seven spreads beat five by +0.0104<!--claim:k7_delta--> pass@80, decisively
-  (p<0.0001) — and was declined, because at corpus scale that point costs thousands of extra operator pages. Five
-  spreads is the default (k=5<!--claim:k_default-->) as a cost decision, not a significance one (ADR 0006).
-### Review routing
-A flip-consistency router sends the least trustworthy sides to a person. It flags a side when its direct and
-flipped passes disagree by more than 87<!--claim:route_threshold--> canonical px (`review.absolute`), the
-deployed absolute policy.
-At that operating point, calibrated confirmation quality rises from 0.8334<!--claim:route_conf_baseline--> before
-review to 0.9248<!--claim:route_conf_routed-->, sending 28.5%<!--claim:route_conf_share--> of page sides to a
-reviewer. On the sealed cohort the same fixed rule lifts 0.8278<!--claim:route_sealed_baseline--> to
-0.9151<!--claim:route_sealed_routed-->. **The review share is an in-archive operating point, not a portable
-promise**; a different archive's distribution moves it.
-Reaching a higher 0.930 target under a coverage *sweep* — explicitly not the deployed rule — needs
 34.5%<!--claim:route_sweep_930_sealed--> review on sealed and 30.5%<!--claim:route_sweep_930_conf--> on
-confirmation. The preregistration warns against quoting a sweep point as the deployed one; they are different
-questions and the two were once mixed to produce a claim that had to be withdrawn (ADR 0009).
 ## Training data and attribution
-The model was trained on completed page crops from the **Ibteda corpus** — books finished by hand by the
-archive's operators, whose crops are the target the model regresses toward. That labour is the ground truth here;
-the model reproduces a finishing convention, it does not invent one. Any use of this model should credit the
-Ibteda corpus and its operators as the source of the training crops.
 That convention is narrow and measured as such: 98.7%<!--claim:canvas_share--> of finished pages used a single
-2048×2897 Photoshop template (ADR 0002), and the export round-trip reconstructs that historical output to a
-median residual of 3.0<!--claim:canvas_residual--> px. The high share is a fact about one archive's finishing
-habit, not a universal page ratio, and the residual describes the convention's reproducibility, not the model's
-quality.
-## Known failure modes — read before deploying off-corpus
-**The disagreement score's level does not transfer.** Off-distribution, the flip-consistency gap is large,
-signed and material-dependent: on foreign captures it has overstated a small error by more than an order of
-magnitude on one book and tracked a large median on another. `review.score.disagreement_px` is a routing signal,
-not a page error, and off-corpus it must not be read as one in either direction. Its *ordering* was thought to
-survive even where its level did not — until a fourth book gave essentially zero rank correlation and routed its
-two most nearly perfect sides last. Ranking isolates a single-page outlier; it cannot see a systematic per-side
-bias. Do not size a foreign operator budget from this score; ADR 0011 records the figures and the reasoning.
-**A documented manuscript blind spot.** The deployed model reads printed material well at correct framing and
-reads handwritten manuscript pages badly: 0.0727<!--claim:manuscript_blindspot--> calibrated pass@80 on the
-manuscript sides evaluated, with the routing signal actively anti-ranked rather than merely uninformative. A
-composition-swap model built to fix this was measured and refused — it learned its manuscripts almost perfectly
-and still failed both foreign reads — so the bridge model ships with the blind spot named rather than papered
-over. Anything quoting this model as adequate for manuscript work should quote that number instead (ADR 0012).
-**Framing is the input the model never sees corrected.** For a capture from outside the archive, the paper block
-must be located and centred at the corpus area fraction before the model has a fair chance; the release measures
-and reports this but applies no such transform on the ordinary path, and whether letterboxing or centring is
-better for a frame with no surround is unmeasured (ADR 0011).
 ## Intended use, and what would be misuse
-Intended: locating and cropping page geometry for photographed spreads **within the archive it was trained on**,
-as the first stage of a three-stage loop whose other two stages are a human's calibration crops and a human's
-review of the routed sides.
-Misuse: quoting any single number here as "accuracy"; deploying on manuscript pages or foreign captures without
-re-measuring against a local operator's crops; citing a review share as portable; or reading the disagreement
-score as a per-page error. The model does not finish a page on its own, and this card exists so that no figure in
-it is read as a claim that it does.
 ## Licence and authorization

 # Model card — pageimprover geometry model
+A ResNet-34 regression model that predicts the two page crops of a photographed two-page spread. It is the first of
+three stages: the model proposes the crops, a per-book calibration of five spreads corrects them to the book's own
+margins, and a review queue sends the least trustworthy page sides to a person. It was trained and measured on one
+archive's books and is built to be pointed at anyone's; the calibration is what carries it to a book it has never
+seen. On its own it does not produce a finished page, and the figures below only mean anything as a trio.
+Every performance number here is a row in [`docs/claims.md`](https://github.com/ibteda-org/pageimprover/blob/main/docs/claims.md), which names the cohort, the
+calibration state and the evidence behind it. A build gate refuses any figure that drifts from that ledger.
+## What it takes as input
+| | works | does not work |
+|---|---|---|
+| Capture | one photograph per spread, both facing pages in the frame | one page per photograph: the model predicts two crops per image and has nothing to predict on a single leaf |
+| Material | printed books | handwritten manuscripts, a documented blind spot: 0.0727<!--claim:manuscript_blindspot--> calibrated pass@80 |
+| Framing | the paper filling most of the frame, roughly centred, as in the training captures | a small paper block in a wide surround, which the release measures and reports but does not correct |
+| Image size and format | any size; the package letterboxes the image to the model's 704<!--claim:model_input--> px tile itself; JPEG, PNG, TIFF and WebP | |
+Closed covers are recognised by the framing check and cropped from the detected paper block as single pages; the
+model is not asked about them.
+## Getting the model
+Install `pageimprover` and run `pageimprover fetch model`. It downloads `ibteda.pt`, checks the bytes against the
+pinned digest and refuses any others. Nothing downloads on its own, and the model is loaded only through the
+package: it is a TorchScript archive carrying its own preprocessing contract, not a state dict.
+## How the three stages work
+**1. The model.** Given one spread, it predicts the left and right page rectangles as a single fixed-length
+vector. Geometric regression, not detection: no anchors, no proposals, no class labels. The 704 px input was chosen
+to fit a T4's memory, not for accuracy, and has never been tuned against the acceptance bar; a larger input is an
+untested lever, not a rejected one.
+A prediction is scored by its **worst corner**. A page side passes when the worst of its four corners lands within
+80<!--claim:pass_bar--> px of the operator's crop, measured in the fixed 2048×2897 finishing frame. This is
+`metrics.PASS_PX`, a geometry rule and nothing more: not an OCR score, not an aesthetic judgement, and not
+"accuracy" in any sense a reader should generalise.
+**2. Per-book calibration.** The dominant residual is a per-book size constant: the margins a book was finished at
+leave no trace in a single image, so no model recovers them at any capacity. Ten operator crops do. Five spreads
+(k=5<!--claim:k_default-->), both sides, 10<!--claim:cal_crops--> crops per book, pooled into one offset
+(`calibrate/pooled_median.py`). The offset is applied only when its magnitude reaches 40<!--claim:cal_gate-->
+canonical px (`refusals.CALIBRATION_GATE_PX`); below that it is indistinguishable from an operator's own
+self-consistency and can only add noise.
+Two design choices are recorded as costs, not victories. Pooled, not per-side: splitting the same ten crops by page
+side harmed 8<!--claim:cal_harm_perside--> held-out books; pooling them harmed 2<!--claim:cal_harm_pooled-->, same
+crops, estimator the only difference. Five spreads, not seven: seven beat five by +0.0104<!--claim:k7_delta-->
+pass@80, decisively (p<0.0001), and was declined because at corpus scale that point costs thousands of extra
+operator pages.
+**3. Review routing.** A flip-consistency check runs the model on the spread and on its horizontally flipped copy
+and flags a side when the two disagree by more than 87<!--claim:route_threshold--> canonical px
+(`review.absolute`). Those sides go to a person; the rest ship.
+## Measured performance
+Two held-out cohorts, both inside the training archive. "Sealed" is a 54-book sensitivity test held back from every
+design decision; "confirmation" is a 192-book cohort unexposed during development. All figures are pass@80.
+| stage | sealed cohort | confirmation cohort |
+|---|---|---|
+| model alone | 0.6770<!--claim:geo_raw_sealed--> | — |
+| + per-book calibration | 0.8278<!--claim:route_sealed_baseline--> | 0.8334<!--claim:route_conf_baseline--> |
+| + review of the routed sides | 0.9151<!--claim:route_sealed_routed--> | 0.9248<!--claim:route_conf_routed-->, with 28.5%<!--claim:route_conf_share--> of page sides sent to a person |
+The calibration gain was measured on its own, larger confirmation cohort: held-out pass@80 rose from
+0.7107<!--claim:cal_baseline--> uncalibrated to 0.8271<!--claim:cal_pooled--> with the single pooled offset, a gain
+bought with five spreads of operator work per book.
+The review share is an in-archive operating point, not a portable promise; a different collection's distribution
+moves it. Reaching a higher 0.930 target under a coverage *sweep*, which is explicitly not the deployed rule, needs
 34.5%<!--claim:route_sweep_930_sealed--> review on sealed and 30.5%<!--claim:route_sweep_930_conf--> on
+confirmation. A sweep point and the deployed point answer different questions, and the two were once mixed to
+produce a claim that had to be withdrawn.
+## Using it on another collection
+The model works on printed-book spreads from other scanning setups, on three conditions.
+- **Calibrate every book.** Five spreads, ten crops, as above. This is not optional away from the training archive:
+  the calibration is what adapts the model to a binding and a finishing convention it has never seen. On a public
+  seventeenth-century atlas photographed by another collection, the model located the text block on every spread
+  and the remaining error was one consistent margin offset, exactly what the ten crops remove. That run is a worked
+  example, not a measurement: no accuracy figure is claimed for other collections, and any figure you need must be
+  measured against a local operator's crops.
+- **Frame the paper as the training captures did.** The paper should fill most of the frame, roughly centred. The
+  release measures the framing of every frame and reports it, but applies no correction, so a capture with a wide
+  surround needs cropping upstream. Whether letterboxing or centring is better for a frame with no surround is
+  unmeasured.
+- **Do not carry the review numbers across.** The disagreement score's level does not transfer, its ordering was
+  found not to either, and the deployed threshold and review share describe one archive. Read the routed queue as
+  a list of sides to look at, not as a forecast of operator workload.
+## Known failure modes
+**Handwritten manuscripts.** The deployed model reads printed material well at correct framing and reads
+handwritten pages badly: 0.0727<!--claim:manuscript_blindspot--> calibrated pass@80 on the manuscript sides
+evaluated, with the routing signal actively anti-ranked rather than merely uninformative. A model built to fix this
+was measured and refused: it learned its manuscripts almost perfectly and still failed both foreign reads, so this
+model ships with the blind spot named rather than papered over. Anything quoting this model as adequate for
+manuscript work should quote that number instead.
+**The disagreement score away from the archive.** On foreign captures the flip-consistency gap is large, signed
+and material-dependent: it has overstated a small error by more than an order of magnitude on one book and tracked
+a large median on another, and on a fourth book its ranking put the two most nearly perfect sides last.
+`review.score.disagreement_px` is a routing signal, not a page error, and away from the archive it must not be read
+as one in either direction.
+**Framing is never corrected.** The paper block must sit at roughly the training framing before the model has a
+fair chance. The release reports framing; it does not fix it.
 ## Training data and attribution
+The model was trained on completed page crops from the **Ibteda corpus**: books finished by hand by the archive's
+operators, whose crops are the target the model regresses toward. That labour is the ground truth here; the model
+reproduces a finishing convention, it does not invent one. Any use of this model should credit the Ibteda corpus
+and its operators as the source of the training crops.
 That convention is narrow and measured as such: 98.7%<!--claim:canvas_share--> of finished pages used a single
+2048×2897 Photoshop template, and the export round-trip reconstructs that historical output to a median residual
+of 3.0<!--claim:canvas_residual--> px. The high share is a fact about one archive's finishing habit, not a universal
+page ratio, and the residual describes the convention's reproducibility, not the model's quality.
 ## Intended use, and what would be misuse
+Intended: locating and cropping page geometry for photographed two-page spreads of printed books, from the archive
+it was trained on or from any other scanning setup, as the first stage of the three-stage loop whose other two
+stages are a person's calibration crops and a person's review of the routed sides.
+Misuse: quoting any single number here as "accuracy"; treating the figures as a promise about another collection
+without re-measuring against a local operator's crops; deploying on handwritten manuscript pages; citing a review
+share as portable; or reading the disagreement score as a per-page error. The model does not finish a page on its
+own, and this card exists so that no figure in it is read as a claim that it does.
 ## Licence and authorization