MHamdan/calidoc-layoutlmv3-large-cord

CaliDoc: CORD

LayoutLMv3-Large fine-tuned for key information extraction on CORD, plus the CaliDoc calibration artifacts fitted for this exact checkpoint. Paper: CaliDoc: Calibrated Field-Level Risk and Annotation Auditing for Document Information Extraction, WACV 2027 (Evaluations and Datasets Track). Code: https://github.com/MHHamdan/CaliDoc · Supplement: PDF · Collection: CaliDoc — WACV 2027

Token-level reliability is not complete-field reliability. CaliDoc calibrates the object a downstream system consumes: the complete field.

Model Description

• Extractor. LayoutLMv3ForTokenClassification (microsoft/layoutlmv3-large), 54 BIO labels, trained on Indonesian restaurant receipts (CORD v2), 29 field types / 54 BIO labels derived from the data (no O class). This is a dataset-specific checkpoint, and CaliDoc does not provide a universal model. Each dataset has its own repository in the collection.
• Checkpoint. This repository contains the seed-42 canonical checkpoint, the one the paper uses for every ablation, SpanCRC and audit result. Headline WACV results report mean ± standard deviation over three training seeds (42, 123, 456). Those aggregates are not metrics of this single checkpoint.
• Calibration (post hoc, no retraining). Temperature scaling followed by field-conditional SpanCRC, a conformal risk control over max-token span scores. The neural extractor and the calibration are separate: the weights are a standard Transformers model, and the thresholds in calibration/ turn its probabilities into calibrated label sets.

Intended Use

• Research on uncertainty quantification, conformal prediction and annotation auditing for document information extraction.
• Reproducing and extending the WACV 2027 results.
• Prototyping human-in-the-loop review routing on documents from the same distribution as CORD.

Out-of-Scope Use

• Commercial use: the weights inherit CC BY-NC-SA 4.0 from LayoutLMv3.
• Documents from other domains, layouts, languages or OCR engines. The calibration guarantee assumes exchangeability with the calibration documents and does not transfer to out-of-domain inputs.
• Autonomous decisions in consequential workflows (finance, legal, medical, identity). CaliDoc is meant to decide what a human should check, not to replace that check.
• Treating calibrated-set status as a per-field probability of correctness.

How CaliDoc Works

1. Input: a document image, plus OCR words and word boxes on LayoutLMv3's 0–1000 grid. OCR is not included: the paper uses the dataset-provided OCR.
2. Extractor: LayoutLMv3 token classification. Each word is scored by its first sub-word, and probabilities are softmax(logits / T) with T = 0.8946.
3. Calibration: the calibrated set of word i is C(x_i) = { y : 1 − p(y|x_i) ≤ λ_type(y) }. A gold field is fully covered exactly when its worst word clears the threshold, which makes this a max-token complete-field score.
4. Reliability: taking the document as the exchangeable unit, SpanCRC bounds the expected per-document fraction of missed gold fields by α = 0.1. The bound holds on fresh documents exchangeable with the calibration documents that contain at least one gold field.
5. Audit (corpus level): at the pooled threshold, a boundary-escape test asks whether threshold escapes concentrate at field edges, which is a symptom of alignment and annotation errors. It runs over a calibration corpus with Holm control and a document-blocked bootstrap. It is a dataset diagnostic, not a per-document error detector, and it is not part of single-document inference.

Expect almost every CORD field to be flagged for review. CORD has 29 field types, and most are rare: in the released calibration split only 9 field types reach m0 = 25 calibration spans and get their own threshold, while 20 fall back to the pooled threshold (including types never seen in calibration). Some supported long-tail types need very large thresholds (MENU.SUB.CNT has λ = 1.000, MENU.SUB.NM λ = 0.834), so their labels enter nearly every word's calibrated set. The paper reports this as CaliDoc's weakest case: average set size 0.90 → 2.92 under the field-conditional rule, with span coverage 0.872 → 0.886. Under this release, the useful review signal on CORD is which extra labels a field's set contains (other_labels_in_set), not whether it is ambiguous. The paper's interpolated variant, which removes most of this set-size cost, is not what this repository ships.

The seed-42 CORD checkpoint is the low tail of the three training seeds (93.44/95.70/96.29 test F1): early stopping fired at epoch 10 on a 50-document split. It is released because seed 42 is the paper's predeclared ablation checkpoint, not because it is the best one.

Quick Start

pip install "calidoc @ git+https://github.com/MHHamdan/CaliDoc"

Plain Transformers (the extractor only):

from transformers import AutoModelForTokenClassification, AutoProcessor

model_id = "MHamdan/calidoc-layoutlmv3-large-cord"
processor = AutoProcessor.from_pretrained(model_id, apply_ocr=False)
model = AutoModelForTokenClassification.from_pretrained(model_id)
print(model.config.id2label)

CaliDoc calibrated extraction, given an image and an OCR JSON {"words": [...], "boxes": [[x0, y0, x1, y1], ...], "box_format": "pixel" | "normalized_1000"}:

import json
from PIL import Image
from calidoc.inference import CaliDocPredictor, normalize_boxes

predictor = CaliDocPredictor.from_pretrained("MHamdan/calidoc-layoutlmv3-large-cord")
image = Image.open("document.png").convert("RGB")
ocr = json.load(open("ocr.json"))
boxes = ocr["boxes"] if ocr["box_format"] == "normalized_1000" else normalize_boxes(ocr["boxes"], *image.size)
result = predictor.predict(image, ocr["words"], boxes)
for field in result["fields"]:
    print(field["type"], "|", field["text"], "|", field["status"], field["set_sizes"])
print(result["summary"]["route"])

or from the command line:

python -m calidoc.predict --model MHamdan/calidoc-layoutlmv3-large-cord --image document.png --ocr ocr.json --out prediction.json

A license-cleared CORD test receipt with its OCR file and the full output is in the GitHub repository under assets/examples/cord/.

Each predicted field gets one of these statuses:

status	meaning	suggested action
covered	every word's calibrated set is exactly the predicted label	pass downstream
ambiguous_review	the predicted label is in every set, but some set also admits other labels	validate / review
conflict_review	some word's set is non-empty but excludes the predicted label	review
empty_abstain	some word's set is empty	abstain / review

unassigned_field_candidates lists words predicted O whose set still admits a field label, which are possible missed fields. These statuses are properties of calibrated prediction sets, not confidence scores.

Calibration Artifacts

file	content
calidoc_config.json	α = 0.1, m0 = 25, T, pooled λ, supported/fallback field types, released-checkpoint test metrics
calibration/temperature.json	T = 0.894607, fitted by NLL on the val_es early-stopping split
calibration/pooled_threshold.json	pooled SpanCRC λ = 0.189033 and the token-marginal q̂ for comparison
calibration/field_thresholds.json	λ per field type with its calibration support
calibration/split_manifest.json	calibration document IDs (identifiers only, no data)

The thresholds were fitted on the fixed 50-receipt calibration split (the paper's first calibration unit) and match the committed paper artifacts exactly. Field types with fewer than m0 = 25 calibration spans fall back to the pooled λ, so they inherit only the pooled document-level bound and get no per-type guarantee.

field type	λ	calibration spans	threshold
MENU.CNT	0.0221	108	own threshold
MENU.DISCOUNTPRICE	0.1890	0	pooled fallback
MENU.ETC	0.1890	0	pooled fallback
MENU.ITEMSUBTOTAL	0.1890	0	pooled fallback
MENU.NM	0.0667	108	own threshold
MENU.NUM	0.1890	2	pooled fallback
MENU.PRICE	0.0606	108	own threshold
MENU.SUB.CNT	1.0000	31	own threshold
MENU.SUB.NM	0.8338	41	own threshold
MENU.SUB.PRICE	0.1890	9	pooled fallback
MENU.SUB.UNITPRICE	0.1890	0	pooled fallback
MENU.UNITPRICE	0.1890	19	pooled fallback
MENU.VATYN	0.1890	1	pooled fallback
SUB_TOTAL.DISCOUNT_PRICE	0.1890	2	pooled fallback
SUB_TOTAL.ETC	0.1890	2	pooled fallback
SUB_TOTAL.OTHERSVC_PRICE	0.1890	0	pooled fallback
SUB_TOTAL.SERVICE_PRICE	0.1890	5	pooled fallback
SUB_TOTAL.SUBTOTAL_PRICE	0.3120	33	own threshold
SUB_TOTAL.TAX_PRICE	0.1890	21	pooled fallback
TOTAL.CASHPRICE	0.0594	31	own threshold
TOTAL.CHANGEPRICE	0.0299	34	own threshold
TOTAL.CREDITCARDPRICE	0.1890	6	pooled fallback
TOTAL.EMONEYPRICE	0.1890	2	pooled fallback
TOTAL.MENUQTY_CNT	0.1890	13	pooled fallback
TOTAL.MENUTYPE_CNT	0.1890	1	pooled fallback
TOTAL.TOTAL_ETC	0.1890	2	pooled fallback
TOTAL.TOTAL_PRICE	0.0149	49	own threshold
VOID_MENU.NM	0.1890	0	pooled fallback
VOID_MENU.PRICE	0.1890	0	pooled fallback

Evaluation

Paper: three training seeds (headline, mean ± s.d. over seeds 42/123/456, full test split):

metric	value
entity F1	95.15 ± 1.23

Paper: CaliDoc on this seed-42 checkpoint (90% target, α = 0.10, mean over three calibration splits; the paper's SpanCRC table):

	token-marginal threshold	field-conditional SpanCRC
realized field-miss risk (the bounded quantity)	0.115	0.108
full-span coverage (empirical)	0.872	0.886
average set size	0.90	2.92

This released checkpoint and calibration (computed from the release artifacts):

metric	value
test entity F1 / precision / recall	93.44 / 92.71 / 94.19 (100 test documents)
test ECE (after temperature)	1.57%
SpanCRC on the evaluation side of the released calibration split: field-miss risk / span coverage / set size	0.108 / 0.886 / 2.92 (1309 gold spans)
same split, token-marginal threshold: token / span coverage	0.879 / 0.872

The released weights reproduce the paper pipeline's cached probabilities to within 3e-5 and the recorded test F1 exactly (see docs/release/HF_MODEL_ARTIFACT_AUDIT.md in the GitHub repository).

Limitations

• Dataset and domain specific. One checkpoint per benchmark. Results do not transfer to other document types, and the guarantee assumes exchangeability with the calibration documents.
• English-script benchmarks only (CORD receipts are Indonesian with mostly Latin-script menu text). One backbone family (LayoutLMv3).
• OCR dependence. Inputs are OCR words and boxes, and the paper uses dataset-provided OCR. A different OCR engine changes tokenization and boxes and voids the calibration.
• Max-score conservatism. The worst word decides a field, so long fields (FUNSD answers, SROIE addresses) need larger thresholds and larger sets.
• Support-limited field conditioning. Types below m0 = 25 calibration spans fall back to the pooled threshold and receive no per-type guarantee (most of CORD).
• Short-span audit blind spot. The boundary-escape test cannot see spans of two words or fewer, so a clean result on short fields means untested.
• Audit independence assumption. The randomization null assumes independence across spans, which the method states rather than derives. Findings are claimed only when a document-blocked bootstrap agrees.
• The guarantee is in expectation, over calibration draws and fresh documents. A single document, or a single calibration split, can exceed α.
• Not a replacement for human review in consequential workflows.

License

• Model weights: CC BY-NC-SA 4.0, inherited from microsoft/layoutlmv3-large (CC BY-NC-SA 4.0). This repository fine-tunes that model for token classification and adds calibration files. Non-commercial use only, with attribution, under the same license.
• CaliDoc source code: Apache-2.0, in the GitHub repository.
• Training data: CORD v2 is released under CC BY 4.0 by its authors. Lineage used: naver-clova-ix/cord-v2 at revision 7f0115a4b758a71d6473b8d085751692da2fef98 (800/100/100; the 100 validation receipts are split 50 early-stopping / 50 conformal calibration). Cite Park et al., CORD: A Consolidated Receipt Dataset for Post-OCR Parsing, Document Intelligence Workshop at NeurIPS 2019. No dataset images or annotations are included in this repository.

Citation

WACV 2027 proceedings metadata (pages, DOI) will be added once published.

@inproceedings{hamdan2027calidoc,
  author    = {Hamdan, Mohammed and Cheriet, Mohamed},
  title     = {{CaliDoc}: Calibrated Field-Level Risk and Annotation Auditing for Document Information Extraction},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year      = {2027}
}

@inproceedings{huang2022layoutlmv3,
  author    = {Huang, Yupan and Lv, Tengchao and Cui, Lei and Lu, Yutong and Wei, Furu},
  title     = {{LayoutLMv3}: Pre-training for Document {AI} with Unified Text and Image Masking},
  booktitle = {Proceedings of the 30th ACM International Conference on Multimedia},
  year      = {2022}
}