IMPLEMENTATION_PLAN.md

Packaging Material Difference Mining — Implementation Plan

Status note. This document is the original design rationale, written before any training run. It is kept as-is so the reasoning behind each choice stays readable. For current results, what was actually built, what failed, and what to do next, read HANDOVER.md — that file supersedes the phase plan in §4 below.

Target metric: global F1 (TP/FP/FN accumulated over all 100 test pairs, IoU ≥ 0.5 matching).

0. Measured dataset facts driving the design

Fact	Value	Design consequence
Pairs	200 train (1407 boxes, 3–11 per image, median 7) / 100 test	Data synthesis is the main lever
Image size	template and photo identical per pair; mostly 1654×2339, 1700×2200	Work at native resolution with tiling
Alignment	block phase-correlation residual |Δ| p95 ≤ 0.26 px	No registration stage required
Box size	median 22×22 px (~1% of width); 419/1407 under 16 px; 83 exactly 8×8	Small-object regime; sub-pixel box regression
Polarity	842 additions, 558 modifications, 0 deletions; 96.5% more ink in photo	Signed difference channel + polarity filter
Degradation	photo is blur + noise + tone shifted (Laplacian variance 2–6× lower); ~18% of pairs carry heavy global shadow	Blur matching + local background normalization
Naive diff baseline	recall 0.57, precision 0.015, F1 0.03	Precision is problem #1
Threshold-blob box vs GT	median IoU 0.58, 34% within ±2 px	Box tightness is problem #2

An 8×8 box at IoU ≥ 0.5 tolerates only ~2.7 px of shift, so localization accuracy is worth as much as detection accuracy.

1. Architecture

Three stages plus metric-aware post-processing.

Stage 0 — preprocessing (CPU, cached to volume)

1. Per-pair alignment check with phase correlation; ECC homography fallback if the response is low.
2. Blur matching: estimate the photo's blur from the Laplacian variance ratio and apply the matched Gaussian to the template. Removes the largest false-positive class (sharp-vs-soft glyph edges).
3. Local background normalization (img − GaussianBlur(img, σ=31)) on both images to remove shadows and tone shift.
4. Stack an 8-channel tensor: template RGB (3), photo RGB (3), signed difference (1), gradient-magnitude difference (1).

Stage 1 — siamese change detector

• Shared-weight convnext_tiny encoder (timm, ImageNet initialization) over both streams, with concat + absolute-difference fusion at every scale and Changer-style stream exchange.
• U-Net decoder to output stride 2, since the objects are tiny.
• CenterNet-style heads: center heatmap, width/height, sub-pixel offset, plus an auxiliary change mask.
• Loss: Gaussian focal on the heatmap + L1 on width/height and offset + Dice/BCE on the auxiliary mask.
• Tiles of 768 px with 25% overlap, sampled around ground truth plus hard-mined negatives.

Stage 2 — patch verifier and refiner

• 96×96 crops from both images around each stage-1 candidate, ResNet-18 with a 6-channel stem.
• Two outputs: real-difference probability and a 4-coordinate box delta.
• Trained on stage-1 out-of-fold predictions (matched candidates positive, unmatched negative).

Stage 3 — post-processing

• Weighted Boxes Fusion to merge tile predictions (better tiny-box coordinates than NMS).
• Polarity filter: drop candidates where the template has ink and the photo does not (0 of 1404 ground-truth boxes look like that).
• Ink snapping: fit the box to the photo-side ink connected component, then apply the measured constant margin (+0, +0, +1, +1).
• Global threshold sweep over out-of-fold predictions, optimizing global F1 with a single threshold shared by all images.

2. Data strategy

1. Synthetic pair generator replicating the dataset's own construction: clean template → paste a synthetic difference (glyph swap, character insertion, 8×8 mark, region content change, sized to match the observed distribution) → degradation chain (Gaussian blur, sensor noise, JPEG, tone curve, shadow map, ±0.3 px warp). Target 10k+ pairs from the 200 templates plus additional OmniDocBench pages.
2. Copy-paste augmentation of real annotated difference patches into other pairs.
3. 5-fold cross-validation on the 200 real pairs; synthetic data is added to every training fold, never to validation.

3. Modal execution plan

Modal function	Hardware	Purpose
preprocess	CPU ×8	Alignment check, blur match, cache preprocessed pairs and tile index to the volume
synth	CPU ×16, parallel map	Generate synthetic pairs into the volume
train_stage1	A100-40GB	Train the siamese detector for one fold
predict_oof	A10G	Out-of-fold tiled inference, produces verifier training data
train_verifier	A10G	Train the stage-2 patch verifier
predict_test	A10G	Tiled test inference, WBF, verifier rescoring
sweep	CPU	Global-F1 threshold and post-processing calibration

Two Modal volumes: pmdm-data (raw + preprocessed + synthetic) and pmdm-ckpt (checkpoints, predictions, submissions). Code ships via add_local_python_source, so edits take effect without rebuilding the image.

4. Phases

Phase 1 — harness (no GPU)

• Official-equivalent scorer, 5-fold split, preprocessing, tiling, and the naive diff baseline as a floor.
• Exit criterion: the scorer reproduces the F1 definition in the task description and the baseline reports F1 ≈ 0.03.

Phase 2 — stage-1 model

• Train one fold on Modal, decode, sweep the threshold, measure out-of-fold global F1.
• Exit criterion: out-of-fold F1 substantially above the baseline, with the precision/recall split logged.

Phase 3 — synthetic data

• Build the generator, pretrain on synthetic pairs, fine-tune on real folds.
• Exit criterion: measurable out-of-fold F1 gain over Phase 2 at equal decode settings.

Phase 4 — verifier and calibration

• Train the stage-2 verifier on out-of-fold candidates, add ink snapping and the margin calibration, re-sweep.
• Exit criterion: precision gain without a matching recall loss.

Phase 5 — ensemble and submission

• 5 folds × 2 backbones (ConvNeXt-UNet, SegFormer), WBF merge, tile-offset and multi-scale test-time augmentation (no flips — text is chiral), final threshold sweep, write submission.csv.

5. Repository layout

TASK1_CYBER/
├── IMPLEMENTATION_PLAN.md
├── modal_app.py            # Modal app: image, volumes, all remote entrypoints
├── pyproject.toml
└── src/pmdm/
    ├── config.py           # paths, hyperparameters
    ├── metric.py           # global F1 scorer, greedy IoU matching
    ├── folds.py            # 5-fold split
    ├── preprocess.py       # alignment, blur match, local normalization, channel stack
    ├── tiles.py            # tiling and box bookkeeping
    ├── dataset.py          # torch datasets, augmentation, target encoding
    ├── model.py            # siamese ConvNeXt-UNet with CenterNet heads
    ├── losses.py           # focal, L1, Dice
    ├── decode.py           # heatmap decode, WBF, ink snapping, polarity filter
    ├── train.py            # stage-1 training loop
    ├── verifier.py         # stage-2 model, training, application
    ├── synth.py            # synthetic pair generator
    ├── infer.py            # tiled inference, out-of-fold and test
    └── baseline.py         # classical diff baseline (floor)

6. Cost control

• Development runs on a single fold with a reduced tile budget before any 5-fold sweep.
• Every Modal function has an explicit timeout and writes checkpoints to the volume after each epoch, so a preempted run resumes rather than restarts.
• GPU functions are invoked explicitly; nothing trains on import.