documentation.md

HLT Geometric Vision — technical documentation

Reference document for anyone implementing the inference model: input contract, tensor preprocessing, output decoding, and the algorithmic specification of rectangle extraction plus the geometric transforms. Training figures live in README.md; everything here is defined precisely enough to be reimplemented in a language other than Python.

Contents: 1. Two ways to use the model · 2. Input: exact specification · 3. Output: contract of the ONNX artifact · 4. Preprocessing, step by step · 5. Decoding the output · 6. Rectangle extraction — algorithmic specification · 7. Geometric transforms and coordinate mapping · 8. Extractor parameter table · 9. Known limitations · 10. Examples in the repository

---

1. Two ways to use the model

The ONNX artifact (model.onnx) does not do everything on its own: it produces heatmaps and the vertices of one single polygon. Which path to use depends on the domain.

Path	What runs	Result	When to use it
A — full pipeline	ONNX image -> vertices (differentiable Hough + keypoint initialization + neural refiner all inside the graph)	1 polygon, up to 8 vertices, sub-pixel	Training domain: i.e. a clean, well-framed rectangle, one object per frame. Measured vertex RMSE: 0.81 px (val), 0.38 px (test) at 512²
B — heatmaps + external geometry	ONNX (uses only edge_map) + the geometric extractor hlt.postproc (numpy/OpenCV, no weights)	N rectangles/quadrilaterals, nested and cropped-at-border ones included	Real photographs. This is the path used by the examples in this card

Why path B exists: on real imagery the edge_map transfers, the keypoint_map does not (domain gap, documented below), and pipeline A — trained with max_polygons=1 — merges the corners of different objects into one cloud and outputs a "star-shaped" polygon. On real photos A is discarded up front; B keeps only the geometry downstream of the network.

Practical rule: if the target is a well-framed rectangle on a clean background use A (sub-pixel accuracy, a single run). For indoor/outdoor photos with several frames or windows use B and ignore vertices.

---

2. Input: exact specification

The artifact exposes one single input tensor:

Name	Shape	Type	Range	Notes
input_image	(B, 3, 512, 512)	float32	[0.0, 1.0]	RGB channels, not normalized (no ImageNet mean/std), not BGR

Non-negotiable constraints (checked at export time by export_pipeline):

1. Fixed 512×512 resolution. The Hough accumulator inside the graph is sized on H,W: a rectangular or differently sized input is not supported (no dynamic spatial axis is declared). This export was produced at 512² from runs/g3_keypoint/best.pt (data.img_size: 512). For a different resolution re-export the model (--img-size, a multiple of 32) and re-calibrate the extractor thresholds.
2. Dynamic batch axis (dynamic_axes on axis 0 of the input and of every output). B can be any integer ≥ 1; the export parity check runs at B = 2 precisely to exercise the declared axis.
3. Contiguous NCHW layout. The ONNXRuntime binding needs a contiguous buffer: in C/C++ no unusual strides; in numpy use np.ascontiguousarray.
4. No state or mask tensor is fed in. The padding and the valid-area mask used by path B live outside the model: they are needed by the geometric extractor, not by the network.

Training used a procedurally rendered synthetic dataset (convex polygons with 4–6 vertices over a background bank, radius_range: [0.16, 0.38], center_margin: 0.24, geometric and photometric augmenting). That is what defines what the model "knows".

---

3. Output: contract of the ONNX artifact

Fixed order of the output bindings (7 tensors), shapes shown for B = 1 (a larger batch replaces the leading 1):

#	Name	Shape	Type	Content
0	vertices	(B, 1, 8, 2)	float32	refined vertices, in pixels of the input tensor (the 512² frame)
1	vertex_mask	(B, 1, 8)	float32	1 = active vertex, 0 = empty slot. This is the field to use to slice vertices
2	confidence	(B, 1, 8)	float32	uncalibrated head (≈0.49 for this run): informational, never use as an acceptance threshold
3	coarse_vertices	(B, 1, 8, 2)	float32	P₀ before the neural refiner: useful for diagnostics
4	coarse_mask	(B, 1, 8)	float32	mask of coarse_vertices
5	edge_map	(B, 1, 512, 512)	float32	per-pixel edge probability in [0,1] ← input of path B
6	keypoint_map	(B, 1, 512, 512)	float32	per-pixel corner probability in [0,1] (transfers poorly to real photos)

Implementation caveats:

• Slice by mask, never by a fixed count: read vertex_mask > 0.5 and keep only the active slots. Here max_vertices = 8 but the actual number of active vertices is typically 4–6.
• One polygon per sample (max_polygons = 1 in the configuration): the "object" dimension of the output is a single polygon per input image.
• No softmax/sigmoid to apply: the tensors already contain probabilities.
• If vertex_mask is all zeros (no Hough peak above hough_min_strength = 0.5) the output is not "a rectangle with zeroed vertices": it is a not detected.
• Numerical PyTorch↔ONNXRuntime parity was verified at export: vertex_max_diff_px ≈ 2·10⁻³ px, edge_map 7.7·10⁻⁶, keypoint_map 2.8·10⁻⁶ (see the build-generated appendix for this artifact).
• Do not test the export with random inputs. With noise input the edge map is flat, the Hough peaks are numerically tied but TopK/Sort/argmax pick different candidates (hard indices, no guaranteed tie-break across runtimes) → tens of px of fake difference. The parity check uses a crisp drawn quadrilateral (see reference_input).

---

4. Preprocessing, step by step

The model performs no preprocessing inside the graph. This is the exact sequence — identical in training, prediction and this release:

1. Decode the image as BGR (OpenCV) or RGB; if it has an alpha channel, composite it over white.

2. Letterbox to a 512² square, aspect preserved, with a replicated border (cv2.BORDER_REPLICATE) — not a constant gray fill:

   scale = 512 / max(h0, w0)
   nw = round(w0 * scale); nh = round(h0 * scale)      # cv2.INTER_AREA
   ox = (512 - nw) // 2 ; oy = (512 - nh) // 2         # offset of the valid window
   

   Measured reason: a constant gray fill creates two artificial lines as long as the image in the edge map, which saturate the Hough line pool and shift the NMS.

3. Valid-area mask valid (512×512, True inside the real-image window), used by the extractor only: eroding it by 5 px (np.ones((5,5))) stops the four artificial frame borders from contributing votes.

4. Tensor conversion: RGB, float32, /255.0, HWC -> CHW, ascontiguousarray, add the batch axis.

Nothing else: no non-uniform resize, no CLAHE, no denoising, no per-channel normalization.

---

5. Decoding the output

Path A (full pipeline):

outs = sess.run(None, {"input_image": x})
vertices, vmask, conf, coarse, cmask, edge_map, kp_map = outs
v = vertices[0][0][vmask[0][0] > 0.5]          # (K,2) in 512-frame coordinates

When model vertices are used on a non-square image, map them back to the original frame with the inverse mapping in §7. Path B uses only edge_map.

---

6. Rectangle extraction — algorithmic specification

Reference implementation: hlt/postproc/quads.py (extract_quads, hybrid_edge_map, extract_quads_hybrid). Dependencies: numpy + OpenCV (headless). No weights, no training. Input: edge map float32 (H,W) in [0,1] + optional grayscale image (H,W) for the contrast test. Output: list of {corners(4,2), score, edge_cov(4), contrast_cov(4), polarity, area, refine_shift_px}, sorted by score.

Steps, in order:

1. Hybrid edge map (path B). edge = max(edge_map, Canny(blur(gray, σ=1.2), 20, 53)/255), zeroed outside valid. The network misses low-contrast sides; with no line for a side the candidate is never born, so classic Canny fills the holes. Canny thresholds: lower 20, upper 53 (ratio ≈ 1:2.6).
2. Threshold and dilation: near = dilate(edge > edge_thresh) with a disk of radius band_px (spatial tolerance of the support, 3 px).
3. Weighted, oriented Hough: accumulator with n_theta = 360 bins over [0, π) (0.5°) and rho at rho_step = 1.0 px. Each edge pixel votes only for the theta values within ±vote_halfwidth_deg (6°) of its own local normal, and only if local orientation coherence is ≥ vote_coherence (0.5). Reason: without oriented voting, textural curves (wood grain, brush strokes, railings) accumulate votes like a straight line and saturate the pool — measured: 200 lines were not enough on a real photo with 29 horizontal background lines against 9 useful vertical ones.
4. Line NMS: bins ±nms_theta_bins (3 bins = ±3°) and ±nms_rho_bins (6 bins = ±6 px), then the min_line_votes filter (25, the sum of edge_prob along the line) and min_rel_votes (5% of the strongest line). Surviving lines (≤ n_lines = 500) are refitted to the edge pixels within refit_band_px (3 px) by weighted least squares, 2 iterations.
5. Candidate generation: take all line quadruples (a,b,c,d) in cyclic order; the 4 corners are the consecutive intersections (a∩b, b∩c, c∩d, d∩a), closed form. Vectorized block-wise enumeration (chunk = 80000 combinations per block: bounded memory, never Python lists). Cost: C(N,4) × 3 cyclic orders with N ≤ n_lines; this is why the filter that matters is line quality (step 4), not line count.
6. Side coverage in O(1): cumulative sums along each line; the coverage of a segment is the fraction of samples with edge inside the band_px band. The trim = 6% fractions at each end of the side are ignored (corners are the region most damaged by the refit). Thresholds: min_side_cov (the worst side) and min_mean_cov (mean over the perimeter) — see §8. Sides flush with the image border are not penalized: outside the canvas there is no pixel, and measuring them used to count as "no edge" (the cause of false rejections on cropped objects).
7. Shape test: shape="rect" requires the 4 interior angles to lie inside corner_angle_deg = [50°, 130°] → rectangles in perspective too (a frame shot obliquely passes, a degenerate trapezoid does not). shape="quad" accepts any convex quadrilateral. Geometric filters: min_side_px = 30, area within [min_area_frac, max_area_frac] = [1%, 60%] of the image, max_aspect = 6, corners tolerated up to border_margin_px = 8 outside the canvas.
8. Gap test (max_gap_frac = 0.20): if along a side there is a stretch longer than 20% of the side length with no aligned edge, the candidate is a stitched group (several neighbouring objects faking one rectangle) and is rejected.
9. Coherent-polarity contrast test (only with gray): for each side, measure the gray-level difference between an inner band and an outer band (contrast_px = (3,8)), and require the sign to be coherent on all 4 sides (contrast_min = 12 levels). A rectangle invented by joining uncorrelated edges has mixed polarity. Non-measurable samples (band outside the canvas) count as passed. With min_contrast_cov = 0 the test does not filter (default); with min_contrast_cov > 0 it rejects below threshold, and the score becomes mean(contrast_cov) + 0.5 * mean(edge_cov).
10. Rectangle NMS: IoU nms_iou = 0.30, containment nms_contain = 0.70. With prefer_outer = True, near-duplicates keep the outer contour (the outer edge of the frame, not the mount board). With nested = True nested rectangles are kept as well (glass inside a frame, sash inside a window); a box that "extends" a child (stretch_margin) or that is covered by ≥ group_cover (60%) by ≥ 2 disjoint children is dropped. At most max_quads = 64 outputs.
11. Final refinement: every side is refitted to the edge pixels inside the refine_band_px = 4 band (different from refit_band_px = 3, used in step 4 for the individual lines) and corners are recomputed as the intersections of the sides; a shift larger than refine_max_shift_px = 10 px is refused (the corner keeps its original position). refine_shift_px reports the mean shift applied, useful as a quality diagnostic.

Corner order in the output (canonical_order): clockwise in image coordinates (y downwards), first corner = the vertex with the smallest x + y. This is a canonicalization: it lets predictions and ground truth be compared without rotation/direction ambiguity.

Complexity and memory

Hough: O(edge pixels × voted bins per pixel). Enumeration: dominated by C(n_lines,4), while the max_eval = 6000 candidate cut (sorted by coverage) bounds the number of expensive checks; the expensive checks are contrast + gap. The real cost depends on the density of valid lines in the scene, not on resolution alone: for the two examples in this release, measured in build on a 4 thread CPU, extraction cost 0.57 s (framed picture) and 6.6 s (window with many panes and muntins, hence many competing lines). Per-image numbers are in each example JSON and in BUILD_REPORT.json. If bounded time is required, reduce n_lines (enumeration grows like ~N⁴ up to where the max_eval cut kicks in) and/or max_eval (expensive checks grow linearly with it): both cut cost, at the price of coverage on the weaker sides.

---

7. Geometric transforms and coordinate mapping

Conventions: pixel centre (0,0) = top-left corner of pixel 0,0; continuous coordinates with a half pixel offset between index and centre (cv2.LINE_AA and INTER_AREA follow this convention).

Forward (§4): p_512 = p_orig * scale + [ox, oy] - 0.5 + 0.5.

Inverse (mapping extracted vertices from the 512 frame back to the original image):

p_orig = (p_512 + 0.5 - np.array([ox, oy])) / scale - 0.5

The half-pixel correction is not cosmetic: on 4000×3000 photos the mapping error reaches ~1.6 px, above the extractor's own signature (refit ±0.5 px). Skipping it makes it impossible to tell whether a 1.5 px residual is an algorithm error or a rectangle-axis error.

Rectification: extracted vertices are quadrilaterals in perspective, not axis-aligned rectangles. To get the metric rectangle, use a homography estimated from the 4 canonical_order vertices → the 4 target corners (0,0), (w,0), (w,h), (0,h), where w/h are the mean lengths of the opposite sides (cv2.getPerspectiveTransform + cv2.warpPerspective). For a frontal rectification of a window/frame the canonical order (§6, end) is already consistent with this target.

Estimated side rotation (to label "horizontal/vertical"): mean of the arguments of the 4 sides, modulo 90°.

---

8. Extractor parameter table

QuadConfig defaults and the recommended QuadConfig.hybrid() set (used by the examples in this release). hybrid() raises the coverage thresholds because missing sides are filled in by Canny:

Parameter	Default	hybrid()	Meaning
edge_thresh	0.5	0.5	edge pixel = edge > threshold
n_theta / rho_step	360 / 1.0 px	=	Hough grid (0.5° / 1 px)
n_lines	500	=	lines kept after NMS
min_line_votes / min_rel_votes	25 / 0.05	=	lines that are too weak are dropped
min_side_cov	0.25	0.60	minimum coverage of the worst side
min_mean_cov	0.50	0.75	mean coverage of the perimeter
shape	rect	rect	rect = angles inside [50,130]°
band_px / trim	3 px / 6%	=	support band and corner trim
min_area_frac / max_area_frac	1% / 60%	=	usable area
min_side_px / max_aspect	30 / 6	=	minimum side, maximum aspect
border_margin_px	8	=	corners tolerated outside the canvas
min_contrast_cov	0 (off)	0 (off)	0 = polarity does not filter (fragile on paintings)
max_gap_frac	0.20	=	maximum gap: "stitched group"
prefer_outer / nested	True / False	True / True in the examples with frames	outer contour vs nested ones
max_quads	64	=	maximum number of outputs

The hybrid() thresholds are calibrated on a small set of real photographs: before using them in production, re-tune them on a set with your own ground truth (see scripts/evaluate.py in the source repository). They are the most delicate point of the whole chain: a min_side_cov that is too high loses backlit sides, one that is too low re-opens the "stitched group" false positives.

Full example (ONNXRuntime + extractor, ~30 lines):

import cv2, numpy as np, onnxruntime as ort
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
inp = sess.get_inputs()[0].name
outs = [o.name for o in sess.get_outputs()]

def pad_square(img, size=512):
    scale = size / max(img.shape[:2])
    nw, nh = round(img.shape[1] * scale), round(img.shape[0] * scale)
    small = cv2.resize(img, (nw, nh), interpolation=cv2.INTER_AREA)
    ox, oy = (size - nw) // 2, (size - nh) // 2
    disp = cv2.copyMakeBorder(small, oy, size-nh-oy, ox, size-nw-ox, cv2.BORDER_REPLICATE)
    valid = np.zeros((size, size), bool); valid[oy:oy+nh, ox:ox+nw] = True
    return disp, valid, scale, ox, oy

disp, valid, scale, ox, oy = pad_square(cv2.imread("photo.jpg"))
x = np.ascontiguousarray(cv2.cvtColor(disp, cv2.COLOR_BGR2RGB).astype(np.float32)
                          .transpose(2, 0, 1) / 255.0)[None]
out = dict(zip(outs, sess.run(outs, {inp: x})))
gray = cv2.cvtColor(disp, cv2.COLOR_BGR2GRAY)
vmask = cv2.erode(valid.astype(np.uint8), np.ones((5, 5), np.uint8)).astype(bool)
edge = hybrid_edge_map(out["edge_map"][0, 0], gray, valid=vmask)      # §6 step 1
quads = extract_quads(edge, QuadConfig.hybrid(nested=True), gray=gray.astype(np.float32))
for q in quads:
    corners = (q["corners"] + 0.5 - np.array([ox, oy])) / scale - 0.5  # §7 inverse
    print(np.round(corners, 1).tolist(), round(q["score"], 3), q["edge_cov"])

---

9. Known limitations (what does not work)

Read these before integrating: they are measured limits, not hypothetical ones.

1. Domain gap. The model is trained on synthetic data (rendered convex polygons over an image bank background, photometric augmenting). On real photographs the edge_map transfers, the keypoint_map does not. Do not expect the model vertices (output 0) to be correct on real images: on real imagery path B uses the edge map only.
2. One polygon per frame in path A: several objects → "star-shaped" polygon.
3. Uncalibrated confidence (≈0.49 on every vertex): it is not a correctness probability.
4. Fixed 512² resolution: small objects in a large image are shrunk by the letterbox; below ~30 px of side the extractor discards them (min_side_px).
5. hybrid() thresholds tuned on a handful of photos: scene-to-scene variability is the main source of false negatives (backlit sides, low-frequency edges).
6. No perspective model: in scenes rich in parallel lines (the window example shows it) one proposal in four comes out warped or on the wrong sash — the corner_angle_deg angle test accepts very oblique quadrilaterals too, so a side running across a neighbouring sash can pass. Available mitigations: prefer_outer, nms_contain and, application-side, a mask on the object of interest.
7. The reported metrics belong to the synthetic domain: vertex RMSE 0.81 px (val, epoch 43) and 0.38 px (test) hold only on the synthetic dataset drawn from the same training distribution.
8. Cropping at the canvas border: handled (border_margin_px, samples outside the canvas are not penalized), but a rectangle with two sides outside the canvas is inferred, not measured.

---

10. Examples in the repository

examples/ contains public-domain photographs (source metadata re-read at build time) together with the pipeline debug files:

• example_1_picture_frame.* — framed pictures on a plain wall: the outer frame and the mount-board edge of each picture are separate rectangles;
• example_2_window.* — a framed window with many nested rectangles: panes, muntins and the frame are rectangles, so the legitimate candidates are in the dozens; the release keeps the best-scoring max_quads = 12 (a value declared in the example JSON);
• _debug.png suffix = mosaic network edge_map | hybrid map | 512² frame with the extracted sides; .json suffix = license/attribution, preprocessing parameters, vertices in original pixels, scores and per-side coverage.

The number of extracted rectangles and the scores in the examples are measured on this artifact (see the build-generated appendix); the JSON files also record the thresholds used, so you can reproduce the chain without having to guess it.

---

Appendix A — facts measured on this artifact

Generated by scripts/build_hf_release.py on 2026-11-02 16:11 (builder: python -u scripts/build_hf_release.py). Sections 1–9 are the specification, the tables below are measurements: regenerate this appendix whenever you regenerate the model.

A.1 The artifact

Field	Value
File	model.onnx
Bytes	58451191 (58.5 MB)
SHA-256	a351dc1ff555a9e8e6fb2dacffadb2b92f91aa7c7e16fb06eed6aceb949e2e89
Opset	18 / effective default:18
Single file (external data inlined)	true (57520896 bytes of external weights recovered)
Input	input_image ('batch', 3, 512, 512) tensor(float)
Resolution	fixed 512×512, dynamic batch
onnxruntime (build check)	ok=true, active vertices on the reference input=4, edge map in [0.0000, 1.0000]
B=1 vs B=2 consistency	0.000000 px (the declared batch axis is correct)
ONNX CPU latency	median 137.9 ms, p95 155.9 ms (4 threads, batch 1, real input)
Export parity PyTorch→ONNX	vertices 0.00198 px, edge_map 7.75e-06, keypoint_map 2.80e-06

A.2 Actual output binding shapes (verified with onnxruntime)

Output	Shape
vertices	(1, 1, 8, 2)
vertex_mask	(1, 1, 8)
confidence	(1, 1, 8)
coarse_vertices	(1, 1, 8, 2)
coarse_mask	(1, 1, 8)
edge_map	(1, 1, 512, 512)
keypoint_map	(1, 1, 512, 512)

A.3 Numbers of the published examples

The 3 highest-scoring rectangles of each example (example_1_picture_frame: 2 released, example_2_window: 12 released); the complete per-side coordinates are in each example JSON.

Example	Rectangle	score	side coverage	area [original px]	refit [px]
example_1_picture_frame	#1	1.006	1.00, 1.00, 1.00, 1.00	60776	0.968
example_1_picture_frame	#2	1.003	1.00, 1.00, 1.00, 1.00	34589	1.762
example_2_window	#1	1.003	1.00, 1.00, 1.00, 1.00	33953	0.619
example_2_window	#2	1.002	1.00, 1.00, 1.00, 1.00	19530	2.879
example_2_window	#3	1.0	0.98, 1.00, 1.00, 1.00	44091	1.726

Extractor cost measured during the build: example_1_picture_frame: ONNX 169.9 ms + extraction 3638.0 ms, example_2_window: ONNX 148.8 ms + extraction 2534.2 ms.

A.4 Ablation (run data, not hypotheses)

Initialization	best val RMSE [px]	best epoch
keypoint (the released model)	0.81	43
hybrid (Hough + keypoint)	39.64	20
keypoint, no structural losses	39.90	26
pure differentiable Hough	71.26	45

test split of the released model (0.384 px, IoU 0.9719, recall 0.9762, precision 1.0000): runs/g3_keypoint/report_test.json, 50 epochs of train history in the CSV, 20 epochs of val history in summary.json (restored run).

A.5 Build environment

Python 3.14.5 · Windows 11 · numpy 2.5.3 · onnxruntime 1.31.0 · onnx 1.23.2 · torch 2.14.1+cu130 · cv2 5.0.0 · huggingface_hub 2.3.0 · matplotlib 3.11.2