Hugging Face's logo

Spaces:
ZainabEman
/
OCR-Based-Spare-Part-Localization
Configuration error

App Files Community
OCR-Based-Spare-Part-Localization / PSD.md
ZainabEman's picture
ZainabEman
Upload 152 files
546c207 verified
preview code
|
Raw History Blame Contribute Delete
55.5 kB

A newer version of the Streamlit SDK is available: 1.67.0

Upgrade

Project Specification Document (PSD)

Project: Spare-Parts Diagram Locator for the BQ188 Warp Tying Machine catalogue Source document: 1.pdf (project root) Status: Phase 1 — specification complete, implementation not started Last updated: 2026-08-30

This document is the permanent source of truth for the project. Every statement in section 4 ("PDF / document structure discovered") was verified programmatically against 1.pdf. Nothing in that section is assumed.

1. Project overview

A small, entirely local desktop tool. The user types a machine spare-part number (e.g. 30642). The tool finds the catalogue page that contains that part, renders the exploded-view diagram from that page, and draws a red rectangle around the physical component the number refers to — not merely around the printed number — with the part number and name shown next to it.

The catalogue is a fixed, known, 55-page PDF. The document never changes at runtime. Therefore all expensive work (text extraction, OCR, geometry) happens once, offline, in a build step. The interactive application only performs a dictionary lookup and draws an overlay. Lookup is deterministic and instant.

This is a document-processing / computer-vision project, not a retrieval or question-answering project. See section 19 for what is explicitly out of scope.

2. Exact problem being solved

Given a 5-digit part number P:

Step Requirement
R1 Determine every catalogue page on which P appears.
R2 Retrieve the part's name (German / English / French) and the section it belongs to.
R3 Render that page's exploded-view diagram as an image.
R4 Find every callout label reading P inside that diagram.
R5 For each label, resolve the component the label points to, and produce a bounding box around that component's drawn geometry.
R6 Draw a red rectangle on the component, and print P (plus name) next to it.
R7 Report page number, section number, section title, and part name.

The hard requirement is R5. R4 (finding the printed text) is easy and already proven. R5 requires following the drawing's leader line from the label to the part.

Why this is not a text-search problem

The number 30642 appears twice on page 34: once as a row in the machine-readable parts table on the left half of the page, and once as a raster callout inside the diagram on the right half. Boxing either of those occurrences boxes text. The component itself — a worm gear (Schnecke) — carries no text at all. It is identified only by a thin line drawn from the callout to the part outline.

3. User workflow 

1. User launches the app          →  streamlit run app.py
2. User types a part number       →  "30642"
3. App validates / autocompletes  →  suggests from 705 known numbers
4. App shows result header        →  30642 · Schnecke,kpl. / worm,cpl. / vis sans fin,cpl
                                     Section 07 "Motor-Antrieb / Motor-drive" · PDF page 34
5. App shows the diagram          →  full-page diagram image
      · RED rectangle on the physical component
      · part number drawn next to the rectangle
      · thin blue rectangle on the printed callout label (so the user can verify)
      · confidence badge: "located" / "approximate" / "label only"
6. If the part appears on several pages, a page selector appears (61 parts do).
7. If the part has several instances on one page, all instances are boxed.
8. User can zoom / download the annotated PNG.

Failure is always visible, never silent: if the component cannot be resolved, the app still shows the page and boxes the label, and says so.

4. PDF / document structure discovered

All facts below were extracted from 1.pdf with PyMuPDF 1.28.2.

4.1 File-level

Property Value
Pages 55
Page size 1190 × 842 pt (A4 landscape), rotation 0, uniform on every page
PDF version 1.7
Producer / Creator Microsoft® Excel® 2019
Created 2022-08-21
Encrypted No
Bookmarks / outline None (0 entries)
File size ~39.8 MB
Machine BQ188 Warp Tying Machine — Jingong Industrial Co., Ltd

The PDF was exported from an Excel workbook. This matters: the parts tables are real vector text in a real table grid, while every diagram is a flat embedded JPEG. There is no vector geometry for the drawings.

4.2 Page map

PDF pages Content Text layer Images
1 Title page (BQ188, 4 languages, company address) Yes (376 chars) 3
2 – 23 Scanned operation manual — prose + photographs No real text; only the running footer (98 chars/page) 2 per page
24 NEEDLE TABLE — yarn-count / needle-number table Yes (vector text, 74 vector drawings) 0
25 Spare-parts catalogue cover — ordering instructions in DE/EN/FR/ES Yes 2
26 – 55 Spare-parts catalogue — 30 section pages Yes (table) 1 visible diagram + decoys

The spare-parts catalogue begins at PDF page 25 (cover); the first section page with parts and a diagram is PDF page 26. Pages 1–24 contain no part numbers and are out of scope for indexing (they may still be viewable in the UI).

4.3 Section structure (pages 26–55)

Sections are numbered 01–27 with three sub-sections. The mapping is complete and was read from the first table row of each page:

Page Sec German English Models
26 01 Abdeckhaube Cover PU-ELA-Z
27 02 Lampe+Gebläse Kurze Knoten Lamp+fan short knot
28 03 Abdeckblech,hinten Cover plate,rear
29 04 Abdeckblech,links+Abschlußblende Kurbeldeckel Cover plate,left+Crank end cover cap
30 04-1 Abdeckblech,link. für kurze knoten Cover plate,left+for short knot
31 04-2 Kurze Knoten Short knot
32 05 Gehäuseplatte,oben,unten Housing plate,above,below
33 06 Handantrieb+Fadenabstreifer+Schutzblech Hand drive+Thread wipers+Fender PU-ELA/PU-ELA-Z/PU-S/PU-DK/PU-DKU
34 07 Motor-Antrieb Motor-drive
35 08 Tastfühler+Tastfühler steuerung Feeler+Feeler control PU
36 09 Fadenschieber+Bürstenhalter Thread spool+Brush holder PU
37 10 Kreuzhebel Cross lever PU-ELA/PU-ELA-Z/PU-DK/PU-C/PU-CC/PU-S
38 11 Führungsplatte Guide plate PU
39 12 Nadelanschlag+Nadelhebelwelle Needle+Needle stop lever shaft
40 13 Mittelgruppe-Kipper Central group-tipper PU-ELA
41 13-1 Mittelgruppe Kipper Middle group tipper PU-ELA
42 14 Kurze Knoten Schere Short knot scissors
43 15 Fadenklemme+Klemmenhebellager Thread clamp+clamping lever bearing
44 16 Knotenhelfer+Knüpferlager+Knüpfernadel Lever hinge+Knotter support+Knotter needle
45 17 Fadenhalter+Knotenauszieher+Fadenkanal Thread holder+Knot extractor+Thread channel
46 18 Antriebslager+Transport Drive bearing+Transport
47 19 Kipperwelle Tipper Shaft PU-ELA/PU-ELA-Z/PU-S/PU-DK/PU-DKU
48 20 Gehäuseplatte,re. li. Housing plate,left,right
49 21 Antriebs+Mittel+Untere+Welle Drive+Medium+Low+Shaft
50 22 Kreuzkurvenscheibe Cross cam PU
51 23 Knüpfrahmen Knotting frame PU
52 24 Zubehör für Knüpfrahmen Accessories for knotting frame
53 25 Innentraverse-Knüpfrahmen Knotting frame inner traverse
54 26 Maschinenwagen Machine cart
55 27 Ölwanne Oil pan

4.4 Layout of a catalogue page

Every one of the 30 pages has the identical two-column layout:

 (0,0)                                                        (1190,0)
  +-------------------------------+-------------------------------+
  |  PARTS TABLE  (vector text)   |   EXPLODED DIAGRAM  (JPEG)    |
  |  x ~  17 ... 607              |   x ~ 607 ... 1140            |
  |  y ~  28 ... (varies)         |   y ~  28 ... 800             |
  |                               |                               |
  |  row 0 : section header       |   header: model codes, sec no |
  |  row 1 : Art.Nr | Bezeichnung |   body:   line-art drawing    |
  |          | Description        |           + 5-digit callouts  |
  |          | Denomination       |           + leader lines      |
  |  rows 2+: one part per row    |   footer: drawing path, Seite |
  +-------------------------------+-------------------------------+
                        page footer: company / URL          (1190,842)

4.5 The parts table (left column) — machine-readable

  • page.find_tables() returns exactly one table on every catalogue page, always with 4 columns, and the Art. Nr. header is always at row index 1 (row 0 is the section header). This was verified on all 30 pages.
  • Columns: Art. Nr. | Bezeichnung (DE) | Description (EN) | Dénomination (FR).
  • 861 part rows in total, 705 distinct part numbers.
  • Part numbers are exactly 5 digits, no letters, no separators. Verified length distribution: {5 digits: 860, 6 digits: 1, 2 digits: 27}. The 27 two-digit values are the section-header cells (01…27). The single 6-digit value is a typo in the source document: page 47, 351159 / Kipperhebel / tipper / culbuteur — and the diagram on that same page prints it as 35159, confirming the table cell is wrong.
  • Observed number prefixes: 73 (281), 74 (239), 75 (73), 30 (48), 33 (44), 70, 76, 78, 77, 32, 35, 72.
  • 61 part numbers appear on more than one page (62 have more than one occurrence; the extra one is 33275, listed twice on page 43 — see below). Examples: 73290 on 9 pages (31, 35, 36, 37, 39, 42, 45, 47, 49); 73289 on 5 pages; 70507 on 4. A lookup therefore returns a set of pages, not one page.
  • Only one within-page duplicate table row exists: 33275 on page 43.
  • Non-part rows must be filtered. Pages 42, 51 and 52 end with footnote rows whose Art. Nr. cell is empty or contains *, **, ***: * = "nur bei Ausführung kurze Knoten" (only for short knot), ** = "Länge angeben" (indicate length), *** = "Länge und Durchmesser (18 oder 17) angeben", S = Sonderzubehör (special accessories). Some rows appear after the footnotes and are still real parts (e.g. page 52 30343–30346, page 51 73013), so filtering must be per-row, not "stop at first footnote".
  • The description columns already contain OCR-era typos inherited from whoever built the Excel file: lsolierplatte (lowercase L for I), Kreuzsohnur for Kreuzschnur, Acoesories, and full-width ． instead of .. These are cosmetic. The Art. Nr. column is clean. Descriptions are displayed as-is; they are never used for matching.
  • A leading *, ** or *** inside the Bezeichnung cell is a footnote marker (e.g. *Handkamm, **Kluppe) and should be parsed off into a footnote field.

4.6 The diagram (right column) — raster only

  • Each catalogue page has exactly one visible diagram image, embedded as a DCTDecode (JPEG), DeviceRGB object.
  • Native resolution is 1689 px wide × 2248–2367 px tall on all 30 pages (1689 × 2352 for page 34). Placed at ≈ 533 × 770 pt → effective ≈ 228 dpi. Rendering the page at 300 dpi would upsample past native; the correct move is to extract the embedded image at native resolution and OCR that.
  • Despite the RGB container, content is essentially bilevel line art: dark-pixel fraction (gray < 128) is 2.2 % – 10.6 % per page, mid-tones 0.8 % – 3.0 %. There are no halftone photographs in the catalogue section (unlike pages 2–23 of the manual).
  • Decoy images. Excel left extra image objects on some pages whose placement rectangles fall partly or wholly outside the page box — e.g. page 26 has a second image at y = 802 … 1576 (page height is 842), pages 29 and 54 have one at negative y. Additionally the same xref is reused across consecutive pages (xref 152 is the second image of page 36 and the visible diagram of page 37). → The diagram must be selected as the image rect with the largest intersection area with page.rect, and cached per page, not per xref.

4.7 How part numbers are represented inside the diagram

Verified by rendering and by OCR:

  • Callouts are the actual 5-digit part numbers (30642, 74506, 73631), drawn in a clean ISO-style technical font, roughly 90 px wide × 30 px tall at native resolution. There is no separate item-number scheme — no "item 1, 2, 3" that would need a second lookup table. This is the single most favourable property of this document.
  • Some callouts carry a leading asterisk (*30638, *74012, *76352). The asterisk is the footnote marker described in 4.5 and must be stripped before matching.
  • Some callouts are enclosed in a drawn rectangle (31269, 32357, 33333, 35037, 36962, 30907, 34983, 30168, 30169). Cross-checking these against the table shows they are consistently the kpl. / cpl. (complete assembly) entries — e.g. 31269 Abdeckhaube,kpl. / covering hood cpl., 33333 Mittelgruppe,kpl. / middle group cpl.. A boxed callout therefore denotes a sub-assembly, and its "component" is the whole group shown, not one item. The box also means the label glyphs are not isolated components — the drawn rectangle around them must be recognised and excluded, or it will be mistaken for the leader line.
  • The same part number can appear several times on one page when the part is used in several places — page 26 has 73060 ×2, 74014 ×3, 74025 ×2, 79038 ×2; page 34 has 74506 ×4. Every instance must be indexed and boxed.
  • Non-part text also appears in the diagram and must be rejected:
    • top-left: model codes — PU-ELA, PU-ELA-Z, PU-JET
    • top-right: section number — 07, K 13
    • bottom-left: drawing path — D / PU-S-12 / PU-ELA / ET / ET- 07 Motor-Antrieb
    • bottom-right: the manufacturer's internal page number — Seite 18 (this is not the PDF page number: PDF page 34 is "Seite 18")
    • small revision/date codes such as 061101, 051201, 110702

4.8 How labels relate spatially to components — leader lines

This was investigated directly on the pixel data of page 34.

  • Callouts sit in white space and are joined to their component by a thin solid leader line, typically 3–5 px wide at native resolution. Most are a single straight segment; some are kinked polylines; a few are dashed (page 40: 73158, 71058).
  • The line starts a few pixels away from the label's bounding box (it does not touch the glyphs — verified: the digits of 30642 are their own connected components, distinct from the line) and terminates touching the outline of the target component.
  • Consequence, and the key finding of Phase 1: because the leader line touches its target and nothing else, the 8-connected component containing the leader line is the target component. Running cv2.connectedComponentsWithStats on gray < 150 for page 34 gave 586 components; the component containing the 30642 leader line has bbox x[310:520] y[1472:1671], which is an exact, tight fit around the worm gear. No template matching, no segmentation model, no training data required.
  • Some callouts have no leader line at all and simply sit adjacent to their part (visible on page 40). These need a nearest-component fallback.
  • Nothing in the diagram uses arrowheads in the catalogue section (arrowheads do appear in the manual, pages 2–23, which is out of scope).

So the relationship is confirmed as:

part number (table)  ->  identical printed callout (raster)  ->  leader line  ->  component outline

4.9 What makes localization difficult (measured, not assumed)

Running the prototype over all 34 validated labels on page 34 gave roughly 80 % tight, correct component boxes. The failures are of four kinds:

  1. Leader-line bleed. If a leader line crosses a large neighbouring outline on its way to a small part, the line, the neighbour and the target all merge into one connected component and the box balloons. On page 34, labels 74506, 74903 and 79105 all resolved to the large gearbox housing component (347 × 533 px, 4.7 % of the image) instead of their small rings.
  2. Frame merge. Any component that touches the diagram's outer border merges with it. Page 34's border component has a bbox covering 64 % of the image. A size guard is mandatory.
  3. Nearest-neighbour capture. When another part's outline happens to be closer to the label than the label's own leader line, the wrong component wins. On page 34, 74025 (a screw) captured the bracket that belongs to 76352.
  4. Assembly callouts. A boxed kpl. callout points at a whole group; the "right" answer is a group of components, and any single-component box is arguably wrong.

None of these need machine learning to fix — they need a leader-line-aware tracing step (section 10) plus a small manual-override file.

4.10 OCR feasibility — measured

OCR was actually run, not assumed. Engine: RapidOCR (ONNX Runtime, PP-OCRv4 weights), CPU only, on the natively extracted JPEG, for all 30 catalogue pages.

  • Runtime ≈ 11 s per diagram on this machine; ≈ 6 minutes for all 30 pages, once, offline.

  • Page 34: 39 text boxes returned, every one of the 34 real callouts recognised, confidence ≥ 0.92 for all of them and 1.00 for most. Only two character-level errors, both trivially repaired by stripping non-digits: *7.4012 → 74012 (spurious dot), -74506 → 74506 (leader-line dash absorbed).

  • Full-catalogue recall — distinct part numbers OCR'd from each diagram, matched against that page's own table, summed over all 30 pages:

    Table part rows, pages 26–55 861
    Matched by OCR in the diagram 817
    Recall 94.9 %
    Pages at 100 % 9 of 30
    Worst page 39 → 40/47 = 85 %
    Unrecognised tokens, all 30 pages 69 total (~2.3/page)

    Per-page examples: p28 14/14, p30 19/19, p31 27/27, p36 21/21, p41 25/25, p45 37/37, p50 15/15, p26 33/36, p34 27/29, p40 59/61. The residual "missing" entries are largely parts that are genuinely listed in the table but not drawn in that diagram, not OCR misses — the catalogue does this deliberately for hardware variants. 94.9 % is therefore a lower bound on true OCR recall.

  • The reverse case also exists: callouts appear in a diagram that are absent from that page's table — page 40 shows 73182, 73636, 74167; page 36 shows 33134, 33141; page 33 shows 73007, 70507. So validating OCR output against the page's table alone would silently drop real labels. Validation must be against the global 705-number set, with the page's own table used only as a confidence bonus.

  • The 69 unrecognised tokens are almost entirely the known furniture from 4.7 — 061101-style revision codes and the Seite NN + section-number pair merging into things like 1207 — plus a handful of truncated reads (4639, 7497, 4401) that recover under edit-distance-1 snapping.

Conclusion: OCR works directly on this document and needs no deskewing, no binarization tuning and no custom training. Pages do not need to be rendered — the embedded image is extracted at native resolution instead, which is both faster and sharper.

5. Core technical approach

Build once, look up instantly.

  1. Offline indexer (python -m spareparts.build_index) processes 1.pdf end to end and writes a SQLite database plus a PNG cache. Runs in ~10 minutes. Re-runnable, idempotent, and each stage is independently re-runnable so OCR is not repeated when only the localization tuning changes.
  2. Streamlit app reads the database, does a dictionary lookup, loads the cached PNG, draws the overlay with OpenCV, and displays it. No OCR, no PDF parsing at runtime.

Three deliberately deterministic decisions:

  • The parts table is the authority. Names, sections and the canonical part list all come from the PDF's real text layer, never from OCR.
  • OCR only ever produces coordinates, never facts. Every OCR token is snapped to a known part number or discarded.
  • Localization is classical CV — binarize, connected components, leader-line trace. No detector is trained, no model is fine-tuned, nothing is learned from data.

6. Proposed architecture 

+-------------------------- BUILD TIME (offline, once) --------------------------+
|                                                                                |
|  1.pdf                                                                         |
|    |                                                                           |
|    +--> pdf_loader.py     classify pages, pick the visible diagram per page,   |
|    |                      extract it at native resolution -> data/pages/p##.png|
|    |                                                                           |
|    +--> table_parser.py   find_tables() -> section header + part rows          |
|    |                      -> parts, occurrences                                |
|    |                                                                           |
|    +--> ocr.py            RapidOCR on each diagram -> raw text boxes           |
|    |                                                                           |
|    +--> matcher.py        normalize, reject non-part text, snap to the 705-    |
|    |                      number set (exact -> edit-distance-1) -> labels      |
|    |                                                                           |
|    +--> localizer.py      binarize -> connected components -> leader-line trace|
|    |                      -> component box + method + confidence -> components |
|    |                                                                           |
|    +--> overrides.json    hand corrections, applied last, always win           |
|    |                                                                           |
|    +--> index_db.py       -> data/catalog.db  (SQLite)                         |
|                                                                                |
+--------------------------------------------------------------------------------+
                                       |
+---------------------------- RUN TIME (Streamlit) ------------------------------+
|                                                                                |
|  app.py  -->  lookup.py (SQLite queries)  -->  overlay.py (OpenCV drawing)     |
|                                                                                |
+--------------------------------------------------------------------------------+

There is no server, no API, no queue, no container. Two entry points: one script, one Streamlit app.

7. Data flow

Build 

1.pdf
 +- for page in 26..55:
      diagram_rect = argmax over image rects of area(rect ∩ page.rect)      # 4.6
      png = extract_image(xref)                            -> data/pages/p34.png
      table = page.find_tables()[0].extract()
        row0            -> section code / DE / EN / model
        row1            -> header, skipped
        rows2..         -> part rows, footnote rows filtered  -> parts, occurrences
      ocr_boxes = RapidOCR(png)
        strip non-digits; drop len != 5
        drop boxes in header/footer strips (top 4 %, bottom 6 %)
        snap to global part set (exact, then edit-distance 1)  -> labels
      bw = (gray(png) < 150)
      cc = connectedComponentsWithStats(bw, 8)
      glyph_ids = components fully inside any label box (+ their enclosing box, if boxed)
      for each label:
          anchor = trace_leader_line(label, bw, cc, glyph_ids)
          box, method, conf = resolve_component(anchor, cc)   -> components
 +- apply overrides.json
 +- write catalog.db

Query 

"30642"
 +- SELECT part                     -> name_de/en/fr
 +- SELECT occurrences              -> [page 34, section 07 "Motor-Antrieb"]
 +- SELECT labels + components      -> [{label_box, component_box, conf}]
 +- load data/pages/p34.png
 +- draw red rect + caption per instance, blue rect on label
 +- st.image(annotated)

8. OCR strategy

Engine: RapidOCR (rapidocr-onnxruntime).

Rationale, and why not the stack originally guessed:

  • PaddleOCR uses the same PP-OCR models but drags in paddlepaddle, which is a large, frequently painful wheel on Windows and pins specific Python versions. RapidOCR ships the identical PP-OCRv4 detection + recognition models as ONNX and runs on onnxruntime CPU. It installed cleanly here in one pip install with no system dependencies, and was verified working on this exact document at 94.9 % recall. If PaddleOCR is later preferred, ocr.py exposes one function and the engine is swappable.
  • Tesseract was not chosen: it needs a system binary outside pip, and it is weaker on sparse, isolated numerals scattered on a mostly-empty canvas, which is precisely this document's layout.

Settings and rules:

  • OCR the natively extracted JPEG (1689 px wide, ~228 dpi effective), never a re-rendered page. No upscaling — measured accuracy is already ≈ 100 % on callouts.
  • No deskew, no rotation correction: every diagram is upright and axis-aligned.
  • Run once per page at build time; results are cached to data/ocr_cache/p##.json so that re-tuning localization never re-runs OCR.
  • Post-processing pipeline per detected box:
    1. re.sub(r'\D', '', text) — removes the * footnote marker, the spurious . from *7.4012, and the leading - from -74506.
    2. Reject if length ≠ 5.
    3. Reject if the box lies in the header strip (top 4 % of image height) or footer strip (bottom 6 %) — this removes Seite 18, drawing paths and revision codes.
    4. Snap to the global part set (section 9).
  • Confidence from the engine is stored per label and surfaced in the UI.

9. Part-number extraction strategy

Two independent sources, with clearly separated authority:

A. Canonical list — from the PDF text layer (authoritative). page.find_tables() on pages 26–55. Per row: Art. Nr. must match ^\d{5}$ after stripping whitespace; rows failing that (empty, *, **, ***) are recorded as footnotes and excluded from the part list. The Bezeichnung cell's leading *s are split off into a footnote field. This yields 705 distinct part numbers / 861 occurrences, and is the only source of names and sections.

One known data defect to carry as-is with a flag: page 47's 351159 is 6 digits and the diagram prints 35159, so it is a source typo. It is stored, marked is_malformed = 1, and excluded from OCR snapping.

B. Label positions — from OCR (positional only). Every OCR token is snapped to the canonical list before it is allowed to become a label:

1. exact match against the 705-number set                -> accept, conf = ocr_conf
2. else Levenshtein distance 1 to exactly one number     -> accept, conf = ocr_conf * 0.8
                                                            (single candidate only —
                                                             ambiguous ties are dropped)
3. else                                                  -> discard, log to build report

A token that also appears in its own page's table gets a confidence bonus; a token that appears only in the global list is still accepted, because section 4.10 proved that diagrams legitimately show parts absent from their page's table.

Because part numbers are exactly 5 digits with no letters, no checksum and no separators, no other normalization is needed.

10. Component localization strategy

This is the heart of the project. Four tiers, tried in order, each recording which one fired so the UI can be honest about it.

Preparation (once per page) 

gray  = cv2.cvtColor(img, COLOR_BGR2GRAY)
bw    = (gray < 150).astype(uint8)                # verified threshold; art is bilevel
n, cc, stats, _ = cv2.connectedComponentsWithStats(bw, 8)
glyph_ids  = components whose bbox lies inside a label bbox (+ small pad)
frame_ids  = components whose bbox area > 25 % of the image  (border, page frame)
box_ids    = for boxed assembly callouts: the thin rectangle enclosing the glyphs

Tier A — leader-line trace (primary)

  1. Cast rays from the label's bounding box outward in 5° increments, from 4 px to ~40 px away from the box edge, looking for dark pixels that belong to a component that is neither a glyph nor the frame. This finds the leader line's near end.
  2. Verify the candidate is line-like: skeletonize the local neighbourhood and check the stroke width is ≤ ~6 px and locally elongated. This is what distinguishes the real leader line from a neighbouring part's outline that happens to be closer (failure mode 3 in 4.9).
  3. Walk the line. Follow the stroke pixel by pixel away from the label, allowing direction changes for kinks and a small pixel gap for dashed lines, until the stroke either ends or opens out into a wide (non-line-like) region. That transition point is the anchor.
  4. Take the connected component containing the anchor. Its bbox is the component box. method = "leader".

Tier B — bounded fallback (when Tier A's component is implausible)

Triggered when the Tier A component's bbox exceeds ~8 % of the image area (leader-line bleed / frame merge, failure modes 1 and 2 in 4.9), or when no line-like stroke is found within 40 px.

  • If an anchor exists: emit a fixed-radius box centred on the anchor, sized from the page's median component size (roughly 120 × 120 px at native resolution).
  • If no anchor exists: take the nearest non-glyph, non-frame component to the label, with the same size cap.
  • method = "anchor_radius" / "nearest", and confidence is lowered. The UI shows these as "approximate".

Tier C — assembly callouts

A callout enclosed in a drawn rectangle (kpl. / cpl. parts, section 4.7) points at a whole group. For these: trace the leader line as in Tier A, then take the union of all non-glyph components whose bbox centre lies within a generous radius of the anchor, capped at 30 % of the image. method = "assembly". The UI labels these "assembly (group)".

Tier D — manual override

data/overrides.json, keyed "<pdf_page>:<part_no>:<instance_index>", holding a box in native-image pixel coordinates plus a note. Applied after everything else and always wins. This is the pragmatic escape hatch for the residual ~10–15 % — a small build-time review tool (section 18, Phase 5) renders every auto box so they can be eyeballed and corrected quickly.

If a label has no resolvable component at all

The record is kept with component_box = NULL. The app still shows the page and boxes the label in blue, stating "label located, component not resolved". The tool is never allowed to draw a red box it cannot justify.

Why not other approaches

  • Hough transform for leader lines — considered and rejected as the primary method: it finds line segments globally but does not tell you which end is the label end, handles kinked polylines badly, and still needs the component-lookup step. It remains a reasonable Tier-A alternative if pixel-walking proves fragile.
  • Training an object detector — needs annotated data that does not exist, and would be non-deterministic and unexplainable for a 30-page fixed document.
  • Full manual annotation of all 861 occurrences — reliable but ~15 hours of work and defeats the point. The tiered approach automates the ~85 % and leaves a reviewable remainder.

11. Data / schema design

SQLite, single file data/catalog.db. All geometry is stored in native diagram-image pixel coordinates (origin top-left of the extracted JPEG). PDF-page coordinates are derivable from pages.rect_* when needed; keeping one coordinate system avoids an entire class of bugs.

CREATE TABLE pages (
    pdf_page      INTEGER PRIMARY KEY,   -- 1-based, as the user sees it
    section_code  TEXT,                  -- '07', '13-1'
    section_de    TEXT,
    section_en    TEXT,
    models        TEXT,                  -- 'PU-ELA/PU-ELA-Z/...'
    img_xref      INTEGER,
    img_w         INTEGER,               -- native px, e.g. 1689
    img_h         INTEGER,               -- native px, e.g. 2352
    rect_x0       REAL, rect_y0 REAL,    -- placement on the PDF page, in points
    rect_x1       REAL, rect_y1 REAL,
    png_path      TEXT                   -- 'data/pages/p34.png'
);

CREATE TABLE parts (
    part_no       TEXT PRIMARY KEY,      -- '30642'
    name_de       TEXT,
    name_en       TEXT,
    name_fr       TEXT,
    is_malformed  INTEGER DEFAULT 0      -- the 6-digit '351159'
);

CREATE TABLE occurrences (               -- part listed in a page's table
    id            INTEGER PRIMARY KEY,
    part_no       TEXT REFERENCES parts(part_no),
    pdf_page      INTEGER REFERENCES pages(pdf_page),
    row_index     INTEGER,
    footnote      TEXT,                  -- '', '*', '**', '***'
    UNIQUE(part_no, pdf_page, row_index)
);

CREATE TABLE labels (                    -- callout found in the diagram raster
    id            INTEGER PRIMARY KEY,
    pdf_page      INTEGER REFERENCES pages(pdf_page),
    part_no       TEXT REFERENCES parts(part_no),
    instance      INTEGER,               -- 0,1,2... for repeated callouts on one page
    lx0 INTEGER, ly0 INTEGER, lx1 INTEGER, ly1 INTEGER,   -- label bbox, native px
    has_asterisk  INTEGER,               -- callout was drawn as '*30638'
    is_boxed      INTEGER,               -- callout enclosed in a rectangle -> assembly
    ocr_text      TEXT,                  -- raw, before normalization
    ocr_conf      REAL,
    match_kind    TEXT                   -- 'exact' | 'fuzzy1'
);

CREATE TABLE components (                -- resolved physical component
    label_id      INTEGER PRIMARY KEY REFERENCES labels(id),
    cx0 INTEGER, cy0 INTEGER, cx1 INTEGER, cy1 INTEGER,   -- may be NULL
    anchor_x      INTEGER, anchor_y INTEGER,              -- leader-line endpoint
    method        TEXT,   -- 'leader'|'anchor_radius'|'nearest'|'assembly'|'override'
    confidence    REAL,
    note          TEXT
);

CREATE INDEX ix_occ_part   ON occurrences(part_no);
CREATE INDEX ix_lab_part   ON labels(part_no);
CREATE INDEX ix_lab_page   ON labels(pdf_page);

data/overrides.json:

{
  "34:74506:1": { "box": [1001, 1448, 1106, 1540], "note": "was capturing the housing" }
}

data/build_report.json — per page: OCR token count, accepted/rejected labels, table parts with no label, labels with no table row, per-method counts. Written on every build; this is the project's own quality dashboard.

12. Proposed project folder structure 

ImageOCR/
├── 1.pdf
├── PSD.md                     <- this document
├── README.md
├── requirements.txt
├── app.py                     <- Streamlit entry point
├── spareparts/
│   ├── __init__.py
│   ├── config.py              <- paths, thresholds, page ranges (one place)
│   ├── pdf_loader.py          <- page classification, diagram extraction
│   ├── table_parser.py        <- parts tables -> parts / occurrences
│   ├── ocr.py                 <- RapidOCR wrapper + cache
│   ├── matcher.py             <- token normalization + snap to part list
│   ├── localizer.py           <- binarize, CC, leader-line trace, tiers A-D
│   ├── index_db.py            <- schema, writes, migrations
│   ├── lookup.py              <- runtime queries
│   ├── overlay.py             <- OpenCV drawing of boxes + captions
│   └── build_index.py         <- CLI: python -m spareparts.build_index
├── tools/
│   └── review.py              <- build-time QA: render every box, write overrides
├── data/                      <- generated, git-ignored except overrides.json
│   ├── catalog.db
│   ├── pages/p26.png ... p55.png
│   ├── ocr_cache/p26.json ... p55.json
│   ├── build_report.json
│   └── overrides.json         <- committed; it is hand-authored input
└── tests/
    ├── test_table_parser.py
    ├── test_matcher.py
    ├── test_localizer.py
    ├── test_index_integrity.py
    └── fixtures/
        ├── page26_expected.json
        ├── page34_expected.json
        └── gold_boxes.json    <- manual ground truth for evaluation

Flat, one package, two entry points. No src/ indirection, no plugin system, no service boundaries.

13. Dependencies

requirements.txt:

pymupdf>=1.24              # PDF parsing, table extraction, image extraction
rapidocr-onnxruntime>=1.3  # OCR (PP-OCRv4 via ONNX Runtime, CPU, no system deps)
opencv-python-headless>=4.9
numpy>=1.26
pillow>=10
streamlit>=1.30

Dev only: pytest.

Notes:

  • Python 3.11 (the interpreter present in this workspace; 3.11.9 verified).
  • opencv-python-headless rather than opencv-python: no GUI windows are ever opened, and headless avoids Qt/GTK baggage.
  • scipy is not required — cv2.connectedComponentsWithStats covers it.
  • No pandas: the data volumes are trivial and SQLite is the store.
  • Everything runs offline on CPU. No network calls at build or run time. Total install footprint is a few hundred MB, dominated by the ONNX models and OpenCV.
  • pymupdf version note: find_tables() prints a suggestion to install pymupdf-layout. It is not needed here — the plain extractor produced correct 4-column output on all 30 pages.

14. UI design

Single-page Streamlit app, no navigation, no login.

+----------------------------------------------------------------------+
|  BQ188 Spare-Parts Locator                                           |
|                                                                      |
|  Part number  [ 30642            v ]   [ Find ]                      |
|               (autocomplete over 705 known numbers)                  |
+----------------------------------------------------------------------+
|  30642 · Schnecke,kpl.                                               |
|  EN worm,cpl.        FR vis sans fin,cpl                             |
|  Section 07 — Motor-Antrieb / Motor-drive     PDF page 34            |
|  1 instance on this page          * located                          |
+----------------------------------------------------------------------+
|  [ page 34 ]                              <- tabs if part is on many |
|                                                                      |
|        +---------------------------------------------+               |
|        |                                             |               |
|        |      +---------+                            |               |
|        |      | #######<--- red box on the component |               |
|        |      +---------+                            |               |
|        |       30642  <-- caption drawn on the image |               |
|        |      +------+                               |               |
|        |      |30642 |<-- blue box on the callout    |               |
|        |      +------+                               |               |
|        +---------------------------------------------+               |
|                                                                      |
|  [ Download annotated PNG ]   [ ] show label boxes   [ ] show all    |
+----------------------------------------------------------------------+

Details:

  • Red rectangle, 4–6 px at native resolution, on the component. Blue thin rectangle on the printed callout label (toggleable, on by default) so the user can see why the tool chose that component. These two colours are never swapped.
  • The part number is drawn on the image next to the red box, on a white plate for legibility against line art.
  • Confidence badge next to the header, driven by components.method: located (leader) / approximate (anchor_radius, nearest) / assembly (group) / label only (no component box).
  • Multi-page parts (61 of them) → one Streamlit tab per page.
  • Multi-instance parts → every instance boxed and numbered #1, #2, … in the caption; a count is shown in the header.
  • "Show all parts on this page" checkbox renders every label on the page at once — genuinely useful for browsing, and it doubles as the QA view.
  • Unknown input → "No part xxxxx in this catalogue" plus the nearest 5 numbers by edit distance.
  • Images are served from the PNG cache; drawing is a few OpenCV calls, so response is effectively instant.

15. Error and edge cases

# Case Evidence Handling
E1 Part appears on several pages 61 parts; 73290 on 9 pages Return all; one UI tab per page
E2 Part appears several times on one page 74506 ×4 on p34 Index every instance; box all
E3 Part in the table but not drawn p39: 7 of 47 Show the page, box nothing, say "listed in table; not drawn on this diagram"
E4 Callout drawn but not in that page's table p40 73182, 73636, 74167 Accept via global part list; flag in build report
E5 Callout prefixed * *30638, *74012 Strip in normalization; store has_asterisk; show the footnote text
E6 Boxed assembly callout 31269, 33333, 34983 Tier C; badge "assembly (group)"
E7 Footnote rows in the table p42, p51, p52 (*, **, ***, blank) Filtered out of parts; text kept for E5 display
E8 Real parts listed after the footnote rows p52 30343–30346, p51 73013 Filter per row, never "stop at first footnote"
E9 Malformed part number in source p47 table 351159 vs diagram 35159 Stored with is_malformed; excluded from OCR snapping
E10 Off-page / decoy images p26 second image at y 802–1576; p29, p54 negative y Select by max intersection with page.rect
E11 Same image xref on two pages xref 152 on p36 and p37 Cache per page, never per xref
E12 OCR noise: Seite 18, 061101, D / PU-S-12 / …, 07 Every page Reject by header/footer strip + 5-digit rule + part-list snap
E13 OCR char errors: *7.4012, -74506, 4639, 7497 p34, p27, p48 re.sub(r'\D','') then exact/fuzzy-1 snap
E14 Leader line merges component with the frame p34 border component = 64 % of image Frame components (>25 % area) excluded; Tier B size cap
E15 Leader line crosses a big neighbour p34 74506/74903/79105 → housing Line-likeness check; >8 % area triggers Tier B
E16 Wrong nearest component wins p34 74025 captured 76352's bracket Line-likeness check; then override
E17 Callout with no leader line p40 Tier B "nearest"
E18 Dashed leader line p40 73158, 71058 Gap-tolerant walk in Tier A
E19 User searches a number in the manual pages (1–24) — Those pages are not indexed; report "not found"
E20 User enters non-numeric / wrong length — Validate ^\d{5}$; suggest nearest known numbers
E21 catalog.db missing or stale — App checks for the DB on start and prints the exact build command

16. Testing strategy

pytest, run against the real 1.pdf — it is fixed and committed, so it is the fixture.

Unit

  • test_table_parser.py — pages 26, 34 and 52 parsed against committed golden JSON: exact row counts, exact part numbers, correct footnote filtering (52), correct handling of real rows after footnotes (52). Asserts the global totals: 30 pages, 861 rows, 705 distinct numbers.
  • test_matcher.py — *7.4012 → 74012, -74506 → 74506, Seite 18 → rejected, 061101 → rejected, 07 → rejected; fuzzy-1 accepts a single candidate and rejects ambiguous ties.
  • test_localizer.py — on synthetic images (a label, a leader line, a target square, a decoy square): the target is chosen, the decoy is not; the frame guard fires; the size cap triggers Tier B; a dashed line is still followed.
  • test_pdf_loader.py — the diagram picked for pages 26, 29, 36, 37 and 54 is the on-page one, not a decoy or a shared xref (this pins E10 and E11).

Integration

  • test_index_integrity.py — over the built catalog.db:
    • every labels.part_no exists in parts
    • every occurrences.part_no exists in parts
    • every label's bbox lies inside its page's image bounds
    • every component box, when present, lies inside the image bounds and is smaller than the Tier-B cap
    • overall label recall stays ≥ 93 % and no page drops below 85 % — a regression guard on OCR settings, anchored to the Phase-1 measurement of 94.9 % / worst 85 %

Regression

  • test_known_parts.py — a handful of hand-verified pins that must never break: 30642 → page 34, section 07, Schnecke,kpl. / worm,cpl., component box overlapping the worm; 31269 → page 26, boxed assembly; 73290 → 9 pages.

Manual

  • tools/review.py renders every page with all boxes drawn into data/review/, which is how localization is actually judged (section 17) and how overrides.json gets written.

17. Accuracy and evaluation strategy

Three metrics, each measured against something real.

M1 — Label detection recall (automatic, no manual work). For each page: distinct part numbers OCR'd from the diagram ÷ part numbers in that page's table. Already measured in Phase 1 over all 30 pages: 817 / 861 = 94.9 %, 9 pages at 100 %, worst page 85 % (page 39). It over-counts misses, because some listed parts are genuinely not drawn, so it is a lower bound. Target: ≥ 95 % overall, ≥ 85 % on every page — i.e. do not regress from the Phase-1 baseline.

M2 — Label precision (automatic). Fraction of accepted labels that snap exactly to a known part number, and the count of rejected tokens per page (Phase-1 baseline: 69 rejects over 30 pages ≈ 2.3/page, nearly all of them known page furniture). Target: ≥ 98 % exact matches; < 3 rejected tokens/page.

M3 — Component localization accuracy (manual gold set). Build a ground-truth set of 80 (page, part, instance) triples across 8 pages chosen for varied difficulty — 26 (dense hardware), 34 (clean), 39 and 40 (very dense), 45, 49, 51 (accessories), 55 (nearly empty) — by hand-drawing the correct component box in tests/fixtures/gold_boxes.json using tools/review.py.

Scored two ways:

  • Strict: IoU(predicted, gold) ≥ 0.5.
  • Useful: the predicted box's centre lies inside the gold box and the predicted area is ≤ 3× the gold area. This is the metric that matters — a slightly loose box that unambiguously indicates the right part still answers the user's question.

Phase-1 prototype baseline on page 34, judged by eye: ~80 % correct, with the four failure modes catalogued in 4.9. Targets after Tier A/B are properly implemented:

Metric Target
M3 useful, Tier A+B automatic ≥ 85 %
M3 strict (IoU ≥ 0.5), automatic ≥ 70 %
After overrides.json review ≥ 97 % useful

Honesty requirement: the build report records the per-method distribution (leader / anchor_radius / nearest / assembly / override / none), and the UI shows the method for the box the user is looking at. The tool never presents a Tier-B guess as if it were a Tier-A result.

18. Implementation phases

Phase Deliverable Definition of done
1 (done) This PSD PDF inspected, structure documented, OCR and localization prototyped and measured
2 Index skeleton: config, pdf_loader, table_parser, index_db, build_index catalog.db holds 30 pages, 705 parts, 861 occurrences; 30 PNGs cached; test_table_parser and test_pdf_loader pass
3 ocr.py + matcher.py + OCR cache labels populated for all 30 pages; M1 ≥ 95 %; build report written
4 localizer.py Tiers A–C components populated; per-method counts in the report; no box exceeds the size cap
5 tools/review.py, gold set, overrides.json 80-box gold set exists; M3 measured; overrides written for the residual failures
6 app.py + overlay.py + lookup.py 30642 → page 34 with a red box on the worm; multi-page and multi-instance paths work
7 Polish: README, autocomplete, download, "show all", error states Fresh clone → pip install -r requirements.txt → python -m spareparts.build_index → streamlit run app.py works end to end

Phases 2–4 are the substance; 5 is what turns "mostly works" into "trustworthy"; 6–7 are a day's work each.

19. Explicitly excluded technologies and features

Excluded by requirement, and none of them would help here — the catalogue is a fixed, fully enumerable 705-row lookup table.

  • RAG in any form — no retrieval-augmented generation.
  • LangChain / LlamaIndex or any agent framework.
  • Embeddings, vector databases, semantic or fuzzy "meaning" search (FAISS, Chroma, Pinecone, pgvector, …). Lookup is an exact 5-digit key.
  • LLMs anywhere in the pipeline, including for OCR correction or description cleanup.
  • Cloud services — no OCR APIs, no storage buckets, no hosted inference. The tool runs offline.
  • Docker / Kubernetes / CI infrastructure. Single developer, single machine, one pip install. There is no deployment target that would justify it.
  • Microservices, REST APIs, message queues, background workers. The build is a script; the app is a Streamlit process.
  • Training or fine-tuning any model. No annotated data exists, and a fixed 30-page document does not warrant creating it.
  • A general-purpose PDF catalogue engine. This is scoped to 1.pdf and its exact layout. Generalization is explicitly a non-goal; the code should read as document-specific, with the document's quirks named in config.py.
  • User accounts, multi-tenancy, telemetry, persistence of user state.
  • PostgreSQL / MongoDB. SQLite holds ~1600 rows.
  • Indexing pages 1–24 (title, operation manual, needle table). They contain no part numbers. They may optionally be viewable, never searchable.

20. Known uncertainties and risks

# Risk Severity Evidence / mitigation
R1 Leader-line tracing is less reliable than the page-34 sample suggests. Denser pages (39, 40, 51 — up to 10.6 % ink, 61 parts) have crossing lines and touching parts; the "line touches only its target" property may break down. High — this is the project's central risk Measured on one page only (~80 %). Mitigation: Tier B size cap keeps failures conservative rather than wrong-and-confident; Tier D overrides; the UI always shows the label box so the user can self-correct. Validate on pages 39/40/51 first in Phase 4, before investing in Tier A refinements.
R2 Component boxes are systematically too large where the leader line crosses a big outline Medium Observed on p34 (74506, 74903, 79105). Line-likeness check in Tier A; 8 % area cap → Tier B
R3 Assembly (kpl.) callouts have no single correct box Medium ~9 seen on p26/p34/p40. Tier C draws a group box and the UI says "assembly"; accepted as inherently approximate
R4 The ~15 % residual needs manual review; a fresh reviewer may find it tedious Medium 861 occurrences total; the review tool renders a page at a time. Budget ~3–4 hours in Phase 5
R5 OCR misses on the densest pages Low Measured 94.9 % overall, worst page 39 at 40/47, and several of those 7 are genuinely undrawn parts
R6 RapidOCR model download on first run Low Models ship with the wheel; verified — the Phase-1 run needed no network. Pin the version in requirements.txt
R7 Binarization threshold 150 is tuned on one page Low JPEG artifacts vary slightly. Verified ink fraction 2.2–10.6 % across all 30 pages, all bilevel line art. Make the threshold a config.py constant and sanity-check the resulting component count per page in the build report
R8 Part numbers appearing on up to 9 pages could confuse a user Low UI shows one tab per page with the section title, so the user picks by assembly
R9 Source-data defects (351159, lsolierplatte, Kreuzsohnur) Low Displayed as-is; flagged, never silently "corrected". Descriptions are never used for matching
R10 pymupdf.find_tables() behaviour could change across versions Low Pin pymupdf; test_table_parser asserts exact golden output on 3 pages and the global 861/705 totals, so a regression fails loudly
R11 Build time (~10 min) is annoying during development Low OCR results cached to data/ocr_cache/; localization can be re-run alone. Stages are independently invocable

Open questions to settle during implementation

  1. Which Tier-A line-following implementation wins — pixel walk vs. probabilistic Hough restricted to a window around the label. Decide empirically on pages 39/40/51 in Phase 4; do not commit to either now.
  2. The right Tier-B fallback box size. "Median component bbox on the page" is a guess; tune it against the gold set.
  3. Whether boxed-assembly callouts should draw a union box or simply highlight the whole diagram region with an explanatory caption. Decide after seeing Tier C on real pages.
  4. Whether to expose the operation manual (pages 2–23) as browsable images in the UI. Nice-to-have, zero indexing cost, and out of scope until Phase 7.